Fundraising

Latest revision as of 13:32, 22 June 2012

The fundraising infrastructure is made up of quite a few different components to facilitate online user contributions and tracking/storing donor information. For a graphical diagram of the pipeline, see our architecture diagram.

Get information on working with Fundraising Software Development.

[edit] Payment processors

Currently, we have the ability to use three different payment processors for online donations:

Global Collect: http://www.globalcollect.com/
- Global Collect has the ability to handle payments from multiple international systems including: credit card, direct debit, real time bank transfer, eWallets and more.
PayPal donations: Website Payments Pro Payflow Edition (via PayPal, documentation: https://www.x.com/community/ppx/documentation#wpp)
- We get notified about payments from Website Payments Pro via PayPal's IPN service (documentation: https://cms.paypal.com/cms_content/US/en_US/files/developer/IPNGuide.pdf)
- The messages sent via PayPal's IPN service get sent to our IPN listener
Credit Card donations: PayflowPro (via PayPal, documentation: https://www.x.com/community/ppx/documentation#pfp)

Note: In general, note that the PayPal documentation tends to be incorrect, out of date, etc.

[edit] ActiveMQ

We are using the ActiveMQ (http://activemq.apache.org/) message broker/queueing system to keep track of contributions from the point of submission to the point of getting inserted into CiviCRM. When a donation is registered, it is placed into the queueing system. This allows for any verification of that transaction that is needed (e.g. checking the validity of a transaction with PayflowPro) to be handled before we consume the transaction details. The queue also ensures that donations are eventually recorded even if Drupal or CiviCRM go down.

The way that all of the working parts communicate with the ActiveMQ server is via Stomp - a simple messaging protocol (http://activemq.apache.org/stomp.html), the library for which is included with the different pieces of the fundraising system that require it.

To pull transactions from the queue, go to the root directory for the civicrm site and run "drush --user=1 qc". This runs the drush script queue_consume.drush.inc that feeds the transactions to the queue2civicrm Drupal module.

[edit] Jenkins

We are using Jenkins as a cron-replacement for handling some of the automation of certain fundraising tasks. For further internal documentation, see Fundraising.wikimedia.org#Hudson.

To log into Jenkins go to http://fundraising.wikimedia.org:8080/

[edit] Contribution tracking

When a potential donor visits the Wikimedia donation page, a tracking record is created in the drupal.contribution_tracking table. This record includes the user's language, referrer, donation comment, opt-out status, a timestamp, and various other data. The tracking is handled on the MediaWiki side by the DonationInterface and ContributionTracking extensions. If the user makes a successful donation, a contribution record is passed to CiviCRM via ActiveMQ. The queue2civicrm module then inserts the contribution record into the CiviCRM database and updates the tracking record with the id given to the contribution by CiviCRM.

[edit] Stats

[edit] Banner Impression/Landing Page Stats Collection

Banner impressions and landing page stats are collected from the production proxies. Fundraising_Analytics/Impression_Stats

[edit] Reports

Contribution tracking data is surfaced through a stats portal on foundation wiki. All entries have to be white listed under wgAllowedTemplates in CommonSettings.php so that we don't surface any doctored banners.

[edit] CiviCRM modules

The CiviCRM modules live on grosley in /srv/org.wikimedia.civicrm/sites/all/modules. To manually run the cron for the modules, go to https://civicrm-dev.wikimedia.org/cron.php.

[edit] queue2civicrm

This module pulls online contributions from the ActiveMQ queue 'donations' and writes them to the CiviCRM database via the CiviCRM API. It is activated by a periodic Hudson job, which fires an included Drush command (found in queue2civicrm/queue_consume.drush.inc) to replace the former cron method. Output from the execution is captured in the Hudson console logs.

Queue2civicrm also contains a hook (queue2civicrm_import) which is invoked at the end of a singular transaction, passing transactional information on to other modules that will perform other actions on the transaction. The following modules make use of this hook:

It also contains a hook which is invoked at the end of processing a batch of messages, which allows for other modules to perform batch processing. This hook is made use of in the following modules:

recurring

The settings for queue2civicrm can be configured at http://<web root for drupal>/admin/settings/queue2civicrm. You can configure the connection information to ActiveMQ, what queue to consume from and the number of messages to consume at once. Also, you can test the ActiveMQ connection and test pushing messages into the queue.

Formerly, this module interacted directly with the PayflowPro gateway to validate transactions and passed queued items ready for consumption to a custom API for inserting them into CiviCRM. This module no longer performs any gateway-specific actions and uses the built-in CiviCRM API for inserting contributions.

[edit] Recurring

The recurring module is a sub-module to queue2civicrm and shares much of the functionality. As is it is right now, it is built particularly to handle raw IPN messages sent from PayPal pertaining to recurring (or in PayPal's language, 'subscription') contributions. Because CiviCRM does not have an API for recurring contributions and due to the immaturity of some aspects of the Civi APIs, this module performs a lot of custom queries to both the CiviCRM and Drupal databases.

This module can be configured form the queue2civicrm configuration.

It processes messages out of its own queue (as configured in the interface). Queue consumption is invoked by making use of the hook at the end of batch processing in queue2civicrm.

[edit] Execution

There are a number of ways to execute queue consumption with queue2civicrm and queue2civicrm/recurring. The easiest way is with Drush. From the Drupal root,

$ drush --user=1 qc

This executes the queue2civicrm Drush script, located in queue2civicrm/queue_consume.drush.inc

[edit] thank_you

This module sends a thank you message to online donors. It is called via a drupal hook from the queue2civicrm module every time a contribution is successfully recorded in the database. The contents of the thank you message are set from Administer » Site configuration » Thank you settings in CiviCRM.

[edit] exchange_rates

This module handles converting foreign currencies into US Dollars. It consists of two major components. One function handles doing the conversions based on exchange rates stored in the drupal.exchange_rates table. This function is called from the queue2civicrm module when it is processing donations from the queue. Another function, which is activated by a cron job, looks up the exchange rates on the web and updates the exchange_rates table. The exchange rates are currently pulled from oanda.com, with http://www.ecb.int/stats/eurofxref/eurofxref-daily.xml used as a back-up source.

[edit] wmf_premiums

NOTE: We never did implement premiums and have no plans to do so in the future

This is a sub-module of queue2civicrm to handle recording data for premiums (such as t-shirts or stickers) that are offered to donors. It utilizes the queue2civicrm_import hook to write premium order information to a table called civicrm.premiums. This table records the donor's tracking_id, contact_id, contribution_id, order information, shipping address, and donation amount (to double-check against fraud). This data can then be periodically exported to a CSV for fulfillment.

[edit] wmf_owa

NOTE: OWA is no longer used for analytics

This is a module for civi that allows Open Web Analytics integration. When the civi module consumes approved transactions from ActiveMQ, the wmf_owa module sends an API call to the OWA cluster with the OWA session associated with the transaction (via the contribution_tracking table) and with the donation amount. Also, on update to the transaction (in the case of refund) the same OWA session is updated with another transaction, but with a negative value.

[edit] Mediawiki extensions

The following Mediawiki extensions related to fundraising are installed on the payments wiki:

[edit] DonationInterface

This module handles all of the payment gateway specific functionality for the fundraising system at the point the user makes a donation. Other handling (secondary verification, queue managing) is handled on Fundraising.wikimedia.org. DonationInterface It is made up of the following components:

[edit] activemq_stomp

Contains the Stomp library and provides a mechanism for correctly formatting a message for storage in ActiveMQ. This could probably be refactored a bit to allow the other extension pieces to more flexibly rely on this to communicate with ActiveMQ. As it is right now, the functionality exposed by activemq_stomp.php is very limited, however the included Stomp library exposes whatever you might need to communicate with the queue.

[edit] extras

This directory contains several different hooks that can be used as needed at various points in a gateway's donation workflow. Extras include conversion_log, custom_filters (used for fraud prevention), minfraud (which also exists as a custom filter), and recaptcha.

[edit] gateway_common

Generally, the gateway_common directory contains gateway adapter code written to be gateway-agnostic, but which all the gateway-specific classes are descended from.

[edit] gateway_forms

Forms and form classes that are not written to be gateway-specific should be kept here.

[edit] globalcollect_gateway

Everything specific to the globalcollect gateway adapter can be found here. The globalcollect gateway adapter interfaces with Global Collect to make donations in many different payment types and currencies.

[edit] scripts

In the course of a normal globalcollect hosted credit card transaction, a small percentage of donors will complete a transaction on the globalcollect hosted credit card form, and not manage to come back to the globalcollect ResultSwitcher page. The function of the ResultSwitcher page is to both filter and finalize the pending transaction and record it in the case of a successful conversion, so the donor not returning to the ResultSwitcher would effectively strand the otherwise successful transaction in a sort of limbo state. The code contained in the globalcollect scripts directory was written to address the problem of these stranded transactions.

This maintenance script is currently running on a Jenkins job, and uses data pulled from the 'cc-limbo' queue in ActiveMQ.

[edit] modules

Generally speaking, all modules that could be said to be gateway-agnostic should live in this directory. These modules will be loaded as-needed with resource loader.

[edit] payflowpro_gateway

This exposes the credit card donation form (as opposed to the PayPal stuff) and handles the communication between the donation page and PayflowPro. It negotiates the entire process from the point the user submits the donation to insertion of a donation into ActiveMQ - if a donation is considered valid at the point of submission, it is placed in the 'donation' queue. Otherwise, it is either rejected or placed in the 'pending_pfp' queue for review. Further handling happens on pending messages by Fundraising.wikimedia.org#PayflowPro_Pending_Transaction_Verification.

[edit] Fraud filtering

There are a series of extra sub-extensions, or filters, for payflowpro_gateway that perform analysis on credit card transactions to determine the likelihood that a transaction is fraudulent. Each of the filters helps determine the 'risk score' for a transaction. Actions to take based on certain risk scores can be configured for payflowpro_gateway (reject, review, challenge, accept). The filters currently available include:

MaxMind/MinFraud - a third party solution that helps analyze the transaction. They return their own 'risk score' for a transaction which heavily influences our own internal scoring.
Referrer - Regular expressions can be configured to be run on a transaction's 'referrer', and each regex can be configured to apply a different score in the event that the referrer is a match.
utm_source - Same as referrer, but for the utm_source bit in the tracking fields.

[edit] Logging

Our configuration of payflowpro_gateway causes two local logfiles to be generated:

/var/log/mw/minfraud
/var/log/mw/payflow

The 'minfraud' log logs more than just 'minfraud' related information. It tracks a user's transaction through our fraud prevention filters, and if their transaction is considered safe enough to be sent off to PayflowPro, Payflow's response object also gets logged here. This log also gets used to help us determine patterns in fraudulent transactions, analyze more general patterns in our transactions and determine why or why not a certain transaction was accepted/rejected.

The 'payflow' log logs more than just payflow related information. It's more of a debug log, tracking a transactions progress through the different parts of the payflowpro_gateway experience. The only piece of transactional data that actually gets recorded in the log is our internal 'trxn_id', which is considered the 'invoice id' by PayPal/Payflow (this makes it possible to line up transactional information either later on our end [eg from CiviCRM] or from the PayPal manager with the information contained in this log). We primarily use this log to keep track of timing information - how long it takes to communicate with the various third party services (MaxMind and PayflowPro), timeouts, retries, etc.

It is important to note that for our purposes, these log files actually get aggregated and later archived - more details can be found on the documentation for the payments cluster.

[edit] paypal_gateway

Handles the redirect from the Wikimedia site to PayPal for donations.

We are implementing a stand-alone listener to talk to PayPal's IPN and push pending PayPal transactions into the proper queue (see Fundraising.wikimedia.org#PayPal_IPN_Listener).

[edit] ContributionTracking

[edit] VariablePage

This extension makes it possible to provide a user-facing link that will direct the user to n possible URLs x% of the time. For instance, this is useful for doing randomized A/B testing - you can configure the module to send 50% of clicks to one URL, and the other 50% to another URL. The percentages are completely configurable as are the URLs. This was used extensively to make the 'Donate' sidebar link on Wikipedia point to a number of different donation landing pages for testing purposes.

[edit] Public reporting

Real-time donor comments: http://wikimediafoundation.org/wiki/Special:ContributionHistory
Contribution statistics: http://wikimediafoundation.org/wiki/Special:ContributionStatistics

The public reporting data is stored in the civicrm.public_reporting table on db9. This data gets replicated to the slave database, db10, which is where the public reporting pages actually pull the data from.

The code that manages the public reporting data lives in /srv/org.wikimedia.civicrm/sites/all/bin/public_reporting on the grosley server (donate.wikimedia.org).

table.sql - Creates the public_reporting table
trigger.sql - Sets up the database triggers for updating the table
synchronize.sql - Can be used to manually sync the public reporting data with the existing contribution records

The reporting pages are created through the ContributionReporting mediawiki exntension

[edit] Miscellaneous Scripts

There are some miscellaneous scripts to help with things like Paypal Verification, queue handling, etc. Details of which can be found on Fundraising.wikimedia.org.

[edit] High-level Overview of Donation Pipeline

Click the images for further explanation.

From the landing page to ActiveMQ
Inserting to queues in ActiveMQ to CiviCRM

@@ Line 30: / Line 30: @@
 ===Stats===
+====Banner Impression/Landing Page Stats Collection====
+Banner impressions and landing page stats are collected from the production proxies. [[Fundraising_Analytics/Impression_Stats]]
+====Reports====
 Contribution tracking data is surfaced through a [http://wikimediafoundation.org/wiki/Special:ContributionTrackingStatistics stats portal] on foundation wiki. All entries have to be white listed under '''wgAllowedTemplates''' in '''CommonSettings.php''' so that we don't surface any doctored banners.