View source for Grosley-Aluminium-Migration

<pre>
as of Tue Nov  1 16:19:22 EDT 2011

Building Aluminium

1. base+puppet install

2. then a bunch of stuff that hasn't been puppetized
svn co http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraising-civicrm/d620c34 /srv/org.wikimedia.civicrm
svn co http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-civicrm /srv/org.wikimedia.civicrm-dev
svn co http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraising-civicrm/d620c34 /srv/org.wikimedia.civicrm-stage
svn co http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraising-misc /opt/fundraising-misc
mkdir /srv/org.wikimedia.fundraising
puppet --test
svn co http://svn.wikimedia.org/svnroot/wikimedia/vendors/drush/current /opt/drush
chmod a+x /opt/drush/drush
ln -s /opt/drush/drush /usr/local/bin/drush
svn co http://svn.wikimedia.org/svnroot/wikimedia/vendors/django/current/ /usr/local/src/django_src
cd /usr/local/src/django_src
python setup.py install

3. copy over faulkner's django reporting stuff [document, package, put in svn . . . something]

4. some form of python/django config [document!]

fetch httpagentparser-0.8.2.tar.gz
cd /usr/locals/src; tar xzf httpagentparser-0.8.2.tar.gz; cd httpagentparser-0.8.2; python setup.py install


Cutover Prep

1) finish config cleanup [jgreen]
  - stuff that's installed from svn--most is under /srv
  - /etc/php5/apache2/php.ini - puppetize from grosley copy **DONE**
  - /usr/sbin/gmond missing on Aluminium. puppetized and check ganglia. **DONE**
  - dpkg --get-selections double-check **DONE**

2) mail. puppet config dealt with outbound. make sure any inbound mail works. [jgreen] **DONE**

3) jenkins. copy over jobs from grosley to aluminium. strategy for making one active [jgreen, arichards]
  - consider keeping jobs in svn? http://jenkins-ci.org/content/keeping-your-configuration-and-data-subversion
  - jenkins cron jobs: 
   0 12 * * * /root/jenkins.tarring.sh
   */5 * * * * /usr/local/scripts/jenkins_watcher

4) dump storage3 faulkner database and take offline [jgreen]
  - disable Faulkner's automatic analytic scripts by disabling cronjob [rfaulk] **DONE**
  - copy user-mysql permissions from storage3 to db1008
     * dumped to storage3:/archive/backups/20111103-storage3-mysql-mysql.sql.gz
  - flush/lock and dump faulkner database to storage3:/archive/backups/20111103-storage3-mysql-faulkner.sql.gz **DONE**
  - stop mysql on storage3 **DONE**
  - tar/gz storage3 database to storage3:/archive/backups/20111103-storage3-mysql.sqldata.tgz **DONE**

5) prep db1008 to be master for faulkner db, otrs-free, but still middle-master for civi/fundraising 
  - configure db1008 to skip slaving of faulkner, otrs db's from db9 and restart **DONE**
  - drop otrs db from db1008 **DONE**
  - restore faulkner db to db1008 **DONE**

6) reconfigure storage3 as slave of db1008
  - apt-get upgrade and reboot storage3 (punted on dist-upgrade) **DONE**
  - update storage3 to the -facebook mysql version, configure like db1025 **DONE**
  - use db1025 to re-image storage3 **DONE**

7) move faulkner's scripts and analytics to aluminium+db1008/db1025 (dns switch, cron jobs, etc) [jgreen, rfaulk]


Cutover Event

1) stop everything on grosley/aluminium that uses db9
   - apache **DONE**
   - jenkins **DONE**

2) stop slave and remove slave settings on db1008 so it becomes fundraising master
   - STOP SLAVE; **DONE**
   - CHANGE MASTER TO MASTER_HOST=''; **DONE**

3) discover that with the puppet switch, we picked up skip-name-resolve and must redo mysql auth IP-based **DONE**

4) hit mysql bad behavior around privileges tables, attempt fetch from db9 + upgrade and fail, rebuild privileges table from scratch **DONE**

  GRANT LOCK TABLES,SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES ON `civicrm`.* TO 'civicrm'@'blah' IDENTIFIED BY 'bleh';
  ERROR 1133 (42000): Can't find any matching row in the user table

5) discover there was stuff on payments* that uses fundraising db's on db9, reconfigure **DONE**
  - /srv/www/org/wikimedia/payments/.LocalSettingsNoBackup.php

6) discover there was stuff on prod that uses fundraising db's on db9, reconfigure **DONE**
  - /home/wikipedia/common/wmf-config/contribution-tracking-setup.php

7) flip various DNS to point to aluminium
  - web hosts **DONE**
  - donate.wikimedia.org mx record [jgreen] **DONE**

8) reconfigure everything on grosley/aluminium that points to db9/db10 to use instead db1008/db1025 [jgreen/arthur]
   - /srv/org.wikimedia.civicrm/sites/default/settings.php **DONE**
   - /srv/org.wikimedia.civicrm/sites/default/civicrm.settings.php **DONE*
   - /srv/org.wikimedia.civicrm-dev/sites/default/settings.php **DONE**
   - /srv/org.wikimedia.civicrm-dev/sites/default/civicrm.settings.php **DONE**
   - /opt/fundraising-misc/public_reporting/update.php **DONE**
   - /opt/fundraising-misc/queue_handling/paypal/IPN/IPNListener_Recurring.php **DONE**
   - /opt/fundraising-misc/queue_handling/paypal/IPN/IPNListener_Standalone.php **DONE**

9) reconfigure/modify faulkner's analytics stuff to use new master/slave db's [faulkner]
   - /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/config/settings.py
   - /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/scripts/settings.py (settings file for shell scripts)
   - /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/web_reporting/settings.py (no references to dbs but a config file not stored in svn)
   - /srv/org.wikimedia.fundraising-analytics/classes/DataLoader.py (references to storage3 appear to be isolated in code comments)

   - creation of /srv/org.wikimedia.fundraising-analytics/mplconfigdir with full access rights -> python matplotlib module dependency (part of env setup)
   - /srv/org.wikimedia.fundraising-analytics/Fundraiser_Tools/web_reporting/static/images/ full access rights -> data image files are written here (part of env setup)
   - /srv/org.wikimedia.fundraising-analytics/fundraiser-analysis -- checkout http://svn.wikimedia.org/svnroot/wikimedia/branches/deployment/fundraiser-analysis/ here

10) restart services on aluminium with new settings **DONE**

11) test test test fix test test fix test fix test


Post-Cutover Cleanup

1) get storage3 my.cnf into puppet and reenable puppet on storage3

2) clean up puppetization of my.cnf for db1008 **DONE**

3) fix the config mess for /opt/fundraising-misc/public_reporting/update.php

4) Check out stomp library from svn into /opt/stomp (svn co http://svn.wikimedia.org/svnroot/wikimedia/vendors/stomp_php/pre-1.0.0 /opt/stomp)

5) puppetize grosley:/etc/php5/cli/php.ini

6) figure out who still used storage3, get their privileges back into db1025/storage3

7) restore stored procedures to mysql.civicrm **DONE**

  http://svn.wikimedia.org/viewvc/wikimedia/branches/deployment/fundraising-civicrm/d620c34/sites/all/bin/public_reporting/triggers.sql?revision=161&view=co

8) get faulkner's js libraries and configs into puppet, svn, whatever

</pre>