Mwlib

Revision as of 17:46, 1 May 2012

Notes

Notes on building mwlib 'offline collection generator' hosts . . .

ocg1 installed - testing cherrypicked-to-production puppet work here
ocg2 installed and are awaiting puppetization
ocg3 has a firmware issue that needs resolution

http://mwlib.readthedocs.org/en/latest/renderserver.html
http://mwlib.readthedocs.org/en/latest/installation.html#ubuntu-install
http://hynek.me/articles/python-deployment-anti-patterns/

Ref. http://wiki.debian.org/Python

• use site-packages?
• use dh_python2?

Should these be isolated from stock python packages, i.e. in an alternate library path, etc?
Misc

• debuild -us -uc
• dpkg-buildpackage -rfakeroot -b

virtualenv

this may get a lot easier if we wait until precise is out

Labs pediapress project

bob - build machine, ip 10.4.0.90
pediapress-ocg1 - ubuntu precise test host
pediapress-ocg2 - ubuntu lucid test host
pediapress-packager - jgreen working with packaging tools/process

To Be Determined

1) daemonizing: runit, init, upstart?

2) package individual modules or entire homedir heap as a single package?

3) depending on #2, install modules system-wide or under a user?

4) other user and filesystem layout specifics not covered in 1-3

Build process

puppet does this already

 apt-get install -y dvipng g++ gcc git-core imagemagick libfreetype6-dev libjpeg-dev liblcms-dev \
    libxml2-dev libxslt-dev libz-dev make ocaml-nox pdftk ploticus python python-dev python-docutils \
    python-greenlet python-imaging python-lxml python-pyparsing python-pypdf python-virtualenv \
    texlive-latex-recommended
 groupadd pp
 useradd -m -c "pediapress" -g pp -d /opt/pp pp

this part needs puppetification

 su - pp
 virtualenv --distribute ~/local; export PATH=~/local/bin:$PATH; hash -r; export PIP_INDEX_URL=http://pypi.pediapress.com/simple/
 pip install pil       # provided by python-imaging
 pip install pyfribidi # no additional dependencies
 pip install mwlib     # pulls odfpy-0.9.4, gevent-1.0b2, timelib-0.2.3, bottle-0.10.9, apipkg-1.0, qserve-0.2.7, roman-1.4.0, py-1.4.7, sqlite3dbm-0.1.4
                       # note python-docutils includes roman.py
 pip install mwlib.rl  # pulls mwlib.ext
 pip install mwlib.zim # pulls pyzim
 set up daemons

Packages to Build

Stock Packages That Should Work

apipkg-1.0
 ubuntu: python-apipkg (lucid:n/a precise:1.0-1.1build1)
bottle-0.10.9
 ubuntu: python-bottle (lucid:0.6.4-1 precise:0.10.6-1)
 source package: git://git.debian.org/collab-maint/python-bottle.git
 http://bottlepy.org/docs/0.10/changelog.html
 stock package looks tricky to backport due to debian/control build dependencies
pygments-1.5
 ubuntu: python-pygments (lucid:1.2.2+dfsg-1ubuntu1 precise:1.4+dfsg-2)
 http://pygments.org/docs/changelog/#version-1-5
 source: http://pypi.python.org/packages/source/P/Pygments/Pygments-1.5.tar.gz
 on precise, it's pulled in as a python-docutils dependency
c-ares
 ubuntu: libc-ares-dev (lucid:1.7.0-1-1 precise:1.7.5-1)
 ubuntu: libc-ares2 (lucid:1.7.0-1 precise:1.7.5-1)
libev
 ubuntu: libev-dev (lucid:1:3.8-1 precise:1:4.11-1)
 ubuntu: libev3 (lucid:1:3.8-1)
 ubuntu: libev4 (precise:1:4.11-1)
reportlab
 ubuntu: python-reportlab (lucid:2.4-1 precise:2.5-1.1build1)
reportlab-accel
 ubuntu: python-reportlab-accel (lucid:2.4-1 precise:2.5-1.1build1)
cython
 ubuntu: cython (lucid:0.11.2-1 precise:0.15.1-1ubuntu1)
python-lxml
 ubuntu: python-lxml (lucid:2.2.4-1 precise:2.3.2-1)
cxxtools
 ubuntu: libcxxtools-dev (lucid:1.4.8-3 precise:2.0-4ubuntu2)
tntdb
 ubuntu: libtntdb-dev (lucid:1.0.1-3build1 precise:1.1-1)
xz
 ubuntu: (?) xz-utils (lucid:4.999.9beta+20091116-1 precise:5.1.1alpha+20110809-3)

New Packages That Build Easily

odfpy-0.9.4
 * built for lucid & precise
py-1.4.7
 * built for lucid & precise
qserve-0.2.7
 * built for lucid & precise
sqlite3dbm-0.1.4
 * built for lucid & precise
pyfribidi-0.11.0
 ubuntu: python-pyfribidi (lucid:0.10.0-2 precise:0.10.0-4), both patched for buffer overflow
 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=663189
 http://pypi.python.org/pypi/pyfribidi/
 upstream appears dead, pediapress released 0.11.0 but changelog ends at 0.10.1
 comment: 0.11 and 0.10.1 have the same API. 0.10.0 won't work and is still broken on ubuntu precise
 notes: replace stock package? package under different name? remove fribidi-src before packaging

New Packages That Require Significant Refactoring Before Build

pyzim
 source: http://www.openzim.org/Releases
 http://pypi.python.org/pypi/pyzim
 build requires: cxxtools =2.0, tntdb =1.1 (for zimwriter), tntnet =2.0 (for zimreader), libxz-dev, autoconf, automake, libtool
 uses a Makefile to build cxxtools, tntdb, xz, zimlib and zimwriter so you'll want remove the vendor directory
 recommend: try to build against stock cxxtools/tntdb on precise
zimlib
zimwriter

Mess

gevent-1.0b2
 ubuntu: python-gevent (lucid:n/a precise:0.13.6-1ubuntu1)
 stable: gevent-0.13.7
 beta: gevent-1.0b2 requires greenlet, libev & c-ares, release tarball includes dependencies
 significant changes between 0.13.7 and 1.0b2
 remove included libev and c-ares
 http://packages.debian.org/experimental/python/python-gevent
 recommend: create python-pediapress-gevent from 1.0b2 source
mwlib-0.13.6
 "mwlib itself also includes a slightly patched apipkg (v1.1) module. I think I already
 got the changes merged upstream. mwlib also includes a patched EasyTimeline.pl script."
mwlib.ext-0.12.4
 provides external dependencies needed by the mwlib library
 essentially reportlab 2.5 with some patches
 http://pypi.python.org/pypi/mwlib.ext/0.12.4 (no Makefile)
mwlib.rl-0.12.10
 check with Pediapress re. font install
mwlib.zim-0.2.0
 requires cython, lxml, mwlib, pyzim (pyzim requires zimlib, zimwriter)
 http://pypi.python.org/packages/source/m/mwlib.zim/mwlib.zim-0.2.0.zip (no Makefile)
 recommend: either build as monolithic heap, or attempt to fulfill all requirements on precise
timelib-0.2.3
 ./ext-date-lib is an import of php-5.2.10/ext/date/lib for use in static linking
 "timelib includes unmodified sources from php. you should also break those out
 and link against libphp (if there such a thing in ubuntu)."
 * first attempt package failed, won't build as fakeroot
 recommend: rewrite? dynamic link against stock php? build with current php
texvc
 - pediapress way
   git clone https://github.com/pediapress/texvc
   cd texvc; make; make install PREFIX=~/local
 - mediawiki way (?)
   http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/Math/math
   version #? use mediawiki version #? what?
   is debian/copyright sane?
   is version sane for pediapress folks?
 recommend: consult mw and pediapress devs for instructions on what to package