Revision as of 17:46, 2 March 2012

Usage

lucene-search is a search extension for MediaWiki based on the "Apache Lucene" search engine. This page attempts to give some information about the extension and how it is set up in the WikiMedia cluster, and to give details about the Lucene search engine.

Overview

Software

The system has two major software components, Extension:MWSearch and lsearchd. The configuration can be found at http://noc.wikimedia.org/conf/lsearch-global-2.1.conf

The version of Lucene is 2.1 and jdk is sun-j2sdk1.6_1.6.0+update30 .

Extension:MWSearch

Extension:MWSearch is a MW extension that overrides default search backend and send requests to lsearchd.

lsearchd

lsearchd (Extension:Lucene-search) is a versatile java daemon that can act as frontend, backend, searcher, indexer, highlighter, spellchecker, ... we use it to searches, highlight, spell-checks and act as an incremental indexer

Essentials

• configuration files:
  • /etc/lsearch.conf - per-host local configuration
  • /home/wikipedia/conf/lucene/lsearch-global-2.1.conf - cluster-wide shared configuration
• started via /etc/init.d/lsearchd
• search frontent port 8123, index frontend port 8321; backend - RMI (RMI registry port 1099)
• logs in /a/search/logs
• indexes in /a/search/indexes
• jar in /a/search/lucene-search
• test with curl http://localhost:8123/search/enwiki/test

Installation

Scripts in /home/rainman/build:

• build - run on searchidx2 to build jar from sources
• deploy - run on target host to make directory structure, deploy jar, copy config template and start lsearchd
• deploy-jar - run on target host to only update jar, start lsearchd

Configuration

There is a shared configuration file /home/wikipedia/conf/lucene/lsearch-global-2.1.conf that contains information about the roles hosts are assigned in the search cluster. This way lsearchd daemons can communicate with each other to obtain the latest index versions, forward request if necessary, search over many hosts if the index is split, etc..

The per-host local configuration file is at /etc/lsearch.conf. Most importantly it defines SearcherPool.size, which should be set to local number of CPUs+1 if only one index is searched. This prevents CPUs from locking each other out. The other important property is Search.updatedelay which prevents all searches from trying to update their working copies of the index at the same time, and thus generate noticeable performance degradation.

Indexing

• searchidx2 serves as the indexer for the cluster
• searchidx2's lsearchd daemon is configured to act as indexer
• other indexing jobs, like indexing private wikis, spell-check rebuilds etc are in rainman's crontab on searchidx2
• searchidx2 runs rsyncd to allow cluster members to fetch indexes
• other cluster hosts fetch indexes by rsync every (what is the interval?)

Search Cluster: Shards, Pools, and Load Balancing Oh My!

This section has been derived from the following configuration:

• /home/wikipedia/common/wmf-config/lucene.php
• /home/wikipedia/conf/lucene/lsearch-global-2.1.conf
• /home/wikipedia/conf/pybal/pmtpa/search_pool[1-3]

Index Sharding

We shard search indexes across hosts in the cluster to accomodate index data footprint, hardware limitations, and utilization. As of Feb 2012, indexes were blatted across the cluster like so: see SearchShards.

Pools

We use a mixture of single-host and multi-host pools to direct requests to the servers that host the appropriate indexes. Where multi-hosts pools are employed we use pybal/LVS load balancing (running on lvs3) or in-code load balancing. As of Feb 2012 we have the following pool configuration:

Administration

Dependencies

 [content needed]

Health/Activity Monitoring

 [content needed]

Software Updates

The following script will build the latest version of lucene-search and deploy it to all searchers:

 /home/rainman/salsa

 (sync-all-lucene-search)

Stopping and fall back to MediaWiki's search

To disable lucene and fall back to MediaWiki's search, set $wgUseLuceneSearch = false in CommonSettings.php.

Adding new wikis

When a new wiki is created an initial index build needs to be made. First restart the indexer on searchidx2 to make sure the indexer knows about the new wikis, and then run the build-new script on appropriate wiki database name (i.e. replace wikidb with the wiki database name, e.g. wikimania2012wiki).

Run on searchidx2 as user rainman:

 root@searchidx2:~# sudo -u rainman nohup /home/rainman/scripts/search-restart-indexer
 root@searchidx2:~# sudo -u rainman /home/rainman/scripts/build-new wikidb

New install of searchidx1

To set up a new indexer we need:

• /etc/rsyncd.conf, /etc/lsearch.conf, and /etc/default/rsync will all be pushed out via puppet.
• the contents of /a/search copied over. this has a local copy of sun-java. sun java is required (according to rainman) as open-jdk can corrupt indexes. may need to recompile sun-java.
• /home, /mnt/thumbs, and /mnt/upload6 mounted and added to /etc/fstab. all the scripts run out of rainman's home dir so he should have an account too.
• (probably no longer the case) /home/wikipedia/conf/lucene/lsearch-global-2.1.conf may need updating
• steal the crontab for rainman.
• ( probably no longer the case due to /home/ all being on nfs) copy over (keeping perms etc) the stuff in /home/ariel/searchidx on fenari. The bad thing about this stuff is that the most recent of any two subdirs in import/* or snapshot/* are really supposed to be hardlinks to the same dir in /a/search/indexes/index, so that needs to be fixed up after the copy. Before going through this see if we can get by without that, ask rainman what he thinks. We need the files no matter what though.

Once the service is up and running the lsearchds on the other search hosts will all have to be restarted. (They run from the script in /home so they will automagically be aware of the new host.)

Trouble

Main indexer on searchidx2 is stuck

Run this script this script as user rainman (so he can restart later if necessary):

 root@searchidx2:~# sudo -u rainman /home/rainman/scripts/search-restart-indexer

Individual lsearchd processes are crashing or nonresponsive

• Try starting the lsearch process in the foreground so you can watch what it does:

 start-stop-daemon --start --user lsearch --chuid lsearch --pidfile /var/run/lsearchd.pid --make-pidfile --exec /usr/bin/java -- -Xmx3000m -Djava.rmi.server.codebase=file:///a/search/lucene-search/LuceneSearch.jar -Djava.rmi.server.hostname=$HOSTNAME -jar /a/search/lucene-search/LuceneSearch.jar

• Check log at /a/search/log/log for indications of obvious issues

  root@search3:~# grep "^Caused by" /a/search/log/log|tail -20
  Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
  Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
  Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

  (oops, we hit java's memory limit)

Space Issues on Indexer Host

Probably deprecated b/c searchidx1 is no longer in use.

The primary indexer searchidx1 has been low on space. To help it limp along you can do:

 cd /a/search/log/
 rm -rf log-all log-prefix
 cd /a/search/indexes/import/
 rm -rf *.spell 
 rm -rf *.prefix (citation needed; this ideally would be done after the file has been processed by the updater...)
 cd /a/search/indexes/snapshot/
 rm -rf *.spell 
 rm -rf *.prefix (citation needed; this ideally would be done after the file has been processed by the updater...))
 su - rainman
 nohup /home/rainman/scripts/search-restart-indexer &

The last restart makes the search indexer write to new log files so the old unlinked ones are actually removed. We do it as rainman so he can shoot and restart the process later if needed.

Space Issues on Cluster Host

• check /a/search/indexes for unintended indexes, i.e. cruft from previous configurations
• [what else?]
• [what's the procedure for adjusting sharding?]

Cluster Host Hardware Failure

• [what's the procedure for deploying a replacement cluster host?]

Indexer Host Hardware Failure

• [what's the procedure for deploying a replacement indexer?]

Excess Load on a Cluster Host

• [which logs etc. to check for evidence of i.e. abuse, configuration issues, etc]