OTRS

Latest revision as of 12:16, 22 February 2013

This page page may be outdated and could contain incorrect details. Please update it if you can.

OTRS is installed on williams.wikimedia.org.

Source is in /opt/otrs
Config file is /opt/otrs/Kernel/Config.pm
URL is https://ticket.wikimedia.org/otrs/index.pl
The root user/pass is in /home/wikipedia/doc/otrs
Primary db is on db48, replica on db49. OTRS uses only the primary DB, they apparently don't support slaves.
The codebase is patched, e.g. for supporting one-click-spam. The patches can be found at http://svn.wikimedia.org/svnroot/mediawiki/trunk/otrs
We use mod_perl with ModPerl::Registry, so whenever you change a file, you need to /etc/init.d/apache2 reload
The "News" messages on the main OTRS login screen can be editing by modifying /opt/otrs/Kernel/Output/HTML/Standard/Motd.dtl.

You no longer need to invoke a specific script or update config files to add email addresses to OTRS; mchenry will automatically see that the queue exists or has disappeared.

It is possible (due to negative caching at the secondary mail exchangers) that new addresses will take up to two hours to begin working.

[edit] Configuration

Configuration can be done by examining Kernel/Config/Files/ZZZAuto.pm and Kernel/Config/Files/ZZZAAuto.pm for default values, and then making a corresponding addition to Kernel/Config.pm. When you change the config, reload apache to clear the mod_perl cache.

OTRS configuration is meant to be primarily done via the Kernel/Config/Files/*.xml files. These are XML files edited by the SysConfig module. Viewing the package manager or running bin/otrs.RebuildConfig.pl will read the XML files and then regenerate the ZZZ*.pm files. Then at runtime, the ZZZ*.pm files are read, followed by the Config.pm overrides. So far so good.

The problem is that the XML files are a mixture of user-level configuration, technical data such as core module registrations, and interface text. They are distributed with OTRS, and trying to use old XML files with new OTRS versions will break horribly, because there is no other registry of defaults (except for a few special cases in the code), so vital interface text and module registrations will be missing.

Diffing and merging is difficult due to the lack of comments and the spurious changes introduced when changing things in the web interface.

So save yourself the hassle and edit Config.pm.

[edit] Upgrading

Fetch new OTRS code
Decompress into /opt/otrs-X.Y.Z
Get the patches and icons with svn export --force http://svn.wikimedia.org/svnroot/mediawiki/trunk/otrs /opt/otrs-X.Y.Z
Apply the patches with quilt push -a
Copy in var/log, var/stats, var/sessions
Symlink the templates...
- cd Kernel/Output/HTML && ln -s Standard OTRS
Set permissions on the dir, eg:
- bin/SetPermissions.pl --secure --otrs-user=otrs --web-user=apache --otrs-group=otrs --web-group=apache /opt/otrs-X.Y.Z
Stop exim4. You don't want to have mail coming in while OTRS is broken.
Follow the database upgrade instructions as documented in OTRS's UPGRADING file
change the /opt/otrs symlink to the new version
/etc/init.d/apache2 reload
start exim4
send a mail to e.g. info-en and check that it shows up in OTRS

Bug 22622 is about upgrading the current version of Bugzilla. Philippe Beaudette is the business owner of OTRS at WMF and Martin Edenhofer is the volunteer who is working on it with Jeff Green.

[edit] Mail setup

In the new OTRS setup, OTRS is installed on williams, and e-mail is sent and received through a special Exim instance on this server. Its configuration follows the lines of the setup described in Mail, but OTRS specific configuration is listed below.

[edit] Exim

The Exim setup for OTRS on williams is somewhat comparable to the one on sanger. It does not follow the concept of serving some specific domains, but rather accepts any mail from a select sets of hosts only (mchenry). It will then check the OTRS database for validity of the recipient addresses, and pipe the message to an OTRS command in case of success.

Locally submitted mail (e.g. by OTRS) is forwarded to mchenry.

The top of /etc/exim4/exim4.conf]] contains some OTRS specific macros:

OTRS_POSTMASTER=/opt/otrs/bin/PostMaster.pl OTRS_HOME=/opt/otrs-home OTRS_USER=otrs OTRS_GROUP=otrs [edit] Main configuration The following line defines the database(s) queried for OTRS addresses: # MySQL lookups (OTRS) hide mysql_servers = srv7.wikimedia.org/otrs/exim/password : \ srv8.wikimedia.org/otrs/exim/password

The hostlist <tt>otrs_mail_from_hosts lists the MXes from which williams accepts mail:

hostlist otrs_mail_from_hosts = mchenry.wikimedia.org

OTRS must be able to set any sender address:

# Allow OTRS to use any sender address
untrusted_set_sender = *
local_from_check = false

[edit] ACLs

The mail server on williams only needs to accept mail from and send mail to the Wikimedia mail relay(s), so any other connections can be denied in the acl_smtp_connect acl:

acl_smtp_connect = acl_check_connect

acl_check_connect:
        # We only accept mail from our own mail relays
        require message = This server does not accept external mail
                hosts = <; 127.0.0.0/8 ; ::1 ; +wikimedia_nets

        accept

In the acl_smtp_rcpt ACL, mail from the hosts listed in otrs_mail_from_hosts is explicitly accepted without any further checks:

accept hosts = +otrs_mail_from_hosts : +relay_from_hosts

[edit] SpamAssassin scanning

Spam filtering is done using SpamAssassin and an Exim ACL which is run at the DATA phase during the SMTP connection. Should SpamAssassin fail for some reason, mail is let through.

acl_smtp_data = acl_check_data

acl_check_data:
        # Run a SpamAssassin check on any non-locally submitted messages.
        # If the message is estimated to be spam, we add a header for OTRS
        # to sort it into its Junk queue 
        
        accept hosts = :

        warn    spam = otrs/defer_ok
                set acl_m0 = $spam_score ($spam_bar)
                set acl_m1 = $spam_report

        accept

These variables are then used by a system filter, in a similar way as described in Mail. However, additionally any X-OTRS-* control headers are removed to prevent external injection/tampering. And if the spam score is sufficiently high, the system filter adds a header X-OTRS-Queue: Junk, instructing OTRS to sort the mail into its Junk queue.

[edit] OTRS mail routing

Mail destined for OTRS is served by a simple accept router otrs, which does a MySQL database query to determine the validity of the recipient address being routed, similar to the check done earlier by mchenry.

# Mail destined for OTRS

otrs:
        driver = accept
        condition = ${lookup mysql{SELECT value0 FROM system_address WHERE value0='${quote_mysql:$local_part@$domain}'}{true}fail}
        transport = otrs

On success, the message is handed over to the otrs pipe transport:

# OTRS pipe transport

otrs:
        driver = pipe
        command = OTRS_POSTMASTER
        current_directory = OTRS_HOME
        home_directory = OTRS_HOME
        user = OTRS_USER
        group = OTRS_GROUP
        freeze_exec_fail
        log_fail_output
        timeout = 1m
        timeout_defer

This transport pipes the full contents of the message to the command/path specified in the macro OTRS_POSTMASTER (defined at the top of the file). A current and home directory will be set as specified, and the command will be run as the otrs user and group. If the actual execution/invocation fails for some reason, the message will be frozen on the queue with a warning message sent to root. If the command invocation succeeds, but the return code is EX_TEMPFAIL (e.g. when OTRS cannot access the database), the message is deferred/queued, and will be retried later. Any output will be logged.

[edit] Outbound mail

Any mail destined for an address that is not an OTRS address, e.g. mail submitted by OTRS itself, will be forwarded to mchenry using the manualroute router:

# Send all mail not destined for the local machine via a set of
# mail relays ("smart hosts")

smart_route:
        driver = manualroute
        transport = remote_smtp
        route_list = *  mchenry.wikimedia.org:lists.wikimedia.org

[edit] SpamAssassin

williams runs its own SpamAssassin instance, so sa-learn can be used to train it from the OTRS Junk queue. The normal Ubuntu spamassassin package is used, with a few configuration modifications as listed below.

Multiple user profiles are not really used, SpamAssassin only reads global configuration settings. Training data is stored under the otrs home directory, /opt/otrs-home/.spamassassin/

[edit] /etc/default/spamassassin

Make sure spamd is enabled:

# Change to one to enable spamd
ENABLED=1

By default it runs as root, which is unnecessary. Since it's only used by OTRS, we can run it as the OTRS user. User preferences are disabled, spamd listens on the loopback interface only.

OPTIONS="--max-children 5 --nouser-config --listen-ip=127.0.0.1 -u otrs -g otrs"

Don't let spam filtering eat all resources:

# Set nice level of spamd
NICE="--nicelevel 10"

Automatically update SpamAssassin rules:

# Cronjob
# Set to anything but 0 to enable the cron job to automatically update
# spamassassin's rules on a nightly basis
CRON=1

[edit] /etc/spamassassin/local.cf

Allow SpamAssassin to trust these IP ranges:

trusted_networks 208.80.152.0/22 91.198.174.0/24 203.212.189.192/26

@@ Line 1: / Line 1: @@
-[http://otrs.org OTRS] is installed on zwinger.
+{{Old-doc}}
-*URL is http://download.wikimedia.org/otrs/index.pl
+[http://otrs.org OTRS] is installed on williams.wikimedia.org.
-*Installed from the command-line, using the documentation at [http://doc.otrs.org/1.2/en/html/install-cli.html].
+*Source is in <tt>/opt/otrs</tt>
+*Config file is <tt>/opt/otrs/Kernel/Config.pm</tt>
+*URL is https://ticket.wikimedia.org/otrs/index.pl
+*The root user/pass is in /home/wikipedia/doc/otrs
+*Primary db is on db48, replica on db49. OTRS uses only the primary DB, they apparently don't support slaves.
+*The codebase is patched, e.g. for supporting one-click-spam. The patches can be found at http://svn.wikimedia.org/svnroot/mediawiki/trunk/otrs
+* We use mod_perl with ModPerl::Registry, so whenever you change a file, you need to <tt>/etc/init.d/apache2 reload</tt>
+*The "News" messages on the main OTRS login screen can be editing by modifying <tt>/opt/otrs/Kernel/Output/HTML/Standard/Motd.dtl.</tt>
+You no longer need to invoke a specific script or update config files to add email addresses to OTRS; [[mchenry]] will automatically see that the queue exists or has disappeared.
+It is possible (due to negative caching at the secondary mail exchangers) that new addresses will take up to two hours to begin working.
+== Configuration ==
+Configuration can be done by examining Kernel/Config/Files/ZZZAuto.pm and Kernel/Config/Files/ZZZAAuto.pm for default values, and then making a corresponding addition to Kernel/Config.pm. When you change the config, reload apache to clear the mod_perl cache.
+OTRS configuration is meant to be primarily done via the Kernel/Config/Files/*.xml files. These are XML files edited by the SysConfig module. Viewing the package manager or running bin/otrs.RebuildConfig.pl will read the XML files and then regenerate the ZZZ*.pm files. Then at runtime, the ZZZ*.pm files are read, followed by the Config.pm overrides. So far so good.
+The problem is that the XML files are a mixture of user-level configuration, technical data such as core module registrations, and interface text. They are distributed with OTRS, and trying to use old XML files with new OTRS versions will break horribly, because there is no other registry of defaults (except for a few special cases in the code), so vital interface text and module registrations will be missing.
+Diffing and merging is difficult due to the lack of comments and the spurious changes introduced when changing things in the web interface.
+So save yourself the hassle and edit Config.pm.
+== Upgrading ==
+# Fetch new OTRS code
+# Decompress into /opt/otrs-X.Y.Z
+# Get the patches and icons with svn export --force http://svn.wikimedia.org/svnroot/mediawiki/trunk/otrs /opt/otrs-X.Y.Z
+# Apply the patches with quilt push -a
+# Copy in var/log, var/stats, var/sessions
+# Symlink the templates...
+#* cd Kernel/Output/HTML && ln -s Standard OTRS
+# Set permissions on the dir, eg:
+#* bin/SetPermissions.pl --secure --otrs-user=otrs --web-user=apache --otrs-group=otrs --web-group=apache /opt/otrs-X.Y.Z
+# Stop exim4. You don't want to have mail coming in while OTRS is broken.
+# Follow the database upgrade instructions as documented in OTRS's UPGRADING file
+# change the /opt/otrs symlink to the new version
+# <tt>/etc/init.d/apache2 reload</tt>
+# start exim4
+# send a mail to e.g. info-en and check that it shows up in OTRS
+[https://bugzilla.wikimedia.org/show_bug.cgi?id=22622 Bug 22622] is about upgrading the current version of Bugzilla.  Philippe Beaudette is the business owner of OTRS at WMF and Martin Edenhofer is the volunteer who is working on it with Jeff Green.
+== Mail setup ==
+In the new OTRS setup, OTRS is installed on [[williams]], and e-mail is sent and received through a special Exim instance on this server. Its configuration follows the lines of the setup described in [[Mail]], but OTRS specific configuration is listed below.
+=== Exim ===
+The Exim setup for OTRS on [[williams]] is somewhat comparable to the one on [[sanger]]. It does not follow the concept of serving some specific ''domains'', but rather accepts any mail from a select sets of hosts only ([[mchenry]]). It will then check the OTRS database for validity of the recipient addresses, and pipe the message to an OTRS command in case of success.
+Locally submitted mail (e.g. by OTRS) is forwarded to [[mchenry]].
+The top of <tt>/etc/exim4/exim4.conf]] contains some OTRS specific macros:
+ OTRS_POSTMASTER=/opt/otrs/bin/PostMaster.pl
+ OTRS_HOME=/opt/otrs-home
+ OTRS_USER=otrs
+ OTRS_GROUP=otrs
+==== Main configuration ====
+The following line defines the database(s) queried for OTRS addresses:
+ # MySQL lookups (OTRS)
+ hide mysql_servers = srv7.wikimedia.org/otrs/exim/''password'' : \
+                      srv8.wikimedia.org/otrs/exim/''password''
+The hostlist <tt>otrs_mail_from_hosts</tt> lists the MXes from which williams accepts mail:
+ hostlist otrs_mail_from_hosts = mchenry.wikimedia.org
+OTRS must be able to set any sender address:
+ # Allow OTRS to use any sender address
+ untrusted_set_sender = *
+ local_from_check = false
+==== ACLs ====
+The mail server on [[williams]] only needs to accept mail from and send mail to the Wikimedia mail relay(s), so any other connections can be denied in the ''acl_smtp_connect'' acl:
+ acl_smtp_connect = acl_check_connect
-A workaround is needed to deal with a known problem with perl and UTF-8 ([https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=106228 Redhat bug report]).  The 1st and 3rd lines in the below code snippet were added in /opt/otrs/Kernel/cpan-lib/Email/Valid.pm, which is the CPAN module <tt>Email::Valid</tt>.
 <pre>
-Encode::_utf8_off($addr);
+acl_check_connect:
-($addr) = Mail::Address->parse( $addr );
+        # We only accept mail from our own mail relays
-Encode::_utf8_on($addr);
+        require message = This server does not accept external mail
+                hosts = <; 127.0.0.0/8 ; ::1 ; +wikimedia_nets
+        accept
 </pre>
-[http://doc.otrs.org/1.2/en/html/install-cli.html Installation docs]
+In the ''acl_smtp_rcpt'' ACL, mail from the hosts listed in ''otrs_mail_from_hosts'' is explicitly accepted without any further checks:
+ accept hosts = +otrs_mail_from_hosts : +relay_from_hosts
+==== SpamAssassin scanning ====
+Spam filtering is done using [[#SpamAssassin|SpamAssassin]] and an Exim ACL which is run at the DATA phase during the SMTP connection. Should SpamAssassin fail for some reason, mail is let through.
+ acl_smtp_data = acl_check_data
+<pre>
+acl_check_data:
+        # Run a SpamAssassin check on any non-locally submitted messages.
+        # If the message is estimated to be spam, we add a header for OTRS
+        # to sort it into its Junk queue
+        accept hosts = :
+        warn    spam = otrs/defer_ok
+                set acl_m0 = $spam_score ($spam_bar)
+                set acl_m1 = $spam_report
+        accept
+</pre>
+These variables are then used by a ''system filter'', in a similar way as described in [[Mail]]. However, additionally any X-OTRS-* control headers are removed to prevent external injection/tampering. And if the spam score is sufficiently high, the system filter adds a header <tt>X-OTRS-Queue: Junk</tt>, instructing OTRS to sort the mail into its Junk queue.
+==== OTRS mail routing ====
+Mail destined for OTRS is served by a simple accept router ''otrs'', which does a MySQL database query to determine the validity of the recipient address being routed, similar to the check done earlier by mchenry.
+<pre>
+# Mail destined for OTRS
+otrs:
+        driver = accept
+        condition = ${lookup mysql{SELECT value0 FROM system_address WHERE value0='${quote_mysql:$local_part@$domain}'}{true}fail}
+        transport = otrs
+</pre>
+On success, the message is handed over to the ''otrs'' pipe transport:
+<pre>
+# OTRS pipe transport
+otrs:
+        driver = pipe
+        command = OTRS_POSTMASTER
+        current_directory = OTRS_HOME
+        home_directory = OTRS_HOME
+        user = OTRS_USER
+        group = OTRS_GROUP
+        freeze_exec_fail
+        log_fail_output
+        timeout = 1m
+        timeout_defer
+</pre>
+This transport pipes the full contents of the message to the command/path specified in the macro OTRS_POSTMASTER (defined at the top of the file). A current and home directory will be set as specified, and the command will be run as the otrs user and group. If the actual execution/invocation fails for some reason, the message will be frozen on the queue with a warning message sent to root. If the command invocation succeeds, but the return code is <tt>EX_TEMPFAIL</tt> (e.g. when OTRS cannot access the database), the message is deferred/queued, and will be retried later. Any output will be logged.
+==== Outbound mail ====
+Any mail destined for an address that is ''not'' an OTRS address, e.g. mail submitted by OTRS itself, will be forwarded to mchenry using the manualroute router:
+<pre>
+# Send all mail not destined for the local machine via a set of
+# mail relays ("smart hosts")
+smart_route:
+        driver = manualroute
+        transport = remote_smtp
+        route_list = *  mchenry.wikimedia.org:lists.wikimedia.org
+</pre>
+=== SpamAssassin ===
+[[williams]] runs its own SpamAssassin instance, so <tt>sa-learn</tt> can be used to train it from the OTRS Junk queue. The normal Ubuntu <tt>spamassassin</tt> package is used, with a few configuration modifications as listed below.
+Multiple user profiles are not really used, SpamAssassin only reads global configuration settings. Training data is stored under the ''otrs'' home directory, <tt>/opt/otrs-home/.spamassassin/</tt>
+==== /etc/default/spamassassin ====
+Make sure ''spamd'' is enabled:
+ # Change to one to enable spamd
+ ENABLED=1
+By default it runs as root, which is unnecessary. Since it's only used by OTRS, we can run it as the OTRS user.
+User preferences are disabled, spamd listens on the loopback interface only.
+ OPTIONS="--max-children 5 --nouser-config --listen-ip=127.0.0.1 -u otrs -g otrs"
+Don't let spam filtering eat all resources:
+ # Set nice level of spamd
+ NICE="--nicelevel 10"
+Automatically update SpamAssassin rules:
+ # Cronjob
+ # Set to anything but 0 to enable the cron job to automatically update
+ # spamassassin's rules on a nightly basis
+ CRON=1
+==== /etc/spamassassin/local.cf ====
+Allow SpamAssassin to trust these IP ranges:
+ trusted_networks 208.80.152.0/22 91.198.174.0/24 203.212.189.192/26
+[[Category:Mail]]

OTRS

Latest revision as of 12:16, 22 February 2013

Contents

[edit] Configuration

[edit] Upgrading

[edit] Mail setup

[edit] Exim

[edit] Main configuration

[edit] ACLs

[edit] SpamAssassin scanning

[edit] OTRS mail routing

[edit] Outbound mail

[edit] SpamAssassin

[edit] /etc/default/spamassassin

[edit] /etc/spamassassin/local.cf

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Ops documentation

Wiki

Toolbox