View source for Diff hell

{{old}}

We've had these intermittently for ages, but reporting has gone way up the last few days. Diffs with mysteriously repeating lines, like this:

:[[Image:Diff-hell.png|500px|border]]

To help track it down, I added some debug comments, then did a batch test hitting and re-hitting a bad diff that was reported in IRC...

== Testing ==

Did 1000 purge hits to [http://en.wikipedia.org/w/index.php?title=Talk%3AHaidinger%27s_brush&diff=208207415&oldid=208207297 this diff].

8 of these hits gave the bogus, extra-long renderings:

<pre>
diff-144:<!-- diff generator: wikidiff2 srv141 2008-04-25 23:13:49 -->
diff-149:<!-- diff generator: wikidiff2 srv99 2008-04-25 23:13:52 -->
diff-214:<!-- diff generator: wikidiff2 srv98 2008-04-25 23:14:38 -->
diff-326:<!-- diff generator: wikidiff2 srv129 2008-04-25 23:15:57 -->
diff-484:<!-- diff generator: wikidiff2 srv132 2008-04-25 23:17:50 -->
diff-755:<!-- diff generator: wikidiff2 srv88 2008-04-25 23:20:59 -->
diff-774:<!-- diff generator: wikidiff2 srv119 2008-04-25 23:21:13 -->
diff-946:<!-- diff generator: wikidiff2 srv83 2008-04-25 23:23:12 -->
</pre>

Spot checks confirm that the same server will render perfectly correct ones as well:
<pre>
$ ls -l `grep -l srv83 diff*`
-rw-r--r--  1 brion  staff  20833 Apr 25 16:14 diff-224
-rw-r--r--  1 brion  staff  20833 Apr 25 16:19 diff-609
-rw-r--r--  1 brion  staff  20833 Apr 25 16:20 diff-723
-rw-r--r--  1 brion  staff  20833 Apr 25 16:21 diff-792
-rw-r--r--  1 brion  staff  20833 Apr 25 16:23 diff-936
-rw-r--r--  1 brion  staff  23284 Apr 25 16:23 diff-946
</pre>

Notes:
* The exact number of repeated lines was variable mostly 5x, some 2x
* all bad hits were Fedora machines (8 of 515 hits; the other 485 were to Ubuntu machines srv150-189)
* all hits reported wikidiff2
* a further test running the diff directly in a command-line script on srv83 25,000 times in one process had encountered no failures.