[PATCH] Make the highlight helper multibyte safe

The highlight helper takes multiple terms now, so this doesn't apply cleanly.

It also looks like it removes case-insensitivity for non-utf8 strings.

Here's a new patch that should apply cleanly to trunk. I've also added a test to show that case-insensitivity for non-utf8 strings still works.

Updated for revision 7064.

Tested against 7187, generated from trunk

Oh yeah, almost forgot, +1. Seems like a really good idea.

Definitely a must!

+1

Norbert just told me he had a disccusion with Koz about the patch. Koz raised some concerns about performance with doing u.chars.upcase and u.chars.downcase for every character.

Unfortunately I don't know a faster or better way to do it. The default regular expression engine in Ruby doesn't have a complete upcase / downcase table for every Unicode character. Note that the expensive part of the loop is only performed when the character consists of more than one byte.

Here's a short benchmark to show how slow it is. The benchmarks calculate each operation a 100 times, so the time for a single call would be a 100 times faster.

      user     system      total        real
Old - 20 ASCII characters     :  0.010000   0.000000   0.010000 (  0.000656)
Old - 20 Multibyte characters :  0.000000   0.000000   0.000000 (  0.000633)
Old - 40 ASCII characters     :  0.000000   0.000000   0.000000 (  0.000684)
Old - 40 Multibyte characters :  0.000000   0.000000   0.000000 (  0.000764)
Old - 100 ASCII characters    :  0.000000   0.000000   0.000000 (  0.000820)
Old - 100 Multibyte characters:  0.000000   0.000000   0.000000 (  0.000955)
New - 20 ASCII characters     :  0.010000   0.000000   0.010000 (  0.005869)
New - 20 Multibyte characters :  0.680000   0.030000   0.710000 (  0.731969)
New - 40 ASCII characters     :  0.010000   0.000000   0.010000 (  0.011309)
New - 40 Multibyte characters :  1.100000   0.050000   1.150000 (  1.159216)
New - 100 ASCII characters    :  0.020000   0.000000   0.020000 (  0.026393)
New - 100 Multibyte characters:  2.650000   0.110000   2.760000 (  2.792602)

Xcuzme, but why do you need to casefold per char in this case? Additionally - _yes_ unicode casefolding is slow. Live with it.

Could we use separate codepaths depending on whether the highlight string is multibyte?