toLower is slow

[noughtmare]

See this discourse post.

Take this program:

import qualified Data.Text as T
import qualified Data.Text.IO as T
import System.IO
import Control.Monad

main :: IO ()
main = isEOF >>= \b -> unless b $
  foldr seq main . T.words . T.toLower =<< T.getLine

It is almost 2x slower than equivalent Python code:

import sys

for line in sys.stdin:
    words = line.lower().split()

The benchmark I used is the kjvbible.txt.

Results (using hyperfine):

Benchmark 1: ./simple-hs <kjvbible.txt
  Time (mean ± σ):     156.7 ms ±   4.9 ms    [User: 141.2 ms, System: 6.9 ms]
  Range (min … max):   149.1 ms … 164.8 ms    19 runs
 
Benchmark 1: python3 simple.py <kjvbible.txt
  Time (mean ± σ):      81.0 ms ±   2.4 ms    [User: 73.5 ms, System: 5.4 ms]
  Range (min … max):    78.3 ms …  87.8 ms    36 runs

Removing the toLower step from both programs makes the running time approximately equal:

Benchmark 1: ./simpler-hs <kjvbible.txt
  Time (mean ± σ):      70.2 ms ±   6.9 ms    [User: 54.9 ms, System: 7.0 ms]
  Range (min … max):    61.4 ms …  92.7 ms    44 runs
 
Benchmark 1: python3 simpler.py <kjvbible.txt
  Time (mean ± σ):      70.6 ms ±   1.3 ms    [User: 62.8 ms, System: 5.3 ms]
  Range (min … max):    69.0 ms …  77.0 ms    40 runs

[Lysxia]

Maybe CaseMapping.hs should be made into a proper lookup table.

[noughtmare]

Maybe it also has to do with the stream/unstream fusion overhead?

[Bodigrim]

Python supports multiple internal representations of String. When a string contains only ASCII characters, Python runtime marks it as ASCII-only and uses optimized implementations of basic operations. For example, toLower on ASCII strings preserves length (so one can immediately allocate output buffer) and just adds 32 to bytes in [65..90] (which is a very fast operation).

The first step to narrow the gap is indeed to rewrite toLower without streaming. Then one can add a shortcut for ASCII characters and likely come pretty close to Python performance.

[noughtmare]

Even if I insert a unicode character somewhere in the benchmark text, I still get much worse performance in Haskell (now on a slower machine):

Benchmark 1: .\T-hs.exe < .\kjvbible.txt
  Time (mean ± σ):     262.6 ms ±   5.2 ms    [User: 4.0 ms, System: 4.1 ms]
  Range (min … max):   257.8 ms … 275.4 ms    10 runs
Benchmark 1: python .\T.py < .\kjvbible.txt
  Time (mean ± σ):     112.5 ms ±   2.0 ms    [User: 3.6 ms, System: 5.2 ms]
  Range (min … max):   108.7 ms … 116.9 ms    24 runs

Or can it even track that I only inserted one unicode character?

Edit: It's still fast even if I replace all O with Ö (Ö LÖRD :P).

[Bodigrim]

Dunno, I'm no expert in Python. Anyway, the first step is to avoid streaming in Haskell implementation.