Full build with PDF is taking more than 24h

[JulienPalard]

Today we're building 6 versions for 13 languages (that's 78 builds).

I'm doing some tests on my machine to get an idea:

• An HTML build takes 55 s.
• A full build (html + text + PDF A4 + PDF letter + epub + texinfo) takes around 21 min, composed of:
  • 1 min for make html composed of:
    • 50 s for sphinx-build
    • 10 s of html archiving
  • 40 s for make text
  • 9 min for the A4 PDF build composed of:
    • 1 min for make latex PAPER=a4
    • 8 min for make all-pdf (in build/latex/) (less if run again, like 1 s if not removing PDFs, or 1 min after removing PDFs). Can be cut down to 6 min with -j 4.
  • 8 min 30 for the letter PDF build, roughly similar to A4 unsurprisingly
  • 1 min for the epub build
  • 1 min 30 s for the texinfo build

So a complete rebuild should take ~27h (on my machine, the server may have a different CPU).

[egeakman]

Building only the changed branches and languages, how does it sound?

I don't have much information about the server but here are some ideas:

• We can store the commit hashes for each branch and language, and check if the head has changed, if changed we can clone and build.
• If we already have all language repos cloned, and we do git pull everytime we build; we can instead git fetch in each repo and see if there are any changes, and if that's the case we can pull and build.

[egeakman]

Additional question for @JulienPalard: Are non-bugfix and non-stable releases only rebuilt manually or is there a cron?

[JulienPalard]

Are non-bugfix and non-stable releases only rebuilt manually or is there a cron?

EOL and security-fixes branches are only built manually, yes.

[egeakman]

What do you think about the idea in general @JulienPalard?

[JulienPalard]

I don't have much information about the server

• The server stores a unique clone of cpython.
• The server keeps all translation repositories.

(see the git_clone function in the script).

Building only the changed branches and languages, how does it sound?

Maybe, yes. But to do it cleanly we should only rebuild if the Doc/ directory changed, but doable and cheap as we keep the cpython clone.

Currently a git log shows that (for the Doc/ directory) changes are:

• the main branch has many updates daily
• the 3.11 branch has daily updates
• the 3.10 branch has 3 updates per month
• the 3.9 branch has 1 update per month

A change cpython side should trigger a rebuild of all languages for the given branch (while a change for a language should not trigger a rebuild for all branches: translation repo has a translation branch per cpython branch).

Translation side I expect less changes, except for a few crons like for python-docs-ja which synchronizes daily. Also changes are for a single branch so they invalidate a single cpython branch.

So instead of rebuilding 78 docs per day, just looking at the cpython side, we'd rebuild like 26 docs daily (main and 3.11) which should take like 9h.

About how to do this, we could store at build time the cpython commit sha and the translation commit sha in a file, and at build time call a dedicated function to check the shas against the repositories to see if a build is needed or not.

Seems doable.

[picnixz]

@JulienPalard Someone opened an issue on Sphinx in order to know whether we could speed-up things on our side. I am not quite confident in that since most of the build time is actually the PDF build. It is possible to know why make all-pdf is so slow? is it only because there are a lot of pages to write? or is it because of the underlying language? (iirc, jp builds don't build with plain pdflatex).

Another possibility is to change the way the PDF are created. We (Sphinx) could technically add a builder which, instead of outputting LaTeX code, outputs some other kind of typesetting language that can still be converted to PDF faster than LaTeX.

[JulienPalard]

Hi @picnixz, thanks for jumping in!

It is possible to know why make all-pdf is so slow?

I'm just a user of LaTeX, I don't know much. I know our biggest file is library.tex with 288k lines of LaTeX, generating a 11MB PDF file.

make all-pdf runs latexmk for each .tex files, which in turn runs xelatex. So I tried just running time latexmk -pdf -dvi- -ps- library.tex, it takes 1 min 16s (on the same laptop used for the timings in the first post). same timing for the underlying command: xelatex -recorder library.tex.

Those timings can be reproduced on a cpython clone after building latex file:

make -C Doc/build/latex/ clean
time make -C Doc/build/latex/ library.pdf  # 4 min 30s
time (cd Doc/build/latex; make clean; XINDYOPTS="-L english -C utf8 -M sphinx.xdy" latexmk -pdf -dvi- -ps- library.tex) # 4 min 13s
time (cd Doc/build/latex; xelatex -recorder library.tex`  # 1 min 16s

It reminds me of something about running it in a loop until the output does not changes..., which strace confirms:

$ grep bin/xelatex make.strace 
15622 1696969290.322763 execve("/usr/bin/xelatex", ["xelatex", "-recorder", "library.tex"], 0x55b2cfd2a5d8 /* 90 vars */) = 0
15719 1696969388.530816 execve("/usr/bin/xelatex", ["xelatex", "-recorder", "library.tex"], 0x5651c759b5d8 /* 90 vars */) = 0
15766 1696969471.969531 execve("/usr/bin/xelatex", ["xelatex", "-recorder", "library.tex"], 0x55ddc43a15d8 /* 90 vars */) = 0

Which also can be confirmed by reading the console log (I choose a smaller file to get readable output):

------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'xelatex   -recorder  "howto-cporting.tex"'
------------

[...]

Rule 'pdflatex':  Reasons for rerun
Changed files or newly in use/created:
  howto-cporting.aux
  howto-cporting.ind
  howto-cporting.out
  howto-cporting.toc

------------
Run number 2 of rule 'pdflatex'
------------
------------
Running 'xelatex   -recorder  "howto-cporting.tex"'
------------

Also noticed, from xelatex output:

LaTeX Warning: Label(s) may have changed. Rerun to get cross-references right.

Package rerunfilecheck Warning: File `library.out' has changed.
(rerunfilecheck)                Rerun to get outlines right
(rerunfilecheck)                or use package `bookmark'.

this is probably less than ideal.

I tried to use perf to get some insights about what's slow inside xetex, if I read this correctly that's the parsing of the input file that's slow, can be reproduced using:

sudo sysctl kernel.perf_event_paranoid=-1
sudo apt install texlive-binaries-dbgsym  # needs `deb https://deb.debian.org/debian-debug trixie-debug main`
time (cd Doc/build/latex/; make clean; perf record -F 1000 -g --call-graph dwarf -o perf.data xelatex -recorder library.tex)
(cd Doc/build/latex/; perf report --tui)

it looks like this:

-   80.59%     0.00%  xelatex    xetex                     [.] main                                               ▒
   - main                                                                                                         ▒
      - 80.54% mainbody                                                                                           ▒
         - 80.37% maincontrol                                                                                     ▒
            - 35.42% getxtoken                                                                                    ▒
               - 18.35% expand                                                                                    ▒
                  - 12.60% conditional                                                                            ▒
                     + 9.10% scanint                                                                              ▒
                     + 1.37% passtext                                                                             ▒
                  + 3.13% expand                                                                                  ▒
                    0.54% passtext                                                                                ▒
               - 14.03% macrocall                                                                                 ▒
                    6.20% getnext                                                                                 ▒
                    1.92% endtokenlist                                                                            ▒
               + 2.67% getnext                                                                                    ▒
            - 35.25% prefixedcommand                                                                              ▒
               - 29.63% zscantoks                                                                                 ▒
                  - 26.62% expand                                                                                 ▒
                     - 19.90% macrocall                                                                           ▒
                        - 10.50% getnext                                                                          ▒
                             3.33% endtokenlist                                                                   ▒
                          1.77% endtokenlist                                                                      ▒
                          0.63% getavail                                                                          ▒
                     + 5.68% conditional                                                                          ▒
                    1.87% getnext                                                                                 ▒
               + 3.12% zdoregistercommand                                                                         ▒
                 0.52% zeqdefine                                                                                  ▒
            + 3.87% measure_native_node                                                                           ▒
              0.86% unsave                                                                                        ▒
            + 0.60% zshipout                                                                                      ▒
            + 0.58% endgraf.part.0                                                                                ▒
              0.51% zbeginbox                                                                                     ▒

is it only because there are a lot of pages to write?

Probably yes, library.pdf is 2318 pages, or lots of latex to parse if my perf reading is good.

or is it because of the underlying language? (iirc, jp builds don't build with plain pdflatex).

Looking at the server logs, yes jp is slower than all the others. It takes like 1h30 while the other take like 30mn (for a full build of text, html, pdfs, ...). All builds are using xelatex while jp builds are using lualatex. As it's the only one I won't focus on this one.

Another possibility is to change the way the PDF are created. We (Sphinx) could technically add a builder which, instead of outputting LaTeX code, outputs some other kind of typesetting language that can still be converted to PDF faster than LaTeX.

It would be possible to build PDF using weasyprint.

[m-aciek]

It looks like a potential alternative for PDFs building is using Rinohtype. It provides a drop-in replacement for Sphinx PDF builder and is a direct PDF builder. Still in beta though.

[picnixz]

Thank you very much for your report !

Now I'm a bit confused about these timings:

time make -C Doc/build/latex/ library.pdf # 4 min 30s
time (cd Doc/build/latex; make clean; XINDYOPTS="-L english -C utf8 -M sphinx.xdy" latexmk -pdf -dvi- -ps- library.tex) # 4 min 13s
time (cd Doc/build/html; xelatex -recorder library.tex` # 1 min 16s

Maybe I misunderstood, but why does the second command takes 4 min 13s but you told me it took 1min 16s? I also think the last command should be cd Doc/build/latex instead of cd Doc/build/html.

if I read this correctly that's the parsing of the input file that's slow

Yes, it's because of how LaTeX is structured with this token based approach + expansions. Depending on how expansions are done, it may enormously slow down the build. I am wondering whether breaking down library.tex into smaller pieces that are put together at the end would improve or not the timings.

---

In conclusion, I don't think we could do much except:

• writing to another format than LaTeX in order to convert it to PDF (e.g., Typst) but I am not aware of any typesetting language as powerful as LaTeX. Also, if we were to include that, we'll need to add an extension for that (or ask the community to help us).
• try to see if breaking down huge files may help or not.

By the way, how much is allocated to build the documentation on Python servers? because a straightforward solution is simply to allocate more resources and run everything in parallel (but again, funds are required and I don't know if the server is like a super-mega-huge-powerful server). In the end, if you need to rebuild 26 docs per day (assuming you can reduce from 70+), the you "simply" need a more powerful unit.

[JulienPalard]

Now I'm a bit confused about these timings:

time make -C Doc/build/latex/ library.pdf # 4 min 30s
time (cd Doc/build/latex; make clean; XINDYOPTS="-L english -C utf8 -M sphinx.xdy" latexmk -pdf -dvi- -ps- library.tex) # 4 min 13s
time (cd Doc/build/html; xelatex -recorder library.tex` # 1 min 16s

Maybe I misunderstood, but why does the second command takes 4 min 13s but you told me it took 1min 16s?

That's because latexmk runs xelatex in a while True: loop until the output stabilizes. Looks like it had to run it 4 times to reach stabilization (I have a few paragraph in my last message about it, search for 'strace' and 'rerun').

I really don't know if there's a way to forge a latex file that necessitates less re-runs.

I also think the last command should be cd Doc/build/latex instead of cd Doc/build/html.

Probably just me manually fixing the commands for readability (and breaking them while doing so, haha).

[picnixz]

That's because latexmk runs xelatex in a while True: loop until the output stabilizes

Ah sorry! yes I overlooked that (I was a bit confused actually because I assumed that the ~1min was the output of time). Now, the question is: is it possible to avoid using xelatex actually and only pdflatex?

Also, since it tells us "or use package bookmark", maybe this could solve the issue (though I don't know how). Nevertheless, yet another alternative is to run the latex command only once and check whether more compilation is needed (and not let latexmk decides by itself). I'm not sure whether the 4 runs are actually needed to solve all the references (in general, we need 2 reruns but here I'm wondering why we actually need 4).

Btw, I'm sorry but I cannot really reproduce it myself because I need to install fonts that I don't have (and move them around files + adding paths or so) (but since you've got everything running on your side + timings are for your machine, the comparison is more fair).

[JulienPalard]

IIRC we use xetex instead of pdflatex for its unicode awareness.

Tried:

$ make -C build_root/cpython/Doc PYTHON=build_root/venv-3.11/bin/python SPHINXBUILD=build_root/venv-3.11/bin/sphinx-build BLURB=build_root/venv-3.11/bin/blurb VENVDIR=build_root/venv-3.11 'SPHINXOPTS=-D latex_engine=pdflatex -q' SPHINXERRORHANDLING= autobuild-stable

[...]

LaTeX Warning: Hyper reference `howto/regex:the-backslash-plague' on page 6 und
efined on input line 816.

[6]

! LaTeX Error: Unicode character ſ (U+017F)
               not set up for use with LaTeX.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.962 Latin small letter dotless i), ‘ſ
                                          ’ (U+017F, Latin small letter lo...

?