Sitemaps not generating

[cdrini]

Related to #7580

Eg https://openlibrary.org/static/sitemaps/sitemap_authors_0000.xml.gz . Seems like they've been erroring since June...

How to Test

ssh -A ol-home0
cd /opt/openlibrary
docker exec -it -uopenlibrary openlibrary_cron-jobs_1 bash
cd /1/var/tmp/sitemaps  # make sure it is empty
python /openlibrary/scripts/sitemaps/sitemap.py /1/var/tmp/dumps/ol_dump_2023-02-28/ol_dump_2023-02-28.txt.gz

Then check /1/var/tmp/sitemaps

Evidence / Screenshot (if possible)

Relevant URL?

https://openlibrary.org/static/sitemaps/sitemap_authors_0000.xml.gz

Steps to Reproduce

1. Go to https://openlibrary.org/static/sitemaps/sitemap_authors_0000.xml.gz

• Actual: 404
• Expected: sitemaps

Details

• Logged in (Y/N)? Y
• Browser type/version? FF
• Operating system? Win10
• Environment (prod/dev/local)? prod

Proposal & Constraints

Related files

Stakeholders

[cclauss]

I have run /openlibrary/scripts/sitemaps/sitemap.py a few times today with no trouble. Takes hours but runs just fine.

• Separately, I also created Delete sitemaps before generating them, not after #7747 (merged)
• I put those (minor) changes into the openlibrary_cron-jobs_1 image.
• Tonight a month-end data dump will run and we will see if the sitemaps are uploaded to nginx or not.

I will make another PR adding logging and minor optimizations.
I sense that we should do this processing in a Pandas dataframe (seconds, not hours).

[cclauss]

After landing #7747 and the monthly dumps cron job...

• Delete sitemaps before generating them, not after #7747

% ls -hal /1/var/tmp/sitemaps

total 7.4G
drwxr-xr-x 3 systemd-coredump systemd-coredump 4.0K Apr  1 22:27 .
drwxr-xr-x 8 systemd-coredump systemd-coredump 4.0K Apr  1 22:29 ..
drwxr-xr-x 2 systemd-coredump systemd-coredump 248K Apr  1 22:40 sitemaps
-rw-r--r-- 1 systemd-coredump systemd-coredump 3.7G Apr  1 22:27 sitemaps_data.txt
-rw-r--r-- 1 systemd-coredump systemd-coredump 3.7G Apr  1 22:29 sitemaps_data.txt.sorted
-rw-r--r-- 1 systemd-coredump systemd-coredump    0 Apr  1 21:46 sitemaps.log

[cclauss]

The sitemaps are generating just fine. They were getting immediately deleted but that was fixed in #7747