Tweak tagger/nginx rate limit logic

[cdrini]

See also https://github.com/internetarchive/olsystem/pull/292

Technical

Testing

Screenshot

Stakeholders

[Copilot]

Pull request overview

This PR refines the nginx rate limiting and request filtering logic to better detect and handle suspicious traffic patterns, with coordinated changes to the tagger system in the related olsystem repository.

Key changes:

• Enhanced referer detection to catch clients sending literal "-" as referer value
• Expanded rate limiting to include suspicious IPs in addition to non-identifying crawlers
• Added dedicated cover image rate limits for non-identifying crawlers to prevent abuse while allowing legitimate batch cover loading

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
docker/web_nginx.conf	Enhanced referer detection to recognize "-" as empty referer; expanded paths requiring referer to include subjects, authors, and search pages; removed direct $is_sus_referer enforcement (delegated to tagger.js)
docker/nginx.conf	Updated file extension from .map to .conf for consistency; added is_sus_ip.conf include; extended crawler rate limit logic to incorporate suspicious IPs; increased global crawler rate limit from 15r/s to 17r/s; added dedicated 150r/s cover rate limit for non-identifying crawlers
docker/covers_nginx.conf	Applied new global_crawler_cover_limit zone to cover endpoints

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.