Add shortlist of user-agent rate limit

[cdrini]

Moves this from prod onto here.

Requires https://github.com/internetarchive/olsystem/pull/317

Technical

Testing

This was roughly tested, but has since been cleaned up and tightened, so there might be adjustments that need to be made.

Screenshot

Stakeholders

[Copilot]

Pull request overview

Adds an additional nginx rate-limiting layer keyed off a shortlist of user agents (intended to be more permissive than the default), and tightens some request classification rules in the nginx frontend configs used by the dockerized deployment.

Changes:

• Add a new ua_rate_limit limit zone and apply it to / and API/JSON requests.
• Make “sus user agent” matching case-insensitive for specific blocked UAs.
• Treat mode=all requests as requiring referrer checks (via $requires_referer mapping).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
docker/web_nginx.conf	Updates UA matching, adds mode=all to the “requires referrer” classifier, and applies the new ua_rate_limit in key locations.
docker/nginx.conf	Adds an include expected to define $ua_rate_limit_key and defines the new ua_rate_limit zone.