"✖ DuckDuckGo blocked this request" almost all the time

[viniciusmuller]

Describe the bug
Hi! Thank you for creating this awesome tool. I can make some requests but after 4 or 5 requests I get this "duckduckgo blocked" this message all the time. I think it could probably be a block/rate limit but other engines such as google work just fine. I can use stackexchange/google but the duckduckgo search results are far superior to those.

To Reproduce
Steps to reproduce the behavior:

1. Do 5+ requests using the default settings

Environment

• OS: NixOS
• Terminal: Kitty
• so --version: 0.4.5

[samtay]

I just reproduced this too after a couple requests. I should really add optional verbose/debug logging, but in lieu of that, manually printing the response body from DDG shows

If this error persists, please let us know: error-lite@duckduckgo.com

which isn't a unique error; this seems like a common occurrence when attempting to automate requests to DDG. But I'm not sure why the first few requests work, and then subsequent ones fail. Hopefully I can dig into this and try a few different things. My first thought is to create and cycle through a few mock user agents instead of just using the same one every time.

[viniciusmuller]

Are you going to try doing it this weekend? If not, I'm not really a rusty guy but I can give it a try.

Edit: Did some experimentation using the fake-useragent crate and most of the runs were fails. Not sure if the library is generating bad user agents or duckduckgo simply denies them. Probably a better approach would be cherry-pick valid user agents and find a way to rotate them.

[samtay]

Thanks for giving it a shot! I took a look today, this appears to be a recent issue; I suppose DDG recently got an upgrade and can better suss out which requests are not coming from a browser (which technically is against their TOS of course). E.g. gleitz/howdoi#404 is also having problems.

I've been playing around with curl, basically hammering DDG with variations of

curl -L -H 'User-Agent: $AGENT' 'https://duckduckgo.com/?q=(site:stackoverflow.com)what'

including variations of origin / referer headers as one commenter suggested on another howdoi issue - this doesn't seem to have any effect.

I agree with that approach you mentioned, for now. I'll probably come up with 10-20 hard-coded user agents and pick them at random. Hopefully it will mitigate the blocked requests slightly, but it won't solve this for good, since howdoi already does that as well. In any case I'll keep this issue open until I discover a better solution.

[0atman]

Just got this. No fix yet?

✖ DuckDuckGo blocked this request

[samtay]

No, but I thought I had already changed the default search engine to Google, so thanks for reminding me. Then this at least won't happen to people by default.

[forthrin]

Fighting automation prevention is a Sisyphean, Pareto-defying undertaking. Why not SE by default?

That said, Stack's spineless, decade-long bystander non-handling of abuse have dug their own grave, thus suggesting:

1. Duplicate all knowledge before their servers spin down
2. Use AI to remove all traces of the cause for their demise (BOFHs, downvoters and other CBGs)
3. Build a database from the prettified remains and index this through this tool