Cloudflare doesn't like the reader user agent

[lemon24]

I am getting bozo_exception "xml.sax._exceptions.SAXParseException: :7:2: mismatched tag" for http://randsinrepose.com/feed/.

According to kurtmckee/feedparser#117, installing sgmllib3k should allow feedparser to handle (some?) bad XML. reader does have sgmllib3k as a dependency.

Should this specific feed work? Does the error show because we're failing fast when we have a bozo_exception? Should we? If we don't fail, do we want to signal the feed had issues? How?

[lemon24]

Turns out there's an issue in our parser; feedparser alone works fine for that feed, with or without sgmllib3k.

Test script:

pip install -q feedparser
pip freeze

python << "EOF"
import feedparser
d = feedparser.parse('http://randsinrepose.com/feed/')
print(len(d.entries), d.bozo, d.get('bozo_exception'))
EOF

pip install -q sgmllib3k
pip freeze

python << "EOF"
import feedparser
d = feedparser.parse('http://randsinrepose.com/feed/')
print(len(d.entries), d.bozo, d.get('bozo_exception'))
EOF

pip install -q reader 
pip freeze

python << "EOF"
from reader import make_reader
import logging
logging.basicConfig(level=logging.INFO)
r = make_reader('')
r.add_feed('http://randsinrepose.com/feed/')
r.update_feeds()
print(len(list(r.get_entries())))
EOF

Output:

+ pip install -q feedparser
+ pip freeze
feedparser==5.2.1
+ python
10 0 None
+ pip install -q sgmllib3k
+ pip freeze
feedparser==5.2.1
sgmllib3k==1.0.0
+ python
10 0 None
+ pip install -q reader
+ pip freeze
certifi==2020.6.20
chardet==3.0.4
feedparser==5.2.1
idna==2.10
reader==1.5
requests==2.24.0
sgmllib3k==1.0.0
typing-extensions==3.7.4.2
urllib3==1.25.10
+ python
ERROR:reader:update feed 'http://randsinrepose.com/feed/': error while getting/parsing feed, skipping; exception: SAXParseException('mismatched tag')
Traceback (most recent call last):
  File ".../lib/python3.7/xml/sax/expatreader.py", line 217, in feed
    self._parser.Parse(data, isFinal)
xml.parsers.expat.ExpatError: mismatched tag: line 7, column 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/feedparser.py", line 3956, in parse
    saxparser.parse(source)
  File ".../lib/python3.7/xml/sax/expatreader.py", line 111, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File ".../lib/python3.7/xml/sax/xmlreader.py", line 125, in parse
    self.feed(buffer)
  File ".../lib/python3.7/xml/sax/expatreader.py", line 221, in feed
    self._err_handler.fatalError(exc)
  File ".../lib/python3.7/site-packages/feedparser.py", line 2063, in fatalError
    raise exc
xml.sax._exceptions.SAXParseException: <unknown>:7:2: mismatched tag

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/reader/core.py", line 385, in _parse_feed_for_update
    return feed, self._parser(feed.url, feed.http_etag, feed.http_last_modified)
  File ".../lib/python3.7/site-packages/reader/_parser.py", line 228, in __call__
    return self._parse_http(url, http_etag, http_last_modified)
  File ".../lib/python3.7/site-packages/reader/_parser.py", line 329, in _parse_http
    feed, entries = _process_feed(url, result)
  File ".../lib/python3.7/site-packages/reader/_parser.py", line 185, in _process_feed
    raise ParseError(url) from exception
reader.exceptions.ParseError: http://randsinrepose.com/feed/
0

[lemon24]

OK, it seems Cloudflare doesn't like the reader user agent, but likes the feedparser one (or any UA that starts with it, see output below).

Test script + output:

import requests
import feedparser
import reader._parser

URL = 'http://randsinrepose.com/feed/'

def print_feed(d, name):
    print('---', name)
    print(d.status, d.headers['Server'], repr(d.version), d.bozo, len(d.entries))
    print(d.feed.get('summary'))

d = feedparser.parse(URL)
print_feed(d, 'feedparser')

headers = {
    'Accept': feedparser.ACCEPT_HEADER, 
    'A-IM': 'feed',
    'User-Agent': reader._parser.Parser.user_agent,
}
response = requests.get(URL, headers=headers, stream=True)
with response:
    # FIXME: in reader._parser, we're mistakenly using response_headers=headers
    d = feedparser.parse(response.raw, response_headers=response.headers)
print_feed(d, 'requests, User-Agent: ' + headers['User-Agent'])

headers['User-Agent'] = feedparser.USER_AGENT
response = requests.get(URL, headers=headers, stream=True)
with response:
    d = feedparser.parse(response.raw, response_headers=response.headers)
print_feed(d, 'requests, User-Agent: ' + headers['User-Agent'])

headers['User-Agent'] = f'{reader._parser.Parser.user_agent} {feedparser.USER_AGENT}'
response = requests.get(URL, headers=headers, stream=True)
with response:
    d = feedparser.parse(response.raw, response_headers=response.headers)
print_feed(d, 'requests, User-Agent: ' + headers['User-Agent'])

headers['User-Agent'] = f'{feedparser.USER_AGENT.partition(" ")[0]} {reader._parser.Parser.user_agent}'
response = requests.get(URL, headers=headers, stream=True)
with response:
    d = feedparser.parse(response.raw, response_headers=response.headers)
print_feed(d, 'requests, User-Agent: ' + headers['User-Agent'])

--- feedparser
200 cloudflare 'rss20' 0 10
None
--- requests, User-Agent: python-reader/1.6.dev0 (+https://github.com/lemon24/reader)
403 cloudflare '' 1 0
<center><h1>403 Forbidden</h1></center>
<hr /><center>nginx</center>
--- requests, User-Agent: UniversalFeedParser/5.2.1 +https://code.google.com/p/feedparser/
200 cloudflare 'rss20' 0 10
None
--- requests, User-Agent: python-reader/1.6.dev0 (+https://github.com/lemon24/reader) UniversalFeedParser/5.2.1 +https://code.google.com/p/feedparser/
403 cloudflare '' 1 0
<center><h1>403 Forbidden</h1></center>
<hr /><center>nginx</center>
--- requests, User-Agent: UniversalFeedParser/5.2.1 python-reader/1.6.dev0 (+https://github.com/lemon24/reader)
200 cloudflare 'rss20' 0 10
None

We could use a user agent of the form "$feedparser_ua $reader_ua", but that would be somewhat against RFC 7231, since:

By convention, the product identifiers are listed in decreasing order of their significance for identifying the user agent software.

If we do do this, I'd prefer it's done via a plugin, instead of changing the default reader UA.

It may be possible to verify reader with Cloudflare as a good bot, see I run a good bot and want for it to be added to the allowlist here.

Some notes:

• This isn't related to ill-formed feeds; however, we should still have an answer for Should ill-formed feeds work? (and a test).
• We're mistakenly passing the request headers to feedparser.parse(), instead of the response ones:
  

  reader/src/reader/_parser.py

  Line 318 in ff1be41

  result = parse(response.raw, response_headers=headers)

  • Update: Wrong, a variable was getting reused. I fixed that, and added some tests for headers getting passed around correctly, though (there weren't any).
• Checking if stuff works the same on my database after reader should use a custom User-Agent #154 would have probably caught this.
• We're getting a 403 in the failure case, so Fail fast for bad HTTP status codes #182 would have made the root cause clearer.

[lemon24]

I filled in the Cloudflare Bot Verification form for reader.

[lemon24]

/remind me to check on the verification thing in 2 weeks from now.

[reminders]

@lemon24 set a reminder for Aug 14th 2020

[reminders]

👋 @lemon24, check on the verification thing in .