bpo-32861: urllib.robotparser fix incomplete __str__ methods. #5711
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The RobotFileParser's string representation was incomplete and missing some valid rule lines.
Member
serhiy-storchaka
left a comment
serhiy-storchaka
left a comment
There was a problem hiding this comment.
LGTM. I have added just few suggestions. And the two unnecessary trailing newlines should be kept in maintained releases.
Please add a news entry.
Lib/urllib/robotparser.py
Outdated
|
|
||
| def __str__(self): | ||
| return ''.join([str(entry) + "\n" for entry in self.entries]) | ||
| ret = [str(entry) for entry in self.entries] |
Member
There was a problem hiding this comment.
This may be faster:
entries = self.entries
if self.default_entry is not None:
entries = entries + [self.default_entry]
return '\n\n'.join(map(str, entries))
Lib/urllib/robotparser.py
Outdated
| ret = [] | ||
| for agent in self.useragents: | ||
| ret.extend(["User-agent: ", agent, "\n"]) | ||
| ret.append("User-agent: {0}".format(agent)) |
Member
There was a problem hiding this comment.
f-strings can be used in 3.6+.
Lib/urllib/robotparser.py
Outdated
| for line in self.rulelines: | ||
| ret.extend([str(line), "\n"]) | ||
| return ''.join(ret) | ||
| ret.append(str(line)) |
Member
There was a problem hiding this comment.
Or just
ret.extend(map(str, self.rulelines))
serhiy-storchaka
added
type-bug
Contributor
Author
|
All suggestions have been implemented, I have no strong opinions on any of them. I also added a news entry. Is backporting part of this PR, or do you merge this into 3.8 and then create separate issues for the other python branches? Is that something that I can help with? |
serhiy-storchaka
approved these changes
May 13, 2018
Member
serhiy-storchaka
left a comment
serhiy-storchaka
left a comment
There was a problem hiding this comment.
LGTM.
Just add your credits.
| @@ -0,0 +1,3 @@ | |||
| The urllib.robotparser's ``__str__`` representation now includes wildcard | |||
| entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra | |||
| newlines that were being appended to the end of the string. | |||
Member
There was a problem hiding this comment.
Please add "Patch by yourname." and add your name into Misc/ACKS.
Contributor
Author
|
Cool, I added my name to the news entry. I'm already in the ACKS file so all good there. |
Contributor
|
Thanks @michael-lazar for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 2.7, 3.6, 3.7. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
May 14, 2018
…GH-5711) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra newlines that were being appended to the end of the string. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]>
|
GH-6795 is a backport of this pull request to the 3.7 branch. |
Contributor
|
Sorry, @michael-lazar and @serhiy-storchaka, I could not cleanly backport this to |
|
GH-6796 is a backport of this pull request to the 3.6 branch. |
miss-islington
pushed a commit
to miss-islington/cpython
that referenced
this pull request
May 14, 2018
…GH-5711) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. Also removes extra newlines that were being appended to the end of the string. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]>
serhiy-storchaka
pushed a commit
that referenced
this pull request
May 14, 2018
…H-5711) (GH-6795) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]>
miss-islington
added a commit
to miss-islington/cpython
that referenced
this pull request
May 14, 2018
…ythonGH-5711) (pythonGH-6795) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]> (cherry picked from commit c3fa1f2) Co-authored-by: Miss Islington (bot) <[email protected]>
serhiy-storchaka
pushed a commit
to serhiy-storchaka/cpython
that referenced
this pull request
May 14, 2018
…ythonGH-5711) (pythonGH-6795) The robotparser's __str__ representation now includes wildcard entries. (cherry picked from commit c3fa1f2) Co-authored-by: Michael Lazar <[email protected]>.
miss-islington
added a commit
to miss-islington/cpython
that referenced
this pull request
May 14, 2018
…ythonGH-5711) (pythonGH-6795) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit bd08a0a) Co-authored-by: Michael Lazar <[email protected]> (cherry picked from commit c3fa1f2) Co-authored-by: Miss Islington (bot) <[email protected]>
serhiy-storchaka
pushed a commit
that referenced
this pull request
May 14, 2018
…H-5711) (GH-6795) (GH-6818) The urllib.robotparser's __str__ representation now includes wildcard entries and the "Crawl-delay" and "Request-rate" fields. (cherry picked from commit c3fa1f2) Co-authored-by: Michael Lazar <[email protected]>
serhiy-storchaka
added a commit
that referenced
this pull request
May 14, 2018
GH-6795) (GH-6817) The robotparser's __str__ representation now includes wildcard entries. (cherry picked from commit c3fa1f2) Co-authored-by: Michael Lazar <[email protected]>.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The RobotFileParser's string representation was incomplete and missing some valid rule lines.
https://bugs.python.org/issue32861
https://bugs.python.org/issue32861