Skip to content

bpo-32583: Fix possible crashing in builtin Unicode decoders #5325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhangyangyu merged 2 commits into from
Jan 31, 2018
Merged

bpo-32583: Fix possible crashing in builtin Unicode decoders #5325

zhangyangyu merged 2 commits into from
Jan 31, 2018

Conversation

@zhangyangyu
Copy link
Member

@zhangyangyu zhangyangyu commented Jan 25, 2018 •
edited by bedevere-bot
Loading

@zhangyangyu zhangyangyu added type-bug An unexpected behavior, bug, or error needs backport to 3.6 labels Jan 25, 2018
@zhangyangyu zhangyangyu changed the title bpo-32583: Fix possible crashing in decode error handler bpo-32583: Fix possible crashing in builtin Unicode decoders Jan 26, 2018
@zhangyangyu
bpo-32583: Fix possible crashing in builtin Unicode decoders
1c7c83a 
When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash.
@zhangyangyu zhangyangyu merged commit 2c7fd46 into python:master Jan 31, 2018
@bedevere-bot
Copy link

bedevere-bot commented Jan 31, 2018

@zhangyangyu: Please replace # with GH- in the commit message next time. Thanks!

@miss-islington
Copy link
Contributor

miss-islington commented Jan 31, 2018

Thanks @zhangyangyu for the PR 🌮🎉.. I'm working now to backport this PR to: 3.6.
🐍🍒⛏🤖

@zhangyangyu zhangyangyu deleted the issue32583 branch January 31, 2018 12:48
@miss-islington
Copy link
Contributor

miss-islington commented Jan 31, 2018

Sorry, @zhangyangyu, I could not cleanly backport this to 3.6 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 2c7fd46e11333ef5e5cce34212f7d087694f3658 3.6

zhangyangyu added a commit to zhangyangyu/cpython that referenced this pull request Jan 31, 2018
@zhangyangyu
[3.6] bpo-32583: Fix possible crashing in builtin Unicode decoders (p…
61f7c0a 
…ythonGH-5325)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash..
(cherry picked from commit 2c7fd46)
@bedevere-bot
Copy link

bedevere-bot commented Jan 31, 2018

GH-5459 is a backport of this pull request to the 3.6 branch.

zhangyangyu added a commit that referenced this pull request Jan 31, 2018
@zhangyangyu
[3.6] bpo-32583: Fix possible crashing in builtin Unicode decoders (G…
ea94fce 
…H-5325) (#5459)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash..
(cherry picked from commit 2c7fd46)
sibiryakov
sibiryakov reviewed Jan 31, 2018
View reviewed changes
Objects/unicodeobject.c
}
new_inptr = *input + newpos;
if (*inend - new_inptr > remain) {
/* We don't know the decoding algorithm here so we make the worst
Copy link

@sibiryakov sibiryakov Jan 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error handler registered with codecs.register_error can produce any string as an output. This output will be inserted in the buffer you allocate. So I propose to expect that decoded output could be bigger and either check that and reallocate the buffer or raise a MemoryError exception with proper description.

Copy link
Member Author

@zhangyangyu zhangyangyu Jan 31, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review @sibiryakov . But the situation you mention has already been handled. See the previous condition replen - 1, if the handler returns any string longer than one, need_to_grow will be set and we will allocate more memory.

Copy link

@sibiryakov sibiryakov Jan 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great @zhangyangyu! Thank you!

ned-deily pushed a commit to ned-deily/cpython that referenced this pull request Jan 31, 2018
@zhangyangyu @ned-deily
bpo-32583: Fix possible crashing in builtin Unicode decoders (python#…
86fdad0 
…5325)

When using customized decode error handlers, it is possible for builtin decoders
to write out-of-bounds and then crash.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner Awaiting requested review from vstinner

@serhiy-storchaka serhiy-storchaka Awaiting requested review from serhiy-storchaka

1 more reviewer

@sibiryakov sibiryakov sibiryakov left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

type-bug An unexpected behavior, bug, or error

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zhangyangyu @bedevere-bot @miss-islington @sibiryakov @the-knights-who-say-ni