Skip to content

Conversation

@yihong0618
Copy link
Contributor

@yihong0618 yihong0618 commented Sep 11, 2025

This make python -m base64 the same behavior as ast or json

@yihong0618 yihong0618 changed the title gh-138775: fix handle python -m base64 stdin correct with EOF single. gh-138775: fix handle python -m base64 stdin correct with EOF signal. Sep 11, 2025
@picnixz picnixz changed the title gh-138775: fix handle python -m base64 stdin correct with EOF signal. gh-138775: fix handle python -m base64 stdin correct with EOF signal Sep 11, 2025
@aisk
Copy link
Contributor

aisk commented Sep 11, 2025

I’m concerned that the current implementation reads the input in chunks, this allows the function to process the input as a stream via a pipe and avoids using too much memory with large inputs:

cpython/Lib/base64.py

Lines 531 to 537 in 4978bfc

def encode(input, output):
"""Encode a file; input and output are binary files."""
while s := input.read(MAXBINSIZE):
while len(s) < MAXBINSIZE and (ns := input.read(MAXBINSIZE-len(s))):
s += ns
line = binascii.b2a_base64(s)
output.write(line)

With this change, however, all input is read into memory, which breaks the chunked behavior.

@yihong0618
Copy link
Contributor Author

I’m concerned that the current implementation reads the input in chunks, this allows the function to process the input as a stream via a pipe and avoids using too much memory with large inputs. :

cpython/Lib/base64.py

Lines 531 to 537 in 4978bfc

def encode(input, output):
"""Encode a file; input and output are binary files."""
while s := input.read(MAXBINSIZE):
while len(s) < MAXBINSIZE and (ns := input.read(MAXBINSIZE-len(s))):
s += ns
line = binascii.b2a_base64(s)
output.write(line)

With this change, however, all input is read into memory, which breaks the chunked behavior.

Thanks for the comments

I will checked if isatty to check
if
echo 1234 | ./python.exe -m base64
or
./python.exe -m base64
and do not break the old method

and in stdin old behavior is wrong, so I think is fine.
will check isatty if can check or not.

@yihong0618
Copy link
Contributor Author

keep the old behavior and use isatty to check
as doc https://docs.python.org/3/library/sys.html#sys.stdin

@yihong0618
Copy link
Contributor Author

Thank you very much for the review again, I will try to learn how to express the change more clearly.
and do better next time, thanks again

@picnixz
Copy link
Member

picnixz commented Sep 11, 2025

I will try to learn how to express the change more clearly.
and do better next time, thanks again

Don't worry. Considering English isn't your first language, you can just leave formulation matters to others.

@yihong0618

This comment was marked as resolved.

@ZeroIntensity

This comment was marked as resolved.

@yihong0618

This comment was marked as resolved.

Signed-off-by: yihong0618 <[email protected]>
@gpshead gpshead added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Nov 11, 2025
@gpshead gpshead enabled auto-merge (squash) November 11, 2025 23:22
@gpshead
Copy link
Member

gpshead commented Nov 11, 2025

I went with this PR after playing around with the BufferedReader connected to a terminal - it really is difficult to tell when you have an EOF in that scenario vs what the base64.encode() code wants to do when reading fixed sized blocks at a time in order to produce the fixed sized output lines. subsequent .read() calls after a short read or even after a 0 length read (the normal EOF indicator) still block and could later return more data (ugh, terminals!). So doing the isatty check does seem most practical here given that the io stack.

@gpshead gpshead merged commit f5c2a41 into python:main Nov 11, 2025
54 checks passed
@miss-islington-app
Copy link

Thanks @yihong0618 for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 11, 2025
… signal (pythonGH-138776)

* fix: handle  stdin correct with EOF single.
* fix: flollow the comments when pipe stdin use buffer
* Apply suggestions from code review
* fix: apply review comments in Lib/base64.py
* fix: address comments
* Reword comment and NEWS entry.

---------
(cherry picked from commit f5c2a41)

Co-authored-by: yihong <[email protected]>
Signed-off-by: yihong0618 <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
@bedevere-app
Copy link

bedevere-app bot commented Nov 11, 2025

GH-141432 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Nov 11, 2025
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 11, 2025
… signal (pythonGH-138776)

* fix: handle  stdin correct with EOF single.
* fix: flollow the comments when pipe stdin use buffer
* Apply suggestions from code review
* fix: apply review comments in Lib/base64.py
* fix: address comments
* Reword comment and NEWS entry.

---------
(cherry picked from commit f5c2a41)

Co-authored-by: yihong <[email protected]>
Signed-off-by: yihong0618 <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
@bedevere-app
Copy link

bedevere-app bot commented Nov 11, 2025

GH-141433 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Nov 11, 2025
gpshead added a commit that referenced this pull request Nov 12, 2025
…F signal (GH-138776) (#141433)

gh-138775: fix handle `python -m base64` stdin correct with EOF signal (GH-138776)

* fix: handle  stdin correct with EOF single.
* fix: flollow the comments when pipe stdin use buffer
* Apply suggestions from code review
* fix: apply review comments in Lib/base64.py
* fix: address comments
* Reword comment and NEWS entry.

---------
(cherry picked from commit f5c2a41)

Signed-off-by: yihong0618 <[email protected]>
Co-authored-by: yihong <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
gpshead added a commit that referenced this pull request Nov 12, 2025
…F signal (GH-138776) (#141432)

gh-138775: fix handle `python -m base64` stdin correct with EOF signal (GH-138776)

* fix: handle  stdin correct with EOF single.
* fix: flollow the comments when pipe stdin use buffer
* Apply suggestions from code review
* fix: apply review comments in Lib/base64.py
* fix: address comments
* Reword comment and NEWS entry.

---------
(cherry picked from commit f5c2a41)

Signed-off-by: yihong0618 <[email protected]>
Co-authored-by: yihong <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
… signal (pythonGH-138776)

* fix: handle  stdin correct with EOF single.
* fix: flollow the comments when pipe stdin use buffer
* Apply suggestions from code review
* fix: apply review comments in Lib/base64.py
* fix: address comments
* Reword comment and NEWS entry.

---------

Signed-off-by: yihong0618 <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Peter Bierma <[email protected]>
Co-authored-by: Gregory P. Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants