fileinput.hook_compressed returning bytes from gz file

[mnewman]

BPO	5758
Nosy	@amauryfa, @jaraco, @merwok, @bitdancer, @methane
PRs	bpo-43712 : fileinput: Add encoding parameter #25272
Files	example.zip: ZIP file containing test example issue5758_fileinput_gzip_with_encoding.patch: Fixed issue fileinput.hook_compressed returning bytes from gz file #50008. fileinput.hook_compressed returns string (and not bytes) from gz file again. issue5758_fileinput_gzip_with_encoding_v2.patch: New version of the patch after reviewer's considerations

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = <Date 2021-04-14.05:13:39.981>
created_at = <Date 2009-04-14.23:48:01.306>
labels = ['type-bug', 'library', '3.10']
title = 'fileinput.hook_compressed returning bytes from gz file'
updated_at = <Date 2021-05-26.03:46:18.079>
user = 'https://bugs.python.org/mnewman'

bugs.python.org fields:

activity = <Date 2021-05-26.03:46:18.079>
actor = 'methane'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2021-04-14.05:13:39.981>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2009-04-14.23:48:01.306>
creator = 'mnewman'
dependencies = []
files = ['13688', '24807', '24815']
hgrepos = []
issue_num = 5758
keywords = ['patch']
message_count = 21.0
messages = ['85978', '111063', '155471', '155532', '155538', '155540', '155543', '155548', '155553', '155554', '155577', '155595', '155618', '155622', '155624', '348642', '394286', '394287', '394290', '394338', '394413']
nosy_count = 7.0
nosy_names = ['amaury.forgeotdarc', 'jaraco', 'eric.araujo', 'r.david.murray', 'mnewman', 'methane', 'tati_alchueyr']
pr_nums = ['25272']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5758'
versions = ['Python 3.10']

[mnewman]

The attached ZIP file contains "test.bat" which runs "test.py" with
Python 2.6 and Python 3.0.

Python 2.6 behaves as expected (see "py26.out"), since it returns
strings from both "mike.txt" and "mike.txt.gz". However, the same test
with Python 3.0 returns bytes from "mike.txt.gz", as shown in "py30.out":
Output: Hello from Mike.
Output: This is the second line.
Output: Why did the robot cross the road?
Output: b'Hello from Mike.'
Output: b'This is the second line.'
Output: b'Why did the robot cross the road?'

For reference, I tested this on Python versions:
Python 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit
(Intel)] on win32
Python 3.0.1 (r301:69561, Feb 13 2009, 20:04:18) [MSC v.1500 32 bit
(Intel)] on win32

[amauryfa]

gzip.open() only implements the "rb" mode, and returns bytes.
fileinput could certainly wrap it with a io.TextIOWrapper.

Then the encoding issue arises.
fileinput.FileInput should grow an "encoding" parameter instead of always relying on the default encoding.

[tatialchueyr]

Working on this

[tatialchueyr]

I had a lot of help of R. David Murray, could someone review our patch, please?

We are specially concerned about backward compatibility.

[merwok]

Thanks for the patch. I commented on not-so-important things on the code review site (which should have sent an email to Tatiana) and wanted to raise an issue here.

Is it best that the encoding used defaults to Python’s default encoding (UTF-8) or the I/O default encoding (depends on the OS and locale), like what open does?

[bitdancer]

I think it should use the same default as open, but frankly I couldn't remember the right way to achieve that. Suggestions welcome :)

[merwok]

We can use locale.getpreferredencoding, or be smarter and just pass down encoding=encoding to TextIOWrapper and let *it* call getpreferredencoding if it’s given a None :)

[tatialchueyr]

Excellent! Thanks for all remarks, Éric! I'll follow your suggestions and will submit a new patch.

[bitdancer]

Duh. I should have remembered that the io module used None to represent the default encoding. Thanks, Éric.

[merwok]

Nobody can remember everything: I tried in a shell and looked at the docs to find it :)

[tatialchueyr]

Improved patch, according to eric.araujo's suggestions and mnewman's guidance.

[merwok]

Thanks. Unless another core dev wants to do a complementary review I will slightly tweak the patch and commit it. I need to finish waking up and eat some food before I do that :)

Technically adding a new argument means that this is a new feature and cannot be applied to the stable 3.2 version, but something needs to be done for this bug in 3.2 too, like a recipe in the docs for a hook_compressed that returns strings (i.e. a function that calls fileinput.hook_compressed and wraps it in a TextIOWrapper), or at least a note to warn about this bug.