site: Potential UnicodeDecodeError when handling pth file

[methane]

BPO	43214
Nosy	@vstinner, @methane
PRs	bpo-43214: site: Use UTF-8 for .pth files. #24837
Superseder	bpo-37364: Use io.open_code for .pth files

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-17.08:54:25.835>
created_at = <Date 2021-02-13.08:53:26.246>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.10', 'invalid', 'library']
title = 'site: Potential UnicodeDecodeError when handling pth file'
updated_at = <Date 2021-03-17.10:30:46.450>
user = 'https://github.com/methane'

bugs.python.org fields:

activity = <Date 2021-03-17.10:30:46.450>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-17.08:54:25.835>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2021-02-13.08:53:26.246>
creator = 'methane'
dependencies = []
files = []
hgrepos = []
issue_num = 43214
keywords = ['patch', 'easy', '3.8regression']
message_count = 3.0
messages = ['386916', '388911', '388916']
nosy_count = 2.0
nosy_names = ['vstinner', 'methane']
pr_nums = ['24837']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = '37364'
type = 'behavior'
url = 'https://bugs.python.org/issue43214'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[methane]

cpython/Lib/site.py

Line 160 in 4230bd5

f = io.TextIOWrapper(io.open_code(fullname))

        f = io.TextIOWrapper(io.open_code(fullname))

When default text encoding is not UTF-8 and pth file contains non-ASCII character, it will raise UnicodeDecodeError.

[methane]

locale-specific encoding is not good especially for Windows.
But we used it for a long time. Changing the encoding for pth files is breaking change.