test_marshal: crash in Python 3.7b5 on Windows 10

[vstinner]

BPO	33720
Nosy	@pfmoore, @vstinner, @tjguk, @ned-deily, @zware, @serhiy-storchaka, @zooba, @miss-islington
PRs	bpo-33720: Improve test for the stack overflow in marshal.loads(). #7336 bpo-33720: Reduces maximum marshal recursion depth on release builds. #7401 [3.7] bpo-33720: Reduces maximum marshal recursion depth on release builds. (GH-7401) #7405 bpo-33720: Refactor marshalling/unmarshalling floats. #8071 [3.7] bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336) #8104 [3.6] bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336) #8105 [2.7] bpo-33720: Improve tests for the stack overflow in marshal.loads(). (GH-7336). #8107

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2018-07-24.08:07:53.063>
created_at = <Date 2018-05-31.15:13:35.091>
labels = ['3.8', '3.7', 'tests', 'OS-windows', 'type-crash']
title = 'test_marshal: crash in Python 3.7b5 on Windows 10'
updated_at = <Date 2018-07-24.08:07:53.062>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-07-24.08:07:53.062>
actor = 'serhiy.storchaka'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2018-07-24.08:07:53.063>
closer = 'serhiy.storchaka'
components = ['Tests', 'Windows']
creation = <Date 2018-05-31.15:13:35.091>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 33720
keywords = ['patch']
message_count = 29.0
messages = ['318323', '318325', '318326', '318327', '318328', '318329', '318330', '318331', '318442', '318497', '318689', '318690', '318693', '318695', '318696', '318698', '318714', '318771', '320053', '320221', '321012', '321054', '321055', '321059', '321084', '321087', '321089', '321092', '322279']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ned.deily', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'miss-islington']
pr_nums = ['7336', '7401', '7405', '8071', '8104', '8105', '8107']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue33720'
versions = ['Python 3.7', 'Python 3.8']

[vstinner]

Follow-up of bpo-33719.

C:\Users\vstinner\AppData\Local\Programs\Python\Python37>python.exe -m test test_marshal -v
== CPython 3.7.0b5 (v3.7.0b5:abb8802389, May 31 2018, 01:54:01) [MSC v.1913 64 bit (AMD64)]
== Windows-10-10.0.16299-SP0 little-endian
== cwd: C:\Users\vstinner\AppData\Local\Temp\test_python_3836
== CPU count: 2
== encodings: locale=cp1252, FS=utf-8
Run tests sequentially
0:00:00 [1/1] test_marshal
(...)
test_loads_2x_code (test.test_marshal.BugsTestCase) ... Windows fatal exception: stack overflow

Current thread 0x000003a0 (most recent call first):
File "C:\Users\vstinner\AppData\Local\Programs\Python\Python37\lib\unittest\case.py", line 178 in handle
File "C:\Users\vstinner\AppData\Local\Programs\Python\Python37\lib\unittest\case.py", line 743 in assertRaises
File "C:\Users\vstinner\AppData\Local\Programs\Python\Python37\lib\test\test_marshal.py", line 215 in test_loads_2x_code
(...)

Crashes in test_marshal is on old topic:

• bpo-1050
• bpo-2286
• bpo-25264
• bpo-22734
• bpo-27019

Current stack size: 2 million bytes (1.9 MiB)

PCbuild/python.vcxproj: <StackReserveSize>2000000</StackReserveSize>
PCbuild/pythonw.vcxproj: <StackReserveSize>2000000</StackReserveSize>

[vstinner]

I compiled the master branch of Python in release mode using VS2015 (MSC v.1912 64 bit) and I failed to reproduce the crash:

• PCbuild/build.bat -e -p x64
• python -m test -v test_marshal
• no crash

[vstinner]

I compiled the master branch of Python in release mode using VS2015 (MSC v.1912 64 bit) and I failed to reproduce the crash

I also failed to reproduce the crash in the 3.7 branch.

I guess that the python.org binary has been compiled differently.

[zooba]

The uploaded binary is compiled with PGO enabled (and trained on most of the test suite). I'll check it out - hopefully we don't need to do anything drastic and can get away with either a compiler update or disabling optimizations on a single function.

[zooba]

Ned, FYI

[vstinner]

priority: normal -> release blocker

I don't think that it's a release blocker. test_marshal does only crash on corner cases which should not occur on usual "valid" data.