Windows: 3.3.0-rc2.msi: test_buffer fails

[skrah]

BPO	15993
Nosy	@loewis, @vstinner, @larryhastings, @tjguk, @briancurtin, @zware, @zooba
Files	profile.bat profiletests.txt issue15993.diff ull_vctest.diff: Quick-and-dirty alternative version of PyLong_AsUnsignedLongLong(). Just for testing.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-07-05.02:07:22.785>
created_at = <Date 2012-09-20.20:31:49.341>
labels = ['interpreter-core', 'build', 'OS-windows']
title = 'Windows: 3.3.0-rc2.msi: test_buffer fails'
updated_at = <Date 2015-07-05.02:07:22.784>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2015-07-05.02:07:22.784>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2015-07-05.02:07:22.785>
closer = 'steve.dower'
components = ['Interpreter Core', 'Windows']
creation = <Date 2012-09-20.20:31:49.341>
creator = 'skrah'
dependencies = []
files = ['27251', '27252', '27253', '35681']
hgrepos = []
issue_num = 15993
keywords = ['patch']
message_count = 32.0
messages = ['170844', '170858', '170873', '170916', '170917', '170918', '170919', '170920', '170923', '170924', '170929', '170935', '170971', '170973', '170981', '170985', '170988', '170989', '220180', '220183', '220531', '220545', '220551', '220557', '220894', '220932', '220992', '221380', '221381', '242548', '246282', '246287']
nosy_count = 9.0
nosy_names = ['loewis', 'vstinner', 'larry', 'tim.golden', 'nadeem.vawda', 'brian.curtin', 'BreamoreBoy', 'zach.ware', 'steve.dower']
pr_nums = []
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue15993'
versions = ['Python 3.5']

[skrah]

I've installed 3.3.0-rc2 on Windows-7 64-bit using the msi installer.
I'm getting these failures in test_buffer, but I can *not* reproduce
them when I build Win-32/pgo/python.exe from source:

======================================================================
FAIL: test_memoryview_assign (test.test_buffer.TestBufferProtocol)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python33\lib\test\test_buffer.py", line 2863,
    self.assertEqual(m[i], 8)
AssertionError: 34359738368 != 8

======================================================================
FAIL: test_memoryview_struct_module (test.test_buffer.TestBufferProtocol)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Program Files (x86)\Python33\lib\test\test_buffer.py", line 2476,
    self.assertEqual(m[0], nd[0])
AssertionError: 15080797365275624638 != 6838299039298601293

[skrah]

Both lzma and memoryview use PyLong_AsUnsignedLongLong() in the
affected code paths. I get this with the msi installed python.exe:

>>> import array
>>> a = array.array('Q', [1,2,3,4])
>>> m = memoryview(a)
>>> m[0] = 4
>>> m[0]
17179869184
>>>

And the correct result with the self-compiled (PGO) python.exe:

>>> import array
>>> a = array.array('Q', [1,2,3,4])
>>> m = memoryview(a)
>>> m[0] = 4
>>> m[0]
4

[skrah]

The high and low words of the 64-bit value are switched:

>>> a = array.array('Q', [1])
>>> m = memoryview(a)
>>> m[0]= 2**32+5
>>> m[0]
21474836481
>>> struct.unpack_from('8s', m, 0)
(b'\x01\x00\x00\x00\x05\x00\x00\x00',)

Can anyone reproduce this in a source build? I think this should
be a blocker.

[vstinner]

Issue bpo-15995 was marked as a duplicate of this issue. Copy of its initial message (msg170853):

This is similar to bpo-15993: With the installed Python from the rc2-msi
test_lzma fails. I cannot reproduce the failure with python.exe (PGO)
compiled from source:

======================================================================                                   
ERROR: test__decode_filter_properties (test.test_lzma.MiscellaneousTestCase)                             
----------------------------------------------------------------------                                   
Traceback (most recent call last):                                                                       
  File "C:\Python33\lib\test\test_lzma.py", line 1105, in test__decode_filter_properties                 
    lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00")                                                             
ValueError: Invalid filter ID: 4611686018427387905                                                       
                                                                                                         
======================================================================                                   
ERROR: test_filter_properties_roundtrip (test.test_lzma.MiscellaneousTestCase)                           
----------------------------------------------------------------------                                   
Traceback (most recent call last):                                                                       
  File "C:\Python33\lib\test\test_lzma.py", line 1114, in test_filter_properties_roundtrip               
    lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00")                                                             
ValueError: Invalid filter ID: 4611686018427387905

---

[vstinner]

I fail to reproduce the issue on Windows 7 (Version 6.0.1, number 7601, Service Pack 1):
---
Microsoft Windows [version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. Tous droits réservés.

C:\Users\haypo>cd \python33

C:\Python33>python.exe
Python 3.3.0rc2 (v3.3.0rc2:88a0792e8ba3, Sep  9 2012, 10:13:38) [MSC v.1600 64 b
it (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\Python33>python.exe -m test test_buffer test_lzma
[1/2] test_buffer
[2/2] test_lzma
All 2 tests OK.
---
I'm running Windows 7 in KVM, my host CPU is a Intel i7-2600.

I'm getting these failures in test_buffer,
but I can *not* reproduce them when I build
Win-32/pgo/python.exe from source:

The issue looks to be specific to 64 bits binaries. You need the professional version of Visual Studio 10. The express version doesn't support 64 bits. I only have the Express version.

[loewis]

Declaring this a release blocker is technically difficult. If it is a release blocker, further releases cannot be done until it is resolved. Since it is an issue with the binary only, the only possible way to resolve this is with a release. So declaring this a release blocker essentially deadlocks the release.

[loewis]

For the record, the released binary is not just a PGO build, but has been trained with the attached training script.

[vstinner]

You need the professional version of Visual Studio 10.
The express version doesn't support 64 bits. I only have the
Express version.

Ah yes, I now remember my issue with VS10 Express: when I set the project to 64 bits, I get such error popup:
http://www.haypocalc.com/tmp/visual_studio_64bits.png

Which version should I try? Ultimate? Premium? Professional?

[skrah]

STINNER Victor <report@bugs.python.org> wrote:

Which version should I try? Ultimate? Premium? Professional?

Try Ultimate, it's AFAIK the only version these days that supports PGO.

[skrah]

Martin v. Löwis <report@bugs.python.org> wrote:

For the record, the released binary is not just a PGO build, but has
been trained with the attached training script.

Thanks. Now I can reproduce the issue with a source build.

[loewis]

I'm using Ultimate, but I think Professional should provide you with all required tools.

[skrah]

It's a bit late here already, but unless I'm missing something I think
this is an optimizer bug. I'm attaching a workaround for memoryview.c
and _lzmamodule.c.

[skrah]

System: Windows 7 64-bit
Build (unpatched): PCBuild\Win32-pgo\python.exe, trained with profile.bat

In the unpatched version, I stepped through this test case in the debugger:

import array
a = array.array('Q', [1])
m = memoryview(a)
m[0] = 1

At Objects/memoryobject.c:1572: llu == 1

At Objects/memoryobject.c:1782: pylong_as_llu returned value 4294967296

So I think it's pretty safe to say that this is indeed an optimizer bug.