test_str.py crashes

[donmez]

BPO	1608
Nosy	@gvanrossum, @loewis, @theller, @gpshead
Files	backtrace.txt valgrind.txt valgrind-supp.txt fwrapv.patch wrap.patch wrap.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2007-12-13.22:38:59.753>
created_at = <Date 2007-12-13.10:09:56.859>
labels = ['tests', 'type-crash']
title = 'test_str.py crashes'
updated_at = <Date 2008-01-09.17:46:04.525>
user = 'https://bugs.python.org/donmez'

bugs.python.org fields:

activity = <Date 2008-01-09.17:46:04.525>
actor = 'gvanrossum'
assignee = 'loewis'
closed = True
closed_date = <Date 2007-12-13.22:38:59.753>
closer = 'gvanrossum'
components = ['Tests']
creation = <Date 2007-12-13.10:09:56.859>
creator = 'donmez'
dependencies = []
files = ['8936', '8937', '8940', '8942', '8944', '8945']
hgrepos = []
issue_num = 1608
keywords = []
message_count = 35.0
messages = ['58525', '58526', '58529', '58540', '58546', '58549', '58550', '58551', '58555', '58558', '58560', '58561', '58562', '58563', '58564', '58566', '58568', '58570', '58571', '58572', '58573', '58574', '58575', '58576', '58577', '58578', '58579', '58583', '58584', '58586', '58597', '58603', '58604', '58616', '59613']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'loewis', 'theller', 'gregory.p.smith', 'donmez']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue1608'
versions = ['Python 2.5']

[donmez]

Checkout Python 2.5 from release25-maint branch, revision 59479

Compiled with gcc 4.3.0 20071212 , make test crashes with the following
output

[... snip ...]
test_socket_ssl
test_socket_ssl skipped -- Use of the network' resource not enabled test_socketserver test_socketserver skipped -- Use of the network' resource not enabled
test_softspace
test_sort
test_sqlite
test_startfile
test_startfile skipped -- cannot import name startfile
test_str
make: *** [test] Segmentation fault

[donmez]

gdb backtrace, segfaulting test is Lib/test/test_str.py

[donmez]

Valgrind output, shows lots of invalid reads.

[gvanrossum]

What hardware and OS? 32 or 64 bit? What optimization level? Debug
build or not?

NB. Unless you used /Misc/valgrind-python.supp, the valgrind output is
useless.

[donmez]

Linux 2.6.18, x86, 32bit .

Executed valgrind with

valgrind --suppressions=./Misc/valgrind-python.supp -v ./python
./Lib/test/test_str.py

attached as valgrind-supp.txt it still shows lots of invalid reads.

Optimization level is -g -O3 which seems to be default as I didn't
specify CFLAGS.

[donmez]

--enable-pydebug fixes the crash it might be that some uninitialized
variable doesn't take affect unless optimized as valgrind output shows
many of this.

[gvanrossum]

Looks like expandtabs() has a problem. Can you boil it down to a single
call?

[gvanrossum]

BTW is this a released version of GCC? If not, you might want to file
the bug with the GCC project...

[donmez]

This is a soon to be released GCC though I won't deny it has
regressions, but note that extra optimizations already uncovered bugs in
other software.

And unless I can get a minimal C testcase, GCC bug will be worthless.

Exact crashling call is string_tests.py line 255 :

self.checkraises(OverflowError,
                             '\ta\n\tb', 'expandtabs', sys.maxint)

Commenting out this fixes the crash.

[gvanrossum]

This is a soon to be released GCC though I won't deny it has
regressions, but note that extra optimizations already uncovered bugs in
other software.

And the GCC authors always win these cases, C standard in hand.

And unless I can get a minimal C testcase, GCC bug will be worthless.

Exact crashling call is string_tests.py line 255 :

self.checkraises(OverflowError,
'\ta\n\tb', 'expandtabs', sys.maxint)

Commenting out this fixes the crash.

If you want for me to debug this myself it'll be ages. it looks like
the crashing call is

'\ta\n\tb'.expandtabs(2147483647)

Can you confirm that this crashes? If it does, you should be able to
use gdb to step through expandtabs() and hopefully analyze the
problem.

[gvanrossum]

Actually, looking at the sample code and the string_expandtabs()
implementation it's clear what happened: the test for overflow on line
3318 or 3331 or 3339 must have been optimized out by GCC.

This is very inconvenient because lots of buffer overflow protection
uses similar code; this means that code that has been audited and fixed
in the past will again be vulnerable after compilation by GCC 4.3.

I'm going to ask Martin von Loewis to give an opinion on this.

Thanks for bringing this up!

[donmez]

Indeed you are correct,

>> '\ta\n\tb'.expandtabs(2147483647)

Program received signal SIGSEGV, Segmentation fault.
string_expandtabs (self=0xb7ba7c60, args=0xb7ba7dec) at
Objects/stringobject.c:3358
3358 *q++ = ' ';
(gdb) bt
#0 string_expandtabs (self=0xb7ba7c60, args=0xb7ba7dec) at
Objects/stringobject.c:3358
#1 0xb7e1b6dd in PyCFunction_Call (func=0xb7ba72ec, arg=0xb7ba7dec,
kw=0x0) at Objects/methodobject.c:73
#2 0xb7e6d05b in PyEval_EvalFrameEx (f=0x80cc40c, throwflag=0) at
Python/ceval.c:3569
#3 0xb7e6ec35 in PyEval_EvalCodeEx (co=0xb7b987b8, globals=0xb7bceacc,
locals=0xb7bceacc, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0,
defcount=0,
closure=0x0) at Python/ceval.c:2832
#4 0xb7e6ee53 in PyEval_EvalCode (co=0xb7b987b8, globals=0xb7bceacc,
locals=0xb7bceacc) at Python/ceval.c:494
#5 0xb7e8ea8d in PyRun_InteractiveOneFlags (fp=0xb7d48420,
filename=0xb7ec589c "<stdin>", flags=0xbff3c6a8) at Python/pythonrun.c:1273
#6 0xb7e8ecc6 in PyRun_InteractiveLoopFlags (fp=0xb7d48420,
filename=0xb7ec589c "<stdin>", flags=0xbff3c6a8) at Python/pythonrun.c:723
#7 0xb7e8f427 in PyRun_AnyFileExFlags (fp=0xb7d48420,
filename=0xb7ec589c "<stdin>", closeit=0, flags=0xbff3c6a8) at
Python/pythonrun.c:692
#8 0xb7e9a347 in Py_Main (argc=0, argv=0xbff3c774) at Modules/main.c:523
#9 0x080485a2 in main (argc=538976288, argv=0x20202020) at
./Modules/python.c:23

Though I am not exactly sure how to proceed from here.