int(s), float(s) and others may cause segmentation fault

[fenrrir]

BPO	34087
Nosy	@vstinner, @ezio-melotti, @fenrrir, @methane, @jonashaag, @serhiy-storchaka, @miss-islington
PRs	bpo-34087: Fix buffer overflow in int(s) and similar functions #8274 [3.7] bpo-34087: Fix buffer overflow in int(s) and similar functions (GH-8274) #8279 [3.6] bpo-34087: Backport tests added by (GH-8274) #8280
Files	testproj.tar.gz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-14.03:59:56.073>
created_at = <Date 2018-07-10.19:36:06.259>
labels = ['interpreter-core', '3.7', '3.8', 'type-crash']
title = 'int(s), float(s) and others may cause segmentation fault'
updated_at = <Date 2018-07-14.07:38:21.790>
user = 'https://github.com/fenrrir'

bugs.python.org fields:

activity = <Date 2018-07-14.07:38:21.790>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-14.03:59:56.073>
closer = 'methane'
components = ['Interpreter Core']
creation = <Date 2018-07-10.19:36:06.259>
creator = 'fenrrir'
dependencies = []
files = ['47688']
hgrepos = []
issue_num = 34087
keywords = ['patch']
message_count = 34.0
messages = ['321394', '321404', '321406', '321407', '321450', '321452', '321491', '321533', '321560', '321561', '321566', '321567', '321577', '321578', '321581', '321586', '321588', '321589', '321590', '321591', '321593', '321594', '321596', '321602', '321603', '321605', '321613', '321616', '321617', '321618', '321621', '321641', '321642', '321645']
nosy_count = 7.0
nosy_names = ['vstinner', 'ezio.melotti', 'fenrrir', 'methane', 'jonash', 'serhiy.storchaka', 'miss-islington']
pr_nums = ['8274', '8279', '8280']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue34087'
versions = ['Python 3.7', 'Python 3.8']

[fenrrir]

• thread Support "bpo-" in Misc/NEWS #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x656d6f6e2236)
  frame #0: 0x000000010014c819 python3`visit_decref(op=0x0000656d6f6e222e, data=0x0000000000000000) at gcmodule.c:271 [opt]
  268 visit_decref(PyObject *op, void *data)
  269 {
  270 assert(op != NULL);
  -> 271 if (PyObject_IS_GC(op)) {
  272 PyGC_Head gc = AS_GC(op);
  273 / We're only interested in gc_refs for objects in the
  274 * generation being collected, which can be recognized
  Target 0: (python3) stopped.
  (lldb) bt
• thread Support "bpo-" in Misc/NEWS #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x656d6f6e2236)
  • frame #0: 0x000000010014c819 python3visit_decref(op=0x0000656d6f6e222e, data=0x0000000000000000) at gcmodule.c:271 [opt] frame #1: 0x000000010003af38 python3list_traverse(o=0x0000000116971608, visit=(python3visit_decref at gcmodule.c:269), arg=0x0000000000000000) at listobject.c:2574 [opt] frame #2: 0x000000010014aca3 python3collect [inlined] subtract_refs at gcmodule.c:296 [opt]
    frame bpo-29403: Fix mock's broken autospec behavior on method-bound builtin functions #3: 0x000000010014ac72 python3collect(generation=2, n_collected=0x00007ffeefbff0a8, n_uncollectable=0x00007ffeefbff0b0, nofail=0) at gcmodule.c:853 [opt] frame #4: 0x000000010014a8f5 python3PyGC_Collect [inlined] collect_with_callback(generation=2) at gcmodule.c:1028 [opt]
    frame Convert READMEs to rst #5: 0x000000010014a8cc python3PyGC_Collect at gcmodule.c:1573 [opt] frame #6: 0x000000010011f7a8 python3Py_FinalizeEx at pylifecycle.c:1087 [opt]
    frame bpo-29527: Disable broken Travis docs job #7: 0x0000000100149605 python3pymain_main(pymain=<unavailable>) at main.c:2664 [opt] frame #8: 0x000000010014a280 python3_Py_UnixMain(argc=5, argv=0x00007ffeefbff4b8) at main.c:2697 [opt]
    frame Fix some deprecation warnings in Doc/conf.py #9: 0x00007fff5869a015 libdyld.dylibstart + 1 frame #10: 0x00007fff5869a015 libdyld.dylibstart + 1

[methane]

It's difficult to investigate without reproducible example.

Do you use 3rd party extension module?
Typical cause of these crash is bug in 3rd party extensions.

[fenrrir]

Sorry for that. I’m not able to make a little example to reproduce this bug. It’s happens during Django tests on a very large code base. A interest thing is that not happens with “-X dev” parameters. Please, any suggestions how I can get more details about this bug?

[methane]

If example cannot be small, you can share the project on Github or other place.
If your project is private, you can create OSS version of reproducer.
If you can't create OSS reproducer, we can't debug it.

3rd party library list may helpful, but I'm not sure.

[fenrrir]

Running with -X faulthandler�����

Fatal Python error: Segmentation fault

Current thread 0x00007fff89cf2380 (most recent call first):
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/urls/resolvers.py", line 526 in resolve
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/urls/resolvers.py", line 500 in resolve
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/base.py", line 113 in _get_response
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/reversion/views.py", line 43 in do_revision_view
File "/Users/rodrigo/root/lib/python3.7/site-packages/reversion/middleware.py", line 51 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/utils/deprecation.py", line 95 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/exception.py", line 35 in inner
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/handlers/base.py", line 81 in get_response
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 138 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 467 in request
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 404 in generic
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 332 in get
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/client.py", line 517 in get
File "/Users/rodrigo/root/lib/python3.7/unittest/case.py", line 615 in run
File "/Users/rodrigo/root/lib/python3.7/unittest/case.py", line 663 in __call__
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/testcases.py", line 206 in __call__
File "/Users/rodrigo/root/lib/python3.7/unittest/suite.py", line 122 in run
File "/Users/rodrigo/root/lib/python3.7/unittest/suite.py", line 84 in __call__
File "/Users/rodrigo/root/lib/python3.7/unittest/runner.py", line 176 in run
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/runner.py", line 569 in run_suite
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/test/runner.py", line 603 in run_tests
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/commands/test.py", line 59 in handle
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/base.py", line 335 in execute
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/base.py", line 288 in run_from_argv
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/commands/test.py", line 26 in run_from_argv
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/init.py", line 365 in execute
File "/Users/rodrigo/root/lib/python3.7/site-packages/django/core/management/init.py", line 371 in execute_from_command_line
File "manage.py", line 10 in <module>
Segmentation fault: 11

The Django code:

    def resolve(self, path):
        path = str(path)  # path may be a reverse_lazy object
        tried = []
        match = self.pattern.match(path)
        if match:
            new_path, args, kwargs = match
            for pattern in self.url_patterns:
                try:
                    sub_match = pattern.resolve(new_path)
                except Resolver404 as e:
                    sub_tried = e.args[0].get('tried')
                    if sub_tried is not None:
                        tried.extend([pattern] + t for t in sub_tried)
                    else:
                        tried.append([pattern])
                else:
                    if sub_match:
                        # Merge captured arguments in match with submatch
                        sub_match_dict = dict(kwargs, **self.default_kwargs)
                        # Update the sub_match_dict with the kwargs from the sub_match.
                        sub_match_dict.update(sub_match.kwargs)
                        # If there are *any* named groups, ignore all non-named groups.
                        # Otherwise, pass all non-named arguments as positional arguments.
                        sub_match_args = sub_match.args
                        if not sub_match_dict:
                            sub_match_args = args + sub_match.args
                        return ResolverMatch(
                            sub_match.func,
                            sub_match_args,
                            sub_match_dict,
                            sub_match.url_name,
                            [self.app_name] + sub_match.app_names,
                            [self.namespace] + sub_match.namespaces,
                        )
                    tried.append([pattern]) # Segfault happens here!
            raise Resolver404({'tried': tried, 'path': new_path})
        raise Resolver404({'path': path})

According to faulthandler the segmentation fault happens on tried.append([pattern]).

[vstinner]

Please try to set PYTHONMALLOC=debug environment variable and run again your test. I hope that Python will be able to catch a memory corruption somewhere. If it's the case, you can try:

PYTHONMALLOC=debug python3 -X tracemalloc=10 (...)

To see where the corrupted object has been allocated.

[fenrrir]

Unfortunately with 'PYTHONMALLOC=debug' the segmentation fault do not happen.

[methane]

Would you provide list of extension modules?
What happens when removing some of them?

[jonashaag]

I also have a segfault that goes away with malloc debugging. Not sure if it's the same issue.

My extension modules are

venv/lib/python3.7/site-packages//_yaml.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//netifaces.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_webp.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_imagingft.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_imagingcms.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_imaging.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_imagingmath.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_imagingtk.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//PIL/_imagingmorph.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//lxml/builder.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//lxml/_elementpath.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//lxml/html/diff.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//lxml/html/clean.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//lxml/etree.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//lxml/objectify.cpython-37m-darwin.so
venv/lib/python3.7/site-packages//coverage/tracer.cpython-37m-darwin.so

Unfortunately I can't test the application without any of these, but maybe we can match my list with fenrrir's.

[jonashaag]

Btw my segfault is from Django too, but that may just be a coincidence

[fenrrir]

Extensions:

./_cffi_backend.cpython-37m-darwin.so
./_mssql.cpython-37m-darwin.so
./_yaml.cpython-37m-darwin.so
./Crypto/Cipher/_AES.cpython-37m-darwin.so
./Crypto/Cipher/_ARC2.cpython-37m-darwin.so
./Crypto/Cipher/_ARC4.cpython-37m-darwin.so
./Crypto/Cipher/_Blowfish.cpython-37m-darwin.so
./Crypto/Cipher/_CAST.cpython-37m-darwin.so
./Crypto/Cipher/_DES.cpython-37m-darwin.so
./Crypto/Cipher/_DES3.cpython-37m-darwin.so
./Crypto/Cipher/_XOR.cpython-37m-darwin.so
./Crypto/Hash/_MD2.cpython-37m-darwin.so
./Crypto/Hash/_MD4.cpython-37m-darwin.so
./Crypto/Hash/_RIPEMD160.cpython-37m-darwin.so
./Crypto/Hash/_SHA224.cpython-37m-darwin.so
./Crypto/Hash/_SHA256.cpython-37m-darwin.so
./Crypto/Hash/_SHA384.cpython-37m-darwin.so
./Crypto/Hash/_SHA512.cpython-37m-darwin.so
./Crypto/PublicKey/_fastmath.cpython-37m-darwin.so
./Crypto/Util/_counter.cpython-37m-darwin.so
./Crypto/Util/strxor.cpython-37m-darwin.so
./cryptography/hazmat/bindings/_constant_time.abi3.so
./cryptography/hazmat/bindings/_openssl.abi3.so
./cryptography/hazmat/bindings/_padding.abi3.so
./Cython/Compiler/Code.cpython-37m-darwin.so
./Cython/Compiler/FlowControl.cpython-37m-darwin.so
./Cython/Compiler/FusedNode.cpython-37m-darwin.so
./Cython/Compiler/Lexicon.cpython-37m-darwin.so
./Cython/Compiler/Parsing.cpython-37m-darwin.so
./Cython/Compiler/Pythran.cpython-37m-darwin.so
./Cython/Compiler/Scanning.cpython-37m-darwin.so
./Cython/Compiler/Visitor.cpython-37m-darwin.so
./Cython/Plex/Actions.cpython-37m-darwin.so
./Cython/Plex/Scanners.cpython-37m-darwin.so
./Cython/Runtime/refnanny.cpython-37m-darwin.so
./Cython/StringIOTree.cpython-37m-darwin.so
./Cython/Tempita/_tempita.cpython-37m-darwin.so
./lazy_object_proxy/cext.cpython-37m-darwin.so
./lxml/_elementpath.cpython-37m-darwin.so
./lxml/builder.cpython-37m-darwin.so
./lxml/etree.cpython-37m-darwin.so
./lxml/html/clean.cpython-37m-darwin.so
./lxml/html/diff.cpython-37m-darwin.so
./lxml/objectify.cpython-37m-darwin.so
./markupsafe/_speedups.cpython-37m-darwin.so
./PIL/_imaging.cpython-37m-darwin.so
./PIL/_imagingcms.cpython-37m-darwin.so
./PIL/_imagingmath.cpython-37m-darwin.so
./PIL/_imagingmorph.cpython-37m-darwin.so
./PIL/_imagingtk.cpython-37m-darwin.so
./psycopg2/_psycopg.cpython-37m-darwin.so
./pymssql.cpython-37m-darwin.so
./reportlab/graphics/_renderPM.cpython-37m-darwin.so
./reportlab/lib/_rl_accel.cpython-37m-darwin.so
./setproctitle.cpython-37m-darwin.so
./simplejson/_speedups.cpython-37m-darwin.so
./wrapt/_wrappers.cpython-37m-darwin.so

requirements.txt

alabaster==0.7.11
amqp==2.2.2
appnope==0.1.0
arrow==0.12.0
asn1crypto==0.24.0
astroid==1.6.5
Babel==2.6.0
beautifulsoup4==4.6.0
billiard==3.5.0.3
boto3==1.7.24
botocore==1.10.24
celery==4.1.0
certifi==2018.4.16
cffi==1.11.2
chardet==3.0.4
colorama==0.3.9
colorful==0.4.0
configparser==3.5.0
coreapi==2.3.3
coreschema==0.0.4
cryptography==2.2.2
cssselect==1.0.3
Cython==0.28.2
decorator==4.3.0
dj-database-url==0.4.2
Django==2.0.6
django-braces==1.12.0
django-celery-beat==1.1.1
django-celery-results==1.0.1
django-ckeditor==5.2.1
django-colorful==1.2
django-crispy-forms==1.6.1
django-extensions==1.9.8
django-extra-views==0.9.0
django-filter==1.0.4
django-filters==0.2.1
django-formset-js==0.5.0
django-formtools==2.1
django-fsm==2.6.0
django-jquery-js==3.1.1
django-js-asset==1.1.0
django-localflavor==1.6.2
django-model-utils==3.1.1
django-mptt==0.9.0
django-pagination==1.0.10
django-querysetsequence==0.8
django-recaptcha2==1.0.3
django-rest-swagger==2.1.2
django-reversion==2.0.13
django-reversion-compare==0.8.4
django-storages==1.6.6
django-tables2==1.17.1
django-tables2-reports==0.1.3
django-taggit==0.22.1
django-wkhtmltopdf==3.1.0
djangorestframework==3.7.3
djangorestframework-filters==0.10.2
djangorestframework-jwt==1.11.0
docutils==0.14
feedparser==5.2.1
google-api-python-client==1.6.4
gunicorn==19.7.1
html5lib==1.0.1
httplib2==0.10.3
idna==2.6
imagesize==0.7.1
ipaddress==1.0.19
ipdb==0.10.3
ipython==6.2.1
ipython-genutils==0.2.0
isort==4.2.15
itypes==1.1.0
jedi==0.12.1
Jinja2==2.10
jmespath==0.9.3
kombu==4.2.1
lazy-object-proxy==1.3.1
ldap3==2.4
lxml==4.1.1
MarkupSafe==1.0
mccabe==0.6.1
MechanicalSoup==0.10.0
minio==4.0.0
model-mommy==1.5.0
oauth2client==4.1.2
olefile==0.45.1
openapi-codec==1.3.2
parso==0.3.1
pathlib2==2.3.0
pbr==3.1.1
pdfrw==0.4
pexpect==4.3.1
pickleshare==0.7.4
Pillow==4.3.0
prompt-toolkit==1.0.15
psycopg2==2.7.3.2
ptyprocess==0.5.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.18
pycpfcnpj==1.2
pycrypto==2.6.1
Pygments==2.2.0
PyJWT==1.6.4
pylint==1.8.1
pyOpenSSL==17.5.0
PyPDF2==1.26.0
pyquery==1.3.0
python-dateutil==2.6.1
python-logstash==0.4.6
python-magic==0.4.15
python-memcached==1.58
pytz==2018.4
PyYAML==3.13
qrcode==5.3
raven==6.4.0
reportlab==3.4.0
requests==2.18.4
rsa==3.4.2
s3transfer==0.1.13
setproctitle==1.1.10
simplegeneric==0.8.1
simplejson==3.13.2
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.6.5
sphinxcontrib-websupport==1.1.0
suds-py3==1.3.3.0
traitlets==4.3.2
typing==3.6.4
Unidecode==0.4.21
uritemplate==3.0.0
urllib3==1.22
vine==1.1.4
wcwidth==0.1.7
webencodings==0.5.1
wrapt==1.10.11
xlrd==1.1.0
xlutils==2.0.0
xlwt==1.3.0
xmljson==0.1.9