xmlrpclib raises when trying to convert an int to string when unicode is available

[NathanWilliams]

BPO	26873
Nosy	@loewis, @serhiy-storchaka
Files	pybug.txt: server response xmlrpc_unsupported.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-05-04.08:29:17.563>
created_at = <Date 2016-04-28.01:49:22.674>
labels = ['type-bug', 'library']
title = 'xmlrpclib raises when trying to convert an int to string when unicode is available'
updated_at = <Date 2017-02-09.07:40:51.026>
user = 'https://bugs.python.org/NathanWilliams'

bugs.python.org fields:

activity = <Date 2017-02-09.07:40:51.026>
actor = 'Zsolt Endreffy'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-05-04.08:29:17.563>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2016-04-28.01:49:22.674>
creator = 'Nathan Williams'
dependencies = []
files = ['42636', '42643']
hgrepos = []
issue_num = 26873
keywords = ['patch']
message_count = 10.0
messages = ['264407', '264411', '264416', '264420', '264436', '264444', '264505', '264612', '264795', '287385']
nosy_count = 5.0
nosy_names = ['loewis', 'python-dev', 'serhiy.storchaka', 'Nathan Williams', 'Zsolt Endreffy']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26873'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[NathanWilliams]

I am using xmlrpclib against an internal xmlrpc server.
One of the responses returns integer values, and it raises an exception in "_stringify"

The code for _stringify is (xmlrpclib.py:180 in python2.7):

if unicode:
    def _stringify(string):
        # convert to 7-bit ascii if possible
        try:
            return string.encode("ascii")
        except UnicodeError:
            return string
else:
    def _stringify(string):
        return string

So when "unicode" is available, .encode is called on the parameter (which are the returned objects from the server) which fails for ints.

Without the unicode path it works fine, proven with the following monkey-patch:
xmlrpclib._stringify = lambda s: s

I am using the above patch as a workaround, but a fix to the library should be straightforward, simply checking for AttributeError in the except clause would solve it while retaining the existing functionality.

The traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1387, in _parse_response
    p.feed(response)
  File "/usr/lib/python2.6/xmlrpclib.py", line 601, in feed
    self._parser.Parse(data, 0)
  File "/usr/lib/python2.6/xmlrpclib.py", line 868, in end
    return f(self, join(self._data, ""))
  File "/usr/lib/python2.6/xmlrpclib.py", line 935, in end_struct
    dict[_stringify(items[i])] = items[i+1]
  File "/usr/lib/python2.6/xmlrpclib.py", line 176, in _stringify
    return string.encode("ascii")
AttributeError: 'int' object has no attribute 'encode'

[serhiy-storchaka]

Could you provide a response of your server (pass verbose=True to ServerProxy)?

[NathanWilliams]

I have attached the response.
As it is coming from our UMS, I had to redact a few values, but that shouldn't matter.

For reference, they were the host name of my email address, and the hashes of passwords etc.
Our UMS is a bit too chatty!

[serhiy-storchaka]

Thank you. Your server produces a response containing nonstandard type tag "ex:nil" [1].

<member><name>authority</name><value>ROLE_USER</value></member>
<member><name>fromDate</name><value><ex:nil/></value></member>
<member><name>userId</name><value><i4>15</i4></value></member>

xmlrpclib is unable to handle this tag, and error handling is poor. xmlrpclib can handle only standard types and "nil" [2].

What is worse, xmlrpclib silently ignores unsupported tag and interprets following name "userId" as a value of the name "fromDate", and following userId's value 15 as next name and so forth. Your workaround doesn't help since the result is totally broken. You get values as keys and keys as values.

What can we do with this issue?

1. Add better handling of unsupported tags. There is a bug, xmlrpc should detect this case and fail instead of producing unreliable result. This is applicable to 2.7.

2. Add support of some Apache extension tags. This is new feature and can be added in 3.6 only. Not all extension tags can be supported, "ex:serializable" is Java specific. This is separate issue.

Martin, do you agree?

[1] http://ws.apache.org/xmlrpc/types.html
[2] http://web.archive.org/web/20130120074804/http://ontosys.com/xml-rpc/extensions.php

[serhiy-storchaka]

Minimal reproducer:

>>> xmlrpclib.loads('<params><param><value><struct><member><name>a</name><value><ex:nil/></value></member><member><name>b</name><value><ex:nil/></value></member></struct></value></param></params>')
(({'a': 'b'},), None)

The workaround for your particular case, Nathan:

xmlrpclib.Unmarshaller.dispatch['ex:nil'] = xmlrpclib.Unmarshaller.dispatch['nil']

[serhiy-storchaka]

Proposed patch makes xmlrpc unmarshaller to be more strong and raise ValueError instead of returning incorrect value when encounters with unsupported value type. The unmarshaller still skips unknown tags silently if they are occurred outside of the "value" element.

[serhiy-storchaka]

Opened bpo-26885 for adding support of "ex:nil" and other types.

[NathanWilliams]

Serhiy, that workaround worked for my needs, thanks.

[python-dev]

New changeset 0d015f6aba8b by Serhiy Storchaka in branch '3.5':
Issue bpo-26873: xmlrpc now raises ResponseError on unsupported type tags
https://hg.python.org/cpython/rev/0d015f6aba8b

New changeset 8f7cb3b171f3 by Serhiy Storchaka in branch 'default':
Issue bpo-26873: xmlrpc now raises ResponseError on unsupported type tags
https://hg.python.org/cpython/rev/8f7cb3b171f3

New changeset 7050c9fc1f72 by Serhiy Storchaka in branch '2.7':
Issue bpo-26873: xmlrpclib now raises ResponseError on unsupported type tags
https://hg.python.org/cpython/rev/7050c9fc1f72