Failure when uploading binary files on Python 3 when resumable=False

[Rufflewind]

Seems like MediaFileUpload (or something else deep inside) is failing to handling binary files properly on Python 3. My guess is that the contents of the file is mistakenly passed as a Unicode string (rather than as a byte-string) into http.client, which dies as soon as it encounters a non-Latin-1 character. This only happens when resumable=False.

Code to reproduce the problem:

import apiclient
service = apiclient.discovery.build("drive", "v2", …)
filename = "some-file.tar.xz"
media = apiclient.http.MediaFileUpload(filename)
service.files().insert(body={"title": os.path.basename(filename)}, media_body=media).execute()

Typical error:

Traceback (most recent call last):
  …
  File "/usr/lib/python3.5/site-packages/oauth2client/util.py", line 142, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/googleapiclient/http.py", line 722, in execute
    body=self.body, headers=self.headers)
  File "/usr/lib/python3.5/site-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/lib/python3.5/site-packages/httplib2/__init__.py", line 1313, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python3.5/site-packages/httplib2/__init__.py", line 1063, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python3.5/site-packages/httplib2/__init__.py", line 987, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1127, in _send_request
    body = body.encode('iso-8859-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' in position 350: ordinal not in range(256)

(Using Python 3.5.0 on Arch Linux 4.2.2.)

[nathanielmanistaatgoogle]

I see that the stack trace goes through oauth2client and httplib2 code as well as this library's code - with what versions of the three libraries are you seeing this problem?

[Rufflewind]

apiclient 1.4.2
oauth2client 1.5.1
httplib2 0.9.2

[nathanielmanistaatgoogle]

There are ways you could have answered that question that would have made my life easier and that is not one of them. :-P And all three libraries formally support Python 3, so that doesn't lead to an easy answer either. I agree with you that "Seems like MediaFileUpload (or something else deep inside) is failing to handling binary files properly on Python 3.". I won't soon have time to investigate this, but I suspect that a proper investigation is warranted. Are you able to look into it?

[journeytosilius]

This bug still exist, just to let you know. If you want output I can drop it ... let me know if you are still interested. I don't have enough experience to fix something like this by myself.

[eyadsibai]

I am facing same issue will test it with this current commit and let you know

[nathanielmanistaatgoogle]

@chromafunk and @eyadsibai: yes, by all means, if you believe this issue is still active please describe how you're still seeing it (and if you can describe how 26178fc fails to fully remedy the problem).

[eyadsibai]

@nathanielmanistaatgoogle I did install the package from master ... it's working now!

thanks