Avoid using `pb` internals for datastore on App Engine

[dhermes]

See #282 for context.

From @pcostell:

Just a random note about proto usage:
If you plan to let users run gcloud-python in the App Engine python runtime, proto access is
really slow (I believe because of the sandbox). I would limit proto access to serialization
and deserialization only.

and

Not 100% sure, but I think it uses a C proto library so it goes outside the process boundary.

[dhermes]

Currently digging into https://www.simonmweber.com/2013/06/18/python-protobuf-on-app-engine.html to see what I can find out.

[dhermes]

@pcostell I'm not sure your previous comments are relevant to gcloud. I just created a test application and the proto used is via the external protobuf library, not the proto libraries provided in App Engine.

I added a debug print to check Entity.save() times in a single request and got:

• 0.197650 seconds
• 0.113310 seconds
• 0.169830 seconds

I was worried this was slow so did the same locally with the version on PyPI (0.3.0) and got similar times:

• 1.152835 seconds (app was cold)
• 0.233535 seconds
• 0.359164 seconds
• 0.217317 seconds

ISTM the App Engine speed-up is due to the fast-path between the App Engine app and the Cloud Datastore machines.

UPDATE: I'm fairly certain Patrick was referring to google.net.proto and google.net.proto2 as the libraries to avoid using. I checked to make sure we don't ever import from proto libraries included in GAE.

SECOND UPDATE: I broke down Entity.save() into lower and lower level component pieces until the network was the only thing left. The times of each show that the impact of using the protobuf library is negligible.

[dhermes]

@silvolu @tseaver FYI gcloud works just fine deployed in App Engine.

[tseaver]

Great analysis!

[pcostell]

Awesome, good to know. Something that may be important: The reason GAE protos have slow access is that they use the C++ library, which requires a process boundary cross. This is used instead of a pure python library because serialization and deserialization is much faster in C++ than in python. In may be worthwhile to still minimize proto access in order to minimize SWIG-code entry points when users do have the C++ libraries (running on VMs or if GAE adds google.protobuf to the runtime).

[dhermes]

Luckily the protobuf library only uses C-code from the standard library. As @tseaver pointed out earlier today, struct is plenty fast.

I analyzed the imports, and the only imports from outside the library are:

StringIO.StringIO
cStringIO
cStringIO.StringIO
copy_reg
copyreg  # Py3, comment added by dhermes
io.BytesIO  # Py3, comment added by dhermes
operator
os
re
struct
sys
uuid
weakref

[pcostell]

Currently, when using the externally available protobuf library on App Engine, it is using the python-only version of the protobuf library (it's impossible to upload a C library to the python runtime). However, once we are able to get the external C library into the App Engine runtime it will start using the C version instead of the python version. The C version has faster encode/decode times however you do pay a penalty when accessing protocol buffer fields because it has to cross a process boundary. We saw this in NDB's Key implementation which tried to store everything internally in protobuf rather than storing the fields as native python types.

I went through and it looks like you only access the protobuf on serialization and deserialization, which is perfect, so I'm leaving this as closed.

[tseaver]

@pcostell we just recently dropped the "long-lived protobuf" inside Query: glad that helps.