ImportError on SDK provided libraries since 1.9.37 [35900562]

Fixed

Bug

More Info Required

Status Update

No update yet.

Description

rd...@gpcsolutions.fr

created issue #1

May 9, 2016 06:22PM

* The problem affect apps running in the
development server. Didn't test production.
* SDK informations
* version 1.9.37
* Python
* Arch Linux (Kernel 4.5.2)
* Package from the AUR (I'm the maintainer)

Since 1.9.37 libraries usually provided by the SDK can't be imported (ImportError).

The sys.path generated by the SDK for the libraries is wrong and points one directory before the actual SDK.

The SDK is installed in /opt/google_appengine.

Sample defective sys.path (Personal directories redacted):
['/opt/google_appengine', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-dynload', '/opt/lib/jinja2-2.6', '/opt/lib/pycrypto-2.6', '/opt/lib/ssl-2.7', '/opt/lib/webapp2-2.5.2', '/opt/lib/yaml-3.10', /opt/lib/markupsafe-0.15', '/opt/lib/setuptools-0.6c11', '/opt/lib/protorpc-1.0', /opt/lib/webob-1.1.1']

Sample working sys.path from SDK 1.9.36 (Personal directories redacted):
['/opt/google_appengine', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-dynload', '/opt/google_appengine/lib/jinja2-2.6', '/opt/google_appengine/lib/pycrypto-2.6', '/opt/google_appengine/lib/ssl-2.7', '/opt/google_appengine/lib/webapp2-2.5.2', '/opt/google_appengine/lib/yaml-3.10', '/opt/google_appengine/lib/markupsafe-0.15', '/opt/google_appengine/lib/setuptools-0.6c11', '/opt/google_appengine/lib/protorpc-1.0', '/opt/google_appengine/lib/webob-1.1.1']

Comments

ar...@google.com <ar...@google.com> #2May 13, 2016 09:37PM

This is indeed strange. I'm testing Python SDK 1.9.37 as well and I don't see the same thing. I'm on Debian though, Python 2.7.9, Kernel 3.16.0-4. Here's an example of my sys.path, SDK installed to '/opt/devel/google/appengine/python':

['/opt/devel/google/appengine/python', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-dynload', '/opt/devel/google/appengine/python/lib/protorpc-1.0', '/opt/devel/google/appengine/python/lib/webapp2-2.3', '/opt/devel/google/appengine/python/lib/webob-1.1.1', '/opt/devel/google/appengine/python/lib/yaml-3.10']

Could you share which Python version you're on, and also the method you're testing (I assume just a one-line get handler that prints sys.path)?

el...@getaround.com <el...@getaround.com> #3May 15, 2016 10:33PM

I have a similar issue; running Python 2.7.11 (from homebrew) on Mac OS 10.11.3.

rd...@gpcsolutions.fr <rd...@gpcsolutions.fr> #4May 16, 2016 07:03PM

I'm running Python 2.7.11.
I simply printed sys.path at the first line of my main.py as a means to debug my ImportError and found the offending paths there.

ar...@google.com <ar...@google.com> #5May 17, 2016 04:32PM

Assigned to ar...@google.com.

Thanks, I'll continue to investigate.

ke...@gmail.com <ke...@gmail.com> #6May 18, 2016 04:58AM

I have a similar issue; running Python 2.7.8 (virtualenv settings) on Mac OS 10.10.5.
: I've reverted SDK version by using gcloud component update version==108.0.0

sys.path are
['$HOME/$PROJECT_DIR/ziplib', '$HOME/$PROJECT_DIR/lib', '$HOME/$PROJECT_DIR', '$HOME/google-cloud-sdk/platform/google_appengine', '$HOME/google-cloud-sdk/platform/google_appengine', '$HOME/google-cloud-sdk/platform/google_appengine', '$HOME/gae/lib/python2.7', '$HOME/gae/lib/python2.7/lib-dynload', '$HOME/.pyenv/versions/2.7.8/lib/python2.7', '$HOME/google-cloud-sdk/platform/lib/pycrypto-2.6', '$HOME/google-cloud-sdk/platform/lib/webapp2-2.5.2', '$HOME/google-cloud-sdk/platform/lib/jinja2-2.6', '$HOME/google-cloud-sdk/platform/lib/markupsafe-0.15', '$HOME/google-cloud-sdk/platform/lib/setuptools-0.6c11', '$HOME/google-cloud-sdk/platform/lib/webob-1.2.3', '$HOME/google-cloud-sdk/platform/lib/PIL-1.1.7', '$HOME/google-cloud-sdk/platform/lib/ssl-2.7', '$HOME/google-cloud-sdk/platform/lib/protorpc-1.0', '$HOME/google-cloud-sdk/platform/lib/yaml-3.10', '$HOME/$PROJECT_DIR/lib/PIL']

jo...@google.com <jo...@google.com> #7May 18, 2016 09:29PM

This is intriguing. I spun up an Arch Linux instance on GCE[1], updated via `pacman -Syu` and grabbed the `base-devel` packages, then ran the following two tests with a (modified) helloworld-ish example app[2] with the appropriate dev_appserver:

1) GAE Python SDK from official source zip.
sys.path contents
/home/$USER$/src/syspathtest
/home/$USER$/src/google_appengine
/home/$USER$/src/google_appengine
/usr/lib/python2.7
/usr/lib/python2.7/lib-dynload
/home/$USER$/src/google_appengine/lib/protorpc-1.0
/home/$USER$/src/google_appengine/lib/webapp2-2.3
/home/$USER$/src/google_appengine/lib/webob-1.1.1
/home/$USER$/src/google_appengine/lib/yaml-3.10

2) GAE Python SDK built from AUR
sys.path contents
/home/$USER$/src/syspathtest
/opt/google_appengine
/opt/google_appengine
/usr/lib/python2.7
/usr/lib/python2.7/lib-dynload
/opt/google_appengine/lib/protorpc-1.0
/opt/google_appengine/lib/webapp2-2.3
/opt/google_appengine/lib/webob-1.1.1
/opt/google_appengine/lib/yaml-3.10

Neither version produced import errors, which I would've expected to see in the dev_appserver output upon hitting the localhost endpoint. Do you have any other ideas on what may be required to trigger this? I'd like to be able to reproduce this myself.

[1]

https://github.com/GoogleCloudPlatform/compute-archlinux-image-builder
[2]

https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/modules/main.py

rd...@gpcsolutions.fr <rd...@gpcsolutions.fr> #8May 20, 2016 09:59AM

Looking at the diff between 1.9.36 and 1.9.37, the most obvious change that may affect sys.path is:
+ if 'google' in sys.modules:
+ google_path = os.path.join(os.path.dirname(__file__), 'google')
+ google_module = sys.modules['google']
+ google_module.__path__.append(google_path)

-
-
- if 'google' in sys.modules:
- del sys.modules['google']
+ google_module.__file__ = google_path

This change appears in 4 files (dev_appserver.py, php_cli.php, _python_runtime.py and _php_runtime.py)

Looking around using pdb, I was able to pinpoint the problem to a conflict with the python2-protobuf archlinux package providing a built-in google module that seem to upset the new code.

Could you please install this package and work from there?

Thanks

in...@csgactuarial.com <in...@csgactuarial.com> #9May 26, 2016 05:09PM

This appears to be happening on Ubuntu 16.04 as well. Please let us know if you need help testing.

jo...@google.com <jo...@google.com> #10May 26, 2016 06:06PM

I can reproduce this when installing the python2-protobuf package. We're currently investigating.

ja...@gmail.com <ja...@gmail.com> #11Jun 1, 2016 10:28PM

The same is happening on Ubuntu 14.04 also.

I printed sys.path from the sdk interactive console to confirm that, as the OP noted, the first part of the path is missing.

Output of sys.path (redacted):
'/home/jack/google_appengine', '/home/jack/lib/MySQLdb-1.2.5', '/home/jack/lib/jinja2-2.6', '/home/jack/lib/webapp2-2.5.2', '/home/jack/lib/ssl-2.7', '/home/jack/lib/markupsafe-0.15', '/home/jack/lib/setuptools-0.6c11', '/home/jack/lib/protorpc-1.0', '/home/jack/lib/webob-1.1.1', '/home/jack/lib/yaml-3.10'

mi...@drip.com <mi...@drip.com> #12Jun 2, 2016 07:21PM

Related to this?

https://groups.google.com/forum/#!searchin/google-appengine/webapp2/google-appengine/LucknWk8iaQ/5wGFJfQ8CAAJ

rd...@gpcsolutions.fr <rd...@gpcsolutions.fr> #13Jun 2, 2016 08:12PM

Yes, sounds like the same issue.

su...@gmail.com <su...@gmail.com> #15Jun 21, 2016 05:17AM

Possibly related:

https://code.google.com/p/googleappengine/issues/detail?can=2&start=0&num=100&q=&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log&groupby=&sort=&id=13084

ar...@artooro.com <ar...@artooro.com> #17Jun 22, 2016 07:44PM

Uninstalling the following PIP packages resolved this issue for me.

sudo pip uninstall gcloud
sudo pip uninstall googleapis-common-protos
sudo pip uninstall protobuf

Credit to this thread:

https://groups.google.com/forum/?hl=nl#!topic/google-appengine/LucknWk8iaQ

ge...@gmail.com <ge...@gmail.com> #18Jul 24, 2016 01:12PM

Run into same problem.

GAE: 1.9.38
Python: 2.7.11

Used solution in #17. I didn't have gcloud and googleapis-common-prtos installed through pip. so only uninstalled protobuf

$ pip uninstall protobuf
Uninstalling protobuf-3.0.0b2:
...

Now problem solved.

ji...@google.com <ji...@google.com> #19Sep 17, 2016 10:05PM

I am unable to uninstall protobuf as it is used by system applications on my installation.

What is the ETA for the fix?

sf...@gmail.com <sf...@gmail.com> #20Jan 31, 2017 04:37PM

Same problem for me:
ImportError: No module named google.appengine.dist27.threading

it's a python import conflict with namespace google with protobuf
=> sudo pip uninstall protobuf

But what if we really need to use appengine SDK and protobuf on the same machine ?

js...@google.com <js...@google.com> #21Mar 17, 2017 10:16PM

Marked as fixed.

This issue is being closed as all blocking work is complete. The issue has likely been resolved for a while, however thanks to the recent integration with our internal tracker we can more easily detect cases with no remaining blockers.

If this issue is not actually resolved, please open a new issue which references this one and we'll take a look.