Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install protobuf should default to cpp_implementation if system headers present #539

Closed
dhermes opened this issue Jun 24, 2015 · 18 comments

Comments

@dhermes
Copy link

dhermes commented Jun 24, 2015

As of now it requires calling

python setup.py install --cpp_implementation

or

pip install protobuf==${VERSION} --install-option="--cpp_implementation"

But

  1. This flag is not documented well or transparent to users
  2. Including protobuf as a dependency via
setup(
    ...
    install_requires=['protobuf', ...],
    ...
)

precludes passing a flag along to setup.py.

(I realize this is somewhat problematic / won't work for users that already have protobuf installed without the flag, but at least this will help people in fresh installs.)

/cc @tseaver @jgeewax @tbetbetbe

@xfxyjwf
Copy link
Contributor

xfxyjwf commented Mar 7, 2017

We now distribute python C++ implementations to PyPI and it should be selected automatically by pip.

@dhermes
Copy link
Author

dhermes commented Mar 7, 2017

Awesome news! Thanks

@swangsc
Copy link

swangsc commented May 10, 2017

I am running sudo pip install protobuf --install-option="--cpp_implementation" on Mac but get errors:
en-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I. -I../src -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c google/protobuf/pyext/map_container.cc -o build/temp.macosx-10.12-intel-2.7/google/protobuf/pyext/map_container.o -Wno-write-strings -Wno-invalid-offsetof -Wno-sign-compare -std=c++11
cc -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I. -I../src -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c google/protobuf/pyext/message.cc -o build/temp.macosx-10.12-intel-2.7/google/protobuf/pyext/message.o -Wno-write-strings -Wno-invalid-offsetof -Wno-sign-compare -std=c++11
google/protobuf/pyext/message.cc:69:10: fatal error: 'google/protobuf/stubs/strutil.h' file not found
#include <google/protobuf/stubs/strutil.h>
^
1 error generated.
error: command 'cc' failed with exit status 1

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/private/tmp/pip-build-_Rgqlb/protobuf/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-TifxqZ-record/install-record.txt --single-version-externally-managed --compile --cpp_implementation" failed with error code 1 in /private/tmp/pip-build-_Rgqlb/protobuf/

@xfxyjwf
Copy link
Contributor

xfxyjwf commented May 10, 2017

@swangsc To install python cpp implementation on Mac, you will need to install the C++ runtime first.

@swangsc
Copy link

swangsc commented May 11, 2017

@xfxyjwf I have installed the C++ package following https://github.com/google/protobuf/tree/master/src how can I make sure if the c++ runtime is installed?

@xfxyjwf
Copy link
Contributor

xfxyjwf commented May 11, 2017

Hmm, I just checked, and it turns out the struil.h header used by pyext/message.cc is not actually installed with protobuf C++ runtime (i.e., it's treated as an internal header). I guess we will need to either make it a public header or update python cpp code to not use it.

To workaround it you can install from source by following instructions here:
https://github.com/google/protobuf/tree/master/python

@swangsc
Copy link

swangsc commented May 11, 2017

Hi X:
Thanks for explaining this. But when I run the following command, it still failed:
$ sudo python setup.py test --cpp_implementation
Password:
running test
running egg_info
writing requirements to protobuf.egg-info/requires.txt
writing protobuf.egg-info/PKG-INFO
writing namespace_packages to protobuf.egg-info/namespace_packages.txt
writing top-level names to protobuf.egg-info/top_level.txt
writing dependency_links to protobuf.egg-info/dependency_links.txt
reading manifest file 'protobuf.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'google/protobuf/internal/.proto'
warning: no previously-included files matching '
_test.py' found under directory 'google'
warning: no previously-included files matching '_test.proto' found under directory 'google'
warning: no previously-included files matching '
.dll' found anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '
.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
writing manifest file 'protobuf.egg-info/SOURCES.txt'
running build_ext
copying build/lib.macosx-10.12-intel-2.7/google/protobuf/pyext/_message.so -> google/protobuf/pyext
copying build/lib.macosx-10.12-intel-2.7/google/protobuf/internal/_api_implementation.so -> google/protobuf/internal
Traceback (most recent call last):
File "setup.py", line 260, in
ext_modules=ext_module_list,
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "build/bdist.macosx-10.12-intel/egg/setuptools/command/test.py", line 211, in run
File "build/bdist.macosx-10.12-intel/egg/setuptools/command/test.py", line 234, in run_tests
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in init
self.parseArgs(argv)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 149, in parseArgs
self.createTests()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 158, in createTests
self.module)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
return self.loadTestsFromModule(obj)
File "build/bdist.macosx-10.12-intel/egg/setuptools/command/test.py", line 43, in loadTestsFromModule
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'any_test_pb2'

@xfxyjwf
Copy link
Contributor

xfxyjwf commented May 11, 2017

@swangsc Can you run "./setup.py build --cpp_implementation" and then try test again?

@swangsc
Copy link

swangsc commented May 11, 2017

Hi X,

I finally installed the c++ implementation based python runtime successfully. I can install it even the tests failed. The performance is awesome.

Thanks!

@BreakTom
Copy link

BreakTom commented Jun 7, 2017

hi,@swangsc, what's your protobuf version?

@danluu
Copy link

danluu commented Jun 26, 2017

@BreakTom I'm not swangsc, but I believe I'm seeing similar errors or maybe even the same error.

On OS X 10.12.5, building from HEAD (097bfb5):

======================================================================
FAIL: testInvalidTimestamp (google.protobuf.internal.json_format_test.JsonFormatTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/danluu/dev/external/protobuf/python/google/protobuf/json_format.py", line 496, in _ConvertFieldValuePair
    self.ConvertMessage(value, sub_message)                                                                                                                   [0/1279]
ValueError: year 0 is out of range

During handling of the above exception, another exception occurred:

google.protobuf.json_format.ParseError: Failed to parse value field: year 0 is out of range.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/danluu/dev/external/protobuf/python/google/protobuf/internal/json_format_test.py", line 772, in testInvalidTimestamp
    json_format.Parse, text, message)
AssertionError: "Failed to parse value field: year is out of range." does not match "Failed to parse value field: year 0 is out of range."

======================================================================
FAIL: testInvalidTimestamp (google.protobuf.internal.well_known_types_test.TimeUtilTest)
----------------------------------------------------------------------
ValueError: year 0 is out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/danluu/dev/external/protobuf/python/google/protobuf/internal/well_known_types_test.py", line 289, in testInvalidTimestamp
    '0000-01-01T00:00:00Z')
AssertionError: "year is out of range" does not match "year 0 is out of range"

----------------------------------------------------------------------
Ran 614 tests in 0.873s

FAILED (failures=2, skipped=14)
Test failed: <unittest.runner.TextTestResult run=614 errors=0 failures=2>
error: Test failed: <unittest.runner.TextTestResult run=614 errors=0 failures=2>

I first built the C++ version, which did not report failures on make check, and then after that I ran

python3 setup.py build --cpp_implementation
python3 setup.py test --cpp_implementation

which gave me the failures mentioned above.

@abhimanyu1289
Copy link

I am getting error: option --cpp_implementation not recognized on doing
pip install protobuf --install-option="--cpp_implementation"
Am I doing something wrong.

@swangsc
Copy link

swangsc commented Jul 17, 2017

@abhimanyu1289 I think you need to install c++ implementation from source code. PIP doesn't support it.

@abhimanyu1289
Copy link

@swangsc I did install it. But I get the same error. I gave up installing via pip. I installed using setup.py but I am getting this error when I am trying to use my protobufs.
ImportError: /home/ubuntu/test-env/local/lib/python2.7/site-packages/protobuf-3.3.0-py2.7-linux-x86_64.egg/google/protobuf/pyext/_message.so: undefined symbol: _ZNK6google8protobuf15FieldDescriptor4typeEv

@swangsc
Copy link

swangsc commented Jul 17, 2017

have you do this: export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp

@abhimanyu1289
Copy link

yes. I did.

@cristicbz
Copy link
Contributor

By the way, I have a PR that should sort out the strutil.h issues in #2576, but it's been languishing there since Jan :( . I'd love to get it merged, but I have no idea how to get the attention of any of the maintainers.

@josegonzalez
Copy link

If anyone is still having issues here, ensure your python is compiled for wide-unicode - compile flag --enable-unicode=ucs4 -- not narrow-unicode. The former is standard on many linux distributions, but protobuf doesn't distribute a narrow-unicode package, so you'll be stuck with the pure-python implementation.

taoso pushed a commit to taoso/protobuf that referenced this issue Aug 1, 2018
…s#539)

protoc-gen-go: add test for various generation params

Add tests for import_prefix and Ma.proto=package.
adellahlou pushed a commit to adellahlou/protobuf that referenced this issue Apr 20, 2023
rinarakaki pushed a commit to rinarakaki/protobuf that referenced this issue Aug 30, 2023
First draft of new mini-table building API
bithium pushed a commit to bithium/protobuf that referenced this issue Sep 4, 2023
First draft of new mini-table building API
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants