protobuf - Project Hosting on Google Code

Today 99 min ago issue 126 (corrupted protobuf-java-2.2.0.jar in maven repository) commented on by gk5...@kickstyle.net - All of this cleanup is in my queue for January. Sorry about the really, really long delay. All of this cleanup is in my queue for January. Sorry about the really, really long delay. 104 min ago issue 126 (corrupted protobuf-java-2.2.0.jar in maven repository) commented on by bs1984 - Hi, the same checksum issue was reported in the google-guice issue tracker when a user tried downloading artifacts using Apache Ivy: http://code.google.com/p/google-guice/issues/detail?id=421 > This issue is caused when using Maven 2.2 to deploy. > It doesn't occur with Maven 2.1. Hi, the same checksum issue was reported in the google-guice issue tracker when a user tried downloading artifacts using Apache Ivy: http://code.google.com/p/google-guice/issues/detail?id=421 > This issue is caused when using Maven 2.2 to deploy. > It doesn't occur with Maven 2.1. 4 hours ago issue 145 (Generated Java code: extension access fields are public, sta...) reported by timw.at.work - What steps will reproduce the problem? 1. Create two messages and assign one as an extended field of the other. 2. Generate java code from the protocol 3. Check the resulting code with the FindBugs static analysis tool ( http://findbugs.sourceforge.net ) What is the expected output? What do you see instead? Actual result: I see a warning, 'extensionField isn't final and can't be protected from malicious code' Expected result: there should be no 'public static' non-final fields. What version of the product are you using? On what operating system? Protocol Buffers 2.2.0, Ubuntu 9.04, 64-bit, JDK 1.6.0_16 Please provide any additional information below. The attached protocol definition results in a field as follows: public static com.google.protobuf.GeneratedMessage.GeneratedExtension< Extension.base, Extension.ext> extensionField; As the FindBugs warning indicates, such fields are unsafe to use where untrusted code could run (such as an applet). A Google search for 'public static non-final' will bring up some of those explanations. If the presence of that field is a legacy wart, what should I be using? The documentation at http://code.google.com/apis/protocolbuffers/docs/reference/java-generated.html seems to indicate that the field is the preferred (perhaps only) way to access/set extension fields. Note that I observed no difference when moving the 'extend [message]' block to the top-level (instead of nesting it within the extension message definition). I bring this up because the documentation for ExtensionRegistry explicitly calls out security concerns of this nature: http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/ExtensionRegistry.html What steps will reproduce the problem? 1. Create two messages and assign one as an extended field of the other. 2. Generate java code from the protocol 3. Check the resulting code with the FindBugs static analysis tool ( http://findbugs.sourceforge.net ) What is the expected output? What do you see instead? Actual result: I see a warning, 'extensionField isn't final and can't be protected from malicious code' Expected result: there should be no 'public static' non-final fields. What version of the product are you using? On what operating system? Protocol Buffers 2.2.0, Ubuntu 9.04, 64-bit, JDK 1.6.0_16 Please provide any additional information below. The attached protocol definition results in a field as follows: public static com.google.protobuf.GeneratedMessage.GeneratedExtension< Extension.base, Extension.ext> extensionField; As the FindBugs warning indicates, such fields are unsafe to use where untrusted code could run (such as an applet). A Google search for 'public static non-final' will bring up some of those explanations. If the presence of that field is a legacy wart, what should I be using? The documentation at http://code.google.com/apis/protocolbuffers/docs/reference/java-generated.html seems to indicate that the field is the preferred (perhaps only) way to access/set extension fields. Note that I observed no difference when moving the 'extend [message]' block to the top-level (instead of nesting it within the extension message definition). I bring this up because the documentation for ExtensionRegistry explicitly calls out security concerns of this nature: http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/ExtensionRegistry.html 12 hours ago r256 (Ensure that 'once' objects are declared using the macro. Th...) committed by temporal - Ensure that 'once' objects are declared using the macro. This is either the third or fourth time I've screwed this up when down-integrating, because our internal code does not require the macro (it's not portable) and on Linux a pthread_once_t that is zero-initialized just happens to work. So, I only discover the problem when I test on Mac, then kick myself for making the same mistake yet again. No more! This time, I have renamed GoogleOnceType to ProtobufOnceType, thus making the type name differ from our internal code. As a result, if you don't update the decls to use the macros, they won't compile. Hah! Take that, future self! Ensure that 'once' objects are declared using the macro. This is either the third or fourth time I've screwed this up when down-integrating, because our internal code does not require the macro (it's not portable) and on Linux a pthread_once_t that is zero-initialized just happens to work. So, I only discover the problem when I test on Mac, then kick myself for making the same mistake yet again. No more! This time, I have renamed GoogleOnceType to ProtobufOnceType, thus making the type name differ from our internal code. As a result, if you don't update the decls to use the macros, they won't compile. Hah! Take that, future self! 13 hours ago r255 (Fix Cygwin build.) committed by kenton@google.com - Fix Cygwin build. Fix Cygwin build. 14 hours ago issue 144 (getExtension() throws ClassCastException on absent repeated ...) changed by kenton@google.com - Status: Accepted Owner: ken...@google.com Status: Accepted Owner: ken...@google.com 14 hours ago issue 143 (When passing an invalid fstream to .SerializeToOstream() it ...) changed by kenton@google.com - Status: Accepted Owner: ken...@google.com Status: Accepted Owner: ken...@google.com 15 hours ago r254 (Commit missing files from r251. Sigh.) committed by ken...@google.com - Commit missing files from r251. Sigh. Commit missing files from r251. Sigh. 17 hours ago r253 (Set version to 2.3.0-pre.) committed by ken...@google.com - Set version to 2.3.0-pre. Set version to 2.3.0-pre. 18 hours ago r252 (Point test_plugin at the correct gtest headers.) committed by ken...@google.com - Point test_plugin at the correct gtest headers. Point test_plugin at the correct gtest headers. 18 hours ago r251 (Massive roll-up of changes. See CHANGES.txt.) committed by ken...@google.com - Massive roll-up of changes. See CHANGES.txt. Massive roll-up of changes. See CHANGES.txt. 18 hours ago r250 (Some sort of emacs protobuf mode fix.) committed by ken...@google.com - Some sort of emacs protobuf mode fix. Some sort of emacs protobuf mode fix. Yesterday 24 hours ago issue 144 (getExtension() throws ClassCastException on absent repeated ...) reported by timw.at.work - What steps will reproduce the problem? 1. Create two messages, Container and Item, with an extension to Container adding a repeated Item field 2. Obtain a message instance with no instances of the repeated extension field. 3. Attempt to call container.getExtension(...) and use the resulting List. What is the expected output? What do you see instead? Expected: I get an empty List, much like already happens with repeated non-extension fields. Actual: I get a ClassCastException: Item cannot be cast to java.util.List. What version of the product are you using? On what operating system? Protocol Buffers 2.2.0 on Ubuntu 9.04, 64-bit (downloaded from google code and built with the default 9.04 64-bit gcc). Please provide any additional information below. The problem appears to be in the generated getExtension() code: FieldDescriptor descriptor = extension.getDescriptor(); final Object value = extensions.getField(descriptor); if (value == null) { if (descriptor.getJavaType() == FieldDescriptor.JavaType.MESSAGE) { return (Type) extension.getMessageDefaultInstance(); } else { return (Type) extension.fromReflectionType( descriptor.getDefaultValue()); } } else { return (Type) extension.fromReflectionType(value); } Type is java.util.List, because the field is a repeated extension field. I did not step through the code (logically or with a debugger), but it seems like it might be hitting the JavaType.MESSAGE branch and attempting to return a default message instance instead of an empty list. Attached is a simple protocol definition and some java code that invokes the bug. A work-around, as indicated in the attached source, is to use getExtensionCount to guard against using a 0-count repetition. What steps will reproduce the problem? 1. Create two messages, Container and Item, with an extension to Container adding a repeated Item field 2. Obtain a message instance with no instances of the repeated extension field. 3. Attempt to call container.getExtension(...) and use the resulting List. What is the expected output? What do you see instead? Expected: I get an empty List, much like already happens with repeated non-extension fields. Actual: I get a ClassCastException: Item cannot be cast to java.util.List. What version of the product are you using? On what operating system? Protocol Buffers 2.2.0 on Ubuntu 9.04, 64-bit (downloaded from google code and built with the default 9.04 64-bit gcc). Please provide any additional information below. The problem appears to be in the generated getExtension() code: FieldDescriptor descriptor = extension.getDescriptor(); final Object value = extensions.getField(descriptor); if (value == null) { if (descriptor.getJavaType() == FieldDescriptor.JavaType.MESSAGE) { return (Type) extension.getMessageDefaultInstance(); } else { return (Type) extension.fromReflectionType( descriptor.getDefaultValue()); } } else { return (Type) extension.fromReflectionType(value); } Type is java.util.List, because the field is a repeated extension field. I did not step through the code (logically or with a debugger), but it seems like it might be hitting the JavaType.MESSAGE branch and attempting to return a default message instance instead of an empty list. Attached is a simple protocol definition and some java code that invokes the bug. A work-around, as indicated in the attached source, is to use getExtensionCount to guard against using a 0-count repetition. Last 30 days Dec 10, 2009 r249 (Fix UTF-8 validity checks to not do unaligned reads.) committed by ken...@google.com - Fix UTF-8 validity checks to not do unaligned reads. Fix UTF-8 validity checks to not do unaligned reads. Dec 10, 2009 r248 (fix SONAME in trunk) committed by ken...@google.com - fix SONAME in trunk fix SONAME in trunk Dec 10, 2009 issue 143 (When passing an invalid fstream to .SerializeToOstream() it ...) reported by lance.earwood - What steps will reproduce the problem? 1. Create valid message to serialize MyCollection collection; // of type google::protobuf::Message 2. Create fstream that will fail (e.g.: directory /foo does not exist) fstream outs("/foo/mycollection.pb", ios::out\|ios::binary); bool success = outs.good(); // will return false 3. Attempt to serialize the message bool serializedOK = collection.SerializeToOstream(); // will return true What is the expected output? What do you see instead? I foolishly forgot to check the status of the fstream before calling .SerializeToOstream() and was using its return value as success/failure of serializing the message to disk. I expected to .SerializeToOstream() to return false when passed an invalid fstream object but it returned true, indicating the Message object was written to disk (when it had not been). I realize this was a mistake on my part, but I fail to see a reason why one would even _attempt_ to serialize something that obviously won't work. This may be a low priority item but the fix is simple: add an fstream.good() check in .SerializeToOStream() method body. What version of the product are you using? On what operating system? 2.0.2 Please provide any additional information below. What steps will reproduce the problem? 1. Create valid message to serialize MyCollection collection; // of type google::protobuf::Message 2. Create fstream that will fail (e.g.: directory /foo does not exist) fstream outs("/foo/mycollection.pb", ios::out\|ios::binary); bool success = outs.good(); // will return false 3. Attempt to serialize the message bool serializedOK = collection.SerializeToOstream(); // will return true What is the expected output? What do you see instead? I foolishly forgot to check the status of the fstream before calling .SerializeToOstream() and was using its return value as success/failure of serializing the message to disk. I expected to .SerializeToOstream() to return false when passed an invalid fstream object but it returned true, indicating the Message object was written to disk (when it had not been). I realize this was a mistake on my part, but I fail to see a reason why one would even _attempt_ to serialize something that obviously won't work. This may be a low priority item but the fix is simple: add an fstream.good() check in .SerializeToOStream() method body. What version of the product are you using? On what operating system? 2.0.2 Please provide any additional information below. Dec 08, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) commented on by dtr...@davidtrott.com - No problem, my patch/workaround is fine for now. I will take another look at it once the plug-in infrastructure is available. No problem, my patch/workaround is fine for now. I will take another look at it once the plug-in infrastructure is available. Dec 08, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) commented on by ken...@google.com - Well, frankly, I'm not very eager to spend time adding features to the existing system knowing that we plan to deprecate it anyway. Can I convince you to wait a couple weeks while I get the next release out? Well, frankly, I'm not very eager to spend time adding features to the existing system knowing that we plan to deprecate it anyway. Can I convince you to wait a couple weeks while I get the next release out? Dec 07, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) commented on by dtr...@davidtrott.com - It sounds like the plugin will solve the issue I currently have. However you have alluded to a more general issue, specifically that two modules running within the same Classloader may need different callback wrappers generated. This could be somewhat cumbersome if the two modules are using the same .proto file. Specifically it would be necessary to generate two "java_outer_classname" wrappers, each one calling a different version of RpcUtil. Fundamentally I see this as an issue with RpcUtil, (because all the methods in RpcUtil are declared static and can't be overridden). An alternative would be to convert RpcUtil to an Interface that defines the same methods as the current static methods. In order to hook this up, the generated service methods: + newStub(...) + newBlockingStub(...) + newReflectiveService(...) + newReflectiveBlockingService(...) Would each need to take an additional RpcUtil parameter, that way you could decide at the time you create a stub or service which RpcUtil implementation you wanted to use. Now I hear you saying that, once the plugin is available I could generate the code to do exactly this. However another alternative would be to change the default code generation to generate two versions of each of these methods; one that takes the extra RpcUtil parameter and one that doesn't. The one that don't can default to using the "standard" implementation of RpcUtil, however it would allow others to call the second version to override the RpcUtil implementation. Thoughts? Is it worth me coding up a PoC for this? It sounds like the plugin will solve the issue I currently have. However you have alluded to a more general issue, specifically that two modules running within the same Classloader may need different callback wrappers generated. This could be somewhat cumbersome if the two modules are using the same .proto file. Specifically it would be necessary to generate two "java_outer_classname" wrappers, each one calling a different version of RpcUtil. Fundamentally I see this as an issue with RpcUtil, (because all the methods in RpcUtil are declared static and can't be overridden). An alternative would be to convert RpcUtil to an Interface that defines the same methods as the current static methods. In order to hook this up, the generated service methods: + newStub(...) + newBlockingStub(...) + newReflectiveService(...) + newReflectiveBlockingService(...) Would each need to take an additional RpcUtil parameter, that way you could decide at the time you create a stub or service which RpcUtil implementation you wanted to use. Now I hear you saying that, once the plugin is available I could generate the code to do exactly this. However another alternative would be to change the default code generation to generate two versions of each of these methods; one that takes the extra RpcUtil parameter and one that doesn't. The one that don't can default to using the "standard" implementation of RpcUtil, however it would allow others to call the second version to override the RpcUtil implementation. Thoughts? Is it worth me coding up a PoC for this? Dec 07, 2009 issue 142 (compile problem due to strutil.cc (linux)) changed by ken...@google.com - This has been fixed (by explicitly including stdio.h) in newer versions. The latest version is 2.2.0. Status: Fixed Owner: ken...@google.com This has been fixed (by explicitly including stdio.h) in newer versions. The latest version is 2.2.0. Status: Fixed Owner: ken...@google.com Dec 07, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) commented on by ken...@google.com - Unfortunately your patch introduces a global, mutable singleton. Singletons are known to be a design anti-pattern that tend to lead to massive headaches and even security problems down the road. In this case, imagine what would happen if you wanted to use two different Java libraries each of which needed to install their own RpcCallbackProvider. It doesn't work, because only one provider can be installed at a time. Here's an alternative: We're planning to introduce a feature into protoc which allows you to write "code generator plugins". You might write a plugin to support a new language, or to extend the code generated by an existing language. Our plan is to deprecate the existing abstract RPC interfaces in protocol buffers and instead suggest that RPC implementations provide their own code generator plugins to generate whatever code is most appropriate to them. In your case, your RPC system could generate code which uses serializeable callbacks. Does this sound like it will suit your needs? Unfortunately your patch introduces a global, mutable singleton. Singletons are known to be a design anti-pattern that tend to lead to massive headaches and even security problems down the road. In this case, imagine what would happen if you wanted to use two different Java libraries each of which needed to install their own RpcCallbackProvider. It doesn't work, because only one provider can be installed at a time. Here's an alternative: We're planning to introduce a feature into protoc which allows you to write "code generator plugins". You might write a plugin to support a new language, or to extend the code generated by an existing language. Our plan is to deprecate the existing abstract RPC interfaces in protocol buffers and instead suggest that RPC implementations provide their own code generator plugins to generate whatever code is most appropriate to them. In your case, your RPC system could generate code which uses serializeable callbacks. Does this sound like it will suit your needs? Dec 06, 2009 issue 142 (compile problem due to strutil.cc (linux)) reported by jubilationJune - What steps will reproduce the problem? 1. ./configure 2. make 3. What is the expected output? What do you see instead? Wno-sign-compare -g -O2 -MT strutil.lo -MD -MP -MF .deps/strutil.Tpo -c google/protobuf/stubs/strutil.cc -fPIC -DPIC -o .libs/strutil.o google/protobuf/stubs/strutil.cc: In function 'int google::protobuf::CEscapeInternal(const char, int, char, int, bool)': google/protobuf/stubs/strutil.cc:433: error: 'sprintf' was not declared in this scope google/protobuf/stubs/strutil.cc: In function 'char* google::protobuf::DoubleToBuffer(double, char)': google/protobuf/stubs/strutil.cc:982: error: 'snprintf' was not declared in this scope google/protobuf/stubs/strutil.cc: In function 'char google::protobuf::FloatToBuffer(float, char)': google/protobuf/stubs/strutil.cc:1037: error: 'snprintf' was not declared in this scope google/protobuf/stubs/strutil.cc: In function 'std::string google::protobuf::LocalizeRadix(const char, const char)': google/protobuf/stubs/strutil.cc:1071: error: 'sprintf' was not declared in this scope make[1]: [strutil.lo] Error 1 make[1]: Leaving directory `/home/junez/Desktop/protobuf-2.0.1/src' make: * [install-recursive] Error 1 What version of the product are you using? On what operating system? protobuf-2.0.1 running on Fedora R12 gcc version 4.4.2 20091027 (Red Hat 4.4.2-7) Please provide any additional information below. What steps will reproduce the problem? 1. ./configure 2. make 3. What is the expected output? What do you see instead? Wno-sign-compare -g -O2 -MT strutil.lo -MD -MP -MF .deps/strutil.Tpo -c google/protobuf/stubs/strutil.cc -fPIC -DPIC -o .libs/strutil.o google/protobuf/stubs/strutil.cc: In function 'int google::protobuf::CEscapeInternal(const char, int, char, int, bool)': google/protobuf/stubs/strutil.cc:433: error: 'sprintf' was not declared in this scope google/protobuf/stubs/strutil.cc: In function 'char* google::protobuf::DoubleToBuffer(double, char)': google/protobuf/stubs/strutil.cc:982: error: 'snprintf' was not declared in this scope google/protobuf/stubs/strutil.cc: In function 'char google::protobuf::FloatToBuffer(float, char)': google/protobuf/stubs/strutil.cc:1037: error: 'snprintf' was not declared in this scope google/protobuf/stubs/strutil.cc: In function 'std::string google::protobuf::LocalizeRadix(const char, const char)': google/protobuf/stubs/strutil.cc:1071: error: 'sprintf' was not declared in this scope make[1]: [strutil.lo] Error 1 make[1]: Leaving directory `/home/junez/Desktop/protobuf-2.0.1/src' make: * [install-recursive] Error 1 What version of the product are you using? On what operating system? protobuf-2.0.1 running on Fedora R12 gcc version 4.4.2 20091027 (Red Hat 4.4.2-7) Please provide any additional information below. Dec 05, 2009 issue 141 (CodedStream (at least in Java) behaves unpredictable when s...) Status changed by ken...@google.com - OK, re-opening bug as a documentation issue. Status: Accepted OK, re-opening bug as a documentation issue. Status: Accepted Dec 05, 2009 issue 141 (CodedStream (at least in Java) behaves unpredictable when s...) commented on by ismaelj - I was under the impression that pushLimit would effectively set a limit on how much would be read from the underlying stream. I would at least add something about this to the documentation as I know of a few people were misled by this (independently). I was under the impression that pushLimit would effectively set a limit on how much would be read from the underlying stream. I would at least add something about this to the documentation as I know of a few people were misled by this (independently). Dec 04, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) commented on by dtr...@davidtrott.com - The specific case is that I am building an RPC library for Project Darkstar http://projectdarkstar.com More generally; processing a long running distributed task can be viewed as a set of small task separated by calls to remote systems (the task ends when the remote call starts the a new task starts when the remote system responds). Given the remote system may take time to respond it may be useful to either: + Write the callback out to disk. + Process the callback on a different machine (in a cluster). In either case Java (or a custom) Serialization may be used to provide this functionality, especially if versioning of the callback itself is not a requirement. In short I want to be able to override the generation of the callbacks so that I can provide an implementation that plays nicely with other systems. The specific case is that I am building an RPC library for Project Darkstar http://projectdarkstar.com More generally; processing a long running distributed task can be viewed as a set of small task separated by calls to remote systems (the task ends when the remote call starts the a new task starts when the remote system responds). Given the remote system may take time to respond it may be useful to either: + Write the callback out to disk. + Process the callback on a different machine (in a cluster). In either case Java (or a custom) Serialization may be used to provide this functionality, especially if versioning of the callback itself is not a requirement. In short I want to be able to override the generation of the callbacks so that I can provide an implementation that plays nicely with other systems. Dec 04, 2009 issue 139 (Go implementation of Protocol Buffers.) commented on by ken...@google.com - That issue is probably obsolete. The next version of protobufs (to be released this month, I hope) will include support for protoc generator plugins. With that, it will actually not make sense to bundle language support into the main implementation. In fact, I would like to spin off the Java and Python implementations into independent packages. That issue is probably obsolete. The next version of protobufs (to be released this month, I hope) will include support for protoc generator plugins. With that, it will actually not make sense to bundle language support into the main implementation. In fact, I would like to spin off the Java and Python implementations into independent packages. Dec 04, 2009 issue 139 (Go implementation of Protocol Buffers.) commented on by lost.goblin - Hmmm... how is this different from Issue 102 (C# merge support)? Hmmm... how is this different from Issue 102 (C# merge support)? Dec 04, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) Owner changed by ken...@google.com - You want your callbacks to be serializeable? Why? Owner: ken...@google.com You want your callbacks to be serializeable? Why? Owner: ken...@google.com Dec 04, 2009 issue 139 (Go implementation of Protocol Buffers.) changed by ken...@google.com - This is the responsibility of the Go team. Status: OutOfScope Owner: ken...@google.com This is the responsibility of the Go team. Status: OutOfScope Owner: ken...@google.com Dec 04, 2009 issue 141 (CodedStream (at least in Java) behaves unpredictable when s...) Owner changed by ken...@google.com - Owner: ken...@google.com Owner: ken...@google.com Dec 04, 2009 issue 141 (CodedStream (at least in Java) behaves unpredictable when s...) Status changed by ken...@google.com - For efficiency, CodedInputStream reads large blocks of data at a time, then parses them. If the InputStream has more data after the end of the message, it is normal for CodedInputStream to read some of that data. You need to either: (a) Create a single CodedInputStream for your stream and reuse it to parse each individual message, rather than create a new one for each message parsed. (b) Wrap your underlying InputStream in some sort of wrapper that prevents it from reading beyond a certain limit, then give that to the CodedInputStream. So, this would make it look like the end of the message was the end of the whole stream. I'd be open to a patch which adds a new constructor for CodedInputStream which takes an InputStream and an overall byte limit, and refuses to read past that limit. The implementation should just wrap the InputStream in some sort of limiter like I described above. Status: WorkingAsIntended For efficiency, CodedInputStream reads large blocks of data at a time, then parses them. If the InputStream has more data after the end of the message, it is normal for CodedInputStream to read some of that data. You need to either: (a) Create a single CodedInputStream for your stream and reuse it to parse each individual message, rather than create a new one for each message parsed. (b) Wrap your underlying InputStream in some sort of wrapper that prevents it from reading beyond a certain limit, then give that to the CodedInputStream. So, this would make it look like the end of the message was the end of the whole stream. I'd be open to a patch which adds a new constructor for CodedInputStream which takes an InputStream and an overall byte limit, and refuses to read past that limit. The implementation should just wrap the InputStream in some sort of limiter like I described above. Status: WorkingAsIntended Dec 04, 2009 issue 138 (Maven Protoc Plugin fails when a jar file contains a .proto ...) Owner changed by ken...@google.com - Owner: gk5885 Owner: gk5885 Dec 04, 2009 issue 83 (protobuf does not compile cleanly in 64-bit mode in Visual S...) commented on by robertshawnmitchell - This seems like a real problem and not something that should be addressed with a pragma. There will be data loss in these implicit casts so this bug should be fixed. This seems like a real problem and not something that should be addressed with a pragma. There will be data loss in these implicit casts so this bug should be fixed. Dec 02, 2009 issue 141 (CodedStream (at least in Java) behaves unpredictable when s...) reported by feinberg - CodedStream (at least in Java) behaves unpredictable when subsequent messages are sent over a Buffered Stream. I used the following approach: From server to client 1. Send a integer indicating the serialized size 2. Send the serialized message The client would attempt to read/decode the message until it encounters a size of -1. When using zero-copy method (CodedStream) this would behave unpredictably (confusing the boundary between the integer indicating message size and the message itself): public static <T extends Message.Builder> TreadToBuilderZc (DataInputStream inputStream, T builder,int size) throws IOException { CodedInputStream codedIn = CodedInputStream.newInstance(inputStream); codedIn.pushLimit(size); builder.mergeFrom(codedIn); return builder; } However, I was able to work around this issue by using a byte array: public static <T extends Message.Builder> T readToBuilder(DataInputStream inputStream, T builder, int size) throws IOException { byte[] input = new byte[size]; read(inputStream, input); builder.mergeFrom(input); return builder; } public static void read(InputStream stream, byte[] buffer) throws IOException { int read = 0; while(read < buffer.length) { int newlyRead = stream.read(buffer, read, buffer.length - read); if(newlyRead == -1) throw new EOFException("Attempt to read " + buffer.length + " bytes failed due to EOF."); read += newlyRead; } } The full sample source, including unit tests is attached (you can invoke the unit tests by doing "mvn test"): see src/test/java/TestStreaming.java; ideally both testNoZerocopy and testZerocopy should pass, however testNoZerocopy passes and testZerocopy fails. Thanks, CodedStream (at least in Java) behaves unpredictable when subsequent messages are sent over a Buffered Stream. I used the following approach: From server to client 1. Send a integer indicating the serialized size 2. Send the serialized message The client would attempt to read/decode the message until it encounters a size of -1. When using zero-copy method (Coded*Stream) this would behave unpredictably (confusing the boundary between the integer indicating message size and the message itself): public static <T extends Message.Builder> TreadToBuilderZc (DataInputStream inputStream, T builder,int size) throws IOException { CodedInputStream codedIn = CodedInputStream.newInstance(inputStream); codedIn.pushLimit(size); builder.mergeFrom(codedIn); return builder; } However, I was able to work around this issue by using a byte array: public static <T extends Message.Builder> T readToBuilder(DataInputStream inputStream, T builder, int size) throws IOException { byte[] input = new byte[size]; read(inputStream, input); builder.mergeFrom(input); return builder; } public static void read(InputStream stream, byte[] buffer) throws IOException { int read = 0; while(read < buffer.length) { int newlyRead = stream.read(buffer, read, buffer.length - read); if(newlyRead == -1) throw new EOFException("Attempt to read " + buffer.length + " bytes failed due to EOF."); read += newlyRead; } } The full sample source, including unit tests is attached (you can invoke the unit tests by doing "mvn test"): see src/test/java/TestStreaming.java; ideally both testNoZerocopy and testZerocopy should pass, however testNoZerocopy passes and testZerocopy fails. Thanks, Dec 01, 2009 issue 136 (serializedSize apparently not set) Status changed by jas...@google.com - Status: Fixed Status: Fixed Dec 01, 2009 issue 136 (serializedSize apparently not set) commented on by jas...@google.com - Fixed at r247. It's a good thing you caught this when you did - with Evan Jones's performance optimizations, writeTo(OutputStream output) started calling getSerializedSize(), which might have masked this problem even longer... Fixed at r247. It's a good thing you caught this when you did - with Evan Jones's performance optimizations, writeTo(OutputStream output) started calling getSerializedSize(), which might have masked this problem even longer... Dec 01, 2009 r247 (Fix Issue 136: the memoized serialized size for packed field...) committed by jasonh+p...@google.com - Fix Issue 136 : the memoized serialized size for packed fields may not be properly set. writeTo() may be invoked without a call to getSerializedSize(), so the generated serialization methods would write a length of 0 for non-empty packed fields. Just call getSerializedSize() at the beginning of writeTo(): although this means that we may compute the byte size needlessly when there are no packed fields, in practice, getSerializedSize() will already have been called - all of the writeTo() wrappers in AbstractMessageLite invoke it. Tested: new unittest case in WireFormatTest.java now passes Fix Issue 136 : the memoized serialized size for packed fields may not be properly set. writeTo() may be invoked without a call to getSerializedSize(), so the generated serialization methods would write a length of 0 for non-empty packed fields. Just call getSerializedSize() at the beginning of writeTo(): although this means that we may compute the byte size needlessly when there are no packed fields, in practice, getSerializedSize() will already have been called - all of the writeTo() wrappers in AbstractMessageLite invoke it. Tested: new unittest case in WireFormatTest.java now passes Nov 30, 2009 issue 137 (Runtime error in python bindings) Status changed by ken...@google.com - I'm not entirely sure what the problem is but I suspect it will go away in 2.3.0 -- much of the surrounding code has been rewritten. Status: Accepted I'm not entirely sure what the problem is but I suspect it will go away in 2.3.0 -- much of the surrounding code has been rewritten. Status: Accepted Nov 30, 2009 issue 137 (Runtime error in python bindings) commented on by Fenimore - Here's the stack trace: File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ File "build/bdist.linux-i686/egg/google/protobuf/descriptor.py", line 77, in GetOptions File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ File "build/bdist.linux-i686/egg/google/protobuf/descriptor.py", line 77, in GetOptions File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ File "build/bdist.linux-i686/egg/google/protobuf/descriptor.py", line 77, in GetOptions File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ .... Here's the stack trace: File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ File "build/bdist.linux-i686/egg/google/protobuf/descriptor.py", line 77, in GetOptions File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ File "build/bdist.linux-i686/egg/google/protobuf/descriptor.py", line 77, in GetOptions File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ File "build/bdist.linux-i686/egg/google/protobuf/descriptor.py", line 77, in GetOptions File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 309, in init File "build/bdist.linux-i686/egg/google/protobuf/reflection.py", line 1468, in __init__ .... Nov 29, 2009 issue 140 (Java Binding: Provide Factory Method to Generate RpcCallback...) reported by dtr...@davidtrott.com - What steps will reproduce the problem? 1. Attempt to Serialize the RpcUtil.newOneTimeCallback(). What is the expected output? What do you see instead? I want to be able to provide my own factory for generating the wrappers (in this case) so that I can generate Serializable wrappers. However it may be necessary to implement other interfaces in the future. What version of the product are you using? On what operating system? Protocol Buffers: 2.2.0a Linux: 2.6.21.5-smp #2 SMP i686 Intel(R) Core(TM)2 Duo CPU Java: SE Runtime Environment (build 1.6.0_16-b01) Please provide any additional information below. I am attaching a patch file that allows the wrapper generation to be replaced. Additionally I have included my Serialized wrapper provider for reference. What steps will reproduce the problem? 1. Attempt to Serialize the RpcUtil.newOneTimeCallback(). What is the expected output? What do you see instead? I want to be able to provide my own factory for generating the wrappers (in this case) so that I can generate Serializable wrappers. However it may be necessary to implement other interfaces in the future. What version of the product are you using? On what operating system? Protocol Buffers: 2.2.0a Linux: 2.6.21.5-smp #2 SMP i686 Intel(R) Core(TM)2 Duo CPU Java: SE Runtime Environment (build 1.6.0_16-b01) Please provide any additional information below. I am attaching a patch file that allows the wrapper generation to be replaced. Additionally I have included my Serialized wrapper provider for reference. Nov 28, 2009 issue 139 (Go implementation of Protocol Buffers.) reported by lost.goblin - It would be great to have Protocol Buffers support for the Go Programming language. It would be great to have Protocol Buffers support for the Go Programming language. Nov 28, 2009 issue 138 (Maven Protoc Plugin fails when a jar file contains a .proto ...) reported by dtr...@davidtrott.com - What steps will reproduce the problem? 1. Build project (A) so the that .proto file is embedded in the jar file. 2. Create project (B) that also has a .proto file. 3. Add (A) as a dependency of (B) 4. In project (B) Try to: mvn -X clean install What is the expected output? What do you see instead? I expect a successful build instead I get: [INFO] ---------------------------------------------------------------- [ERROR] BUILD FAILURE [INFO] ---------------------------------------------------------------- [INFO] Protoc failed to execute because: null [INFO] --------------------------------------------------------------------- [DEBUG] Trace XXXXX TRACE EDITED XXXXXXX Caused by: java.lang.IllegalArgumentException com.google.common.base.Preconditions.checkArgument(Preconditions.java:70) com.google.protobuf.maven.Protoc$Builder.addProtopathElement(Protoc.java:178) com.google.protobuf.maven.Protoc$Builder.addProtopathElements(Protoc.java:188) com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:109) ... 20 more What version of the product are you using? On what operating system? Linux with JDK 6 I took the code from: http://protobuf.googlecode.com/svn/branches/maven-plugin/tools/maven-plugin/ Please provide any additional information below. Fix: Change Line 213 of AbstractProtocMojo.java From: protoDirectories.add(uncompressedCopy); To: protoDirectories.add(uncompressedCopy.getParentFile()); This line adds the directory for inclusion into other .proto files however the bug is that it is adding the file itself. What steps will reproduce the problem? 1. Build project (A) so the that .proto file is embedded in the jar file. 2. Create project (B) that also has a .proto file. 3. Add (A) as a dependency of (B) 4. In project (B) Try to: mvn -X clean install What is the expected output? What do you see instead? I expect a successful build instead I get: [INFO] ---------------------------------------------------------------- [ERROR] BUILD FAILURE [INFO] ---------------------------------------------------------------- [INFO] Protoc failed to execute because: null [INFO] --------------------------------------------------------------------- [DEBUG] Trace XXXXX TRACE EDITED XXXXXXX Caused by: java.lang.IllegalArgumentException com.google.common.base.Preconditions.checkArgument(Preconditions.java:70) com.google.protobuf.maven.Protoc$Builder.addProtopathElement(Protoc.java:178) com.google.protobuf.maven.Protoc$Builder.addProtopathElements(Protoc.java:188) com.google.protobuf.maven.AbstractProtocMojo.execute(AbstractProtocMojo.java:109) ... 20 more What version of the product are you using? On what operating system? Linux with JDK 6 I took the code from: http://protobuf.googlecode.com/svn/branches/maven-plugin/tools/maven-plugin/ Please provide any additional information below. Fix: Change Line 213 of AbstractProtocMojo.java From: protoDirectories.add(uncompressedCopy); To: protoDirectories.add(uncompressedCopy.getParentFile()); This line adds the directory for inclusion into other .proto files however the bug is that it is adding the file itself. Nov 27, 2009 issue 66 (cannot install using easy_install) commented on by pokorf1 - OLA-0.5.0 needs protobuf version at least 2.1.0 ubuntu has protobuf ( and libprotoc ) version only 2.0.3 which too old. Do you have any solution for this please? I've tried to bypas this test in configure script, but it is dirty and not working. http://code.google.com/p/linux-lighting/ OLA-0.5.0 needs protobuf version at least 2.1.0 ubuntu has protobuf ( and libprotoc ) version only 2.0.3 which too old. Do you have any solution for this please? I've tried to bypas this test in configure script, but it is dirty and not working. http://code.google.com/p/linux-lighting/ Nov 27, 2009 issue 66 (cannot install using easy_install) commented on by pokorf1 - OLA-0.5.0 needs protobuf version at least 2.1.0 ubuntu has protobuf ( and libprotoc ) version only 2.0.3 which too old. Do you have any solution for this please? I've tried to bypas this test in configure script, but it is dirty and not working. OLA-0.5.0 needs protobuf version at least 2.1.0 ubuntu has protobuf ( and libprotoc ) version only 2.0.3 which too old. Do you have any solution for this please? I've tried to bypas this test in configure script, but it is dirty and not working. Nov 24, 2009 issue 136 (serializedSize apparently not set) commented on by jas...@google.com - Ah, I will take a look. Guess I ported this from C++ a little too literally. Ah, I will take a look. Guess I ported this from C++ a little too literally. Nov 24, 2009 issue 137 (Runtime error in python bindings) Owner changed by ken...@google.com - Can you provide some of the stack trace? Presumably it involves some sort of infinite recursion, so just cut out the repeating part. Owner: ken...@google.com Can you provide some of the stack trace? Presumably it involves some sort of infinite recursion, so just cut out the repeating part. Owner: ken...@google.com Nov 24, 2009 issue 136 (serializedSize apparently not set) changed by ken...@google.com - Ah, I see the problem. Indeed, it looks like the generated writeTo(CodedOutputStream) method is incorrectly assuming that getSerializedSize() has been called at least once. This is only an issue with packed fields. Jason, I believe this is your code. Want to fix it? :) The easiest solution would be to call getSerializedSize() at the start of writeTo() if any packed fields are present, though maybe there's a more elegant solution. Status: Accepted Owner: jas...@google.com Cc: ken...@google.com Ah, I see the problem. Indeed, it looks like the generated writeTo(CodedOutputStream) method is incorrectly assuming that getSerializedSize() has been called at least once. This is only an issue with packed fields. Jason, I believe this is your code. Want to fix it? :) The easiest solution would be to call getSerializedSize() at the start of writeTo() if any packed fields are present, though maybe there's a more elegant solution. Status: Accepted Owner: jas...@google.com Cc: ken...@google.com Nov 24, 2009 issue 137 (Runtime error in python bindings) reported by Fenimore - What steps will reproduce the problem? 1. Create a .proto with the following content: import "google/protobuf/descriptor.proto"; package a; extend google.protobuf.MessageOptions { optional string response_type = 50002; optional string expected_response = 50003; } extend google.protobuf.FieldOptions { optional string param_name = 51234; optional string type_name = 51235; } message DoFoo { optional int32 correlationId = 1; } 2. Create a python script as follows: from a_pb2 import DoFoo mh=DoFoo() mh.correlationId = 1 mh.SerializeToString() 3. Generate the proto file 4. Run the python script What is the expected output? What do you see instead? The python script should terminate normally, instead you see a huge stacktrace that ends with the error message "RuntimeError: maximum recursion depth exceeded" What version of the product are you using? On what operating system? 2.2.0 on fedora 10 Please provide any additional information below. Note that if you remove the FieldOptions extensions OR the MessageOptions extensions, things work just fine. Note also, that if you import bindings from a different proto file that does NOT use FileOptions and MessageOptions, you can proceed to use messages from that file just fine - it is only once you import messages from a proto with FileOptions AND MessageOptions being used. What steps will reproduce the problem? 1. Create a .proto with the following content: import "google/protobuf/descriptor.proto"; package a; extend google.protobuf.MessageOptions { optional string response_type = 50002; optional string expected_response = 50003; } extend google.protobuf.FieldOptions { optional string param_name = 51234; optional string type_name = 51235; } message DoFoo { optional int32 correlationId = 1; } 2. Create a python script as follows: from a_pb2 import DoFoo mh=DoFoo() mh.correlationId = 1 mh.SerializeToString() 3. Generate the proto file 4. Run the python script What is the expected output? What do you see instead? The python script should terminate normally, instead you see a huge stacktrace that ends with the error message "RuntimeError: maximum recursion depth exceeded" What version of the product are you using? On what operating system? 2.2.0 on fedora 10 Please provide any additional information below. Note that if you remove the FieldOptions extensions OR the MessageOptions extensions, things work just fine. Note also, that if you import bindings from a different proto file that does NOT use FileOptions and MessageOptions, you can proceed to use messages from that file just fine - it is only once you import messages from a proto with FileOptions AND MessageOptions being used. Nov 23, 2009 issue 136 (serializedSize apparently not set) commented on by josh.t.burdick - Should have done that originally... If you have protoc and protobuf.jar installed somewhere, then running "run.sh" below should show the issue. When I run it originally, "od" indicates that "foo.pb" contains: $ ./run.sh 0000000 0a 00 03 c8 03 12 00 0e d0 0f 0000012 But when I uncomment the call to getSerializedSize(), I get: $ ./run.sh size = 10 0000000 0a 03 03 c8 03 12 03 0e d0 0f 0000012 and the "payload sizes" are set correctly. I'm guessing that just calling getSerializedSize() in the writeTo() methods would fix this. Should have done that originally... If you have protoc and protobuf.jar installed somewhere, then running "run.sh" below should show the issue. When I run it originally, "od" indicates that "foo.pb" contains: $ ./run.sh 0000000 0a 00 03 c8 03 12 00 0e d0 0f 0000012 But when I uncomment the call to getSerializedSize(), I get: $ ./run.sh size = 10 0000000 0a 03 03 c8 03 12 03 0e d0 0f 0000012 and the "payload sizes" are set correctly. I'm guessing that just calling getSerializedSize() in the writeTo() methods would fix this. Nov 23, 2009 issue 85 (Emacs does not recognize Protocol Buffers specification file...) commented on by ken...@google.com - Can you explain the purpose of the patch to me? When is it needed? (Mostly it's setting off a red flag for me that I have no idea what to write in the SVN commit log. :) ) Can you explain the purpose of the patch to me? When is it needed? (Mostly it's setting off a red flag for me that I have no idea what to write in the SVN commit log. :) )

issue 144 (getExtension() throws ClassCastException on absent repeated ...) reported by timw.at.work - What steps will reproduce the problem? 1. Create two messages, Container and Item, with an extension to Container adding a repeated Item field 2. Obtain a message instance with no instances of the repeated extension field. 3. Attempt to call container.getExtension(...) and use the resulting List. What is the expected output? What do you see instead? Expected: I get an empty List, much like already happens with repeated non-extension fields. Actual: I get a ClassCastException: Item cannot be cast to java.util.List. What version of the product are you using? On what operating system? Protocol Buffers 2.2.0 on Ubuntu 9.04, 64-bit (downloaded from google code and built with the default 9.04 64-bit gcc). Please provide any additional information below. The problem appears to be in the generated getExtension() code: FieldDescriptor descriptor = extension.getDescriptor(); final Object value = extensions.getField(descriptor); if (value == null) { if (descriptor.getJavaType() == FieldDescriptor.JavaType.MESSAGE) { return (Type) extension.getMessageDefaultInstance(); } else { return (Type) extension.fromReflectionType( descriptor.getDefaultValue()); } } else { return (Type) extension.fromReflectionType(value); } Type is java.util.List, because the field is a repeated extension field. I did not step through the code (logically or with a debugger), but it seems like it might be hitting the JavaType.MESSAGE branch and attempting to return a default message instance instead of an empty list. Attached is a simple protocol definition and some java code that invokes the bug. A work-around, as indicated in the attached source, is to use getExtensionCount to guard against using a 0-count repetition.

What steps will reproduce the problem? 1. Create two messages, Container and Item, with an extension to Container adding a repeated Item field 2. Obtain a message instance with no instances of the repeated extension field. 3. Attempt to call container.getExtension(...) and use the resulting List. What is the expected output? What do you see instead? Expected: I get an empty List, much like already happens with repeated non-extension fields. Actual: I get a ClassCastException: Item cannot be cast to java.util.List. What version of the product are you using? On what operating system? Protocol Buffers 2.2.0 on Ubuntu 9.04, 64-bit (downloaded from google code and built with the default 9.04 64-bit gcc). Please provide any additional information below. The problem appears to be in the generated getExtension() code: FieldDescriptor descriptor = extension.getDescriptor(); final Object value = extensions.getField(descriptor); if (value == null) { if (descriptor.getJavaType() == FieldDescriptor.JavaType.MESSAGE) { return (Type) extension.getMessageDefaultInstance(); } else { return (Type) extension.fromReflectionType( descriptor.getDefaultValue()); } } else { return (Type) extension.fromReflectionType(value); } Type is java.util.List, because the field is a repeated extension field. I did not step through the code (logically or with a debugger), but it seems like it might be hitting the JavaType.MESSAGE branch and attempting to return a default message instance instead of an empty list. Attached is a simple protocol definition and some java code that invokes the bug. A work-around, as indicated in the attached source, is to use getExtensionCount to guard against using a 0-count repetition.

Today

Yesterday

Last 30 days