protobuf-perlxs - Project Hosting on Google Code

Older Oct 26, 2009 issue 6 (optimize_for = LITE_RUNTIME causes compilation errors) reported by mika.raento - I have .proto files that I want to use with a mobile client talking to a perl server. The mobile code needs to be compiled with optimize_for = LITE_RUNTIME but the generated wrappers are wrong as they don't take this into account as they unconditionally try to link in calls to non-lite methods. The workaround is to preprocess the .protos in Makefile.PL to strip out the LITE_RUNTIME for the perl target. Would be nice to either make the perl target also understand LITE_RUNTIME. (though I think having LITE_RUNTIME be a property of a .proto rather than a protoc argument is probably a sub-optimal design). I have .proto files that I want to use with a mobile client talking to a perl server. The mobile code needs to be compiled with optimize_for = LITE_RUNTIME but the generated wrappers are wrong as they don't take this into account as they unconditionally try to link in calls to non-lite methods. The workaround is to preprocess the .protos in Makefile.PL to strip out the LITE_RUNTIME for the perl target. Would be nice to either make the perl target also understand LITE_RUNTIME. (though I think having LITE_RUNTIME be a property of a .proto rather than a protoc argument is probably a sub-optimal design). May 29, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) Status changed by d...@daveb.net - Status: Fixed Status: Fixed May 29, 2009 protobuf-perlxs-0.8.tar.gz (protobuf-perlxs 0.8) file uploaded by d...@daveb.net - Labels: Featured Type-Source OpSys-Linux Labels: Featured Type-Source OpSys-Linux May 29, 2009 r23 (0.8) committed by d...@daveb.net - 0.8 0.8 Apr 24, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) commented on by dkindlund - Great! Thanks for your help. -- Darien Great! Thanks for your help. -- Darien Apr 24, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) commented on by dbaile - I'll have the generated code call IsInitialized() before serializing, and croak() if the IsInitialized() returns false. This should take care of it - as far as I know, the only area where we need to worry about abort() is when serializing with a missing required field. This should be ready this weekend. -dave I'll have the generated code call IsInitialized() before serializing, and croak() if the IsInitialized() returns false. This should take care of it - as far as I know, the only area where we need to worry about abort() is when serializing with a missing required field. This should be ready this weekend. -dave Apr 22, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) commented on by dkindlund - Hi Dave, Thanks for the updates; sorry about the delayed ACK. For temporary solution #1, is there a recommended place to put the "#define NDEBUG" in the existing code? Perhaps a quick patch? Also, if you're close to implementing #2, then I'll hold off on using NDEBUG and wait for your reply. -- Darien Hi Dave, Thanks for the updates; sorry about the delayed ACK. For temporary solution #1, is there a recommended place to put the "#define NDEBUG" in the existing code? Perhaps a quick patch? Also, if you're close to implementing #2, then I'll hold off on using NDEBUG and wait for your reply. -- Darien Mar 23, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) commented on by d...@daveb.net - Actually, I didn't look into this carefully enough, and it turns out there are a couple of workarounds that might be better, possibly even a lot better... 1) Kenton Varda informed me that generating a #define NDEBUG will prevent the abort(). 2) He also mentioned that I can generate an IsInitialized() before the serialization, and use this to catch errors before they become fatal. I think I am going to look through the code and see if judicious use of (2) will allow me to trap all of these would-be fatal errors and then croak() with the appropriate message so it can be caught in an eval {}. -dave Actually, I didn't look into this carefully enough, and it turns out there are a couple of workarounds that might be better, possibly even a lot better... 1) Kenton Varda informed me that generating a #define NDEBUG will prevent the abort(). 2) He also mentioned that I can generate an IsInitialized() before the serialization, and use this to catch errors before they become fatal. I think I am going to look through the code and see if judicious use of (2) will allow me to trap all of these would-be fatal errors and then croak() with the appropriate message so it can be caught in an eval {}. -dave Mar 21, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) changed by d...@daveb.net - Hi Darien, It looks like libprotobuf is calling abort() instead of throwing an exception (Google apparently frowns on the use of C++ exceptions in their code). It's harder to catch an abort(), but I think we can do it with a signal handler and setjmp/longjmp. I tried this out in the Person example you gave (by modifying the generated XS code) and got it to work, such that the program seemed to recover without any side effects. I'll put this into the next release; the generation of this extra code may or may not be controlled by some sort of protoxs command line option, or something like that. -dave Status: Accepted Owner: d...@daveb.net Hi Darien, It looks like libprotobuf is calling abort() instead of throwing an exception (Google apparently frowns on the use of C++ exceptions in their code). It's harder to catch an abort(), but I think we can do it with a signal handler and setjmp/longjmp. I tried this out in the Person example you gave (by modifying the generated XS code) and got it to work, such that the program seemed to recover without any side effects. I'll put this into the next release; the generation of this extra code may or may not be controlled by some sort of protoxs command line option, or something like that. -dave Status: Accepted Owner: d...@daveb.net Mar 20, 2009 issue 5 (eval { } doesn't catch protobuf->pack() fatal errors) reported by dkindlund - What steps will reproduce the problem? 1. Create a protobuf message with incomplete data in ->new() 2. Call the usual message->pack() 3. libprotobuf FATAL google/protobuf/message.cc:212] CHECK failed: What is the expected output? What do you see instead? When I wrap the message->pack() routing inside an eval {} block, I expect to be able to catch the exception. Instead, it seems that the exception thrown is somehow bypassing the eval {} block. What version of the product are you using? On what operating system? protobuf-perlxs v0.6 perl, v5.8.8 built for i486-linux-gnu-thread-multi Please provide any additional information below. Here's an example code: Using protobuf: message Person { required int32 id = 1; required string name = 2; optional string email = 3; } ---- Perl Code: use Person; my $person = Person->new({id => 1}); eval { $person->pack(); }; ---- The exception should be handled. Please advise on how to gracefully handle these exceptions from XS code. Thanks. -- Darien What steps will reproduce the problem? 1. Create a protobuf message with incomplete data in ->new() 2. Call the usual message->pack() 3. libprotobuf FATAL google/protobuf/message.cc:212] CHECK failed: What is the expected output? What do you see instead? When I wrap the message->pack() routing inside an eval {} block, I expect to be able to catch the exception. Instead, it seems that the exception thrown is somehow bypassing the eval {} block. What version of the product are you using? On what operating system? protobuf-perlxs v0.6 perl, v5.8.8 built for i486-linux-gnu-thread-multi Please provide any additional information below. Here's an example code: Using protobuf: message Person { required int32 id = 1; required string name = 2; optional string email = 3; } ---- Perl Code: use Person; my $person = Person->new({id => 1}); eval { $person->pack(); }; ---- The exception should be handled. Please advise on how to gracefully handle these exceptions from XS code. Thanks. -- Darien Feb 25, 2009 r22 (Branch SET_VERSION. Implement setting the version of a gen...) committed by google....@sethdaniel.org - Branch SET_VERSION. Implement setting the version of a generated .pm using protobuf custom options. Branch SET_VERSION. Implement setting the version of a generated .pm using protobuf custom options. Jan 27, 2009 r21 (version info patch from Jeremy Leader) committed by d...@daveb.net - version info patch from Jeremy Leader version info patch from Jeremy Leader Jan 23, 2009 protobuf-perlxs-0.7.tar.gz (protobuf-perlxs 0.7) file uploaded by d...@daveb.net - Labels: Featured Type-Source OpSys-Linux Labels: Featured Type-Source OpSys-Linux Jan 23, 2009 issue 4 (String and bytes types are truncated when using NUL characte...) changed by d...@daveb.net - Thanks (again) Jason! I've committed your fix with a slight change to avoid the extra copy of the field type is 'bytes' (since there are methods available that don't require the construction of an intermediate string to hold the data). This will go out in 0.7, due shortly. -dave Status: Fixed Owner: d...@daveb.net Thanks (again) Jason! I've committed your fix with a slight change to avoid the extra copy of the field type is 'bytes' (since there are methods available that don't require the construction of an intermediate string to hold the data). This will go out in 0.7, due shortly. -dave Status: Fixed Owner: d...@daveb.net Jan 23, 2009 r20 (Issue 4, prep for 0.7) committed by d...@daveb.net - Issue 4 , prep for 0.7 Issue 4 , prep for 0.7 Jan 22, 2009 r19 (remove unused code) committed by d...@daveb.net - remove unused code remove unused code Jan 21, 2009 r18 (fix compile error) committed by d...@daveb.net - fix compile error fix compile error Jan 21, 2009 r17 (fix bug in generated ZeroCopyStream code) committed by d...@daveb.net - fix bug in generated ZeroCopyStream code fix bug in generated ZeroCopyStream code Jan 21, 2009 issue 4 (String and bytes types are truncated when using NUL characte...) reported by jason.hord - I noticed that strings with embedded NUL ("\0") characters in them were being truncated when using either the HASHREF form of the ->new() constructor or the ->set_FIELD() accessor for the generated XS class. The problem appears to be in the conversion of the char * type to the string type without referencing the length of the char * array. I've included a patch that performs the necessary steps to extract the length of the char * type from the SV pointer and use both values when constructing string types. Please note that my experience with both XS and C++ is extremely limited, so the patch is probably quite sub-optimal and is provided as a proof-of-concept. You can test this issue with something like the following: [person.proto] message Person { required string name = 1; required bytes image = 2; } [person.pl] use Person; my $p = Person->new({ name => "jason\0hord" }); $p->set_image( "binary\0goop" ); print 'name: ', $p->name, "\n", 'image: ', $p->image, "\n", join( ' ', unpack '(H2)', $p->pack ), "\n" ; [pre-patch output] name: jason image: binary 0a 05 6a 61 73 6f 6e 12 06 62 69 6e 61 72 79 [post-patch output] name: jason^@hord image: binary^@goop 0a 0a 6a 61 73 6f 6e 00 68 6f 72 64 12 0b 62 69 6e 61 72 79 00 67 6f 6f 70 I noticed that strings with embedded NUL ("\0") characters in them were being truncated when using either the HASHREF form of the ->new() constructor or the ->set_FIELD() accessor for the generated XS class. The problem appears to be in the conversion of the char type to the string type without referencing the length of the char * array. I've included a patch that performs the necessary steps to extract the length of the char * type from the SV pointer and use both values when constructing string types. Please note that my experience with both XS and C++ is extremely limited, so the patch is probably quite sub-optimal and is provided as a proof-of-concept. You can test this issue with something like the following: [person.proto] message Person { required string name = 1; required bytes image = 2; } [person.pl] use Person; my $p = Person->new({ name => "jason\0hord" }); $p->set_image( "binary\0goop" ); print 'name: ', $p->name, "\n", 'image: ', $p->image, "\n", join( ' ', unpack '(H2)*', $p->pack ), "\n" ; [pre-patch output] name: jason image: binary 0a 05 6a 61 73 6f 6e 12 06 62 69 6e 61 72 79 [post-patch output] name: jason^@hord image: binary^@goop 0a 0a 6a 61 73 6f 6e 00 68 6f 72 64 12 0b 62 69 6e 61 72 79 00 67 6f 6f 70 Jan 21, 2009 issue 3 (Off-by-one error in XS generated code for initializing repea...) Status changed by d...@daveb.net - Hi Darien, I just uploaded protobuf-perlxs 0.6 to the downloads area. -dave Status: Fixed Hi Darien, I just uploaded protobuf-perlxs 0.6 to the downloads area. -dave Status: Fixed Jan 21, 2009 protobuf-perlxs-0.6.tar.gz (protobuf-perlxs 0.6) file uploaded by d...@daveb.net - Labels: Featured Type-Source OpSys-Linux Labels: Featured Type-Source OpSys-Linux Jan 21, 2009 r16 (prep for 0.6) committed by d...@daveb.net - prep for 0.6 prep for 0.6 Jan 20, 2009 issue 3 (Off-by-one error in XS generated code for initializing repea...) commented on by dkindlund - Any chance 0.6 is getting released shortly? I was going to wait for 0.6 because of this bug, but I'll apply the patch manually if 0.6 is getting delayed. Thanks, -- Darien Any chance 0.6 is getting released shortly? I was going to wait for 0.6 because of this bug, but I'll apply the patch manually if 0.6 is getting delayed. Thanks, -- Darien Jan 16, 2009 issue 3 (Off-by-one error in XS generated code for initializing repea...) changed by dbaile - Thanks Jason! I'll put this fix into 0.6, which should be ready over the weekend. -dave Status: Accepted Owner: d...@daveb.net Thanks Jason! I'll put this fix into 0.6, which should be ready over the weekend. -dave Status: Accepted Owner: d...@daveb.net Jan 14, 2009 issue 3 (Off-by-one error in XS generated code for initializing repea...) reported by jason.hord - After doing some experimentation I found an issue trying to initialize repeated elements when passing a HASHREF to ->new() for a protoxs generated class. I was able to track it down to an off-by-one condition in the for() loop that initializes the repeated item in the from_hashref() helper. I've attached a patch that solves the problem on my end. After doing some experimentation I found an issue trying to initialize repeated elements when passing a HASHREF to ->new() for a protoxs generated class. I was able to track it down to an off-by-one condition in the for() loop that initializes the repeated item in the from_hashref() helper. I've attached a patch that solves the problem on my end. Jan 14, 2009 protobuf-perlxs-0.5.tar.gz (protobuf-perlxs 0.5) file uploaded by d...@daveb.net - Labels: Featured Type-Source OpSys-Linux Labels: Featured Type-Source OpSys-Linux Jan 14, 2009 r15 (prep for 0.5) committed by d...@daveb.net - prep for 0.5 prep for 0.5 Jan 11, 2009 r14 (fixed small error) committed by d...@daveb.net - fixed small error fixed small error Jan 11, 2009 r13 (fixed erroneous usage example) committed by d...@daveb.net - fixed erroneous usage example fixed erroneous usage example Jan 11, 2009 protobuf-perlxs-0.4.tar.gz (protobuf-perlxs 0.4) file uploaded by d...@daveb.net - Labels: Featured Type-Source OpSys-Linux Labels: Featured Type-Source OpSys-Linux Jan 11, 2009 r12 (prepare for 0.4) committed by d...@daveb.net - prepare for 0.4 prepare for 0.4 Nov 19, 2008 issue 1 (Undefined symbols when compiling on Mac OSX.) Status changed by d...@daveb.net - Status: Fixed Status: Fixed Nov 19, 2008 issue 2 (Nested messages have incorrect class) Status changed by d...@daveb.net - Status: Fixed Status: Fixed Nov 19, 2008 protobuf-perlxs-0.3.tar.gz (protobuf-perlxs 0.3) file uploaded by d...@daveb.net - Labels: Featured Type-Source OpSys-Linux Labels: Featured Type-Source OpSys-Linux Nov 19, 2008 r11 (prep for 0.3) committed by dbaile - prep for 0.3 prep for 0.3 Nov 19, 2008 r10 (Issue 2 - wrong class name for embedded messages) committed by dbaile - Issue 2 - wrong class name for embedded messages Issue 2 - wrong class name for embedded messages Nov 05, 2008 issue 2 (Nested messages have incorrect class) changed by d...@daveb.net - Thanks John! I will include this in the next release (0.3, due this weekend). -dave Status: Accepted Owner: d...@daveb.net Thanks John! I will include this in the next release (0.3, due this weekend). -dave Status: Accepted Owner: d...@daveb.net Nov 03, 2008 issue 2 (Nested messages have incorrect class) reported by j...@mediaOnFire.com - What steps will reproduce the problem? 1. Use a nested message construct. 2. After compilation and use, do an ISA on the nested message and the class is that of the nesting class, not the nested class. What is the expected output? What do you see instead? Here's a proto that will show the issue: ------------------ package Report; message Statistic { optional string key = 1; optional string val = 2; } message Query { optional string name = 1; repeated Statistic stats = 2; } ------------------ What version of the product are you using? On what operating system? 0.2 build on Linux 2.6.21/gcc 4.2.4/perl 5.8.8 Please provide any additional information below. I attached a patch - simple fix. Thanks for coding this up. -John What steps will reproduce the problem? 1. Use a nested message construct. 2. After compilation and use, do an ISA on the nested message and the class is that of the nesting class, not the nested class. What is the expected output? What do you see instead? Here's a proto that will show the issue: ------------------ package Report; message Statistic { optional string key = 1; optional string val = 2; } message Query { optional string name = 1; repeated Statistic stats = 2; } ------------------ What version of the product are you using? On what operating system? 0.2 build on Linux 2.6.21/gcc 4.2.4/perl 5.8.8 Please provide any additional information below. I attached a patch - simple fix. Thanks for coding this up. -John Oct 07, 2008 issue 1 (Undefined symbols when compiling on Mac OSX.) changed by d...@daveb.net - I believe the missing symbol(s) can be found in libprotobuf.so. I committed a new src/Makefile.am with the extra library on the protoxs_LDADD line. This change will be included in version 0.3, but if you are able to test it in your environment before I make that release, please do so and let me know if the change gives you a successful build. You may need to add --enable-maintainer-mode to ./configure after you update the src/Makefile.am. -dave Status: Started Owner: d...@daveb.net Labels: OpSys-OSX I believe the missing symbol(s) can be found in libprotobuf.so. I committed a new src/Makefile.am with the extra library on the protoxs_LDADD line. This change will be included in version 0.3, but if you are able to test it in your environment before I make that release, please do so and let me know if the change gives you a successful build. You may need to add --enable-maintainer-mode to ./configure after you update the src/Makefile.am. -dave Status: Started Owner: d...@daveb.net Labels: OpSys-OSX Oct 07, 2008 r9 (Issue 1 - kTypeToCppTypeMap is in libprotobuf) committed by d...@daveb.net - Issue 1 - kTypeToCppTypeMap is in libprotobuf Issue 1 - kTypeToCppTypeMap is in libprotobuf Oct 06, 2008 issue 1 (Undefined symbols when compiling on Mac OSX.) reported by a...@wardley.org - What steps will reproduce the problem? 1. ./configure --with-protobuf=/usr/local/ 2. make What is the expected output? What do you see instead? Making all in src g++ -g -O2 -L/usr/local/lib -o protoxs perlxs_generator.o perlxs_helpers.o main.o -lprotoc Undefined symbols: "google::protobuf::FieldDescriptor::kTypeToCppTypeMap", referenced from: (and lots, lots more undefined symbols - see attached file) It looks like it can't find libprotoc.a but it's installed there in my /usr/local/lib What version of the product are you using? On what operating system? protobuf 2.0.2, protobuf-perlxs-0.2, Mac OXS 10.5.4 Please provide any additional information below. typescript attached showing build session What steps will reproduce the problem? 1. ./configure --with-protobuf=/usr/local/ 2. make What is the expected output? What do you see instead? Making all in src g++ -g -O2 -L/usr/local/lib -o protoxs perlxs_generator.o perlxs_helpers.o main.o -lprotoc Undefined symbols: "google::protobuf::FieldDescriptor::kTypeToCppTypeMap", referenced from: (and lots, lots more undefined symbols - see attached file) It looks like it can't find libprotoc.a but it's installed there in my /usr/local/lib What version of the product are you using? On what operating system? protobuf 2.0.2, protobuf-perlxs-0.2, Mac OXS 10.5.4 Please provide any additional information below. typescript attached showing build session Oct 02, 2008 r8 (prep for 0.2) committed by d...@daveb.net - prep for 0.2 prep for 0.2 Sep 28, 2008 r7 (eliminate the use of CamelCase) committed by dbaile - eliminate the use of CamelCase eliminate the use of CamelCase Sep 19, 2008 r6 (added Storable comparison) committed by dbaile - added Storable comparison added Storable comparison Sep 19, 2008 r5 (added license) committed by dbaile - added license added license Sep 19, 2008 r4 (add examples to EXTRA_DIST) committed by dbaile - add examples to EXTRA_DIST add examples to EXTRA_DIST Sep 19, 2008 r3 (added simple example) committed by dbaile - added simple example added simple example Sep 18, 2008 r2 (initial work) committed by dbaile - initial work initial work Aug 28, 2008 Project protobuf-perlxs created by dbaile - Protocol Buffers for Perl/XS Protocol Buffers for Perl/XS

issue 4 (String and bytes types are truncated when using NUL characte...) reported by jason.hord - I noticed that strings with embedded NUL ("\0") characters in them were being truncated when using either the HASHREF form of the ->new() constructor or the ->set_FIELD() accessor for the generated XS class. The problem appears to be in the conversion of the char * type to the string type without referencing the length of the char * array. I've included a patch that performs the necessary steps to extract the length of the char * type from the SV pointer and use both values when constructing string types. Please note that my experience with both XS and C++ is extremely limited, so the patch is probably quite sub-optimal and is provided as a proof-of-concept. You can test this issue with something like the following: [person.proto] message Person { required string name = 1; required bytes image = 2; } [person.pl] use Person; my $p = Person->new({ name => "jason\0hord" }); $p->set_image( "binary\0goop" ); print 'name: ', $p->name, "\n", 'image: ', $p->image, "\n", join( ' ', unpack '(H2)*', $p->pack ), "\n" ; [pre-patch output] name: jason image: binary 0a 05 6a 61 73 6f 6e 12 06 62 69 6e 61 72 79 [post-patch output] name: jason^@hord image: binary^@goop 0a 0a 6a 61 73 6f 6e 00 68 6f 72 64 12 0b 62 69 6e 61 72 79 00 67 6f 6f 70

I noticed that strings with embedded NUL ("\0") characters in them were being truncated when using either the HASHREF form of the ->new() constructor or the ->set_FIELD() accessor for the generated XS class. The problem appears to be in the conversion of the char * type to the string type without referencing the length of the char * array. I've included a patch that performs the necessary steps to extract the length of the char * type from the SV pointer and use both values when constructing string types. Please note that my experience with both XS and C++ is extremely limited, so the patch is probably quite sub-optimal and is provided as a proof-of-concept. You can test this issue with something like the following: [person.proto] message Person { required string name = 1; required bytes image = 2; } [person.pl] use Person; my $p = Person->new({ name => "jason\0hord" }); $p->set_image( "binary\0goop" ); print 'name: ', $p->name, "\n", 'image: ', $p->image, "\n", join( ' ', unpack '(H2)*', $p->pack ), "\n" ; [pre-patch output] name: jason image: binary 0a 05 6a 61 73 6f 6e 12 06 62 69 6e 61 72 79 [post-patch output] name: jason^@hord image: binary^@goop 0a 0a 6a 61 73 6f 6e 00 68 6f 72 64 12 0b 62 69 6e 61 72 79 00 67 6f 6f 70

Older