
kryo - issue #5
It is not possible to serialize classes without a zero-argument-constructor
What steps will reproduce the problem? 1. generate a class without default constructor 2. serialize it with kryo
What is the expected output? What do you see instead? expected: no output output: Exceptions about missing zero-argument constructor
What version of the product are you using? On what operating system? 0.92
why not using the internal way of creating an empty object? example source attached
- createObject.java 1.12KB
Comment #1
Posted on Dec 1, 2009 by Quick GiraffeYou can modify newInstance method in Serializer class to solve problem:
try {
Constructor<T> cons = null;
cons = type.getDeclaredConstructor(new Class[0]);
cons.setAccessible(true);
return cons.newInstance(new Object[0]);
} catch (Exception e) {
throw new RuntimeException("Class cannot be created (missing
no-arg constructor): " + type.getName(), e); }
Comment #2
Posted on Jan 6, 2010 by Grumpy CamelTo properly support serializing a class that has no zero-arg constructor, whatever constructor does exist must be called with proper parameters. This is not easily done. How would Kryo know which constructor to use if there are multiple?
Neither of the two proposed solutions allow serializing a class that has no zero-arg constructor.
To do it, you will need to implement a Serializer for the specific class you want to serialize. Eg: http://code.google.com/p/kryo/source/browse/trunk/test/com/esotericsoftware/kryo/serialize/SerializerTest.java#207 Note that you can make use of existing serializers.
Comment #3
Posted on Feb 26, 2010 by Happy ElephantI want to explain the Java 'original' way to serialize/deserialize Objects.
Given that we have the following class hierarchy: Object -> A -> B -> CS -> DS -> ES (The classes with S implement Serializable) at creating the Object ES with new ES(...) the constructors will be called in this order: Object, A, B, CS, DS, ES
at deserializing the Object ES you have to look for the first class which is not Assignable to Serializable. This class must to have a zero-arg constructor! Then with that constructor and the class info from ES, an Object will be created for that given class. The constructors that will be called are only: Object, A, B
The Solution to the original Problem would be to "interpret" a registered class as "Serializable" and look for the first class which is not registered, which must contain a zero-arg constructor.
On the long run it would be better to not use registered classes and instead implement some marker-interface.
Some sample code for a possible solution attached
- Serializer.java 1.83KB
Comment #4
Posted on Feb 27, 2010 by Happy BirdComment deleted
Comment #5
Posted on Feb 27, 2010 by Grumpy CamelComment deleted
Comment #6
Posted on Feb 27, 2010 by Grumpy CamelJava's default serialization has a lot of problems and "gotchas". One of these is that it provides an extralinguistic mechanism to construct classes. When the default deserialization occurs, it doesn't call a constructor to create an instance of a class. This must be carefully planned for and increases the chances of bugs and security holes.
Even if we wanted Kryo to support object construction without calling constructors, the mechanism Java's default serialization uses to do so is not made available for our use. It is not possible any other way. To create an instance of a class, a constructor must be called.
The code you posted doesn't compile, and I'm not really sure where you are going with it. [edit] Ahh, I see from the code in your original post that you are using sun.reflect.ReflectionFactory. I don't think this is appropriate for Kryo, since it is a general purpose library. Even if using a sun.* class was acceptable, I don't like the "magic backdoor" that the default serialization uses.
It is not hard to write a serializer for problematic classes. Please see the "testNoDefaultConstructor" method in the SerializerTest.java file linked above. I have just now added a more complex example to that test showing how to use FieldSerializer with a class lacking a zero argument constructor.
Comment #7
Posted on Mar 24, 2010 by Grumpy BearYou wrote:
Even if we wanted Kryo to support object construction without calling constructors, the mechanism Java's default serialization uses to do so is not made available for our use. It is not possible any other way. To create an instance of a class, a constructor must be called. With a sun jdk it's possible via sun.reflect.ReflectionFactory:
final Constructor constructor = ReflectionFactory.getReflectionFactory().newConstructorForSerialization( cls, Object.class.getDeclaredConstructor( new Class[0] ) ); constructor.setAccessible( true ); return (Collection) constructor.newInstance( new Object[0] );
For other jvms there should be similar possibilities.
Cheers, Martin
Comment #8
Posted on Mar 24, 2010 by Grumpy CamelAgain, I don't feel sun.* classes are appropriate for use in a general purpose library. Even if they were, the "magic backdoor" that the default serialization uses is a bad idea.
Comment #9
Posted on Mar 24, 2010 by Grumpy BearThis (ReflectionFactory) btw. also solves deserialization of private classes...
I'd also be interested in deserialization of classes with no default constructor and private classes, I'm still interested in kryo to integrate as serialization strategy into the memcached-session-manager.
Cheers, Martin
Comment #10
Posted on Mar 25, 2010 by Grumpy CamelReflectionFactory doesn't call a constructor to create an instance of a class. This must be carefully planned for and increases the chances of bugs and security holes. In fact, the serialization proxy pattern is recommended (Effective Java by Joshua Bloch, item 78) to bypass the complexities and potential problems with Java's built-in serialization. IMO, if you have to go to extra trouble anyway, you might as well just write a Kryo serializer that handles reading/writing constructor parameters and non-zero arg construction.
If you really want ReflectionFactory functionality, Kryo serializers are extensible. I believe comment 3 above provides the code to extend FieldSerializer to create instances with ReflectionFactory.
Regarding using Kryo for memcached-session-manager, I haven't forgotten about you! We have come part way as Kryo now supports serializing unregistered classes (though I feel it still needs some refinement). We don't yet have forward/backward compatibility, but it is on my todo list.
Comment #11
Posted on Mar 25, 2010 by Grumpy BearGreat, thanx for this info! Really looking forward to using kryo! :-)
Comment #12
Posted on Mar 26, 2010 by Quick HippoI'm thinking loud here regarding "Best effort discover, record and invoke of appropriate constructor to call". People can add to this idea/steps and see if we can do better than invoke newInstance() and fail if there isn't a no-arg constructor.
In writeObject(): - If class has no arg constructor do the usual, if not then below
- Identify a least argument constructor whose parameters (by type and name) can be mapped uniquely to the object fields. 1) Look for all one arg constructors and inspect the fields (type and names) and see if any of them can uniquely be mapped to the constcutor param. If so, that is the candidate constructor to be recorded for use by readObject(). 2) If found, serialize the Constructor object (name, params) with the values of the params to be passed for construction to the buffer (may be some standardized marker can separate constructor, constructor params from the object fields). 3) If not, repeat (1) and (2) for two arg, three arg and so on progressively till all declaredConstructors are exhausted.
readObject(): - If after reading the Class info, if there is a Standard marker that indicates a constructor related data then read the Constructor info, read params and invoke the constructor. - Once the object is constructed with no exceptions/errors, go about de-serializing the object.
Feedback/criticism welcome
Thanks, -Prasad
Comment #13
Posted on Mar 27, 2010 by Grumpy CamelThat could work in some situations, but fails when there are two fields the same type as a single argument constructor, or if multiple constructor arguments are of the same type. Even when the proposed approach can be applied, there is no guarantee that the chosen fields really should be applied to the constructor. I think we have to be sure. If we guess then sometimes we will be wrong, which will cause a major and possibly difficult to detect problem.
The official way to map fields to properties is with the java.beans.ConstructorProperties annotation, which is Java 1.6+ only. A second mechanism would be needed both for Java 1.5 and for third party classes that don't use the annotation. I suppose it would work something like FieldSerializer#setConstructorFields(String...). However, it seems that this is only slightly less work than just writing a small serializer to construct the object. Still, if users would find this useful and feel it best solves the problem, I can reopen this bug and implement this proposed solution.
Comment #14
Posted on Mar 29, 2010 by Quick HippoI'd vote for Kryo supporting the @ConstructorProperties annotation. Those of us using Java 1.6 can use this. Others have nothing to lose w.r.t current functionality. Also, I'm not too sure if implicitly deriving the constructor props by type and name match will be an issue in most applications. The object (being constructed) by itself may not have much impact as its state will be updated from the serialized state after the construction. It can however have side effects viz. initializing the state of another object or opening a resource not needed etc. If we provide a way to turn-on/off the implicit derivation of constructor props then applications that don't see such side effects can turn-on and others can keep it off.
Comment #15
Posted on Apr 5, 2010 by Grumpy CamelWill look into some sort of constructor properties support.
Comment #16
Posted on Apr 7, 2010 by Massive Oxhow about a zero arg private constructor? this could be accessed using reflection and would allow an effectively immutable object to be serialized without requiring a public zero arg constructor that would violate immutability.
Comment #17
Posted on Apr 7, 2010 by Grumpy CamelI think that is quite a clever idea! Support for private constructors is in SVN, r98. Thanks!
People probably still want support for mapping fields to constructor parameters for classes were the source cannot be modified. This will get implemented eventually.
Comment #18
Posted on Apr 7, 2010 by Massive OxWow that's got to be the fastest response i've seen ever! Certainly encurages comments. Could I suggest another method?
private void defensivelyCopyState();
It could be called using reflection, optionally allowing an object to make defensive copies of internal mutable state, eg: guard against stolen references, so contained mutable objects can be effectively immutable.
N.B. I've just found your project, looking for a new serialization framework after realising ObjectStreamClass prevents ClassLoaders from being garbage collected, while trying to work out how to handle package versioning and ClassLoader isolation for remote objects.
Thanks,
Peter
Comment #19
Posted on Apr 8, 2010 by Grumpy CamelSo Kryo would call defensivelyCopyState, if it exists? When would this happen? One issue with using setAccessible is that it is not available in some environments, such as applets.
I've thought of adding an interface named Deserialization, which would have a "deserialized" method which is called after an object has been completely deserialized. Currently the only place to take action after deserialization is in the registered serializer.
Comment #20
Posted on Apr 8, 2010 by Massive OxYes, if it exists, if not or where the environment didn't support it (if private), it wouldn't be called. The only consequence of not calling it would be that the state would not be defensively copied if it wasn't called, the references would still be valid. I wonder if it would be possible to register a null serializer that calls the method when no serializer exists and or insert a serializer into the inheritance hierachy for serializers that calls the method if it exists?
By the sound of your Deserializer interface you've already thought about this and it sounds like a good decision.
If the method is public the implementor should set a volatile boolean variable to ensure it is only executed once, especially where immutability is used for thread safety.
I'm going to have to study your code to get a better understanding.
Cheers & Thanks,
Peter.
Comment #21
Posted on Apr 8, 2010 by Massive OxN.B Looks like you've done heaps of work here, the code looks really clean too. I see you've got a Serializer class, could it be used to execute your Deserializer method?
Comment #22
Posted on Apr 8, 2010 by Grumpy CamelLet's move the "deserialized" method discussion over to the discussion group: http://groups.google.com/group/kryo-users
Comment #23
Posted on Oct 10, 2010 by Grumpy Camel(No comment was entered for this change.)
Comment #24
Posted on Oct 3, 2011 by Quick HippoIs it possible yet to deserialize classes without 0-arg ctors? Ran into this via http://stackoverflow.com/questions/7590557/simple-hassle-free-zero-boilerplate-serialization-in-scala-java-similar-to-pyth. (Bear in mind that your users aren't necessarily the authors of the classes they want to serialize, so even if we were willing to pollute our codebase with superfluous 0-arg ctors, it would not be possible for third-party code.)
Comment #25
Posted on Oct 3, 2011 by Grumpy BearAlso added as comment on SO: If you're using a sun/oracle jvm you can use http://github.com/magro/kryo-serializers for deserializing objects without a 0-arg constructor: just change "new Kryo()" to "new KryoReflectionFactorySupport()"
Comment #26
Posted on Apr 17, 2012 by Grumpy Camelv2 supports Objenesis, which can create classes without a zero argument constructor.
Status: Fixed
Labels:
Type-Enhancement
Priority-Medium