IssueTracker

i think we'd need to see the actual dx command line. my assumption is that you're not actually forcing it to run in jumbo-only mode.

Hello. Thank you for responding.

Do you mean the command that eclipse run when I execute the app?
I'm confident that the dex.force.jumbo=true makes a difference since without I get:
[2012-11-28 21:33:36 - Dex Loader] Unable to execute dex: Cannot merge new index 67974 into a non-jumbo instruction!
While with it I do get the application to try and install on the phone (then it fails with the above dexopt error).

I have enabled verbose build output, though I don't see any explicit dex command on the eclipse console. Just stuff like:
...
[2012-11-28 21:39:31 - Find In Files] Dx processing org/apache/poi/openxml4j/opc/TargetMode.class...
...
[2012-11-28 21:40:19 - Find In Files] Dx processing classes.dex...\

What further data could I provide?
Thank you.

did you make a clean build after forcing jumbo mode?
i'm wondering if you could have a jar file that wasn't in jumbo mode (not sure how the dex merger deals with this0

I have a main project that references 3 android libraries projects.
Each library has various 3rd party jars in its build path.
Also, one library is referencing two Java projects (which are not android libraries themselves).

In fact, I only had dex.force.jumbo=true in the main project project.properties file.
Per your suggestion, I've added dex.force.jumbo=true to all 3 android libraries project.properties files.
then cleaned up with: Project > Clean ... > Clean all projects.
Uninstalled the app on the device.
Tried running the main project, it built for 2-3 minutes, tried to install, but :( installation failed much like before (see below).

I assume that setting the flag on a library takes care of all of this library's jar/java artifacts in regards to compiling the right Dex - Any other use case of a jar file not in jumbo mode I should worry about?

Would you like me to upload any log/dump/app files?
Thanks.

12-02 00:26:23.597: I/PackageManager(209): Running dexopt on: com.mypackage.myapp
12-02 00:26:24.093: I/power(209): *** set_screen_state 0
12-02 00:26:24.148: D/SurfaceFlinger(134): About to give-up screen, flinger = 0xcf0918
12-02 00:26:24.585: D/Sensors(209): Smb380Sensor::~enable(0, 0)
12-02 00:26:24.585: D/Sensors(209): Smb380Sensor::~enable(0, 0) open /sys/class/input/event1/device/enable
12-02 00:26:24.589: D/Sensors(209): Smb380Sensor::~enable(0, 0) opened /sys/class/input/event1/device/enable
12-02 00:26:24.601: D/Sensors(209): Smb380Sensor::~setDelay(0, 66667000)
12-02 00:26:26.328: D/AccelerometerListener(584): enable(false)
12-02 00:26:27.250: D/OpenGLRenderer(14733): Flushing caches (mode 2)
12-02 00:26:27.250: D/OpenGLRenderer(14733): Flushing caches (mode 0)
12-02 00:26:27.250: W/ManagedEGLContext(14733): doTerminate failed: EGL count is 2 but managed count is 1
12-02 00:26:29.867: V/TransportControlView(209): Create TCV com.android.internal.widget.TransportControlView@423660a8
12-02 00:26:29.878: E/dalvikvm(15148): Out-of-order method_idx: 0x2ae2 then 0
12-02 00:26:29.894: E/dalvikvm(15148): Trouble with item 832 @ offset 0x11e22a8
12-02 00:26:29.894: E/dalvikvm(15148): Swap of section type 2006 failed
12-02 00:26:29.894: E/dalvikvm(15148): ERROR: Byte swap + verify failed
12-02 00:26:30.117: E/dalvikvm(15148): Optimization failed
12-02 00:26:30.125: W/installd(139): DexInv: --- END '/data/app/com.mypackage.myapp-2.apk' --- status=0xff00, process failed
12-02 00:26:30.125: E/installd(139): dexopt failed on '/data/dalvik-cache/data@app@com.mypackage.myapp-2.apk@classes.dex' res = 65280

Also, regarding priority. I believe it should be higher than Medium, as this issue is a clear shipping blocker. Thanks.

ah, i hadn't noticed there were two problems here.

problem two, then, is this:

12-02 00:26:29.878: E/dalvikvm(15148): Out-of-order method_idx: 0x2ae2 then 0

or in the original example:

11-18 20:11:08.577: E/dalvikvm(868): Out-of-order method_idx: 0x2ae0 then 0x1

there are two places in libdex that can output that error: swapMethodAnnotations and swapParameterAnnotations. so it looks like dx is outputting method and/or parameter annotations unsorted. http://source.android.com/tech/dalvik/dex-format.html says they must be sorted.

i think that DexMerger.transformAnnotationDirectory is wrong, because it potentially rewrites every field/method index but doesn't resort the field/method/parameter annotation lists.

i'm not sure if there's a better way to fix this than changing transformAnnotationDirectory to construct temporary field/method/parameter collections, sort them, and then write out the sorted collections?

i'm also not sure how to change DexMergeTest to ensure that some renumbering occurs.

in the meantime, your workaround -- if you don't need them -- is to strip annotations. dexdump and the method indexes in question should point you in the direction of the ones that are causing you trouble.

Thanks for the analysis enh.

While the Dalvik team works on a fix, I'm going to allow disabling of the new dex merger features. Builds will be slower (actually back like they were before) but they'll work at least. We were looking at a 21.0.1 release so I'm going to do this now and make sure this gets in.

[Comment deleted]

seems unlikely to be @Override --- those annotations only have a SOURCE RetentionPolicy, so they should be discarded by the compiler.

[Comment deleted]

OK, I might be wrong, what's sure is that SPEED generated protocol buffers java files are the cause of this problem. If interested I can post the differences between both kind of java files, if that can help debug the issue a bit more.

Note that it would be really good to have more specific error messages like for instance specifying what class triggered the error at first if that's ever possible.

Any news regarding this bug?
This is really a show stopper right now for me.

For me as well.

Finally, I did a lots of test, running dx on the command-line including some or all or no dexedLibs or classes from my projects. I also tried to order differently the input classes/libs at no avail.

Ultimately, I found that it doesn't really depends on the classes that are dexed, but more on the numbers of dexed classes (or maybe their complexities?).

So, I'm now doubting this is related to annotations.

What is sure is that a broken classes.dex can't be read on the platform or by dexdump on the host.

What is sure is that I get a various range of differents dalvikvm errors:
* either the "Out o order method_idx" one as described in this bug
* or "Invalid class_data_item"
* or I get a valid classes.dex (valid in the sense dexopt is able to digest it), but when "executed" on a device, when trying to load some classes the device dalvik errors with 'this' arg class 'some random class of the project' is not an instance of 'Landroid/bitmap/Bitmap' (sorry the error message might be incomplete, reproducing it from memory).

All those happens with either jumbo mode or not, optimize or not. This was dexed with dx version 1.7 from sdk r21.

One thing I couldn't try is to use an older dx from an older sdk (is there and archive somewhere?)

I'm currently trying to rebuild dx to debug the issue, so wish me luck.

So, I dug a lot today to try to fix the problem.

First thing, I really think it is not related to annotations. I stripped all the dx java code related to annotations and now the problem always appear with a "Invalid class_data_item".

I also resurrected various platform tools (v12, v10) just to see, and always get either an "Invalid class_data_item" error or a failure when generating the dex (because a short is too limited to write a large number).

Here's the error while dexdump'ing our dex generated by the r12 (which I believe didn't support annotations at that time):
E/dalvikvm(15810): Invalid class_data_item
E/dalvikvm(15810): Trouble with item 3770 @ offset 0x137764
E/dalvikvm(15810): Cross-item verify of section type 0006 failed
E/dalvikvm(15810): ERROR: Byte swap + verify failed

I'm open to send privately my generated classes.dex if that can help the dalvik team to understand the issue.

@18: i think you're confused. there are probably multiple bugs. as i explained in comment 7, the original error is caused by out-of-order annotations. the number of annotations isn't relevant to that bug; all that matters is whether the method indexes they refer to happen to come out in order or not when the merge basically appends them without sorting. (you are though correct that the annotations problem has nothing to do with jumbo mode --- it's the merge step that's the problem.)

looking at libdex, the reason for your new error is a bit more obscure. it seems to mean that there are references to more than one class in a class_data_item. my assumption is that that means that one of the fields or methods referred to in the class_data_item doesn't belong to the class we're supposedly defining. i'm afraid i don't know enough about the merge process to know how/why that might happen.

@19,

Well I won't pretend I understand how dx works and what it does.

I really think it is a size related issue. I'm dealing with about 15 jars, some very big or containing large classes (especially the protobuf generated ones).

I tried to individually dex the jars. then merge them two by two, and I could dexdump the resulting dex everytime.

Then I tried merging the dexes one by one, checking with dexdump the resulting dex. It works fine until I reach the point where I get any of the aforementioned errors.
I tried reversing the order of the jars on the dx command-line (I don't know if that matters) but I still get non-working dex until I remove some jars.

So, it seems I'm reaching a kind of critical mass. Maybe I have too many classes/methods/whatever and dx overflows some offsets and thus generate an invalid dex?

@20: i've yet to see any evidence that that's the case.

Hello, I'm the one who opened this bug report. It's been open for several weeks now.
Could anyone post an official status report on the path to solve this issue?
Thx.

@21,

The only evidence I have is that the problem would appear as soon as I'd merge two dexes if it was only related to a distinct set of classes/annotations that doesn't mix together.

Since it does only happen when a certain number of items are merged (I couldn't really identify what this number is precisely), then that reinforces my feeling that the problem happens only on large projects involving more than 10 merges.

I'd love to have evidence of my assumption, because that'd mean I'd understand the issue :) and would be able to fix it :)

What could the OP and I do to help resolve the issue?
What kind of debug information or dump can we provide (and how can we get it)?

If I'm able to rebuild dexdump (it doesn't look that easy apparently), I'll try to debug it reading my large classes.dex to see how/why it chokes on it.

Hi,

I'm posting a follow-up after a day tracing dexdump and dx.
So, I found that the move_idx error is caused because when generating the dex file, we overflow the method_id used in the method part of the annotation directory (which dx outputs like a short while in the dex spec it's an uint).
Once this problem was fixed, I stumbled upon a different problem with an invalid class_data_item, which resolved to a different instance of the same issue.

Finally, with the attached dx patch, we were almost able to generate our large classed.dex.

The last error we finally now get is the famous one the OP got first:
com.android.dx.util.DexException: Cannot merge new index 65627 into a non-jumbo instruction!
        at com.android.dx.merge.InstructionTransformer.jumboCheck(InstructionTransformer.java:108)
        at com.android.dx.merge.InstructionTransformer.access$4(InstructionTransformer.java:106)
        at com.android.dx.merge.InstructionTransformer$MethodVisitor.visit(InstructionTransformer.java:101)
        at com.android.dx.io.CodeReader.callVisit(CodeReader.java:114)
        at com.android.dx.io.CodeReader.visitAll(CodeReader.java:89)
        at com.android.dx.merge.InstructionTransformer.transform(InstructionTransformer.java:48)
        at com.android.dx.merge.DexMerger.transformCode(DexMerger.java:809)
        at com.android.dx.merge.DexMerger.transformMethods(DexMerger.java:780)
        at com.android.dx.merge.DexMerger.transformClassData(DexMerger.java:753)
        at com.android.dx.merge.DexMerger.transformClassDef(DexMerger.java:654)
        at com.android.dx.merge.DexMerger.mergeClassDefs(DexMerger.java:526)
        at com.android.dx.merge.DexMerger.mergeDexBuffers(DexMerger.java:168)
        at com.android.dx.merge.DexMerger.merge(DexMerger.java:186)
        at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:300)
        at com.android.dx.command.dexer.Main.run(Main.java:232)
        at com.android.dx.command.dexer.Main.main(Main.java:174)
        at com.android.dx.command.Main.main(Main.java:91)

So, I double-checked I was using dex.force.jumbo on my eclipse project and indeed I am. I cleaned everything, and ran dx on the command-line on all the dependencies and classes folders of my hierarchy with "--force-jumbo".
Surely that should force jumbo, right?

Well, apparently no, because I still get the aforementioned errors.

So the question is: is there really a 65535 limit of methods in a given dex file?
Reading the .dex specs, I couldn't find any evidence of this...

Now, I guess the only solution we have is to strip all the deadcode (ie proguard) during development runs...

So, to summarize, even though the .dex specs allows more than 65535 classes and methods, the vm doesn't support large number of classes/methods, there wasn't any explicit error message.

Regarding the linked proposed commit, that'd be great to have a more explicit error message (one that would indeed make the user understand what the root cause is and what work-arounds exist).

Brice, you got it. Merging pushed your .dex file over the 65,535 method limit, but it didn't detect this error and thus produced a bogus file. You need to shrink your application, either with proguard, or by pruning the parts you aren't using.

do we still need to change transformAnnotationDirectory to construct temporary field/method/parameter collections, sort them, and then write out the sorted collections? (as i guessed in comment 7.) or is there some other reason why those lists would always remain sorted in a merged dex file?

I know I must have more than 65535 classes/methods. See the original message on this defect.
But, isn't dex.force.jumbo=true there to allow more than 65535 methods?

no. there are no jumbo opcodes for method invocation.

Ufnortunately method call is encoded with a method_id being a short int (so only 65535 different methods), unlike const string access.

We need an invoke virtual jumbo  :)  

As a *workaround* then, what would be the easiest way to figure out what unused classes I can delete of the 3rd party libraries I use? so that my the # of methods in my dex file will drop under 65K?
For example: Would it be possible to run the app with the android equivalent of verbose class loading in order to tell what classes are used and which are not?
Or is this something that proguard can tell?
Or is there some other way that you would recommend?

What's the intended usage for dex.force.jumbo=true?

Given the above declared limit, do you think this defect should be change to an enhancement request?

Proguard can tell you what your project is using (check the usage.txt output file), provided proguard is correctly configured.

To my knowledge, dex.force.jumbo activates the const-string-jumbo opcode which allows to refer to static strings when you have more than 65k strings in your dex file.

usually, dx will use the shortest instruction it can. this can make merging impossible if an instruction would need to be widened to fit more bits of string index (or whatever). dex.force.jumbo says "always use the wide form, even if you don't need to", to improve the chances of being able to merge later.

I am also using a lot of 3rd party libraries (.jar), so removing unused code (identified via proguard) is not possible.
Is there a possible in this case ?

Is there an estimate on when this defect will be fixed ?

I have got the same error when using emma to collect code coverage.

For my project, ant clean release / ant clean debug is ok. "ant clean instrument" results in a "too many methods" error from dx in android tools r20.

After upgrading to r21 and using dex.force.jumbo=true, I am able to build the apk, which however cannot be installed. The error is also "E/dalvikvm(  192): Invalid class_data_item".

I think using "ant instrument" on some large project could be a good way to reproduce the bug.

I got the same error, is there a solution now?

for debug or instrument build, you can modify the build.xml file to enable proguard, which can remove useless methods before dex.

D/dalvikvm( 132): DEX prep '/system/framework/ext.jar': unzip in 42ms, rewrite 625ms
D/dalvikvm( 132): DexOpt: --- BEGIN 'framework.jar' (bootstrap=1) ---
I/ServiceManager( 131): Waiting for service broadcom.tvservice...
E/dalvikvm( 366): Out-of-order entry types: 0x12fb then 0x12fb
E/dalvikvm( 366): Trouble with item 611 @ offset 0x1593a8
E/dalvikvm( 366): Cross-item verify of section type 1003 failed
E/dalvikvm( 366): ERROR: Byte swap + verify failed
I/ServiceManager( 288): Waiting for service media.audio_policy...
E/dalvikvm( 366): Optimization failed
W/dalvikvm( 132): DexOpt: --- END 'framework.jar' --- status=0xff00, process failed
E/dalvikvm( 132): Unable to extract+optimize DEX from '/system/framework/framework.jar'
D/dalvikvm( 132): Unable to process classpath element '/system/framework/framework.jar'
I/ServiceManager( 134): Waiting for service broadcom.tvservice...
D/dalvikvm( 132): DexOpt: --- BEGIN 'framework2.jar' (bootstrap=1) ---
E/dalvikvm( 367): outsSize (6) > registersSize (4)
E/dalvikvm( 367): Trouble with item 3051 @ offset 0x168d10
E/dalvikvm( 367): Swap of section type 2001 failed
E/dalvikvm( 367): ERROR: Byte swap + verify failed
E/dalvikvm( 367): Optimization failed
W/dalvikvm( 132): DexOpt: --- END 'framework2.jar' --- status=0xff00, process failed
E/dalvikvm( 132): Unable to extract+optimize DEX from '/system/framework/framework2.jar'
D/dalvikvm( 132): Unable to process classpath element '/system/framework/framework2.jar'
Here is the same error i"m facing! Any one solved?? Is it because of Big size of framework.jar??

I found DEX failed when bytecode file total size is larger than 11542831 bytes ,is there a solution?
Who can help me?

Thank you for submitting this issue. It was last modified before Android Studio 1.0 Launched in 2014. We have made a large number of improvements in Android Studio since 2014 and hope this issue is already resolved. In order to correctly prioritize issues in our upcoming release, we are closing this bug. If you continue to experience your issue on the latest stable version of Android Studio, please submit a new bug report. We apologize for any inconvenience this may cause you.

If you wait long enough, all bugs become obsolete. Why bother fixing any of them?