| Issue 2025: | replication fails with org.eclipse.jgit.errors.MissingObjectException: Missing unknown <SHA> | |
| 32 people starred this issue and may be notified of changes. | Back to list |
Affected Version: 2.6.1 What steps will reproduce the problem? Maybe the following: 1. Set up Gerrit to replicate a repository, where the destination has branches with unmerged commits which are not known to Gerrit What is the expected output? What do you see instead? Replication should work. Instead I get exceptions like the following: [2013-07-22 07:20:24,177] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to git@github.com:beaker-project/beaker.git org.eclipse.jgit.errors.TransportException: git@github.com:beaker-project/beaker.git: Missing unknown c15bf378569d64ce8677e417e14504f5fce1e061 at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:217) at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152) at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165) at org.eclipse.jgit.transport.Transport.push(Transport.java:1167) at org.eclipse.jgit.transport.Transport.push(Transport.java:1213) at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:307) at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:252) at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:207) at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:71) at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:186) at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:183) at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222) at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201) at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75) at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:183) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown c15bf378569d64ce8677e417e14504f5fce1e061 at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148) at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229) at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809) at org.eclipse.jgit.internal.storage.pack.PackWriterBitmapWalker.findObjects(PackWriterBitmapWalker.java:95) at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPackUsingBitmaps(PackWriter.java:1847) at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1665) at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:797) at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:760) at org.eclipse.jgit.transport.BasePackPushConnection.writePack(BasePackPushConnection.java:300) at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:197) ... 23 more Please provide any additional information below. I have set up Gerrit to replicate this repo to GitHub. There were a bunch of old dead branches on GitHub which were not present in Gerrit. Some of those branches had commits which were not merged, so the commits were totally unknown to Gerrit. The commit from the exception, c15bf378569d64ce8677e417e14504f5fce1e061, was one such commit. I deleted all the unwanted branches from GitHub (they shouldn't have been there) so I am now getting a 404 for that commit on GitHub as expected, but the exceptions still happen in Gerrit. I guess GitHub is continuing to reference that commit as part of the receive-pack process. Gerrit replication shouldn't barf just because the remote contains objects which are not known to Gerrit.
Jul 22, 2013
#1
icee...@googlemail.com
Jul 22, 2013
To be more specific: It was a tag that was deleted in gerrit again, but survived in the mirror repo.
Jul 22, 2013
Same for us here: branch with commit was on remote, but not in Gerrit.
Jul 24, 2013
I meet same error message when replicate repository to mirror after I run gerrit gc command on master server.
Jul 24, 2013
We did a gerrit gc command before this replication error message occurred, too!
Jul 29, 2013
Same issue here. We did the git gc command recently too. Analyzing the problem deeper..
Aug 8, 2013
Same issue. Here is a workaround: [as the gerrit user] cd base/git/<repo>.git git remote add github <github url> git fetch github
Sep 6, 2013
Getting the same error after upgrading to 2.6.1. I thought it was the upgrade, but upon further inspection it would seem that only the repos that gerrit gc were run on are affected.
Sep 16, 2013
I witness the same problem and we suspect the unwanted change is erased when `git gc` runs, but still sits at the destination for mirror. I would said if we don't want this change, running `git gc` on Github will be fine.
Sep 16, 2013
Just to add more info: I did run `git gc` on the destination once I narrowed it down to the repos that I had run `gerrit gc` on. Unfortunately it did not resolve the issue - same errors filling up the log and the downstream repo not getting updated
Sep 16, 2013
OK, I guess that doesn't work. :-( I wish there's a better fix for this. We have scheduled `git gc` ran on gerrit and this will happen again the next time we run it.
Oct 10, 2013
We faced a similar issue due to some incomptability between the jgit generated bitmaps with those generated with git gc. There was a fix in JGit 3 to ignore such error, but still Gerrit 2.6.1 uses jgit 2.3. The workaround we applied was to get a backup repo which didn't have gerrit gc run on and fetch from the latest repository which had gerrit gc run on. After applying this workaround we stopped running git gc and we run now gerrit gc instead ..
Oct 21, 2013
After a little debug, I found that when JGit uses bitmaps (which is the case after a gc), the push command doesn't go through the code where MissingObjectExceptions are caught for remote objects not found locally. I could fix this by removing earlier non-locally-found remote objects, see attached patch.
Oct 24, 2013
The patch in #13 fixes the problem. I've prepared a "hot-patch" for Gerrit 2.6 and Gerrit 2.7 with it. Add the following JAVA_OPTIONS to /etc/default/gerritcodereview and restart Gerrit: export JAVA_OPTIONS=-javaagent:<path to gerrit-jgit-patch-1.0-SNAPSHOT.jar>
Oct 25, 2013
I guess our issue is different then as I've applied your "hot-patch" on 2.6.1 with the JAVA_OPTIONS set and I still have the same errors for replication
Oct 25, 2013
My apologies, I forgot to point out that /etc/default/gerritcodereview is only applied if you start Gerrit with bin/gerrit.sh. If you start Gerrit through Jetty or Tomcat, you have to add the JAVA_OPTIONS to their corresponding configuration files instead. Nevertheless, before spending more time figuring out, which config file to change, you can always just test the patch like by starting Gerrit like this: $ java -javaagent:/tmp/gerrit-jgit-patch-1.0-SNAPSHOT.jar -jar gerrit2/bin/gerrit.war daemon -d gerrit2 It should immediately print "PatchingAgent initialized!" Once you run trigger replication it should print: "org/eclipse/jgit/transport/BasePackPushConnection patched!" If it still doesn't work for you than its definitely a different problem.
Nov 4, 2013
Issue 2236 has been merged into this issue.
Dec 4, 2013
A new flavor of this occurred for me last night (that the above mentioned patch didn't fix: [2013-12-04 08:35:29,644] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to gerrit@server:repo.git org.eclipse.jgit.errors.TransportException: gerrit@server:repo.git: Missing tree 39276ded80c4686a67aaeefdda216d60eb2ce1fc at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:217) at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152) at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165) at org.eclipse.jgit.transport.Transport.push(Transport.java:1167) at org.eclipse.jgit.transport.Transport.push(Transport.java:1213) at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:361) at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:335) at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:262) at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:78) at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:235) at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:232) at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222) at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201) at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75) at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 39276ded80c4686a67aaeefdda216d60eb2ce1fc at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:149) at org.eclipse.jgit.revwalk.ObjectWalk.newTreeVisit(ObjectWalk.java:728) at org.eclipse.jgit.revwalk.ObjectWalk.nextObject(ObjectWalk.java:379) at org.eclipse.jgit.internal.storage.pack.PackWriterBitmapWalker.findObjects(PackWriterBitmapWalker.java:110) at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPackUsingBitmaps(PackWriter.java:1847) at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1665) at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:797) at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:760) at org.eclipse.jgit.transport.BasePackPushConnection.writePack(BasePackPushConnection.java:303) at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:197) ... 23 more Looking into the cause, I found the change ID that was associated with the tree was no longer in the database again, leading me to suspect a return of this bug... though when I've tried to reproduce it, I've been unable so far, and it's only happening on one repo now.
Jan 16, 2014
We have this issue reproduced with Gerrit 2.8 (more specifically 2.8-29-gd367fe1). This version have JGit library version 3.1.0.201310021548-r, and still the issue is reproducible spoardically.
Jan 17, 2014
The patch seems to work also in 2.8.1. Is the change to Jgit a workaround for a problem in Gerrit or is this a bug in Jgit? In the later case an issue should be opened at https://bugs.eclipse.org and the patch should be uploaded to https://git.eclipse.org/r.
Jan 18, 2014
It seems to me this is indeed a JGit bug. I filed it under https://bugs.eclipse.org/bugs/show_bug.cgi?id=426044
Jan 26, 2014
Seeing this in 2.8.1 release. Could do a fix. Tried the fetch trick, that didn't work. Thankfully we only replicate out to Stash. So I could drop/remake the effected repo - but that's not really a fix.
Jan 26, 2014
Yes, to patch it, you do need to have Gerrit built with a custom JGit -- the issue will come back any time you delete a draft patch or changeset. Alternatively, don't garbage collect any repos for the time being.
Mar 28, 2014
Don't use the "hot-patch" from #14 with Gerrit later than 2.8.1. org/eclipse/jgit/transport/BasePackPushConnection.java has changed in jgit-3.2.0.201312181205 which is used in Gerrit 2.8.2 and 2.8.3.
Apr 2, 2014
We get this error on both replication and pushing code to Gerrit Projects.
May 1, 2014
#14 - any chance you could publish the source to your patch so it can be updated
May 1, 2014
Thanks for reminding me... We upgraded to Gerrit 2.8.4 at work a couple of weeks ago. At that time i rebased my patch as well to match jgit in Gerrit 2.8.4. I'll upload it tomorrow (CET) along with the simple maven project, which is quite useful for cleanly patching any third party libs included in webapps like Gerrit.
May 1, 2014
Thanks! Good news, though, JGit 3.4 (currently under development) has the change applied and the upstream bug has been resolved, so hopefully we can resolve this officially in the near future.
May 2, 2014
Hi guys! Here is the patch for jgit 3.2.0.201312181205-r (in gerrit 2.8.2 - 2.8.4). You can use the attached maven project to port the patch to another jgit (gerrit) revision. Here is how you proceed: - Download the proper jgit source (look in your gerrit.war) - Apply the patch: https://git.eclipse.org/r/#/c/21094/ - Build jgit. - Replace src/main/resources/patch/BasePackPushConnection.class in the attached gerrit-jgit-patch project with your own patched/built version. - Update the revision in gerrit-jgit-patch/pom.xml. - Rebuild the patch: $ mvn clean package to build Good luck and let me know if something not working as it should. P.S. sorry for all different mail addresses, I've been using different devices, and forget to check, who is currently logged onto google :). Anyhow, #14, #16, #27, and #29 its the same person...
May 9, 2014
Great! Is it too risky to backport to stable-2.8 as well?
May 9, 2014
Probably not very risky. The question is rather if there will be another 2.8 release.
Status:
Submitted
Labels: FixedIn-2.9
Jul 18, 2014
(No comment was entered for this change.)
Status:
Released
|
|
| ► Sign in to add a comment |