My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 1582: Issue with git push-ing to Gerrit
57 people starred this issue and may be notified of changes. Back to list
Status:  Submitted
Owner:  ----
Closed:  Dec 2014


Sign in to add a comment
 
Reported by YuriyPad...@gmail.com, Sep 26, 2012
************************************************************
***** NOTE: THIS BUG TRACKER IS FOR GERRIT CODE REVIEW *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, INTERNAL *****
***** ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.    *****
***** THOSE ISSUE BELONG IN DIFFERENT ISSUE TRACKERS!  *****
************************************************************

Affected Version: 2.4.2

What steps will reproduce the problem?
1. git push
2.
3.

What is the expected output? What do you see instead?

We expect successfull push, instead we get:
o$ git push
Counting objects: 5389, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1801/1801), done.
Writing objects: 100% (5190/5190), 2.27 MiB | 15 KiB/s, done.
Total 5190 (delta 3401), reused 5006 (delta 3275)
remote: Resolving deltas: 100% (3401/3401)
error: unpack failed: error Missing tree bfe8d24e0232d8e8969fdadddea7b81aca863664
fatal: Unpack error, check server log
To ssh://**********:29418/********.git
 ! [remote rejected] HEAD -> refs/for/cmedia_3.0.9 (n/a (unpacker error))
error: failed to push some refs to 'ssh://************8:29418/ticketco.git'

Please provide any additional information below.

Mar 20, 2013
#2 aleksey....@gmail.com
Hi guys, we have the same issue with 2.5.1.

Direct push or push to review or draft fails sometimes for some developers with error:

$ git review master
Counting objects: 152, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66/66), done.
Writing objects: 100% (90/90), 15.94 KiB, done.
Total 90 (delta 33), reused 30 (delta 13)
remote: Resolving deltas: 100% (33/33)
error: unpack failed: error Missing tree 42d769150decc8ee96bd9eb7afacc01fe14ffc55
fatal: Unpack error, check server log
To ssh://....
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://....'

In logs/error_log

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 42d769150decc8ee96bd9eb7afacc01fe14ffc55
        at org.eclipse.jgit.transport.BaseReceivePack.checkConnectivity(BaseReceivePack.java:996)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:756)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:167)
        ... 15 more

Missing tree sha was different, but every time git ls-tree showed not-corrupted existed tree.
git fsck showed danglings only.

After investigation I found if I set Read right on /ref/* for Registered Users, issue would disappear.

What is the link between Read right on /ref/* for Registered Users on time-to-time missing tree exception? 




Apr 17, 2013
#3 presich....@gmail.com
Hello guys,
we are exiriencing the same issues with 2.5.1 and 2.5.2.

Aleksey's workaround hasn't helped.

Any suggestions on cause of this issue and possible workarounds?
Apr 18, 2013
#4 presich....@gmail.com
Hello again,
we started to receive such errors very often.

Push fails with such errors (even push force!! through the Gerrit):
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree
or Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob

Any thoughts what could we check?

Those missing trees and blobs are really exist in both local and gerrit repository and i can look in them by: git show

I mentioned that one of missing trees was in Packed objects, but can't say for all of them.

Also I can suggest that such errors come up with commits, that was not uploaded to the Gerrit review previously.

The only way we were able to push changes, is to push them through the pure Git on port 22.




Nov 20, 2013
#5 KNibb...@gmail.com
Good day. I've tried to configure local Gerrit for evaluation purposes and got the same problem. Seems that there is some problem in git protocol implementation.
I've checked on git 1.8.3.2 and everything worked fine. When I've updated to 1.8.4.3 problems on commit changes were similar to the one in this topic:

k.stolyarov@DGIS-184 ~/test $ git clone ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
Cloning into 'v2'...
The authenticity of host '[dgis-184.dvlp.2gis.local]:29418 ([10.54.200.106]:29418)' can't be established.
RSA key fingerprint is c7:4d:d5:23:d1:b3:02:24:c6:21:2c:1d:3e:4e:de:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[dgis-184.dvlp.2gis.local]:29418,[10.54.200.106]:29418' (RSA) to the list of known hosts.
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 2 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (2/2), done.
skjdhjksdf
Checking connectivity... done
k.stolyarov@DGIS-184 ~/test $ cd v2
k.stolyarov@DGIS-184 ~/test/v2 $ vim 1 
k.stolyarov@DGIS-184 ~/test/v2 $ scp -p -P 29418 a.baskanov@dgis-184.dvlp.2gis.local:hooks/commit-msg .git/hooks/
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
commit-msg                                                                                                                                                 100% 4367     4.3KB/s   4.3KB/s   00:00    
k.stolyarov@DGIS-184 ~/test/v2 $ git add 1
k.stolyarov@DGIS-184 ~/test/v2 $ git commit
[master dd26813] sdfhsdf
 1 file changed, 1 insertion(+)
 create mode 100644 1
k.stolyarov@DGIS-184 ~/test/v2 $ git push origin HEAD:refs/for/master
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
Counting objects: 4, done.
Writing objects: 100% (3/3), 269 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   http://dgis-184.dvlp.2gis.local:8080/1
remote: 
To ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
 * [new branch]      HEAD -> refs/for/master
k.stolyarov@DGIS-184 ~/test/v2 $ vim  2
k.stolyarov@DGIS-184 ~/test/v2 $ git add 2
k.stolyarov@DGIS-184 ~/test/v2 $ git commit -a  --amend
[master 6026dbb] sdfhsdf
 2 files changed, 2 insertions(+)
 create mode 100644 1
 create mode 100644 2
k.stolyarov@DGIS-184 ~/test/v2 $ git push origin HEAD:refs/for/master
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 302 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
error: unpack failed: error Missing blob e89ef0eb5feb4db1e4c757374b83ed1d8e6081ca
fatal: Unpack error, check server log
To ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2'

I've checked this on both Gerrit 2.7 and 2.8-rc2 versions from two different linux machines (Gentoo and Ubunto builds). Latest git version bakes review updates.

Hope that this information will help you to fix the issue.
Nov 21, 2013
#6 joseph.r...@gmail.com
This is much more than a minor issue as it is leading us to have corrupted repos at this time. Can someone please take a deeper look into why this might be happening?
Jan 29, 2014
#7 sschuberth
 Issue 2296 , which seems to duplicate this issue, suggests to set receive.checkReferencedObjectsAreReachable = false (available since Gerrit 2.6) to work around this.
Jan 29, 2014
#8 joseph.r...@gmail.com
Why in the world would one turn that off - it sounds like the suggestion is to allow what looks like corruption in the reference tree...?
Jan 29, 2014
#9 sschuberth
Just consult the Gerrit 2.6 release notes [1] for why it would make sense to turn off that option:

"New config option receive.checkReferencedObjectsAreReachable

If set to true, Gerrit will validate that all referenced objects that are not included in the received pack are reachable by the user.

Carrying out this check on Git repositories with many refs and commits can be a very CPU-heavy operation. For non public Gerrit servers it may make sense to disable this check, which is now possible."

[1] http://gerrit-documentation.googlecode.com/svn-history/r63/ReleaseNotes/ReleaseNotes-2.6.html
Jan 29, 2014
#10 sschuberth
Also, I don't believe turning off that option allows for corruption in the reference tree. It's just that Gerrit's JGit implementation cannot handle [1] yet (see  issue 2296 ), which is why you get this bogus error with recent Git clients.

[1] https://github.com/git/git/commit/fbd4a7036dfa71ec89e7c441cef1ac9aaa59a315
Apr 23, 2014
#11 onlyn...@gmail.com
Is there any work around for this from the client side? If say I can't upgrade the version of gerrit being used, or change server settings? Maybe there's an option to git I could set to not do the optimization that sends fewer trees?
Jun 10, 2014
#12 mpe...@gmail.com
Per a coworker, sounds like specifying --no-thin on the git client command line avoids this optimization without having to set receive.checkReferencedObjectsAreReachable = false on Gerrit.
Sep 5, 2014
Project Member #14 jrn@google.com
 Issue 2296  has been merged into this issue.
Sep 9, 2014
Project Member #15 david.pu...@sonymobile.com
https://gerrit-review.googlesource.com/#/c/59935/
Status: ChangeUnderReview
Oct 8, 2014
#16 mdemp...@gmail.com
I'm using gerrit 2.9.1, which already has jgit update (change 59935), but I still can reproduce the missing tree error. It seems to happen only for pushes where *only commit message is changed* without any other commit modifications o.O

$ echo 'a' > a
$ git commit -a -m 'msg'
$ git push origin HEAD:refs/for/master
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: New Changes:
remote:   https://gerrit/2202

$ echo 'b' > a
$ git commit -a --amend
/ possibly some changes in msg in editor/
$ git push origin HEAD:refs/for/master
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: updated: 1, refs: 1, done    
remote: Updated Changes:
remote:   https://gerrit.sio2project.mimuw.edu.pl/2202

$ git commit -a --amend
/ changes in msg/
$ git push origin HEAD:refs/for/master 
error: unpack failed: error Missing tree a9596a3d9e2f4fa81912124d3dacd58113f11e85
fatal: Unpack error, check server log
To ssh://gerrit/project
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://gerrit/project'

This is not too common to change only commit msg, also it can be done from gerrit web ui, but still is annoying.

Can someone try to reproduce and confirm it?
Thanks!
Oct 8, 2014
Project Member #17 nas...@codeaurora.org
How did you determine that 2.9.1 has 59935? 59935 was uploaded and merged to master. The only JGit update I see on stable-2.9 is from May: https://gerrit-review.googlesource.com/#/q/status:merged+project:gerrit+branch:stable-2.9+message:%22JGit%22
Oct 8, 2014
#18 mdemp...@gmail.com
Oops, right, I thought I've seen it in some changelog, but apparently it's not there.
Sorry!
Nov 12, 2014
#19 silvio.h...@gmail.com
Is there gonna be a new release soon? We're plagued by this bug in various teams.
Or is installing 2.10.rc0 a good alternative to waiting for the release?
Nov 14, 2014
#20 jlst...@gmail.com
I've been hitting this issue for quite a while and it still exists in 2.9.1. A simple case to duplicate is to push a change, merge, and try to push the git reverted change. I've been able to reproduce this in more complicated cases where the commit tree ends up being the same as one that was pushed prior.
Dec 2, 2014
Project Member #21 bassem.rabil
If you are using git 1.8.4.2 or later, there is some incompatibility between jgit 2.3 shipped with Gerrit 2.7 and jgit 3.1 shipped with Gerrit 2.8 and this version of git due to this enhancement, i.e. https://github.com/git/git/commit/fbd4a7036dfa71ec89e7c441cef1ac9aaa59a315

With this enhancement, if git finds out that the tree sha1 already exists in the server, it wont send it again, but gerrit wants to search the tree sha1 associated with the commit sha1 in the uploaded pack. There is a bug report on jgit library for this issue at: https://bugs.eclipse.org/bugs/show_bug.cgi?id=426044. This was reported to be fixed for jgit 3.4 which is shipped with Gerrit 2.9, but there is still the same behavior reported with Gerrit 2.9.1.

A proposed workaround for this issue is to go to the local branch you are trying to push, and pull from origin and then retry pushing, or use older version of git to push such branches. 
Dec 2, 2014
Project Member #22 jrn@google.com
Fix submitted and should be part of 2.10-rc1.
Status: Submitted
Dec 2, 2014
Project Member #23 jrn@google.com
(The fix I'm referring to is https://gerrit-review.googlesource.com/59935.)
Feb 12, 2015
#24 jlst...@gmail.com
Can anyone verify this is fixed in 2.10?
Apr 12, 2015
#25 fraczwoj...@gmail.com
Got this error in 2.9.3. After update to 2.10 works.
Sep 2, 2015
#26 johannes...@gmail.com
I have this problem in Gerrit 2.11.3. git push --no-thin solves the problem.
Sep 2, 2015
#27 fabio.po...@gmail.com
@johannes: Wich version of the git client are you using?

On Gerrit 2.10.4 i solved this issue by using Git for Windows 2.x instead of version 1.9.5
Jan 5 (5 days ago)
#28 johannes...@gmail.com
@fabio: Sorry for late reply!
I just saw this with gerrit 2.12 from a windows client running git version 1.9.4.msysgit.2
Using --no-thin solved the problem, next time I will try to upgrade git (I am using the version supplied by git extensions which is getting old).
Sign in to add a comment

Powered by Google Project Hosting