My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 2131: Running gerrit gc - garbage collection for project "<project_name>" failed
6 people starred this issue and may be notified of changes. Back to list
Status:  Submitted
Owner:  ----
Closed:  Sep 2013


Sign in to add a comment
 
Reported by Vladimir...@gmail.com, Sep 18, 2013
Affected Version: 2.6.1

What steps will reproduce the problem?
1. Create a new permissions-only project or a new simple project with no master branch (no empty commit)
2. Add at least one Access control option
3. Run gerrit gc on new project
4. Run git gc on new project
5. Run gerrit gc on new project

What is the expected output? What do you see instead?

Expected a successful garbage collection and a bitmap on step 5, however second gerrit gc command fails with the following output:
 ssh -p 29418 <gerrit_url> gerrit gc <project0name>
collecting garbage for "<project0name>":
Pack refs:              100% (1/1)
Counting objects:       8
Finding sources:        100% (8/8)
Getting sizes:          100% (4/4)
Writing objects:        100% (8/8)
failed.

error: garbage collection for project "<project_name>" failed



Please provide any additional information below.

- New pushes to these repos have replication processes stalled and the change doesn't reach mirrors (common scenario for permissions-only parent projects).
- In some cases users lose access to their repos inherited from parent project Access Controls.

- git gc or fsck doesn't report any issue. 
- git clone works well
- jgit clone fails with the following output:

Initialized empty Git repository in /apps/gerrit/tools/error/new-all/.git
fatal: Missing unknown <SHA-ID>


Workaround to fix the issue:

1. git clone --bare "</path/project_name>"
2. Replace the original repo with the new bare copy
3. Run gerrit gc - command runs successfully


Sep 19, 2013
#1 Vladimir...@gmail.com
Described workaround is not a reliable one, refs/meta/config is lost using described steps.
Sep 19, 2013
Project Member #2 cran...@google.com
Just before (or after) you do the git gc , delete the *.bitmap files and everything will work.

It sounds like git created a pack with the same file name but a different pack checksum, so the .bitmap cannot be used. The "fix" here is probably to just log the bitmap as being invalid and forgetting about it.
Sep 19, 2013
Project Member #3 cran...@google.com
I've added the fix to jgit: https://git.eclipse.org/r/#/c/16618/
Status: Submitted
Sep 20, 2013
Project Member #4 bassem.rabil
Deleting the .bitmap files does not fix the gerrit gc errors using jgit. I see Gerrit 2.6 is using Jgit 2.3, will using Jgit 3.1 with Gerrit 2.6 introduce incompatibility issues ?
Sep 20, 2013
Project Member #5 bassem.rabil
Here below the exception we got after removing the bitmap file, putting an empty file would result is a similar exception.

Caused by: java.io.FileNotFoundException: <Path to .bitmap file> (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.eclipse.jgit.internal.storage.file.PackBitmapIndex.open(PackBitmapIndex.java:96)
        at org.eclipse.jgit.internal.storage.file.PackFile.getBitmapIndex(PackFile.java:1063)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.getBitmapIndex(WindowCursor.java:110)
        at org.eclipse.jgit.internal.storage.pack.PackWriterBitmapPreparer.<init>(PackWriterBitmapPreparer.java:107)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.prepareBitmapIndex(PackWriter.java:2031)

Oct 4, 2013
#6 casta...@motorola.com
So if we never do a "git gc" once we've started using "gerrit gc" then would this issue ever occur?

And why would I do "gerrit gc" -> "git gc" -> "gerrit gc"?  What's the point of doing that?  Not criticizing here... I actually am curious what that does.
Oct 15, 2013
#7 Vladimir...@gmail.com
We don't run this on regular basis however we had some issues with running git gc in the past even with nice and ionice options. Application used to sporadically fail or even the entire server crushed.
We finally managed to have control more or less with git gc (gits are on NFS and we used to run git gc on a separate machine). With 2.6 we had valid concerns switching to gerrit gc right away.
Basically we had a few days of running one and then another gc, while running gerrit gc manually and watching closely the system behavior. This created this unfortunate scenario. We run gerrit gc only now.

BTW: We ran git gc again at least on one more occasion as result of another jgit bug:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=394078

Basically we had to add a new bigFileThreshold value for a larger repo:
bigFileThreshold = 20M
And run git gc --aggressive which fixed the issue similar to what is described in above link.


 

Sign in to add a comment

Powered by Google Project Hosting