Issue 1068: Large commit results in broken UI and unable to do anything with the review
Status:  New
Owner: ----
Reported by jhans...@myyearbook.com, Jul 27, 2011
Affected Version:

2.2.1

What steps will reproduce the problem?
1. Make a large commit (e.g., in this case it was a 150k-line diff at 5.8MB just from "git show", and 64MB from "git format-patch", which includes binaries)
2. Push for review
3. Note that the newly created review cannot be opened in the web UI
4. Note that `ssh gerrit -p 29418 gerrit approve --abandon <change>` hangs (doesn't appear to affect either db or git tree at all) and does not modify the review.

What is the expected output? What do you see instead?

Should see the review load, with (in this particular case) "533 deleted files", "527 added files", 8 modified files.

Instead, when trying to load that review, we see the black curtained "Server Unavailable" error when trying to load that review.

Please provide any additional information below.

I believe this was the cause of our database.poolLimit problem from issue #1066.

I had to manually delete the review object from gsql and delete the refs/changes/ entries for the change (followed by git prune).  After those changes, the review no longer shows up in the web UI.

Is this expected behavior?

What is the expected "largest commit" that gerrit is known to work with in byte size?  -- in diff size?


To get an idea of the contents of the change:

Number of added/deleted/modified files:
$ git whatchanged origin/universal..HEAD | egrep '^:' | cut -c 37-38 | sort | uniq -c
 527  A
 533  D
   8  M

Number of lines, words, characters in the `git show` diff output:
$ git show | wc
  154934  574887 5869190

Number of deleted and added PNG images:
$ git whatchanged origin/universal..HEAD |  grep ' D' | grep -c png
245
$ git whatchanged origin/universal..HEAD |  grep ' A' | grep -c png
202

Full patch (via "git format-patch"):
66654063 bytes

Unfortunately I can't attach the patch itself, but I can provide answers if there are any questions about the diff.
Jul 27, 2011
#1 jhans...@myyearbook.com
FYI:  I finally found the single culprit here...  The iOS "universal" library for Burstly (advertising library) is 55MB (that's 55MB out of the 58MB that the entire repository currently consists of).

When that file is present in a review (add, modify, delete, upgrade, etc...), gerrit absolutely falls on its face.  Bypassing review ("git push gerrit master") works, no problem.  But if it has to create a review from the push, it causes the rest of the server to become unresponsive.

To reproduce the problem, see the Burstly downloads page [1], and specifically the "All Networks" variation [2].

[1]: https://github.com/burstly/Burstly-iPhone/downloads
[2]: https://github.com/downloads/burstly/Burstly-iPhone/libBurstly_AllNetworks.tar.gz
Jan 6, 2012
#2 jhans...@myyearbook.com
Any suggestions?  We just had another developer get into this state today, however it seems to be handling it a bit differently now (we're still running 2.2.1).  Now the rest of the interface works fine, but it fails when trying to open the one huge review.  When trying to open the review, the ajax request to load information fails with:

> Application Error
> Server Unavailable
> 502 Proxy Error

Again we were able to bypass review by pushing directly into master, but the reviews still exist, cluttering up the interface with reviews that will never work.  I have to manually delete them from the database in order for it to work.
Mar 22, 2012
#3 alan.ol...@gmail.com
I have encountered this as well. Large prebuilt binaries tend to trigger this. Any solution yet?
Jul 10, 2012
#4 gustavo....@gmail.com
I also have the same symptoms with a 45MB glassfish zip
Aug 1, 2012
#5 copilotC...@googlemail.com
Happens to us as well, always reproducible and even with just very few files (just now < 20 at a total size 20 Megabytes) (gerrit 2.3). 


Mar 19, 2013
#6 jhans...@meetme.com
This is still a problem even in Gerrit 2.5.2!  Even pushing the file directly to the git repository ("git push origin master") causes Gerrit to hang.

The same file is the culprit (updating the Burstly iPhone library).  The file is 46MB.  Previously we were able to push the commit that contained that file directly to the git repository, and Gerrit was fine with reviews thereafter.  But now it is not even letting the file be pushed directly to the branch.

Is there any way we can get this prioritized?  Any workaround?  I guess we could push directly to the filesystem (git push gerrit@server:/mnt/gerrit/git/Project.git master)?
Mar 19, 2013
#7 jhans...@meetme.com
I was able to start a thread profiler via New Relic, and after a 5 minute sample, got this result (screenshot is also attached, since the tree structure doesn't copy/paste well):

100.0%  - java.lang.Thread.run:619
7 Collapsed methods (show)
98.0%  - java.util.concurrent.Executors$RunnableAdapter.call:441
69.0%  - com.google.gerrit.sshd.BaseCommand$TaskThunk.run:430
17 Collapsed methods (show)
69.0%  - org.eclipse.jgit.storage.pack.DeltaStream.read:213
44.0%  - org.eclipse.jgit.storage.pack.DeltaStream.seekBase:333
6 Collapsed methods (hide)
44.0%  - org.eclipse.jgit.util.IO.skipFully:329
44.0%  - org.eclipse.jgit.lib.ObjectStream$Filter.skip:199
44.0%  - java.io.BufferedInputStream.skip:349
44.0%  - java.util.zip.InflaterInputStream.skip:191
43.0%  - java.util.zip.InflaterInputStream.read:135
43.0%  - java.util.zip.Inflater.inflate:223
43.0%  - java.util.zip.Inflater.inflateBytesnative
25.0%  - org.eclipse.jgit.storage.pack.DeltaStream.seekBase:339
6 Collapsed methods (show)
24.0%  - java.util.zip.Inflater.inflateBytesnative
29.0%  - com.googlesource.gerrit.plugins.replication.PushOne.run:178
28 Collapsed methods (show)
27.0%  - org.eclipse.jgit.storage.pack.DeltaStream.read:213
18.0%  - org.eclipse.jgit.storage.pack.DeltaStream.seekBase:333
6 Collapsed methods (show)
17.0%  - java.util.zip.Inflater.inflateBytesnative
9.3%  - org.eclipse.jgit.storage.pack.DeltaStream.seekBase:339
Screen Shot 2013-03-19 at 4.23.27 PM.png
79.3 KB   View   Download
Mar 19, 2013
#8 sop@google.com
Have you tried setting core.streamFileThreshold to some huge value like 2047m in your gerrit.config?
Mar 19, 2013
#9 jhans...@meetme.com
I was just about to make this change and try it again, but before doing that (since this requires a restart of the server), I had him commit .gitattributes with the -delta flag, rebase, git-gc, and then try the push again directly to the branch, and that went through almost immediately.  So I wasn't able to confirm if the streamFileThreshold change would have fixed it.  Seems like something that can be easily reproduced on a non-production instance though.

Would still be good to know why Gerrit hangs for so long on this (1.5 hours).  Attaching a cluster of 3-hour graphs from New Relic monitoring, showing the issue occurring between 14:30 until 16:00 when gerrit was restarted.
Screen Shot 2013-03-19 at 4.52.43 PM.png
120 KB   View   Download
Apr 10, 2013
#10 vivek.a...@gmail.com
I can confirm that this still occurs with 2.5.2. Here's a snippet of the exception:

      at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        ... 76 more
[2013-04-10 12:11:06,498] ERROR com.google.gerrit.httpd.RestApiServlet : Error in GET /changes/?q=is:open+owner:self&q=is:open+reviewer:self+-owner:self&q=is:closed+owner:self+-age:1w+limit:10&O=1
com.google.inject.ProvisionException: Guice provision errors:

1) Cannot open ReviewDb
  while locating com.google.gerrit.server.config.RequestScopedReviewDbProvider
  at com.google.gerrit.server.config.GerritRequestModule.configure(GerritRequestModule.java:68)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error

Jul 10, 2013
#12 aleksand...@gmail.com
Hello, I can confirm that, on Gerrit 2.6.1, setting core.streamFileThreshold to adequately large value solves the problem with large blobs.

I had a problem with a 112 MB push to a Gerrit remote (pushin both over HTTP and SSH stalled at around 40 MB), but after setting the core.streamFileThreshold to 1 GB,  push went through (it took 360 seconds to complete).

I've added the following clause to etc/gerrit.config, then restarted Gerrit, then retried the push:

[core]
        streamFileThreshold = 1024m

Sep 5, 2013
#13 tvaug...@onyxpoint.com
I'm using 2.6.1 and having a similar issue.

I have no problem at all pushing items into Gerrit for review.

However, I do have an issue being able to approve them afterwards. I checked in a file that was around 75M and have been waiting for over 4 hours for the command line review to complete.