My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 3300: Severe performance degradation on Git operations in 2.10 compared to 2.9
8 people starred this issue and may be notified of changes. Back to list
Status:  Submitted
Owner:  ----
Closed:  Apr 2015


Sign in to add a comment
 
Reported by org...@gmail.com, Apr 15, 2015
Affected Version: 2.11

What steps will reproduce the problem?
1. Submit a change ;-)

What is the expected output? What do you see instead?
With 2.9 on the same server and settings it used to take 1-2 seconds.

With 2.11 it takes about 6-8 seconds.

Apr 15, 2015
#1 org...@gmail.com
Sorry, it was my fault. I upgraded gerrit but didn't upgrade its plugins (reviewnotes caused an exception).

There should be a warning on the init phase though...
Apr 15, 2015
Project Member #2 david.pu...@sonymobile.com
If you run the init from a release WAR that contains the plugins, it will prompt you to upgrade them.

Status: AwaitingInformation
Apr 15, 2015
Project Member #3 david.pu...@sonymobile.com
Did you upgrade from the WAR file for one of the RC builds, or something you built locally?
Apr 15, 2015
#4 org...@gmail.com
I built it locally with the submodules updated. I used init --batch for the upgrade.
Apr 15, 2015
Project Member #5 david.pu...@sonymobile.com
how did you build it?

If you only used 'buck build gerrit' the plugins are not included in the WAR file.  You have to build with 'buck build release' to include the plugins.
Apr 15, 2015
#6 org...@gmail.com
buck build release. Maybe --batch doesn't upgrade the plugins?
Apr 15, 2015
Project Member #7 david.pu...@sonymobile.com
It probably doesn't.

Note that it also doesn't download the updated bouncycastle libs, so be aware of that if you're using bouncycastle crypto.
Apr 16, 2015
Project Member #8 david.pu...@sonymobile.com
(No comment was entered for this change.)
Status: WontFix
Apr 19, 2015
#9 org...@gmail.com
Well, it still slow, and now the log shows nothing...
Apr 21, 2015
Project Member #10 david.pu...@sonymobile.com
(No comment was entered for this change.)
Status: New
Apr 24, 2015
Project Member #11 huga...@gmail.com
Where are you experiencing the slowness, on your production server where there is traffic or on a test server where you submit only one change at the time?
Apr 25, 2015
#12 florent....@intersec.com
Gerrit 2.11 get blocked during 2 minutes while I submitted 2 fast-forwards on a repository. The performance of the submission seems pretty random (maybe, but not sure, dependent on the size of the repository).
Apr 25, 2015
#13 thesa...@gmail.com
We feel this slowness too, both in the UI (following submitting code, rebasing), and through SSH. I have just submitted a commit through SSH, it took a lot longer (maybe 20 seconds) than what it would normally take (2-3 seconds).  The CPU usage on the server remained high for a few minutes, I took some stacktraces and it looks like the replication threads were very busy following the submit:

"ReplicateTo-local repos-1" prio=10 tid=0x00007fc8cc01f000 nid=0x261 runnable [0x00007fc8c1b65000]
   java.lang.Thread.State: RUNNABLE
        at org.eclipse.jgit.internal.storage.file.PackIndexV2.binarySearchLevelTwo(PackIndexV2.java:302)
        at org.eclipse.jgit.internal.storage.file.PackIndexV2.findOffset(PackIndexV2.java:217)
        at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:258)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:417)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:386)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:378)
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
        at org.eclipse.jgit.lib.ObjectReader$1.open(ObjectReader.java:302)
        at org.eclipse.jgit.revwalk.RevWalk$2.next(RevWalk.java:977)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1714)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:799)
        at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:762)
        at org.eclipse.jgit.transport.BasePackPushConnection.writePack(BasePackPushConnection.java:304)
        at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:197)
        at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1200)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1246)
        at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:399)
        at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:371)
        at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:286)
        at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:81)
        at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:256)
        at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:253)
        at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
        at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
        at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75)
        at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:253)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Wishfully thinking, there is a single cause for all the slow operations that touch Git under Gerrit 2.11, maybe something related to JGit has changed?
Apr 26, 2015
#14 org...@gmail.com
I don't use the replication plugin. Even when I removed all the plugins it was still slow (~5-6 seconds).

It happens on both my test and production server (test on linux, production on Windows). Both are not loaded, and we don't have multiple simultaneous submits.
Apr 26, 2015
Project Member #15 David.Os...@gmail.com
https://gerrit-review.googlesource.com/67465
Summary: Severe preformance degradation on Git operations in 2.10 compared to 2.9 (was: Submit is very slow with 2.11 compared to 2.9)
Status: ChangeUnderReview
Labels: -Priority-Minor Priority-Critical
Apr 26, 2015
Project Member #16 David.Os...@gmail.com
(No comment was entered for this change.)
Summary: Severe performance degradation on Git operations in 2.10 compared to 2.9 (was: Severe preformance degradation on Git operations in 2.10 compared to 2.9)
Apr 27, 2015
Project Member #17 ziv...@gmail.com
Fixed in https://gerrit-review.googlesource.com/67465

Alternatively, set the receive.checkReferencedObjectsAreReachable=false in gerrit.config.
Apr 27, 2015
#18 org...@gmail.com
receive.checkReferencedObjectsAreReachable=false doesn't help here.
Apr 27, 2015
Project Member #19 David.Os...@gmail.com
(No comment was entered for this change.)
Status: Submitted
Apr 27, 2015
Project Member #20 edwin.ke...@gmail.com
(No comment was entered for this change.)
Labels: FixedIn-2.10.4
May 10, 2015
#21 org...@gmail.com
Please reopen. It's become worse with JGit 4.0...
May 11, 2015
#22 org...@gmail.com
jstack trace with 2.11-gce6c4bc7: https://gist.github.com/orgads/942671d3e2a9ee11ac15
May 12, 2015
Project Member #23 ziv...@gmail.com
Two questions:

1. I don't see the thread names (and other thread data) in the thread dump. Have you edited the output of jstack before posting it?

2. In your comment on [1] you said that it was tested on 2.11. However, this issue is about the performance issue of 2.10 compared to 2.9. Have you actually tried the 2.10.4?

[1] https://gerrit-review.googlesource.com/67653
May 12, 2015
Project Member #24 ziv...@gmail.com
As confirmed in 3363 this is an issue affecting 2.11, not 2.10.

This issue can stay closed.
Sign in to add a comment

Powered by Google Project Hosting