| Issue 3300: | Severe performance degradation on Git operations in 2.10 compared to 2.9 | |
| 8 people starred this issue and may be notified of changes. | Back to list |
Affected Version: 2.11 What steps will reproduce the problem? 1. Submit a change ;-) What is the expected output? What do you see instead? With 2.9 on the same server and settings it used to take 1-2 seconds. With 2.11 it takes about 6-8 seconds.
Apr 15, 2015
#1
org...@gmail.com
Apr 15, 2015
If you run the init from a release WAR that contains the plugins, it will prompt you to upgrade them.
Status:
AwaitingInformation
Apr 15, 2015
Did you upgrade from the WAR file for one of the RC builds, or something you built locally?
Apr 15, 2015
I built it locally with the submodules updated. I used init --batch for the upgrade.
Apr 15, 2015
how did you build it? If you only used 'buck build gerrit' the plugins are not included in the WAR file. You have to build with 'buck build release' to include the plugins.
Apr 15, 2015
buck build release. Maybe --batch doesn't upgrade the plugins?
Apr 15, 2015
It probably doesn't. Note that it also doesn't download the updated bouncycastle libs, so be aware of that if you're using bouncycastle crypto.
Apr 16, 2015
(No comment was entered for this change.)
Status:
WontFix
Apr 19, 2015
Well, it still slow, and now the log shows nothing...
Apr 21, 2015
(No comment was entered for this change.)
Status:
New
Apr 24, 2015
Where are you experiencing the slowness, on your production server where there is traffic or on a test server where you submit only one change at the time?
Apr 25, 2015
Gerrit 2.11 get blocked during 2 minutes while I submitted 2 fast-forwards on a repository. The performance of the submission seems pretty random (maybe, but not sure, dependent on the size of the repository).
Apr 25, 2015
We feel this slowness too, both in the UI (following submitting code, rebasing), and through SSH. I have just submitted a commit through SSH, it took a lot longer (maybe 20 seconds) than what it would normally take (2-3 seconds). The CPU usage on the server remained high for a few minutes, I took some stacktraces and it looks like the replication threads were very busy following the submit:
"ReplicateTo-local repos-1" prio=10 tid=0x00007fc8cc01f000 nid=0x261 runnable [0x00007fc8c1b65000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.jgit.internal.storage.file.PackIndexV2.binarySearchLevelTwo(PackIndexV2.java:302)
at org.eclipse.jgit.internal.storage.file.PackIndexV2.findOffset(PackIndexV2.java:217)
at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:258)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:417)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:386)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:378)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
at org.eclipse.jgit.lib.ObjectReader$1.open(ObjectReader.java:302)
at org.eclipse.jgit.revwalk.RevWalk$2.next(RevWalk.java:977)
at org.eclipse.jgit.internal.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1714)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:799)
at org.eclipse.jgit.internal.storage.pack.PackWriter.preparePack(PackWriter.java:762)
at org.eclipse.jgit.transport.BasePackPushConnection.writePack(BasePackPushConnection.java:304)
at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:197)
at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152)
at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165)
at org.eclipse.jgit.transport.Transport.push(Transport.java:1200)
at org.eclipse.jgit.transport.Transport.push(Transport.java:1246)
at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:399)
at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:371)
at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:286)
at com.googlesource.gerrit.plugins.replication.PushOne.access$000(PushOne.java:81)
at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:256)
at com.googlesource.gerrit.plugins.replication.PushOne$1.call(PushOne.java:253)
at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75)
at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:253)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Wishfully thinking, there is a single cause for all the slow operations that touch Git under Gerrit 2.11, maybe something related to JGit has changed?
Apr 26, 2015
I don't use the replication plugin. Even when I removed all the plugins it was still slow (~5-6 seconds). It happens on both my test and production server (test on linux, production on Windows). Both are not loaded, and we don't have multiple simultaneous submits.
Apr 26, 2015
https://gerrit-review.googlesource.com/67465
Summary:
Severe preformance degradation on Git operations in 2.10 compared to 2.9
(was: Submit is very slow with 2.11 compared to 2.9)
Status: ChangeUnderReview Labels: -Priority-Minor Priority-Critical
Apr 26, 2015
(No comment was entered for this change.)
Summary:
Severe performance degradation on Git operations in 2.10 compared to 2.9
(was: Severe preformance degradation on Git operations in 2.10 compared to 2.9)
Apr 27, 2015
Fixed in https://gerrit-review.googlesource.com/67465 Alternatively, set the receive.checkReferencedObjectsAreReachable=false in gerrit.config.
Apr 27, 2015
receive.checkReferencedObjectsAreReachable=false doesn't help here.
Apr 27, 2015
(No comment was entered for this change.)
Status:
Submitted
Apr 27, 2015
(No comment was entered for this change.)
Labels:
FixedIn-2.10.4
May 10, 2015
Please reopen. It's become worse with JGit 4.0...
May 11, 2015
jstack trace with 2.11-gce6c4bc7: https://gist.github.com/orgads/942671d3e2a9ee11ac15
May 12, 2015
Two questions: 1. I don't see the thread names (and other thread data) in the thread dump. Have you edited the output of jstack before posting it? 2. In your comment on [1] you said that it was tested on 2.11. However, this issue is about the performance issue of 2.10 compared to 2.9. Have you actually tried the 2.10.4? [1] https://gerrit-review.googlesource.com/67653
May 12, 2015
As confirmed in 3363 this is an issue affecting 2.11, not 2.10. This issue can stay closed. |
|
| ► Sign in to add a comment |