| Issue 3363: | Submit is very slow with 2.11 compared to 2.10 | |
| 4 people starred this issue and may be notified of changes. | Back to list |
Affected Version: 2.11 What steps will reproduce the problem? 1. Submit a change What is the expected output? What do you see instead? With 2.9 on the same server and settings it used to take 1-2 seconds. With 2.11 it takes about 6-8 seconds. jstack trace with 2.11-gce6c4bc7: https://gist.github.com/orgads/942671d3e2a9ee11ac15 See discussion on https://gerrit-review.googlesource.com/67653
May 12, 2015
Project Member
#1
edwin.ke...@gmail.com
May 12, 2015
Cherry-Pick. I don't use other submit types, and can't change it for this project (other projects are not as slow).
May 12, 2015
Title of this issue is "compared to 2.10". In description you are saying: "With 2.9 on the same server and settings it used to take 1-2 seconds." Could you test the submit operation for the same project and setting on recently released 2.10.4 and post the result?
May 12, 2015
Is it possible to downgrade gerrit? How?
May 12, 2015
How many open changes do you have on the branch your are submitting to? Can you close all open changes and test submit performance again?
May 12, 2015
I have 116 open changes. Closing all the changes fixes performance. I can try running 2.10 on my test server.
May 12, 2015
>Is it possible to downgrade gerrit? How? This would be hard. * 2.10: schema version 98 * 2.11: schema version 107 You would need to undo these version upgrades. Don't you have some backups from your 2.9 or 2.10 production databases, so that you can restore this in test environment?
May 12, 2015
I have a backup of the DB from 2.9. I'll use it on my test environment.
May 12, 2015
By the way. On the branch I tested against there are only 11 open changes. I doubt it only reindexes this branch.
May 12, 2015
Works much better with 2.10 (back to ~2 seconds).
May 12, 2015
Hmm... here you say that it works much better with 2.10. Which 2.10? 2.10.4? Do we have the same issue in 2.10? Is the issue 3300 still relevant?
May 12, 2015
2.10 => origin/stable-2.10 (that is v2.10.4). > Is the issue 3300 still relevant? Well, I opened it against 2.11, not 2.10 :) It was changed later.
May 12, 2015
OK then 3300 can be closed (again).
May 12, 2015
> Works much better with 2.10 (back to ~2 seconds)
Like I said in my comment in [1] I suspect that's because of reindexing of all open changes during push. That's the only difference between 2.10 and 2.11. JGit version in 2.10.4 and in stable-2.11 is exactly the same: 4.0.0.201505050340-m2. Can you patch the 2.11 and disable Git ReferenceUpdatedListener and report if this solved your performance problem on 2.11?
Here is the diff:
diff --git a/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java b/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java
index 6ad264b..5bd86cc 100644
--- a/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java
+++ b/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java
@@ -255,7 +255,7 @@ public class GerritGlobalModule extends FactoryModule {
DynamicSet.setOf(binder(), ProjectDeletedListener.class);
DynamicSet.setOf(binder(), HeadUpdatedListener.class);
DynamicSet.setOf(binder(), UsageDataPublishedListener.class);
- DynamicSet.bind(binder(), GitReferenceUpdatedListener.class).to(ReindexAfterUpdate.class);
+ //DynamicSet.bind(binder(), GitReferenceUpdatedListener.class).to(ReindexAfterUpdate.class);
DynamicSet.bind(binder(), GitReferenceUpdatedListener.class)
.to(ProjectConfigEntry.UpdateChecker.class);
DynamicSet.setOf(binder(), EventListener.class);
Interessting enough, with the diff applied all unit tests are still passing.
May 12, 2015
OK just checked the code:
ListeningExecutorService getBatchIndexExecutor(
@IndexExecutor(INTERACTIVE) ListeningExecutorService interactive,
@GerritServerConfig Config config,
WorkQueue workQueue) {
if (batchExecutor != null) {
return batchExecutor;
}
int threads = config.getInt("index", null, "batchThreads", 0);
if (threads <= 0) {
threads = config.getInt("changeMerge", null, "threadPoolSize", 0);
}
if (threads <= 0) {
return interactive;
}
return MoreExecutors.listeningDecorator(
workQueue.createQueue(threads, "Index-Batch"));
}
The problem is probably, that both:
* index.batchThreads
* changeMerge.threadPoolSize
aren't set on your site, in which case the reindexing would happen interactively. In your case as you reported, you have 116 open changes on this branch. 6-8 sec. to reindex 116 changes is fast.
Just set index.batchThreads (Note that changeMerge.threadPoolSize is deprecated) to some sane value and you should be OK.
We should probably warn in ReleaseNotes, that since 2.11 this setting is critical, otherwise the user would see dramatic performance degradation on push operation. And this degradation is linear to the number of open changes on the branch.
Status:
AwaitingInformation
May 12, 2015
Thanks! Now it's down to ~4 seconds, which is much better (in the original post it was 6-8, recently it became closer to 15 seconds). I suggest setting a sensible default (2? auto-detection by number of processors?) in addition to adding this information to the release notes.
May 13, 2015
>>Is it possible to downgrade gerrit? How? > >This would be hard. > >* 2.10: schema version 98 >* 2.11: schema version 107 > >You would need to undo these version upgrades. Turn out, that OpenStack guys already did that and downgraded from 2.10 t0 2.8: https://review.openstack.org/#/c/182113/2/tools/gerrit-rollback-2.10-2.8.sql
May 13, 2015
(No comment was entered for this change.)
Status:
Submitted
Labels: FixedIn-2.11.1
Jun 21, 2015
(No comment was entered for this change.)
Status:
Released
|
|
| ► Sign in to add a comment |