My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 3634: reindexing fails with 2.11
3 people starred this issue and may be notified of changes. Back to list
Status:  New
Owner:  ----


Sign in to add a comment
 
Reported by sjakub, Oct 28, 2015

Affected Version: 2.11.*

What steps will reproduce the problem?
1. Update from earlier versions (2.8.6.1 and 2.10 tested)
2. java -jar gerrit-2.11.4.war reindex -d review_site

I tried upgrading straight from 2.8.6.1 to 2.11.4, and in the result most of the changes were missing.
So I tried upgrading first to 2.9, and then to 2.10. Up to this point everything was working fine.
Upgrading from 2.10 to 2.11 or 2.11.4 loses most of the changes.

The problem seems to be with reindexing with 2.11+ versions, when it tries to create an index version 14.
I tried reindexing with 2.10 and it worked fine.

But reindexing using 2.11 results in a lot of errors.
I tried reindexing when index version was 8, and when it was 11 (after I reindexed using 2.10).
In both cases I was seeing the same errors.

I am using postgres (with max_connections set to 100), and Lucene index type.
I tried changing the database.poolMaxWait option (up to 5 minutes) and it didn't help.

This is the output:

$ java -jar gerrit-2.11.4.war reindex -d review_site
[2015-10-28 14:07:50,168] WARN  com.google.gerrit.pgm.util.ThreadLimiter : Limiting program to 8 threads due to database.poolLimit
[2015-10-28 14:07:50,222] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 228m
[2015-10-28 14:07:50,794] INFO  com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /home/gerrit2/review_site/cache
Collecting projects:    26
Reindexing changes: projects: 3% (1/26), 0% (60/7270) (-)[2015-10-28 14:08:22,312] WARN  com.google.gerrit.server.git.ScanningChangeCacheImpl : Cannot fetch changes for MyProject
java.util.concurrent.ExecutionException: com.google.gwtorm.server.OrmException: Cannot open database connection
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2348)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2320)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at com.google.gerrit.server.git.ScanningChangeCacheImpl.get(ScanningChangeCacheImpl.java:78)
        at com.google.gerrit.server.index.SiteIndexer$3.call(SiteIndexer.java:244)
        at com.google.gerrit.server.index.SiteIndexer$3.call(SiteIndexer.java:234)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:130)
        at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
        at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22.<init>(Unknown Source)
        at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$22_Factory_GwtOrm$$23.open(Unknown Source)
        at com.google.gwtorm.jdbc.Database.open(Database.java:122)
        at com.google.gerrit.server.util.ManualRequestContext.<init>(ManualRequestContext.java:37)
        at com.google.gerrit.server.util.OneOffRequestContext.open(OneOffRequestContext.java:48)
        at com.google.gerrit.server.git.ScanningChangeCacheImpl$Loader.load(ScanningChangeCacheImpl.java:99)
        at com.google.gerrit.server.git.ScanningChangeCacheImpl$Loader.load(ScanningChangeCacheImpl.java:85)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        ... 17 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at com.google.gwtorm.jdbc.Database.newConnection(Database.java:128)
        ... 27 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        ... 29 more

Followed by many more like this.

It eventually succeeds, and the last message is:
Reindexing changes: projects: 100% (26/26), 1% (115/7270), done    
Reindexed 115 changes in 120.4s (1.0/s)

And most of the changes are missing.

Oct 28, 2015
Project Member #1 huga...@gmail.com
in gerrit.config, what is your database.poolLimit ?
Oct 28, 2015
#2 sjakub
I didn't set it. I think the default is 8.
Oct 28, 2015
Project Member #4 huga...@gmail.com
It looks like some change reindexing are timing out trying to get a db pooled connection but this is not seen as a reindex failure, that is why you end up with most of the changes missing.

The strange thing is reindex is limiting itself to 8 thread to prevent this timeout error but obviously, something is not working as intended.

On our side, we use postgres and we did upgrade from 2.10 to 2.11 without any issue.
Here are our db related configuration:
        poolLimit = 576
        poolMaxWait = 5000ms
        poolMaxIdle = 10

I suggest you set poolLimit to 100, since you allow 100 connection from the db server side, delete index 14 and retry reindex (maybe allowing more connection will work around the issue).
Oct 29, 2015
#5 leand...@motorola.com
I'm seeing a very similar problem, upgrading from 2.8.4 to 2.11.3:

https://groups.google.com/forum/#!topic/repo-discuss/jPMnfYfTiZU

I did some tests with 2.10.2 and the indexing command from the changes REST API was working fine, now with 2.11.3 it is failing for 99% of the changes.
Sign in to add a comment

Powered by Google Project Hosting