My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 3094: Projects become invisible under certain circumstances
3 people starred this issue and may be notified of changes. Back to list
Status:  Released
Owner:  ----
Closed:  Jan 2015


Sign in to add a comment
 
Project Member Reported by ziv...@gmail.com, Jan 9, 2015
Copied from the initial post in [1] as written by Gustaf Lundh:

In certain circumstances Gerrit cannot load refs/meta/config, hence the projects become invisible since Gerrit cannot calculate the ACL for that Git. 

The logs are _full_ of these: 

[2014-12-15 05:35:14,276] WARN  com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project platform/git/project 
java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 40bcb7aa806d2790bbf0f7d2ee6a42cd9ee2f03a 
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) 
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) 
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) 
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135) 
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2344) 
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316) 
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) 
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) 
        at com.google.common.cache.LocalCache.get(LocalCache.java:3932) 
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) 
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) 
        at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:122) 
        at com.google.gerrit.server.project.ProjectControl$GenericFactory.controlFor(ProjectControl.java:80) 
        at com.google.gerrit.server.project.ChangeControl$GenericFactory.controlFor(ChangeControl.java:74) 
        at com.google.gerrit.server.query.change.IsVisibleToPredicate.match(IsVisibleToPredicate.java:62) 
        at com.google.gerrit.server.query.change.IsVisibleToPredicate.match(IsVisibleToPredicate.java:27) 
        at com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:75) 
        at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:130) 
        at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:94) 
        at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:261) 
        at com.google.gerrit.server.query.change.QueryChanges.query0(QueryChanges.java:153) 
        at com.google.gerrit.server.query.change.QueryChanges.query(QueryChanges.java:141) 
        at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:108) 
        at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:41) 
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:306) 
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) 
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) 
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) 
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) 
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) 
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) 
        [Removed some of the stack trace here] 
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 40bcb7aa806d2790bbf0f7d2ee6a42cd9ee2f03a 
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148) 
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229) 
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:839) 
        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:752) 
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:122) 
        at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:98) 
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:274) 
        at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:258) 
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) 
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) 
        ... 56 more 

It is always a commit in the refs/meta/config that is missing (but not necessary the tip). The issue does not seem appear on refs/heads/*. 

When the first "Missing unknown" happens in a Git, it will continue to happen every time you access the Git. No clearing of caches helps. 

As time pass, more Gits gets affected. After just two days we had 15 inaccessible gits. 

Since we cannot recover from the issue, we think an IOException happens in 
        openPackedObject(WindowCursor curs, AnyObjectId objectId), and the packFile is removed from future access. 

Interesting enough this happened on a Git with just an empty initial commit, and it had not been touched for months. So it does _not_ seem to be a racing condition where we are trying to read a pack-file while writing to the git directory. 

I cannot tie the initial MissingObjectException to a certain user interaction, it just suddenly happens when Gerrit is trying to read objects from refs/meta/config. I cannot reproduce it on our staging environment, which makes the issue very hard to debug. So I'm asking for help. This issue makes me very hesitant to update to 2.10. 

Rolling back to 3.2.0.201312181205-r solved all issues. 

[1] https://groups.google.com/forum/#!topic/repo-discuss/Qdmbl-YZ4NU
Jan 18, 2015
#1 saxo...@gmail.com
I'm not sure if this is related or not, but we had this happen last week with a push to refs/for. We were using 2.9.2. There was no obvious difference between this push and any other except that the object didn't end up in .git/objects. There's nothing unusual in the logs preceding the event.

I copied in the first exception below. We had several others re: cannot get full commit message, cannot get author email, replication errors, etc. Fortunately the author had the object in their clone so we were able to work around this by just copying it in manually.

[2015-01-13 12:36:28,995] ERROR com.google.gerrit.server.index.Schema : error getting field tr of ChangeData{22455}
com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown b575bf3e0583c82966b3e797f896a99eb6c5fe0f
        at com.google.gerrit.server.index.ChangeField$16.get(ChangeField.java:272)
        at com.google.gerrit.server.index.ChangeField$16.get(ChangeField.java:264)
        at com.google.gerrit.server.index.Schema$1.apply(Schema.java:103)
        at com.google.gerrit.server.index.Schema$1.apply(Schema.java:98)
        at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:646)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at com.google.gerrit.lucene.LuceneChangeIndex.toDocument(LuceneChangeIndex.java:496)
        at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:296)
        at com.google.gerrit.server.index.ChangeIndexer$Task.call(ChangeIndexer.java:246)
        at com.google.gerrit.server.index.ChangeIndexer$Task.call(ChangeIndexer.java:199)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
        at com.google.gerrit.server.index.ChangeIndexer.submit(ChangeIndexer.java:196)
        at com.google.gerrit.server.index.ChangeIndexer.indexAsync(ChangeIndexer.java:119)
        at com.google.gerrit.server.change.PostReviewers.addReviewers(PostReviewers.java:238)
        at com.google.gerrit.server.change.PostReviewers.putAccount(PostReviewers.java:155)
        at com.google.gerrit.server.change.PostReviewers.apply(PostReviewers.java:137)
        at com.google.gerrit.server.change.PostReviewers.apply(PostReviewers.java:68)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:303)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1539)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1110)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:453)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1044)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:459)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:280)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229)
        at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown b575bf3e0583c82966b3e797f896a99eb6c5fe0f
        at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148)
        at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
        at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:814)
        at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:727)
        at com.google.gerrit.server.query.change.ChangeData.loadCommitData(ChangeData.java:440)
        at com.google.gerrit.server.query.change.ChangeData.commitFooters(ChangeData.java:422)
        at com.google.gerrit.server.index.ChangeField$16.get(ChangeField.java:269)
        ... 59 more

Jan 21, 2015
#3 sven.sel...@sonymobile.com
I don't believe it is related. In our case the object was in the git, even thought jgit suggested otherwise. We believe jgit just disregarded the pack file due to an IOException of unknown origin.

/Sven

Jan 21, 2015
Project Member #4 david.pu...@sonymobile.com
https://gerrit-review.googlesource.com/#/c/63453/
Status: ChangeUnderReview
Jan 22, 2015
Project Member #5 ziv...@gmail.com
(No comment was entered for this change.)
Status: Submitted
Jan 22, 2015
Project Member #6 ziv...@gmail.com
(No comment was entered for this change.)
Labels: FixedIn-2.10
Jan 27, 2015
Project Member #7 ziv...@gmail.com
(No comment was entered for this change.)
Status: Released
Sign in to add a comment

Powered by Google Project Hosting