| Issue 3094: | Projects become invisible under certain circumstances | |
| 3 people starred this issue and may be notified of changes. | Back to list |
Copied from the initial post in [1] as written by Gustaf Lundh: In certain circumstances Gerrit cannot load refs/meta/config, hence the projects become invisible since Gerrit cannot calculate the ACL for that Git. The logs are _full_ of these: [2014-12-15 05:35:14,276] WARN com.google.gerrit.server.project.ProjectCacheImpl : Cannot read project platform/git/project java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 40bcb7aa806d2790bbf0f7d2ee6a42cd9ee2f03a at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:135) at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2344) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2316) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) at com.google.common.cache.LocalCache.get(LocalCache.java:3932) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:122) at com.google.gerrit.server.project.ProjectControl$GenericFactory.controlFor(ProjectControl.java:80) at com.google.gerrit.server.project.ChangeControl$GenericFactory.controlFor(ChangeControl.java:74) at com.google.gerrit.server.query.change.IsVisibleToPredicate.match(IsVisibleToPredicate.java:62) at com.google.gerrit.server.query.change.IsVisibleToPredicate.match(IsVisibleToPredicate.java:27) at com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:75) at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:130) at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:94) at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:261) at com.google.gerrit.server.query.change.QueryChanges.query0(QueryChanges.java:153) at com.google.gerrit.server.query.change.QueryChanges.query(QueryChanges.java:141) at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:108) at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:41) at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:306) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [Removed some of the stack trace here] Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 40bcb7aa806d2790bbf0f7d2ee6a42cd9ee2f03a at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148) at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229) at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:839) at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:752) at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:122) at com.google.gerrit.server.git.VersionedMetaData.load(VersionedMetaData.java:98) at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:274) at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:258) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ... 56 more It is always a commit in the refs/meta/config that is missing (but not necessary the tip). The issue does not seem appear on refs/heads/*. When the first "Missing unknown" happens in a Git, it will continue to happen every time you access the Git. No clearing of caches helps. As time pass, more Gits gets affected. After just two days we had 15 inaccessible gits. Since we cannot recover from the issue, we think an IOException happens in openPackedObject(WindowCursor curs, AnyObjectId objectId), and the packFile is removed from future access. Interesting enough this happened on a Git with just an empty initial commit, and it had not been touched for months. So it does _not_ seem to be a racing condition where we are trying to read a pack-file while writing to the git directory. I cannot tie the initial MissingObjectException to a certain user interaction, it just suddenly happens when Gerrit is trying to read objects from refs/meta/config. I cannot reproduce it on our staging environment, which makes the issue very hard to debug. So I'm asking for help. This issue makes me very hesitant to update to 2.10. Rolling back to 3.2.0.201312181205-r solved all issues. [1] https://groups.google.com/forum/#!topic/repo-discuss/Qdmbl-YZ4NU
Jan 21, 2015
I don't believe it is related. In our case the object was in the git, even thought jgit suggested otherwise. We believe jgit just disregarded the pack file due to an IOException of unknown origin. /Sven
Jan 22, 2015
(No comment was entered for this change.)
Status:
Submitted
Jan 22, 2015
(No comment was entered for this change.)
Labels:
FixedIn-2.10
Jan 27, 2015
(No comment was entered for this change.)
Status:
Released
|
|
| ► Sign in to add a comment |
I'm not sure if this is related or not, but we had this happen last week with a push to refs/for. We were using 2.9.2. There was no obvious difference between this push and any other except that the object didn't end up in .git/objects. There's nothing unusual in the logs preceding the event. I copied in the first exception below. We had several others re: cannot get full commit message, cannot get author email, replication errors, etc. Fortunately the author had the object in their clone so we were able to work around this by just copying it in manually. [2015-01-13 12:36:28,995] ERROR com.google.gerrit.server.index.Schema : error getting field tr of ChangeData{22455} com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown b575bf3e0583c82966b3e797f896a99eb6c5fe0f at com.google.gerrit.server.index.ChangeField$16.get(ChangeField.java:272) at com.google.gerrit.server.index.ChangeField$16.get(ChangeField.java:264) at com.google.gerrit.server.index.Schema$1.apply(Schema.java:103) at com.google.gerrit.server.index.Schema$1.apply(Schema.java:98) at com.google.common.collect.Iterators$8.transform(Iterators.java:794) at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:646) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at com.google.gerrit.lucene.LuceneChangeIndex.toDocument(LuceneChangeIndex.java:496) at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:296) at com.google.gerrit.server.index.ChangeIndexer$Task.call(ChangeIndexer.java:246) at com.google.gerrit.server.index.ChangeIndexer$Task.call(ChangeIndexer.java:199) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) at com.google.gerrit.server.index.ChangeIndexer.submit(ChangeIndexer.java:196) at com.google.gerrit.server.index.ChangeIndexer.indexAsync(ChangeIndexer.java:119) at com.google.gerrit.server.change.PostReviewers.addReviewers(PostReviewers.java:238) at com.google.gerrit.server.change.PostReviewers.putAccount(PostReviewers.java:155) at com.google.gerrit.server.change.PostReviewers.apply(PostReviewers.java:137) at com.google.gerrit.server.change.PostReviewers.apply(PostReviewers.java:68) at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:303) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132) at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129) at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1539) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1110) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:453) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1044) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:459) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:280) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229) at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:745) Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown b575bf3e0583c82966b3e797f896a99eb6c5fe0f at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:148) at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229) at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:814) at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:727) at com.google.gerrit.server.query.change.ChangeData.loadCommitData(ChangeData.java:440) at com.google.gerrit.server.query.change.ChangeData.commitFooters(ChangeData.java:422) at com.google.gerrit.server.index.ChangeField$16.get(ChangeField.java:269) ... 59 more