My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 3727: Related Changes OutOfMemoryError
1 person starred this issue and may be notified of changes. Back to list
Status:  Released
Owner:  dborowitz@google.com
Closed:  Dec 19


Sign in to add a comment
 
Reported by exceptio...@gmail.com, Dec 17, 2015
Affected Version: 2.12_rc2

What steps will reproduce the problem?
1. Open issue which has related changes

What is the expected output? What do you see instead?
I expect to see related changes. Instead I get Waiting... message and 500 server error several minutes later.

I have tried to increase -Xmx1000m to -Xmx3000m but it did not help.

Error message

Dec 17 16:06:53 gerrit gerrit: [2015-12-17 16:06:53,189] [HTTP-56] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /review/changes/7644/revisions/4909fdd5127983f7714cab5c9825befd21bdad6f/related
Dec 17 16:06:53 gerrit gerrit: java.lang.OutOfMemoryError: GC overhead limit exceeded
Dec 17 16:06:53 gerrit gerrit: at org.eclipse.jgit.lib.AnyObjectId.toHexCharArray(AnyObjectId.java:462)
Dec 17 16:06:53 gerrit gerrit: at org.eclipse.jgit.lib.AnyObjectId.name(AnyObjectId.java:498)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.server.change.GetRelated$ChangeAndCommit.<init>(GetRelated.java:153)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.server.change.GetRelated.getRelated(GetRelated.java:99)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:63)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:44)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:322)
Dec 17 16:06:53 gerrit gerrit: at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
Dec 17 16:06:53 gerrit gerrit: at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
Dec 17 16:06:53 gerrit gerrit: at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
Dec 17 16:06:53 gerrit gerrit: at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
Dec 17 16:06:53 gerrit gerrit: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

Dec 18, 2015
#1 exceptio...@gmail.com
More details. This issue happened after upgrade from 2.11 to 2.12_rc2. Increasing memory limit to 10G still did not help. I "fixed" it by deleting code for getting related changes entirely and recompiling gerrit. Still I hope  it will get fixed since I need related changes functionality.
Dec 18, 2015
#2 dborowitz@google.com
Hm. Just looking at that stack trace, I would guess there are just too many objects being allocated in that path. It can't be an infinite loop, because everything has been materialized to a List at that point.

How many related changes (opened changes pushed together) do you think there are? Can you share the repository?

If not, I would be interested in seeing a few values from a debugger:
cd.size() from the top of GetRelated#getRelated(RevisionResource)
-ancestors.size() and descendants.size() from the bottom of RelatedChangesSorter#sort(List, PatchSet)

Dec 18, 2015
#3 dborowitz@google.com
(No comment was entered for this change.)
Owner: dborowitz@google.com
Dec 18, 2015
#4 dborowitz@google.com
Also are there any merge commits involved?
Dec 18, 2015
#5 exceptio...@gmail.com
OK, so let me answer one by one

How many related changes (opened changes pushed together) do you think there are?
At that time there was like 300 related changes. The same changeset worked fine with 2.11, the issue started to happen just after upgrade to 2.12_rc2

Can you share the repository?
Unfortunately I cannot share the repository, this is a private project

If not, I would be interested in seeing a few values from a debugger:
Currently I fixed it by deleting related changes functionality completely because I need a working gerrit instance for our team to be able to work. I can however create another node with the same data and original gerrit 2.12_rc2 (though it will take some time).

Also are there any merge commits involved?
Yes, there are many merge commits actually. Unfortunately I cannot remove them from the current workflow. But again it worked just fine in 2.11.
Dec 18, 2015
#6 dborowitz@google.com
(No comment was entered for this change.)
Labels: -Priority-Minor Priority-Blocker Blocking-2.12
Dec 18, 2015
#7 dborowitz@google.com
How large is the total size of project history?

Is your topology something like:

--X1--X2-...-Xn--A--B1-...-Bn--M
       \------C1-...-Cn-------/

where:
X1-Xn: lots of old history
A: current branch tip
B1-Bn & C1-Cn: two long branches of open changes
M: open merge commit

If so, which change is related changes overflowing on? Bi, Ci, or M?
Dec 18, 2015
#8 exceptio...@gmail.com
There are like 8000 changes in gerrit. I imported existing git repo, so the total commits count in this repo is around 37000.

I think I need to explain my current workflow a bit more because I am not quite sure my topology is like you showed (or I just misunderstood it).

Currently just a few members of our team are using Gerrit its intended way by pushing their own commits for review in some A project.

The remaining majority of our team are pushing to git on other node bypassing gerrit. But their commits need to be reviewed too, so I created separate B project for this purpose and this issue happened in B project.

My workflow of reviewing changes in B project

1) reset to some point in git history by taking a few days of unreviewed commits after current master branch tip on Gerrit.
2) push to HEAD:refs/for/master
3) reviews changes in Gerrit, but don't submit for merge them
4) push to Gerrit master, so the changes will be marked as merged automatically.

Using this workflow I usually had a lot of related changes in 2.11 (it showed 500 most of the times).

Dec 18, 2015
#9 dborowitz@google.com
I think it's possible in step (2) to get a topology like the one I describe. If you're just importing unreviewed changes that were pushed directly to another repo, you don't really know what they might contain.

All that being said, it's still surprising that this would eat up 10G of heap, even if we were creating several thousand objects.
Dec 18, 2015
#10 exceptio...@gmail.com
Yes, I agree. I am going to setup a working copy with original Gerrit 2.12_rc2 to be able to debug then. If you could point me to some instructions on how to debug Gerrit (or java app in general), that would speed up the process considerably. Otherwise I will find it on my own. 
Dec 18, 2015
#11 dborowitz@google.com
For the first step, just those couple of .size() things I asked for, you don't really need a debugger, we'll just start with good ol' printf debugging. Just apply this change to the 2.12-rc2 tag and rebuild gerrit:

https://gerrit-review.googlesource.com/73480
Dec 18, 2015
#12 exceptio...@gmail.com
Ok I cloned the working node and did the following

1) removed gerrit
2) restored gerrit from backup of Dec 15th (made before upgrading to 2.12_rc2)
3) confirmed related changes work fine
4) compiled gerrit from branch stable-2.12 with your changes
5) opened random open change
6) waited for OOM error

This was tested with -Xmx2800m

Stacktrace

Dec 18 22:04:06 gerrit-temp gerrit: === GetRelated: 309 input ChangeDatas
Dec 18 22:04:24 gerrit-temp gerrit: === RelatedChangesSorter: 17253805 ancestors
Dec 18 22:04:24 gerrit-temp gerrit: === RelatedChangesSorter: 673 descendants
Dec 18 22:04:24 gerrit-temp gerrit: === GetRelated: 17254478 total PatchSetDatas
Dec 18 22:04:44 gerrit-temp gerrit: === RelatedChangesSorter: 17253805 ancestors
Dec 18 22:04:44 gerrit-temp gerrit: === RelatedChangesSorter: 673 descendants
Dec 18 22:08:16 gerrit-temp gerrit: [2015-12-18 22:08:16,152] [HTTP-29] WARN  org.eclipse.jetty.servlet.ServletHandler : Error for /review/changes/7795/revisions/c2f7e6f417257fa5d7032f82dcb4886efb3ff84e/related
Dec 18 22:08:16 gerrit-temp gerrit: java.lang.OutOfMemoryError: GC overhead limit exceeded
Dec 18 22:08:16 gerrit-temp gerrit: at org.eclipse.jgit.lib.AnyObjectId.toHexCharArray(AnyObjectId.java:462)
Dec 18 22:08:16 gerrit-temp gerrit: at org.eclipse.jgit.lib.AnyObjectId.name(AnyObjectId.java:498)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated$ChangeAndCommit.<init>(GetRelated.java:152)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.getRelated(GetRelated.java:102)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:63)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:44)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:322)
Dec 18 22:08:16 gerrit-temp gerrit: at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
Dec 18 22:08:16 gerrit-temp gerrit: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
Dec 18 22:08:16 gerrit-temp gerrit: [2015-12-18 22:08:16,153] [HTTP-29] ERROR com.google.gerrit.pgm.http.jetty.HiddenErrorHandler : Error in GET /review/changes/7795/revisions/c2f7e6f417257fa5d7032f82dcb4886efb3ff84e/related
Dec 18 22:08:16 gerrit-temp gerrit: java.lang.OutOfMemoryError: GC overhead limit exceeded
Dec 18 22:08:16 gerrit-temp gerrit: at org.eclipse.jgit.lib.AnyObjectId.toHexCharArray(AnyObjectId.java:462)
Dec 18 22:08:16 gerrit-temp gerrit: at org.eclipse.jgit.lib.AnyObjectId.name(AnyObjectId.java:498)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated$ChangeAndCommit.<init>(GetRelated.java:152)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.getRelated(GetRelated.java:102)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:63)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:44)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:322)
Dec 18 22:08:16 gerrit-temp gerrit: at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
Dec 18 22:08:16 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
Dec 18 22:08:16 gerrit-temp gerrit: at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
Dec 18, 2015
#13 dborowitz@google.com
> Dec 18 22:04:24 gerrit-temp gerrit: === RelatedChangesSorter: 17253805 ancestors

Umm yeah that'll do it. And you're sure there are not 17M commits in this repo right? :)
Dec 18, 2015
#14 exceptio...@gmail.com
I don't think there are 17M commits. Unless I screwed up something. But it works in 2.11 anyway.

bash-4.2$ git rev-list --all --count
31949
Dec 18, 2015
#15 dborowitz@google.com
Yeah, I understand that this is a regression in 2.12, the related changes handler was rewritten from scratch.

I'm still not sure how the set could grow that big, here's another patch set with more debugging:
https://gerrit-review.googlesource.com/73480
Dec 18, 2015
#16 exceptio...@gmail.com
ok new stacktrace

Dec 18 22:35:37 gerrit-temp gerrit: === GetRelated: 309 input ChangeDatas
Dec 18 22:35:37 gerrit-temp gerrit: === RelatedChangesSorter: 309 total revisions
Dec 18 22:35:37 gerrit-temp gerrit: === RelatedChangesSorter: 309 patch sets total
Dec 18 22:35:37 gerrit-temp gerrit: === RelatedChangesSorter: 367 entries in parent map
Dec 18 22:35:37 gerrit-temp gerrit: === RelatedChangesSorter: 301 keys in parent map
Dec 18 22:35:37 gerrit-temp gerrit: === RelatedChangesSorter: 367 in children map
Dec 18 22:35:37 gerrit-temp gerrit: === RelatedChangesSorter: 308 keys in children map
Dec 18 22:35:54 gerrit-temp gerrit: === RelatedChangesSorter: 17253805 ancestors
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 673 descendants
Dec 18 22:35:55 gerrit-temp gerrit: === GetRelated: 17254478 total PatchSetDatas
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 309 total revisions
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 309 patch sets total
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 367 entries in parent map
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 301 keys in parent map
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 367 in children map
Dec 18 22:35:55 gerrit-temp gerrit: === RelatedChangesSorter: 308 keys in children map
Dec 18 22:36:15 gerrit-temp gerrit: === RelatedChangesSorter: 17253805 ancestors
Dec 18 22:36:15 gerrit-temp gerrit: === RelatedChangesSorter: 673 descendants
Dec 18 22:39:49 gerrit-temp gerrit: [2015-12-18 22:39:48,886] [HTTP-29] WARN  org.eclipse.jetty.servlet.ServletHandler : Error for /review/changes/7795/revisions/c2f7e6f417257fa5d7032f82dcb4886efb3ff84e/related
Dec 18 22:39:49 gerrit-temp gerrit: java.lang.OutOfMemoryError: GC overhead limit exceeded
Dec 18 22:39:49 gerrit-temp gerrit: at java.nio.ByteBuffer.wrap(ByteBuffer.java:369)
Dec 18 22:39:49 gerrit-temp gerrit: at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.java:963)
Dec 18 22:39:49 gerrit-temp gerrit: at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:930)
Dec 18 22:39:49 gerrit-temp gerrit: at org.eclipse.jgit.util.RawParseUtils.parsePersonIdent(RawParseUtils.java:751)
Dec 18 22:39:49 gerrit-temp gerrit: at org.eclipse.jgit.revwalk.RevCommit.getAuthorIdent(RevCommit.java:404)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated$ChangeAndCommit.<init>(GetRelated.java:159)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.getRelated(GetRelated.java:102)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:63)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.server.change.GetRelated.apply(GetRelated.java:44)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:322)
Dec 18 22:39:49 gerrit-temp gerrit: at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:82)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:73)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:117)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:136)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:105)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
Dec 18 22:39:49 gerrit-temp gerrit: at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
Dec 18, 2015
#17 dborowitz@google.com
Here are two separate potential fixes:
https://gerrit-review.googlesource.com/73484
https://gerrit-review.googlesource.com/73442

Those are both kind of stabs in the dark, but if one of them works, I can try to puzzle out why.
Dec 18, 2015
#20 dborowitz@google.com
No, we'll just stick with the first one. Glad it works.

Still not sure why though, unless your topology has like lots of octopus merges, or there are a bunch of commits that are each a parent of a bunch of different merge commits.

I would be interested in seeing the output of "gitk c2f7e6f417257fa5d7032f82dcb4886efb3ff84e", if you have the patience to photoshop out the commit messages.
Dec 18, 2015
#21 dborowitz@google.com
Actually I think "a bunch" really means "some". You don't need that many to make the number of distinct paths blow up real big.
Dec 18, 2015
#22 exceptio...@gmail.com
well, you're right. though I cannot use gitk atm, I used git log --graph and output looked scary

| * | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |   commit 209256095762704a1ed8022c4a9e2912a4714f57
| |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  Merge: 30b8d88 f1fb572
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Author: someauthor
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Thu Nov 26 11:36:42 2015 +0200
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |     Merge branch 'master' into somebranch
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |   commit 30b8d88c20467ae1e54c84a14abf58d26da454db
| |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  Merge: 47c91c9 7c8749e
| | | |_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|_|/ / / /  Author: someauthor
| | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |   Date:   Thu Nov 26 11:13:38 2015 +0200
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |       Merge branch 'master' into somebranch
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | commit 47c91c9858c47a3f28fb8b6ddb913326b4c4881d
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Author: someauthor
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 25 21:10:51 2015 +0200
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |     cleanup and optimization
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |   commit 2a6e50af017fa5c4b6d1d39bdd1da05d84f39466
| |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \  Merge: a1cb6fa 11287b9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Author: someauthor
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Wed Nov 25 17:36:02 2015 +0200
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |     Merge branch 'master' into somebranch
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |     # Conflicts:
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |     #     css/critical.css
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | commit a1cb6fa96f92473e7793a81d3642e6ac2f60d948
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Author: someauthor
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Date:   Tue Nov 24 21:58:40 2015 +0200
Dec 18, 2015
#23 dborowitz@google.com
Yeah. The issue was that we were walking all *distinct paths* from the open change to the branch tip, rather than walking each commit exactly once. More merge commits == exponential blowup of paths.

In addition to the performance problem this was also a real correctness issue in that changes could appear multiple times if there were multiple paths. It's my bad for writing 600 lines of tests but no tests for merge commits ;)

Anyway, the change was cleaned up for review and should make it into 2.12 final. Thanks again for the report, and for testing out my dumb patches.
Dec 18, 2015
#24 exceptio...@gmail.com
thank you very much for your help!
Dec 19, 2015
Project Member #25 david.pu...@sonymobile.com
https://gerrit-review.googlesource.com/#/c/73484/
Status: Submitted
Labels: -Blocking-2.12 FixedIn-2.12
Dec 19, 2015
Project Member #26 david.pu...@sonymobile.com
(No comment was entered for this change.)
Labels: -Priority-Blocker
Dec 21, 2015
Project Member #27 david.pu...@sonymobile.com
(No comment was entered for this change.)
Status: Released
Sign in to add a comment

Powered by Google Project Hosting