Issue 2160: 500 internal server error on "has:draft" query
Project Member Reported by david.pu...@sonymobile.com, Oct 2, 2013
************************************************************
***** NOTE: THIS BUG TRACKER IS FOR GERRIT CODE REVIEW *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, INTERNAL *****
***** ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.    *****
***** THOSE ISSUE BELONG IN DIFFERENT ISSUE TRACKERS!  *****
************************************************************

Affected Version: 2.7-1686-gdaf8bd4

What steps will reproduce the problem?
1. https://gerrit-review.googlesource.com/#/q/has:draft,n,z
2. https://android-review.googlesource.com/#/q/has:draft,n,z
3.

What is the expected output? What do you see instead?

Expected: returns a page with list of changes having draft comments.

Actual: "Working..." is displayed and then after a while "500 internal server error".  If you F5 the page while it's hung on "Working..." it reports "Server unavailable" briefly.

Unable to reproduce with local build.

Oct 2, 2013
Project Member #1 david.pu...@sonymobile.com
(No comment was entered for this change.)
Blocking: gerrit:2161
Oct 2, 2013
Project Member #2 david.pu...@sonymobile.com
(No comment was entered for this change.)
Blocking: -gerrit:2161
Oct 3, 2013
Project Member #3 david.pu...@sonymobile.com
(No comment was entered for this change.)
Labels: Blocking-2.8
Oct 9, 2013
#4 sop@google.com
I think the has:draft bug is unique to our system. The server stack trace when I reproduced the query shows the database timing out on the user's starred changes:

	at com.google.gerrit.server.IdentifiedUser.getStarredChanges(IdentifiedUser.java:303)
	at com.google.gerrit.server.change.ChangeJson.toChangeInfo(ChangeJson.java:294)
	at com.google.gerrit.server.change.ChangeJson.toChangeInfo(ChangeJson.java:271)
	at com.google.gerrit.server.change.ChangeJson.formatList2(ChangeJson.java:255)

This is... unexpected. I will try to debug our database code and see what might be wrong.
Status: Accepted
Owner: sop@google.com
Labels: Environment-Google
Oct 10, 2013
#5 sop@google.com
Since has:draft is not supported by the secondary indexing system the actual query was rewritten to be something more complicated.

Initial query:

  (draftby:1000000 sortkey_before:z limit:26 visibleto:1000000)

Rewritten:

  (index{p=sortkey_before:z, limit=27} draftby:1000000 limit:26 visibleto:1000000)

The index call is first so it gets pumped. For each open change the draft comments are loaded into the ChangeData and examined for the caller. This is not fast in our system, as the database reads are running sequentially at 20-60ms latencies. gerrit-review has thousands of changes open, so the secondary index paginates through candidates 27 at a time hoping to get a match.

In reality the draftby predicate has a lower cardinality and should be the one driving the AND join with the secondary index. I think the problem here is the cardinality/cost choices configured on the secondary index and the draftby predicates.
Oct 10, 2013
#6 sop@google.com
https://gerrit-review.googlesource.com/50711
Status: ChangeUnderReview
Oct 11, 2013
Project Member #7 david.pu...@sonymobile.com
(No comment was entered for this change.)
Status: Submitted
Labels: -Blocking-2.8 FixedIn-2.8
Dec 9, 2013
Project Member #8 david.pu...@sonymobile.com
(No comment was entered for this change.)
Status: Released