My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 1700: gerrit fails due to duplicated comment messages UUIDs
4 people starred this issue and may be notified of changes. Back to list
Status:  Duplicate
Merged:  issue 851
Owner:  ----
Closed:  May 2013


Sign in to add a comment
 
Reported by andrey.k...@gmail.com, Dec 3, 2012
Hi,

Affected Version: 2.5
backing store: mysql
OS: FreeBSD
Java: Diablo Java(TM) SE Runtime Environment (build 1.6.0_07-b02)

we've experienced pretty weird issues over here which most likely connected with a generation of "non-unique" change messages' UUIDs (field change_messages.uuid in the database). Issue has been observed few times already and usually it can be traced down only via error_log. The consequences of it are undetermined: starting from harmless messages loss (from CI server or so) up to "out of the blue" approvals (for instance today we've got a review where R=+2 was set, however there is no evidence that reviewer ever set it: neither emails, nor messages on the Web interface). The latter case for instance is pretty confusing for the end user since it looks like gerrit code review tool allows to push unapproved commits.

It's pretty hard to obtain reliable STR (because it happens sporadically), however I've managed to reproduce by sending quite a big amount of comments to the single patchset. Here is my STR:

1) create review in the gerrit as usual. (say commitid=deadbeef for reference);

2) try to send _many_ comments for the patchset(commitid=deadbeef). It's pretty hard to do this via web interface (since it may take you dozens or most likely hundreds of attempts before you'll get an error). So below is how I did it (just use CLI tool: gerrit review):

sh -e -c 'n=1; while [ $n -lt 1000 ]; do ssh <LOGIN>@<GERRITHOST> gerrit review -m m$n deadbeef; echo $n; n=$((n+1)); done'

3) observe the output of command above (or tail -f  /path/to/gerrit/review_site/logs/error_log);

4) sooner or later there will be smth. like:
...
500
501
502
fatal: internal server error while approving 242,1
one or more approvals failed; review output above

and log will contain:
===
[2012-12-03 23:20:13,969] ERROR com.google.gerrit.sshd.commands.ReviewCommand : internal error while approving 242,1
com.google.gwtorm.server.OrmException: insert failure on change_messages
        at com.google.gwtorm.schema.sql.SqlDialect.convertError(SqlDialect.java:151)
        at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:448)
        at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:160)
        at com.google.gerrit.server.patch.PublishComments.message(PublishComments.java:288)
        at com.google.gerrit.server.patch.PublishComments.publishApprovals(PublishComments.java:259)
        at com.google.gerrit.server.patch.PublishComments.call(PublishComments.java:140)
        at com.google.gerrit.sshd.commands.ReviewCommand.approveOne(ReviewCommand.java:203)
        at com.google.gerrit.sshd.commands.ReviewCommand.run(ReviewCommand.java:165)
        at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:430)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.BatchUpdateException: Duplicate entry '242-AAAAM3///Z8=' for key 'PRIMARY'
        at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2007)
        at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1443)
        at com.google.gwtorm.schema.sql.SqlDialect.executeBatch(SqlDialect.java:370)
        at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:438)
        at com.google.gwtorm.jdbc.JdbcAccess.insertAsBatch(JdbcAccess.java:202)
        at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:155)
===

There is the following entry in the database:

mysql> SELECT change_id, uuid FROM change_messages WHERE uuid = 'AAAAM3///Z8=';
+-----------+--------------+
| change_id | uuid         |
+-----------+--------------+
|       242 | AAAAM3///z8= |
+-----------+--------------+
1 row in set (0.00 sec)


I'm not quite sure that issue relates to UUIDs generation. Since it is all about mysql and perhaps setting case sensitive collation on the change_messages.uuid field should help to fix it. OTOH it may be worth to improve the method for generating unique uids.


May 10, 2013
#1 docw...@gmail.com
We're having the same problems on 2.5.2 with nearly identical data.

Are the UUIDs supposed to be case sensitive?

I just changed the collation from 'utf8_general_ci' to 'utf8_bin' and it seems to have fixed my problem.

Link to how to change a column collation: http://dev.mysql.com/doc/refman/5.6/en/charset-column.html

Are there any other places that have the same issue?

To be honest, I was surprised that this changed the way a 'WHERE' worked. :-/
May 10, 2013
#2 sop@google.com
Yes the UUIDs are case sensitive.
Having the column case-insensitive in the database is going to lead to failures.
May 12, 2013
#3 docw...@gmail.com
Then may I suggest that Gerrit enforce that by setting the column to ASCII-BIN on mysql?  Are there any other columns with the same problem?
May 17, 2013
#4 sop@google.com
(No comment was entered for this change.)
Status: Duplicate
Mergedinto: 851
Sign in to add a comment

Powered by Google Project Hosting