| Issue 1700: | gerrit fails due to duplicated comment messages UUIDs | |
| 4 people starred this issue and may be notified of changes. | Back to list |
Hi,
Affected Version: 2.5
backing store: mysql
OS: FreeBSD
Java: Diablo Java(TM) SE Runtime Environment (build 1.6.0_07-b02)
we've experienced pretty weird issues over here which most likely connected with a generation of "non-unique" change messages' UUIDs (field change_messages.uuid in the database). Issue has been observed few times already and usually it can be traced down only via error_log. The consequences of it are undetermined: starting from harmless messages loss (from CI server or so) up to "out of the blue" approvals (for instance today we've got a review where R=+2 was set, however there is no evidence that reviewer ever set it: neither emails, nor messages on the Web interface). The latter case for instance is pretty confusing for the end user since it looks like gerrit code review tool allows to push unapproved commits.
It's pretty hard to obtain reliable STR (because it happens sporadically), however I've managed to reproduce by sending quite a big amount of comments to the single patchset. Here is my STR:
1) create review in the gerrit as usual. (say commitid=deadbeef for reference);
2) try to send _many_ comments for the patchset(commitid=deadbeef). It's pretty hard to do this via web interface (since it may take you dozens or most likely hundreds of attempts before you'll get an error). So below is how I did it (just use CLI tool: gerrit review):
sh -e -c 'n=1; while [ $n -lt 1000 ]; do ssh <LOGIN>@<GERRITHOST> gerrit review -m m$n deadbeef; echo $n; n=$((n+1)); done'
3) observe the output of command above (or tail -f /path/to/gerrit/review_site/logs/error_log);
4) sooner or later there will be smth. like:
...
500
501
502
fatal: internal server error while approving 242,1
one or more approvals failed; review output above
and log will contain:
===
[2012-12-03 23:20:13,969] ERROR com.google.gerrit.sshd.commands.ReviewCommand : internal error while approving 242,1
com.google.gwtorm.server.OrmException: insert failure on change_messages
at com.google.gwtorm.schema.sql.SqlDialect.convertError(SqlDialect.java:151)
at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:448)
at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:160)
at com.google.gerrit.server.patch.PublishComments.message(PublishComments.java:288)
at com.google.gerrit.server.patch.PublishComments.publishApprovals(PublishComments.java:259)
at com.google.gerrit.server.patch.PublishComments.call(PublishComments.java:140)
at com.google.gerrit.sshd.commands.ReviewCommand.approveOne(ReviewCommand.java:203)
at com.google.gerrit.sshd.commands.ReviewCommand.run(ReviewCommand.java:165)
at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:430)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.BatchUpdateException: Duplicate entry '242-AAAAM3///Z8=' for key 'PRIMARY'
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:2007)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1443)
at com.google.gwtorm.schema.sql.SqlDialect.executeBatch(SqlDialect.java:370)
at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:438)
at com.google.gwtorm.jdbc.JdbcAccess.insertAsBatch(JdbcAccess.java:202)
at com.google.gwtorm.jdbc.JdbcAccess.insert(JdbcAccess.java:155)
===
There is the following entry in the database:
mysql> SELECT change_id, uuid FROM change_messages WHERE uuid = 'AAAAM3///Z8=';
+-----------+--------------+
| change_id | uuid |
+-----------+--------------+
| 242 | AAAAM3///z8= |
+-----------+--------------+
1 row in set (0.00 sec)
I'm not quite sure that issue relates to UUIDs generation. Since it is all about mysql and perhaps setting case sensitive collation on the change_messages.uuid field should help to fix it. OTOH it may be worth to improve the method for generating unique uids.
May 10, 2013
#1
docw...@gmail.com
May 10, 2013
Yes the UUIDs are case sensitive. Having the column case-insensitive in the database is going to lead to failures.
May 12, 2013
Then may I suggest that Gerrit enforce that by setting the column to ASCII-BIN on mysql? Are there any other columns with the same problem?
May 17, 2013
(No comment was entered for this change.)
Status:
Duplicate
Mergedinto: 851 |
|
| ► Sign in to add a comment |