| Issue 3730: | OrmConcurrencyException: Concurrent modification detected - when executing 'gerrit review' command for different Lables | |
| 3 people starred this issue and may be notified of changes. | Back to list |
*****************************************************************
***** *****
***** !!!! THIS BUG TRACKER IS FOR GERRIT CODE REVIEW !!!! *****
***** *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, CYANOGENMOD, *****
***** INTERNAL ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.*****
***** *****
***** THOSE ISSUES BELONG IN DIFFERENT ISSUE TRACKERS *****
***** *****
*****************************************************************
Affected Version: 2.10
What steps will reproduce the problem?
1. Execute 'gerrit review --label A=' and 'gerrit review --label B=' to update 2 different labels simultaneously
2. There's a chance (1 in 10) you would get 'Cannot post review' error
What is the expected output? What do you see instead?
I expect Gerrit to update both labels without concurrency issues.
bash-3.2$ ssh robot@gerrit.foo.int gerrit review --label Verified=-1 --message '"Verified .. -1."' 2463,9 & ssh robot@gerrit.foo.int gerrit review --label Tested=+1 --message '"Tested .. +1."' 2463,9 &
[3] 29329
[4] 29330
bash-3.2$
bash-3.2$ error: Cannot post review
one or more reviews failed; review output above
[3] Done ssh robot@gerrit.foo.int gerrit review --label Verified=-1 --message '"Verified .. -1."' 2463,9
[4] Exit 1 ssh robot@gerrit.foo.int gerrit review --label Tested=+1 --message '"Tested .. +1."' 2463,9
Please provide any additional information below.
If I slightly change the code of RevisionApiImpl and debug the exception there, I get an exception:
[2015-12-21 13:41:04,447] WARN com.google.gerrit.server.api.changes.RevisionApiImpl : Review command ended with an exception
com.google.gwtorm.server.OrmConcurrencyException: Concurrent modification detected
at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:449)
at com.google.gwtorm.jdbc.JdbcAccess.updateAsBatch(JdbcAccess.java:270)
at com.google.gwtorm.jdbc.JdbcAccess.update(JdbcAccess.java:223)
at com.google.gerrit.server.change.PostReview.apply(PostReview.java:166)
at com.google.gerrit.server.api.changes.RevisionApiImpl.review(RevisionApiImpl.java:102)
at com.google.gerrit.sshd.commands.ReviewCommand.applyReview(ReviewCommand.java:194)
at com.google.gerrit.sshd.commands.ReviewCommand.reviewPatchSet(ReviewCommand.java:242)
at com.google.gerrit.sshd.commands.ReviewCommand.run(ReviewCommand.java:171)
at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:368)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Jan 7 (2 days ago)
(No comment was entered for this change.)
Status:
Accepted
|
|
| ► Sign in to add a comment |
reproduced with gerrit v2.11.5 in my test environment, too. Here is the stacktrace on throwing RestApiException("cannot post review",e) com.google.gwtorm.server.OrmConcurrencyException: Concurrent modification detected at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:451) at com.google.gwtorm.jdbc.JdbcAccess.updateAsBatch(JdbcAccess.java:271) at com.google.gwtorm.jdbc.JdbcAccess.update(JdbcAccess.java:224) at com.google.gerrit.server.change.PostReview.apply(PostReview.java:175) at com.google.gerrit.server.change.PostReview.apply(PostReview.java:136) at com.google.gerrit.server.api.changes.RevisionApiImpl.review(RevisionApiImpl.java:141) at com.google.gerrit.sshd.commands.ReviewCommand.applyReview(ReviewCommand.java:246) at com.google.gerrit.sshd.commands.ReviewCommand.reviewPatchSet(ReviewCommand.java:301) at com.google.gerrit.sshd.commands.ReviewCommand.run(ReviewCommand.java:220) at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35) at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) It seems that regardless of the "--label" option , it can be occured. Actually it is exposed from gerrit-trigger plugin from jenkins with multiple jobs running. If it occurs, gerrit user cannot be aware whether jobs are successfly triggerd or not. inconvinience. sometimes jenkins also suffers from this error: FYI) exception from jenkins logs: java.io.IOException: Error during sending command at com.sonymobile.tools.gerrit.gerritevents.workers.cmd.AbstractSendCommandJob.sendCommand2(AbstractSendCommandJob.java:118) at com.sonymobile.tools.gerrit.gerritevents.workers.cmd.AbstractSendCommandJob.sendCommand(AbstractSendCommandJob.java:79) at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.GerritNotifier.buildStarted(GerritNotifier.java:88) at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.job.ssh.BuildStartedCommandJob.run(BuildStartedCommandJob.java:80) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.sonymobile.tools.gerrit.gerritevents.ssh.SshException: error: Cannot post review ←←←←←←← one or more reviews failed; review output above (1) ←←←←←←← at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionImpl.executeCommand(SshConnectionImpl.java:254) at com.sonymobile.tools.gerrit.gerritevents.workers.cmd.AbstractSendCommandJob.sendCommand2(AbstractSendCommandJob.java:116) ... 8 more It seems that in my environment, only specific ssh-client may face this symptom, ssh review attempt from A server happens, but B server does not sshd_log from gerrit: [2015-11-10 10:39:19,874 +0900] 54c9526f hudson_user a/1002679 LOGIN FROM 10.129.12.218 [2015-11-10 10:39:19,875 +0900] f4c7265b hudson_user a/1002679 LOGIN FROM 10.129.12.218 [2015-11-10 10:39:20,764 +0900] f4c7265b hudson_user a/1002679 gerrit.review.1374323,4.--message.Build Started http://jenkins-host-name/job/test-2/1/ (1/2).--verified.0.--code-review.0 1ms 7ms 1 ←←←←←←← (exited with 1) [2015-11-10 10:39:20,838 +0900] 54c9526f hudson_user a/1002679 gerrit.review.1374323,4.--message.Build Started http://jenkins-host-name/job/test-1/4/ (2/2).--verified.0.--code-review.0 1ms 83ms 0 [2015-11-10 10:39:21,096 +0900] f4c7265b hudson_user a/1002679 LOGOUT [2015-11-10 10:39:21,161 +0900] 54c9526f hudson_user a/1002679 LOGOUT