My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 3730: OrmConcurrencyException: Concurrent modification detected - when executing 'gerrit review' command for different Lables
3 people starred this issue and may be notified of changes. Back to list
Status:  Accepted
Owner:  ----


Sign in to add a comment
 
Reported by vavra.stepan, Dec 21, 2015
*****************************************************************
*****                                                       *****
***** !!!! THIS BUG TRACKER IS FOR GERRIT CODE REVIEW !!!!  *****
*****                                                       *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, CYANOGENMOD,  *****
***** INTERNAL ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.*****
*****                                                       *****
*****   THOSE ISSUES BELONG IN DIFFERENT ISSUE TRACKERS     *****
*****                                                       *****
*****************************************************************

Affected Version: 2.10


What steps will reproduce the problem?
1. Execute 'gerrit review --label A=' and 'gerrit review --label B=' to update 2 different labels simultaneously
2. There's a chance (1 in 10) you would get 'Cannot post review' error



What is the expected output? What do you see instead?
I expect Gerrit to update both labels without concurrency issues.

bash-3.2$ ssh robot@gerrit.foo.int gerrit review --label Verified=-1 --message '"Verified .. -1."' 2463,9 & ssh robot@gerrit.foo.int gerrit review --label Tested=+1 --message '"Tested .. +1."' 2463,9 &
[3] 29329
[4] 29330
bash-3.2$
bash-3.2$ error: Cannot post review
one or more reviews failed; review output above

[3]   Done                    ssh robot@gerrit.foo.int gerrit review --label Verified=-1 --message '"Verified .. -1."' 2463,9
[4]   Exit 1                  ssh robot@gerrit.foo.int gerrit review --label Tested=+1 --message '"Tested .. +1."' 2463,9




Please provide any additional information below.


If I slightly change the code of RevisionApiImpl and debug the exception there, I get an exception:

[2015-12-21 13:41:04,447] WARN  com.google.gerrit.server.api.changes.RevisionApiImpl : Review command ended with an exception
com.google.gwtorm.server.OrmConcurrencyException: Concurrent modification detected
        at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:449)
        at com.google.gwtorm.jdbc.JdbcAccess.updateAsBatch(JdbcAccess.java:270)
        at com.google.gwtorm.jdbc.JdbcAccess.update(JdbcAccess.java:223)
        at com.google.gerrit.server.change.PostReview.apply(PostReview.java:166)
        at com.google.gerrit.server.api.changes.RevisionApiImpl.review(RevisionApiImpl.java:102)
        at com.google.gerrit.sshd.commands.ReviewCommand.applyReview(ReviewCommand.java:194)
        at com.google.gerrit.sshd.commands.ReviewCommand.reviewPatchSet(ReviewCommand.java:242)
        at com.google.gerrit.sshd.commands.ReviewCommand.run(ReviewCommand.java:171)
        at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:368)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Jan 6 (4 days ago)
#2 ryoichi....@sonymobile.com
reproduced with gerrit v2.11.5 in my test environment, too.
Here is the stacktrace on throwing RestApiException("cannot post review",e) 

com.google.gwtorm.server.OrmConcurrencyException: Concurrent modification detected
	at com.google.gwtorm.jdbc.JdbcAccess.execute(JdbcAccess.java:451)
	at com.google.gwtorm.jdbc.JdbcAccess.updateAsBatch(JdbcAccess.java:271)
	at com.google.gwtorm.jdbc.JdbcAccess.update(JdbcAccess.java:224)
	at com.google.gerrit.server.change.PostReview.apply(PostReview.java:175)
	at com.google.gerrit.server.change.PostReview.apply(PostReview.java:136)
	at com.google.gerrit.server.api.changes.RevisionApiImpl.review(RevisionApiImpl.java:141)
	at com.google.gerrit.sshd.commands.ReviewCommand.applyReview(ReviewCommand.java:246)
	at com.google.gerrit.sshd.commands.ReviewCommand.reviewPatchSet(ReviewCommand.java:301)
	at com.google.gerrit.sshd.commands.ReviewCommand.run(ReviewCommand.java:220)
	at com.google.gerrit.sshd.SshCommand$1.run(SshCommand.java:35)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:445)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:379)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	

It seems that regardless of the "--label" option , it can be occured.
Actually it is exposed from gerrit-trigger plugin from jenkins with multiple jobs running.
If it occurs, gerrit user cannot be aware whether jobs are successfly triggerd or not. inconvinience.

sometimes jenkins also suffers from this error:

FYI) exception from jenkins logs:

java.io.IOException: Error during sending command
        at com.sonymobile.tools.gerrit.gerritevents.workers.cmd.AbstractSendCommandJob.sendCommand2(AbstractSendCommandJob.java:118)
        at com.sonymobile.tools.gerrit.gerritevents.workers.cmd.AbstractSendCommandJob.sendCommand(AbstractSendCommandJob.java:79)
        at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.GerritNotifier.buildStarted(GerritNotifier.java:88)
        at com.sonyericsson.hudson.plugins.gerrit.trigger.gerritnotifier.job.ssh.BuildStartedCommandJob.run(BuildStartedCommandJob.java:80)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        
Caused by: com.sonymobile.tools.gerrit.gerritevents.ssh.SshException: error: Cannot post review   ←←←←←←←
one or more reviews failed; review output above (1)                                               ←←←←←←←
        at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionImpl.executeCommand(SshConnectionImpl.java:254)
        at com.sonymobile.tools.gerrit.gerritevents.workers.cmd.AbstractSendCommandJob.sendCommand2(AbstractSendCommandJob.java:116)
        ... 8 more
        

It seems that in my environment, only specific ssh-client may face this symptom, ssh review attempt from A server happens, but B server does not
sshd_log from gerrit:

[2015-11-10 10:39:19,874 +0900] 54c9526f hudson_user a/1002679 LOGIN FROM 10.129.12.218
[2015-11-10 10:39:19,875 +0900] f4c7265b hudson_user a/1002679 LOGIN FROM 10.129.12.218
[2015-11-10 10:39:20,764 +0900] f4c7265b hudson_user a/1002679 gerrit.review.1374323,4.--message.Build Started http://jenkins-host-name/job/test-2/1/ (1/2).--verified.0.--code-review.0 1ms 7ms 1 ←←←←←←← (exited with 1)
[2015-11-10 10:39:20,838 +0900] 54c9526f hudson_user a/1002679 gerrit.review.1374323,4.--message.Build Started http://jenkins-host-name/job/test-1/4/ (2/2).--verified.0.--code-review.0 1ms 83ms 0
[2015-11-10 10:39:21,096 +0900] f4c7265b hudson_user a/1002679 LOGOUT
[2015-11-10 10:39:21,161 +0900] 54c9526f hudson_user a/1002679 LOGOUT



Jan 7 (3 days ago)
Project Member #3 david.pu...@sonymobile.com
(No comment was entered for this change.)
Status: Accepted
Sign in to add a comment

Powered by Google Project Hosting