| Issue 1593: | internal server error | |
| 10 people starred this issue and may be notified of changes. | Back to list |
Affected Version: 2.5-rc0 Pushing a commit (whatever branch, whatever machine) which was working just the day before, now displays an error. $ git push origin HEAD:refs/for/master Counting objects: 39, done. Delta compression using up to 4 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (24/24), 2.25 KiB, done. Total 24 (delta 22), reused 20 (delta 19) remote: Resolving deltas: 100% (22/22) remote: Processing changes: refs: 2, done To ssh://xxxx@example.com:29418/Xxxxx ! [remote rejected] HEAD -> refs/for/master (internal server error) error: failed to push some refs to 'ssh://xxxx@example.com:29418/Xxxxx' The error_log shows: [2012-10-03 09:47:51,013] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in Xxxxx; aborting Tried: - Rebooting the server machine - git fsck all projects on the Gerrit server - Clear the logs - Clear the tmp
Oct 3, 2012
#1
wer...@beroux.com
Oct 3, 2012
Full startup, try push, and shutdown log shows: [2012-10-03 10:00:14,620] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache C:\Gerrit\cache [2012-10-03 10:00:19,121] WARN com.google.gerrit.sshd.SshDaemon : Disabling cipher aes192-cbc: Illegal key size; try installing unlimited cryptography extension [2012-10-03 10:00:19,121] WARN com.google.gerrit.sshd.SshDaemon : Disabling cipher aes256-cbc: Illegal key size; try installing unlimited cryptography extension [2012-10-03 10:00:19,167] WARN com.google.gerrit.httpd.GitWebConfig : gitweb not installed (no \usr\lib\cgi-bin\gitweb.cgi found) [2012-10-03 10:00:20,777] INFO com.google.gerrit.server.plugins.PluginLoader : Loading plugins from C:\Gerrit\plugins [2012-10-03 10:00:20,855] INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD on *:29418 [2012-10-03 10:00:20,855] INFO org.eclipse.jetty.util.log : jetty-7.2.1.v20101111 [2012-10-03 10:00:21,597] INFO org.eclipse.jetty.util.log : Started SelectChannelConnector@0.0.0.0:8080 [2012-10-03 10:00:21,769] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.5-rc0 ready [2012-10-03 10:01:03,225] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in Elysian; aborting [2012-10-03 10:01:08,554] INFO com.google.gerrit.pgm.Daemon : caught shutdown, cleaning up [2012-10-03 10:01:08,554] INFO org.eclipse.jetty.util.log : Graceful shutdown SelectChannelConnector@0.0.0.0:8080 [2012-10-03 10:01:09,206] INFO org.eclipse.jetty.util.log : Graceful shutdown ServletContextHandler@420c93d1@420c93d1/,file:/C:/.gerritcodereview/tmp/gerrit_8066764001741925564_app/gerrit_war/ [2012-10-03 10:01:10,287] INFO com.google.gerrit.sshd.SshDaemon : Stopped Gerrit SSHD
Oct 3, 2012
Deleting another temp folder located at C:\.gerritcodereview solved this issue. However it doesn't explain why it happened.
Oct 5, 2012
This issue happened again. Again in the morning but not the first commit of the day. Actually the second, right after another commit.
Oct 8, 2012
This keeps happening again and again.
Oct 8, 2012
Now it displays in the error log: Only 0 of 2 new change refs created in Foo; aborting.
Oct 8, 2012
Same problem via SSH or HTTPD. Now deleting the temporary folder doesn't seem to fix it.
Oct 8, 2012
Same issue with the snapshot version as of 2010-10-08.
Oct 8, 2012
The only way I could get it to work was by restoring an older H2 database backup.
Oct 8, 2012
Not a fix but it seems this issue is partly dependent to the Change-Id: line generated by commit-msg hook. I got a log like with 1 commit having that change id and not the other: Only 1 of 2 new change refs created in Foo; aborting.
Oct 8, 2012
I've added some more logging on top the current master branch. Once it fails again (hoping it doesn't), I'll report which error ReceiveCommand returned. This should be in the logs anyway.
Oct 12, 2012
CreateRequest retured error LOCK_FAILURE. Cannot do much more. The last weird thing is that sometimes after a service restart and/or a while it works again. Still this issue is new in and wasn't there before.
Oct 12, 2012
I found this error is the error log a few minutes before: [2012-10-12 11:31:31,888] ERROR com.google.gerrit.server.git.ChangeMergeQueue : Merge attempt for Foo,refs/heads/master failed com.google.gerrit.server.git.MergeException: Cannot merge f0fd5a839d3e9c8b341cb7bf5434d361483ff2e8 at com.google.gerrit.server.git.MergeOp.cherryPickChanges(MergeOp.java:794) at com.google.gerrit.server.git.MergeOp.preMerge(MergeOp.java:309) at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:281) at com.google.gerrit.server.git.ChangeMergeQueue.mergeImpl(ChangeMergeQueue.java:196) at com.google.gerrit.server.git.ChangeMergeQueue.merge(ChangeMergeQueue.java:116) at com.google.gerrit.server.changedetail.Submit.call(Submit.java:202) at com.google.gerrit.httpd.rpc.changedetail.SubmitAction.call(SubmitAction.java:59) at com.google.gerrit.httpd.rpc.changedetail.SubmitAction.call(SubmitAction.java:34) at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65) at com.google.gerrit.httpd.rpc.changedetail.ChangeManageServiceImpl.submit(ChangeManageServiceImpl.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91) at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:379) at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:265) at com.google.gerrit.httpd.rpc.GerritJsonServlet.service(GerritJsonServlet.java:118) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:921) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:856) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:59) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1069) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Failed to create ref refs/changes/65/465/2 in Foo: LOCK_FAILURE at com.google.gerrit.server.git.MergeOp.writeCherryPickCommit(MergeOp.java:984) at com.google.gerrit.server.git.MergeOp.cherryPickChanges(MergeOp.java:764) ... 67 more
Oct 12, 2012
Possibly fixed by c0083226ab935ee982e3c1fa65121e2320e40017 and 204669c2e14d4b1d08e704d843e40703831c645d.
Oct 12, 2012
I think you are seeing the change_id sequence reset to too small of a value and the server is repeating change identifiers. Did you copy the refs/changes/ names from one server to another in this repository? If so that will cause periodic failures when a change id is assigned that was already used.
Oct 12, 2012
Still having that issue.
Oct 13, 2012
I'm not sure where they are. The only thing that could have gone wrong is an upgrade rollback. Instead of using an older backup version of Gerrit, we: 1. Reinstalled an older Gerrit (fresh install) 2. Copied the latest Git repository (which had a few changes waiting but were thus lost) 3. Setup again the users and SSH I'm under the impression those IDs are in the H2 database. If that's the case, it shouldn't have any idea about the old ones. Another thing we do quiet a lot, are Cherry-Picks instead of merges (for some older branches).
Oct 15, 2012
Is there some H2 query I could execute or some operation I could do to check that (to be sure).
Oct 15, 2012
But doesn't this explain there error? You reinstalled Gerrit freshly, which means that you start with a new change_id sequence. At the same time you copied the existing git repositories into this new Gerrit server. These git repositories already contain change refs from the old server which may now be conflicting with the change_ids generated by your new change_id sequence.
Oct 15, 2012
In other words, I should 'import' all the previous Change-Id in the H2 database or remove them from the log?
Oct 15, 2012
Why not simply set the change_id sequence to a high enough value so that the new change_id's are not conflicting with the old ones?
Oct 16, 2012
How can I do that? Looking at PATCH_SETS it contains all used Change-Id: in the H2 database. I found also a way to increment the changeid number (not the full Change-Id:): SELECT MAX(change_id) + 1 FROM changes; INSERT INTO change_id VALUES (...that value...); I doubt however this would fix as our current changeid numbers as higher than those before the Gerrit reinstall.
Oct 17, 2012
Anything I can do?
Oct 23, 2012
Running RC1 now. Let's wait and see...
Oct 25, 2012
Happened again today.
Oct 26, 2012
The error changed a bit: [2012-10-26 10:03:00,230] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 2 of 0 new change refs created in Foo; aborting [2012-10-26 10:03:07,936] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 0 new change refs created in Foo; aborting [2012-10-26 10:03:12,226] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 0 new change refs created in Foo; aborting ... Using a new Change-Id: will result in: [2012-10-26 10:34:44,506] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in Foo; aborting Using a new Change-Id: in a commit 3 commits ago (but still under review): [2012-10-26 10:36:48,944] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 3 new change refs created in Foo; aborting In the gerrit_service.log (Windows-specific) I get (possibly unrelated): Fri Oct 26 10:05:23 2012 Continuing after GetQueuedCompletionStatus() ;n = 6;key = 0;po = 00000FB0 Sometimes the only way to fix it is by repeating: 1. Delete cache and restart service 2. Amend and or remove Change-Id: 3. push
Nov 6, 2012
I tried also flushing the cache via Gerrit CLI but no use. Could someone explain to me the cause of this issue and possibly hints on how to solve this? We have it like twice or three times a week. It's nearly a showstopper.
Nov 14, 2012
Possibly 'gerrit flush-caches' also fixes the issue temporarily. gerrit receive-pack shows: 00a0e616a1bc566be4e79125b6d12157dba9c9dd975a refs/cache-automerge/0d/48bfe98d24ca1cfaca4d7cba0c001233c4f5ba^@ side-band-64k delete-refs report-status ofs-delta 006c70adddb3d926351c778766fe6f6c94df385e9c6d refs/cache-automerge/12/4fe86e0456fce7e9670d4e45acdeadb25a9d4c ... 003ff97dae5b9f8ac8e96ab8362bb6f6bcdc9d7d87cb refs/heads/master ... 00427b2482883a2518e94d7637a126fd13e4df9799e0 refs/tags/v1.8.0.938 004238d0c71acea16390424fcef680438459db2a60d3 refs/tags/v1.8.0.951 004502bf646877b68bf99ad7a9064c15bc7b12d6ff13 refs/tags/v1.8.0.826 00422514ac5a5b75129bedf9e0b8d413e9ec26d39c21 refs/tags/v2.0.0.786 00330c71f1794d814d36f33369296ef14ca0fdb707b7 .have 00331343affe4359823f075e7eaf86a5c966f6400d16 .have 00331a4f35c8da2891b960d69ad3bbf03c6fb66d6490 .have 0033cc7e22f863f086b6eafeaa01828c8181d3ec6bfa .have 00334a9609dfb9568c96101fef9b5e57368cd70df944 .have 0033d782d22c4a249ce3164f3a2b1915053c7d4e2194 .have 0033190392a7b33c19c2a5137b00c8b14ac25679e84b .have 0033c7b9f14385e675272d8db3cc960758627bfa647b .have 0033a780a56108724060e8736f81dbee0b75e71770a1 .have 0033eb348e29d5a8708fdc88e5c61925d09c28ddc02d .have 0033f60c03e87a719060b05cdfde58851040c4d9a2cf .have 0000 And hangs there until some keys are pressed and then displays 'fatal: internal server error' which may make sense (I haven't check how to use this command). May be it can give some insight.
Nov 22, 2012
Deleting the cache or flushing it seems only to help in some cases. So far the best way to get it back working seems to be only the last step: Repeat trying to submit a commit until it succeeds. Then other commits can be submitted again. Below is the tiny script that does just that.
touch _dummy && git add _dummy && git ci -m dummy
while [[ true ]]
do
git push origin HEAD:refs/drafts/master
[ $? -eq 0 ] && break
done
Nov 27, 2012
I'm having this problem after patset-created hook failed: [2012-11-27 08:56:01,494] WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning [2012-11-27 08:56:01,512] WARN com.google.gerrit.server.git.AsyncReceiveCommits : Error in ReceiveCommits java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at com.google.gerrit.server.git.WorkQueue$Task.get(WorkQueue.java:315) at com.google.gerrit.server.git.MultiProgressMonitor.waitFor(MultiProgressMonitor.java:234) at com.google.gerrit.server.git.AsyncReceiveCommits.onPreReceive(AsyncReceiveCommits.java:163) at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:179) at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:142) at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:98) at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:106) at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:34) at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:72) at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:430) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.NullPointerException at com.google.gerrit.server.ApprovalsUtil.copyVetosToLatestPatchSet(ApprovalsUtil.java:95) at com.google.gerrit.server.git.ReceiveCommits$ReplaceRequest.insertPatchSet(ReceiveCommits.java:1597) at com.google.gerrit.server.git.ReceiveCommits.insertChangesAndPatchSets(ReceiveCommits.java:642) at com.google.gerrit.server.git.ReceiveCommits.processCommands(ReceiveCommits.java:498) at com.google.gerrit.server.git.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:90) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:196) at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:174) at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator$1.call(ThreadLocalRequestScopePropagator.java:51) at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:81) at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:113) ... 9 more [2012-11-27 08:56:01,903] INFO com.google.gerrit.common.ChangeHookRunner : hook[patchset-created.bat] output: Traceback (most recent call last): ... [2012-11-27 08:58:13,539] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 0 new change refs created in Project; aborting Now I can't upload new patch sets for this commit or its parents. Tried to manually remove these patch sets from the database (including corresponding git refs), but it didn't help.
Jan 8, 2013
Most weirdly it seems this error doesn't occur anymore for us.
Jan 8, 2013
Should the ticket be closed then?
Jan 9, 2013
Possibly. If the database is more to Git I guess this problem will soon be non-reproducible and then it should be closed. If not, ideally it should remain in background for reference until someone finds a fix. Personally it's closed and next time I'll back up more but I'm still not confident in restoring a database without the exact Gerrit database backup.
Jan 9, 2013
I got it today on gerrit v2.5 running linux with postgres db. Not sure if this has any significance but the commit that fails to be pushed is third patchset to a change. Author of commit changed from patchset one to patchset two (amend with --reset-author). But author of patchset three is the same as patchset two. Workaround that worked for me was to upload the commit as new chage. That is unfortunate a bit due to lost review history - because the original change had to be abandoned. This is what gerrit log says: [2013-01-09 15:02:09,564] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 0 new change refs created in rpox; aborting On client side I get 'internal server error'.
Jan 9, 2013
I have the feeling that different problems get mixed in this issue. 'Internal Server Error' is a very generic error message that is displayed for all kind of errors. I propose to open new issues for each problem and then describe in these issues how to reproduce the problem. I'm closing this issue, but of course it stays for further reference and can also be reopened if needed.
Status:
CannotReproduce
Jan 9, 2013
I moved my bug report here: https://code.google.com/p/gerrit/issues/detail?id=1751.
Mar 21, 2013
I just ran into this issue as well. I moved from an H2 database to PostgreSQL. My fix was looking at the following line from the H2 sql export: CREATE SEQUENCE PUBLIC.CHANGE_ID START WITH 48; Noting the number associated (ie 48) and running the following on the reviewdb: ALTER SEQUENCE CHANGE_ID RESTART WITH 48;
Apr 26, 2013
I've just had a user of my gerrit 2.5.2 test install report this error which, prima facie, look similar to reports in this discussion: snip--- fb@a11202-debian:~/code/internal-gerrit-test/internal/arndale$ git push origin HEAD:refs/for/3.4-android Counting objects: 13, done. Delta compression using up to 4 threads. Compressing objects: 100% (7/7), done. Writing objects: 100% (7/7), 938 bytes, done. Total 7 (delta 6), reused 0 (delta 0) remote: Resolving deltas: 100% (6/6) remote: Processing changes: refs: 2, done To ssh://junin.wolfsonmicro.main:29418/kernel/arndale ! [remote rejected] HEAD -> refs/for/3.4-android (internal server error) error: failed to push some refs to 'ssh://junin.wolfsonmicro.main:29418/kernel/arndale' snip--- The server log reported: [2013-04-25 13:37:53,558] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 0 new change refs created in kernel/arndale; aborting But nothing else. I would appreciate knowing how to fix this. I use an external postgresql server back end.
Jun 14, 2013
I have been experiencing this problem for many months now, and have finally discovered a solution. On my Gerrit server, inside the directory that houses all my repositories (/git for me), I browsed to: [ /git/REPO.git/refs/changes/ ] and noticed there was a single file with some irrelevant text inside of it. Gerrit is expecting a number of directories inside the /refs/changes/ directly, so I was a little perplexed as to why it was there. However, deleting that file completely resolved the problem - Gerrit is now able to create patchsets with no problem!
Nov 22, 2013
We are seeing this after upgrading to 2.7 (from 2.4.something). We're using MySQL as the database: $ git push origin HEAD:refs/for/master Counting objects: 29, done. Delta compression using up to 8 threads. Compressing objects: 100% (15/15), done. Writing objects: 100% (16/16), 7.76 KiB, done. Total 16 (delta 12), reused 0 (delta 0) remote: Resolving deltas: 100% (12/12) remote: Processing changes: refs: 2, done To ssh://source.myhost.com/myproject.git ! [remote rejected] HEAD -> refs/for/master (internal server error) error: failed to push some refs to 'ssh://source.myhost.com/myproject.git' and the log shows: [2013-11-22 11:42:49,879] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in myproject; aborting I'm in the process of testing various solutions proposed here but if anyone has any debugging advice I'd appreciate it.
Nov 22, 2013
Addition to my previous comment: Flushing all caches seems to have cleared this for us for now. There's no indication in the logs of a problem other than the error message given.
Nov 26, 2013
We're also seeing this issue. It occurs with all of our systems (Mac and PC).
After a push to refs/for/master, the gerrit error log shows:
[2013-11-26 18:02:05,533] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in xxxxxxx; aborting
[2013-11-26 18:02:05,892] WARN org.apache.sshd.server.session.ServerSession : Exception caught
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
at sun.nio.ch.IOUtil.read(IOUtil.java:224)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:280)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:44)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:677)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1111)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
This started occurring with Gerrit 2.7, and is still occurring with Gerrit 2.8rc0. More on our configuration:
* Gerrit's database is backed by Postgresql (we've attempted bumping the sequence number to max change_id +1).
* We are using LDAP authentication.
* We have also attempted flushing the cache.
Our maximum change ID in our DB in the changes table is 91, but the maximum sequence number for 'change_id' is now 111.
Nov 26, 2013
Can this be re-opened, please? We are seeing this frequently, and the only way to clear the condition and get things moving again seems to be to restart gerrit. We are running 2.7, using MySQL as the back end, running from the embedded Jetty. Flushing the cache *sometimes* helps, but the only consistent fix so far has been to restart the server.
Nov 26, 2013
I believe this issue is too much messed up with mixing different problems. Better open a new issue.
Nov 27, 2013
Thanks, Edwin, I wasn't sure what the protocol was. I will open a new issue.
Mar 18, 2014
Hallo Guys! I got the problem with internal error exactly when I moved all git repos into new gerrit installation. Could not push. Looks like comment #41 helped to solve this particular issue in my case. Thanks a lot
May 9, 2014
I have the same problem. Upgrade 2.7 -> 2.9-rc0. [2014-05-09 15:41:16,818] WARN com.google.gerrit.sshd.GerritServerSession : Exception caught java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(Unknown Source) at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) at sun.nio.ch.IOUtil.read(Unknown Source) at sun.nio.ch.SocketChannelImpl.read(Unknown Source) at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:273) at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:44) at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:690) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664) at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
Jul 3, 2014
TESTING
Dec 3, 2014
Hi All, We have the LOCK_FAILURE problem here on gerrit 2.9.2 (Just installed) I don't know if they are related, but when ref-update hook exit with non-zero value, the error log for LOCK_FAILURE can be found just few lines below ===================== [2014-12-03 15:30:14,204] INFO com.google.gerrit.common.ChangeHookRunner : hook[ref-update] exitValue:2 [2014-12-03 15:30:14,206] INFO com.google.gerrit.common.ChangeHookRunner : hook[ref-update] output: [^[[31;1mPush Rejected^[[0m] ^[[33;1mRead your mail for more information: ^[[0m^[[34;1muser@mycompany.com^[[0m [^[[31;1merr: 2^[[0m] [2014-12-03 15:34:09,275] INFO com.google.gerrit.common.ChangeHookRunner : hook[ref-update] exitValue:2 [2014-12-03 15:34:09,276] INFO com.google.gerrit.common.ChangeHookRunner : hook[ref-update] output: [^[[31;1mPush Rejected^[[0m] ^[[33;1mRead your mail for more information: ^[[0m^[[34;1muser@mycompany.com^[[0m [^[[31;1merr: 2^[[0m] [2014-12-03 15:39:01,023] ERROR com.google.gerrit.server.git.ReceiveCommits : Command CREATE on kernel/msm-3.10:refs/changes/94/94/1 not completed: LOCK_FAILURE [2014-12-03 15:39:01,023] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in kernel/msm-3.10; aborting [2014-12-03 15:39:10,379] ERROR com.google.gerrit.server.git.ReceiveCommits : Command CREATE on kernel/msm-3.10:refs/changes/95/95/1 not completed: LOCK_FAILURE [2014-12-03 15:39:10,379] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in kernel/msm-3.10; aborting =========================
Dec 11, 2014
Update my status and hope the below will help: Finally I found that I forget to remove all references under refs/changes/ before reinstalling the gerrit 2.9.2. Now I had removed all the refs/changes/* and problem is resolved.
Jun 24, 2015
I have the similar problem, but have something different. I installed a plugin "reviewer", the problem happened when I added a default reviewer by this plugin. The problem is gone when I removed this plugin.
Jun 24, 2015
> I have the similar problem, but have something different. I installed a plugin > "reviewer", the problem happened when I added a default reviewer by this plugin. > The problem is gone when I removed this plugin. Verify that the version of the "reviewer" plugin is compatible with your Gerrit server version.
Aug 13, 2015
I have got a similar error log:
```
[2015-08-13 17:13:46,843] WARN com.google.gerrit.sshd.GerritServerSession : Exception caught
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:273)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:44)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:690)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
[2015-08-13 17:21:15,646] WARN com.google.gerrit.sshd.GerritServerSession : Exception caught
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:273)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:44)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:690)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
[2015-08-13 17:22:18,621] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in apps/LenovoSystemUI; aborting
[2015-08-13 17:22:48,933] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in apps/LenovoSystemUI; aborting
[2015-08-13 17:23:45,576] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in apps/LenovoSystemUI; aborting
[2015-08-13 17:25:13,714] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in apps/LenovoSystemUI; aborting
[2015-08-13 17:28:10,641] ERROR com.google.gerrit.server.git.ReceiveCommits : Only 0 of 1 new change refs created in apps/LenovoSystemUI; aborting
```
|
|
| ► Sign in to add a comment |