Issue 1870: Upgrade from 2.1.8 to 2.6.0-rc0 fails
Status:  Submitted
Owner: ----
Closed:  May 2013
Reported by jos...@whamcloud.com, Apr 17, 2013
Affected Version: 2.6.0-rc0

What steps will reproduce the problem?
1. Have an existing 2.1.8 install.  Convert all tables to InnoDB, and all tables to UTF-8. Change connect string to use utf-8 encoding and unicode=true.
2. Run:

sudo -u gerrit java -jar gerrit-2.6-rc0.war init --batch -d /opt/local/gerrit/review.whamcloud.com

I expect it to start upgrading. Instead I see:

fatal: DbInjector failed
fatal: null

Java is java-1.7.0-openjdk-1.7.0.9-2.3.3.2.el6_3.x86_64 on CentOS 6.3.
Apr 17, 2013
#1 jos...@whamcloud.com
An upgrade process from this 2.1.8 setup to 2.5.2 works without issue.
Apr 24, 2013
#2 jos...@whamcloud.com
This bug still applies:

sudo -u gerrit java -jar gerrit-2.6-rc1.war init --batch -d /opt/local/gerrit/review.whamcloud.com
fatal: DbInjector failed
fatal: null

Apr 24, 2013
#3 sop@google.com
Adding --show-stack-trace will get more data back from init that can help narrow searching in the code.
Labels: Blocking-2.6
Apr 24, 2013
#4 jos...@whamcloud.com
As requested. :)

# sudo -u gerrit java -jar gerrit-2.6-rc1.war init --show-stack-trace --batch -d /opt/local/gerrit/review.whamcloud.com
com.google.gerrit.pgm.util.Die: null
	at com.google.gerrit.pgm.util.AbstractProgram.die(AbstractProgram.java:95)
	at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:221)
	at com.google.gerrit.pgm.Init.createSysInjector(Init.java:302)
	at com.google.gerrit.pgm.Init.createSiteRun(Init.java:290)
	at com.google.gerrit.pgm.Init.run(Init.java:78)
	at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:168)
	at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:92)
	at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
	at Main.main(Main.java:25)
Caused by: java.lang.RuntimeException: DbInjector failed
	... 13 more
Caused by: com.google.inject.CreationException: Guice creation errors:

1) Error in custom provider, java.lang.NullPointerException
  while locating com.google.gerrit.server.schema.ReviewDbDatabaseProvider
  at com.google.gerrit.server.schema.DatabaseModule.configure(DatabaseModule.java:31)
  while locating com.google.gwtorm.jdbc.Database<com.google.gerrit.reviewdb.server.ReviewDb>
  at com.google.gerrit.server.schema.DatabaseModule.configure(DatabaseModule.java:29)
  while locating com.google.gwtorm.server.SchemaFactory<com.google.gerrit.reviewdb.server.ReviewDb>
Caused by: java.lang.NullPointerException
	at com.google.gwtorm.jdbc.Database.<init>(Database.java:78)
	at com.google.gerrit.server.schema.ReviewDbDatabaseProvider.get(ReviewDbDatabaseProvider.java:39)
	at com.google.gerrit.server.schema.ReviewDbDatabaseProvider.get(ReviewDbDatabaseProvider.java:28)
	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.Scopes$1$1.get(Scopes.java:65)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.Scopes$1$1.get(Scopes.java:65)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
	at com.google.inject.Guice.createInjector(Guice.java:95)
	at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:192)
	at com.google.gerrit.pgm.Init.createSysInjector(Init.java:302)
	at com.google.gerrit.pgm.Init.createSiteRun(Init.java:290)
	at com.google.gerrit.pgm.Init.run(Init.java:78)
	at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:168)
	at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:92)
	at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
	at Main.main(Main.java:25)

2) Error in custom provider, java.lang.NullPointerException
  while locating com.google.gerrit.server.schema.ReviewDbDatabaseProvider
  at com.google.gerrit.server.schema.DatabaseModule.configure(DatabaseModule.java:31)
  while locating com.google.gwtorm.jdbc.Database<com.google.gerrit.reviewdb.server.ReviewDb>
Caused by: java.lang.NullPointerException
	at com.google.gwtorm.jdbc.Database.<init>(Database.java:78)
	at com.google.gerrit.server.schema.ReviewDbDatabaseProvider.get(ReviewDbDatabaseProvider.java:39)
	at com.google.gerrit.server.schema.ReviewDbDatabaseProvider.get(ReviewDbDatabaseProvider.java:28)
	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.Scopes$1$1.get(Scopes.java:65)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:204)
	at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:198)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
	at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:198)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:179)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
	at com.google.inject.Guice.createInjector(Guice.java:95)
	at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:192)
	at com.google.gerrit.pgm.Init.createSysInjector(Init.java:302)
	at com.google.gerrit.pgm.Init.createSiteRun(Init.java:290)
	at com.google.gerrit.pgm.Init.run(Init.java:78)
	at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:168)
	at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:92)
	at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
	at Main.main(Main.java:25)

2 errors
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:183)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
	at com.google.inject.Guice.createInjector(Guice.java:95)
	at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:192)
	... 12 more

Apr 25, 2013
#5 sop@google.com
Getting NPE here means the DataSource is broken.

What is in your [database] block of gerrit.config?
Status: AwaitingInformation
Apr 25, 2013
#6 sop@google.com
Any issue must be a MySQL specific problem; a 2.1.8 -> 2.6-rc1+ using H2 is fine.

I suspect the URL you configured is not recognized by the MySQL driver being used by Gerrit. A bug in gwtorm means we don't throw an exception and instead return null, which can cause the crash we see in this bug report.

Does $site_path/lib have two copies of the MySQL driver?
Apr 25, 2013
#7 sop@google.com
Even with https://gerrit-review.googlesource.com/45040 we have a problem
upgrading the MySQL database driver. The running init program has the old
library already in the classpath and will try to use that instead of the
newer driver that was just downloaded.
Apr 25, 2013
#8 jos...@whamcloud.com
So, here is what I have.

Upgrade executed via:

sudo -u gerrit java -jar <NAME>.war init --batch -d /opt/local/gerrit/review.whamcloud.com

Before upgrade is run:
[database]
	url = jdbc:mysql://localhost:3306/reviewdb2?user=gerrit2&password=PW&useUnicode=true&characterEncoding=utf8


After I run a 2.5.2 upgrade:

[database]
	type = H2
	database = db/ReviewDB
	url = jdbc:mysql://localhost:3306/reviewdb2?user=gerrit2&password=PW&useUnicode=true&characterEncoding=utf8

After an attempted 2.6-rc1 upgrade

[database]
	type = h2
	database = db/ReviewDB
	url = jdbc:mysql://localhost:3306/reviewdb2?user=gerrit2&password=PW2&useUnicode=true&characterEncoding=utf8

# ls review.whamcloud.com/lib/
bcprov-jdk16-144.jar  mysql-connector-java-5.1.10.jar

A 'database' and 'type' is added in both cases, but that didn't seem to bother the 2.5.2 upgrade. Only one MySQL driver.  I upgraded to Connector/J 5.1.24, and got the same error.
Apr 25, 2013
#9 sop@google.com
https://gerrit-review.googlesource.com/45033 fixes init to set the type to be jdbc when there is no type named in the database block.

If you did this on your 2.1.8 server:

  git config --file gerrit.config database.type jdbc

I think the 2.1.8->2.6 upgrade would work correctly, so 45033 tries to do that.
Apr 25, 2013
#10 sop@google.com
(No comment was entered for this change.)
Status: ChangeUnderReview
Apr 25, 2013
#11 jos...@whamcloud.com
Well, that gets me farther (?)

# sudo -u gerrit java -jar gerrit-2.6-rc1.war init --batch -d /opt/local/gerrit/review.whamcloud.com
Exception in thread "main" com.google.inject.ProvisionException: Guice provision errors:

1) Error injecting constructor, java.lang.IllegalArgumentException: No database.driver configured
  at com.google.gerrit.server.schema.JDBC.<init>(JDBC.java:29)
  while locating com.google.gerrit.server.schema.JDBC
  while locating com.google.gerrit.server.schema.DataSourceType annotated with @com.google.inject.name.Named(value=jdbc)

1 error
	at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1009)
	at com.google.gerrit.pgm.util.SiteProgram.createDbInjector(SiteProgram.java:178)
	at com.google.gerrit.pgm.Init.createSysInjector(Init.java:302)
	at com.google.gerrit.pgm.Init.createSiteRun(Init.java:290)
	at com.google.gerrit.pgm.Init.run(Init.java:78)
	at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:168)
	at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:92)
	at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
	at Main.main(Main.java:25)
Caused by: java.lang.IllegalArgumentException: No database.driver configured
	at com.google.gerrit.server.config.ConfigUtil.getRequired(ConfigUtil.java:292)
	at com.google.gerrit.server.schema.JDBC.<init>(JDBC.java:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at com.google.inject.internal.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:85)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54)
	at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
	at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)

# ls review.whamcloud.com/lib/
bcprov-jdk16-144.jar  mysql-connector-java-5.1.10.jar

Were there other changes I was supposed to make?
Apr 26, 2013
#12 sop@google.com
The driver class also needs to be set:

  git config --file=gerrit.config database.driver com.mysql.jdbc.Driver

Change https://gerrit-review.googlesource.com/45090 was created to
perform this automatically on upgrades.
Apr 26, 2013
#13 jos...@whamcloud.com
Hmm...seems a lot of things have changed between versions.  After I run my upgrade script (before running the gerrit upgrade), I have this:

[database]
        url = jdbc:mysql://localhost:3306/reviewdb2?user=gerrit2&password=PW&useUnicode=true&characterEncoding=utf8
        type = jdbc
        driver = com.mysql.jdbc.Driver

Then I run the upgrade:

[root@vcs-staging gerrit]# sudo -u gerrit java -jar gerrit-2.6-rc1.war init --batch -d /opt/local/gerrit/review.whamcloud.com
fatal: DbInjector failed
fatal: Unable to determine SqlDialect
fatal:   caused by java.sql.SQLException: Access denied for user 'gerrit'@'localhost' (using password: YES)

And My config file looks like this:

[database]
        username = gerrit
        url = jdbc:mysql://localhost:3306/reviewdb2?user=gerrit2&password=PW&useUnicode=true&characterEncoding=utf8
        type = jdbc
        driver = com.mysql.jdbc.Driver

Why is it assuming my user name is gerrit? Granted, I'm running (via sudo) as the gerrit user, but it has nothing to do with my MySQL user name.  My MySQL user name is in the url. Does gerrit even use the 'url' setting any more? Should I break out all the settings into their own lines?
Apr 29, 2013
#14 jos...@whamcloud.com
I also discovered that when I put the 'type' and 'driver' fields in the config file before a 2.5.2 upgrade, I also get the "username = gerrit" added when I try to run the upgrade.
May 2, 2013
#15 jos...@whamcloud.com
If I add

username = gerrit2

to the database stanza, the upgrade succeeds.
May 8, 2013
#16 pedah...@gmail.com
Still an issue with gerrit-2.6-rc2

sudo -u gerrit java -jar gerrit-2.6-rc2.war init --batch -d /opt/local/gerrit/review.whamcloud.com
fatal: DbInjector failed
fatal: Unable to determine SqlDialect
fatal:   caused by java.sql.SQLException: Access denied for user 'gerrit'@'10.18.13.134' (using password: NO)

Seems the DB and driver are added, but the username is still assumed to be gerrit, when it is gerrit2 in the JDB URL.

Here is what it starts with:

        url = jdbc:mysql://prd1whamc2db01:3306/reviewdb2?user=gerrit2&password=PW&useUnicode=true&characterEncoding=utf8

And this is what it adds:

        type = jdbc
        driver = com.mysql.jdbc.Driver
        username = gerrit

The type and driver are right, but not the user name.
May 13, 2013
#17 pedah...@gmail.com
So, upgrading from 2.1.8 to 2.5.2 adds this to the DB stanza:

        type = H2
        database = db/ReviewDB

But it leaves the url alone, so it still works properly.  This means that upgrading from 2.5.2 to 2.6.x is going to have the same problem, as the values for db, type, and user are not in the stanza.
May 13, 2013
#18 sop@google.com
https://gerrit-review.googlesource.com/45658 corrects an issue seen in comment #16 where init added a username property that confused the connection.
May 13, 2013
#19 pedah...@gmail.com
Cool. I do not have a build environment set up. Will this be cut as -rc3?

But even if it is, I'm afraid we won't be able to upgrade to 2.6 due to the issue I raised today on the mailing list about the mod_rewrite NoDecode issue.
May 13, 2013
#20 sop@google.com
Yes, there will be an rc3 that includes this change.
May 14, 2013
Project Member #21 edwin.ke...@gmail.com
(No comment was entered for this change.)
Status: Submitted
Labels: -Blocking-2.6 FixedIn-2.6