expired Google G2 cert in urlfetch_cacerts.txt causes exception "Invalid and/or missing SSL certificate" [35901176]

Obsolete

Bug

More Info Required

Status Update

No update yet.

Description

de...@nicemice.net

created issue #1

Jan 2, 2017 06:05AM

The problem affects dev_appserver, not apps deployed in the cloud.

It affects (at least) App Engine Python SDK versions 1.9.40 and 1.9.49
on Linux with Python 2.7.11 and libssl 1.0.2j.

My app does this:

from googleapiclient import discovery
import httplib2
credentials = ...
script_service = discovery.build(
'script', 'v1', credentials=credentials,
http=httplib2.Http(timeout=15))

This worked until 2017-Jan-01, when two "Google Internet Authority G2"
certs (serial=023A76, serial=023A83) in lib/cacerts/urlfetch_cacerts.txt
expired, at which point discovery.build() started raising this exception:

HTTPException: Invalid and/or missing SSL certificate for URL:

https://www.googleapis.com/discovery/v1/apis/script/v1/rest?userIp=127.0.0.1

That exception is a few levels of indirection away from the original
exception, which I tracked down by adding logging statements
in the SDK. The original exception is in wrap_socket() in
lib/fancy_urllib/fancy_urllib/__init__.py. It raises an ssl.SSLError
with library=SSL reason=CERTIFICATE_VERIFY_FAILED.

This happens in both SDK 1.9.40 (where urlfetch_cacerts.txt lacks any
newer version of the cert) and in SDK 1.9.49 (where urlfetch_cacerts.txt
contains the two expired versions and also the current version,
serial=023A92, which is also sent by

www.googleapis.com in the SSL
handshake).

If the expired certs are simply removed from urlfetch_cacerts.txt,
everything works once again. That's true regardless of whether
urlfetch_cacerts.txt contains the new cert (like in 1.9.49) or not (like
in 1.9.40).

If a certificate validates with a given CA file, adding more certs
(expired or not) to the file should not invalidate it. There must be a
bug either in the way the SDK is using some Python libraries, or in one
of the Python libraries, or in libssl.

I don't know what is essential difference between dev_appserver and the
real App Engine that makes this bug affect one and not the other. That
might be a clue.

While that bug is being worked on, removing expired certs from
urlfetch_cacerts.txt is a workaround.

Also, I'm not sure whether this certificate validation
is supposed to be happening at all.

Issue 12129

(

https://code.google.com/p/googleappengine/issues/detail?id=12129)
suggests that maybe it's not, in which case fixing that bug would also
work around this one.

Comments

jm...@google.com <jm...@google.com> #2Jan 2, 2017 05:39PM

Accepted by jm...@google.com.

Our engineering team is aware of this issue and they are working on the fix. Note there is currently no ETA for its release.

kk...@google.com <kk...@google.com> #4Jan 11, 2017 10:40PM

As a temporary workaround, you can replace the expired urlfetch_cacerts.txt with

https://curl.haxx.se/ca/cacert.pem

On linux you can use the following command:
wget

https://curl.haxx.se/ca/cacert.pem -O {Root of Python SDK}/lib/cacerts/urlfetch_cacerts.txt

We apologize for the inconvenience caused.

mi...@gmail.com <mi...@gmail.com> #5Jan 24, 2017 11:58PM

For those on windows, below is the default location of the file that needs to be replaced:

C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\cacerts\urlfetch_cacerts.txt

I didn't need to restart my dev server to get it working.

[Deleted User] <[Deleted User]> #6Feb 9, 2017 05:20PM

[Comment deleted]

ty...@google.com <ty...@google.com> #7Feb 9, 2017 06:21PM

On OSX the file that needs to be replaced is:
google-cloud-sdk/platform/google_appengine/lib/cacerts/urlfetch_cacerts.txt

The location of the folder google-cloud-sdk is dependent on how you installed the SDK.

js...@google.com <js...@google.com> #8Mar 17, 2017 10:16PM

Marked as fixed.

This issue is being closed as all blocking work is complete. The issue has likely been resolved for a while, however thanks to the recent integration with our internal tracker we can more easily detect cases with no remaining blockers.

If this issue is not actually resolved, please open a new issue which references this one and we'll take a look.

[Deleted User] <[Deleted User]> #9Mar 23, 2017 10:14PM

Hi,

This still appears to be an issue. I've replaced my urlfetch_cacerts.txt with the one recommended above and I am still having issues.

[Deleted User] <[Deleted User]> #10Mar 23, 2017 10:21PM

Quick update: I had to update the urlfetch_cacerts.txt in a different google_appengine instance and not the one in google-cloud-sdk and my problem was resolved. Thanks

[Deleted User] <[Deleted User]> #11Apr 17, 2017 11:20PM

Where was the different instance? I'm having the same problem using the google appengine python sdk (not the cloud sdk)

pr...@gmail.com <pr...@gmail.com> #12May 30, 2017 01:27AM

Can I ask what CA are using the domains for your urlfetch requests?

Because I just updated my SDK, and apparently Google removed the Symantec certs of this file. I wonder if it's related to all the news on the Symantec vs Google, and if the production environments will stop working after this.

ge...@gmail.com <ge...@gmail.com> #13Jun 19, 2017 12:16AM

I am still having the same issue. I replaced urlfetch_cacerts.txt using the following command and still have the issue:
wget

https://curl.haxx.se/ca/cacert.pem -O {Root of Python SDK}/lib/cacerts/urlfetch_cacerts.txt

kk...@google.com <kk...@google.com> #14Jun 19, 2017 03:34PM

Hi getcontigo,

Your issue may belong to the corner cases that local urlfetch cannot 100% match deployed app.

Did the same url work locally before? If yes, would you mind tell us the latest SDK version it worked?

Also, would you tell us which url you are trying to fetch? Thanks

jm...@google.com <jm...@google.com> Jun 19, 2017 03:36PM

Reassigned to gc...@google.com.

jo...@gmail.com <jo...@gmail.com> #15Jul 28, 2017 09:18PM

I encountered this issue today, and replacing urlfetch_cacerts.txt with the .pem file did not work for me either.

I upgraded to the current release of Cloud SDK (158.0.0) and still had the issue.

However, I noticed that buried a page or more back in the stack trace for the exception was a message that I was running Python 2.7.5 and that I should consider upgrading. I upgraded to 2.7.12, which is the current Python runtime (as of mid-May). Issue resolved.

Hope this helps someone!

aa...@digismith.net <aa...@digismith.net> #16Sep 10, 2017 06:14AM

Just started happening to me, was working fine earlier today. I tried "validate_certificate=False", updating to "

https://curl.haxx.se/ca/cacert.pem", and upgrading python to recent...

python -V
Python 2.7.13

gcloud --version
Google Cloud SDK 170.0.1
app-engine-python 1.9.57
bq 2.0.25
cloud-datastore-emulator 1.2.1
core 2017.09.08
gcloud
gsutil 4.27

Security is tough! I hope this gets updated soon. Thank you!

Message last modified on Sep 10, 2017 06:15AM

ma...@tech4trades.com <ma...@tech4trades.com> #17Nov 15, 2017 04:31AM

Also having this issue.
Removing urlfetch_cacerts.txt not working.
Updating it not working.

Python 2.7.10
Google Cloud SDK 179.0.0
app-engine-python 1.9.62
bq 2.0.27
core 2017.11.06
gsutil 4.28

ch...@fonoma.com <ch...@fonoma.com> #18Nov 21, 2017 12:49AM

Same problem here
python 2.7.14,
Google Cloud SDK 180.0.0
app-engine-python 1.9.63

MacOS 10.12.6

su...@gmail.com <su...@gmail.com> #19Nov 29, 2017 12:27PM

this error occurred in my local environment

Python 2.7.10
Google Cloud SDK 180.0.1
alpha 2017.09.15
app-engine-python 1.9.63
beta 2017.09.15
bq 2.0.27
core 2017.11.20
gcloud
gsutil 4.28

sv...@gmail.com <sv...@gmail.com> #20Dec 21, 2017 12:46PM

This error keeps happening. It renders local development useless.

Google Cloud SDK 183.0.0
app-engine-python 1.9.64
app-engine-python-extras 1.9.63
bq 2.0.27
core 2017.12.08
gsutil 4.28

gs...@google.com <gs...@google.com> #21Jan 13, 2018 04:06AM

Status: Assigned (reopened)

This issue manifests itself again, lately in Google Groups:

https://groups.google.com/forum/#!topic/google-appengine/TU3LwvRClCU . Re-opening this issue, in consequence.

[Deleted User] <[Deleted User]> #22Jan 24, 2018 08:04AM

Very much an issue!

Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 12:01:12)
Google Cloud SDK 185.0.0
alpha 2017.09.15
app-engine-python 1.9.65
app-engine-python-extras 1.9.63
beta 2017.09.15
bq 2.0.28
cloud-datastore-emulator 1.3.0
core 2018.01.12

br...@percolate.com <br...@percolate.com> #23Jan 29, 2018 07:26PM

Having the same issue with cdn.contentful

Google Cloud SDK 186.0.0
app-engine-python 1.9.65
app-engine-python-extras 1.9.63
bq 2.0.28
core 2018.01.22
gcloud
gsutil 4.28

in...@csgactuarial.com <in...@csgactuarial.com> #24Apr 25, 2018 04:22PM

Also happening within Local:
Python 2.7.14
Google Cloud SDK 199.0.0
app-engine-python 1.9.69
bq 2.0.33
core 2018.04.20
gsutil 4.30

za...@gmail.com <za...@gmail.com> #25Nov 29, 2018 10:19PM

same issue,sometime it works and a minute later got SSL error

ca...@google.com <ca...@google.com> #26Sep 4, 2019 06:42PM

Status: Won't Fix (Obsolete)

This issue has been closed by the engineering team as there has been no activity on it for quite some time.

If the issue persists you may feel free to create a new issue here [1].
You may also reference this issue for context should you create a new issue with us.

[1]

https://cloud.google.com/support/docs/issue-trackers

[Deleted User] <[Deleted User]> #27Mar 28, 2020 07:13AM

getting this problem , can anyone help me ?
I'm using mac OS

WARNING:root:/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/lib/cacerts/urlfetch_cacerts.txt missing; without this urlfetch will not be able to validate SSL certificates.
Traceback (most recent call last):
File "/Users/ast/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 96, in <module>
_run_file(__file__, globals())
File "/Users/ast/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 90, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Users/ast/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 35, in <module>
from google.appengine.tools.devappserver2 import cli_parser
ImportError: cannot import name cli_parser

mi...@gmail.com <mi...@gmail.com> #28Nov 22, 2020 03:30PM

Same problem again but on production server.

From server logs:

2020-11-22 16:14:17.048 CET
/base/alloc/tmpfs/dynamic_runtimes/python27g/dbe7f5cd5f74360/python27/python27_lib/versions/1/lib/cacerts/urlfetch_cacerts.txt missing; without this urlfetch will not be able to validate SSL certificates.
Warning
2020-11-22 16:14:17.048 CET
No ssl package found. urlfetch will not be able to validate SSL certificates.