IssueTracker

[Comment deleted]

Is this the average request response time or the startup time of a new instance (e.g.,
directly after appcfg.py update)?

If it's indeed only 40ms then your website is probably not much bigger than "hello
world"? Loading just one big module can take 40ms and if you've got a non-trivial
website the initial load times for your own modules (without the framework) will easily
be more than 1000ms.

Jack has been using the default VM encoding, which on windows is windows-1252.

This has been fixed for Jack out-of-process. For Jack in-process we need an API to set the source file encoding from the Gradle plugin. Not sure if this will go into Gradle plugin 2.2 or 2.3 preview.

Hi Guido,
when that problem occurred, we often got a DeadlineExceededError in the middle of a
module import before the WSGI handler was activated (via run_wsgi_app()), so there's
no RPC going on at that point.

I didn't save any profiling information, but the results showed that almost all of
the execution time is spent loading modules *before* run_wsgi_app() and then another
profiling session revealed that even when in run_wsgi_app() almost all of the time is
spent loading the remaining modules and quite a bit of time is spent on file reads
(though, not as dramatic as __import__). We have quite a few i18n files and
templates, so probably it has something to do with that. We just installed a simpler
site without i18n and that one loads really fast, ATM, though load times vary by a
factor of 3 (up to 4s).

In another round of profiling I added logging calls with a timestamp that were
executed whenever a model was defined (I intercepted db.PropertiedClass.__init__) and
the result was that when the instance took much longer to load than usual the
increase time distance between model definitions was distributed evenly across all
models (I don't know if it was a proportional amount, but it was significantly more
than in the normal case). IOW, if Model A was defined at time t=0 and Model B at t=1
then on a slow request it would be more like A:t=0 and B:t=2. Basically, the whole
server seemed to become much slower - or maybe it just was a slower server. :)

Could you try to somehow profile a simple Django site and find out why there is a
high variance in startup time, even for the same web page?

Ok, my issue(218892) was merged with this because duplicate.
So, just in case, i attach my screenShotes.

Assigning to Ivan for Gradle plugin side

Lukas, Gradle Plugin is not able to set the encoding with JackInProcess true for now. I think it is in the Jack side for now.

ah sorry, didn't realize the Jack API is still needed

Internal build of Jack provides this API, reassigning to Ivan for Gradle side.

It is on test in Jack side. Please, wait a little.

Need support in Gradle Plugin. Need to talk with Ivan for API.

I think due to the extra imports Django just makes this problem more likely to
surface. I see occasional bursty periods with long import times on a webapp
application also. They coincide with periods when datastore operations take a long
time and are likely to Timeout.

I've considered using fast ajax timeouts and automatic retries to help deal with this
issue, but unfortunately it usually happens that they occur in bursts such that
multiple requests in a short period of time are all likely to be affected.

ag/I4cb2185d2774cb51628fa5f116f808287ef47ea5

I add '-Dfile.encoding=UTF-8' in the 'Android Studio\bin\studio64.exe.vmoptions file' to change the default VM encoding.It's ok before the 'Target-2.3 Release'

Google please test your updates before pushing them, you have just ruined my entire bloody project.

upgrading to api 28 and to the new java 1.8 damaged all my files, I opened the style file and I found java codes from another files in it, I opened a java file and i found lines from gradle setting, I opened gradle settings and I found weird logos and shit, I worked my ass for a fucking year on this project, test your fucking shit. lucky I have an old backup., but I lost weeks of work.

how on earth you can open a new bloody empty project, and you open main activity layout and you find the stupid error telling you failed to render the layout, you can't even put a text box, not even HELLO WORLD is visible..A NEW PROJECT..you shouldn't push your stupid broken update to the public.

R

Here is a thread where I try to track down slow import issues in a webapp app:
http://groups.google.com/group/google-appengine/browse_thread/thread/205dd45cda130c4d# So,
clearly switching from django->webapp will not automatically solve the problem.

[Comment deleted]

I mentioned this in a forum post, but I want to make sure to note that this issue exists for apps using Django
0.96 as well, not just 1.0.

http://groups.google.com/group/google-
appengine/browse_thread/thread/7487e5f429a2789d/56f93701188cb5ff

Here is a sample stack trace from my app: gqueues.appspot.com

========

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
  File "/base/data/home/apps/gqueues/beta2-7-9.336205351043982571/gqueues.py", line 23, in
<module>
    from controllers.admin import AdminHandler
  File "/base/data/home/apps/gqueues/beta2-7-9.336205351043982571/controllers/admin.py", line 10, in
<module>
    from controllers.baserequest import BaseRequestHandler
  File "/base/data/home/apps/gqueues/beta2-7-9.336205351043982571/controllers/baserequest.py", line
6, in <module>
    from google.appengine.ext.webapp import template
  File "/base/python_lib/versions/1/google/appengine/ext/webapp/template.py", line 65, in <module>
    import django.template
  File "/base/python_lib/versions/third_party/django-0.96/django/template/__init__.py", line 57, in
<module>
    import re

============

The DeadlineExceededErrors seemed to start cropping up after the Sept 2nd maintenance.  The logs vary, but
that all seem to be related to timeouts when trying to import modules (for django).  My app uses the pytz
library, which I know has a lot of imports. So I tried removing that library to see if that would improve things,
but I still get the same problems.  It's not all the time, but every day or two for several hours my app is
basically inaccessible for all my users because of this error, which is quite frustrating.

I also see this with the built-in Django 0.96.  It seems to affect some app ids but
not others.

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
  File "/base/data/home/apps/gmz/profile.336427364511513903/main.py", line 35, in
<module>
    import django.core.handlers.wsgi
  File
"/base/python_lib/versions/third_party/django-0.96/django/core/handlers/wsgi.py",
line 1, in <module>
    from django.core.handlers.base import BaseHandler
  File
"/base/python_lib/versions/third_party/django-0.96/django/core/handlers/base.py",
line 4, in <module>
    import sys
<class 'google.appengine.runtime.DeadlineExceededError'>:



I have also seen it die here several times:
...
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 1301, in exception
    apply(error, (msg,)+args, {'exc_info': 1})
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 1294, in error
    apply(root.error, (msg,)+args, kwargs)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 1015, in error
    apply(self._log, (ERROR, msg, args), kwargs)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 1101, in _log
    self.handle(record)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 1111, in handle
    self.callHandlers(record)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 1148, in callHandlers
    hdlr.handle(record)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 655, in handle
    self.emit(record)
  File "/base/python_lib/versions/1/google/appengine/api/app_logging.py", line 70, in
emit
    message = self._AppLogsMessage(record)
  File "/base/python_lib/versions/1/google/appengine/api/app_logging.py", line 83, in
_AppLogsMessage
    message = self.format(record).replace("\n", NEWLINE_REPLACEMENT)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 630, in format
    return fmt.format(record)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 426, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/base/python_dist/lib/python2.5/logging/__init__.py", line 398, in
formatException
    traceback.print_exception(ei[0], ei[1], ei[2], None, sio)
  File "/base/python_dist/lib/python2.5/traceback.py", line 125, in print_exception
    print_tb(tb, limit, file)
  File "/base/python_dist/lib/python2.5/traceback.py", line 69, in print_tb
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/base/python_dist/lib/python2.5/linecache.py", line 14, in getline
    lines = getlines(filename, module_globals)
  File "/base/python_dist/lib/python2.5/linecache.py", line 40, in getlines
    return updatecache(filename, module_globals)
  File "/base/python_dist/lib/python2.5/linecache.py", line 128, in updatecache
    fp = open(fullname, 'rU')

I recall seeing in a thread somewhere that the Sep 2 maintenance added some sort of exception management
that reported this particular problem as a DeadlineExceededError. So it could be that this issue pre-dated the
Sep 2 maintenance, but prior to that it surfaced in different ways.

I do know that for our apps, we've previously had issues with slow spin-up that manifested itself in odd ways.
E.g, Django unable to do a reverse lookup on a url or unable to find a particular view. Both of these examples
were transient; the best way that I can think of to explain it is that the Django initialization had the rug pulled
out from under it (by a deadline issue, presumably due to slow imports judging by information on this Issue)
and then the Django instance seemed to be permanently damaged and that particular application instance
always threw exceptions.

Since the Sep 2 maintenance, I now see DeadlineExceededErrors, but I no longer see this "Django in a bad
initialization state" issue. So I suspect Sep 2 maintenance added a check to see if the DeadlineExceededError
happens on a newly inflating application instance, and if so, the application instance is dumped.

But, I still think this particular issue pre-dated the Sep 2 maintenance; we're all just seeing it very visibly now.

Someone in IRC suggested changing app ids as a method of getting around this.
Unfortunately I have keys hard-coded in my app so it is not trivial for me, but it
might work for some of you who have custom domains...

Our site (app id: skrit) has had much higher error rates since Sep 2, so if this issue
predates it, it's also gotten much worse at the same time. We really didn't have any
problems before.

This week it always happens between 6am and 9am (time from logs). As if some scheduled
heavy maintenance operation takes place in the background.

re: 6a to 9a - could be the North American daytime load coming onstream and many application instances are
spinning up which causes contention for import resources and therefore more DeadlineExceededErrors.

(I should stop supposing as I may be clouding the real problem with my own beliefs. ;)

Hi

I have had my app running now since for several months and the last deployment 46
days ago, recently I have seen an increase in the number of Deadline Exceeded error
messages. I only ever experience these on app startup.  Normally my app startup time
is between 3-8 secs, and occasionally 12.  

An example of heavy startup (ie a view that has a query as well is instance spinup)
/lilies/night_flowers/view 200 6868ms 3678cpu_ms 295api_cpu_ms

vs failure
/lilies/night_flowers/view 500 28572ms 1963cpu_ms

These types of failures are always during import phase.
So definately something is going on.

I am not using django by the way.

Rgds

Tim

Here is an example from today:

      09-19 05:47AM 37.673 /admin/tick 200 10488ms 678cpu_ms 56api_cpu_ms 0kb Python-
urllib/2.5,gzip(gfe),gzip(gfe)
      D 09-19 05:47AM 41.381 App loading (main.py)...
      D 09-19 05:47AM 47.567 admin.py loading...
      D 09-19 05:47AM 47.804 Initial imports complete. WSGI application loading...
      D 09-19 05:47AM 47.860 Handler starting...
      D 09-19 05:47AM 48.117 ...
      D 09-19 05:47AM 48.117 Handler is done.

The debug statement "App loading (main.py)..." is at the very top of the python
script described in app.yaml as the handler for this URL. It looks like this:

#!/usr/bin/env python
import logging
logging.debug("App loading (main.py)...")

So, before the very first import has been completed, almost 4 seconds have elapsed.

Before the WSGI application has been loaded, almost 10 seconds have elapsed.

I am still seeing lots of DeadlineExceededError errors this morning, so last night's maintenance didn't address
this issue.

It is absolutely horrible now. Right now, my app is getting server error for more than one hour. My users are very
angry along with myself at the moment.

Im also seeing lots of DeadlineExceededError

I can't say I'm angry because it is an (awesome) beta product, but it sure would be nice to hear something from
Google on this issue...

Please reply with:
a) which runtime you are using
b) when you began to notice an excess of these exceptions -- did these start after
yesterday's maintenance or earlier in the month?
c) any imports you have for the handlers/servlets that are timing out

a) Python
b) September 2nd maintenance
c) I haven't made narrow test cases, so I can't say it definitely happens on requests
smaller than importing Django 1.0, our models.py, and some views.

AppIDs: myfrontsteps, steprep, mfs-demo, steprep-demo are all affected

(a) Python
(b) DeadlineExceededError started after Sep 2 maintenance, but I believe that this was just a change in how a
pre-existing problem was being reported by the framework
(c) we use Django 1.0 (provided by Google, not uploaded) and the appengine_django helper (r90). The timeouts
usually occur when importing different parts of Django, i.e., before it reaches our code at all.

(a) Python
(b) earlier in the month
(c) The exceptions usually raise importing parts of Django, but not always

application: dhadharu
runtime: python
api_version: 1


I use appengine-patch with django 1.1.

I am getting this from the beginning of this month.

We have made some changes to address this issue so the situation should be improved.
Please to continue to report instances of timeouts during initial load imports so
that we can investigate.

Nice I notice the improve, i haven't saw any Load exception in the last two days.
Thanks

App ID: StepRep
Google-supplied-lib: Django 1.0
appengine_django helper: r90

DeadlineExceededErrors at:

 - 09-28 03:01PM 53.677
 - 09-28 03:01PM 53.664
 - 09-28 03:01PM 53.645
 - 09-28 02:16PM 37.373

The DeadlineExceededErrors were happening quite regularly for AppID StepRep at around:

  - 09-28 05:48PM 47.929

Maybe it is just a coincidence but on a www.cloudstatus.com can be seen pretty
amazing picture from 09/25/09.

It measures response time of a some python code that is resides in AppEngine and
usually (before Sep 25) there was fluctuations in response time between 280 ms and
360 ms.
But now it is almost straight graph with a sustained response time of 280ms.

I mean if mentioned changes were made a couple of days ago - they did the trick (at
least for a test code of CloudStatus).

I am not affiliated with them - just an observation.

Thought everything was great, but DeadLine Exceeded today during imports.

09-29 12:08AM 30.704 appid svfalf

I haven't seen hardly any slow imports for around a week now, very happy with the
changes here!

What exactly got changed? Do you now compile the source to bytecode? Or could that provide another
performance boost?

hi there

app id: bny-imageapp

i am using pylons. first page is almost a static page and do have around 8-13 sec startup time.

This is occurring quite frequently again. App IDs myfrontsteps, steprep, vendastaweb.

It even occurs occasionally on a new, exceedingly simple webapp-based app (i.e., no Django).

(a) Python
(b) earlier in the month
(c) The exceptions usually raise importing parts of Django, but not always

Using App-Engine-Patch, and importing Django-1.1.zip

[Comment deleted]

Loaded with DeadlineExceeded errors today, esp, between 6-6:35 AM Pacific time, then
again at 7AM.

I have seen this more than 40 times from yesterday. And right now, my app is totally
inaccessible. All the requests, except static files, are failing.

Having the same problems with massive timeouts. App is not usable.

(a) python
(b) today (uploaded for testing)
(c) raise during various django imports. Am using app-engine-patch and django1-1.zip.
Most times does not even get to my code. Many times dies in url importing....

deadline exceptions are somewhat random but guaranteed to happen if you let the app
set 10-15 seconds and then try any action. Once you try numerous actions it seems to
'catch up' until either a random dealine exception comes along or you let the app
'set' for 10 - 15 then it's dead again.

we also got a lot of errors the last days, appid: wegif-gae

I concurr, the frequency of DeadLineExceeded errors has gone way up in the last day
or so,

The system status page should really be showing the status of loading a heavy stack
like Django too.

I am also seeing this A LOT.

My details:
a) Python
b) Since maybe 2 weeks ago
c) I'm often having it from within google-app-engine-django imports (which will load
Django 1.1 by default) but sometimes within normal imports and sometimes even into my
own code (e.g. importing simplejson).

A new issue, 2683, asks for a new pricing model to allow longer HTTP requests.  This
would at least keep our applications from sending users errors until this is
resolved.  BTW, we would also need to increase the CPU limits for the purposes of
starting a new instance.  My app is regularly using 45 seconds of CPU to start a new
instance.  This causes an exception as well.  

So basically, 2683 would solve the 30 second request limit.  We would need another
issue to handle the CPU limit.  

Of course this isn't ideal.  It would just help us until we figure out a real
solution to the problem.  A slow response is much better than an error.

It seems that the loading time reduces a lot recently.

Indeed, we've made numerous improvements to the loading time of instances over the past year.  We've also launched warming requests and the Always On feature which is meant to help those who have long initial startup times.

Is there any solid asks in this issue, outside of the pricing model change request that is captured in Issue 2683?  If not, I am going to close this.

I just wanted to note that I still experience problems with loading causing DeadlineExceededError.  Sometimes my loading requests finish in 5 seconds or so, other times it takes upwards of 30+ seconds.

I am currently hitting this issue on my app mind-well.  There are no problems reported on the App Engine System Status page.  Currently the app is unusable due to this problem.  

These issues with slow imports have been popping up lately again. Sometimes a fix is to deploy several versions of the application since a version is not always affected.

We've recently seen a few of the same timeout-on-import issues as well. appid:khanexercises

I experience this issue very frequently and in a quite reproductible way.

My app (paid appid: upcache) is under development with Djangoappengine for it's intended to be a control/command IU for a software engines running elsewhere.

It's receiving informations from engines via HTTP POST requests handled by Djangoappengine (I do know that it's far from the most efficient way, but I need Django database model for subsequent treatments).

In development it receives 3 requests per minutes (+ a cron warmup per minute, quite inefficient) plus tests user requests.

The result is clearly instable.

Requests last between 70 and 400ms when instances are warm (at least I suppose). Suddenly everything goes wrong and for a few minutes all requests fail with a DeadlineExceededError (I even saw this error thrown after 1248121ms - 20 minutes, yes).

After some kind of struggle with the system, the app become responsive again.

User requests often failed or take 10 to 30 secondes to be treated ; at other times they take only some hundreds of milliseconds.

Appstats doesn't seem to be of any help as it seems to start only after the code is launched so figures are always under the second.

I hardly imagine running this app for commercial purpose...

Except giving up DjangoAppengine, does anyone know how to speed-up its start-up?

Thank you very much for your help!

Florent

Reluctantly I am marking this issue as NotRepeatable.  That's because there is no single bug that can be fixed -- this issue has become a catch-all place to gripe about DeadlineExceededError.  There are many different underlying causes, often unique to the specific app (or the packages it loads), and in most cases not enough information is given to be able to explain the reason for the failure observed by a particular user.  I recommend that instead of adding a "me too" comment to this tracker issue, people who are experiencing this problem post to the support group (http://groups.google.com/group/google-appengine-python) so they can be helped either by other users who can explain their symptoms or by an App Engine devrel engineer.

I am experiencing the same issue as indicated above.

@David, can you open a new issue per instruction at [1] so that one of our support staff look into your specific issue as suggested at #64 since there may be many different underlying causes?

[1] https://cloud.google.com/support/docs/issue-trackers