My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
  Advanced search   Search tips   Subscriptions
Issue 5774: Class not found with async sessions
92 people starred this issue and may be notified of changes. Back to list
Status:  Fixed
Owner:  ----
Closed:  Mar 2012

Sign in to add a comment
Reported by, Sep 3, 2011
2011-09-03 14:43:56.361

javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassNotFoundException:$DeferredSave

The task keeps retrying and keeps failing.
Sep 6, 2011
Project Member #1
Can you provide more context (for example sample code) on how to reproduce this issue ?

Is this happening in the SDK, in production or both ?

Thanks in advance.
Status: NotRepeatable
Labels: Component-TaskQueue
Sep 6, 2011
I just enabled session for my app and I get this intermittent for errors. 
Please do not mark this bug as closed / non repeatable. It is clearly an issue as other people have reported it (in the forums and 8 people starred the issue in less than 2 days).

It also only happens in production.

You can check my app id logs for version

    2011-09-04 10:34:41.643 /_ah/queue/__deferred__ 415 10ms 9cpu_ms 0kb AppEngine-Google; (+ - - [04/Sep/2011:07:34:41 -0700] "POST /_ah/queue/__deferred__ HTTP/1.1" 415 0 "" "AppEngine-Google; (+" "" ms=10 cpu_ms=9 api_cpu_ms=0 cpm_usd=0.000314 queue_name=sessions task_name=6797441778698926271 instance=00c61b117c9a0ca10640cb55f285c79935466398

    I 2011-09-04 10:34:41.640

    javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassNotFoundException:$DeferredDelete

Sep 6, 2011
Project Member #3
(No comment was entered for this change.)
Status: Acknowledged
Sep 12, 2011
I have the same error in production enviroment. The origin of the error is when configure asyn session persistence feature.

Example code:
<async-session-persistence enabled="true" queue-name="default"/>

Sep 12, 2011
What the hell is this google support. Now response in 6 days. Is this how you guys want to run service. I am never developing on google app engine ever again...
Sep 15, 2011
javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassNotFoundException:$DeferredSave
Sep 16, 2011
In deed - I asked about this issue several months ago - no solution in 3 releases since then... see

Google people, at least please explain why the error is occuring.

When will something be done here?

Sep 16, 2011

I too am very interested in obtaining a little feedback on this problem... even if its just a recommendation to stop using it.

I only turned it on following advice from the following URL (which quoted it as a method of reducing the additional cost burdens of the price changes)

I am currently receiving 100% failures.
Oct 6, 2011
This error continues to be present in Java SDK 1.5.4. This happens in production as well. Something to fix asap, given it is one of the recommended options for optimizing instance usage!
Oct 6, 2011
Project Member #10
Sorry for the late followup, 

Is this error also happening with 1.5.5 pre-release ?

Thanks in advance.
Oct 6, 2011
I have not yet tried this with 1.5.5 pre-release, and will not be able to do so anytime soon. If anyone else who is following this issue has done so, please comment.

Oct 6, 2011
Hi guys,

We deciding to turn on async sessions and sure enough we found this issue. Googled the error and here we are.

SDK 1.5.4 .... 
Oct 6, 2011
(No comment was entered for this change.)
Labels: log-5425806
Oct 28, 2011
Hi all,

I am using appengine-sdk 1.5.5 pre-release.With 1.5.5 also, this issue is persisting.

code used in my appengine-web.xml:
<async-session-persistence enabled="true" queue-name="default"/>
Oct 28, 2011
I am using 1.5.5 and it is still a problem.  Is there any reason why it would have been fixed?  Has someone worked on this?
Nov 2, 2011
Project Member #16
(No comment was entered for this change.)
Status: Accepted
Nov 2, 2011
Project Member #17
I should be fixed with 1.6.0 pre-release SDK.

Feel free to report here if that's not the case.

Thanks in advance.

Status: Fixed
Nov 2, 2011
Thanks for fixing this! I am excited to see what impact async sessions will have on my app.
Nov 2, 2011
Likewise for the fix. We were holding off testing this until it was fixed. So, looking forward to 1.6.0 and to test this.

If people who are using it already or test it in 1.6.0 can post some results and/or recommendations that would e great for the rest of us. Thanks
Nov 13, 2011
I don't believe this issue is fixed. I am using the sdk 1.6 since 7th Novemver 2011, on 12th Nov I enabled     <async-session-persistence enabled="true" />. Left the system running overnight and in the morning the logs are full of the following

_ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassCastException:$DeferredSave cannot be cast to

Since google has moved app engine out of preview and created an SLA. The least we can expect is very rigorous testing before adding any new features. After all I am relying on app engine to be my production quality system. How can we monitor logs 24/7 . Had I not checked logs this might have gone un noticed for days.
Nov 13, 2011
Plus one vote for better system for more immediate reporting of production issues. I have setup a Caron task that downloads, formats and emails exceptions. Minimum period for log download is 1 day.

So my frequency of log analysis (without starting to parse exception date/time), and hence maximum "latency" in detecting an issue is one day.

Need a system that can send exceptions by email, or xmpp chat to logged-in admins, in real time... with rules for severity, who to send to, escalation path if not delivered etc.... all configurable from dashboard.

As this is off-topic, if you submit an enhancement request, please let us know number.
Nov 13, 2011
Confirmed, this is NOT fixed. 

2011-11-13 15:45:35.953 /_ah/queue/__deferred__ 415 425ms 0kb AppEngine-Google; (+ - - [13/Nov/2011:15:45:35 -0800] "POST /_ah/queue/__deferred__ HTTP/1.1" 415 60 "" "AppEngine-Google; (+" "" ms=426 cpu_ms=631 api_cpu_ms=0 cpm_usd=0.018270 queue_name=session-queue task_name=18206908635556665143 instance=00c61b117c80eea93a19906ffa7093a657947d
I 2011-11-13 15:45:35.811
javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassCastException:$DeferredSave cannot be cast to
Nov 17, 2011
(No comment was entered for this change.)
Status: Acknowledged
Nov 17, 2011
Project Member #24
 Issue 5845  has been merged into this issue.
Nov 30, 2011
(No comment was entered for this change.)
Labels: -log-5425806 log-5386803
Nov 30, 2011
I can only reproduce this error when I have AppStats running. Can anyone reproduce this without AppStats?

Max (Google)
Dec 1, 2011
I do not have AppStats enabled and I can still see this happening all the time.
Dec 1, 2011
I too don't have stats and still:
2011-12-01 10:53:53.178 /_ah/queue/__deferred__ 415 11ms 0kb AppEngine-Google; (+ - - [01/Dec/2011:07:53:53 -0800] "POST /_ah/queue/__deferred__ HTTP/1.1" 415 0 "" "AppEngine-Google; (+" "" ms=11 cpu_ms=38 api_cpu_ms=0 cpm_usd=0.001336 queue_name=sessions task_name=13870281791854850281 instance=00c61b117ca93d202e52ba77e94504291cf8
I 2011-12-01 10:53:53.175
javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassCastException:$DeferredSave cannot be cast to
Dec 1, 2011
Yes, I see it with stats turned off. Using 1.6. Same log message as above:  _ah_queue_deferred: ....ClassCastException...
Dec 1, 2011
This issue seems to be consuming a lot of CPU. I hope Google will do the right thing and not charge me for CPU consumed by this thread. I don't see why I should pay real $$ for this bug.

Dec 1, 2011
javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task 
failed exception: 
$DeferredTaskException: java.lang.ClassCastException:$DeferredSave 
cannot be cast to

Dec 3, 2011
I am seeing these errors in the task queue in the admin console with SDK 1.6.0 in production as well (app id yama-test).  The "Last http response code" is 415.  It seems to be happening when we call session.invalidate().
Dec 20, 2011
 Issue 6339  has been merged into this issue.
Dec 22, 2011
How the heck do we stop it once it's started?  I've disabled "async-session-persistence" and I'm still getting the errors (5 per second).
Dec 22, 2011
In case anyone else has the same problem - I managed to stop it by going to the "default" task queue and purging it (after turning off "async-session-persistence" of course).  Seems to have worked.
Jan 4, 2012
I can confirm that I am seeing this with sdk 1.6.1 in production.
Jan 5, 2012
as am I - 1.6.1
Jan 14, 2012
Still an issue with 1.6.1.
Could you plz give us tips to handle this, and what could be the timeline for its FIX.

2012-01-14 23:06:02.247 javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassCastException:$DeferredSave cannot be cast to

Jan 20, 2012
Please provide a fix for this asap. I am seeing this with 1.6.1 as posted by other as well. At least provide an ETA on when it will be fixed or some workaround...
Feb 11, 2012
5 months..and it still exists..
Feb 12, 2012
Deployed my application into production and I am getting this:

javax.servlet.ServletContext log: _ah_queue_deferred: Deferred task failed exception:$DeferredTaskException: java.lang.ClassCastException:$DeferredSave cannot be cast to

Really appalling work by the Googlers for them not to be fixing this issue.  What the **** is going on !?  
Feb 20, 2012
Same issue here, which was an unpleasant surprise when switching to production and enabling billing: the volume of these I get is eating up an indecent amount of instances/hours - I have disabled billing for the time required to fix this.
Feb 21, 2012
I am getting this issue on a new application and did some digging.  Looking at the task queues, they contain serialized representations of$DeferredDelete.  I presume that this is some form of scheduled cleanup for sessions.

Given the ClassCastException listed in previous comments, I would surmise that an ObjectInputStream somewhere in DeferredTaskServlet is loading the DeferredDelete class from a ClassLoader different than what it is defined in.  It then tries to cast to a DeferredTask and gets a ClassCastException.

To try to provide some additional context, I uploaded some code that dumps information about the classes in question.  The output is here:
Class Loader:
  Found class interface
    (Defined in ClassLoader
    (URL: jar:file:/base/java_runtime/api/appengine-api.jar!/com/google/appengine/api/taskqueue/DeferredTask.class)
  Found class class
    (Defined in ClassLoader
    (URL: jar:file:/base/java_runtime/api/appengine-api.jar!/com/google/apphosting/utils/servlet/DeferredTaskServlet.class)
  Found class class
    (Defined in ClassLoader null)
    (URL: jar:file:/base/java_runtime/runtime-shared.jar!/com/google/apphosting/runtime/jetty/DeferredSessionTasks.class)
Checking system ClassLoader:
Error checking system class loader: access denied (java.lang.RuntimePermission getClassLoader)
The code that generates this is attached.

Based on this, we see that both DeferredTask and DeferredTaskServlet are defined in the UserClassLoader along with the rest of the application while the DeferredSessionTasks$DeferredDelete class which is being deserialized is in the "bootstrap" class loader (or at least the fuzzy sandbox below the level of app visibility).  The DeferredDelete class must be implementing a DeferredTask interface from this same bootstrap class loader.

When the DeferredTaskServlet deserializes it gets a DeferredDelete instance from the bootstrap loader implementing a DeferredTask from the bootstrap and tries to cast it to a DeferredTask from the user class loader... and gets a ClassCastException.

At least, that is what I figure is going on after a long history with ClassLoader type things :)

This analysis shows that this is likely not an environmental issue, but here are my details:
   - Only occurs when deployed on production app engine
   - App Engine SDK 1.6.1 deployed from the Eclipse Plugin
   -   <threadsafe>true</threadsafe>
   -  <sessions-enabled>true</sessions-enabled>
   -  <async-session-persistence enabled="true" />

Let me know if I can be of any further help tracking this down.  Some lucky engineer has some cross-eyed time staring at ClassLoader hierarchies ahead of them.
1.7 KB   View   Download
Feb 21, 2012
And for the lack of feedback from the billing team - while the issue has cost me a dollar in performance I have note used (and contributed to a downtime of several hours, thankfully, while in Beta), I have had to come up with a workaround, fairly simple: silence the default queue using a queue.xml

<!--Change the refresh rate of the default queue to 0/s-->

I am, however, disappointed so far in the support. Profoundly so, for the single point of direct contact I have been given has showed a total absence of concern about my (billed) services. *Shivers*, the other workaround I am considering is using one of the available App Engine emulation layers for other cloud providers.

If Google doesn't release its custom domain SSL support options rapidly, I will have no choice. App Engine could be the cream but it's light years away from it because of very poor customer care.
Feb 22, 2012
Great comment Mauro - I fully agree. Google people: get it done. This issue and SSL on custom domain are the main issues for many of your clients. Please, get it done. Just do it, please.
Feb 22, 2012
Heartfelt Goran, thank you. For years I have implemented ITIL-compliant support processes in corporate environments, emails to the App Engine billing team have gone unanswered so far, not even an acknowledgment. 

I have experienced other similar communication problems in the past with Google, granted, I imagine they are in demand, but Microsoft has a completely automated support service of the highest quality for example. I can't be left wishing on a star when it comes to business.
Mar 1, 2012
Is this still broken in 1.6.3? Has anyone tested it?  I see it recommends doing this in the optimization article:
Mar 1, 2012
Just tried.
Still broken :(
Mar 1, 2012
They should probably remove this from the article then since it has never worked.
Mar 21, 2012
Should be fixed in 1.6.4.
Status: Fixed
Mar 22, 2012
Are sessions written to the same datastore as the application (M/S or HRD)?  Do the writes use EVENTUAL CONSISTENCY?
Mar 22, 2012
Sessions are written to the same datastore that your app uses. The _reads_ are strongly consistent. In practice it should be pretty infrequent for your session data to be read from the datastore since we read from memcache first and then only fail over to the datastore if it's not there.
Sign in to add a comment

Powered by Google Project Hosting