I add more info about this bug :

I can reproduce this bug in Production and in Development
You can check the log here :
http://appengine.google.com/logs?app_id=testgapp&version_id=1.6&logs_form=1&severity_level=3

It's random and the error is sometimes diffente :
* KeyError: '0'
* TypeError: 'dict' object is not callable

If i work with my SDK : stop and start Eclipse and it works again. In production, i
upload a new version and it seems to work again.

To raise the error : 
- connect here  http://testgapp.appspot.com/
- upload photos
- check the log (if an image is missing)

I hope it will help you.

Regards

I have uploaded my sources. The error is raise (in the main.py) :

class GetImage(webapp.RequestHandler):
    def get(self, size, key):
        try:
            logging.info('GetImage : size = %s , key = %s' % (size, key))
            img = memcache.get(key) <---- Random Error

----------------

My PC : Win XP Sp2 (French)
Pyhton 2.5 (default install) + PIL
GAE 1.10
My Dev IDE : 3.3.2 + PyDev

source.zip 5.4 KB   Download

Oups : python version = 2.5.2

i've got a similar problem which can be reproduced by 
{{{
memcache.set(key, 1)
memcache.get(key)
}}}
however
{{{
memcache.set(key, '1') 
memcache.get(key)
}}}
works fine. 
it looks like there is some problem with storing int values and such things should be
explicitly casted to a string.

I add the KeyError: '0' stack
-----------------------------
'0'
Traceback (most recent call last):
  File "/base/data/home/apps/testgapp/1.6/main.py", line 141, in get
    img = memcache.get(key)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
370, in get
    self._do_unpickle)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
188, in _decode_value
    value = do_unpickle(stored_value)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
251, in DoUnpickle
    return self._unpickler_instance.load()
  File "/base/python_dist/lib/python2.5/pickle.py", line 852, in load
    dispatch[key](self)
  File "/base/python_dist/lib/python2.5/pickle.py", line 1149, in load_binget
    self.append(self.memo[repr(i)])
KeyError: '0'
------------------

This is killer for me as well, it is happening when I try to retrieve a dictionary
from the memstore.

As requested in the group: here are my object i put in the memcache :

class Image(db.Model):
    name = db.StringProperty()
    type = db.StringProperty()
    ext = db.StringProperty()
    label = db.StringProperty()
    owner = db.UserProperty()
    date = db.DateTimeProperty(auto_now_add=True)
    hits = db.IntegerProperty(default=0)
    original = db.BlobProperty()
    large = db.BlobProperty()
    medium = db.BlobProperty()
    small = db.BlobProperty()
    out_ext = db.StringProperty()
    length = db.IntegerProperty()

and the key is a random string (i.e : npyFWbcjHGyYNsBbPWrr3qYjFTgjPf )

my code : 
im = Image()
im.<my_property_n> = <a_value_n>
im.put()
memcache.set("npyFWbcjHGyYNsBbPWrr3qYjFTgjPf", im)

then in the code i do :
img = memcache.get("npyFWbcjHGyYNsBbPWrr3qYjFTgjPf").

The strange thing is that this error is random.

Now all my errors are KeyError: '0' (see comment '4')

I hope it will help

I am getting the same problem:

Traceback (most recent call last):
  File
"/home/gregf/Servers/google_appengine/google/appengine/ext/webapp/__init__.py", line
499, in __call__
    handler.get(*groups)
  File "/home/gregf/Servers/google_appengine/tawa/tawa_admin.py", line 14, in get
    sess=gmemsess.Session(self)
  File "/home/gregf/Servers/google_appengine/tawa/gmemsess.py", line 48, in __init__
    data=memcache.get(self._sid)
  File
"/home/gregf/Servers/google_appengine/google/appengine/api/memcache/__init__.py",
line 370, in get
    self._do_unpickle)
  File
"/home/gregf/Servers/google_appengine/google/appengine/api/memcache/__init__.py",
line 188, in _decode_value
    value = do_unpickle(stored_value)
  File
"/home/gregf/Servers/google_appengine/google/appengine/api/memcache/__init__.py",
line 251, in DoUnpickle
    return self._unpickler_instance.load()
  File "pickle.py", line 858, in load
    dispatch[key](self)
  File "pickle.py", line 1156, in load_binget
    self.append(self.memo[repr(i)])
KeyError: '5'

This occurs intermittently, but once it starts happening, it is continuous. It occurs
when retrieving a dictionary containing a single key,value pair.

I triggered it once when clicking "Reload" fast in Firefox, so I suppose there is the
possibility of a race condition where two requests are accessing the same data. This
occurred last night when the dev server had been running for several hours, but I
can't reproduce it this morning.

This is on the dev server 1.1.0 running on Ubuntu Gutsy, I have not reproduced it on
production. Let me know if you want sample code.

Marzia asked me to add to this issue. I'm getting a similar error to everyone else, 
working with a dict as a value for memcache.

Traceback (most recent call last):
  File "/root/google_appengine/google/appengine/ext/webapp/__init__.py", line 499, 
in __call__
    handler.get(*groups)
  File "/var/www/svn/appengine-utitlies/main.py", line 35, in get
    self.sess = session.Session()
  File "/var/www/svn/appengine-utitlies/utilities/session.py", line 95, in __init__
    self.memcache = memcache.get("sid-"+self.sid)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 370, 
in get
    self._do_unpickle)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 188, 
in _decode_value
    value = do_unpickle(stored_value)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 251, 
in DoUnpickle
    return self._unpickler_instance.load()
  File "/usr/local/lib/python2.5/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/lib/python2.5/pickle.py", line 1156, in load_binget
    self.append(self.memo[repr(i)])
KeyError: '3'

The code can be browsed at: http://code.google.com/p/appengine-
utitlies/source/browse/trunk/utilities/session.py?r=65

If you checkout that revision of my project, and run it (the entire project is a 
working demo that you can call directly with dev_appserver.py) you can reproduce the 
error by doing the following.

http://localhost:8080/session
refresh page

It appears to have no problems setting the value, however retrieving it creates a 
problem. 

I did some poking around on ummm Google (sometimes I wonder if you know how often we 
nod in approval when interview candidates for a couple open sysadmin positions we 
have, when they mention Google for problem solving)... It appears that the most 
cases I saw where this type of error was popping up was when people were pickling 
functions and/or classes, then trying to unpickle them in an instance where that 
function/class no longer exists. Appearantly pickle stores a reference to the 
function/class object rather than the object itself, creating the problem. Not sure 
if this is helpful or applies in this case, but thought I'd record my findings.

(No comment was entered for this change.)

Thank you for accepting this!

To anyone out there that this is breaking for, my workaround for storing a dictionary
is to call the template that was going to display the data, render it, and then cache
the string output.  If that is applicable in your situation, give it a shot.

bowman.joseph:  that's very helpful, actually!

Further to my post above, It DOES seem to happen on the production system. One of my
users reports it happens...
- Right after uploading the script (and hitting reload). Closing the browser and
starting again solves the problem.
- Sometimes when hitting repetitively the reload button.

Also just want to point out the obvious that there may be two separate issues here:

- the original TypeError: 'dict' object is not callable 
File "/base/python_dist/lib/python2.5/pickle.py", line 1126, in load_reduce
    value = func(*args)

- the KeyError: 'x'
File "pickle.py", line 1156, in load_binget
    self.append(self.memo[repr(i)])

Maybe this is normal behaviour, and if so, please let me know and disregard. But.. I 
think there might be a typo somewhere in memcache __init__.py that's creating the 
problem, but I can't find where. I tried tracking it down with pylint and got 
nothing. The reason I believe this is the case is this.

Run my script, refresh, and you get this error.

Traceback (most recent call last):
  File "/root/google_appengine/google/appengine/ext/webapp/__init__.py", line 499, 
in __call__
    handler.get(*groups)
  File "/var/www/svn/appengine-utitlies/main.py", line 35, in get
    self.sess = session.Session()
  File "/var/www/svn/appengine-utitlies/utilities/session.py", line 95, in __init__
    self.memcache = memcache.get("sid-"+self.sid)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 370, 
in get
    self._do_unpickle)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 188, 
in _decode_value
    value = do_unpickle(stored_value)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 251, 
in DoUnpickle
    return self._unpickler_instance.load()
  File "/usr/local/lib/python2.5/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/lib/python2.5/pickle.py", line 1156, in load_binget
    self.append(self.memo[repr(i)])
KeyError: '22'

Now, open up __init__.py in the memcache directory, and add 3 blank lines to the 
file, then refresh your browser. You'll get the following.

Traceback (most recent call last):
  File "/root/google_appengine/google/appengine/ext/webapp/__init__.py", line 499, 
in __call__
    handler.get(*groups)
  File "/var/www/svn/appengine-utitlies/main.py", line 35, in get
    self.sess = session.Session()
  File "/var/www/svn/appengine-utitlies/utilities/session.py", line 95, in __init__
    self.memcache = memcache.get("sid-"+self.sid)
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 370, 
in get
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 188, 
in _decode_value
    if flags & FLAG_VALUE_UNICODE:
  File "/root/google_appengine/google/appengine/api/memcache/__init__.py", line 251, 
in DoUnpickle
    self._pickle_data.truncate(0)
  File "/usr/local/lib/python2.5/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/local/lib/python2.5/pickle.py", line 1156, in load_binget
    self.append(self.memo[repr(i)])
KeyError: '22'

Note how the line numbers don't change for the error reporting. 370, 188, and 251 
are the lines in __init__.py that appear to cause problems, regardless of what those 
lines actually are?

Well, figured out that the reason the line numbers don't change appears to have 
something with the way code is being cached by the SDK, not that there's a problem 
with the __init__.py script itself.

this is breaking for me as well. trying to implement sessions based on a cached
dictionary.

The work-around for now (and honestly the better solution in any case) is to not put
complex objects that need to be pickled into memcache.  It's something in the App
Engine pickle code that's being causing the errors here.

But I say it's the best solution in any case because putting raw strings (or unicode
strings) will take up less space in memory, so then you'll have more memory to work
with for your other items.

Note that currently there's a bug in the SDK & production where ints and longs are
pickled unnecessarily.  That shouldn't be a problem for depickling since they're so
simple, but it is a problem if you insert an int into memcache and then try to "incr"
it.  Because it's actually stored as a pickled int in memcache, memcache can't
increment it (memcache isn't python code).  So if you want to use counters and use
incr/decr, be sure to initialize your counter with a string version of your number:

memcache.set("counter", str(the_value))

- Brad

Will there be changes made to allow objects in memcache using appengine? Or is this 
going to be one of those "it is, what it is" cases of development? Reason I ask is 
for what I'm working on (sessions) it would be worth waiting on a fix rather than 
developing a work around. Basically the only work around I can see is to come up 
with something like 

if type(value) == type(''):
  memcache.set("sid_"+sid+"_"+keyname, str(value))

Which while possible, starts to get kind of ugly.

It is a simple bug and will be fixed with the next SDK release.

Great, thats very good to hear. I'll be sure to update my code to reflect the 
changes and demonstrate them working after the next release.

Is this likely to fix the KeyError exception too?

I met simliar situation while using memcache.get.

    cleanedContent = memcache.get(url)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
370, in get
    self._do_unpickle)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
188, in _decode_value
    value = do_unpickle(stored_value)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
251, in DoUnpickle
    return self._unpickler_instance.load()
  File "/base/python_dist/lib/python2.5/pickle.py", line 852, in load
    dispatch[key](self)
  File "/base/python_dist/lib/python2.5/pickle.py", line 1071, in load_obj
    self._instantiate(klass, k)
  File "/base/python_dist/lib/python2.5/pickle.py", line 1054, in _instantiate
    value = klass(*args)
TypeError: in constructor for _URLFetchResult: __init__() takes exactly 2 arguments
(1 given)

junliu44:  that looks like the same problem that's fixed and will be included in the
next release.

I am running into a similar Pickle issue but not with memcache. My problem is
unpickling from the datastore.

This error is not present for me in the mac SDK dev environment. I compared the
strings in the Database, and they appear to be the same string generated by the
pickle method in both the local dev environment and in production.

------------------------------------------------

In the following code, I am unpickling a pickled list of dictionaries
(result._resultrolls) from a db.TextProperty.


Traceback (most recent call last):
  File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line
499, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/rollr/1.3/func_rollr.py", line 384, in get
    resultsunpick = pickle.loads(result._resultrolls)
  File "/base/python_dist/lib/python2.5/pickle.py", line 1367, in loads
    return Unpickler(file).load()
  File "/base/python_dist/lib/python2.5/pickle.py", line 852, in load
    dispatch[key](self)
KeyError: '\x00'

jforsthe: that traceback does not look related. Most likely the problem is that you
should not be storing pickled data in Text properties -- you should use Blob
properties. Please use the discussion group to get to the bottom of it -- do not
follow up here.

"will be included in the next release"

Any info when the next release is due? This bug is a showstopper for me.

No, but here's a suspected workaround: instead of memcache.set(...), use
memcache.Client().set(...).  No need to change the get() calls.  You only need to use
the work-around for values that are objects (not strings).

additional note, happening sporratically at mobaganda.com.

I disabled memcaching tonight, as I wait for resolution.

@28 (gvanrossum), any news if this does indeed help with caching objects?

Thanks!

@30 (stirman) - to be honest, I haven't tried, but would expect the time to create a
Client() is still small compared to the RPC latency.  Also you only need to create a
Client() when *writing*.

For everyone having these kind of problems:
I have recently bumped into it, and was able to pin it down to the pickler mechanism
memcached uses. A very quick, and very dirty patch, take it with a grain of salt:

** google\appengine\api\memcache\__init__.py ** line 241- **
    def DoPickle(value):
      self._pickler_instance = pickler(self._pickle_data,pickle.HIGHEST_PROTOCOL)
      self._pickle_data.truncate(0)
      self._pickler_instance.dump(value)
      return self._pickle_data.getvalue()
*** EOF ***
so basically, I just re-initalized the pickler instance on every serialization event.
Works for me.

Note for Google: I'd suggest using the version control for actual, daily builds, and
quick patches; for some of us, developers, the "next SDK release" is waaay too far in
the uncertain future.

-SDr

Guido,

I don't know that that workaround is valid.  I'm using django.core.cache, which uses a wrapper for memcache.Client():


class CacheClass(BaseCache):
    def __init__(self, server, params):
        BaseCache.__init__(self, params)
        self._cache = memcache.Client()

    def add(self, key, value, timeout=0):
        self._cache.add(key.encode('ascii', 'ignore'), value, timeout or self.default_timeout)
...


and I still get sporadic KeyErrors:
<type 'exceptions.KeyError'>: '1'
Traceback (most recent call last):
  File "/base/data/home/apps/doodlewarrior/1.340/main.py", line 42, in main
    util.run_wsgi_app(application)
  File "/base/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 76, in run_wsgi_app
    result = application(env, _start_response)
  File "/base/data/home/apps/doodlewarrior/1.340/django/core/handlers/wsgi.py", line 205, in __call__
    response = self.get_response(request)
  File "/base/data/home/apps/doodlewarrior/1.340/django/core/handlers/base.py", line 64, in get_response
    response = middleware_method(request)
  File "/base/data/home/apps/doodlewarrior/1.340/django/middleware/cache.py", line 64, in process_request
    response = cache.get(cache_key, None)
  File "/base/data/home/apps/doodlewarrior/1.340/appengine_django/cache_backend.py", line 20, in get
    val = self._cache.get(smart_str(key))
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 370, in get
    self._do_unpickle)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 188, in _decode_value
    value = do_unpickle(stored_value)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 251, in DoUnpickle
    return self._unpickler_instance.load()
  File "/base/python_dist/lib/python2.5/pickle.py", line 852, in load
    dispatch[key](self)
  File "/base/python_dist/lib/python2.5/pickle.py", line 1149, in load_binget
    self.append(self.memo[repr(i)])





Separately, even though this is a beta, it is unacceptable to have such a buggy feature included in the API, especially when it's 
being promoted as a best-practice that everyone should be using.  If memcache causes HTTP 500s as often as it does, that 
makes it unusable for any actual app.  If you guys do have a fix for this critical issue, please deploy it.  This isn't a feature 
add, but a showstopper.  If you have to release a new SDK, and all it does is fix this bug, it's worth it. 

Thanks, as always, for your hard word on this service.  I look forward to the day when I can deploy every app I make on GAE 
and not have to think about hosting again. =)

No, releasing just a new SDK would not fix this. The good news is that a new release
including the fix has nearly made it through QA so relief is nearly there. We're very
sorry for this bug and the grief it has caused. Another workaround, BTW, is to only
store string data in memcache. This is guaranteed not to be affected by this bug.

Looking at your example, it seems that the Django wrapper does not use the
workaround. The trick is never to use a memcache.Client() instance more than once.

Thanks Guido.  I couldn't imagine you'd be sitting on an update that would fix this, but your earlier comment 
seemed to imply that.

I'll look at the Django cache backend and see if I can make it instantiate a new Client for each call. =)

For those of you trying to use Django's CacheMiddleware, here's a patch until this bug is fixed:

Add this to your settings.py:
CACHE_BACKEND_MODULES = {'appengine': 'appengine_django.cache_backend'}
CACHE_BACKEND = 'appengine://'

MIDDLEWARE_CLASSES = {
    'django.middleware.cache.CacheMiddleware',
    # . . .
}

Replace this file with the patched version:
./django/core/cache/__init__.py

and add this one:
 ./appengine_django/cache_backend.py

cache_backend.py 1.3 KB   Download

__init__.py 2.4 KB   Download

> Comment #35 by bsimpson:
> Thanks Guido.  I couldn't imagine you'd be sitting on an update that would
> fix this, but your earlier comment seemed to imply that.

The fix I mentioned earlier was pushed out to production yesterday. The fixed SDK
will arrive in a few days, once it's been vetted by QA.

(No comment was entered for this change.)

thanks guido!

In that cache_backend I posted, each of the cache calls are routed through its own instance of 
memcache.Client().  When the new SDK is released, should I change them back to using a single instance of 
memcache?

Thanks guys!

Yes

We ate Sir Robin's Minstrels, and there was much rejoicing.

Thank's, Guido.

This should now be fixed as of our latest release. Refer to our 1.1.1 release notes:
http://code.google.com/p/googleappengine/wiki/SdkReleaseNotes

Still have this bug : 

08-11 06:49AM 59.673

'0'
Traceback (most recent call last):
  File "/base/data/home/apps/testgapp/2.21/main.py", line 143, in get
    img = get_image(key)
  File "/base/data/home/apps/testgapp/2.21/main.py", line 309, in get_image
    img = memcache.get(key)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
396, in get
    self._do_unpickle)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
206, in _decode_value
    return do_unpickle(value)
  File "/base/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line
277, in DoUnpickle
    return self._unpickler_instance.load()
  File "/base/python_dist/lib/python2.5/pickle.py", line 852, in load
    dispatch[key](self)
  File "/base/python_dist/lib/python2.5/pickle.py", line 1149, in load_binget
    self.append(self.memo[repr(i)])
KeyError: '0'

is it possible that that was caused by a really old cache entry (stored before the
bugfix went live)?  The fix prevents the API from storing bad cache entries, but it
doesn't do anything about bad cache entries stored before the fix was live.  If you
don't use timeouts on your cache (or perhaps if your timeouts are measured in weeks)
you might experience this.

Yes, thank you.

I've flushed the cache (flush_all()) and now it works well.

what flush_all()? where?

http://code.google.com/appengine/docs/memcache/functions.html

flush_all()
    Deletes everything in memcache.
    The return value is True on success, False on RPC or server error.