My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 7864: memcache thread race
1 person starred this issue and may be notified of changes. Back to list
Status:  Fixed
Owner:  ----
Closed:  Aug 2012


Sign in to add a comment
 
Reported by flaming...@google.com, Jul 17, 2012
When running with threadsafe: true and high parallelism, reads from memcache sporadically throw an exception in what looks suspiciously like a race:

  File "/base/data/home/apps/s~sc2sandbox/1.360375925982080038/lib/cache.py", line 21, in get
    value = memcache.get(key, namespace=self.__class__.__name__)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 579, in get
    rpc = self.get_multi_async([key], namespace=namespace, for_cas=for_cas)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 632, in get_multi_async
    self.__get_hook, user_key)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 399, in _make_async_call
    rpc.make_call(method, request, response, get_result_hook, user_data)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 519, in make_call
    self.__service, method, request, response, self.__rpc)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 205, in Call
    function(service, call, request, response, rpc)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1200, in pre_call_hook
    recorder_proxy.record_rpc_request(service, call, request, response, rpc)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 378, in record_rpc_request
    self.get_call_stack(trace)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 704, in get_call_stack
    if not self.get_frame_summary(frame, trace):
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 762, in get_frame_summary
    x.set_value(format_value(value))
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 820, in format_value
    return formatting._format_value(val, config.MAX_REPR, config.MAX_DEPTH)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/formatting.py", line 186, in _format_value
    rep = _format_value(dct[nam], limit, level-1)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/formatting.py", line 202, in _format_value
    for elem in series:
RuntimeError: dictionary changed size during iteration
Jul 18, 2012
#1 amis...@google.com
Looks like a bug in appstats.
Status: Acknowledged
Labels: -Language-Python Language-Python27 Component-Tools
Jul 22, 2012
#2 amis...@google.com
(No comment was entered for this change.)
Labels: log-6859447
Jul 22, 2012
#3 amis...@google.com
What's happening is that appstats is traversing the stack and for each stack frame, dumps the method and its arguments. If one of those arguments contains a container (dict, list) that is modified concurrently by different threads, then you can run into this problem.

The following code fragment will trigger this issue:
class Cache(object):
  def __init__(self):
    self._lock = threading.Lock()
    self._cache = {}

  def set(self, key, value):
    with self._lock:
      self._cache[key] = value

cache = Cache()

class MemcacheHandler(webapp.RequestHandler):
  def set_cache(self, cache):
    k = os.urandom(2)
    v = os.urandom(10)
    cache.set(k, v)
    memcache.get('foo')

  def get(self):
    self.set_cache(cache)

So, do you see anything in your code which may look like the above?
Status: MoreInfoRequired
Aug 6, 2012
#4 flaming...@google.com
My code looks similar enough to that that I suspect that's the issue; it's a bit murky with an intercept to db.get in it. Is this a bug regardless, or just an appstats limitation?
Aug 6, 2012
#5 amis...@google.com
Thanks. We looked into this and will be fixing it.
Status: Started
Aug 21, 2012
#6 amis...@google.com
(No comment was entered for this change.)
Status: Fixed
Sign in to add a comment

Powered by Google Project Hosting