There's following exception in the wavereactor-hr log related to the re-index map reduce job.
2012-04-02 00:51:47.942 com.google.walkaround.slob.server.MutationLog reconstruct: Reconstructed requested version null from snapshot at 7 followed by 1 deltas I 2012-04-02 00:51:47.942 com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler$1 run: Re-indexing SlobId(lmTba_6WnUYL7epj) at version 8 I 2012-04-02 00:51:47.942 com.google.walkaround.wave.server.wavemanager.WaveIndex update: Updating index for SlobId(lmTba_6WnUYL7epj): IndexEntry(SlobId(lmTba_6WnUYL7epj), wim@fixnum.org, ,
, 1332402896247, [wim@fixnum.org]) I 2012-04-02 00:51:47.959 com.google.walkaround.util.server.RetryHelper runBodyOnce: Body exited normally, run time: 57ms I 2012-04-02 00:51:47.959 com.google.walkaround.wave.server.wavemanager.ReIndexMapper map: Re-indexing Wavelet("lm_zZygomxSqKFYD") I 2012-04-02 00:51:48.294 com.google.walkaround.util.server.RetryHelper runBodyOnce: Running body VoidBodyWrapper(com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler$1@5476bf) I 2012-04-02 00:51:48.299 com.google.walkaround.util.server.appengine.CheckedDatastore$1 run: Begun transaction CheckedTransactionImpl(Txn [s~wavereactor-hr.-7913504055520821756, BEGUN] (class com.google.appengine.api.datastore.TransactionImpl@1c4c801)) I 2012-04-02 00:51:48.316 com.google.walkaround.slob.server.MutationLog getMetadata: Got {"1":0} I 2012-04-02 00:51:48.324 com.google.walkaround.slob.server.MutationLog getSnapshotEntryAtOrBefore: query SELECT * FROM WaveletSnapshot WHERE ancestor is Wavelet("lm_zZygomxSqKFYD") ORDER BY key DESC returned first result null W 2012-04-02 00:51:51.004 com.google.appengine.api.datastore.QueryResultsSourceImpl logChunkSizeWarning: This query does not have a chunk size set in FetchOptions and has returned over 1000 results. If result sets of this size are common for this query, consider setting a chunk size to improve performance. To disable this warning set the following system property in appengine-web.xml (the value of the property doesn't matter): 'appengine.datastore.disableChunkSizeWarning' I 2012-04-02 00:51:53.459 com.google.walkaround.util.server.RetryHelper runBodyOnce: Body exited abnormally, run time: 5165ms W 2012-04-02 00:51:53.459 com.google.walkaround.util.server.RetryHelper run: Problem on retry 0, millis elapsed so far: 5165 com.google.walkaround.util.server.RetryHelper$RetryableFailure: com.google.appengine.api.datastore.DatastoreTimeoutException: The datastore operation timed out, or the data was temporarily unavailable. at com.google.walkaround.util.server.appengine.CheckedDatastore.safeRun(CheckedDatastore.java:458) at com.google.walkaround.util.server.appengine.CheckedDatastore.access$100(CheckedDatastore.java:67) at com.google.walkaround.util.server.appengine.CheckedDatastore$CheckedIteratorImpl.hasNext(CheckedDatastore.java:109) at com.google.walkaround.slob.server.MutationLog$DeltaIterator.hasNext(MutationLog.java:321) at com.google.walkaround.slob.server.MutationLog.reconstruct(MutationLog.java:858) at com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler$1.run(ReIndexMapper.java:77) at com.google.walkaround.util.server.RetryHelper$3.run(RetryHelper.java:182) at com.google.walkaround.util.server.RetryHelper$3.run(RetryHelper.java:180) at com.google.walkaround.util.server.RetryHelper.runBodyOnce(RetryHelper.java:142) at com.google.walkaround.util.server.RetryHelper.run(RetryHelper.java:156) at com.google.walkaround.util.server.RetryHelper.run(RetryHelper.java:180) at com.google.walkaround.wave.server.wavemanager.ReIndexMapper$Handler.process(ReIndexMapper.java:62) at com.google.walkaround.wave.server.wavemanager.ReIndexMapper.map(ReIndexMapper.java:94) at com.google.walkaround.wave.server.wavemanager.ReIndexMapper.map(ReIndexMapper.java:51) at com.google.appengine.tools.mapreduce.v2.impl.handlers.Worker.processMapper(Worker.java:111) at com.google.appengine.tools.mapreduce.v2.impl.handlers.Worker.handleMapperWorker(Worker.java:289) at com.google.appengine.tools.mapreduce.MapReduceServlet.doPost(MapReduceServlet.java:190) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.walkaround.util.server.servlet.RequestStatsFilter.doFilter(RequestStatsFilter.java:95) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.walkaround.wave.server.servlet.ServerExceptionFilter.doFilter(ServerExceptionFilter.java:108) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.appengine.tools.appstats.AppstatsFilter.doFilter(AppstatsFilter.java:141) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:102) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:249) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135) at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:446) at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:449) at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:455) at com.google.tracing.TraceContext.runInContext(TraceContext.java:695) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:333) at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:325) at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:453) at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251) at java.lang.Thread.run(Thread.java:679) Caused by: com.google.appengine.api.datastore.DatastoreTimeoutException: The datastore operation timed out, or the data was temporarily unavailable. at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:46) at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException(DatastoreApiHelper.java:76) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:106) at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72) <continued in next message> W 2012-04-02 00:51:53.459 <continued from previous message> at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33) at com.google.appengine.api.datastore.QueryResultsSourceImpl.peekQueryResultAndIfFirstRecordIndexList(QueryResultsSourceImpl.java:172) at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:108) at com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:95) at com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:161) at com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:65) at com.google.walkaround.util.server.appengine.CheckedDatastore$CheckedIteratorImpl$1.run(CheckedDatastore.java:111) at com.google.walkaround.util.server.appengine.CheckedDatastore$CheckedIteratorImpl$1.run(CheckedDatastore.java:109) at com.google.walkaround.util.server.appengine.CheckedDatastore.safeRun(CheckedDatastore.java:456) ... 82 more I 2012-04-02 00:51:53.460 com.google.walkaround.util.server.RetryHelper run: Sleeping for 3 millis
Comment #1
Posted on Apr 3, 2012 by Quick LionI hope http://code.google.com/p/walkaround/source/detail?r=ee3b32bed5ad9f58147036ea1ad6099ebbbec33f fixes this, but if not, please re-open this bug.
Status: Fixed
Labels:
Type-Defect
Priority-Medium