Using spymemcached 2.10.0

I saw this happen in the error log of a very long running service:

<pre> {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - Uncaught exception in thread {MemcachedConnection to localhost/127.0.0.1:11211} {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - java.lang.AssertionError: Unknown line type: STORED {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - at net.spy.memcached.protocol.ascii.BaseGetOpImpl.handleLine(BaseGetOpImpl.java:117) {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - at net.spy.memcached.protocol.ascii.OperationImpl.readFromBuffer(OperationImpl.java:153) {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:628) {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:543) {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:322) {MemcachedConnection to localhost/127.0.0.1:11211}] UncaughtExceptionLogger - at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:1047) - Timeout waiting for bulk values: waited 2,500 ms. Node status: Connection Status { localhost/127.0.0.1:11211 active: true, authed: true, last read: 2,501 ms ago } - net.spy.memcached.OperationTimeoutException: Timeout waiting for bulk values: waited 2,500 ms. Node status: Connection Status { localhost/127.0.0.1:11211 active: true, authed - at net.spy.memcached.MemcachedClient.getBulk(MemcachedClient.java:1462) - at net.spy.memcached.MemcachedClient.getBulk(MemcachedClient.java:1495) </pre>

After a while this happened: <pre> - Shutting down - java.lang.IllegalStateException: Shutting down - at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:1035) - at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:825) - at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:315) - at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:833) </pre>

This resulted in all future memcache operations failing until the service was restarted.

This seems like a corrupt communication state between the client and server (some request/responses being dropped or shuffled?) I suppose we can't know if the problem is in memcached or spymemcached, but there could probably be some better way of repairing itself once the problem is detected. (Fail all outstanding requests and reconnect?)