
redis - issue #503
Redis crash: networking.c:64 '!(c->flags & REDIS_CLOSE_AFTER_REPLY)' is not true
What version of Redis you are using, in what kind of Operating System? redis_version:2.2.2 Linux 2.6.32-28-generic on x86_64 Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 8 cores Ubuntu 10.04.2 Server Edition
What is the problem you are experiencing? Redis instance abnormal termination. I need to manually restart it. Data, until redis crash, are preserved (I suppose so!).
What steps will reproduce the problem? Sorry, sporadic problem after a very long run.
Do you have an INFO output? Please past it here.
See below
If it is a crash, can you please paste the stack trace that you can find in the log file or on standard output? This is really useful for us!
[1378] 29 Mar 22:46:08 - 14 clients connected (0 slaves), 2828789880 bytes in use [1378] 29 Mar 22:46:12 - Accepted 127.0.0.1:35165 [1378] 29 Mar 22:46:14 - DB 0: 1930331 keys (1929993 volatile) in 2097152 slots HT. [1378] 29 Mar 22:46:14 - DB 1: 19 keys (0 volatile) in 64 slots HT. [1378] 29 Mar 22:46:14 - 15 clients connected (0 slaves), 2829124064 bytes in use [1378] 29 Mar 22:46:15 * 1000 changes in 300 seconds. Saving... [1378] 29 Mar 22:46:15 * Background saving started by pid 21310 [1378] 29 Mar 22:46:19 - DB 0: 1930480 keys (1930142 volatile) in 2097152 slots HT. [1378] 29 Mar 22:46:19 - DB 1: 19 keys (0 volatile) in 64 slots HT. [1378] 29 Mar 22:46:19 - 15 clients connected (0 slaves), 2829598952 bytes in use [1378] 29 Mar 22:46:23 # === ASSERTION FAILED === [1378] 29 Mar 22:46:23 # ==> networking.c:64 '!(c->flags & REDIS_CLOSE_AFTER_REPLY)' is not true [1378] 29 Mar 22:46:23 # (forcing SIGSEGV in order to print the stack trace) [1378] 29 Mar 22:46:23 # ======= Ooops! Redis 2.2.2 got signal: -11- ======= [1378] 29 Mar 22:46:23 # redis_version:2.2.2 redis_git_sha1:00000000 redis_git_dirty:0 arch_bits:64 multiplexing_api:epoll process_id:1378 uptime_in_seconds:19719 uptime_in_days:0 lru_clock:119734 used_cpu_sys:787.66 used_cpu_user:1622.44 used_cpu_sys_childrens:480.03 used_cpu_user_childrens:97.46 connected_clients:15 connected_slaves:0 client_longest_output_list:0 client_biggest_input_buf:219 blocked_clients:4 used_memory:2829745072 used_memory_human:2.64G used_memory_rss:3758010368 mem_fragmentation_ratio:1.33 use_tcmalloc:0 loading:0 aof_enabled:0 changes_since_last_save:1030520 bgsave_in_progress:1 last_save_time:1301431274 bgrewriteaof_in_progress:0 total_connections_received:986 total_commands_processed:105574691 expired_keys:1931601 evicted_keys:0 keyspace_hits:52793903 keyspace_misses:44007253 hash_max_zipmap_entries:512 hash_max_zipmap_value:64 pubsub_channels:0 pubsub_patterns:0 vm_enabled:0 role:master allocation_stats:2=659398,6=2,7=2,8=1248492,9=1553916,10=45227905,11=7630420,12= [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(_redisAssert+0x56) [0x42d256] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(_redisAssert+0x56) [0x42d256] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(_installWriteEvent+0x7e) [0x41793e] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(addReply+0x19) [0x4183f9] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(replicationFeedMonitors+0x114) [0x41c1d4] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(call+0x55) [0x40f4a5] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(processCommand+0x267) [0x40f787] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(processInputBuffer+0x57) [0x418247] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(readQueryFromClient+0x5d) [0x4182fd] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(aeProcessEvents+0x153) [0x40c053] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(aeMain+0x2e) [0x40c29e] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server(main+0xf7) [0x411307] [1378] 29 Mar 22:46:23 # /lib/libc.so.6(__libc_start_main+0xfd) [0x7fdc9eee6c4d] [1378] 29 Mar 22:46:23 # /usr/local/bin/redis-server() [0x40b5c9] [21310] 29 Mar 22:46:48 * DB saved on disk
Please provide any additional information below.
Comment #1
Posted on Mar 31, 2011 by Grumpy DogHello, this is a bug indeed, related to the MONITOR command.
After a client is in MONITOR state, if QUIT is called against this connection the flag REDIS_CLOSE_AFTER_REPLY is set on the connection, but still as an effect of other clients doing other work some new data may be written against this client, triggering the assert.
Fixing it into unstable and 2.2 branch in minutes, and then reporting back here.
Comment #2
Posted on Mar 31, 2011 by Grumpy DogFixed in both 2.2 and unstable branch. Fix rationale: the assert was not enforceable due to MONITOR, Pub/Sub, and other systems that will try to append data even after a QUIT or Protocol Error. So the assert was removed and conditionals added in the low level code to append to the client output buffer so that further output will be discarded if the close-after-reply flag is set.
I tested the new code and can no longer crash Redis.
Salvatore
Status: Fixed
Labels:
Type-Defect
Priority-Medium