What version of Redis you are using, in what kind of Operating System?

Redis 2.2.8 32bit mode

4 x Dual-Core AMD Opteron(tm) Processor 2214 HE

16GB RAM

uname -a

Linux XXXXX 2.6.18-238.9.1.el5 #1 SMP Fri Mar 18 12:42:39 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release

Red Hat Enterprise Linux Server release 5.6 (Tikanga)

What is the problem you are experiencing?

Possible memory leak using MONITOR command.

I've built a program that opens a TCP connection and sends "MONITOR\r\n". It reads every line and performs some processing. The processing is a bit slow, so I think that there are a lot of things that redis-server is trying to send to the client but the client does not read fast enough.

When this happens, the used_memory_rss grows and never decreases. The used_memory/used_memory_human does not changes.

What steps will reproduce the problem?

Send the "MONITOR" command from a slow client to a very fast server so that the client cant keep up the pace of reading as fast as the server is sending.

Do you have an INFO output? Please past it here.

redis_version:2.2.8 redis_git_sha1:00000000 redis_git_dirty:0 arch_bits:32 multiplexing_api:epoll process_id:18479 uptime_in_seconds:1060 uptime_in_days:0 lru_clock:793616 used_cpu_sys:4.96 used_cpu_user:2.78 used_cpu_sys_childrens:3.81 used_cpu_user_childrens:1.04 connected_clients:38 connected_slaves:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 used_memory:56662460 used_memory_human:54.04M used_memory_rss:67694592 mem_fragmentation_ratio:1.19 use_tcmalloc:0 loading:0 aof_enabled:0 changes_since_last_save:0 bgsave_in_progress:0 last_save_time:1308169970 bgrewriteaof_in_progress:0 total_connections_received:171 total_commands_processed:438735 expired_keys:0 evicted_keys:0 keyspace_hits:229178 keyspace_misses:0 hash_max_zipmap_entries:8 hash_max_zipmap_value:2048 pubsub_channels:0 pubsub_patterns:0 vm_enabled:0 role:master allocation_stats:2=208949,4=629,6=1,8=160,9=1596138,10=992439,11=193,12=5720653,13=38622,14=519522,15=524079,16=1822441,17=1657700,18=921,19=361145,20=1504,21=144175,22=1111,23=3684,24=20614,25=1984,26=1847,27=3391,28=1852,29=2481,30=2515,31=2341,32=90618,33=3620,34=9513,35=3778,36=4277,37=4607,38=6343,39=5845,40=33738,41=6275,42=7174,43=50742,44=8057,45=8669,46=8520,47=9104,48=9144,49=9269,50=10427,51=10136,52=10011,53=10877,54=11623,55=12777,56=11350,57=12320,58=12401,59=12066,60=13071,61=12930,62=12835,63=12626,64=14865,65=13660,66=12196,67=12559,68=62470,69=14594,70=15590,71=17682,72=15743,73=10625,74=11708,75=11840,76=11269,77=9098,78=9093,79=8689,80=9762,81=9598,82=11505,83=7355,84=8502,85=8202,86=7214,87=8233,88=7233,89=6868,90=8082,91=7725,92=11716,93=7825,94=8940,95=11650,96=12187,97=9209,98=9165,99=7919,100=20182,101=21832,102=29795,103=7739,104=7747,105=7790,106=8043,107=6807,108=8841,109=6706,110=5470,111=6008,112=5272,113=4645,114=8918,115=4629,116=5192,117=7502,118=7721,119=6683,120=5843,121=4502,122=4651,123=4955,124=42831,125=4938,126=74052,127=4752,128=6146,129=5451,130=5697,131=4170,132=4485,133=3901,134=3700,135=3630,136=3689,137=3570,138=3505,139=7110,140=3359,141=3410,142=6242,143=3129,144=3209,145=3033,146=2965,147=2933,148=4298,149=5102,150=2817,151=2692,152=2658,153=2726,154=11849,155=5418,156=2939,157=2964,158=3476,159=3497,160=2593,161=2642,162=2880,163=2222,164=2692,165=2310,166=2207,167=2262,168=2213,169=2206,170=2106,171=2129,172=2212,173=2083,174=2063,175=2183,176=2174,177=2123,178=2364,179=2864,180=2566,181=2662,182=3636,183=3913,184=2088,185=2156,186=2067,187=1860,188=1806,189=1778,190=1761,191=1726,192=1517,193=1668,194=1589,195=1521,196=1505,197=1597,198=1513,199=1459,200=1540,201=1485,202=1408,203=1400,204=1542,205=1561,206=1478,207=1495,208=1482,209=1501,210=4193,211=1490,212=1341,213=1395,214=1406,215=1493,216=1479,217=1470,218=1555,219=1646,220=1626,221=1554,222=1767,223=1819,224=2620,225=7290,226=1837,227=2387,228=1645,229=2320,230=2348,231=1705,232=1728,233=2116,234=1846,235=1816,236=1721,237=1791,238=1775,239=1954,240=1830,241=1987,242=1939,243=1971,244=1880,245=1920,246=1951,247=1904,248=1901,249=1823,250=1888,251=1911,252=1749,253=1614,254=1787,255=2280,>=256=111474 db0:keys=208949,expires=0

I have a custom built tool called "redis-top" that show me some info of my redis cluster.

Before the "leak":

INSTANCE PING db0:keys mem mem_real clients reqs/s role 10.1.2.201:63700 0.188ms 208949 54.03M 64.56M 37 20 master 10.1.2.201:63701 0.143ms 209183 54.12M 61.21M 37 15 master 10.1.2.201:63702 0.144ms 208997 53.95M 60.99M 37 15 master 10.1.2.201:63703 0.136ms 209024 54.08M 64.66M 37 14 master 10.1.2.201:63704 0.140ms 208770 53.95M 60.99M 38 13 master 10.1.2.201:63710 0.138ms 208084 53.86M 60.88M 37 20 master 10.1.2.201:63711 0.141ms 208589 53.97M 61.08M 38 14 slave 10.1.2.201:63712 0.138ms 208369 53.96M 61.07M 38 20 slave 10.1.2.201:63713 0.142ms 208297 53.91M 61.02M 38 13 slave 10.1.2.201:63714 0.141ms 208285 53.91M 61.01M 39 15 slave 10.1.2.202:63700 0.131ms 209128 54.04M 61.09M 37 15 master 10.1.2.202:63701 0.111ms 208146 53.83M 60.85M 37 20 master 10.1.2.202:63702 0.135ms 208973 54.04M 61.09M 37 14 master 10.1.2.202:63703 0.134ms 209071 54.12M 61.16M 37 17 master 10.1.2.202:63704 0.136ms 209366 54.18M 61.23M 37 19 master 10.1.2.202:63710 0.129ms 208949 54.04M 61.15M 38 22 slave 10.1.2.202:63711 0.134ms 209183 54.13M 61.27M 38 12 slave 10.1.2.202:63712 0.157ms 208997 53.96M 61.07M 38 18 slave 10.1.2.202:63713 0.125ms 209024 54.09M 61.22M 38 12 slave 10.1.2.202:63714 0.141ms 208770 53.95M 61.07M 38 14 slave 10.1.2.203:63700 0.118ms 208271 53.89M 61.12M 37 16 master 10.1.2.203:63701 0.115ms 209339 54.25M 64.88M 37 15 master 10.1.2.203:63702 0.116ms 209128 54.12M 61.16M 37 14 master 10.1.2.203:63703 0.130ms 208260 53.88M 60.88M 37 13 master 10.1.2.203:63704 0.117ms 208562 54.03M 64.57M 39 16 master 10.1.2.203:63710 0.139ms 209128 54.04M 61.18M 38 17 slave 10.1.2.203:63711 0.117ms 208146 53.85M 60.96M 39 18 slave 10.1.2.203:63712 0.115ms 208973 54.05M 61.16M 38 16 slave 10.1.2.203:63713 0.112ms 209071 54.13M 61.25M 39 20 slave 10.1.2.203:63714 0.115ms 209366 54.19M 61.32M 38 18 slave 10.1.2.204:63700 0.141ms 208615 53.96M 64.43M 37 12 master 10.1.2.204:63701 0.134ms 209099 54.01M 61.05M 37 15 master 10.1.2.204:63702 0.118ms 208322 53.92M 60.96M 37 18 master 10.1.2.204:63703 0.131ms 210035 54.31M 61.39M 37 15 master 10.1.2.204:63704 0.116ms 208871 54.01M 61.06M 37 12 master 10.1.2.204:63710 0.137ms 208271 53.89M 60.98M 38 13 slave 10.1.2.204:63711 0.117ms 209339 54.25M 61.39M 38 15 slave 10.1.2.204:63712 0.131ms 209128 54.13M 61.27M 38 16 slave 10.1.2.204:63713 0.116ms 208260 53.88M 61.00M 38 12 slave 10.1.2.204:63714 0.153ms 208562 54.02M 61.13M 38 17 slave 10.1.2.205:63700 0.116ms 209078 54.07M 61.14M 37 13 master 10.1.2.205:63701 0.114ms 209411 54.13M 61.18M 38 15 master 10.1.2.205:63702 0.118ms 208898 54.00M 64.54M 37 15 master 10.1.2.205:63703 0.115ms 208243 53.82M 64.34M 37 16 master 10.1.2.205:63704 0.115ms 208580 53.98M 64.49M 37 17 master 10.1.2.205:63710 0.116ms 208615 53.96M 61.07M 38 15 slave 10.1.2.205:63711 0.117ms 209099 54.02M 61.16M 38 11 slave 10.1.2.205:63712 0.115ms 208322 53.94M 61.06M 40 19 slave 10.1.2.205:63713 0.130ms 210035 54.32M 61.48M 38 16 slave 10.1.2.205:63714 0.115ms 208871 54.02M 61.16M 38 16 slave 10.1.2.206:63700 0.141ms 207954 53.79M 60.81M 38 17 master 10.1.2.206:63701 0.136ms 209282 54.21M 61.31M 37 13 master 10.1.2.206:63702 0.136ms 208520 53.88M 64.41M 37 13 master 10.1.2.206:63703 0.136ms 208758 54.00M 64.54M 37 18 master 10.1.2.206:63704 0.139ms 208926 54.07M 64.59M 37 19 master 10.1.2.206:63710 0.115ms 209078 54.08M 61.20M 38 12 slave 10.1.2.206:63711 0.134ms 209411 54.13M 61.28M 38 15 slave 10.1.2.206:63712 0.138ms 208898 54.01M 61.16M 38 13 slave 10.1.2.206:63713 0.138ms 208243 53.83M 60.95M 39 17 slave 10.1.2.206:63714 0.136ms 208580 53.98M 61.12M 38 16 slave 10.1.2.207:63700 0.102ms 208555 53.98M 61.27M 38 15 master 10.1.2.207:63701 0.063ms 208060 53.85M 60.89M 37 17 master 10.1.2.207:63702 0.062ms 209535 54.22M 64.79M 37 18 master 10.1.2.207:63703 0.061ms 209087 53.98M 61.05M 37 15 master 10.1.2.207:63704 0.066ms 209219 54.00M 64.52M 37 15 master 10.1.2.207:63710 0.062ms 207954 53.79M 60.89M 38 15 slave 10.1.2.207:63711 0.061ms 209282 54.22M 61.34M 38 14 slave 10.1.2.207:63712 0.063ms 208520 53.89M 60.98M 38 17 slave 10.1.2.207:63713 0.084ms 208758 54.00M 61.12M 38 19 slave 10.1.2.207:63714 0.085ms 208926 54.07M 61.18M 38 17 slave 10.1.2.208:63700 0.181ms 208715 53.93M 64.64M 38 16 master 10.1.2.208:63701 0.135ms 208707 54.03M 61.07M 38 16 master 10.1.2.208:63702 0.127ms 209265 54.17M 61.23M 37 16 master 10.1.2.208:63703 0.130ms 208574 53.97M 61.03M 37 18 master 10.1.2.208:63704 0.130ms 209065 54.04M 61.09M 37 15 master 10.1.2.208:63710 0.116ms 208555 53.98M 61.09M 38 17 slave 10.1.2.208:63711 0.117ms 208060 53.87M 60.96M 39 15 slave 10.1.2.208:63712 0.129ms 209535 54.23M 61.37M 39 18 slave 10.1.2.208:63713 0.115ms 209087 53.98M 61.11M 38 17 slave 10.1.2.208:63714 0.134ms 209219 54.01M 61.13M 38 19 slave 10.1.2.209:63700 0.154ms 209314 54.12M 64.69M 37 17 master 10.1.2.209:63701 0.133ms 208102 53.88M 64.39M 37 18 master 10.1.2.209:63702 0.139ms 208628 53.95M 64.46M 37 13 master 10.1.2.209:63703 0.143ms 208303 53.96M 64.48M 37 16 master 10.1.2.209:63704 0.135ms 208321 53.87M 64.32M 37 14 master 10.1.2.209:63710 0.129ms 208715 53.92M 61.04M 38 14 slave 10.1.2.209:63711 0.133ms 208707 54.03M 61.13M 38 14 slave 10.1.2.209:63712 0.143ms 209265 54.18M 61.29M 38 12 slave 10.1.2.209:63713 0.133ms 208574 53.98M 61.09M 38 20 slave 10.1.2.209:63714 0.133ms 209065 54.05M 61.16M 39 15 slave 10.1.2.210:63700 0.207ms 207991 53.76M 60.79M 37 16 master 10.1.2.210:63701 0.136ms 209199 54.12M 61.18M 37 14 master 10.1.2.210:63702 0.136ms 209260 54.12M 61.17M 37 12 master 10.1.2.210:63703 0.138ms 209156 54.08M 61.13M 38 19 master 10.1.2.210:63704 0.136ms 209348 54.10M 61.15M 37 13 master 10.1.2.210:63705 0.136ms 209205 54.05M 61.11M 37 11 master 10.1.2.210:63706 0.145ms 208738 53.94M 60.95M 37 16 master 10.1.2.210:63710 0.115ms 209314 54.12M 61.25M 38 15 slave 10.1.2.210:63711 0.142ms 208102 53.89M 60.97M 38 17 slave 10.1.2.210:63712 0.114ms 208628 53.96M 61.07M 38 14 slave 10.1.2.210:63713 0.136ms 208303 53.96M 61.06M 38 18 slave 10.1.2.210:63714 0.134ms 208321 53.88M 60.97M 39 18 slave 10.1.2.211:63700 0.115ms 208863 54.12M 61.17M 37 16 master 10.1.2.211:63701 0.114ms 208752 53.98M 64.53M 37 16 master 10.1.2.211:63702 0.119ms 209382 54.17M 64.72M 37 16 master 10.1.2.211:63703 0.115ms 209146 54.04M 64.53M 37 16 master 10.1.2.211:63704 0.115ms 208578 53.93M 64.50M 37 18 master 10.1.2.211:63705 0.123ms 208672 53.99M 64.58M 37 14 master 10.1.2.211:63706 0.115ms 208115 53.89M 64.40M 37 12 master 10.1.2.211:63710 0.111ms 207991 53.77M 60.87M 38 12 slave 10.1.2.211:63711 0.121ms 209199 54.13M 61.26M 38 17 slave 10.1.2.211:63712 0.115ms 209260 54.13M 61.25M 39 12 slave 10.1.2.211:63713 0.115ms 209156 54.08M 61.20M 38 16 slave 10.1.2.211:63714 0.117ms 209348 54.10M 61.23M 38 13 slave 10.1.2.211:63715 0.115ms 209205 54.06M 61.18M 39 14 slave 10.1.2.211:63716 0.115ms 208738 53.94M 61.05M 38 18 slave 10.1.2.212:63700 0.117ms 208084 53.87M 61.02M 38 12 slave 10.1.2.212:63701 0.117ms 208589 53.96M 61.00M 37 19 master 10.1.2.212:63702 0.116ms 208369 53.96M 64.47M 37 17 master 10.1.2.212:63703 0.118ms 208297 53.90M 64.40M 37 10 master 10.1.2.212:63704 0.116ms 208285 53.89M 64.47M 37 16 master 10.1.2.212:63710 0.115ms 208863 54.13M 61.25M 38 15 slave 10.1.2.212:63711 0.118ms 208752 53.99M 61.11M 38 16 slave 10.1.2.212:63712 0.114ms 209382 54.17M 61.32M 38 14 slave 10.1.2.212:63713 0.116ms 209146 54.05M 61.18M 39 17 slave 10.1.2.212:63714 0.115ms 208578 53.94M 61.03M 38 17 slave 10.1.2.212:63715 0.113ms 208672 54.00M 61.11M 39 13 slave 10.1.2.212:63716 0.114ms 208115 53.91M 60.99M 39 15 slave

AVERAGE: 0.124ms 208800 54.01M 61.86M 37 15 -

TOTAL: 15.879ms 26726476 6.75G 7.73G 4824 1992 master[64]/slave[64]

After the "leak":

INSTANCE PING db0:keys mem mem_real clients reqs/s role 10.1.2.201:63700 0.182ms 208921 54.03M 235.95M 37 27 master 10.1.2.201:63701 0.139ms 209155 54.13M 205.77M 37 16 master 10.1.2.201:63702 0.133ms 208970 53.95M 233.67M 37 18 master 10.1.2.201:63703 0.137ms 209016 54.09M 227.52M 37 16 master 10.1.2.201:63704 0.133ms 208751 53.94M 235.67M 37 21 master 10.1.2.201:63710 0.136ms 208030 53.86M 253.34M 38 18 slave 10.1.2.201:63711 0.134ms 208534 53.96M 281.63M 38 20 slave 10.1.2.201:63712 0.135ms 208327 53.97M 254.35M 38 22 slave 10.1.2.201:63713 0.134ms 208259 53.91M 254.46M 38 19 slave 10.1.2.201:63714 0.134ms 208230 53.90M 259.59M 38 19 slave 10.1.2.202:63700 0.177ms 209075 54.03M 229.72M 37 17 master 10.1.2.202:63701 0.131ms 208073 53.82M 185.42M 37 28 master 10.1.2.202:63702 0.131ms 208928 54.04M 237.16M 37 16 master 10.1.2.202:63703 0.129ms 209026 54.12M 234.61M 38 15 master 10.1.2.202:63704 0.135ms 209317 54.17M 179.53M 37 15 master 10.1.2.202:63710 0.165ms 208921 54.05M 233.37M 39 29 slave 10.1.2.202:63711 0.134ms 209155 54.13M 226.41M 38 16 slave 10.1.2.202:63712 0.128ms 208970 53.96M 232.53M 38 20 slave 10.1.2.202:63713 0.130ms 209016 54.10M 229.45M 38 18 slave 10.1.2.202:63714 0.130ms 208751 53.95M 230.12M 38 16 slave 10.1.2.203:63700 0.173ms 208196 53.87M 266.04M 37 16 master 10.1.2.203:63701 0.108ms 209285 54.24M 252.48M 37 18 master 10.1.2.203:63702 0.108ms 209047 54.11M 251.21M 37 13 master 10.1.2.203:63703 0.108ms 208195 53.87M 253.29M 37 19 master 10.1.2.203:63704 0.127ms 208501 54.01M 274.39M 38 14 master 10.1.2.203:63710 0.127ms 209075 54.04M 198.55M 39 16 slave 10.1.2.203:63711 0.108ms 208073 53.82M 182.88M 38 22 slave 10.1.2.203:63712 0.127ms 208928 54.05M 181.68M 38 15 slave 10.1.2.203:63713 0.126ms 209026 54.12M 199.93M 38 18 slave 10.1.2.203:63714 0.110ms 209317 54.18M 182.34M 38 16 slave 10.1.2.204:63700 0.110ms 208554 53.95M 232.17M 37 16 master 10.1.2.204:63701 0.110ms 209045 54.01M 208.48M 38 14 master 10.1.2.204:63702 0.119ms 208280 53.93M 238.91M 38 32 master 10.1.2.204:63703 0.110ms 209972 54.30M 225.23M 37 24 master 10.1.2.204:63704 0.119ms 208801 54.00M 241.37M 37 15 master 10.1.2.204:63710 0.115ms 208196 53.87M 252.14M 38 20 slave 10.1.2.204:63711 0.109ms 209285 54.24M 250.15M 38 14 slave 10.1.2.204:63712 0.109ms 209047 54.11M 252.90M 38 13 slave 10.1.2.204:63713 0.112ms 208195 53.87M 243.80M 38 14 slave 10.1.2.204:63714 0.109ms 208501 54.01M 264.73M 38 19 slave 10.1.2.205:63700 0.129ms 209048 54.07M 237.50M 37 23 master 10.1.2.205:63701 0.129ms 209379 54.13M 241.97M 37 16 master 10.1.2.205:63702 0.108ms 208856 54.00M 232.32M 37 15 master 10.1.2.205:63703 0.129ms 208209 53.82M 245.21M 38 13 master 10.1.2.205:63704 0.129ms 208540 53.98M 238.80M 38 15 master 10.1.2.205:63710 0.129ms 208554 53.95M 231.04M 38 21 slave 10.1.2.205:63711 0.107ms 209045 54.01M 235.06M 38 16 slave 10.1.2.205:63712 0.125ms 208280 53.92M 220.96M 38 35 slave 10.1.2.205:63713 0.129ms 209972 54.31M 240.30M 38 23 slave 10.1.2.205:63714 0.128ms 208801 54.01M 238.94M 38 17 slave 10.1.2.206:63700 0.165ms 207893 53.78M 181.81M 38 17 master 10.1.2.206:63701 0.130ms 209228 54.21M 171.57M 37 19 master 10.1.2.206:63702 0.129ms 208459 53.87M 200.45M 37 16 master 10.1.2.206:63703 0.130ms 208667 53.99M 167.21M 38 18 master 10.1.2.206:63704 0.130ms 208870 54.07M 236.81M 38 20 master 10.1.2.206:63710 0.135ms 209048 54.08M 210.68M 39 19 slave 10.1.2.206:63711 0.136ms 209379 54.13M 235.06M 38 21 slave 10.1.2.206:63712 0.129ms 208856 54.01M 216.14M 38 15 slave 10.1.2.206:63713 0.130ms 208209 53.83M 232.78M 39 20 slave 10.1.2.206:63714 0.130ms 208540 53.98M 234.30M 38 14 slave 10.1.2.207:63700 0.136ms 208534 53.98M 221.18M 37 15 master 10.1.2.207:63701 0.076ms 208019 53.86M 229.02M 38 18 master 10.1.2.207:63702 0.055ms 209484 54.22M 221.63M 37 14 master 10.1.2.207:63703 0.056ms 209032 53.97M 222.16M 37 16 master 10.1.2.207:63704 0.052ms 209176 54.00M 240.24M 37 21 master 10.1.2.207:63710 0.104ms 207893 53.78M 181.30M 38 17 slave 10.1.2.207:63711 0.060ms 209228 54.21M 198.84M 38 16 slave 10.1.2.207:63712 0.063ms 208459 53.88M 197.96M 38 15 slave 10.1.2.207:63713 0.055ms 208667 53.99M 175.25M 38 15 slave 10.1.2.207:63714 0.055ms 208870 54.07M 188.54M 38 17 slave 10.1.2.208:63700 0.143ms 208680 53.92M 215.43M 37 17 master 10.1.2.208:63701 0.142ms 208677 54.03M 221.60M 38 19 master 10.1.2.208:63702 0.109ms 209197 54.16M 230.70M 38 20 master 10.1.2.208:63703 0.108ms 208505 53.97M 230.25M 38 18 master 10.1.2.208:63704 0.110ms 209021 54.03M 241.14M 37 20 master 10.1.2.208:63710 0.126ms 208534 53.99M 220.72M 39 14 slave 10.1.2.208:63711 0.110ms 208019 53.86M 224.62M 38 18 slave 10.1.2.208:63712 0.109ms 209484 54.23M 222.55M 38 16 slave 10.1.2.208:63713 0.123ms 209032 53.99M 216.52M 39 17 slave 10.1.2.208:63714 0.126ms 209176 54.00M 222.39M 38 22 slave 10.1.2.209:63700 0.172ms 209258 54.11M 250.35M 37 20 master 10.1.2.209:63701 0.134ms 208029 53.88M 254.50M 39 35 master 10.1.2.209:63702 0.134ms 208590 53.96M 250.89M 38 17 master 10.1.2.209:63703 0.130ms 208238 53.94M 243.12M 37 18 master 10.1.2.209:63704 0.131ms 208270 53.87M 229.43M 38 13 master 10.1.2.209:63710 0.133ms 208680 53.93M 218.01M 39 17 slave 10.1.2.209:63711 0.138ms 208677 54.03M 218.97M 38 14 slave 10.1.2.209:63712 0.130ms 209197 54.16M 227.55M 38 19 slave 10.1.2.209:63713 0.132ms 208505 53.97M 220.27M 38 13 slave 10.1.2.209:63714 0.132ms 209021 54.05M 219.36M 39 18 slave 10.1.2.210:63700 0.108ms 207878 53.74M 255.69M 38 15 master 10.1.2.210:63701 0.108ms 209065 54.09M 331.42M 37 17 master 10.1.2.210:63702 0.109ms 209139 54.08M 255.67M 37 18 master 10.1.2.210:63703 0.126ms 209023 54.03M 254.51M 37 15 master 10.1.2.210:63704 0.108ms 209224 54.08M 326.20M 38 18 master 10.1.2.210:63705 0.122ms 209079 54.02M 279.91M 37 11 master 10.1.2.210:63706 0.123ms 208619 53.92M 254.12M 39 15 master 10.1.2.210:63710 0.106ms 209258 54.12M 231.06M 38 20 slave 10.1.2.210:63711 0.120ms 208029 53.87M 250.35M 38 36 slave 10.1.2.210:63712 0.181ms 208590 53.96M 246.51M 38 13 slave 10.1.2.210:63713 0.107ms 208238 53.95M 245.25M 38 17 slave 10.1.2.210:63714 0.126ms 208270 53.88M 228.40M 39 16 slave 10.1.2.211:63700 0.125ms 208805 54.12M 249.58M 38 22 master 10.1.2.211:63701 0.109ms 208707 53.98M 250.25M 37 15 master 10.1.2.211:63702 0.128ms 209337 54.17M 327.93M 38 26 master 10.1.2.211:63703 0.130ms 209074 54.02M 252.71M 37 15 master 10.1.2.211:63704 0.107ms 208515 53.93M 250.50M 38 13 master 10.1.2.211:63705 0.106ms 208626 53.98M 248.28M 37 20 master 10.1.2.211:63706 0.106ms 208073 53.89M 242.56M 37 18 master 10.1.2.211:63710 0.106ms 207878 53.74M 277.07M 39 18 slave 10.1.2.211:63711 0.118ms 209065 54.09M 256.86M 38 17 slave 10.1.2.211:63712 0.158ms 209139 54.10M 254.02M 39 14 slave 10.1.2.211:63713 0.108ms 209023 54.05M 249.58M 39 14 slave 10.1.2.211:63714 0.108ms 209224 54.07M 252.80M 38 17 slave 10.1.2.211:63715 0.097ms 209079 54.02M 252.08M 38 14 slave 10.1.2.211:63716 0.109ms 208619 53.91M 250.79M 38 19 slave 10.1.2.212:63700 0.109ms 208030 53.86M 281.98M 37 17 master 10.1.2.212:63701 0.111ms 208534 53.95M 251.40M 37 20 master 10.1.2.212:63702 0.122ms 208327 53.96M 252.13M 37 23 master 10.1.2.212:63703 0.106ms 208259 53.90M 254.01M 37 18 master 10.1.2.212:63704 0.108ms 208230 53.89M 254.90M 37 20 master 10.1.2.212:63710 0.106ms 208805 54.12M 277.70M 38 20 slave 10.1.2.212:63711 0.108ms 208707 53.98M 251.38M 38 15 slave 10.1.2.212:63712 0.104ms 209337 54.17M 279.05M 38 28 slave 10.1.2.212:63713 0.107ms 209074 54.03M 253.14M 38 16 slave 10.1.2.212:63714 0.108ms 208515 53.93M 248.76M 38 18 slave 10.1.2.212:63715 0.105ms 208626 54.00M 272.80M 39 15 slave 10.1.2.212:63716 0.105ms 208073 53.90M 247.99M 39 19 slave

AVERAGE: 0.120ms 208742 54.00M 236.84M 37 18 -

TOTAL: 15.359ms 26719022 6.75G 29.61G 4838 2318 master[64]/slave[64]

It smells a lot like your MONITOR client isn't reading fast enough or your network connection isn't fast enough. To verify, try running "redis-cli MONITOR > /dev/null".

This sounds like the same bug I reported in 525. http://code.google.com/p/redis/issues/detail?id=525 Redis doesn't limit the amount of memory allocated for client buffers, which would make sense that I see it on slow pub/sub clients and you see it with monitor (a kind of subscription).

Yeah, my client may not be reading fast enough, but that does mean that Redis should allocate a large amount of memory and never free it afterward...

This is caused by memory fragmentation. Reclaiming memory after the top of the heap is grown is very difficult, as new allocations may have been done there. Redis itself can't do anything about it, but using a different allocator can decrease fragmentation drastically. We've seen good reports of people using jemalloc. There is a separate branch available that includes jemalloc support: https://github.com/antirez/redis/tree/2.2-jemalloc-static. It is not 100% up to date with the latest 2.2, but good enough (unless you can tell from the changelog that you need the latest 2.2, in that case I can update that branch for you). Can you report back here if Redis with jemalloc has better results in your scenario? Thanks.

Using the jemalloc branch:

INSTANCE PING db0:keys mem mem_real clients reqs/s role 10.1.2.201:63700 0.182ms 208769 256.53M 228.65M 41 22 master 10.1.2.201:63701 0.144ms 208959 257.31M 228.88M 41 15 master 10.1.2.201:63702 0.176ms 208831 256.19M 226.74M 42 12 master 10.1.2.201:63703 0.140ms 208767 256.76M 234.09M 40 14 master 10.1.2.201:63704 0.138ms 208590 256.23M 227.94M 41 16 master 10.1.2.201:63710 0.143ms 207943 218.26M 192.63M 40 16 slave 10.1.2.201:63711 0.145ms 208386 217.91M 205.88M 40 14 slave 10.1.2.201:63712 0.123ms 208219 218.37M 191.70M 43 18 slave 10.1.2.201:63713 0.140ms 208123 218.18M 193.78M 41 13 slave 10.1.2.201:63714 0.147ms 208095 218.06M 194.16M 40 15 slave 10.1.2.202:63700 0.140ms 208920 257.21M 231.10M 41 14 master 10.1.2.202:63701 0.137ms 207886 256.05M 227.17M 41 17 master 10.1.2.202:63702 0.141ms 208815 256.91M 228.70M 41 16 master 10.1.2.202:63703 0.132ms 208891 257.10M 225.17M 42 13 master 10.1.2.202:63704 0.141ms 209182 258.14M 229.68M 39 15 master 10.1.2.202:63710 0.134ms 208769 219.70M 194.41M 44 20 slave 10.1.2.202:63711 0.134ms 208959 219.02M 195.18M 41 11 slave 10.1.2.202:63712 0.128ms 208831 218.85M 191.28M 42 16 slave 10.1.2.202:63713 0.139ms 208767 218.98M 201.70M 43 16 slave 10.1.2.202:63714 0.141ms 208590 218.41M 193.52M 42 14 slave 10.1.2.203:63700 0.135ms 208014 218.13M 188.89M 42 14 slave 10.1.2.203:63701 0.123ms 209169 220.61M 194.12M 40 15 slave 10.1.2.203:63702 0.104ms 208939 219.23M 190.57M 44 19 slave 10.1.2.203:63703 0.125ms 208072 218.51M 191.22M 41 14 slave 10.1.2.203:63704 0.124ms 208414 217.82M 194.94M 40 15 slave 10.1.2.203:63710 0.123ms 208920 218.44M 198.80M 41 15 slave 10.1.2.203:63711 0.125ms 207886 217.50M 192.65M 42 23 slave 10.1.2.203:63712 0.127ms 208815 218.93M 194.02M 41 13 slave 10.1.2.203:63713 0.132ms 208891 219.19M 194.10M 43 11 slave 10.1.2.203:63714 0.130ms 209182 219.21M 194.75M 43 13 slave 10.1.2.204:63700 0.129ms 208450 218.23M 196.43M 41 14 slave 10.1.2.204:63701 0.131ms 208960 218.76M 195.88M 42 13 slave 10.1.2.204:63702 0.121ms 208211 218.71M 195.11M 43 19 slave 10.1.2.204:63703 0.103ms 209791 258.13M 227.56M 42 13 master 10.1.2.204:63704 0.124ms 208643 218.75M 192.89M 42 11 slave 10.1.2.204:63710 0.123ms 208014 255.88M 224.86M 40 15 master 10.1.2.204:63711 0.125ms 209169 257.53M 225.18M 43 16 master 10.1.2.204:63712 0.123ms 208939 257.19M 226.00M 41 19 master 10.1.2.204:63713 0.127ms 208072 255.37M 222.70M 41 13 master 10.1.2.204:63714 0.129ms 208414 256.64M 227.93M 42 14 master 10.1.2.205:63700 0.151ms 208903 219.09M 196.53M 42 16 slave 10.1.2.205:63701 0.173ms 209233 219.32M 195.23M 42 13 slave 10.1.2.205:63702 0.149ms 208794 218.79M 194.59M 41 16 slave 10.1.2.205:63703 0.316ms 208086 217.72M 195.69M 41 15 slave 10.1.2.205:63704 0.152ms 208391 217.91M 205.69M 40 16 slave 10.1.2.205:63710 0.154ms 208450 255.35M 228.25M 40 12 master 10.1.2.205:63711 0.145ms 208960 256.13M 227.52M 42 14 master 10.1.2.205:63712 0.159ms 208211 255.59M 226.98M 44 17 master 10.1.2.205:63713 0.140ms 209791 219.72M 195.15M 41 16 slave 10.1.2.205:63714 0.150ms 208643 256.57M 224.08M 41 12 master 10.1.2.206:63700 0.147ms 207725 256.21M 225.64M 32 10 master 10.1.2.206:63701 0.140ms 208958 256.56M 228.36M 39 9 master 10.1.2.206:63702 0.141ms 208266 256.04M 226.17M 39 11 master 10.1.2.206:63703 0.899ms 208515 256.55M 224.41M 39 12 master 10.1.2.206:63704 0.637ms 208674 256.82M 228.78M 40 16 master 10.1.2.206:63710 0.208ms 208903 256.67M 228.51M 39 16 master 10.1.2.206:63711 0.199ms 209233 256.87M 226.51M 41 17 master 10.1.2.206:63712 0.222ms 208794 257.05M 224.91M 35 9 master 10.1.2.206:63713 0.261ms 208086 255.77M 227.80M 39 14 master 10.1.2.206:63714 0.141ms 208391 256.61M 236.82M 39 15 master 10.1.2.207:63700 0.111ms 208448 219.07M 197.62M 42 17 slave 10.1.2.207:63701 0.077ms 207825 217.59M 200.59M 41 15 slave 10.1.2.207:63702 0.067ms 209181 219.37M 197.64M 40 16 slave 10.1.2.207:63703 0.070ms 208890 218.27M 201.18M 40 13 slave 10.1.2.207:63704 0.061ms 208991 218.85M 196.86M 42 22 slave 10.1.2.207:63710 0.072ms 207725 217.88M 196.32M 42 15 slave 10.1.2.207:63711 0.069ms 208958 219.09M 196.71M 40 14 slave 10.1.2.207:63712 0.071ms 208266 218.65M 196.82M 41 17 slave 10.1.2.207:63713 0.074ms 208515 218.64M 194.20M 41 14 slave 10.1.2.207:63714 0.073ms 208674 218.78M 200.93M 44 11 slave 10.1.2.208:63700 0.132ms 208547 256.70M 227.88M 39 10 master 10.1.2.208:63701 0.110ms 208567 256.65M 228.91M 40 13 master 10.1.2.208:63702 0.126ms 209087 256.39M 228.58M 37 14 master 10.1.2.208:63703 0.106ms 208364 256.75M 225.69M 38 14 master 10.1.2.208:63704 0.123ms 208780 257.42M 227.45M 41 15 master 10.1.2.208:63710 0.127ms 208448 256.74M 229.64M 38 14 master 10.1.2.208:63711 0.124ms 207825 256.21M 230.62M 40 13 master 10.1.2.208:63712 0.119ms 209181 256.74M 228.14M 32 13 master 10.1.2.208:63713 0.132ms 208890 255.82M 230.18M 34 13 master 10.1.2.208:63714 0.135ms 208991 157.47M 147.83M 23 9 master 10.1.2.209:63700 0.142ms 209062 219.46M 204.79M 40 17 slave 10.1.2.209:63701 0.126ms 207861 218.61M 190.75M 41 22 slave 10.1.2.209:63702 0.140ms 208486 218.08M 192.50M 40 12 slave 10.1.2.209:63703 0.127ms 208047 217.94M 191.14M 40 11 slave 10.1.2.209:63704 0.128ms 208136 218.55M 190.40M 42 11 slave 10.1.2.209:63710 0.127ms 208547 218.38M 196.05M 42 15 slave 10.1.2.209:63711 0.147ms 208567 218.83M 196.55M 43 13 slave 10.1.2.209:63712 0.133ms 209087 219.24M 197.19M 41 11 slave 10.1.2.209:63713 0.132ms 208364 218.35M 196.68M 42 16 slave 10.1.2.209:63714 0.135ms 208780 218.80M 196.15M 40 15 slave 10.1.2.210:63700 0.135ms 207757 254.64M 221.63M 39 16 master 10.1.2.210:63701 0.120ms 208971 257.44M 227.80M 40 16 master 10.1.2.210:63702 0.163ms 209144 257.02M 231.55M 41 14 master 10.1.2.210:63703 0.138ms 209009 257.39M 227.63M 39 15 master 10.1.2.210:63704 0.125ms 209095 257.35M 223.91M 41 15 master 10.1.2.210:63705 0.136ms 209037 256.63M 227.74M 39 14 master 10.1.2.210:63706 0.131ms 208478 255.19M 226.62M 39 17 master 10.1.2.210:63710 0.127ms 209062 257.87M 236.33M 41 19 master 10.1.2.210:63711 0.122ms 207861 255.21M 222.44M 39 21 master 10.1.2.210:63712 0.128ms 208486 255.86M 224.68M 40 12 master 10.1.2.210:63713 0.125ms 208047 256.46M 222.67M 40 14 master 10.1.2.210:63714 0.130ms 208136 255.90M 223.52M 39 12 master 10.1.2.211:63700 0.105ms 208787 219.07M 191.36M 40 15 slave 10.1.2.211:63701 0.104ms 208536 218.37M 194.23M 43 16 slave 10.1.2.211:63702 0.105ms 209121 219.19M 193.74M 41 12 slave 10.1.2.211:63703 0.104ms 208945 218.60M 194.22M 41 14 slave 10.1.2.211:63704 0.109ms 208343 218.05M 193.75M 41 12 slave 10.1.2.211:63705 0.129ms 208485 217.80M 193.87M 41 15 slave 10.1.2.211:63706 0.125ms 207975 217.90M 191.52M 41 13 slave 10.1.2.211:63710 0.122ms 207757 217.05M 190.80M 40 13 slave 10.1.2.211:63711 0.129ms 208971 218.65M 194.46M 43 13 slave 10.1.2.211:63712 0.104ms 209144 219.24M 200.27M 41 12 slave 10.1.2.211:63713 0.105ms 209009 219.22M 193.02M 40 14 slave 10.1.2.211:63714 0.104ms 209095 219.00M 191.43M 40 13 slave 10.1.2.211:63715 0.122ms 209037 218.94M 192.92M 40 12 slave 10.1.2.211:63716 0.104ms 208478 216.83M 193.62M 40 11 slave 10.1.2.212:63700 0.106ms 207943 255.54M 224.45M 39 15 master 10.1.2.212:63701 0.105ms 208386 256.36M 239.73M 40 14 master 10.1.2.212:63702 0.115ms 208219 255.92M 224.33M 41 18 master 10.1.2.212:63703 0.104ms 208123 256.87M 232.62M 43 16 master 10.1.2.212:63704 0.102ms 208095 255.89M 226.02M 40 17 master 10.1.2.212:63710 0.104ms 208787 256.54M 224.12M 40 15 master 10.1.2.212:63711 0.104ms 208536 256.46M 226.85M 40 16 master 10.1.2.212:63712 0.104ms 209121 257.63M 223.85M 40 13 master 10.1.2.212:63713 0.104ms 208945 256.44M 228.98M 41 14 master 10.1.2.212:63714 0.143ms 208343 255.59M 226.23M 39 13 master 10.1.2.212:63715 0.103ms 208485 256.17M 226.72M 40 14 master 10.1.2.212:63716 0.105ms 207975 255.95M 225.45M 43 11 master

AVERAGE: 0.139ms 208602 236.78M 210.70M 40 14 -

TOTAL: 17.778ms 26701078 29.60G 26.34G 5176 1853 master[64]/slave[64]

Its is strange that redis is reporting used_memory higher than used_memory_rss.

After a few minutes, the memory usage got to its previous state:

INSTANCE PING db0:keys mem mem_real clients reqs/s role 10.1.2.201:63700 0.209ms 208769 58.11M 61.82M 37 14 master 10.1.2.201:63701 0.139ms 208959 58.19M 62.14M 37 11 master 10.1.2.201:63702 0.141ms 208831 58.01M 63.16M 37 11 master 10.1.2.201:63703 0.139ms 208767 58.11M 61.37M 37 9 master 10.1.2.201:63704 0.133ms 208590 57.98M 62.96M 37 11 master 10.1.2.201:63710 0.138ms 207943 57.93M 62.89M 38 15 slave 10.1.2.201:63711 0.139ms 208386 58.03M 61.74M 38 15 slave 10.1.2.201:63712 0.131ms 208219 58.05M 62.66M 38 12 slave 10.1.2.201:63713 0.131ms 208123 57.96M 62.68M 38 16 slave 10.1.2.201:63714 0.130ms 208095 57.94M 60.80M 38 11 slave 10.1.2.202:63700 0.133ms 208920 58.07M 62.00M 37 13 master 10.1.2.202:63701 0.128ms 207886 57.89M 61.55M 37 14 master 10.1.2.202:63702 0.131ms 208815 58.13M 62.62M 37 13 master 10.1.2.202:63703 0.128ms 208891 58.19M 62.29M 37 11 master 10.1.2.202:63704 0.128ms 209182 58.26M 62.61M 37 12 master 10.1.2.202:63710 0.126ms 208769 58.12M 60.73M 38 15 slave 10.1.2.202:63711 0.111ms 208959 58.20M 62.75M 38 11 slave 10.1.2.202:63712 0.125ms 208831 58.02M 60.84M 38 12 slave 10.1.2.202:63713 0.126ms 208767 58.13M 60.78M 39 9 slave 10.1.2.202:63714 0.129ms 208590 57.99M 62.33M 38 13 slave 10.1.2.203:63700 0.111ms 208014 57.92M 62.32M 38 14 slave 10.1.2.203:63701 0.110ms 209169 58.34M 62.34M 39 14 slave 10.1.2.203:63702 0.126ms 208939 58.18M 62.96M 38 12 slave 10.1.2.203:63703 0.111ms 208072 57.92M 62.41M 38 14 slave 10.1.2.203:63704 0.111ms 208414 58.10M 62.05M 38 13 slave 10.1.2.203:63710 0.111ms 208920 58.08M 61.17M 38 11 slave 10.1.2.203:63711 0.136ms 207886 57.90M 61.51M 38 15 slave 10.1.2.203:63712 0.110ms 208815 58.14M 61.82M 38 14 slave 10.1.2.203:63713 0.124ms 208891 58.20M 63.21M 38 11 slave 10.1.2.203:63714 0.109ms 209182 58.27M 62.02M 38 13 slave 10.1.2.204:63700 0.110ms 208450 58.02M 62.29M 38 12 slave 10.1.2.204:63701 0.110ms 208960 58.08M 62.05M 38 8 slave 10.1.2.204:63702 0.127ms 208211 58.03M 62.37M 38 15 slave 10.1.2.204:63703 0.129ms 209791 58.39M 62.75M 38 14 master 10.1.2.204:63704 0.110ms 208643 58.06M 63.03M 38 13 slave 10.1.2.204:63710 0.109ms 208014 57.91M 61.75M 37 13 master 10.1.2.204:63711 0.109ms 209169 58.33M 62.74M 37 12 master 10.1.2.204:63712 0.110ms 208939 58.18M 63.13M 38 16 master 10.1.2.204:63713 0.126ms 208072 57.92M 62.90M 38 16 master 10.1.2.204:63714 0.131ms 208414 58.10M 62.02M 37 13 master 10.1.2.205:63700 0.148ms 208903 58.15M 63.19M 38 13 slave 10.1.2.205:63701 0.133ms 209233 58.19M 61.92M 38 14 slave 10.1.2.205:63702 0.142ms 208794 58.09M 62.81M 38 11 slave 10.1.2.205:63703 0.129ms 208086 57.88M 62.22M 38 10 slave 10.1.2.205:63704 0.134ms 208391 58.04M 61.08M 38 9 slave 10.1.2.205:63710 0.138ms 208450 58.02M 60.72M 38 12 master 10.1.2.205:63711 0.137ms 208960 58.07M 61.70M 37 12 master 10.1.2.205:63712 0.135ms 208211 58.02M 62.29M 37 19 master 10.1.2.205:63713 0.136ms 209791 58.38M 61.40M 38 14 slave 10.1.2.205:63714 0.147ms 208643 58.05M 63.17M 37 12 master 10.1.2.206:63700 0.161ms 207725 57.82M 61.89M 37 12 master 10.1.2.206:63701 0.176ms 208958 58.23M 63.14M 37 11 master 10.1.2.206:63702 0.132ms 208266 57.90M 62.87M 38 15 master 10.1.2.206:63703 0.133ms 208515 58.04M 62.02M 37 12 master 10.1.2.206:63704 0.151ms 208674 58.09M 62.57M 38 12 master 10.1.2.206:63710 0.140ms 208903 58.14M 62.50M 37 16 master 10.1.2.206:63711 0.136ms 209233 58.20M 63.32M 39 14 master 10.1.2.206:63712 0.137ms 208794 58.09M 62.73M 37 14 master 10.1.2.206:63713 0.131ms 208086 57.88M 61.67M 37 9 master 10.1.2.206:63714 0.130ms 208391 58.03M 61.66M 37 12 master 10.1.2.207:63700 0.085ms 208448 58.07M 60.77M 38 13 slave 10.1.2.207:63701 0.057ms 207825 57.92M 62.25M 38 11 slave 10.1.2.207:63702 0.056ms 209181 58.26M 62.23M 38 8 slave 10.1.2.207:63703 0.061ms 208890 58.06M 61.54M 38 10 slave 10.1.2.207:63704 0.053ms 208991 58.09M 62.55M 39 11 slave 10.1.2.207:63710 0.057ms 207725 57.83M 60.74M 38 11 slave 10.1.2.207:63711 0.056ms 208958 58.24M 61.84M 38 12 slave 10.1.2.207:63712 0.061ms 208266 57.90M 61.41M 38 12 slave 10.1.2.207:63713 0.060ms 208515 58.04M 62.44M 38 11 slave 10.1.2.207:63714 0.070ms 208674 58.09M 62.10M 38 11 slave 10.1.2.208:63700 0.177ms 208547 57.99M 61.75M 37 11 master 10.1.2.208:63701 0.124ms 208567 58.13M 62.24M 37 13 master 10.1.2.208:63702 0.121ms 209087 58.26M 62.99M 38 8 master 10.1.2.208:63703 0.129ms 208364 58.01M 61.77M 37 11 master 10.1.2.208:63704 0.125ms 208780 58.06M 62.57M 37 13 master 10.1.2.208:63710 0.134ms 208448 58.07M 62.11M 37 14 master 10.1.2.208:63711 0.124ms 207825 57.91M 61.87M 37 14 master 10.1.2.208:63712 0.129ms 209181 58.24M 62.91M 37 14 master 10.1.2.208:63713 0.123ms 208890 58.05M 61.91M 37 9 master 10.1.2.208:63714 0.121ms 208991 58.08M 61.81M 37 12 master 10.1.2.209:63700 0.117ms 209062 58.16M 61.90M 39 9 slave 10.1.2.209:63701 0.104ms 207861 57.91M 60.86M 39 13 slave 10.1.2.209:63702 0.124ms 208486 58.05M 62.41M 38 12 slave 10.1.2.209:63703 0.122ms 208047 58.02M 61.35M 38 14 slave 10.1.2.209:63704 0.108ms 208136 57.94M 61.07M 38 13 slave 10.1.2.209:63710 0.105ms 208547 58.01M 61.57M 39 11 slave 10.1.2.209:63711 0.123ms 208567 58.14M 62.58M 38 11 slave 10.1.2.209:63712 0.122ms 209087 58.26M 63.58M 38 9 slave 10.1.2.209:63713 0.122ms 208364 58.03M 62.40M 39 12 slave 10.1.2.209:63714 0.124ms 208780 58.06M 62.62M 38 15 slave 10.1.2.210:63700 0.133ms 207757 57.80M 60.68M 37 13 master 10.1.2.210:63701 0.130ms 208971 58.17M 61.72M 37 12 master 10.1.2.210:63702 0.108ms 209144 58.22M 62.12M 38 13 master 10.1.2.210:63703 0.127ms 209009 58.14M 62.03M 37 13 master 10.1.2.210:63704 0.129ms 209095 58.13M 61.57M 37 13 master 10.1.2.210:63705 0.113ms 209037 58.15M 61.08M 37 10 master 10.1.2.210:63706 0.132ms 208478 57.98M 62.96M 37 14 master 10.1.2.210:63710 0.130ms 209062 58.15M 60.73M 37 10 master 10.1.2.210:63711 0.132ms 207861 57.89M 61.91M 37 11 master 10.1.2.210:63712 0.130ms 208486 58.04M 62.53M 37 14 master 10.1.2.210:63713 0.130ms 208047 58.01M 63.06M 37 11 master 10.1.2.210:63714 0.136ms 208136 57.93M 60.96M 38 11 master 10.1.2.211:63700 0.109ms 208787 58.26M 62.81M 38 13 slave 10.1.2.211:63701 0.107ms 208536 58.04M 62.09M 38 13 slave 10.1.2.211:63702 0.107ms 209121 58.21M 61.96M 38 14 slave 10.1.2.211:63703 0.106ms 208945 58.09M 61.58M 38 10 slave 10.1.2.211:63704 0.107ms 208343 57.96M 62.05M 38 11 slave 10.1.2.211:63705 0.105ms 208485 58.06M 62.93M 38 10 slave 10.1.2.211:63706 0.108ms 207975 57.96M 62.92M 38 15 slave 10.1.2.211:63710 0.108ms 207757 57.81M 60.37M 38 12 slave 10.1.2.211:63711 0.107ms 208971 58.18M 61.21M 38 13 slave 10.1.2.211:63712 0.108ms 209144 58.22M 61.34M 38 13 slave 10.1.2.211:63713 0.107ms 209009 58.15M 62.71M 38 16 slave 10.1.2.211:63714 0.106ms 209095 58.13M 62.75M 38 12 slave 10.1.2.211:63715 0.107ms 209037 58.16M 61.23M 38 11 slave 10.1.2.211:63716 0.113ms 208478 57.99M 62.25M 38 12 slave 10.1.2.212:63700 0.106ms 207943 57.93M 62.39M 38 9 master 10.1.2.212:63701 0.106ms 208386 58.02M 62.79M 37 14 master 10.1.2.212:63702 0.112ms 208219 58.03M 61.49M 37 13 master 10.1.2.212:63703 0.106ms 208123 57.96M 63.39M 37 14 master 10.1.2.212:63704 0.107ms 208095 57.93M 61.50M 37 15 master 10.1.2.212:63710 0.106ms 208787 58.26M 62.25M 37 12 master 10.1.2.212:63711 0.105ms 208536 58.03M 62.08M 37 11 master 10.1.2.212:63712 0.105ms 209121 58.22M 61.39M 38 13 master 10.1.2.212:63713 0.107ms 208945 58.08M 63.06M 37 12 master 10.1.2.212:63714 0.106ms 208343 57.96M 62.23M 37 9 master 10.1.2.212:63715 0.106ms 208485 58.06M 62.51M 38 10 master 10.1.2.212:63716 0.134ms 207975 57.95M 62.66M 37 11 master

AVERAGE: 0.119ms 208602 58.07M 62.11M 37 12 -

TOTAL: 15.266ms 26701078 7.26G 7.76G 4821 1575 master[64]/slave[64]

The "top" command showing some of the instances after the memory returned to its previous state:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP P TIME COMMAND
12949 root 5 -10 170m 63m 664 S 0.0 0.4 0:24.78 107m 2 0:24 redis_B0_47 /etc/redis/redis_B0_47.conf
12944 root 5 -10 158m 63m 672 S 0.0 0.4 0:24.84 95m 2 0:24 redis_B0_46 /etc/redis/redis_B0_46.conf
12955 root 5 -10 174m 63m 680 S 0.0 0.4 0:24.78 111m 2 0:24 redis_B1_41 /etc/redis/redis_B1_41.conf
12948 root 5 -10 162m 63m 672 S 0.0 0.4 0:24.70 99m 2 0:24 redis_B0_48 /etc/redis/redis_B0_48.conf
12947 root 5 -10 198m 62m 664 S 0.0 0.4 0:24.65 135m 2 0:24 redis_B0_51 /etc/redis/redis_B0_51.conf
12946 root 5 -10 170m 62m 668 S 0.0 0.4 0:24.79 107m 2 0:24 redis_B0_49 /etc/redis/redis_B0_49.conf
12952 root 5 -10 194m 62m 668 S 0.0 0.4 0:24.46 132m 2 0:24 redis_B1_44 /etc/redis/redis_B1_44.conf
12951 root 5 -10 146m 62m 660 S 0.0 0.4 0:24.64 84m 2 0:24 redis_B0_50 /etc/redis/redis_B0_50.conf
12950 root 5 -10 166m 61m 660 S 0.0 0.4 0:24.37 104m 2 0:24 redis_B1_42 /etc/redis/redis_B1_42.conf
12945 root 5 -10 170m 61m 668 S 0.0 0.4 0:24.41 108m 2 0:24 redis_B0_45 /etc/redis/redis_B0_45.conf
12954 root 5 -10 142m 61m 664 S 0.0 0.4 0:24.67 80m 2 0:24 redis_B1_40 /etc/redis/redis_B1_40.conf
12953 root 5 -10 142m 61m 660 S 0.0 0.4 0:24.44 81m 2 0:24 redis_B1_43 /etc/redis/redis_B1_43.conf

That is indeed weird.. Not sure why that is happening.. What I forgot to mention in my previous post is that jemalloc only works with 64-bit binaries. I see in your top-post that you use a 32-bit build. The 32-bit build always uses a little less memory than the 64-bit build because of the smaller pointer overhead.

Building a 64-bit version of Redis with jemalloc is not done by default, but requires setting an env var:

USE_JEMALLOC=yes make

By the way: I would try this scenario on a development machine, or as single instance side-by-side with your cluster before deploying this everywhere. You never know what happens for a particular OS/dataset/config.

I have built Redis like this:

make 32bit USE_JEMALLOC=yes

And changed the Makefile to this:

../deps/jemalloc/lib/libjemalloc.a: cd ../deps/jemalloc && ./configure CFLAGS="-std=gnu99 -Wall -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -m32" --with-jemalloc-prefix=je_ --enable-cc-silence && $(MAKE) lib/libjemalloc.a

I'm comfortable using the jemalloc branch in 32 bits mode since I have already use it for quite some time a few weeks ago and it seemed to work fine.

Here is my INFO using jemalloc int 32 bits mode:

redis_version:2.2.8 redis_git_sha1:00000000 redis_git_dirty:0 arch_bits:32 multiplexing_api:epoll process_id:5799 uptime_in_seconds:3544 uptime_in_days:0 lru_clock:800489 used_cpu_sys:32.79 used_cpu_user:6.63 used_cpu_sys_childrens:3.71 used_cpu_user_childrens:1.06 connected_clients:37 connected_slaves:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 used_memory:60960148 used_memory_human:58.14M used_memory_rss:65716224 mem_fragmentation_ratio:1.08 mem_allocator:jemalloc-2.2.1 loading:0 aof_enabled:0 changes_since_last_save:0 bgsave_in_progress:0 last_save_time:1308238911 bgrewriteaof_in_progress:0 total_connections_received:422 total_commands_processed:903752 expired_keys:0 evicted_keys:0 keyspace_hits:483398 keyspace_misses:183 hash_max_zipmap_entries:8 hash_max_zipmap_value:2048 pubsub_channels:0 pubsub_patterns:0 vm_enabled:0 role:master allocation_stats:... db0:keys=208786,expires=0

Cool! I noticed a couple compiler warnings about a field not being large enough, so I assumed it would crash hard when compiled with -m32. So: changing to jemalloc resulted in the memory being reclaimed by the OS after some time, correct?

Redis 2.4 can also be compiled with 32 bit target plus jemalloc, just with: make 32bit.

I tried a few days ago and it passess all the tests without issues.

Salvatore

Btw I think the issue is just the result of a memory spike that is not reclaimed by the allocator, but will be used again when the spike will happen again, so should not be a big deal at all. Even if it is cooler an allocator that is also able to effectively reclaim memory... of course :)

Well, I think I'll have to use the MONITOR command with parsimony then :) Thanks for the discussion.

Daniel: you don't need that actually! What you need is to split your processing in two halves. One reads from MONITOR without waiting for processing, and writing all the MONITOR output into a file. The other one reads the file processing the output. This way your processing can be slower than the MONITOR output without issues.

Swapping in jemalloc appeared to do the trick here. Closing.