My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 312: redis-server hanging when being populated with keys-values w/expire with VM enabled
2 people starred this issue and may be notified of changes. Back to list
Status:  Accepted
Owner:  ----


Sign in to add a comment
 
Reported by themig....@gmail.com, Aug 24, 2010
What steps will reproduce the problem?
1. Start redis-server with 
   vm-enabled yes
   vm-max-memory 2147483648
   vm-page-size 32
   vm-pages 134217728
   etc
2. From a standalone executable issue commands in a loop that will do the following:
   getset random_key random_value
   expire random_key 2592000
3. When the server gets across boundary of its virtual machine (2 G)it stops responding to cli and to "redis-stat vmstat" with about 48% CPU busy on our LINUX system, probably falling into a loopwhole of VM swapping.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

Aug 24, 2010
#1 themig....@gmail.com
Version tested: 2.0.0-rc4 and 2.2-alpha18, same problem
Aug 24, 2010
#2 themig....@gmail.com
It recovers eventually after 10-15 minutes, but is dead for inputs for that duration.
Aug 27, 2010
Project Member #3 anti...@gmail.com
Thanks, this is something we did not tested carefully enough, I'll check this issue better in the next days as this sounds like a blocking bug for 2.0.0 release.

Cheers,
Salvatore
Status: Accepted
Aug 27, 2010
Project Member #4 anti...@gmail.com
Hello again, there is no way I can reproduce it. What was the available memory on the system? Is it possible that the process started to swap (due to OS paging) before?

Was the GETSET/EXPIRE thing running on a single executable or with some kind of parallelization? (I tried with a single threaded approach).

Thanks!
Salvatore
Aug 27, 2010
Project Member #5 anti...@gmail.com
nevermind, problem reproduced... will take you posted on the progresses
Aug 27, 2010
Project Member #6 anti...@gmail.com
@themig.adx if you are around, did you noticed if this happened when the server was about to save a snapshot? Thanks.
Aug 27, 2010
Project Member #7 anti...@gmail.com
I think the bug is fixed in Redis master, please can you confirm? A third party confirmation from you that opened the issue so are surely able to reproduce it will be hugely appreciated!

I'll back port it in 2.0.0 as soon as I get your ACK.

Thanks for reporting,
Salvatore
Aug 30, 2010
#8 themig....@gmail.com
I did test the latest 2.2-alpha0-79. I don't see long freezes as I saw in 2.0. I dosn't get stuck as before. Very happy with it.

However, while swapping (and swapping is happening constantly), reads can take very, very long time. Why is it swapping constantly? Is it because that all the key-value pairs have expiration on it? It seems that in this case its impossible to use VM with expiration for keys, because of excessive swapping. Am I wrong? 
Aug 30, 2010
#9 themig....@gmail.com
I want to make it clear again. read freeze happens in the first several minutes after intensive data population, when the VM memory starts working (mem is getting above the threshold). So, after the population of the DB, the intensive reads are initiated from a multi threaded multisocket client, and the read performance can be very low (400 requests per second) instead of usual 40,000 that we have in a steady situation.
Sign in to add a comment

Powered by Google Project Hosting