
redis - issue #607
Redis server unreachable when uploading a dataset of 800,000 key/value pairs.
What version of Redis you are using, in what kind of Operating System? v. 2.2.11, Ubuntu Lucid Lynx (10.04)
What is the problem you are experiencing? I am trying to upload an example dataset with about 800,000 key value pairs into a sorted Redis set using zadd. My python script (redis-py v 2.2.8), invariably fails when uploading the 28233 item after which the redis-server becomes unavailable (I cannot telnet to it). The server is back again after a while.
What steps will reproduce the problem? Upload key value pairs using a script. The script is unable to connect to the server after a while fails with a connection error.
Do you have an INFO output? Please past it here.
If it is a crash, can you please paste the stack trace that you can find in the log file or on standard output? This is really useful for us!
Please provide any additional information below.
Comment #1
Posted on Jul 22, 2011 by Happy ElephantAre the commands sent in a pipeline, or part of a MULTI/EXEC block? It would help if you provided an excerpt of the commands that are sent to Redis, or even the script you use when you observe this behavior. When the issue is reproducible, it should not be very hard to track down what is causing this. Thanks.
Comment #2
Posted on Jul 22, 2011 by Happy BearNo, the commands are not sent in a pipeline nor is it a part of a MULTI/EXEC block.
The script is very simple: Key/Value pairs are read from a plain text file and zadd is continuously called until EOF.
Comment #4
Posted on Jul 22, 2011 by Happy ElephantYou're creating a new connection for every ZADD. Probably Python can't GC the connections fast enough, and you run out of file descriptors. Use a single connection to perform all ZADD's instead.
Comment #5
Posted on Jul 22, 2011 by Happy BearUsing a single connection fixed it. I could add all the key/value pairs without a hitch.
Status: Invalid
Labels:
Type-Defect
Priority-Medium