What steps will reproduce the problem? 1. Create a set with 100k elements 2. sdiffstore against a much smaller set (basically take some operation which needs to allocate a large chunk of memory) 3. Trigger a bgsave and run the sdiffstore operation again while bgsave is in progress

What is the expected output? What do you see instead? bgsave should not have a significant negative impact on performance. For this operation, the time to complete drops from a few hundred millseconds to several minutes.

What version of the product are you using? On what operating system? redis-2.0.1 on Ubuntu 10.04

Please provide any additional information below.

This is due to hash table sizes being fixed during bgsave, which increases time complexity from linear to quadratic (stepping over many small hashtables in a list)

A quick view of the impact of this on a production system. I have two redis servers running on a single box - the master does everything in memory, the slave persists to disk but accepts no network connections aside from the master/slave link. The amount of additional CPU time taken up by the slave which periodically persists to disk is clear:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17821 root 20 0 569m 561m 880 S 5 7.1 48:58.33 ./redis-server /etc/redis.conf
17793 root 20 0 528m 521m 868 S 1 6.5 87:17.23 ./redis-server /etc/redis-persistence.conf

Thanks for the report. This will be investigated ASAP.

Hello, the bug was fixed in both Redis master and Redis 2.0.0.

In Redis master it was harder to trigger because of the intset patch, and the fact that both my and Pieter script tried to populate the sets with integers, but the bug was indeed preset in Redis master as well.

Now it's fixed (and verified) in both branches.

Cheers, Salvatore

This is great! That's exactly what I'd do in that case.