We moved to Redis.io!

Speaking to your last point about global:timeline, what you have suggested is essentially 'lamport time'. Lamport time requires that each node produce increasing time values (which is where it differs from your suggestion), but I believe that adding this constraint could solve the problem (though I'm not sure if it would be worth the effort).

Mathew Duafala

thanks mduafala, I'll study better lamport time but I still can't figure how it is possible to distribute the list stored in global:timeline without the need of a merge operation later, even if I got an idea from what you said about this issue.

That is, append the element N to server 0, N+1 to server 1, N+2 to server 3, .. adn so on, so you end with an interleaved list that you can merge in O(N).

Yeah, antirez, that's how you'd have to do it. Once you split up a list like that, its all scatter-gather to do stuff with it from then on.

Typo at the end of 'Updates' section: I believe ORDER BY (and not SORT BY) is the correct keyword for sorting in SQL.

@dov.murik: thanks fixed

You can also use an encrypted user_id and user_name stored in the cookie.

Why was the choice made to concatenate UID, posting time, and post rather than using an array? Is there a speed difference?

@smullen.uclick when you don't need to access this fields separately, it's better to use a separator: atomicity of get/set of the three elements, memory saving, speed.

Is there a reason why the PHP client library is written using PHP4?

@videlalvaro backward compatibility and PHP5 features not needed. There is a new PHP lib in development, written in C as PHP module, but the pure-PHP lib will be maintained anyway.

Cheers, Salvatore

@antirez Nice :-) Any release date for this PHP module?

@videlalvaro - http://code.google.com/p/phpredis/

@antirez Amazing example. Is there any Java API release?

In this example:

SADD uid:1000:following 1001
SADD uid:1001:followers 1000

How would you atomically do both commands? What would happen if redis crashed after doing the first operation but before doing the second operation?

@poleris in order to perform this operations atomically currently there is no way. Also in the client crashes after the first operation the effect will be to have just the first set modified.

After Redis-1.0 stable will be released I'll start to work on 1) a LOCKing primitive, that will allow the atomicity. 2) A way to send multiple commands and tell Redis to only exec the commands when we can be sure all can be processed without interruption. This is also a guarantee of atomicity itself. Example:

PREPARE
SADD foo 1
SADD bar 1
COMMIT

What will happen is that the two SADD will be executed only when you call COMMIT, so either both or none will be executed. Also the commands are guaranteed to be performed one after the other without any other command executed in the middle, so atomically.

For now it's simply too important to release 1.0-stable, so this features will be shifted after 1.0 stable. Please feel free to post your exact need in the Redis DB google group, there is almost always a locking free way of doing a lot of things.

Isn't this auth cookie business susceptible to cookie hijacking? If I go to an Internet café of some kind, login to their Wi-Fi, and run a packet sniffer, I'd perhaps find a stray Set-Cookie: line floating around. I could then create the same cookie in my browser and just like that, I'm logged in as the other user.

Posts or login names with Non-Latin1 charaters does not display at all. (I tried hungarian ones) My ugly workaround was: retwis.php

@@ -60,7 +60,7 @@

}

function utf8entities($s) {

}

function goback($msg) {

is it possible and meaningful to design a high level programming language or system which generate redis instructions as low level code? redis as a VM, a funny idea suddenly come out to me

@nkchenz: I guess not, Redis is not turing-complete ;)

Someting is wrong with http://retwis.antirez.com/ application, I guess, redis server is off ;P

Hi! Great app and great help for anyone learning redis. I have a question though:

You write "we don't need to lookup by time or user id in the example application so it is better to compact everything inside the post string."

However in the ruby clone of this app ("retwis-rb"), the author danlucraft took the opposite approach saving each individual piece of information under its own key ("post:id:1:user_id" => "1", "post:id:1:timestamp" => "Tue Nov 24 17:00:27 +0100 2009", "post:id:1:content" => "Hello World!").

You only save one item "post:1" => "1|Tue Nov 24 17:00:27 +0100 2009|Hello World!".

What are the pros/cons of each approach? I'd say, his solution is more general but requires three GETs per post. Yours is more specific and requires custom serialization, but gets by with only one GET per post.

In general, would I rather want many GETs for individual values or fewer gets for serialized data?

Answer to my own question: This article describes both approaches (under "Attributes") http://www.paperplanes.de/2009/10/30/how_to_redis.html and also favors the one-key-to-serialzed-values approach taken here.

Now what happens if a user posts something that contains "|" ? Isn't that encoding somehow unsecure?

@contag

the using of third argument of explode() will prevent all possible secure vulnerabilities

$aux = explode('|', $postdata, 3);

I wrote a Retwis clone in python, based on retwis-rb : http://github.com/pims/retwis-py if someone prefers python to php :)

There is one important thing missing. When you start to follow a person on twitter you get his/hers old updates included in your timeline. It seems to me that that would be impossible with a redis list since your can only insert values from left or right or maybe at a index-position but not at a date.

is it possible to simulate ORDER BY field1, field2 of SQL in redis? Can u give some example patterns for GET option within SORT command (if i want to search all the keys contains string 'abc')...

ok,try

The Redis Cluster project is going to be our next priority after the release of Redis 2.0.0. Redis Cluster is an higher level layer implemented as a different decoupled component using normal Redis instances as "Data Nodes". Redis Cluster will support adding and removing nodes while the system is running and fault tolerance. well, we need Redis Cluster!

Two Questions. a) What was the optimal maximum record size per key for the said benchmark where the 5ms pageview was achieved. b) This question is directed more at the design of the twitter clone rather than the underlying redis architecture. How would this scale when a good subset of users has 1000+ followers. Wouldn't that mean 1000 writes for each post/tweet ?

can you explain some more (with a case) why we need two keys for auth

SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9 
SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000 

why is "uid:1000:auth" not sufficient?

this example is informative - tells how to think about a key-value app.

i have one minor suggestion, in post update the $followers array is modified but not persisted, it is more readable to set the user post separately.

old:

$followers[] = $User['id']; /* Add the post to our own posts too */

foreach($followers as $fid) {
    $r->push("uid:$fid:posts",$postid,false);
}

new:

$uid = $User['id'];

foreach($followers as $fid) {
    $r->push("uid:$fid:posts",$postid,false);
}
$r->push("uid:$uid:posts",$postid,false);
}}

how to move my old data to new redis server when i try making it horizontally scalable?

Please, restart redis for your retwis demo.

test

FYI I threw together a prototype, a clone of this PHP example, in Python using the Tornado framework. Its available on github (http://bit.ly/gYJhp6)

I am trying out redis, with node.js. Here is my code on github(https://github.com/zdwalter/retwis-js-node-express)

still working on it.

great redis !