Comparison of PubSubHubbub to light-pinging protocols

Good stuff.

Please correct me if I'm wrong, but while your representation is accurate, it may give the wrong impression to the less technical, as if anyone else is reading this stuff ;)

What I mean to say is that the subscriber here is most likely not a end-user client but a server whose job is to propagate the new content to end users. In other words, both protocols are server to server.

Therefore, the savings you speak of are true but it's not necessarily orders of magnitude. It may be in some cases, I'll grant you that.

But in many cases, the cloud and/or the hub is notifying thousands of servers of an update. Those thousands then inform the millions of end-users. It's a federated system like XMPP or email, not peer-to-peer.

When that is put into consideration, the extra complexity may outweigh the savings.

Now, I will grant you that anyone implementing either one of the protocols will probably have the requisite technical chops to handle the extra overhead of processing a fat ping, so maybe my point is moot.

On the other hand, RSS is extensible and there is nothing saying that adding a fat ping can't be put into a namespace. Then it would be just a matter of whether a developer wanted to support it or not. Their choice, and not imposed by the protocol from the onset.

Great work you are doing here.

Matt: Thanks for the feedback. Some responses:

1. I'd claim that the last mile is not significant. Topologies (application, network, datacenter) and architectures (long-polling) optimize it away. Think of how Akamai edge caching works. The real cost is federating server-to-server communication and getting data to local nodes.

2. You mentioned "the extra overhead of processing a fat ping." You're missing something crucial here. Processing fat pings requires less effort for both publishers and subscribers (i.e., the ones who matter); they offload all the hard work to hubs. That's the point: complexity in the center.

3. PubSubHubbub supports RSS right now: http://code.google.com/p/pubsubhubbub/wiki/RssFeeds

Everything you say is true to a large degree. My only rebuttals might be:

1. Some (not me) might criticize your claim that long-polling optimizes a certain problem away. It is not yet in widespread use (that's a relative statement) and to scale it well brings complexity back to the edge. Not everyone is running Tornado yet. ;)

2. Others might say, if you wanted complexity at the center you already had XMPP. Not an XMPP joke, but if you can implement PSHB, couldn't you rig up XMPP PubSub??

3. Less work on Publishers and Subscribers. Well Brett, it's more work for me as a subscriber right now, since I don't have code in place to process and aggregate these atomized and multifeed updates. I have code that reads RSS and Atom documents on a whole. But once I do implement reassembling partial documents, I see your point.

The most compelling reason I've heard why publishers don't like fat-pings is that it will likely cause their compete.com scores to drop.

If everyone converts to a realtime network, then the new measurement of success is the number of followers (subscribers). When the number of realtime followers is very high then the "Thundering Herd" problem emerges if the Hub doesn't offload the Publishers website.

The only effective way to build a distributed "twitter-like" web is to insure equal access via large-scale services.

Small clouds dont need excessive optimizations... but a realtime twitter killer has to think about removing every potential source of latency between the Pub and the Sub.

How about that Denial of Service box? There won't even be a Fail Whale to know the Hub is in trouble... just a lack of realtime news. Some say that RSScloud would just fall back to mormal RSS... if you ask for News from every site you subcribe too. They will give you their latest posts. But that's just not realtime is it.

We want it real and timely. News measured in seconds... new news.

Great analysis!

For context, I would just add that there is no one right way to do pub/sub architectures. The case for light pings is in those architectures where the subscriber should decide whether it wants to grab the additional data or not.

Optimal pings will contain the least amount of information necessary for the subscriber to be able to make that decision (weighed, of course, against the additional costs of using "light" pings shown in this doc). In rare pub/sub cases, the near emptiness of RssCloud?'s pings ("I updated something, but not telling you what") might actually suffice. RSS/Atom, however, does not seem to be one of those cases. RSS/Atom itself is already very close to the very header-type info you'd want to include in the "I updated" pings to let the subscriber decide if the actual content should be pulled down. This gets to the old question of how much of an article one should include in RSS. If large articles are embedded into the feeds, with pictures and ad attachments, one might be better off with rssCloud style minimalist pings than full pings. You have to take this into account when calculating the bandwidth tradeoff. Of course, a data-hungry aggregator with many end-users as the subscriber will always want to pull down everything and will always prefer fat pings. A more evenly distributed system, on the other hand, might be more selective about what goes where.

"Fat pings of light RSS/Atom" is probably the most generally optimal solution.

What's "Complex XMPP"? XMPP transporting Atom entries doesn't feel very complex, for example.