Mailing Lists: Apple Mailing Lists
Image of Mac OS face in stamp
 
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Anyone know if RSS <cloud> is used in the real world?

Hi Jens,

Here's what I remember about the scaling issues. Apologies for the length... ;-)

-Jake

On Jan 19, 2006, at 7:40 PM, Jens Alfke wrote:
I'd love to know about the issues with scaling, since the reason I'm interested in this is to try to make things scale better than they currently do with polling.

Your back-of-envelope sketch is fine as far as limiting the effect of polling the feeds themselves, but it doesn't take into account the resources required to do the notification piece. The problem we found happens when you have lots of clients that support the <cloud> element that expect to be notified of changes to lots of feeds, especially when a central server is responsible for doing the actual notifications.

This is all back-of-the-envelope as well, but I'll try to walk through the issues as best I remember them...

Take an example where 100 clients are subscribed to a given feed. Assuming that all of the clients have requested notification, that means that each time the feed is updated, 100 outgoing requests get instantiated by the cloud server. Now say 50 of those clients IP addresses have changed, or are now offline (no route to host). Now you've got 50 or so connections that, in the worst case, stick around until they time out. (The firewall/NAT situation is easier, since in most cases connections get blocked instead of ignored, so they drop right away.)

Next have a look at the user-model. A blogger might update a given post (and its feed) 5 or 10 times in a short period, as they add links, fix typos and spelling mistakes, and do other light editing. If their feed is updated say 5 times in 3 minutes, and the server's notification agent times out after 60 seconds, then you might end up with 150 open connections just hanging around, waiting to time out, and another 350 that hopefully have gotten through. Now say it takes 2K of RAM for each TCP connection, another 2K for the response itself and 1K for a database operation, etc -- now we've allocated and deallocated at least 2.5MB of RAM for just this one feed, and at any given time, 1MB or so might be sitting there in use, waiting to time out.

Now imagine this situation on a server which is hosting 1,000 or more sites with an average of 100 subscribers each, and some percentage of very active sites -- 20 or more updates a day -- which have 1,000 or more subscribers. (More active sites tend to be more popular.) I think you can see the problem pretty clearly. Just one of those 1,000 subscriber sites can generate 25MB of RAM usage (not necessarily concurrent), and 10MB for stalled connections to subscribers who will simply time out.

Another place where you get into trouble is the case where outbound connections are happening at a high enough rate that they start to stack up. If the average update notification takes 1 second, and your server can handle 500 of them concurrently, then you're looking at 3,000 max-possible notifications a minute. But if for some reason 10 of your high-profile 1,000-subscriber feeds update within the same minute, or even within 2m'55s, you're past your bottleneck, and you have to either queue up the requests or drop them. It's not difficult to imagine a situation on a busy server, where your notification queue fills up faster than you can send out the notifications. What's more, a long notification queue delays the workstation's refreshing the feed, reducing the value of the feature. The busier your server, the less value the notification has.

Worse: none of the above takes into account the costs associated with any network connections to a database, Internet bandwidth, or processor requirements. And I didn't even look at the metrics for tracking pleaseNotify requests and timing them out.

Of course there are lots of opportunities to optimize this whole scenario. Notifications will change very little, if at all, so the need to generate a response body for each client could be reduced to 1-response/update from 100/update or 1,000/update. Also, we could limit notification frequency to 1/5min or similar to handle the case of the over-zealous editor.

It might also get easier with a little more decentralization. A central server could have a method that a blogger's client app (workstation) can call to ask who's subscribed for notifications, and then we can shift the notification responsibility to the workstation. This would reduce the server-side piece to handling only the pleaseNotify requests, and communicating a list of subscribers to the workstation that's actually producing a feed update.

But it also introduces an additional layer of complexity and ambiguity, a new point of failure on the network, and lots of additional testing difficulty. And when does the workstation refresh its subscriber list? Does the workstation poll the server? (Remember we're trying to avoid polling in the first place.) Does it ask for a subscribers list after each time it updates its feed? What if it's updating once a minute? Can we trust the workstation to only ask for the list less frequently than that?

What's more, the added indirection still doesn't really solve the problem for large centralized applications like Manila (or TypePad, or Blogger, or whatever). Those apps will probably still have to send all the updates themselves, which might not be too much work for 10 feeds with small numbers of subscribers -- but for hundreds or thousands of feeds with many subscribers, it's a big problem (see above).

I realize that for browser-based apps, there may now be ways to make the centralized setup work in a decentralized way (at least some of the time) using AJAX or REST techniques, but we didn't get that far at UserLand the last time through this loop, since we were working with relatively slow CPUs, on relatively small pipes, running web apps in browsers that didn't know how to make XML-RPC requests, and in many cases couldn't be relied on to run scripts (JavaScript, VBScript or whatever).

The long and the short of it is that the cost/benefit analysis didn't work out at the time, so we turned our <cloud> services off, having many other fish in need of frying.

 -Jake

_______________________________________________
Do not post admin requests to the list. They will be ignored.
syndication-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/syndication-dev/email@hidden

This email sent to email@hidden
References: 
 >Re: Anyone know if RSS <cloud> is used in the real world? (From: Brent Simmons <email@hidden>)
 >Re: Anyone know if RSS <cloud> is used in the real world? (From: Jake Savin <email@hidden>)
 >Re: Anyone know if RSS <cloud> is used in the real world? (From: Jens Alfke <email@hidden>)



Visit the Apple Store online or at retail locations.
1-800-MY-APPLE

Contact Apple | Terms of Use | Privacy Policy

Copyright © 2007 Apple Inc. All rights reserved.