« Architecture of Participation | Main | Web Based »

Using IM for grid computing

Tim Bray:

"My problem is that I’ve been a Unix guy for twenty years and a Web guy for ten. My feeling is that if something says it’s a service, the right way to talk to it is to get a hostname/port-number combination, call it up, send a request, and wait for a response. My feeling is that a good way to have processes work together is for them to exchange streams of text using standard input and standard output. My feeling is that if I’m going to be stressing out the performance of an app on a grid, I don’t want too many layers of abstraction between me and the messages. My feeling is that server failures are going to have to be handled in an application-specific way, not pushed down into the abstraction layer."

XMPP. Especially the Erlang* server, ejabberd, as it can be managed with ejabberdctl the same way you manage Apache with apachectl. And it's fast. Want to create a compute/result farm for all things trivially parallelisable? Use a chatroom to coordinate the workers. Want to listen to interesting arbitary activity on a heteregenous networks? Use PubSub to subscribe to RT data feeds. When I look at the OGSA stack I'm fairly sure XMPP is to Grid and JINI as HTTP was to WS and CORBA. Give it a few years - instant messaging is where real commodity grid action will take place.

MapReduce. Tim mentions hadoop. I susoect hadoop got spun out of Nutch cos it was where all the action was at, and the basic spidering - index - respider - reindex cycle was being given short shrift. But mapreduce sure likes the Linda in() and out() operations, as you see them in the implementation Gelernter and Carrerio outline in How To Write Paralel Programs.

* as in, yeah, we do SMP.

May 26, 2006 12:04 AM


Aristotle Pagaltzis
(May 26, 2006 08:44 AM #)

See also DJabberd.

James Governor
(May 26, 2006 02:50 PM #)

cote started down this route in his jabber coverage.

I think you're onto something. will the world just be ATOM and XMPP? ;-)

That will be what I am starting to call The Synchronised Web... not a moment too soon now 2.0 is actionable... ;-)

Bill de hOra
(May 26, 2006 05:45 PM #)

I've built on RSS over XMPP (back before Atom was specced). It's very fast to build out on, fast to add new nodes and services, and seems to have the scaling characteristics and economics of a p2p grid, insofar it doesn't punish the popular and you can use commodity boxes. Plus we know already IM tech is internet scale. Try sending files via XMPP compared to email or web dropsends - it's stupid fast.

The problem as ever is that most peers are natted and can't communicate directly so they take the hit of going via a server to communicate. If XMPP ever gets its standard port "assumed open" as per port 80, ie sysadmins will automatically leave it accessible instead of you having to argue for weeks to get it opened, that's a game changer.

I think in the medium term, XMPP will become the default internet protocol for heavy MQ integration. Doing MQ over HTTP is... doable, but a bit of a pita, and if you want real speedups you need both sides to run servers to emulate full duplex (ie like Biztalk point to point configurations). That's a non-issue for IM since everyone's a client.