« Generating feed identifiers can be tricky | Main | Agile Testing »

The integrator's dilemma

Mike Champion has a crack at delineating between REST and WS. The question he's responding to is whether Microsoft are ignoring demand for REST tools?.

"The REST approach (as far as I can tell after years of discussion) assumes that all these features are provided at the application level rather than being provided by the infrastructure."

My first reaction was - I didn't know there was a demand for REST tools. My impression of those who are interested in using REST is that they want to be close to wire and to the data and if anything, want to reduce the number of tools in use. REST is a minimalist approach to systems building - when you're using it you're making a decision to do some of the heavy lifting on your own. Or perhaps you realized that you never needed to lift half of what you thought you did. If anything, I would think that selling tools into REST users would be tough work.

"I am convinced that developers with demanding requirements for security, reliability, transactions, etc. generally DO want what WS-* and Indigo promise."

Perhaps they do. Tho' there's not much question that WS-* is in need of rationalizing. Entirely natural in the tech sector, competitive pressures have resulted in duplicated effort and excess choice for technology decision makers, which lessens the value of having standards to begin with. The string 'WS-*' itself indicates a problem. not a solution, for end-users [1]. End users don't benefit by having multiple similar specifications to choose from (implementations are another matter).

Mike goes on the contrast the WS potential with the current state of REST:

"Perhaps there is some way to offer these services within a RESTful toolkit so that application developers don't have to wrestle with them. The obvious answer, however, is that few people seem to have much of a clue how to do this. Those who do, such as the developers at Amazon, have invested fortunes in making it happen."

The position advocated for WS in contrast to REST sounds like this - yes, that's nice for simple things, but WS are for serious work. They exist to service those who need systems where all the 'ilities are turned up to 10.

But it's worth considerign that the sheer cost of solutions for bodies such as Amazon may simply reflect the massive scale of their operations (Werner Vogels, Amazon CTO has remarked that Amazon's scale provides unique challenges for technology and infrastructure). There's no compelling evidence that WS based technology would generally be cheaper or that REST solutions are generally not cost-effective.

Nonetheless, that WS can be positioned primarily for those companies whose requirements are effectively off the bell curve is interesting. The foundation technology of WS is of course SOAP. SOAP is no longer an acronym, but the S used to stand for Simple. The implication is that as of 2005, the simple stuff can be served by inferior technology and WS has progressed in complexity to serve a more rarified market. The rest of this entry runs with that idea.

Points of disruption

We can see the market looking at 3 things to provide value in software solutions:

  1. Simple protocols and formats
  2. Open source infrastructure
  3. Agile methodologies

All of these are disruptive.

The first is what we're talking about here - formats and protocols that are sufficient for most users. The other two amplify the first and are worthy of a brief mention.

Open source as disruptive technology to traditional per seat, per CPU, licensing models has been beaten to death by Tim O'Reilly among many others - today everyone selling products into the integration and middleware space has to factor OSS into their plans.

The reason agile methods are disruptive is because software methodologies are closely aligned to business and procurement models. How you go about delivery has a first order effect on how you get paid, and how much. Agile approaches also favour what Scott Ambler calls 'generalizing specialists', rather than actual specialists, something that runs counter to much of industry's organisational and training practices. Notably, the Standish group is still telling us that 65% of projects are failing in way or another. In that context, providing a process that get the figure down to even 40% and say, halves the post-live defect count and you have in your hands a disruptive business model.

A services dilemma

Using the Christensen playbook, we should consider whether WS specs and tools are being commoditized by disruptive technologies. If so the natural progression for them and the software based on them is to move up the value chain. When the mass market for enterprise technology is either over-served by technology, quality or the price is too high that leaves an opportunity for disruptive technology to take hold at the bottom end of the market.

I worry this may put out some people who feel that WS are a disruptive technology, but when you see RSS usage figures doubling twice a year, you stop and think about what's really disrupting what. But, REST as it appears in the HTTP Web seems to have a broader potential set of uses than was originally imagined by WS proponents, especially in conjunction with simple formats like RSS or Atom.

In the case of WS, it's possible that the broader market is over-served. Statistically speaking, relatively few organizations needs all that serious stuff, all those extreme 'ilites. What most organizations need is cost-effective, stable, flexible software with reasonable TCO. So if the number of people who really need 'serious' high-end systems is relatively small compared to those who can't afford or don't need all the technology, what we can expect is that the market will be served by inferior technology at margins the providers of serious tech will be typically unwilling to meet. The suggestion is that those providing the disruptive technology will enter a growth market.

Naturally, some people will object to classing the Web, REST, RSS et al as inferior technology. The point is that this stuff is good enough for a growth market. And following we should expect that good enough tech will improve and those supplying it will move up the value chain, while the technology itself probably gets cheaper.

What's driving WS?

Should we accept that people are either being supplied excessive technology requirements and architectures through WS or simply can't/won't afford to procure in the first place? I think we should, but there's nothing sinister here, rather it's a quite simple outcome of enough 'what if' questions. What if the database server goes down? What if the transaction fails? What if the network goes down? What if we have 1M users online? What if someone tampers with the data? What if we get slashdotted? It's all about listening to your customers and taking care of them . It should make sense to do this. This is a good thought process to go through.

The problem is that the concerns are often answered without taking cost into consideration, without anything that looks like risk assessment, without a thorough examination of the requirements.

The burn rate for such things fits the classic J curve - a modicum of improvement has a disproportionate cost the rate of which increases as improvements are added. Software people don't always consider the J-curve in answering what if questions - where they do, answering them by climbing to the summit of the curve is not ideal and is something that currently separates software from engineering. Engineers will establish ratios and determine fitness for purpose. Engineers understand the effect, necessity and relative cost of part tolerances. They know the properties of their materials and structures under a variety of environmental conditions. They have a good understanding of solution cost. They know they might get sued. Engineers even have a physics. Yes, engineers do get it wrong, but generally, the software industry isn't strong on this.

Good enough opportunities

Finally, REST advocates like to paint an appealing picture of a Web governed by open standards that mesh well with one another and over which XML documents conforming to industry-standard schemas with well-documented semantics are transferred. Few who actually work down in the trenches suffer from this delusion for long. Even a dirt-simple idea such as the XML format RSS becomes an interoperability mess in practice.

This is easy to agree with. What is less easy to understand is how fifty plus specifications from the WS-* canon will improve the situation.

How are complicated WS specs going to help with interoperability a way the dirt simple stuff does not? There doesn't seem to be anything special about REST that would make it prone to interop problems over WS specifications. The interop story on WS has not been spectacular, but is getting better [2]. I'm neutral on the relative merits of their interoperability and much more interested in how much interopability is actually required case by case. What seems to be special about REST from this point of view is that it's fit for purpose and cheap by comparison, once you establish that you aren't served best with the complexity and capabilities implied by WS.

Once again, it appears that REST is a fine idea for simple scenarios where bad stuff can happen with impunity and corner cases can be disposed of arbitrarily. Building secure and reliable applications for situations in which serious money or human lives are at stake is another matter.

I don't think anyone should read that and draw the conclusion that REST is an irresponsible technology and WS is not.

update: Mike Champion had a comment about this that's worth highlighting:

"...My point, which I think you agreed with at the beginning, is that the WS-* stuff is trying provide standardized interfaces to encryption, signatures, authentication, authorization, etc. services in the infrastructure. AFAIK "REST" puts this responsibility on the application developer. In reality, security has to be a consideration at every level, and I certainly didn't want to imply that WS-* provides any kind of magical protection or that REST is intrinsically insecure."

Which is fair - the quote I took from Mike could be misunderstood out of context.


What's really worth bearing in mind is that serious money is relative, reliability is relative, security is relative. In one sense, the polarization (you're secure or you're not, it's reliable or it's not, it's serious or it's not) and the drive toward higher levels of performance and complexity are entirely rational. As Christensen has pointed out, it's what happens when you listen to your best customers and try to maximise existing profit centres. This attentiveness inevitably forces WS vendors up the value chain and out of areas where they believe it's not worthwhile selling into, leaving a gap for simple technology like REST to step into. It means when you take into account the J-curve, there is a risk of abandoning a growth market by thinking interms of absolutes and overlooking the notion of fit for purpose.


Conclusion

If we accept that WS based technology is being disrupted at lower and mid ends of the market then it becomes clear that there is no absolute technology winner between REST and WS [3]. What to choose becomes a question of establishing your relative requirements. However we can expect that vendors servicing the low end will move up the value chain over the coming years and as a result claw away at WS based revenue - Christensen's model does not predict a steady state of affairs.

If dirt simple equates to good growth and better profits, then the missed opportunity arises when simple and simplistic are conflated. When the WS contingent are looking at the REST and syndication crowd and saying more or less, 'here's a nickel kid', they may want to stop and take a second look at what the kid is doing with that nickel.



[1] I should qualify myself here on the state of WS. There's no doubt that some WS technology will prove useful - for example a standard RM protocol will be immensely valuable whenever the vendors decide to coalesce. SOAP will clearly remain useful for some cases.

[2] The SOAP interop story is getting better, but it's not all there yet. The people I look to for level-headed guidance in this space are Steve Loughran and Sam Ruby - when they say things are good it's an opportunity to re-evaluate. Otherwise I see know reason why you couldn't evolve towards the best WS has to offer as and when the features are needed.

[3] This does not mean the public debate is reduced to a tourniquet fest, only that we need to take a focused look at what the customer actually needs.


February 17, 2005 08:25 PM

Comments

Bob Dionne
(February 18, 2005 01:44 PM #)

Bill,

Having just finished the Web Services 2005 conf. and listened to all the vendor noise about interop I couldn't agree more. It's the vendors who need interop in theory so that they can wave the open standards flag, but of course in practice they all also need to "add value" to justify their price.

One interesting technology to look at is Netkernel ( http://www.1060research.com ). They taken the REST approach to the extreme in providing a server where URIs cross the webserver boundary. It's URIs all the way down. Another way to characterize it is as an XML app server. It's a very radical approach but the simplicity of URIs and XML pipes, coupled with a nice module system, support for scripting engines, and caching make Netkernel is very new approach.

regards,

Bob Dionne

Mike Champion
(February 20, 2005 02:00 PM #)

Great piece! I agree with most of it, but may pick a few nits in my weblog. The only thing I want to go out of my way to clarify is:


> Building secure and reliable applications for situations in which serious money or human
> lives are at stake is another matter.
I don't think anyone should read that and draw the conclusion that REST is an irresponsible technology and WS is not.

That certainly wasn't my intent. My point, which I think you agreed with at the beginning, is that the WS-* stuff is trying provide standardized interfaces to encryption, signatures, authentication, authorization, etc. services in the infrastructure. AFAIK "REST" puts this responsibility on the application developer. In reality, security has to be a consideration at every level, and I certainly didn't want to imply that WS-* provides any kind of magical protection or that REST is intrinsically insecure.

Bill de hOra
(February 20, 2005 06:23 PM #)

Mike,

I can see how that quote I took might look out of context; so I've lifted your comment into the text.

James Orenchak
(February 26, 2005 05:45 PM #)

Thanks. It's been my opinion for a long time that REST is the correct solution for some, but not all, problems. SOAP is under certain condtions simply the better choice. SOAP has it's problems with interoperability, but SOAP doesn't always mean being tied to a certain vender. There's the open source SOAP-Lite in Perl, among other free SOAP implementations. I agree with your conclusion and hope that all REST fanatics read your journal!