« Les_Fran%C3%A7ais_ont_gagn%C3%A9_hier_soir | Main | links for 2006-07-12 »


Steve Loughran: "I felt operations were a barrier on a small project, let alone a big one, because once you have a split between dev and deployment, you have problems. "

Dave Thomas: "In the real world we have this split, we have developers responsible for applications and we have a totally separate part of the organisation that's responsible for administering the application, for making things work. And I hate to say it, but that really is the way it should be."

My money's on Loughran. Web apps, grids, service oriention, are all about continuous deployment. Anyone remember the J2EE Deployer role?

July 10, 2006 11:32 PM


(July 12, 2006 02:07 AM #)

[Also left this comment with Don, but I thought it was worth repeating here]

The best way to completely automate operations is to have to developers be responsible for running the software they develop. It is painful at times, but also means considerable creativity gets applied to a very important aspect of the software stack. It also brings developers into direct contact with customers and a very effective feedback loop starts. There is no separate operations department at Amazon: you build it; you run it.

(July 12, 2006 07:49 AM #)

For every site of every size, deployment should be a 1-click task.
However, installation (which covers routers, firewalls, DDoS protection, certs, load balancing, IDS rules) can never be a 1-click install.

If you've never worked on a significant site (ie >10 servers) then you've likely never needed to worry about these sorts of issues. But when you've worked on sites where DDoS attacks occur daily, you understand the value of the ops guys and what they do.

In response to Werner, who is running the DDoS and IDS at Amazon? The devs? How stupid!

Steve Loughran
(July 12, 2006 12:56 PM #)

RichB: you are right that deployment covers config of routings, load balancing, etc, etc.

Where we differ is that I think it is possible to automate this set up. once you can automate it, someone can roll it out, in one single click, or more likely command line action.

Justin Mason
(July 12, 2006 01:07 PM #)

hi Bill --

I vote for Loughran, too. My experience of ops work indicated that too many packages are written by developers with no idea of how to make code that's actually _usable_ in the real world.

see also http://radar.oreilly.com/archives/2006/07/cloudy_with_a_chance_of_server_1.html .

Adam Constabaris
(July 12, 2006 03:02 PM #)

While I think there's more than a nugget of truth behind "you build it, you run it," if I read the comments correctly, the reasoning seems more obviously applicable to consultant-style work rather than in-house developer style work.

If you stick around the job long enough to have developed many applications, eventually your job will become more about operations than it will about development.

(July 12, 2006 03:53 PM #)

Keeping dev and ops together is a horribly bad idea. It sounds great in theory, but in practice I have seen this fail time and again. For example, I used to work for a very large financial services company that used this methodology for their trading software. The software is hugely popular in its market (it's #1 by a substantial margi, has 10k+ users, and accounts for billions of traded shares per month) yet from an IT management standpoint it is a complete mess.

Perhaps the result is peculiar to financial services where each transaction represents a lot of money, but the developers are protected to such a degree that there isn't much reason for them to tighten up their software or use version control or automate common tasks or write unit tests or do almost anything considered best practice. The software "as is" makes a shit-ton of money. So the implied feedback loop never really gets a chance to take hold. If it did, these developers would do nothing except ops work all day long. No one wants to pay $250k+ for an ops guy.

Lyn Robison
(July 12, 2006 06:01 PM #)

Interesting discussion. Two points come to my mind:

1) I believe the Sarbanes-Oxley act prevents developers from deploying and/or operating the applications that they develop, if any of those apps have anything to do with the company's financial statements. This separation of duties is a big deal for lots of applications.

2) It seems like the idea that developers should run their apps is impractical, because pretty soon, you have no one working in a developer role. The urgent demands of operations will always take precedence over development, so the company's development efforts will wither away over time. If developers are writing software that ops cannot support, ops needs to set standards for what they will and will not accept from dev. Applications should not move from dev to ops until ops says so.

Steve Loughran
(July 12, 2006 10:46 PM #)

I do not advocate that the developers own problems like managing the OS security patches on the servers, or the router config. Nor should they be the one fielding java.io.NoRouteToHost exceptions at 03:00. it aint fun. Being on call 24x7 destroys your freedoms.

At the same time, you can't just hand off to someone else and expect things to work. It wont. They will just call you at 04:00 instead of 03:00, and you still have to come in, only without the on-call cash bonus that ops get.

What you need is a team where dev and ops go hand in hand, to the extent that ops help design the system; their use cases are treated as importantly as those of the end users. The dev team needs to start automating deployment from day one, starting with the vmware images that ops build up and manage for them on their local system, so that each developer has a realistic stack to work with.

Then you need the developers using the management tools to deploy and debug the beast. So they instrument the app properly, give it good logging, etc.

Finally, all config problems are defects. file them in the defect tracker, write tests, then fix. Make your app able to detect problems and point ops at the defect reference.

Bill de hOra
(July 13, 2006 03:59 PM #)

Lyn: "It seems like the idea that developers should run their apps is impractical, because pretty soon, you have no one working in a developer role. "

This is a fallacy, in the same way "The network is reliable" is a fallacy. It's applying shrinkwrap economic thinking in a domain where it falls apart, namely SaaS. In shrinkwrap, developers doing ops is a commercial and organisational problem, because developers are supposed to be coding for the next product, not fixing the old one. But the profitability model for SaaS is different. You get paid for managing the service and making it valuable to users over time, not by reselling the same fixed code over and over. In SaaS maintenance *is* development. That's why it makes sense for Werner Vogels to pin teams to deployed services instead of what almost everyone else does, or has to do, which is pin them to projects/products.

You can't imo do software as a service well, based on shrinkwrap metholodogies. Shrinkwrap methodologies as deployed come from payment models, not process guidelines. Services are *by design* never finished. Shrinkwrap is *supposed* to finish, but rarely does. In the world I live in most of us are doing SaaS, but we don't really know it yet. Steve's years and years ahead of us on this one. He just needs to get an MBA so he can explain it in terms of commercials and business models instead of project methodologies :)

I'm guessing most of the problems people have with deployed services are down to applying shrinkwrap economics to the project - if developers are working on it after release you are de facto losing money. The organisation will not be a good structural fit for the reality of the deployed software. At best it'll be dissonant, at worst it will be a hellish environment to work in, because you'll *never* be giving sufficient time to work on anything that's assumed to be cash out. Over time, that will result in architectural decay - the system will rot from the inside, but it will need to be propped up no matter what.

Steve: "What you need is a team where dev and ops go hand in hand, to the extent that ops help design the system; their use cases are treated as importantly as those of the end users."

Along with test first development, I'll claim operations use cases and bringing depoyment into the life cycle are the most important extension to software methodology in a decade. If you are working to a shrinkwrap model, I expect this will sound like nonsense.

Lyn Robison
(July 14, 2006 01:14 AM #)

I agree with your statement that "SaaS maintenance *is* development". However, maintenance is *not* operations.

Correct me if I am wrong, but Steve Loughran's quote at the top of this post is about operations and deployment - not maintenance. Dave Thomas's quote is about administration, which sounds to me like operations - not maintenance.

So, I agree that it makes sense to keep developers in a maintenance role, but I do not see how it makes sense to put them in an operations, deployment, or adminstration role. In fact, it is often against the law - the Sarbanes-Oxley act forbids putting developers in any operations, deployment, or adminstration roles.

Lyn Robison
(July 14, 2006 02:22 AM #)

Just to be clear: having the person who writes the software be the same person who runs the software is a colossally bad idea.

A developer who writes software can more easily use it to steal money or valuable information if they also run that software. Until the nature of man changes, we will have to worry about these types of things.

Steve Loughran
(August 1, 2006 02:10 PM #)

Lyn, do you really think that I couldn't subvert any system I worked on, even if I didnt have my hand on the box?

Back doors are trivial to put in, and if you think you can find it by doing a code walk through, you'd better make sure that the release of Xerces or Axis doesn't have a back door I put in their source to handle a special XML processing instruction or other pattern I put in. Or you have to find the SQL-injection bug I left in my web service and decide whether it was just incompetence or maliciousness. Or maybe I dont put it in the source, I tweak ant or jikes to patch the binary. Better yet, I dont just 0wn my box that way, I 0wn yours too :)

You can't defend against the malicious by keeping dev away from production. That helps against the mildly malicious, not the skilled.

Similarly, why should I trust the operations team?

Geir Magnusson Jr
(August 4, 2006 03:11 PM #)

I'm with Steve, as I've been bit by this in my past lives. Developers better have a good understanding of operational requirements and support them - they have to be part of a projects deliverables, as they are usually just as important to the business as the business objectives themselves.

We aren't talking operational action here - not deployment or admin. But making software easy and safe to deploy and administer should be an explicit product requirement.

I've personally discovered the fastest way to get a developer to internalize this is put a list of developer's home/mobile/boat/vacationhome phone numbers in the console room of the operations team along with a list of who worked on what software and tell the operations team to feel free to escalate as needed.


Alan, web developer
(August 22, 2006 08:43 AM #)

You can't get one person rsponsible for all the stuff. Of course there should be deversification. The main thing in a team work is to trust each other.