31.1.08

Dig Safe!

Suffice it to say Dig Safe does not provide services in the Mediterranean Sea. The story I linked to (which may not be a permanent link) is about an internet service outage all over the Middle East, that hit Indian tech firms particularly hard today. (They think the cable cut, off the coast of Egypt, was due to a boat anchor, so Dig Safe wouldn't have helped. If there are any entrepreneurs out there with a big boat, a sophisticated GPS system, and a detailed map of international cable runs, I bet you could get funding this week...)

This reminds me that I've been wanting to write about my Doomsday game, which I'll do later this week. But it also points out a largely un-vended requirement for multi-site development tools (repositories, build farms, defect tracking systems, schedule and task allocation systems) with lazy replication.

It also points out that VoIP, while cheap, probably needs a POTS fall back in everyone's DR plan.

Anyway, bad news for some poor sap in Egypt, and big headaches for network engineers all over the planet...

Tool of The Man

I said something the other day that surprised me. I've since pondered my statement enough to believe it's an interesting idea.

To set the stage, I was listening to someone complain about how the employees of their company needed to grow up, act like adults, and start treating communal property with some respect. The rant in question was about a common issue that I suspect plagues most companies -- how do 40 busy engineers share two projectors?

The issue is that people "borrow" the gear and don't return it when they're done. When the next person (who incidentally waits until the last minute) can't find it, and there's a big flap. Both parties get angry and bring the hate, but neither party learns their lesson and starts acting like adults about shared resources.

My knee-jerk reaction to this rant was to give up entirely on the notion of sharing, and to utter this:

When I run my own company, I'm going to equip each of my employees with all the tools they need to do their jobs! They'd each get a projector, big flat-screen monitors, a fast light laptop, a mini-tower under their desk, their own printer, and whatever else they needed to do their job. It might cost $15,000, but it's worth it. They'd get it all on their first day. Or maybe I'd give them a budget and let them shop for themselves. But they would get the tools they needed to do the job I asked of them. If a tool broke, I'd have it fixed. If one got stolen, I'd have it replaced.

I've never worked any place that took this approach to tools. I've always had to fight to get "the good stuff" for me and my teams. It's usually the bean-counters that push back, but the push back is just nuts.

Let's say your US-based engineer costs you $75 an hour. If you're a typical R&D shop, Engineering represents maybe a third of your overall expense base. YMMV. Still, in this scenario, if you're a profitable company, that means you need to make a little over three dollars for every dollar you spend in engineering, in order to remain profitable. If you shift your thinking to opportunity cost rather than General Ledger impact, your engineer has an opportunity cost to your company of over $250 an hour.

Thinking about my own career, I can guarantee that in December alone I spent over 10 hours walking back and forth to the printer, to ensure that highly confidential contracts and performance reviews were picked up (by me) as soon as I printed them. All that walking back and forth and waiting nervously over the laser printer cost my company $2500 in opportunity cost. A brand new Dell personal color laser printer starts at $300. I know, because I tried to buy one. Coincidentally, I was shot down by the IT Purchasing Standards Committee.

And I can not even count the number of hours I've spent trying to track down a projector, or the number of times I've had to cancel or postpone a meeting because the eight or nine participants didn't have a way to look at project plans together. For what it's worth, entry-level projectors run about $600 now.

I can't quantify how much more productive I'd be if I had these two pieces of equipment all to my very own, but my gut feel says plenty more productive. At least enough to justify the expense.

Presume it costs $15,000 to outfit an engineer with the best of everything he or she might need to do a GREAT job. (That number might be low, might be high, depending on the kind of work we're talking about, but let's use it for the sake of argument.) Most companies balk at that kind of expense, and they skimp out. But if you can save that employee one hour a week of productivity... if you can keep them on-task, keep them from having to waste time trying to track down a projector for a meeting, you can actually save money. If you give them back an hour a week, you've broken even from your $15,000 on-boarding investment in 15 months. This says nothing of the message you've sent to your employee -- You're important. We're giving you the best tools possible so you can do your very best at the important work you do for us. We love you, you brilliant but poorly socialized whiz kid. Now, go forth and be excellent so we can book more revenue.

A few years back, when I was setting up a contract for a partnership with an R&D company in India, I put a term into the contract which, apparently in radical departure from the standard of the day, stipulated that technical leads and managers would be issued laptops rather than desktop systems, at my expense. This simple upgrade in tools was viewed as an immensely positive act by the engineers in India. It boosted productivity and morale, and most importantly, it sent the message that the work the team was doing was valuable, and that I wasn't afraid of making capital investments to allow my team to do their best work. It was "revolutionary" at the time, and very hard to get past the accountants. But in retrospect, I should have gone much deeper, and provided a much higher standard of tools. I should have contracted that everyone get the best tools available, not simply "adequate" tools.

Like I said, when I run my own company, it's going to be laptops and laser printers for everyone. Oh, and because... well, because WANT... a new Macbook Air for me, please.

30.1.08

This defies conventional wisdom...

... but I've come to the conclusion that SDLCs (software development life cycles) are rubbish -- universally more trouble than they are worth.

I've worked in big and small companies and sooner or later every company I've spent time at started tinkering around with their SDLC (or their PDLC, or their ISO compliant seven stage development methodology, or their standard project template, or what ever they called it).

I put some thought into this recently. Why is it, I pondered, that organizations sink time and money into a second-order problem space with frequently dubious results? (As an aside, if anyone is reading this blog, I'd love to hear your experiences with "SDLC re-envisioning projects" via the Comments for this post).

Some of the emphasis on this meta-work may stem from deflection tactics -- Revenue is down because our process is broken...

Some of it may be a misguided attempt to scale -- We need to define our process tightly so we can double our team size and still keep the same great culture of innovation that got us here.

Some it may be a wrong-headed attempt to abstract people out of the project teams, and to treat people as fungible resources -- We need process uniformity so senior management can easily and quickly deploy resources to the most strategic projects, with the highest return on invested capital.

None of these reasons make good sense when you weigh them against the amount of work required to get five or six product teams to do their work with uniformity - on five or six disparate products, with disparate technology stacks, disparate maturity levels, and most importantly disparate people.

Plus, those reasons I came up with sound contrived. So I thought a little more about what's behind the contrivance. And I've since come to think, based on my experience, that organizations usually tinker with their SDLC for one of the following reasons:

  • A new executive joins the team, with commitments for a quick turn-around, and with an a priori prescription for greatness
  • An existing executive reads a new book on a new process tweak that promises a quick turn-around, and offers a ready-made prescription for greatness
Again, in my experience, these two potentials have one of the following root causes:
  • Things are going badly - Performance is weak, projects are late, revenue is flagging, customers are unhappy, productivity and morale are down, etc.
  • Things are going well - Performance is strong, the numbers are up, and the team is undergoing massive growth - meaning that there is a dilution of the "core" technical leadership team
That causal chain may not be worth much, as it basically boils down to this: good news = new process. bad news also = new process.

I haven't gone so far afield as to issue a fatwa against process orientation. But SDLC definition is not in any way the same as process orientation, and in fact it's a poor substitute.

Talking about and standardizing how you do your projects is a second order problem space, and tinkering here is almost always an inflection point. And not the good kind. It's frequently a zenith. An apogee. A high point before a precipitous decline.

The best teams I've worked with (and sometimes have been privileged to lead) could have produced great product with high quality, on time and per plan, without any predefined process, or in spite of an utterly broken SDLC. The worst teams I've worked on (or in some cases I'm reluctant to admit, that I've helmed) couldn't have produced great product even if they had a process framework that defined every step along the way, and all they had to do was code-by-numbers.

Again, in my experience it's common to look at that broken team and blame the failure on process. It's more interesting to look at the successful teams and figure out what it is about them that makes them work well, irrespective of process.

If you think back to the great teams you've been on, I bet one thing will pop out as common:

Teaminess.

Yeah, I know, that's not a word, but I bet you get my meaning. Teaminess: everyone knows what they're doing, and why. There aren't any jerks. People get along. They have a shared fate. They know their jobs. They know how to do stuff, when to do it, and why. They have lots and lots of teaminess.

Maybe they have all their spec templates, decision gates, and feedback loops defined in advance, maybe they don't. But with all that teaminess, it really doesn't matter. Because as a team, they will sense if there is a breakdown, a weak spot, a risk, or a lack of definition. And they will fix it, as a team.

I wrote last week that I believed software engineers shouldn't improvise. I still believe that, even though this essay seems to be taking me in an orthogonal direction from there. Bear with me.

It's not import an to have all the SDLC steps, responsibilities, etc. defined externally. It's important to have them defined within the context of the team doing the work. That is, to extend the Jazz metaphor, your team (the band) gets together and does the composition -- they write the score, themselves. They decide the key, the tempo, how much they put on paper, what cadenzas they leave for which virtuosos. But they DO NOT spend several weeks or months fretting about how to make sure their templates and numbering schemes for their specs match the templates and numbering schemes for specs produced by some other product team in some other campus.

The best teams I've worked with developed their own understanding of what they were doing, how, when, and by whom, right in the context of the project. I argue that this is a lot better and cheaper than a meta-process definition, because it's closer to the work.

That led me to the notion that I think is clever -- rather than having an external process definition, and training people to use it, develop a concept of a project glossary. If people don't already grok why they need to do this, point out the obvious -- for people to work together on a team, they need to have a common understanding of (and vocabulary for) a few things. Off the top of my head:
  • What problem are they trying to solve?
  • Who are the beneficiaries of the solution?
  • How do those beneficiaries want to interact with the solution?
  • How will the team document requirements and decisions?
  • What artifacts will the a team produce (specs, test plans, executable product, etc.) on the way to the solution?
  • At what level of granularity or detail will those artifacts be developed?
  • What are the commonly held coding standards and expectations?
  • What tools (and what versions of those tools) will be used?
  • When do you start tracking defects?
  • What level of detail is required for defect reports?
  • What specific project vocabulary do you have, and does everyone understand it?
  • What things are required, and what things are "nice to haves" before you call this thing done?
That's by no means a comprehensive list. There more questions teams want a common answer to, and certainly there will be variance in this list project by project.

But I think the big idea here is to define this in a small local context, very close to the work, with the involvement of all the people in the work. You write all this stuff down, you've got a project glossary. Put it in a wiki, and you've got a really powerful tool to help your team with its teaminess.

I worked with an executive recently who was fond of saying something like: "people will support a universe they help create". I suppose this project glossary approach is the logical extension of that notion.

You could try to define all this stuff in advance, externally to a given project, and impose it from above. But that would be second order work. It would be a waste of time. It wouldn't help the team; it wouldn't work for them; they wouldn't adopt it; and it would give them a ready-made scapegoat for failure.

A better, cheaper and ultimately more successful approach (that I've seen work many times) is to get the team together, the entire team, and spend the first several days (or weeks for big projects) of the project laying this stuff out, and getting everyone on the same page. It may seem like extra work, particularly when most projects in my career have seemed to be three months late to market on the day they started. But this approach will work, and in my experience, externally defined and mandated SDLCs or process models promise a lot, but deliver very little.

Like I said, I'd love to people's thoughts on this, so please offer your comments.

25.1.08

My article from Global Services

This links to my recent article on Global Services, about "faster, cheaper, better, closer". The content is the same as the galley version I blogged a few months back, but the Global Services site itself might be interesting.

Jazz improv

My favorite album is Thelonious Monk - Alone in San Francisco.

Monk was a musical genius - a self-taught master of composition and improvisation. He played with a veritable who's who of jazz giants - Duke Ellington, Dizzy Gillespie, Charlie Parker, Sonny Rollins, Miles Davis...

Alone in San Francisco is, as the title suggests, just Monk, playing his piano. Ostensibly in a room with some recording equipment, somewhere in San Francisco.

I love it because it's improvisational. It's filled with unexpected starts and stops. The phrasing is sublime, and as you listen to it you know that what you're hearing is pure emotion, expressed through the keys and pedals of a piano. You're left with the feeling that this is music, pure and pristine music, that could never be captured in a score and could never be reproduced by anyone, probably including even Monk himself.

Remember, Monk was musical genius.

Geniuses get to improvise.

Neophyte piano students get to play scales, and maybe after a year or two they get to move up to three-four time waltzes and, if you're me, your apogee might be a dumbed-down version of Scott Joplin's The Entertainer.

But neophyte piano students, mere mortals in the musical world, do not, as a rule, get to improvise.

They might try, and it might be fun for them. But to achieve what Monk achieved, to be able to take raw emotion and turn it into beautiful music - that is beyond them. They might try to improvise, but unless they are exceptionally gifted, it won't be fun for anyone who has to listen to them try. And the results will mostly be crap.

As an engineer, I've always thought of music as "just another symbolic language", and mastery of an instrument as "just another human-machine interface." If you accept, for an instant, this overly dismissive broad-brush definition, we can apply my thoughts about Thelonious Monk and the art of improvisation to software.

Most software engineers I have worked with think they're closer to Thelonious Monk than they are to that neophyte kid picking out a C-major scale on the old upright in the basement. That is to say, they like to think of themselves as geniuses, as artists, as masters of the sublime.

Generally, they're none of those things. Most engineers I have worked with like to think they're capable of producing great code, great features, great quality - all as inspired acts of improvisation. Really, they should stick to the score.

To tie this back to tangible advice on software development and global teams:

  • Don't make stuff up as you go.
  • Don't allow engineers to work solo for long periods of time.
  • Document expectations for work norms, coding practices, and standards.
  • Hold everyone on the project accountable to those standards.
That is to say, don't improvise, follow the score.

Leave enough room for creativity, but don't allow people to simply riff on their own.

Unless, of course, you are very very lucky, and you work with a team of geniuses. In which case, by all means, take a theme, develop it, and Go man Go! Who knows? You might enjoy some great riffs, like the collaborative improvisation recorded in Miles and Monk at Newport.