Here's the problem -- I've got about 25 people working on the QA team for our big flag-ship product. The product is a fairly complex client-server application, Windows and SQL Server on the back end, web-services command and control, Windows on the client-side. About a million lines of code, give or take. 10 years old. Lots of crufty features no one remembers except that one customer in Idaho who we did the feature for back in '98.
Here's the rest of the problem --I'm "re-inventing" the QA practice. I want a more methodical approach. I want deeper test coverage. I want the ability to make tradeoffs between risk and comprehensive test coverage. You can only do that if you have a concept of what comprehensive is.
So I'm doing a lot of work with this team to get them on the same page as me. I'm doing it first with my team here in the US. We're planning on bringing the contract team along the same path, phase-offset by about a week or two.
Here's the thought experiment I did with the team:
Thought experiment 1. Fill in the blank:
Our product is __________ deterministic.
I asked around the room, and got: non, mostly, not very, occasionally, purely, non, somewhat and usually.
Then I told them my answer. For starters, I pointed out that I didn't constrain the number of words they could put in the blank. I just asked them to fill it in. (There's an interesting point about conformity vs. "out of the box" thinking here, but we'll save that for another time.) My answers was "hyper-complex, but utterly".
The product is hyper complex. It's a million lines of object-oriented class-based code in at least 3 languages. It is so stateful as to appear, to human processing engines, random.
I used to say of this product that it is a "random crash generator". This is because, after analyzing empirical evidence from multiple product releases, I determined that as long as we keep testing we will keep finding defects at an asymptotically decaying discovery rate forever. Which means that there are infinite defects in the product. Which plotted against a finite number of code paths means it's effectively a random crash generator. I don't think that any more, and I told my team as much. In fact, I regret ever talking about that leap of flawed logic.
It's code. The code is not dynamic. It's very very very stateful, and it's operating against a matrix of highly complex and stateful background noise. But it's code. Code is, well, encoded. It does the same thing over and over, given the same inputs.
What we've failed to do is to understand the inputs. We don't grok all the salient points that can perturb the system. We haven't listed all the variables in this giant system, and we haven't determined all the states those variables can have. So we're treating the product as a black box, and we're treating our inputs as a black box. So it's no wonder the behavior of the product appears random to us. And it's no wonder the testing we're doing isn't having the effect I want, and isn't always giving us a clear insight into the quality of the compiled code.
Thought Experiment 1a.
It is QA's job to test _________________ product capabilities.
I got: major, some, all, all, all, 75% of, interesting, and new.
I said: "A statistically relevant sample set of".
Here's why I believe that: If we define all the variables that could impact this product, and we then map out all the combinations of all the variables, we'll easily conceive of millions and millions of test cases for this product. Because the product is so stateful, and because no one on my team can yet tell me that the combination (Variable1:Value3, Variable2:Value 8, Variable3:Value1) doesn't take the product down a different code-path than (Variable1:Value3, Variable2:Value 8, Variable3:Value2), all combinations of all inputs and environment variables are, unto themselves, valid test-cases. Which means there's millions and millions of test cases. (Take this as a given, since I don't want to write a whole essay about the complexity of the product in question.)
If there are even one million test cases, and they take 5 seconds on average to run, then running them all takes a cumulative 350 days of just compute time, ignoring the tens of thousands of man-days it would take to code the test cases in automation. Oh, and automation the test is your only hope of having each test case take only 5 seconds. Otherwise, you're probably looking at an average of 30 minutes to an hour per test case.
Which means that testing all the features is a nice idea, but not commercially practical.
(I don't have the math to describe this with precision, but we should be able to hire people who do. Here's the lay-version of the math:) So we've got to pick a statistically relevant sample size, and run tests against said sample size, hopefully also hitting common use case scenarios that we expect our costumers to hit on a frequent basis. Based on the number of combinations we tested, plotted against the number we believe to be present in the product, we know how much of the product we think we sampled. Then, based on the number of defects we found, and the severity, we can extrapolate what's left in the product. There are a lot of messy assumptions in this approach, but I think it's the only commercially viable approach given a hyper-complex product.
Given this belief, I think the first step in re-inventing the QA practice for this product is to scope the product, write down QA's view of it, and begin cataloging all the variables that can impact each component, feature or function. Then, we can figure out how to best organize that info, and can get our heads around how many test cases there would be, in theory, if we had forever to work on each of these releases.
It was an interesting thought experiment.
I'm writing it up and asking my team in India to do it as well, without telling them our answers. It will be interesting to see what they come back with.
If you don’t travel internationally that frequently, you should still get a phone to take with you. They're handy, particularly when you don't know the country, and you're counting on strangers and vendors to pick you up at the airport.
You can rent a GSM phone for short money. (Well, short money if it’s expensible.) They've actually gotten the provisioning for these down pretty well. You call the provider, ask for a GSM phone, tell them what countries you’re traveling in, and they ship you your phone. After that, it just works. They take a slug out of your credit card, and you burn down against it until hit hits some threshold, then they take another slug. It’s just like fast pass. It even comes in a box you can ship it back to the provider in…
Given that they do have phones in India, it's perfectly reasonable to ask why I'm traveling 7000 miles to visit my team.
I think its simple.
Relationships build results. In any team setting, you need to know the people involved. You need to have a face to put with the name and the userid and the avatar and the voice. If at all possible, you need to know when someone says, over a bad speaker phone connection "yes, I understand", whether they're saying it confidently, with a smile on their face, or questioningly, with their brow furrowed. You can't get that from IM, e-mail, or phone conversations.
Maybe on my next trip, my folks in India will point to a video conference terminal, and say "Do you have these where you come from?"
So, that extra $3000 a ticket gets you something pretty nice. I’m writing this from the upper deck of a 747 flying between Boston and London. (I'm actually posting this several weeks later, becuase I have a free afternoon...) Some hundreds of the unwashed masses, as I shall henceforth and forever call people who fly coach, sit behind and below me, cramped into their two and a half square feet of real estate. I – on the other hand – am sitting in a luxurious (though still evil and sadistic) 70 inch long sleeper pod.
- I have my own TV.
- I have a wool quilt.
- I have a full sized pillow.
- I have a little kit filled with earplugs and lotion and other apparent travel necessities I’ve foolishly been doing without my whole life.
- I have champagne and macadamia nuts.
Granted, each of these macadamia nuts probably cost my company $40, but they're awfully good.
The simple truth - Traveling to India Business Class, we will arrive in Bangalore in a condition to actually do some work that day. Traveling trans-Atlantic in coach results in a sore back and a 3-day stupor. It kills a day on each end simply recovering from the flight.
A strategic lesson from this is that if a US company wants to make global sourcing work, they better be willing to spend some money on airplane tickets. And if they’re already spending some money on airplane tickets, they should go ahead and pony up the money for business class.
An expert on this new-found luxury after one leg in business class, I have some travel tips:
- If you fly Virgin Atlantic, they give you free limo service to and from the airport.
- No matter who you fly, they give you a little kit with toothpaste, a toothbrush, and other toiletries. So you don't have to worry where you packed your cosmetic case.
- Free Magazines! Real Magazines. About stuff interesting enough to support ad revenue.
- And when you check in, ask if you’re flying in a 747. If you are, ask for an upper-deck seat. It’s quieter, and it’s an upper deck! On an airplane. That's just cool.
- If you fly business or first class, you get to hang out before hand in a lounge filled with other obnoxious rich people (or business people who would rather be somewhere else). It’s many orders of magnitude better than hanging out in the terminal with the unwashed masses.
- People with bratty ill-tempered children do not, it seems, fly Business Class. How sad for them.
This essay is extracted and de-identified from a presentation I gave to the assembled executive staff of my company, which I will herein call Big Co.
The presentation was introduced in the context of:
1) A “trip-report” on our recent tour through India and Sri Lanka, and
2) A follow on presentation to this statement, from our EVP of product development:
Collaborative multi-shore development is core to how we will build software and provide services, now and in the future.
My presentation started right after that. I had 5 slides, but the voice-over was the interesting bit. This is what I intended to say during that presentation. For the sake of this essay, presume that Big Co. bought Little Co., where I used to run the QA team, about 18 months ago. Now I work in Big Co., running a bigger QA team.
I was well prepared, but I was also “in the moment”. I think and hope I said something close to this…
Collaborative Multi-Shore Development
That’s a mouth full. But let’s not mince words. Collaborative multi-shore development means outsourcing and off-shoring. When we say “collaborative multi-shore development” we mean using contractors, somewhere else on the planet, to help us build and manage our technology infrastructure.
I started running multi-shore teams at Little Co. Soon after we were acquired by Big Co., our CEO (nod to the CEO) sat down to talk to the engineering management team from Little Co. When the topic of our then fledgling off-shore efforts came up, he asked me: “Why'd you go off-shore? To do it cheaper, or to do it faster?”
At the time I gave a very weak answer, saying something along the lines of “a little bit of both”. Since that time I've been on a fact-finding mission that's included my recent trip through India. And now I have a stronger answer. “Cheaper and faster? Sure. But we also outsourced to do it better.”
Big Co. shouldn’t struggle obtaining and maintaining skills that aren't core to our business. We should, as all of our customers have done, go out and find partners who can do those tasks for us – faster cheaper and better. That’s the promise of collaborative multi-shore development.
Just for background, India Inc. is how the Indian media refers to the burgeoning corporate economic sector in India. When you say “multi-shore development” today, you de facto reference sourcing work in India.
When we were in India recently, we met with the president of the Indian division of a large financial services company. He gave us a great sound-byte to describe his view of India Inc. “You come for the cost differential. You stay for the people and the culture.”
India Inc. has built itself on the back of an intensely intellectual culture, with free but highly competitive education. The Indian Institutes of Technology and Management (IITs and IIMs) are on par with the best technology and management schools on the planet. Think Cal Tech or MIT. Think Harvard Business School or Sloan. Think culture of innovation.
- India graduates more engineers each year than the all the member nations of the European Union combined.
- It’s a culture with a rapidly developing first-generation middle-class workforce.
- India’s population will pass China’s within the next decade.
- And the Indian IT outsourcing industry has adopted process and quality control standards like CMMI as a technical differentiator over their global competition.
“You come for the cost differential. You stay for the people and the culture.”
This is a big deal. This represents an opportunity and a threat. For all the reasons above, small software startups all the way to technology giants like Cisco, Microsoft and Google, view India Inc. as strategic to their technology roadmap. India Inc. represents a huge opportunity for Big Co. – Smart hardworking people who will do hard jobs for us, for less than it costs in the US? We gotta get us some of that.
But if our competitors beat us to the punch, then our service delivery cost will be too high, and our pace of innovation will be too low, and we’ll run the risk of failing to dominate our market. We can’t let that happen.
Today’s landscape at Big Co.
Just so everyone in the room here today has an understanding of where the company is, here’s a brief snapshot of our current multi-shore collaborative efforts.
(Pause to give folks a few seconds to read through the list of a dozen or so major software projects on the slide.)
Our efforts to date are largely grass-roots.
We have over 100 multi-shore engineers engaged, with at least 4 vendors, in at least 3 countries, doing almost any kind of technology work you could imagine.
This has given us great lift and capacity enhancement in some projects. I don’t mean to minimize the good work that’s been done, but this is a very hard way to make significant impact to the company’s trajectory.
Tomorrow’s landscape at Big Co.
We need to change that landscape. We need to realize the opportunity and avoid the threat of faster cheaper and better.
This is important stuff. It needs to be somebody’s day job.
We need to consolidate our engagements, and avoid a nickel and dime approach to multi-shore development.
We need to manage and on-board our staffing partners consistently.
We need to systematize our approach to multi-shore development. We need to make it frictionless, and make it the default position for all our technology projects.
We’re going to ask our managers to learn new skills – they’re going to have to lead people from a different culture, across 9.5 time zones. We need to train them on how to be effective at that.
We need to change our culture, and ensure that our employees embrace this as opportunity, instead of fearing it as a threat to their jobs. We need to change our grass-roots mode of operation, and do multi-shore collaborative development from the top down.
And most of all, we need to ensure that we assess and manage risk.
What we need from the executive team:
To get all this done, we need your help. You are the thought leaders of Big Co. You set the tone.
We need you to embrace globalization.
We need you to understand that globalization means not just selling and delivering globally – but also making and supporting globally.
We need you to embrace the cultural change we’re trying to effect.
We need you to speak with one voice.
People don’t like outsourcing. It’s hard. It’s nobody’s first choice. If our people sense dissent from you, they’ll drag their feet on these initiatives. They’ll “pocket veto” our multi-shore development efforts, and we will fail to realize the opportunity we have in front of us. And if that happens, we’ll end up with slower, more expensive and worse, instead of faster, cheaper and better.
(Applause and accolades. CEO decides we should consider outsourcing senior staff to India.)
The premise, I suppose, is that visas are supposed to control entry into a country. They fail pretty badly at that. I have a 10 year multi-entry visa for India. Here's the details of how I got it:
I found a company who handles getting visas for people. There are dozens of these on the Internet. I picked one that our corporate travel agent works with.
I downloaded their forms and filled them out.
I went to Kinkos and got two passport photos taken.
I (gulp) sent all the forms, my credit card info, the pictures and my current US passport to the visa company.
Presumably, they took the whole smash to the Indian consulate in NYC, and stood in line on my behalf.
A week or so later, I get a Fedex envelope with my passport and my visa in it.
So if the whole point is to control and regulate entry into the country in question, there are a few problems:
1) Am I who I say I am? My passport was issued 9 years ago. The years have not exactly been kind. 9 years ago I was a skinny kid with long curly sun-bleached hair. Now, I'm a middle-aged software guy, with short cropped thinning hair and 50 extra pounds of "shaped like my chair". You'd be hard pressed to look at my passport photo, then pick me out of a line up. But that's really a commentary on passports in general -- my passport basically proves that I'm a white guy who looks kind of like the white guy in the picture. But the guy in the picture was clearly a US citizen, cause he got a passport.
2) Am I who I say I am? - II: The way you get visas, or at least the way I got mine, further confounds the "am I who I say I am" problem. None of the forms have any certification or validation more sophisticated than a signature. So, the fact that all my forms are filled out and signed means basically that I'm some white guy who looks kind of like the white guy in the picture, and who can scratch a totally illegible scribble that looks like the totally illegible scribble in the passport.
3) Can I come into your country now? Once the forms are done, remember this whole package goes to some company who hires some one to stand in line at the consulate where they secure my visa, all official like. This proves, well, that I've got $150, which is about what that service cost. There were some invite letters required, and these had to be on company letterhead. So we've also proven that I have access to a color laser printer. So. I'm a white guy. Who looks like the white guy who got this passport 9 years ago. And I can scribble. And I had $150. And I might have access to a $300 laser printer. Stellar credentials. May I please come into your country?
Apparently, the answer is usually yes.
Glad we got that straightened out.
So the absurd chain of custody in this alleged "national security document" aside, let's talk about why visas even exist.
Here's what I've been able to figure out:
- Visas are largely reciprocal. If we (the USA) make people coming from Country-A get visas to enter the country, then Country-A makes US citizens get visas to enter their country.
- Unless there's a lot of US tourism going on in Country-A. In which case you can just show up, say you're going to spend lots of US dollars, and they'll let you in. So you can lie about the purpose of your trip, and say that you always vacation in business suits because it makes you feel sexy, and that you're really looking forward to sampling the wireless Ethernet in all the hot beach resorts, which is why you have your laptop with you. And the training manuals are to hand out to all the wee street urchins -- you know, candy rots their teeth.
- But that's unwise, because, well, prison.
- Not intending to start a riot, but just commenting on empirical evidence, the chances are, the more poor people or Muslims there are in Country-A, the tighter the restrictions are for entry from Country-A into the US. Which they can't take sitting down. So, a high quotient of poor people or Muslims correlates directly to more paperwork you'll have to fill out to get a visa to travel to Country-A on business.
- So the whole reason you have to get a visa is to allow Country-A to save face because we embarassed them and made their people get visas. Because we as a nation fear poor people and the statistically peaceful followers of Muhammad (PBUH).
- Once you get a visa, you can come and go as you please. Unless the visa expires. In which case you can get a new one, but you have to be in the USA to do that. And you have to go to the consulate's office. In NYC. Unless you have $150. And a laser printer. And you know somebody who looks like the picture in a valid passport. And you can scribble. Then you can get someone to get your visa for you. Then you're free to come and go as you please again.
- I suppose, if you were clever, you could even get around the whole "being in the USA" thing. But it's very unlikely that you could or would do that. You'd have to have access to, well, Fedex.