Sri Lanka, when these pictures were taken, was called Ceylon. Before that, it was the island of Scerendip, whence the word scerendipity.
A few miles to the North, Chenai used to be Madras.
Mumbai used to be Bombay.
Growing up in the South, I'm well prepared for this dichotomy. The Battle of Antietem used to be the Battle of Sharpsburg. Bull Run, Manassas.
It's all just a binary form of victor's justice.
Anyway, the pictures are very cool.
"write blog entry about small-set intersection theory of macro-economics"
Okay. I'll do that.
Except I have no idea what the small-set intersection theory of macro-economics is.
I do have my own global macro-economic theory, but it has little to do with set theory, much less "small-set intersection theory". Which while being made of words is basically a bunch of gibberish.
I clearly need bigger post-it notes.
Until I get bigger post-it notes, or until I remember what I meant by small-set intersection theory, I'll just post a link to this site.
It's a map company. The maps they make and sell are just really really cool. They show global trends (things like poverty, imports, exports, fuel production, fuel consumption) by inflating or deflating the sizes of each country on the map. Brilliant way to show a lot of quantitative information quickly.
It's real easy to stay up on this kind of thing.
Any number of end-user portal sites can keep you up to date and in the loop. For instance, Google News lets you configure and save a custom news feed page, with feeds from almost anywhere on the planet.
My "Google News" page, which I check every morning when I get in to the office, has links for US Business, India general news, Sri Lanka general news, and Argentina general news. (I work with teams in India and Argentina, and my company has a partner in Sri Lanka...)
I find out about mergers and acquisitions in our space, bombings and floods and cold-war with Pakistan on the Indian sub-continent, Tamil rebels in Colombo, and soccer scores in Argentina. (either not much of merit goes on in Buenos Aires, or I picked a weak news agency...)
5 minutes every morning, skim skim skim, and I'm up to speed. Once every few days I find something worth a deep dive, and I come away better informed.
It really does go a long way with the teams, to pay attention to their worlds.
And as a business owner, you really need to understand geo-political risk, and pay attention to anything that could take your team out of commission for even a day.
Google News (or Google Desktop, or any of a number of other sites or tools) can let you do this for free.
I sat in on a status call with one of my teams today. We had one manager and three subject matter experts on our end, and a team of about 15 engineers and team leads on the conference phone in India.
I'll preface my commentary by saying that Monday morning quarter backs have the easiest job in the NFL. I've been leading calls with teams in India for a few years. I seldom get to sit and listen to my guys lead calls. So this is likely a case of "do as I say, not as I do".
My guy, who was leading the call, was doing a good job. He was holding our India team accountable for some very tough technical stuff. He knew what needed to be done, and clearly wanted to make sure that the guys on the other end of the phone were on the ball. So he asked a bunch of probing questions:
For example, he asked:
- What versions of Outlook are you testing this feature against?
Good question, I thought. Let's see what they say...
Then he continued...
- You're testing with Outlook from an Office 2000 and an Office XP installation, right?
Of course, they answered:
- Yes. We're testing with Outlook from an Office 2000 and an Office XP installation.
What did we learn in that exchange? Not much, really.
I'm cynical by nature. Communication, when based on cynicism, becomes a matter of "trust but verify". The guy leading the call missed a big opportunity. He asked a tough question, and proceeded in the same breath to give the folks he asked the answer. He got assurance that they were doing the testing "right", but he doesn't really know if they get it or not.
A better approach would be something like:
- Can you tell me what versions of Outlook you're testing with?
It's a simple change, but it's critical when attempting to communicate and lead without being in the room with the people doing the work. In that model, you get to find out exactly what the folks on the other end are doing and thinking.
I sum it up with a simple, easy to remember missive: ask, don't answer.
It gives the folks on the other end a chance to be smart, and if they don't get it, it gives you a chance to learn that and correct it, before it's a big problem. It may seem kind of mean spirited and distrustful, but I really think it's a win-win.
Ask, don't answer.
And by the way, do as I say, not as I do.
Here's the problem -- I've got about 25 people working on the QA team for our big flag-ship product. The product is a fairly complex client-server application, Windows and SQL Server on the back end, web-services command and control, Windows on the client-side. About a million lines of code, give or take. 10 years old. Lots of crufty features no one remembers except that one customer in Idaho who we did the feature for back in '98.
Here's the rest of the problem --I'm "re-inventing" the QA practice. I want a more methodical approach. I want deeper test coverage. I want the ability to make tradeoffs between risk and comprehensive test coverage. You can only do that if you have a concept of what comprehensive is.
So I'm doing a lot of work with this team to get them on the same page as me. I'm doing it first with my team here in the US. We're planning on bringing the contract team along the same path, phase-offset by about a week or two.
Here's the thought experiment I did with the team:
Thought experiment 1. Fill in the blank:
Our product is __________ deterministic.
I asked around the room, and got: non, mostly, not very, occasionally, purely, non, somewhat and usually.
Then I told them my answer. For starters, I pointed out that I didn't constrain the number of words they could put in the blank. I just asked them to fill it in. (There's an interesting point about conformity vs. "out of the box" thinking here, but we'll save that for another time.) My answers was "hyper-complex, but utterly".
The product is hyper complex. It's a million lines of object-oriented class-based code in at least 3 languages. It is so stateful as to appear, to human processing engines, random.
I used to say of this product that it is a "random crash generator". This is because, after analyzing empirical evidence from multiple product releases, I determined that as long as we keep testing we will keep finding defects at an asymptotically decaying discovery rate forever. Which means that there are infinite defects in the product. Which plotted against a finite number of code paths means it's effectively a random crash generator. I don't think that any more, and I told my team as much. In fact, I regret ever talking about that leap of flawed logic.
It's code. The code is not dynamic. It's very very very stateful, and it's operating against a matrix of highly complex and stateful background noise. But it's code. Code is, well, encoded. It does the same thing over and over, given the same inputs.
What we've failed to do is to understand the inputs. We don't grok all the salient points that can perturb the system. We haven't listed all the variables in this giant system, and we haven't determined all the states those variables can have. So we're treating the product as a black box, and we're treating our inputs as a black box. So it's no wonder the behavior of the product appears random to us. And it's no wonder the testing we're doing isn't having the effect I want, and isn't always giving us a clear insight into the quality of the compiled code.
Thought Experiment 1a.
It is QA's job to test _________________ product capabilities.
I got: major, some, all, all, all, 75% of, interesting, and new.
I said: "A statistically relevant sample set of".
Here's why I believe that: If we define all the variables that could impact this product, and we then map out all the combinations of all the variables, we'll easily conceive of millions and millions of test cases for this product. Because the product is so stateful, and because no one on my team can yet tell me that the combination (Variable1:Value3, Variable2:Value 8, Variable3:Value1) doesn't take the product down a different code-path than (Variable1:Value3, Variable2:Value 8, Variable3:Value2), all combinations of all inputs and environment variables are, unto themselves, valid test-cases. Which means there's millions and millions of test cases. (Take this as a given, since I don't want to write a whole essay about the complexity of the product in question.)
If there are even one million test cases, and they take 5 seconds on average to run, then running them all takes a cumulative 350 days of just compute time, ignoring the tens of thousands of man-days it would take to code the test cases in automation. Oh, and automation the test is your only hope of having each test case take only 5 seconds. Otherwise, you're probably looking at an average of 30 minutes to an hour per test case.
Which means that testing all the features is a nice idea, but not commercially practical.
(I don't have the math to describe this with precision, but we should be able to hire people who do. Here's the lay-version of the math:) So we've got to pick a statistically relevant sample size, and run tests against said sample size, hopefully also hitting common use case scenarios that we expect our costumers to hit on a frequent basis. Based on the number of combinations we tested, plotted against the number we believe to be present in the product, we know how much of the product we think we sampled. Then, based on the number of defects we found, and the severity, we can extrapolate what's left in the product. There are a lot of messy assumptions in this approach, but I think it's the only commercially viable approach given a hyper-complex product.
Given this belief, I think the first step in re-inventing the QA practice for this product is to scope the product, write down QA's view of it, and begin cataloging all the variables that can impact each component, feature or function. Then, we can figure out how to best organize that info, and can get our heads around how many test cases there would be, in theory, if we had forever to work on each of these releases.
It was an interesting thought experiment.
I'm writing it up and asking my team in India to do it as well, without telling them our answers. It will be interesting to see what they come back with.
If you don’t travel internationally that frequently, you should still get a phone to take with you. They're handy, particularly when you don't know the country, and you're counting on strangers and vendors to pick you up at the airport.
You can rent a GSM phone for short money. (Well, short money if it’s expensible.) They've actually gotten the provisioning for these down pretty well. You call the provider, ask for a GSM phone, tell them what countries you’re traveling in, and they ship you your phone. After that, it just works. They take a slug out of your credit card, and you burn down against it until hit hits some threshold, then they take another slug. It’s just like fast pass. It even comes in a box you can ship it back to the provider in…
Given that they do have phones in India, it's perfectly reasonable to ask why I'm traveling 7000 miles to visit my team.
I think its simple.
Relationships build results. In any team setting, you need to know the people involved. You need to have a face to put with the name and the userid and the avatar and the voice. If at all possible, you need to know when someone says, over a bad speaker phone connection "yes, I understand", whether they're saying it confidently, with a smile on their face, or questioningly, with their brow furrowed. You can't get that from IM, e-mail, or phone conversations.
Maybe on my next trip, my folks in India will point to a video conference terminal, and say "Do you have these where you come from?"
So, that extra $3000 a ticket gets you something pretty nice. I’m writing this from the upper deck of a 747 flying between Boston and London. (I'm actually posting this several weeks later, becuase I have a free afternoon...) Some hundreds of the unwashed masses, as I shall henceforth and forever call people who fly coach, sit behind and below me, cramped into their two and a half square feet of real estate. I – on the other hand – am sitting in a luxurious (though still evil and sadistic) 70 inch long sleeper pod.
- I have my own TV.
- I have a wool quilt.
- I have a full sized pillow.
- I have a little kit filled with earplugs and lotion and other apparent travel necessities I’ve foolishly been doing without my whole life.
- I have champagne and macadamia nuts.
Granted, each of these macadamia nuts probably cost my company $40, but they're awfully good.
The simple truth - Traveling to India Business Class, we will arrive in Bangalore in a condition to actually do some work that day. Traveling trans-Atlantic in coach results in a sore back and a 3-day stupor. It kills a day on each end simply recovering from the flight.
A strategic lesson from this is that if a US company wants to make global sourcing work, they better be willing to spend some money on airplane tickets. And if they’re already spending some money on airplane tickets, they should go ahead and pony up the money for business class.
An expert on this new-found luxury after one leg in business class, I have some travel tips:
- If you fly Virgin Atlantic, they give you free limo service to and from the airport.
- No matter who you fly, they give you a little kit with toothpaste, a toothbrush, and other toiletries. So you don't have to worry where you packed your cosmetic case.
- Free Magazines! Real Magazines. About stuff interesting enough to support ad revenue.
- And when you check in, ask if you’re flying in a 747. If you are, ask for an upper-deck seat. It’s quieter, and it’s an upper deck! On an airplane. That's just cool.
- If you fly business or first class, you get to hang out before hand in a lounge filled with other obnoxious rich people (or business people who would rather be somewhere else). It’s many orders of magnitude better than hanging out in the terminal with the unwashed masses.
- People with bratty ill-tempered children do not, it seems, fly Business Class. How sad for them.
This essay is extracted and de-identified from a presentation I gave to the assembled executive staff of my company, which I will herein call Big Co.
The presentation was introduced in the context of:
1) A “trip-report” on our recent tour through India and Sri Lanka, and
2) A follow on presentation to this statement, from our EVP of product development:
Collaborative multi-shore development is core to how we will build software and provide services, now and in the future.
My presentation started right after that. I had 5 slides, but the voice-over was the interesting bit. This is what I intended to say during that presentation. For the sake of this essay, presume that Big Co. bought Little Co., where I used to run the QA team, about 18 months ago. Now I work in Big Co., running a bigger QA team.
I was well prepared, but I was also “in the moment”. I think and hope I said something close to this…
Collaborative Multi-Shore Development
That’s a mouth full. But let’s not mince words. Collaborative multi-shore development means outsourcing and off-shoring. When we say “collaborative multi-shore development” we mean using contractors, somewhere else on the planet, to help us build and manage our technology infrastructure.
I started running multi-shore teams at Little Co. Soon after we were acquired by Big Co., our CEO (nod to the CEO) sat down to talk to the engineering management team from Little Co. When the topic of our then fledgling off-shore efforts came up, he asked me: “Why'd you go off-shore? To do it cheaper, or to do it faster?”
At the time I gave a very weak answer, saying something along the lines of “a little bit of both”. Since that time I've been on a fact-finding mission that's included my recent trip through India. And now I have a stronger answer. “Cheaper and faster? Sure. But we also outsourced to do it better.”
Big Co. shouldn’t struggle obtaining and maintaining skills that aren't core to our business. We should, as all of our customers have done, go out and find partners who can do those tasks for us – faster cheaper and better. That’s the promise of collaborative multi-shore development.
Just for background, India Inc. is how the Indian media refers to the burgeoning corporate economic sector in India. When you say “multi-shore development” today, you de facto reference sourcing work in India.
When we were in India recently, we met with the president of the Indian division of a large financial services company. He gave us a great sound-byte to describe his view of India Inc. “You come for the cost differential. You stay for the people and the culture.”
India Inc. has built itself on the back of an intensely intellectual culture, with free but highly competitive education. The Indian Institutes of Technology and Management (IITs and IIMs) are on par with the best technology and management schools on the planet. Think Cal Tech or MIT. Think Harvard Business School or Sloan. Think culture of innovation.
- India graduates more engineers each year than the all the member nations of the European Union combined.
- It’s a culture with a rapidly developing first-generation middle-class workforce.
- India’s population will pass China’s within the next decade.
- And the Indian IT outsourcing industry has adopted process and quality control standards like CMMI as a technical differentiator over their global competition.
“You come for the cost differential. You stay for the people and the culture.”
This is a big deal. This represents an opportunity and a threat. For all the reasons above, small software startups all the way to technology giants like Cisco, Microsoft and Google, view India Inc. as strategic to their technology roadmap. India Inc. represents a huge opportunity for Big Co. – Smart hardworking people who will do hard jobs for us, for less than it costs in the US? We gotta get us some of that.
But if our competitors beat us to the punch, then our service delivery cost will be too high, and our pace of innovation will be too low, and we’ll run the risk of failing to dominate our market. We can’t let that happen.
Today’s landscape at Big Co.
Just so everyone in the room here today has an understanding of where the company is, here’s a brief snapshot of our current multi-shore collaborative efforts.
(Pause to give folks a few seconds to read through the list of a dozen or so major software projects on the slide.)
Our efforts to date are largely grass-roots.
We have over 100 multi-shore engineers engaged, with at least 4 vendors, in at least 3 countries, doing almost any kind of technology work you could imagine.
This has given us great lift and capacity enhancement in some projects. I don’t mean to minimize the good work that’s been done, but this is a very hard way to make significant impact to the company’s trajectory.
Tomorrow’s landscape at Big Co.
We need to change that landscape. We need to realize the opportunity and avoid the threat of faster cheaper and better.
This is important stuff. It needs to be somebody’s day job.
We need to consolidate our engagements, and avoid a nickel and dime approach to multi-shore development.
We need to manage and on-board our staffing partners consistently.
We need to systematize our approach to multi-shore development. We need to make it frictionless, and make it the default position for all our technology projects.
We’re going to ask our managers to learn new skills – they’re going to have to lead people from a different culture, across 9.5 time zones. We need to train them on how to be effective at that.
We need to change our culture, and ensure that our employees embrace this as opportunity, instead of fearing it as a threat to their jobs. We need to change our grass-roots mode of operation, and do multi-shore collaborative development from the top down.
And most of all, we need to ensure that we assess and manage risk.
What we need from the executive team:
To get all this done, we need your help. You are the thought leaders of Big Co. You set the tone.
We need you to embrace globalization.
We need you to understand that globalization means not just selling and delivering globally – but also making and supporting globally.
We need you to embrace the cultural change we’re trying to effect.
We need you to speak with one voice.
People don’t like outsourcing. It’s hard. It’s nobody’s first choice. If our people sense dissent from you, they’ll drag their feet on these initiatives. They’ll “pocket veto” our multi-shore development efforts, and we will fail to realize the opportunity we have in front of us. And if that happens, we’ll end up with slower, more expensive and worse, instead of faster, cheaper and better.
(Applause and accolades. CEO decides we should consider outsourcing senior staff to India.)
The premise, I suppose, is that visas are supposed to control entry into a country. They fail pretty badly at that. I have a 10 year multi-entry visa for India. Here's the details of how I got it:
I found a company who handles getting visas for people. There are dozens of these on the Internet. I picked one that our corporate travel agent works with.
I downloaded their forms and filled them out.
I went to Kinkos and got two passport photos taken.
I (gulp) sent all the forms, my credit card info, the pictures and my current US passport to the visa company.
Presumably, they took the whole smash to the Indian consulate in NYC, and stood in line on my behalf.
A week or so later, I get a Fedex envelope with my passport and my visa in it.
So if the whole point is to control and regulate entry into the country in question, there are a few problems:
1) Am I who I say I am? My passport was issued 9 years ago. The years have not exactly been kind. 9 years ago I was a skinny kid with long curly sun-bleached hair. Now, I'm a middle-aged software guy, with short cropped thinning hair and 50 extra pounds of "shaped like my chair". You'd be hard pressed to look at my passport photo, then pick me out of a line up. But that's really a commentary on passports in general -- my passport basically proves that I'm a white guy who looks kind of like the white guy in the picture. But the guy in the picture was clearly a US citizen, cause he got a passport.
2) Am I who I say I am? - II: The way you get visas, or at least the way I got mine, further confounds the "am I who I say I am" problem. None of the forms have any certification or validation more sophisticated than a signature. So, the fact that all my forms are filled out and signed means basically that I'm some white guy who looks kind of like the white guy in the picture, and who can scratch a totally illegible scribble that looks like the totally illegible scribble in the passport.
3) Can I come into your country now? Once the forms are done, remember this whole package goes to some company who hires some one to stand in line at the consulate where they secure my visa, all official like. This proves, well, that I've got $150, which is about what that service cost. There were some invite letters required, and these had to be on company letterhead. So we've also proven that I have access to a color laser printer. So. I'm a white guy. Who looks like the white guy who got this passport 9 years ago. And I can scribble. And I had $150. And I might have access to a $300 laser printer. Stellar credentials. May I please come into your country?
Apparently, the answer is usually yes.
Glad we got that straightened out.
So the absurd chain of custody in this alleged "national security document" aside, let's talk about why visas even exist.
Here's what I've been able to figure out:
- Visas are largely reciprocal. If we (the USA) make people coming from Country-A get visas to enter the country, then Country-A makes US citizens get visas to enter their country.
- Unless there's a lot of US tourism going on in Country-A. In which case you can just show up, say you're going to spend lots of US dollars, and they'll let you in. So you can lie about the purpose of your trip, and say that you always vacation in business suits because it makes you feel sexy, and that you're really looking forward to sampling the wireless Ethernet in all the hot beach resorts, which is why you have your laptop with you. And the training manuals are to hand out to all the wee street urchins -- you know, candy rots their teeth.
- But that's unwise, because, well, prison.
- Not intending to start a riot, but just commenting on empirical evidence, the chances are, the more poor people or Muslims there are in Country-A, the tighter the restrictions are for entry from Country-A into the US. Which they can't take sitting down. So, a high quotient of poor people or Muslims correlates directly to more paperwork you'll have to fill out to get a visa to travel to Country-A on business.
- So the whole reason you have to get a visa is to allow Country-A to save face because we embarassed them and made their people get visas. Because we as a nation fear poor people and the statistically peaceful followers of Muhammad (PBUH).
- Once you get a visa, you can come and go as you please. Unless the visa expires. In which case you can get a new one, but you have to be in the USA to do that. And you have to go to the consulate's office. In NYC. Unless you have $150. And a laser printer. And you know somebody who looks like the picture in a valid passport. And you can scribble. Then you can get someone to get your visa for you. Then you're free to come and go as you please again.
- I suppose, if you were clever, you could even get around the whole "being in the USA" thing. But it's very unlikely that you could or would do that. You'd have to have access to, well, Fedex.
- crash and burn.
- flip it up on deck.
- catch it on the first bounce.
- ping me.
- you're yankin' my chain.
- that dog don't hunt.
If you're reading this in the 'States, you probably know what I mean. But if you grew up in the Punjab, and learned your English from an Anglican Church missionary from Hull, those phrases don't mean dick. Which also doesn't mean dick.
I had this driven home in a very dangerous way in the last few weeks.
"Ad hoc testing" is a pretty simple phrase. Ad hoc is Latin for, literally, "to this". Figuratively, I've always thought it meant "impromptu". That's pretty precise, and not subject to interpretation. You'd think.
When we say "ad hoc testing", we mean testing without a script or plan. Trying to break software. Some folks call it "fault method" testing.
One of my QA managers has had his teams doing periodic "ad hoc testing" during the last 5 months of this release. The idea was to run both the test cases we have, and to just kick the tires. When we went into Beta, we found boatloads of bugs that shouldn't have been missed in QA. Hmm. Not good. We sniffed around a little, and discovered a very fundamental communication disconnect.
We said "do ad hoc testing."
They heard "do ad hoc testing."
We thought we meant "do impromptu testing, exercising various divers parts of the product, in mean-spirited ways, to see if you can break the software."
They thought we meant "run test cases randomly selected from the thousand or so test cases we have written, and which we have already run."
We wanted them doing exploratory work, exercising code paths that were unlikely to have been hit thus far in the project. They ended up doing a whole lot of regression testing.
If you know anything about software, you know that these are very different things.
I hate to be trite, but communication is tough. I've turned some of the text above blue, just to visually highlight where I used imprecise language or idiom. It's really hard to write in a precise way, that is not subject to multiple interpretations.
Throw in a bad speaker phone and nine and a half time zones and the problem can be crippling.
I don't have any non-obvious ideas on how to fix this, though I'm still running a background thread on this, and I expect to post more on this later.
With a Vice President of Engineering, and an Executive Vice President of Product Development. Who have neither of them traveled in AsiaPac before. Visiting two of our staffing vendors. And 50 of our contractors. And a bunch of potential customers. And some sage people in the biz. In 3 cities. In 2 countries. Covering 18,575 miles. On 7 different airlines. In 4 different hotels. Over the span of 11 days. It is like herding cats. It gives me a headache.
And it throws into stark relief some interesting points about doing this outsourcing thing in Asia.
Some thoughts that are front of mind:
- Visas are stupid. More on this later.
- When it comes to visas, vendors are stupid. More on this later too.
- Bangalore India is a boom town. Good luck getting a decent hotel room with less than 6 weeks notice.
- Vendors are a critical in-country support mechanism. All global staffing vendors should become vertically integrated and offer trip-planning as a value-added service for busy software executives. More on this later.
- Traveling alone is a lot easier than traveling with your boss. And his boss.
- Traveling in India, as a software executive, is best treated as an amusement park ride. Buy your ticket, stand in line, get on, and go for a ride. You have little control. That's okay. Become comfortable with it. More on this later.
- Getting shots before a trip to Asia is probably a good idea. Who knew? This fact will likely freak out the average software executive who has never traveled in Asia. More on this later.
- Expedia is better than Travelocity. Faster, more comprehensive, and better at finding flights between cities in Asia.
- As an aside to that point, trusting Expedia, or our own travel agents, makes me very nervous. God knows what our connections and flight transfers are going to be like...