We clearly still don't want no foreign rulers, as we keep crashing spaceships into the martian terra firma because we still don't use a consistent universal system of weights and measurements.
But that's a different story, and not really what I wanted to talk about today. I just thought it was a clever title for a second meditation on the topic of metrics.
Along with my teams, I've been putting a lot of thought into the topic of metrics lately. I think there are a few basic metrics that are absolutely required in order to to provide executives, managers and leaders the high-level date necessary to follow trends within a team, to highlight any areas of concern, and to shine light on areas where the teams have done exceptional work.
The first area is cost. This all presumes that you care about cost, and that you are in a partnership that has negotiated rate structure, that's variable based on skill set. This won't always be the case, of course, but it is for me, so here's what I think makes sense:
- Average blended labor rate -- This is the overall cost, on the "buy" side, of a person-day (or what ever your standard is) of work. This should be averaged across your whole partner team, including management.
- Average blended labor rate, fully burdened -- This is the same as above, but includes hidden costs like air fare, capital equipment, software licenses, bandwidth, T&E, and telco costs. This is important, because in any outsourced partnership or engagement, these hidden costs can represent a significant pile of money.
- Performance against budget -- this is pretty straight forward, though measuring it can be tough in any organization. Basically, you'll want to know if your project, partnership, virtual team, etc. is on track -- What's your budget, how much of it have you burned up so far?
- Run-Rate - Run rate is your "latest" monthly (or any other period of time) invoice amount. This ends up being an important predictor, since it's very easy in offshore partnerships to miss big spikes of growth, particularly in distributed relationships. You'll need to use this metric as an input to a forecast model, that will give you a good guess on where you land at the end of your budget cycle, and also, what your one to three year spend forecast should be for your team or project. (with the understanding that this isn't applicable for all situations...)
- Labor Cost Differential -- This one is only important if you think you care about "cheaper," as in "cheaper than US-based engineers". In this one, you'd use your US fully burdened labor rate (for years, I've used $140,000 a year, and no one has been able to give me a better number, so it's probably pretty close for most East Coast teams...) You'd then take a quotient: (Offshore rate) divided by (Onshore rate) should give you a quotient. In most partnerships, and in most offshore situations, it will be favorable. Please note, this is not a measurement of cost effectiveness, just of how well you've negotiated. To get to cost effectiveness, you have to factor in productivity, which makes it much tougher.
The second area is team composition. This isn't a dynamic metric, in as much as it will only change slowly through time, but it's important to understand the following:
- Team Composition, by function - I think it's important to follow how many people you have on your team, by function. It's just something you should know off the top of your head. By example, I have 16 QA engineers, 2 SysAdmins, and 12 C++ developers.
- Team Composition, by seniority - This is also important. If you're working with or for a really well managed company, with a mature PMO function, you might have tools at your disposal to produce staffing pyramids, that graphically display this information. If not, Excel works well too. You basically want to know how many "freshers" you have, how many engineers with one to three years experience, three to five, etc. It doesn't matter how you slice up the bands (though it would help if it matches the terms of your contract, or your internal "career ladder"), but you need to pay attention to this. I won't opine about the "right" mix, but you need to put some thought into what that right mix is, and you need to staff accordingly. Obviously, a team of 45 engineers straight out of college isn't going to give you great product, and just as obviously, a team of 45 engineers with 20 years experience might be tough to build, keep and wrangle. Seek balance with this one.
- Management structure and hierarchy - It's odd to think of an org chart as a metric, but it kind of is. It's easy, in my experience, to find yourself in a situation where one senior manager may have 17 direct reports. If your team is in India, they may tell you that this is perfectly normal. It might be, but it still feels risky to me, and I still don't support a staff to manager ratio that high. I like to watch the team hierarchy,
The third area is hiring and attrition. This is either a critical success factor, or a red herring, depending on the situation. Either way, it's important to measure.
- On the hiring front, I don't generally care about how many resumes my partners sources, or what their phone-screen to interview or interview to hire ratios are. They should care, so they can become efficient at hiring. What I care about is hiring latency. This is critical to understand, as it impacts how much advance notice I need in order to have a team up and learning. I like to keep an eye on running average -- how long does it take in my market, to go from "start recruiting" to "start date" for a person with the skill set I usually try to hire.
- Hiring ratio -- Contrary to what I wrote above, there is one case where I do care about efficiency in hiring -- that's when I have my front-line managers interviewing all the candidates we bring on to the teams. (As an aside, this is expensive and time consuming, but sometimes it is very useful to do this, as it builds buy-in with the onshore management team, and correctly sets the bar with the offshore management team.) If I do have my managers interviewing, I like to know how many of the people my partner "short-lists" end up getting thumbs-up from my staff. This is a measure of how well my vendor "gets" the staffing requirements my teams are giving them.
- Hire to "on-boarding" ratio - I care about this one because of a problem I've experienced. Often, in my recent experience, my partner will extend an offer to an engineer (in India, by the way), the engineer in question will accept the offer, give us a start date, then "no-show". Everyone stops recruiting, expecting that we've got the talent we need, then they fail to show (usually using the offer they got to drive up their wage with another company). It's not a great practice, and I hope the engineers in question end up somehow punished, if only karmicly, for their misrepresentation, but I like to measure this, as it illustrates 1) how well my partners are selling themselves, and 2) how well they're reading their prospects.
- New Hire Attrition -- If someone quits in their first 90 days on the job, it's usually because they took the wrong job. It represents an annoyance, a set back, and a loss of productivity, but it is a different problem than losing someone who's been with you for three years. I measure this separately, because I think it reflects the partners ability to find the right person for the job. It's really a late-stage "no-show" measurement, and I like to cluster these two metrics together.
- Rank-and-File Attrition -- Obviously, you never tell the rank-and-file that they're rank-and-file, but in any team, there's "standouts" and there's "the team". Attrition from this portion of the team is "just bad". Meaning there are worse people to lose than rank-and-file engineers. This needs to be treated as a vector, to the extent possible. That means you need to track it monthly, for ever, so you can watch for trend lines.
- Key-staff Attrition -- Key-staff can, very privately, know that they are key staff. These are your leads, your high-fliers, you architects, your managers. Losing one of these folks is tough on your whole organization. This is also a vector.
- Managed attrition -- Knowing someone is going to leave, having a relationship with them that allows them to tell you this, and having more than two weeks to find and groom a replacement is really the best case scenario, aside from keeping someone on your team, happy and productive, forever. People change jobs. If they do it professionally, I can live with it. This is a measure of churn on the team, but also of how well the managers build rapport with the team members.
- Surprise Attrition -- In India, and in much of the world, 2-weeks notice is considered "professional". In my world, this is both unprofessional and irresponsible. So, if someone surprises us with a "today's my last day", or "I'm giving my 2-weeks", I don't like it. I count these very carefully, and I like to get root cause on each of them.
- Attrition root cause -- It's not very helpful to look at a high attrition number, declare that it should be lower, and to then consider your job as a leader "done". You need to know why your people are leaving. Is it repetitive work? It is poor treatment by the onshore team? Is it sub-par work environment? Salary and benefits? Don't like their boss? All these are different problems, requiring different solutions. Measure root cause, and commit yourself to fixing the problems reflected therein.
The forth area is performance and execution. There isn't a lot I can say here, except in the abstract. This will change with the team, the function, the product, the SDLC, and every other variant of work and work norms imaginable. Suffice to say you should measure performance, as objectively as possible.
- Performance against expectation - The method will vary, but it's important to know if your team did what you expected them to do, asked them to do, or what they committed to do. Measure this as scalar (i.e., discrete tasks, over discrete time intervals) but watch it as a trend.
- Quality of work product - Again, a million ways to measure this, but it's critical to measure. Maybe you look at code-review comments, maybe you look at bugs inserted, maybe you look at call resolution time, but look at something, consistently. It helps if you have a target here, so if you don't have a target, make one up, and make it 5 or 10% better than what you're currently measuring, and keep the quality trend improving.
On this point, it is worth a bit of soap-boxing. If you have a "virtual" team, i.e., a co-sourced effort, where some of "your" employees are working together with some of your outsource staff, you need to measure both sides of the partnership equally. You can't measure your offshore team, and let your onshore team coast. It isn't fair, and it will never produce a favorable outcome.
The last area is very subjective. You want to measure how your team "feels" about the offshore partner or staff. This is mushy, by necessity. What you care about is consistency in measurement, and the trend line. Here's the three areas where I think it makes sense to have a consistently measured but utterly subjective metric and trend line.
- Trust - Does your management team trust their offshore staff? This can be a simple question you ask every month (Survey Monkey is great for this). It should probably be a slider scale. Some people like a five point scale, some people like a 100 point scale. I don't think it matters. You should just consistently ask the stake holders if they trust the partner. You should de-identify the results, average them, and watch the trend line. If there's an outlier, that is, if someone is way off scale, you should break the "anonymity" rule and intervene, and figure out what's going on.
- Confidence - How confident is your management team in the their offshore team component? Do they believe their team "gets it"? Again on this one, the scalar is arbitrary and subjective. What matters is the trend, and outliers from the trend.
Last bit of proselytizing for this post -- If you choose to measure trust and confidence, you need to measure both sides of the relationship, and get a subjective analysis of how much your offshore team trusts you, and how confident they are in your leadership, work, and partnership activities.
There's a lot more stuff you could measure, and probably a lot more stuff you should measure, but this is my short list, at the moment, of stuff I think you must measure.