25.4.08

Outsourcing and Security


Someone asked me this week if I had any thoughts or opinions about outsourcing and security. It turns out that I have both thoughts and opinions, so I thought I'd write a post about this.

I see the problem of security as related to technology outsourcing breaking down like this:
  • Confidentiality or privacy breach
  • Intellectual property theft
  • Sophisticated or inadvertent attack insertion
  • Physical security
  • Personal security and safety
I'll talk about each of these, and then talk in some detail about what I've seen done to address each.

Start with Trust

Ultimately, outsourcing partnerships are built on trust. Without this you will always have an adversarial relationship with your outsourcing partner and you will never enjoy the productivity and quality of work you might otherwise experience. I preach incessantly (in this blog and my other endeavors) that you have to think of your outsource team as if they were your own employees. In order to get to that level of shared fate you have to have trust. In order to trust, you probably have to first become paranoid about what could go wrong and then make sure you have contingency plans in place for everything you can think of...

After Trust, move on to Verify

"Trust but verify" is a great mantra when thinking about security. While this should not be construed as actionable or legal advice, I believe that you can't have effective security without a great measure of trust. For without this your security measures are likely to paralyze your business (see the photo above for a good example of this).

That said, part of the "verify" is the due diligence and audit aspect of any well run sourcing program. When selecting a vendor, references are key. Sideways references (people you know who work with the vendor, not necessarily references your vendor provides to you) are even more powerful. Before I move in to specific security risks and abatement measures, I'll give a couple of examples of "trust but verify" in action:
  • If your vendor claims to do security background checks on all its employees, verify their assertion: First, discuss with them that you will randomly audit the results of these background checks. Tell them you'd like to see redacted output of the background checks, and give them 24 hours to clear that with their lawyers. Then, give them 5 random names from your team, and ask them to produce the evidence of their security background checks. How they handle this will tell you a great deal about what they're really doing with respect to security checks. If the results are unsatisfactory to you, switch back to "trust" and tell them what they have to do to satisfy your concerns, and give them a time frame to accomplish that. Test them again when the time frame is over.
  • If your vendor claims to have 24-hour a day security cameras in all the data centers, follow the same routine above, and ask to see tape from last Thursday at 02:45 local time. (or any other random but recent time period.) If they can't produce this, figure out why, and give them corrective action to fix the problem.
If you take this approach of trusting what they say, but auditing what they do, you and your vendor will have a good business relationship that will grow further trust through time. (Or you'll figure out before it's too late that you partnered badly...)

Now, on to specific threats and how to manage to them:

Confidentiality or Data Breach

Protection of customer and employee data is highly regulated. Loss or inadvertent exposure of credit card numbers, social security numbers, or even names and addresses can have massive personal impact to the people whose data was disclosed and devastating financial results to the company who lost or disclosed the data. This security concern should be taken very seriously. The concern is compounded when "people who don't even work for us" in "some country half way around the world" are interacting with this data as part of day to day operations.
  • Transactional data within CRM (and other) systems - Think about the data that is present on a computer screen when you call technical support. The screen in any CRM application will have name, address, potentially social security numbers, and enough data to facilitate identify theft. This has to be guarded. CRM transactions should be logged for both reads and writes. Read access should be granular, and where possible critical components should be masked, and revealed only on an as-needed basis. In this, abstraction is your friend - if you have to store things like Social Security Numbers, do so in an abstracted table, and don't make that data element part of your CRM work flow.
  • Test data - This is an often overlooked security hole, prevalent with Offshore Product Development (OPD) teams. Test data sets are often cobbled together from production data. If that's the case, they will have real names, social security numbers, etc. in them. Disclosing this to third parties (or often to test engineers within your own company) can be construed as breach of confidentiality. Hence, you should de-identify all your test data. I won't write the regular expressions for you, but names should become random strings, numbers should be randomized, and for US data sets, any string of the form ###-##-#### (Social Security Number format) should ABSOLUTELY be randomized or redacted. I'll leave it to you to figure out how to do this without compromising your test results, but this is important stuff.
  • Data Residency - Irrespective of compliance concerns it seems intuitive to me that companies should not host sensitive data outside their own domains unless there are very strong security and contractual measures in place. A popular expedient (for American companies) is to host all customer and financial data exclusively in US data centers, to ensure that there is no potential for other jurisdictions to come in to play and seize or otherwise access company or customer data. This space is also highly regulated, and in come cases there is legal obligation around where data must reside, and a lot of complex case law about what levels of out-of-country access are allowable. This is one to talk about with your corporate counsel.
  • Printouts and other movable media - Many high-security offshore operations centers I've toured through the years are "paperless." This is to prevent a purposeful or inadvertent screen dump that results in critical data ending up in a waste bin, or in a briefcase. I've seen further measures in place wherein the offshore operations center is not networked to the public carrier Internet, but only has a route (through secure VPN) to the client's network. In this case, the remote network is effectively "air-gapped" to prevent inadvertent or purposeful file transfer. Obviously where security is paramount laptops are probably to be avoided and I've seen centers that only used diskless network consoles. And there are some great software solutions that prevent the use of USB drives, bluetooth, and IR ports, effectively making it impossible to transfer media to small mobile devices.
  • Third party attacks - This may seem like an unlikely risk, but it should be mitigated nonetheless. Fortunately, best practices in network management cover much of this. If your network traffic is run through a secure encrypted VPN, third party sniffing should be effectively eliminated. One thing to think about though is human to human "phishing" attack. Your offshore staff should be trained not to respond to such attacks, potentially to the point of having a procedure or protocol in place to ensure that no one at the vendor site is fooled by a third party claiming to be a representative of the client.
  • Contractual concerns - Obviously breach of data security is breach of contract. Your contract should be clear on this point. Work with your lawyers to make sure that you have an agreement that protects (and indemnifies) the concerns of the client in this case. This is particularly important where the "local" (China, India, Vietnam) laws might be construed to be in force with respect to data privacy. You probably want to have clauses in your contract that define jurisdiction as "local" to you (the client). Again, I'm not a lawyer, so please talk to yours about this.

Intellectual Property (IP) Theft

This concern gets a lot of play in the press, probably because of highly publicized software piracy in China, dating back 5 or 10 years. Aside from that, I haven't read or heard a great deal of credible instances where outsourcing firms stole trade secrets or IP. In my mind this breaks down into just two facets:
  • Assignment of IP and inventions - If you have an OPD team, and they're good, they will innovate for you. They'll invent stuff. They'll think of new ways to solve old problems. It's important that you work with your lawyers to put contractual vehicles in place to convey that innovation back to you, if that's your intent. Every OPD firm I've worked with has a keen grasp of this, and they actually want the IP to convey back to the client. An obvious tactical measure I've seen almost universally employed is to keep source code repositories in the client site, accessed remotely.
  • Inadvertent cross-contamination of IP - This one is tougher to manage. Say you hire an OPD firm, and they do some work for you, and one of their engineers comes up with a great new optimization algorithm for you. You're very pleased, until your main competitor sues you because that engineer learned said optimization algorithm while working for your competitor on contract through the same outsourcing firm. It's a bit of an unlikely scenario, but worth considering nonetheless. Usually OPD firms will already have a contractual structure that prevents cross-contamination. To implement that protection they may ask you who your main market competitors are, and if they happen to work with your competitors they will embargo transfers between the respective teams to prevent cross-contamination of IP.
Sophisticated or inadvertent attack insertion

This one has always seemed like a Fear, Uncertainty and Doubt (FUD) campaign to me, but in the interest of being thorough, I'll speak to it as well. It is possible with an outsourced technology development or delivery team to be subject to attack insertion. (Though it's just as possible that these attacks will come from badged employees...)
  • Purposeful, sophisticated attacks - I live in New England, and frequently shop at a local supermarket chain that shall herein remain nameless. They recently experienced a malware attack that resulted in the theft of millions of credit card numbers. This is not common, but the results can be devastating. To prevent this kind of exploit, you have to take a holistic approach to security, and ensure best practices in all aspects of your IT infrastructure. You open the door slightly by having a third party contractor in a position of trust. Ensure that your contractors have gone through the same background checks as your employees. Ensure that security access is granular, and is granted provisionally, and that all high security transactions are logged and scrutinized. If you have an OPD team, quietly ensure code inspection of their work, to guard against backdoor attack insertion.
  • Inadvertent attacks - These are more common and insidious. This hit me once, several years ago, and resulted in me having to take my R&D lab down for several days to patch vulnerabilities in a hundred or so SQL servers. A virus entered our lab network through a secure VPN from our OPD partner's test network. The systems there were not running anti-virus software, because we needed to test (for a variety of reasons) on base operating systems. The net result was a damaging virus infection resulting in the loss of two days of productivity. But it could have been worse if the lab network hadn't been isolated. For this one, certainly air-gap your lab networks, and ensure to the extent possible that all systems are up to date with security patches and other protective measures. In short, do what you already do for your own networks.
Physical security

If you have a sizable offshore team, doing ITO or OPD, you will likely end up with capital assets (either owned or leased) in your offshore center. If those assets are stolen, you will lose both the equipment and significant productivity so physical security within your offshore center should be a concern.
  • Access - Physical access to your offshore development or delivery center should be controlled. Your vendor should not allow unauthorized personnel into your center, just as you would probably not allow strangers into your office building. It will be up to you to work with your vendor to define "unauthorized." I've seen this range from casual (with a receptionist just inside the door smiling at people as they walk in) to formal (with armed security guards who inspect ID, issue visitor badges, and go through luggage looking for disallowed media or other devices). I've also toured centers that don't allow cell phones or digital cameras on-site. You'll have to decide what's appropriate for your business. While I wouldn't recommend this because it might be risky, I've worked with people who visit their offshore center unannounced in the middle of the night and try to "scam" their way into the building. This could land you in a (Chinese / Russian / Brazilian) jail, or you could politely and appropriately be escorted off the premises. Be careful with this one, but it could tell you a great deal about how seriously your partner takes physical security.
  • Monitoring - Access (badge swipe) logs and video monitoring are pretty common in this industry. But cameras are useless if the files are thrown away daily. And access logs are useless if the doors are frequently propped open. Ask to see your partner's policy on video retention, and figure out how long you need them to keep tape. Also, if you care to have a record of entry and exit from your center, physically inspect the site to be sure there are cameras appropriately positioned to see all the doors, that all the doors have badge readers, and that all the doors are kept locked.
  • Access Control - I'll throw this one in under physical security. I have read many times, and my own experience confirms the fact, that the most common attack results from social engineering. I've heard stories of OPD firms who forced their employees to record their usernames and passwords in clear text - in a spreadsheet on a shared drive - so management could control and manipulate access to client systems. This is a gross violation of security best practices. But just as bad is the post-it note on the monitor, with usernames and passwords to various systems. People will always do this, but you can train it out of them, you can force them to change their passwords frequently, and you can implement a good identity management system with single sign on.
Personal security and safety

Last but not least, you should think about the physical security and safety of your employees and contractors.
  • Your offshore center - You should inspect and audit the security at your offshore center. If you get a "bad vibe" about the facilities, you should seek corrective action from your vendor. If your company has a Security Officer, involve him or her in the audit, to get a professional opinion about whether your vendor's measures are appropriate.
  • Your staff, when they travel - You should put some thought into the safety of your staff, if they travel to your offshore center for knowledge transfer, employee exchange, etc. I won't give you actionable advice on this point, but personally, I would think twice about setting up an BPO center in Iraq right now, even if it was free.
I'm sure there's important stuff I'm forgetting, but this list is what's top of mind for me on the topic of security and outsourcing.

No comments: