How I stop worrying and let a project ship

Consider a scenario similar to the previous post, where there’s some big security issue in a project that wants to ship soon. Instead of stopping the project in its tracks, you decide to let it go. Why is that?

Reasons to let it ship

  • Investigate further and discover whether the security issue is really exploitable. As a side benefit, maybe this will give mitigation ideas. (More on this later)
  • We’re here to build useful products and make money, not to build a perfect work of art. Maybe after stepping back and thinking about it some more you can convince yourself (and others) that it’s actually a reasonably secure design despite the issue you’ve found.
  • You don’t want to damage the relationship with the product team because it will hurt the overall mission.
  • The number of people on earth that have the skillset or access necessary to exploit the issue is in the single digits, and none of them are going to be motivated to do so.

Mitigations

Rather than just letting the project ship unmodified, maybe you can work out some kind of mitigating controls. Examples:

  • See if the riskiest parts of the system are really necessary for the initial launch. I’ve often had success in modifying a launch to include the bulk of the feature set without some risky component that isn’t a critical feature.
  • Spend more on the initial operating expenditure so that there’s more robust hosting and controls in place, with a plan to ship an improved version that doesn’t require this soon after launch. An example would be to separate risky processing into sandboxes or on different servers, with a plan to move to a more robust and operationally cheaper solution soon after launch. Presumably there will be a slow ramp up of initial users, so it might not be a significant expense.
  • Monitor for abuse through logging and metrics.
  • Plan to launch iterative security mitigations as time goes on. With the right timeline for shipping iterations, you might not give attackers enough time to develop their capabilities to the point where customers are at risk.
  • Label the product as “experimental” or “beta”, to make it clear that it’s not quite production ready.

These are just a few off the top of my head, and it’s nowhere near exhaustive. There are going to be many things to consider depending on the project. The specific mitigations will need to take into account many factors, and working with others to come up with a solution might allow a good compromise.

Authority

Up to here I’ve made assumptions at different points about who has the authority to accept a risk and ship a project or to deny a project from shipping. That’s going to vary significantly between different companies and between different teams within the company. Sometimes you will have absolutely no authority and must lead without authority to convince others that it’s better to mitigate a risk. In other cases you will be empowered to say no.

If you do have the authority to stop a project launch you should consider only using it when absolutely necessary, as you have a significant responsibility not to unnecessarily impede the business.

Ethical responsibility

Don’t let anything I’ve said here discourage you from doing the right thing. A high impact vulnerability that’s trivial to exploit can cause significant harm. As security professionals we have a responsibility to help the business succeed but also to avoid obvious harm to customers. Finding the right balance is sometimes difficult, but by establishing a good working relationship with colleages and developing a reputation for being helpful rather than always being a naysayer you’ll find greater overall success.