Engineering a solution to security

An internet of threats

In today's digital age, a multitude of services are accessible online through apps, websites, and internet connected devices. Whether you're booking a holiday, arranging a flight, shopping, ordering a meal, or adjusting your home's temperature, chances are you're doing it over the internet.

Even if you pop to the shops, or make bookings over the phone, service providers are operating 'digitally'. Nearly every service provided to the public is an integration of a number of platforms, software, and services communicating over public networks. All built from a mix of custom software on a foundation of commodity libraries and supporting systems.

This provides a broad threat surface where malicious actors can look for ways to enter and exploit your systems and the data they hold, for fun and profit.

If you own or run one of these systems, you'll be dealing with these risks on a day to day basis. Odds are, there are more risks in your Risk Log than you have staff to secure them. Not only that, but you might well have a large group of engineers, building, experimenting and shipping novel ideas and new features far faster than you can keep up. All systems built out of commodity software libraries provide a constant stream of vulnerabilities.

The threat surface is always growing - evidenced by the reports of hacks, data exposures and ransom attacks that appear weekly. But you cannot police every risk or change. So what can you do: delegate, enable, or police?

A solution shape

As a Staff Engineer faced with this challenge back in the late 2010s, I pitched and formed a three person Engineering Security team to enable the Product teams to make good security decisions and actions.

We acted as an Enabling team, informing and influencing choices, bringing in new knowledge and building an Internal Developer Platform to raise the knowledge and leverage all our engineering team.

Why this shape

I was inspired to choose this shape for three reasons.

Reason 1. Don't be the brakes

How we delivered was important and I didn't want to damage that success. We had a set of autonomous product teams who experimented, operated and delivered products that our customers loved. Centralising controls, placing external gates, or reducing team ownership of the whole problem was not a recipe for success.

Reason 2: optimise for speed and success

We needed good enough security at the right time in the right places. Putting security demands where the work was, would both provide leverage (we had many engineers) and would allow decisions to be made in context.

Reason 3: a valued pattern

Coming from a software consultancy background, I was used to structuring and delivering organisational change. An enabling team was a common pattern when introducing complex change, able to work from the trenches, understanding the needs and often dog-fooding and trialing changes before scaling them out. Team Topologies has cemented this concept as one of four key team types in modern software delivery.

Beyond this, there was existing work that showed that engineers were a good place to invest in security. In 2015, Shannon Lietz, a Security leader, said that teaching a security person how SW delivery works ”takes a lot longer than teaching a developer how to understand how to make security happen and change how security gets remediated” - (GOTO 2015 • The Road To Being Rugged).

Why not Security champions?

Security Champions is a pattern that has become popular in the last 5 years and is a way of scaling out security. It focuses on having a single point of contact for security in each team. A team Security Champion operates as a communication conduit and a review point of security quality in the team, performing security code reviews and dedicating time to security initiatives.

I don't think it's a great model on its own. While it might complement other strategies, it has several failure points worth avoiding.

Silos: a problem with lone champions

Teams are the unit of software delivery. The members of a product team should have all the talent that is needed to build great products in a domain. We often support them with utility and education platforms - supporting a self service model that removes silos. Silos of knowledge and control slow down getting stuff done by gatekeeping and destroying team ownership and flow.

Just as external security control creates one of these silos, security champions distribute but don't remove the silo, leading to a risk of gatekeeping and a higher bus-factor within the team. To avoid a silo, it places the enablement effort in the champion, something they may not have the skill or drive for, as they likely signed up through an interest in security.

Security Champion alternatives

Every company and situation is different. But, if you want to raise the engineering standards of a set of teams, focus on how you can do so without generating silos and bottlenecks. Given these limitations of Security Champions, focus on enabling teams to do a better job and to be able to build good enough software. Whatever you provide, keep the barriers to entry low, make it on demand and as self service as you can.

What could an enabling team bring that would allow them to scale?

Consider a platform to raise standards

When folk use the phrase 'platform team' it often takes us immediately to a team that provides on demand compute and other operational components that deliver software to users. With the complexities of modern infrastructure, it's a clear place to manage with a platform wrapper to hide the complexity and provide utility.

Internal Developer Platforms can serve different goals, and security is a great candidate. You can read here how our Engineering Security team built a living standard that worked as part of a platform that scaled the company's security expertise. Along with that we built reporting tooling to provide calls to action and feedback and new tooling, as well as providing support and education, engaging with the challenges our teams had.

Consider Communities-of-Practice to sustain interest

Of course, we totally wanted to sustain interest in managing security, and support engineers interested in becoming knowledgeable and skilled in this area. Our Engineering Security team ensured everyone could join in and learn what they could, and that there was a space where folk interested in security could converse and contribute. We were part way to building what would now be called a Community of Practice[1] - a gathering of like-minded people where a skill or interest can be honed and improved.

Foster Collaboration and Empowerment

In wrapping up, it's clear that the digital landscape is fraught with challenges, but by fostering an environment where product teams are empowered to make informed security decisions, we can maintain the agility and creativity that drives success.

The approach of enabling rather than policing; and focusing on education and shared knowledge, ensures that security becomes a seamless part of the development process. This strategy not only mitigates risks but also enhances the overall resilience of our systems. As we continue to navigate the complexities of digital threats, let’s commit to raising the tide of knowledge and capability across all teams, ensuring a secure and thriving digital future.