Hey what happened to my code? | Secure Code Access

What good is a well-designed application, that is vulnerability free, when an attacker can modify the source code?

This is the fourth part in our ‘What the Sec is DevSecOps?’ series.

History has taught us that even a well architected application, designed with security in mind, can be open to exploits. An example we like is from 2013, when the hacktivist group known as ‘Hack The Planet’ (HTP), published an ezine boasting they had, through a backdoor, compromised Nmap, a popular open source network security scanning tool. It’s an amusing example, due to the irony that Nmap is a security tool.

A much more popular example is that of Ken Thompson: “It’s one of the great computer security lessons”. In the early 1980s, widely trusted computer pioneer Ken Thompson, one of the original creators of Unix, admitted that he had inserted a hidden backdoor into a source code compiler. In his acceptance speech for the Association for Computing Machinery’s most prestigious honour, the Turing Award, Thompson admitted that he had created a parasitic C source code compiler that, when it detected source code for login programs it was compiling, would generate a backdoor known only to himself. He could essentially log onto any system that had used his source code compiler for logon access control programs.

In short: Don’t cast aside all your hard work in training, designing and implementing security measures in your DevSecOps pipeline with misconfigured access to your application code and build process that leaves you wide open. In this post, we’ll cover the next phase of the DevSecOps journey – Securing your code & image repositories.

We’ll dive into the need to ensure that a secure access control framework is in place for your application source code, image and artefact repositories and any systems that can access these, for example CI/CD tooling; and effective strategies that you can implement.

Code is a valuable asset!

Protecting your source code should be a core security focus of your organisation. The repositories containing your code, compiled artefacts and images are as important as what gets delivered through your pipeline into production. Code is an asset. Attackers implement modern ‘Tactics, Techniques and Procedures’ (TTPs) that are extremely effective, and an attacker who knows every detail of how an application works will find a way to exploit it. The added danger is that the attacker will possess the ability to access and modify these artefacts and create backdoors – compromises that could easily go undetected.

Every second, millions of requests are sent to popular sites such as GitHub, scanning for code repositories left open and containing the super-secret sauce hackers are hoping to find. Scans of public cloud services (such as the popular recent classic – S3 buckets) are happening continuously to locate and abuse misconfigured and overly permissive access to resources.

Start with Access Control Principles

When securing anything, the principle of least privilege is a great place to start. It is an industry best standard principle, providing standards for Identity Access Management, and segregation of privilege and duty. Its foundational ethos is that users are provided only the access required for their role, and nothing more.

Platforms, cloud services, and applications that you start to use will likely have default passwords and default access profiles that are more permissive than you need. Roles and ClusterRoles provide access to resources, based on sets of objects and actions that can be taken on those objects. We commonly see these created within Kubernetes clusters as a result of users looking to package managers such as Helm, and relying on open source packages and modules. If you’re using a package manager to install say, NGINX or Istio, you may find that the default Roles created ship with wildcard “*” permissions. The use of wildcards is not optimal from a security perspective, as it may allow for inadvertent access to be granted when new resources are added to the Kubernetes API, either as CRDs or in later versions of the product. The use of wildcard rights grants is likely to provide excessive rights to the Kubernetes API. By applying sensible limitations to access control, the damage an attacker could make to your system(s) is limited. Some example scenarios can be found here.

Supply Chain and Allow-listing

In our previous post we covered ‘reinventing the wheel’ and ‘velocity. It’s common practice to reuse code from external contributors – open source software or third-party components. Although it is arguable that community-led initiatives are better and more secure (or have at least been reviewed by many more peers), it is important to be aware of the risks associated with this, such as supply chain attacks. A common cyber-attack pattern is to damage an organisation by targeting less-secure elements in the supply chain network.

‘Allow-listing’ provides guardrails to the team to only allow access to pre-reviewed and pre-approved external sources. Developers can only consume resources from these sources, rather than trying to maintain lists of known bad sites that your teams shouldn’t use.

What needs protecting?

You need to protect the building blocks of your application. Let’s look at some target areas:

Source Code Repositories

Commonly used online code repositories such as Github, BitBucket and GitLab provide an efficient way to manage source code. The site hosts public and private folders, which mirror folders on a developer’s computer, through which they share code with collaborators. This approach is particularly appealing to distributed enterprise DevSecOps teams; however, there are literally dozens of examples where source code has been stolen, and not just from small companies. Login credentials and cutting-edge intellectual property were retrieved in 2016 when hackers penetrated Uber’s source code repository on GitHub. They also came across AWS credentials that yielded personal data of about 7 million Uber drivers and 50 million customers.

Samsung left dozens of critical code repositories on GitLab marked as public, exposing code and secret keys through the likes of the SmartThings app. One repository contained credentials that allowed access to an entire AWS account that held several employees’ private GitLab tokens stored in plaintext as well as S3 buckets containing log and analytics data.

Artefact and Image Repositories

Container/image repositories, like DockerHub, store the artefacts that you’ve built so that they can be easily reused and deployed. This ensures that the artefacts you’re using can be trusted, and can be re-used many times to help speed up secure coding practices. It’s not widely known, but container vulnerabilities can be a major hidden threat in what developers might otherwise assume is a secure platform – with exploits and issues lying in wait to become an issue after code is deployed into production.

The following articles from Aqua Security, provide a deep insight into container vulnerabilities that have been seen in the wild:

How to do it … easily

We’ve found the following practices are proven to be effective and worthwhile of consideration for mandating as a standard across your repositories and their access:

Access Control

Use private Git and Image repositories: Unless you have a specific requirement such as collaborating with the open source community, configure your repositories to be private to protect your source code and IP.
Role Based Access Control (RBAC): Many of popular tools and services used today such as Jenkins, CircleCI, GitLab, etc provide some form of RBAC. Wherever possible, configure appropriate roles that enable the people and teams to use these tools, and prevent those without need from using them.
Single Sign On (SSO): SSO adoption is widespread. Integrating your source code repository and your artefact and image repositories with providers such as Microsoft, Google, Okta, or others is well documented and easy to get up and running. Security and compliance benefits greatly from implementing SSO.
Implement requirements for MFA. Do not use SMS as a form of MFA.
Revoke access of staff no longer working on the project and have strict offboarding guidelines that are easy to follow. Usually off-boarding processes help with this, but where you have a dynamic and often-changing team, some automation around this access will really help.
Periodically rotate SSH keys and deployment keys / tokens. This is a great belt-and-braces example that ensures someone with an old key can’t creep into the system later.

Supply Chain and Testing

Build a private, cached library of modules that has been security tested and approved
Whitelist approved sources (i.e. your own npm or Docker registry) at the stages of that component’s lifecycle. For example, configuring systems where applications run and restricting where Developers can consume resources from.
For untrusted sources including public docker image repositories, we strongly recommend performing security testing and scanning. You may wish to do this in a sandbox environment.

Secret Sauce

Never store sensitive data or information in your repositories, either as code or configuration. Tools such as git-secrets may be useful to prevent secrets ever getting to your repository, or scripting similar to execute as a Git hook. Scanning and auditing can be achieved using tools such as TruffleHog or GitRob. If you are facing a complex issue, a secrets management solution such as HashiCorp Vault may be useful.
Monitor repositories for suspicious activity. Sudden spikes in source code check-ins or image commits, unusually large amount of source code checkouts, successful logins from unusual locations may all indicate compromise.
Don’t leave a .git folder on a publicly accessible server!

There’s a fair bit to consider when setting up your code, container and artefact repositories and libraries to help ensure a secure outcome – but we reckon they’re a better way to fly and to have to re-check every component before deployment each and every time, which imposes a cost on deployment that can slow down your release rate.