The proliferation of Pipeline tools and plugins: A backdoor for Supply Chain Attackers

Codecov hackers gained access to Monday.com source code

Monday.com has recently disclosed that it was impacted by the Codecov supply-chain attack according to BleepingComputer. After their investigation into the Codecov breach, monday.com found that unauthorized actors had gained access to a read-only copy of their source code.

Monday.com is the latest victim of this attack which has already affected hundreds of companies including HashiCorp, Confluent, Twilio and Rapid7, and others.

The breach allowed a malicious third-party to alter a version of the bash uploader script to potentially export information subject to continuous integration (CI) to a third-party server, which according to Codecov, could potentially affect:

• Any credentials, tokens, or keys passing through their CI runner that would be accessible when the Bash Uploader script was executed.
• Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
• The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.

DevOps Tools Proliferation – A whole new world of vulnerabilities

One of the main contributors to the weak security posture of development environments is the complexity and knowledge gap created by the number of tools and services taking part in this process. With more than a hundred CI/CD tools to choose from and hundreds of plugins and services connected to those tools, no wonder security teams are having a hard time grasping the amount of information and security requirements of these environments.

It is not rare to see a CI/CD pipeline which is built with 10 to 20 different tools and services, some are cloud services, some open-source tools, and a variety of plugins. It is impossible to manually keep track of this complexity, which might result in an exposure of your environment, code, secrets, and network through those tools and plugins’ vulnerabilities.

The DevOps tools’ sprawl continues as more and more companies introduce their DevOps products and services. Development teams take advantage of these new CI/CD tools and services to build their pipelines and enhance the process but by that they also increase the exposure of their pipeline to risks. Add to it the limited collaboration between development and security teams; and the lack of visibility and control over these services and there is no surprise that CISOs and application security managers look puzzled when asked about their CI/CD pipelines security.

Minimize risk and complexity, and avoid the pitfalls of Supply Chain Attacks

The recent series of supply chain attacks affected tens of thousands of companies. Nowadays, CI/CD pipelines form the backbone of modern-day DevOps operations and as we see this trend continues, we cannot ignore the urgency in protecting customer’s development environments from these pervasive attacks.

The complexity and collaborative nature of these environments provide an easy target for attackers, who can take advantage of vulnerabilities and misconfigurations within pipeline plugins and services. By gaining access to the CI/CD pipelines attackers can hijack your updates, inject malicious code and get a backdoor to your and your customers’ environments.

The latest Codecov and SolarWinds attacks taught us two alarming facts:

1. Attackers can gain easy access to your most valuable process and data through your pipeline’s many services and plugins which are usually not monitored at all.
2. Those attacks can go unnoticed for months, impact thousands of companies and inflict massive damage.

Organizations must take proactive action to secure their software supply chain from such attacks and prevent attackers from using these backdoors to their environment. This requires taking into account the complexity of the development environments, the various 3^rd party plugins, and services connected to it, and the sophisticated nature of today’s supply chain attacks.

Building a strong CI/CD Pipeline security posture

Security and DevOps teams need to watch their pipeline dependencies closely to identify and respond to vulnerabilities and attacks against those addons services and tools.

Whenever a new service is connected to your pipeline, these services need to be checked and monitored constantly for any vulnerability or suspicious activity. Any suspicion should automatically trigger an alert to the appropriate stakeholders that need to verify the integrity of the service and ensure there is no risk associated with it.

How Argon can help prevent Supply Chain attacks?

The way Argon detect and prevent supply chain attacks like the one that happened to Codecov is through a multi-layered security approach:

• Visibility: Argon is a CI native solution, which means it is integrated into the CI pipeline, it understands the set of instructions composing it and is able to map out all external dependencies. In real-time you get an overview of all CI pipelines in the organization, including every step that is external or that accesses resources from outside of your environment.
• Security: On exiting pipelines, Argon applies a set of security policies and DevOps best practices; which include pipeline analysis abilities, which immediately alerts on dangerous instructions, like the one the Codecove hacker added, to trigger printing and sharing of all environment variables to a remote URL. These sets of security and DevOps policies also apply to the pipeline tools and to external dependencies connected to it.
• Integrity: The final safety mechanism is based on validating the integrity of external dependencies in the pipeline (like the Codecov step, that was compromised). Argon would automatically perform a checksum validation on every release, verifying the authenticity of the used resource, and on a hash mismatch – would alert in real-time or even actively prevent the potentially compromised release.

Eran Orzel, Argon’s Chief Revenue Officer

Topics: Supply Chain Attack, Codecov breach, SolarWinds Sunburst attack, DevOps Pipeline Breach, DevOps, Codecov, DevSecOps, CI/CD Pipeline Security, CICD security best practices, Codecov leak