6/13/2024
Cloud Unfiltered with Cole Kennedy - Attestation is the Key
Author: Cole Kennedy
Recently I was a guest on the Cloud Unfiltered podcast from Outshift by Cisco. I was interviewed by Mike Chenetz, Head of Product Marketing for Cisco’s Emerging Technologies and Incubation.
Mike and I discussed software supply chain security, attestations, the in-toto framework and more. In this blog post I summarize some of the topics that Mike and I covered.
Why SBOM Is Not Enough
A “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. A SBOM is a nested inventory, a list of ingredients that make up software components. The SBOM work has advanced since 2018 as a collaborative community effort, driven by National Telecommunications and Information Administration’s (NTIA) multistakeholder process.
Mike began the episode by asking me if SBOM is sufficient to ensure the security of software supply chains.
Short answer: SBOM is not sufficient.
Let me explain by using a wedding cake analogy. The set of ingredients are only one factor in the quality of the wedding cake. There are other factors, like where the ingredients were sourced from, how those ingredients were mixed, what temperature the oven was set to and how long the cake was in the oven.
We have no way of knowing how the cake was made unless we can observe and verify the process. The same applies to software. An SBOM simply gives you the list of ingredients, but it doesn’t tell you how the cake was made.
Mike used the analogy of a drug recall. Pharmaceutical companies know the “ingredients” of a drug, but when a recall is issued, they need to assess not just the things the drug is made of, but where they were sourced from and how those elements were made.
I compared this to the Tylenol poisonings from the 1980s when seven people died when the painkiller was laced with cyanide. The deaths were not caused by an ingredient, but by a poison (i.e., cyanide) that was inserted. It’s similar to the recent XZ Attack in which a bad actor attempted to insert “ingredients” out of band.
SBOM doesn't account for scenarios like this. What’s missing is the observability of the software development lifecycle.
Introducing the Pipeline Observer
In a traditional CI/CD system you have a pipeline orchestrator. The CNCF defined a new component called the pipeline observer. The observer is where we discover security issues. Without a layer of observability that it provides, we can't ascertain what happened during the build process. We simply have to trust that the orchestrator did what we expected it to do.
Sometimes trust is not sufficient.
For instance, in the XZ Attack, certain package managers pulled down the XZ tarball and trusted it without checking to ensure that everything in it came from that source repository. Imagine what can happen in a private repo in a corporate setting? That repo might be maintained by a contractor who’s subcontracted three levels deep to work on a project.
Now back to the pipeline observer...
We’ve collaborated with the Cloud Native Computing Foundation (CNCF) on a software factory reference architecture, as well as a software supply chain best practices white paper. The result of this collaboration informs our conclusions about the importance of the pipeline observer. When you have this component it solves many of the challenges of software supply chain security.
It observes the inputs, the outputs, the action and the environment in which all the software build steps took place. We codified this with The National Institute of Standards and Technology (NIST) into NIST SP 800-204D (“Implementation of DevSecOps for a Microservices-based Application with Service Mesh”).
NIST SP 800-204D specifies exactly what makes up a supply chain and what we need to do to secure it.
The in-toto Framework
Mike mentioned a need to standardize on what we call things, so we can communicate and talk the same language. I agree! I used DevOps as a good example. In the past, we used to email an artifact and ask a colleague to deploy it on a server. Today with DevOps, we define that process in YAML and Kubernetes manifests.
Those standards of interchange help improve communication between Dev and Ops. In DevSecOps, we also need a way to improve communication. That means standardized, trusted APIs. We've been spending a lot of time with the in-toto project from the CNCF - it's the foundational API of DevSecOps. It's the future of how we communicate security events within the software delivery lifecycle.
in-toto is an open metadata standard that you can implement in your software's supply chain toolchain.
The in-toto Attestation Framework (available at GitHub) “provides a specification for generating verifiable claims about any aspect of how a piece of software is produced. Consumers or users of software can then validate the origins of the software, and establish trust in its supply chain, using in-toto attestations.”
Let me return to the wedding cake analogy.
The people who observe the baking process might have their own language that we don't understand. Instead, if they wrote those steps in a schema, we can pass the schema along to customers and suppliers. And now we have a common language that we can verify all the artifacts on. In-toto is the API that allows us to verify software in a standardized way.
The Attestation Working Group within in-toto is defining a set of atoms of specific information (e.g., “What does it look like when I want to verify that this build took place on a piece of trusted hardware?”). The group is working to define these standards and specify how we pass information around.
What Is Attestation?
Attestation is hiring a trust observer to sit in the bakery as the wedding cake is being made. Every time the baker completes a step towards baking that cake, the observer writes down what happened, takes a picture of the room, signs it (i.e., for certification) and places it in an envelope. The observer repeats these steps each time the baker takes an action.
Now when the cake is delivered, you have external trust and don't have to trust the baker. You have the signed attestations that were completed. With software, you can ask the question, “What was the AWS metadata that was attached to this instance when we verified the timestamp of this build?”
Attestation can tell me that the build happened on AWS GovCloud on an instance that I own. I can say that my build happened on AWS GovCloud on an instance that I know I own. The machine is this Amazon Machine Image (AMI) and I know that it happened on this date.
We have a GitHub Action that makes it easy to create these attestations. We recently announced the Witness GitLab Component.
When Do You Need Attestation?
Mike asked me when organizations need to start considering tools for attestation. My answer? To look at what's running in production right now and see if you can answer the question:
"For a given artifact, can I trace things back to the development process and the security process that created it?”
If the answer is “no,” then consider using in-toto, or open-source tools from CNCF like Witness or Arhivista. If that doesn’t work for you, then open an issue with us and let us know. We can work with you to help achieve your software supply chain security goals.
Final note:
Attestation should not be tied to any specific technology. Witness is a Command Line Interface (CLI) tool that is multi-architecture. Even if you're compiling code on an IBM AS/400, you can capture attestations about that process with Witness. The attestation from the AS/400 and attestations from your microservices architecture all go into one central repository where you can correlate all of the supply chain data
Interview Recording
Here’s a video recording of the Cloud Unfiltered interview with Mike Chenetz: