Remote attestations are useless without reproducible builds

Turnkey spends a lot of time thinking about software builds, and has invested a lot of resources to make them reproducible. When I talk about this externally the first reaction I get is a variation of “why bother?”. Indeed: do you really need your software to yield byte-for-byte identical artifacts? My hope is to convince you that the answer is a resounding YES if you’re planning to use remote attestations.

Brief introduction to reproducible builds

Humans write code in text format. This is generally referred to as “source code”. For compiled languages, source code is processed by a compiler to produce binary artifacts. These artifacts are the actual executables, run by computers.

In the Rust ecosystem, cargo build is the tool to compile a crate or set of crates into binary artifacts. In the Golang ecosystem, go build does this. For C code, gcc. And so on.

A build process is reproducible if given the same source code, it produces identical binary artifacts, byte-for-byte. Given the same source code:

• Building it twice in a row should yield identical artifacts (time independence)
• Building it on the same machine with different users or operating systems should yield identical artifacts (operating system/software independence)
• Building it on two completely different computers should yield identical artifacts (hardware/architecture independence)¹

This problem is a lot harder than it looks: compilers produce binaries which depend on system version, system libraries version, system username, hostname, name of the folder you are building in, filesystem, system time, timezone, language, number of cores, core speed, linux kernel version, container runtime version, CPU brand/model and more.

Trust the source != trust the binaries

Good security practices rely heavily on policing and enforcing source code integrity. The problem? Computers don’t run source code. They run binary artifacts, which humans can’t read or reason about.

The process to go from source code to binary artifacts is generally opaque, run by a single machine or set of machines responsible for “the build”. This could be in the form of Github CI runners which automatically publish docker images, or self-hosted clusters which populate an internal store with tagged binaries. This build process is a major weak point. It’s an inherent single point of failure. You can independently verify and audit the inputs (source code) but can’t trust the outputs (binary artifacts) unless other humans can verify them.

So how do we solve this problem? You guessed it already: reproducible builds! Reproducible builds mean verifiable builds. If multiple machines or humans can independently attest to the fact that a set of source files yields the same artifact, the single point of failure is gone.

Remote attestations and reproducible builds

Remote attestations are touted as a solution to “run any program verifiably”. The reality is: yes, as long as you have a verifiable (aka reproducible) build!

To be concrete, let’s take the example of AWS Nitro Attestations. An attestation document is, at its core, “just” a signed message. The signature is meaningful because it comes from a public key associated with a particular Nitro enclave. And the message contains valuable information about what is running inside of that enclave. The important fields contained in each attestation are:

• PCR0²: Enclave Image File (EIF) hash. EIF files are what the enclave uses to boot.
• PCR1: linux kernel and initial RAM data hash (aka initramfs)
• PCR2: hash of user applications, without the boot ramfs. For Turnkey enclaves, PCR2 == PCR1 because we do not use pivot_root, and simply use initramfs as our final filesystem.
• PCR3: hash of the IAM role assigned to the parent EC2 instance. This lets you verify the AWS account and region an enclave is running in.
• certificate: contains an X.509 certificate specific to the enclave (with the enclave public key)
• cabundle: certificate chain to certify the “certificate” above. This ties the enclave public key to a top-level AWS certificate³.

These fields are encoded as a CBOR struct and signed. This provides the basis for remote attestation: as an external user, I can request an attestation, verify it, and gain confidence that the enclave is running a certain EIF file, in a certain AWS account/region, without trusting the developer who provisioned this enclave (as long as I trust AWS is honest⁴).

Note that AWS attestations provide a proof (among other things) of the hash of the EIF file. Which means you need to trust whoever or whatever builds the EIF file from the source code you provided. And thus: if you can’t reproducibly build the EIF file, AWS Nitro attestations are useless⁵.

A peek at Turnkey builds

This will be a full post at some point in the future but I can’t let you walk away without sketching how Turnkey builds software in practice. EDIT (2024-10-31): the full post is now available at Reproducible builds made easy: introducing StageX.

We’ve thought about this from first principles: today our enclave builds are powered by Stageˣ (“stagex”)⁶, a new distro focused on immutable, reproducible packages distributed in the form of Docker OCI images.

Stagex builds all of its packages from scratch, rooted in 387 bytes of assembly for the “stage 0” compiler. Bootstrap compilers (stage 1, stage 2, stage 3)⁷ and all other importable packages (“stage x”, hence the name) are built reproducibly, and signed by multiple parties.

As a concrete example, take a look at linux-nitro: this is the kernel we use inside of our enclave environment. It is maintained in Stagex because Amazon itself does not provide signed, reproducible, source-bootstrapped or even recent kernels for Nitro enclaves. Thanks to Stagex the kernel we use in Turnkey enclaves is built reproducibly and signed by multiple parties, all the way down to these few hundred bytes of assembly (see stage0).

We use Stagex to power EIF builds for all Turnkey enclaves. EIF files are byte-for-byte identical, every time, regardless of who builds them, regardless of which computer in the world kicks off “the build”. And that’s no small feat: the AWS developer SDK won’t do this for you!

Because we can verify EIF builds, we’re able to use remote attestations meaningfully: PCR0 values can be verified and mapped to source code, and the source code can be independently reviewed and audited by other humans, both inside and outside of Turnkey.

Conclusion

We’ve seen why reproducible builds are needed to use remote attestations: attestations only attest to the actual binaries (or EIF files) run in secure enclaves. Humans can’t review binaries directly, but they can review source code. The connection between source code and binaries is the build process. Without a reproducible build it’s not possible to verify which source code produced a given binary, hence it’s not possible to verify what is running inside of a secure enclave independently.

Next time you hear about remote attestations used in the wild, ask the hard question: “and how do you build and deploy your binaries exactly?”. If the answer doesn’t ring true, you’re within your rights to respond “so why bother?”