Let me tell you about something that quietly broke my brain a while back.

You find an open-source app. You pull up its GitHub repo, scroll through the source code, read it top to bottom. It looks clean. No funny business. You trust it. So you download the binary (aka. The compiled and ready to run version) and run it.

Here’s the thing though: what you just ran is not what you just read. Not necessarily. The binary on your machine was built on someone else’s computer, in someone else’s environment, through a build pipeline you’ve never seen. The source code and the binary are two different things, and the gap between them is exactly where things can go wrong, whether by accident or on purpose.

This isn’t a hypothetical. In 1984, Ken Thompson, one of the people who literally created Unix, gave a Turing Award speech that made a lot of very smart people deeply uncomfortable. He described how he had modified a C compiler to inject a backdoor into every login program it compiled. Not in the source of the login program. Not in the source of the compiler. In the compiled binary of the compiler itself. You could read every line of code involved and find absolutely nothing wrong.

His conclusion: “You can’t trust code that you did not totally create yourself.”

He was making a philosophical point. But in 2026, we ship software through CI/CD pipelines, package registries, and automated build servers, none of which you control, and some of which have been compromised before (SolarWinds, anyone?). Thompson’s thought experiment aged into a genuine threat model.

Reproducible builds are the answer to that gap. This blog post is about what they are, and how you can actually see the problem with your own eyes.

The Gap Between Source and Binary

Before I get into solutions, let’s make the problem concrete. Because it turns out binaries can diverge from source code even when nobody is being malicious at all.

Build your project on Monday, build it again on Wednesday and you might get two different binaries. Same source code, same machine, different output. Why? A few boring but sneaky reasons:

• Timestamps. A lot of build tools bake the current date and time directly into the binary. “This was compiled on May 3rd at 14:32.” Helpful for debugging, catastrophic for reproducibility.
• Build paths. Debug information often includes the full path of every source file on the machine that compiled it. /home/omar/projects/myapp/src/main.c on my machine becomes /home/peter/projects/myapp/src/main.c on Peter’s. Same code, different binary.
• Compiler randomness. Some compilers introduce randomized memory layouts or ordering during optimization. Intentionally. For performance.
• Environment bleed. Your local settings, timezone, even the order files are read from disk. All of these can sneak into the output in ways that are really hard to track down.

Now add a motivated attacker. And I don’t mean a vague theoretical one. I mean the SolarWinds breach of 2020, which is basically Ken Thompson’s thought experiment made real, at enterprise scale.

WTF Actually Happened at SolarWinds

SolarWinds made a product called Orion, an IT monitoring platform used by thousands of companies and U.S. government agencies. Attackers got into SolarWind’s build server and deployed a piece of malware called SUNSPOT, whose entire purpose was to watch for the Orion build process and hijack it at exactly the right moment.

Here’s what actually happened. SUNSPOT monitored the running processes on the build machine, and the moment it detected MsBuild.exe starting an Orion build, it swapped out a single C# source file with a backdoored version, then restored the original file the moment the build finished. By the time any developer opened their editor, the file looked completely normal. Git showed nothing. Code review showed nothing.

Two details here are worth unpacking because they show how carefully this was engineered.

• The AES-encrypted blob. SUNSPOT needed to carry the backdoored source file with it somehow, but storing malicious code as plain text inside the malware would make it trivially detectable by any antivirus scanner doing a string search. So instead, the entire backdoored InventoryManager.cs was stored inside SUNSPOT in AES-encrypted form and locked in a box that only SUNSPOT knew how to open. At build time it decrypted the file, wrote it to disk, waited for the build to finish, then deleted it. Nothing suspicious was ever sitting around in plain sight.

• The MD5 check. Before swapping the file, SUNSPOT would calculate the MD5 hash of the original InventoryManager.cs and compare it against a value it had memorized. If a developer had refactored that file since the last build — meaning the code had changed and the hash no longer matched — SUNSPOT would skip the injection entirely rather than replace a file it didn’t recognize. Why? Because injecting into a modified file could break the build with a compile error, which would immediately raise suspicion. The attackers were being careful not to get caught by their own tool.

The actual backdoor that got injected into the source was a single try/catch block added to a legitimate method:

try {
    if (!OrionImprovementBusinessLayer.IsAlive) {
        Thread th = new Thread(OrionImprovementBusinessLayer.Initialize)
            { IsBackground = true };
        th.Start();
    }
 } catch (Exception) { }

Let’s break down what each part actually does.

• OrionImprovementBusinessLayer is the name of the backdoor class. The attackers named it to blend in with legitimate SolarWinds code as Orion already had a real feature called the “Orion Improvement Program” that collected usage telemetry. A class named OrionImprovementBusinessLayer fits right in with that naming convention. Nothing jumps out.

• The if (!OrionImprovementBusinessLayer.IsAlive) check just makes sure only one instance of the backdoor is running at a time. Standard defensive programming. Completely normal looking.

• The Thread block spins up a background thread that runs the backdoor code. IsBackground = true means the thread dies silently when the main Orion process closes. No orphaned processes, no suspicious cleanup, nothing to notice.

• And the catch (Exception) { } with an empty body is the really sneaky part. It means if the backdoor crashes, throws an error, or fails to start for any reason then nothing happens. No log entry. No error message. No crash. The legitimate Orion code keeps running completely normally. The backdoor either works silently or fails silently. Either way, nobody notices.

If you were code-reviewing this next to a thousand other lines of legitimate refactoring, you’d probably scroll right past. It looks like boilerplate as developers write background thread startup code that looks exactly like this all the time, but The resulting DLL, now carrying a backdoor called SUNBURST, was compiled by SolarWind real build server, signed with their real code-signing certificate, and shipped through their real update infrastructure. About 18,000 organizations downloaded and installed it.

Once running on a victim’s machine, SUNBURST was in no hurry. It waited 12 to 14 days before doing anything at all, long enough for any post-install sandbox scan to complete and flag the software as clean disguising its outbound traffic to look exactly like legitimate Orion telemetry data so network monitors would see nothing unusual.

Those same attackers had initially gained access to SolarWinds systems as far back as September 2019, spent months doing reconnaissance, and ran a test injection with no malicious payload just to confirm the technique worked before deploying the real thing in early 2020. From the first trojanized update shipping to customers to public disclosure was roughly nine months.

The source code was fine. The binary was not. And nobody could tell the difference. That’s the threat model.

So What Are Reproducible Builds, Exactly?

Here’s the definition, and it’s simpler than it sounds:

A build is reproducible if, given the same source code and the same build environment, every independent build produces bit-for-bit identical output. That’s it. If you compile it, and I compile it, and a CI server in a data center somewhere compiles it we all get the exact same binary. Same bytes. Same SHA-256 hash.

Why does this matter? Because now verification becomes social. Instead of “trust that SolarWinds build server wasn’t compromised,” you get “trust that SolarWinds, and Microsoft, and some random developer in Germany, and an automated rebuilder in Debian’s infrastructure all independently got the same result.” Getting all of those to agree requires compromising all of them simultaneously. That’s a very different, and much harder, attack.

This is actually what SolarWinds was missing. After the breach, part of their remediation was running three parallel independent builds with separate credentials and comparing the outputs. Bit-for-bit identical? Ship it. Different? Something is very wrong. That check, had it existed before December 2020, would have caught SUNSPOT immediately; because the build server running the implant would have produced a different binary than the two clean ones.

There are two ingredients that both need to be locked down for this to work:

• The source: pinned to an exact commit hash, not just a branch name
• The environment: the OS, compiler version, timezone, and every dependency, all identical across every machine doing the build

The first one is easy, but the second one is where most of the engineering effort lives.

How Do You Actually Make a Build Reproducible?

Let’s go through the real sources of non-determinism one by one and how the ecosystem has addressed each of them. Some fixes are one-liners. Some require rethinking how you build software entirely.

The Easy Win: Timestamps

The single most common cause of non-reproducible builds is timestamps. Compilers, archivers, documentation generators, etc.. A shocking number of tools call time() during a build and bake the result into the output. Two builds of identical source, one second apart, produce different binaries.

The fix the community landed on is an environment variable called SOURCE_DATE_EPOCH. Set it to any Unix timestamp and tools that respect it will use that fixed value instead of the current time:

# Pin all timestamps to a fixed point in time
 export SOURCE_DATE_EPOCH=1700000000

 # Now build your project — timestamps in the output will be fixed
 make

That’s it. One line. A huge chunk of build tools like GCC, Clang, pip, Maven, Go, Rust’s cargo, and many more now respect SOURCE_DATE_EPOCH out of the box. If you maintain a build tool that calls time() and you haven’t added support for this yet, please do.

The harder part is catching tools that don’t respect it. That’s where diffoscope comes in, and I’ll get to that in a moment.

The Subtler Problem: Build Paths

Even with timestamps fixed, your binary might still differ between machines because of paths. When you compile with debug information enabled, the compiler often records the full path to every source file:

# I build on my machine:
 gcc -g -o myapp main.c
 # Binary contains: /home/omar/projects/myapp/main.c

 # Peter builds on his machine:
 gcc -g -o myapp main.c
 # Binary contains: /home/peter/projects/myapp/main.c

Same source. Same compiler version. Different binary. The fix is a compiler flag called -ffile-prefix-map that rewrites paths at compile time.

The Harder Problem: The Environment Itself

Okay, so you’ve fixed timestamps and paths. You run two builds and get the same binary. But only as long as both builds used the exact same compiler version, the same version of every library, the same locale settings, and the same everything else.

In practice, this is really hard to guarantee across different machines or over time. For example I might use GCC 13.2, but Peter installed GCC 13.3 last week, and your CI runner migh have gotten an OS update last Tuesday and uses a completely different version. Any of these can silently change the output.

The real solution is making the environment itself reproducible, what the industry calls a hermetic build. The build shouldn’t reach out and grab things from the host system at all. Everything it needs from compilers, linkers, libraries, and tools should be explicitly declared and locked to exact versions.

The two most principled tools for this are Nix and Guix. The core idea is that every package is a pure function of its inputs. Instead of “install GCC globally on this machine,” you write a derivation that says exactly which GCC, built from exactly which source, with exactly which dependencies, all the way down. Every dependency is identified by a cryptographic hash of its inputs, not just a name and version number.

Here’s what a minimal reproducible C build looks like in Nix:

# default.nix
 { pkgs ? import <nixpkgs> {} }:

 pkgs.stdenv.mkDerivation {
   name = "myapp";
   src = ./.;                        # Use the current directory as source

   buildInputs = [ pkgs.gcc ];       # Exact GCC version pinned by nixpkgs

   buildPhase = ''
     export SOURCE_DATE_EPOCH=1700000000
     gcc -ffile-prefix-map=$src=. -o myapp main.c
   '';

   installPhase = ''
     mkdir -p $out/bin
     cp myapp $out/bin/
   '';
 }

Anyone running the same nixpkgs commit gets the exact same GCC, the exact same standard library, the exact same everything. Two developers on completely different Linux distributions — or even macOS — can run this and produce identical output.

Seeing It With Your Own Eyes: A diffoscope Walkthrough

Everything so far has been conceptual. Now i’m going to deliberately create a non-reproducible build, watch it happen, diagnose exactly why it’s happening using diffoscope, fix it, and verify the fix works.

To follow along you’ll need GCC and diffoscope installed. On Debian/Ubuntu:

sudo apt install gcc diffoscope

On Arch:

sudo pacman -S gcc diffoscope

Step 1: Create a Simple C Program

Nothing fancy. Save this as main.c:

#include <stdio.h>

 int main() {
     printf("Hello, reproducible world!\n");
     printf("Built: %s %s\n", __DATE__, __TIME__);
     return 0;
 }

The __DATE__ and __TIME__ macros are built into C and get replaced at compile time with the current date and time as string literals, baked directly into the binary. This is exactly the kind of thing real build systems do all the time, whether intentionally for version strings or accidentally through tooling that calls the equivalent of time() somewhere in the chain.

Step 2: Build It Twice and Watch the Hashes Differ

# First build
 gcc main.c -o build1

 # Wait a moment, then build again
 sleep 2 # will work without this
 gcc main.c -o build2

 # Compare the hashes
 sha256sum build1 build2

You’ll see something like this:

a3f82c1d...  build1
 e91b047a...  build2

Different hashes. Same source code. Same compiler. Two seconds apart. Run the binaries and you can see exactly what changed:

./build1
 # Hello, reproducible world!
 # Built: May  5 2026 13:17:52

 ./build2
 # Hello, reproducible world!
 # Built: May  5 2026 13:18:06

Fourteen seconds of wall clock time, permanently embedded in two different binaries.

Step 3: Ask diffoscope Why

This is where it gets satisfying. diffoscope doesn’t just tell you that the files differ, it drills down through the binary format and tells you exactly what changed and why:

diffoscope build1 build2

The output will look something like this:

--- build1
 +++ build2
 │┄ File has been modified after NT_GNU_BUILD_ID has been applied.
 ├── readelf --wide --notes {}
 │ @@ -1,12 +1,12 @@
 │ -  GNU  NT_GNU_BUILD_ID  Build ID: 1dfaa94da27ad9f5f1045c5bd0637b5a45cd9d41
 │ +  GNU  NT_GNU_BUILD_ID  Build ID: 27533c0d3df5021c03eb5975067e287864287dd3
 │
 ├── strings --all --bytes=8 {}
 │ @@ -4,15 +4,15 @@
 │  Hello, reproducible world!
 │ -13:17:52
 │ +13:18:06
 │  May  5 2026
 │
 ├── readelf --wide --decompress --hex-dump=.rodata {}
 │ -  0x00002020 333a3137 3a353200 4d617920 20352032 3:17:52.May  5 2
 │ +  0x00002020 333a3138 3a303600 4d617920 20352032 3:18:06.May  5 2

There’s a lot here but let’s read it top to bottom because every line is telling you something useful.

• The NT_GNU_BUILD_ID is a unique fingerprint GCC generates for each binary based on its contents. The two build IDs are completely different which confirms the binaries differ, and means debuggers and crash reporters will treat them as entirely separate builds.

• The strings section is the most human-readable part. You can see the two timestamps sitting right there as plain text: 13:17:52 in build1 and 13:18:06 in build2. Fourteen seconds difference, permanently embedded.

• The .rodata hex dump is showing you the exact bytes in the read-only data section of the binary where those strings live. The hex 333a3137 3a353200 decodes to the ASCII string 3:17:52. You’re looking directly at the timestamp sitting in the compiled binary.

This is what makes diffoscope so valuable. It doesn’t just say “these files are different.” It parses the binary format, finds every section that changed, and shows you exactly what byte changed and what it means. You’re not staring at a raw hex diff but reading a structured explanation.

This is the tool you reach for whenever you’re chasing down why two builds aren’t matching. It handles ELF binaries, Debian packages, tarballs, zip files, PDFs, and dozens of other formats always recursing into the structure rather than just diffing raw bytes.

Step 4: Fix It With SOURCE_DATE_EPOCH

# Pin the timestamp to a fixed value
 export SOURCE_DATE_EPOCH=1700000000

 # Build twice with the fixed timestamp
 gcc main.c -o build1
 sleep 2
 gcc main.c -o build2

 # Compare again
 sha256sum build1 build2

This time:

9a4c2b8f...  build1
 9a4c2b8f...  build2

Identical hashes. Run it again tomorrow and you will get the same hash. Run it on Peter’s machine and he will get the same hash, as long as he is also building from the same directory path.

Step 5: Fix the Path Problem Too

SOURCE_DATE_EPOCH fixes timestamps but paths are a different problem entirely.

# Make sure SOURCE_DATE_EPOCH is set so timestamps aren't the variable
 export SOURCE_DATE_EPOCH=1700000000

 # Build from your current directory
 gcc -g main.c -o build_here

 # Now build the same file from a subdirectory
 mkdir test && cp main.c test/ && cd test
 gcc -g main.c -o ../build_there
 cd ..

 # Compare
 sha256sum build_here build_there

Different hashes again, even though the timestamps are locked. Run diffoscope build_here build_there and you’ll see why:

│ -    <12>   DW_AT_name        : (line_strp) (offset: 0xb): main.c
 │ -    <16>   DW_AT_comp_dir    : (line_strp) (offset: 0): /home/omar
 │ +    <12>   DW_AT_name        : (line_strp) (offset: 0): main.c
 │ +    <16>   DW_AT_comp_dir    : (line_strp) (offset: 0x7): /home/omar/test

DW_AT_comp_dir is the DWARF debug attribute that records which directory the compiler was run from. It’s different because the binary was built from a different path, even though the source was identical.

Fix it with -ffile-prefix-map, which tells GCC to rewrite paths at compile time so the build directory never leaks into the binary:

export SOURCE_DATE_EPOCH=1700000000

 gcc -g -ffile-prefix-map=$(pwd)=. main.c -o build_here

 mkdir test && cp main.c test/ && cd test
 gcc -g -ffile-prefix-map=$(pwd)=. main.c -o ../build_there
 cd ..

 sha256sum build_here build_there

Who’s Actually Doing This in Production?

Reproducible builds can sound like a theoretical ideal, but that’s not true. Some of the most security-critical software in the world has made reproducibility a hard requirement, and for very concrete reasons.

• Debian has probably done more than any other project to push reproducible builds into mainstream software. They have a dedicated reproducible builds team that has been systematically working through the entire Debian package archive — tens of thousands of packages — and fixing whatever makes each one non-reproducible. You can see their progress live at reproduce.debian.net. At the time of writing they’re at roughly 98% of packages reproducible. They also built a lot of the foundational tooling the rest of the ecosystem now uses, including diffoscope.

• Tor Browser was one of the earliest serious adopters, and their motivation was about as clear as it gets: their users are journalists, activists, and dissidents in hostile environments. A compromised Tor Browser binary that quietly leakes IP addresses could get someone killed. Their threat model essentially demands that no single person or server be trusted to produce the release binary. Multiple developers independently build from the same source and compare hashes before anything ships. If the hashes don’t agree, nothing goes out.

• Bitcoin Core switched its release process to Guix-based reproducible builds in 2021. A backdoored Bitcoin binary that subtly mishandles private keys could silently drain wallets at scale. Any of Bitcoin Core’s hundreds of contributors can independently build the release and verify it matches what was published. Many of them do, every release.

• Arch Linux has an ongoing reproducible builds initiative and runs a service called rebuilderd. A continuous rebuilder that takes published Arch packages, rebuilds them independently from source, and publishes whether the result matched. Any package that suddenly stops being reproducible shows up as a mismatch and gets investigated.

Notice what these projects have in common: they all have a very specific, concrete reason to care. Tor Browser users face physical danger. Bitcoin Core users have real money on the line. Debian is the foundation that half the Linux ecosystem builds on. Reproducible builds tend to get adopted when the cost of a compromised binary is obvious and personal.

Back to Thompson

Remember where it all started. 1984. A compiler that injected a backdoor into every program it compiled, including future versions of itself, with no trace in any source file anywhere. Ken Thompson built it as a thought experiment to make a philosophical point about trust. Forty years later, SUNSPOT did exactly that, not as a thought experiment, but as a real operation, run by a nation-state intelligence agency, against a real software vendor, that reached 18,000 real organizations. The source code was clean. The binary was not. Nobody could tell the difference for nine months.

Reproducible builds don’t solve everything. They don’t protect you if your source repository is compromised. They don’t catch a malicious maintainer who submits a backdoor as a legitimate commit. They don’t eliminate trust entirely, they redistribute it. But that redistribution matters. The difference between “trust that this one build server wasn’t compromised” and “trust that this build server, and that one, and that independent rebuilder in Germany, and Debian’s infrastructure, and your own laptop all got the same result” is the difference between a single point of failure and a consensus. Compromising one is a targeted attack. Compromising all of them simultaneously is nearly impossible.

Thompson ended his 1984 speech with something that’s aged remarkably well: “You can’t trust code that you did not totally create yourself.”

He was right. But reproducible builds are as close as we’ve collectively gotten to a practical answer to that problem. A way to verify, rather than just trust, that what you’re running is what was written.

The tooling is there. diffoscope to diagnose. SOURCE_DATE_EPOCH to fix the easy stuff. Nix or Guix for the hard stuff. rebuilderd to keep watching. The projects doing it right have shown it’s achievable even at scale. The rest is just the industry deciding if it cares enough to do the work.