Rethinking S3 as a backup target: streaming qcow2 at near-wire speed
Instead of forcing S3 to behave like a filesystem, Trilio restructured the pipeline to emit a stream of object-shaped writes from the very first byte. The result is roughly 4× faster backups with no cache tier to babysit.
The premise
S3 is the modern default for backup storage
Object storage — S3 and its API-compatible siblings — has become the default choice for backup storage across cloud and on-prem environments. Nearly nine in ten Trilio customers now target S3-compatible endpoints (AWS, Wasabi, MinIO, Ceph RGW, cloud-provider variants) as their primary backup destination.
The reasons are familiar: essentially unlimited scale, eleven-nines durability, geographic replication, low cost per gigabyte, and a single API contract that decouples the backup software from the underlying storage.
The mismatch
But S3 is a strange fit for backup workflows
S3 speaks a different language than backup software has traditionally spoken. Objects are immutable-once-written blobs: no random-access byte-level API, no POSIX file semantics, no in-place mutation. A product built around qcow2 files, VMDK images, or filesystem streams doesn't naturally map onto “PUT this whole thing, then GET it back later.”
The industry's answer has generally been to bolt on a translation layer — and each one trades off in a different direction without removing the underlying architectural mismatch.
FUSE mounts
s3fs, goofys, or an in-house s3-fuse expose a bucket as a POSIX filesystem. Convenient — but every operation becomes an HTTP round-trip, and qcow2's L1/L2 walks, refcount updates, and header patches generate an enormous number of small ones. Latency compounds.
S3 gateways
MinIO Gateway, JuiceFS, and vendor products cache a filesystem in front of S3. Faster for hot data, but they add a stateful cache tier to size, monitor, and keep consistent — hiding the impedance mismatch behind more infrastructure rather than removing it.
Filesystem-semantic stores
Object stores that offer filesystem semantics work well where available, but they lock you to a specific vendor and often carry a significant price premium.
The approach
Stream native qcow2 straight to S3
Trilio is retiring the FUSE-based path in favor of a native streaming qcow2 subsystem — a Rust implementation that speaks S3 directly, with no filesystem intermediary. The idea is straightforward once you take it seriously: rather than make S3 look like a filesystem, restructure the pipeline so it produces a stream of S3-shaped writes from the start.
- Emit the qcow2 header, L1 table, L2 tables, and data clusters as a linear byte stream.
- Chunk that stream into 8 MiB pieces — the same size whether or not deduplication is enabled.
- Upload chunks in parallel through the AWS SDK with a bounded concurrency window.
- Emit a small manifest at the end describing chunk layout — enough for restore to reconstruct the qcow2 view.
No filesystem in the path, no cache tier to manage, no chatty round-trip pattern. Throughput is bounded by network bandwidth and endpoint concurrency — not by the FUSE layer's overhead.
The numbers
Performance in practice
We benchmarked the streaming subsystem against the FUSE path across multiple customer environments and endpoints. The reference case: a 100 GB VM-disk backup on OpenShift-on-GCE against an S3-compatible endpoint.
- The 4× speedup is consistent across workload types. Dense random data or a mostly-empty freshly-provisioned disk, the streaming path finishes roughly 4× sooner. On sparse disks the upload phase alone accelerates by more than two orders of magnitude — the writer detects and skips zero clusters natively instead of round-tripping them through a FUSE layer that materializes them.
- Restore throughput lands within a few percent of backup at matched concurrency. An earlier 70% gap turned out to be a reader-concurrency artifact, not a PUT/GET asymmetry. Matched configuration, matched throughput.
- Memory footprint is bounded independent of restore size. A 1 TB restore runs in roughly the same ~400 MiB resident envelope as a 1 GiB restore, because the pipeline streams rather than materializing intermediate state.
We cite relative numbers deliberately. Absolute throughput depends heavily on the endpoint's own characteristics — AWS us-east-1 differs from Wasabi differs from on-prem Ceph — and on the network path. The relative improvement over FUSE has held across every endpoint we've measured.
Beyond speed
What the stream makes possible
The streaming subsystem isn't just faster — it's the substrate for backup capabilities that were awkward or impossible under the FUSE-based architecture.
Native qcow2, plain and encrypted
The subsystem produces true qcow2 files on S3 — not raw dumps, not custom formats. Any tool that reads qcow2 can consume a Trilio backup directly with the appropriate access.
Encryption is built in via LUKS: source-side decrypt for OpenStack Cinder LUKS volumes, target-side encrypt for backup-at-rest, or both directions with independent keys under a BYOK trust model. Encrypted backups are byte-identical on restore, and the passphrase never leaves the backup-admin boundary.
Incremental backups via hash-file comparison
Traditional incrementals depend on hypervisor-side change tracking (CBT / dirty bitmaps), which needs platform cooperation and doesn't always survive migrations or restarts. Instead, alongside each backup we store a hash file — a SHA per 1 MiB chunk. The next backup recomputes hashes, compares against the previous file, and emits only changed chunks as an overlay.
It works on any source, survives any source-side disruption because the hash file lives on S3, and composes cleanly with the backup-chain model.
In keeping with the spirit of S3 — objects are written once and never modified in place — forever-incremental chains and synthetic full backups are being deprecated. Building a synthetic full means rewriting existing backups to fold a chain into a new baseline, and that in-place mutation is exactly what S3's write-once model is built to avoid. Update your schedule to take a real full backup after each retention window — a fresh object written from source, never one synthesized by editing backups already on the store.
Global deduplication
Under the streaming architecture, dedup is a natural fit rather than a bolt-on. Each 8 MiB chunk is SHA-256 hashed and stored at a content-addressed S3 key. Identical chunks across backups — the same workload over time, a shared base image across VMs, common OS blocks across a fleet — dedup automatically. Object Lock retention doubles as garbage-collection-by-expiry, so there's no separate dedup GC job to run and audit.
For encrypted backups, dedup is per-key: backups sharing a passphrase dedup against each other; those under different passphrases don't. Tenant isolation is preserved while still delivering meaningful compression within a single backup plan.
The economics
What this means for cost
Faster backups and native S3 semantics aren't only performance wins — they translate directly into lower infrastructure cost across three axes.
100 GB sparse → 13 MB
S3 storage
Sparse workloads store almost nothing — a 100 GB freshly-provisioned disk lands as 13 MB of real data on S3, versus a FUSE path that materializes every zero byte. Content-addressed dedup compounds it across a chain: unchanged clusters are stored once, not repeatedly. And because incrementals upload only changed chunks, the marginal cost of an extra restore point tracks the workload's change rate rather than its total size. On tiered S3 pricing, the smaller footprint also lets you tier more aggressively without hurting restore SLA.
data-mover compute
Data-mover compute
Backup wall-clock is roughly a quarter of the FUSE path on identical workloads. Data-mover pods bill for time-in-flight — Kubernetes pod CPU-seconds, or the equivalent on OpenStack — so a 4× speedup is close to a 4× cut in per-backup compute cost. The bounded ~400 MiB memory envelope also lets pods be sized predictably rather than over-provisioned for tail-latency scenarios, packing more concurrent backups per node.
· staging PVCs to run
Operational infrastructure
The streaming subsystem removes the infrastructure the FUSE path needed to work at scale: no per-node cache tier to size and monitor, no S3-gateway servers in front of the endpoint, no separate staging PVCs for backup overlays. Dropping the staging PVC alone can free terabytes of Filesystem-mode block storage — often the priciest storage class in a Kubernetes deployment. And a shorter backup window collides less with production traffic, cutting the network and storage headroom you'd otherwise over-provision to absorb it.
The combined effect matters at fleet scale. A smaller S3 footprint, shorter data-mover runtimes, and eliminated staging infrastructure add up to a materially different backup TCO than a FUSE deployment — and the gap widens with fleet size.
The landscape
How this compares to established enterprise backup
Two names come up in nearly every backup evaluation: Veeam and Commvault. Both are mature products with strong footprints in traditional VM backup (VMware, Hyper-V) that have extended into cloud-native workloads — Veeam via Kasten K10, Commvault via Commvault Cloud — and both support S3 as a target. The architectural differences worth understanding:
Where the established vendors excel: mature UIs, deep vSphere/Hyper-V integration, decades of enterprise support relationships, and broad protocol coverage for legacy workloads. For a customer whose primary environment is traditional VMware, they remain strong choices.
Where we're differentiated: cloud-native architecture, an open format, a streaming S3 write path, and the substantially lower operational cost of removing the backup-proxy tier entirely. For a customer on KubeVirt, OpenShift Virtualization, or OpenStack — where the workload is already Linux-native and the operator model already Kubernetes-native — those differences compound.
Put concretely: if your infrastructure is Kubernetes-native and your backup target is S3, the architectural fit is far closer for a subsystem designed for both from day one than for one that reached this territory by extension.
Sovereign cloud
Backups that answer to your jurisdiction, not to a vendor
Across the EU, the Middle East, India, and beyond, sovereign-cloud initiatives share a common thread: critical data must remain under local jurisdiction and legal control, and the systems holding it must be built on open standards and open source — so that access to your own data never depends on a foreign vendor's tooling, licensing, or continued goodwill.
Backup is where this requirement bites hardest. A backup is the copy of record for everything an organization runs. If it's written in a proprietary format, the data is only nominally yours: reading it requires the vendor's software, under the vendor's license, for as long as the vendor exists and chooses to support you. The streaming subsystem is built so that this dependency never forms.
qcow2 — publicly documented, community-maintained
Backups land as native qcow2, the open disk-image format specified and maintained by the QEMU project. The specification is public; the reference tooling is open source and packaged in every major Linux distribution. No proprietary reader stands between you and your data — a sovereignty guarantee no closed format (VBK, CVBK, or otherwise) can offer.
Any S3-compatible store, in any region — including yours
The write path speaks the S3 API, not a vendor cloud. Point it at a sovereign hyperscaler region, a national cloud provider, or fully self-hosted object storage — MinIO or Ceph RGW in your own datacenter — and backup data never crosses a border you didn't choose. Encryption keys stay inside your trust boundary under the BYOK model.
Reversibility by design
Sovereign-cloud frameworks increasingly demand demonstrable exit: proof you can leave a provider and keep operating. Because backups are standard qcow2 on standard S3, restore works with open-source qemu tools alone — no license check, no vendor in the loop. You can walk away with your backups and prove it in an audit.
Sovereignty isn't a compliance checkbox bolted onto a proprietary core — it falls out of the architecture. Open format, open tooling, storage you choose, keys you hold.
Rollout
Production-validated today, phased across the portfolio
The subsystem is production-validated at scale now. Rollout across Trilio's product line is phased — each stage builds on the same Rust codebase, changing only the integration surface with the surrounding orchestration.
Streaming qcow2 for KubeVirt / OpenShift Virtualization GA · Q3 2026
The initial GA target. Full backups, incrementals via hash-file, plain and encrypted variants. Replaces the FUSE-based backup path for T4K workloads.
VM instant restore for OCP-V / KubeVirt GA · Q4 2026
An nbdkit-based read plugin serves backup content as a live NBD endpoint, letting a restored VM boot immediately and stream its disk in the background. RTO drops from hours to seconds — customers can be back in service before a traditional restore has finished downloading.
Full support for OpenStack GA · Q4 2026
Extends the same subsystem to T4O — Cinder volumes, Nova boot disks, encrypted volumes, and instant-restore — closing out the FUSE retirement across the entire product portfolio.
Closing thought
FUSE-mounting object storage has served the industry as an expedient bridge — but it's a bridge that costs performance, reliability, and operational complexity on every single backup.
Making S3 a first-class backup target means restructuring the pipeline to talk S3 natively: treating “object storage with immutable blobs” as a first principle rather than a constraint to work around. The streaming qcow2 subsystem is the shape of that restructure — faster by 4×, cheaper to run, and sovereign by construction: open format, open tooling, and storage under your jurisdiction. The numbers and the architecture both say the trade is worth making.