Chapter 2: SpanFS Internals, Distributed Storage, and Cluster Mechanics

SpanFS stores user data as chunks, not files. The IO Engine runs a Rabin rolling hash across the byte stream and slices it at content-defined boundaries. Each resulting chunk is fingerprinted with SHA-1 and looked up in the cluster-wide deduplication hash table.

Three on-disk constructs hold these chunks together:

• Chunk file — smallest unit of user data persisted to disk after dedup and compression.
• Blob file — container that aggregates many chunk files belonging to a logical object.
• Chunk group — the resiliency unit that is RF-replicated or EC-striped as a whole.

The relationship is captured in SnapTree, a distributed B+ tree atop the cluster-wide key-value store. SnapTree supports copy-on-write, so a snapshot or clone is just a new root pointer.

Figure 2.1: Chunk file, Blob file, and Chunk group hierarchy

NVRAM journaling and write coalescing

Every node includes an NVRAM region (battery- or flash-backed SSD) that survives power loss. The mechanism follows a journal-then-checkpoint pattern: append to NVRAM log, mirror to peer NVRAM (matching RF), ACK client, then a background destage coalesces small NVRAM entries into large sequential disk writes.

Animated Write Path: Client to Durable Disk

Read path and locality optimizations

Reads exploit locality. Cohesity prefers to ingest, dedup, and persist data on the same node that received it, so subsequent reads of recent data are local. Read cache (DRAM and SSD) is per-node and warmed by access patterns. For sequential restore workloads, SpanFS issues parallel reads to multiple nodes simultaneously since the chunk group's fragments span the cluster.

Garbage collection and chunk reclamation

Because chunks are deduplicated, deleting a backup or expiring a snapshot does not immediately free space. A chunk is reclaimable only when its reference count in the distributed KV store drops to zero. GC walks the SnapTree, decrements ref counts, compacts blob files, and re-runs erasure coding on cold blob files. GC is throttled, so capacity reclaim typically lags deletion events by 24-72 hours.

A SpanFS cluster keeps data safe through two complementary mechanisms: Replication Factor (RF) and Erasure Coding (EC). Both can coexist within a single cluster — even within a single View Box.

RF2 vs RF3 trade-offs

• RF2: tolerates 1 simultaneous failure, ~50% usable, 100% overhead. Fast block-level rebuild. Best for hot data.
• RF3: tolerates 2 simultaneous failures, ~33% usable, 200% overhead. Required for strict SLAs / small clusters.

Erasure Coding schemes

Headline: EC 4:2 and RF3 tolerate the same number of failures, but EC 4:2 uses half the overhead. The cost is encoding CPU and a more complex rebuild path.

Inline RF2 -> Post-process EC pipeline

Cohesity rarely writes EC-protected chunks directly on the hot ingest path. Instead:

1. Stage 1 (Inline RF2): Hot chunks land in NVRAM and SSD with RF2 protection. Mirroring is cheap, latency is low.
2. Stage 2 (Post-process EC): A background task identifies cold chunk groups, re-encodes under the View Box's EC policy (e.g., 4:2), persists the new EC stripe, and frees the RF2 copies.

Animated RF2 to EC Conversion

Figure 2.3: Erasure coding stripe placement decision flow

Cohesity's headline storage-efficiency multipliers come from global, variable-length deduplication combined with inline compression.

Variable-length sliding-window dedup

1. The IO Engine slides a Rabin rolling hash across the byte stream.
2. When the rolling hash matches a content-defined breakpoint pattern, a chunk boundary is declared.
3. Each resulting chunk gets a SHA-1 fingerprint.
4. The fingerprint is queried against the global hash table; if found, only a metadata reference is stored.

The advantage is insertion-resilience: a 1 KB header prepend produces one new chunk, not all new chunks like fixed-block schemes. This can be a 5-10x dedup ratio difference for incremental and synthetic-full workloads.

Global vs local dedup domains

The dedup hash table lives in the distributed KV store, but its scope is the View Box. Two View Boxes maintain separate hash tables. Architectural consequences:

• Per-tenant View Boxes give isolation but lose cross-tenant dedup.
• Group similar workloads (all VMware backups) in the same View Box to maximize dedup.
• Per-tenant encryption keys break dedup across tenants regardless of View Box settings — identical plaintext encrypts differently.

Inline vs post-process compression

• Inline compression (LZ4 / zstd-class): each unique chunk compressed before disk write. Lower disk usage on first write; small CPU cost on ingest.
• Post-process compression: chunks written uncompressed first (faster ACK), then compressed by background task. Better latency at the cost of transient capacity.

Planning ratios for sizing

A SpanFS cluster is a peer-to-peer system: every node runs the same services. Coordination uses explicit consensus, fault-domain awareness, and disciplined upgrade procedures.

Strict consistency via Paxos

SpanFS guarantees strict consistency for both data and metadata. Reads always observe the most recent durably committed write. With three KV replicas per record, a metadata write requires at least two acks. Two clients writing the same offset never see split-brain.

Fault domain levels

Quorum scenarios

• 3-node, 1 down: 2/3 online, quorum holds, rebuild proceeds.
• 3-node, 2 down: quorum lost, cluster goes read-only/offline.
• 6-node, 2 down: 4/6 online, EC 4:2 still tolerates the second failure.
• Network partition 3/3: neither side has majority — both go offline to prevent split-brain.

Animated Quorum State Machine

Rolling upgrades

SpanFS upgrades are rolling: one node at a time enters maintenance, drains leadership and VIPs, upgrades, reboots, and rejoins. During maintenance: quorum tightens, effective resiliency drops by one, in-flight client connections migrate via VIP failover. For RF2 clusters this means upgrade temporarily reduces fault tolerance to zero. RF3 or EC 4:2 is preferred for mission-critical systems so that maintenance plus an unplanned failure does not equal data loss.