How IPFS Splits Files: Fixed vs. Content-Defined Chunking

Before a file becomes a Merkle DAG, it has to get cut into pieces, and how IPFS chooses where to cut matters more than it looks. IPFS's default chunker slices every file into fixed 256 KiB blocks. It also ships two content-defined alternatives, Rabin fingerprinting and Buzhash, that pick cut points based on the data itself instead of a fixed byte count. The difference sounds academic until you edit a file and re-add it, at which point one approach keeps most of the original blocks and the other regenerates almost all of them.

TL;DR:

• IPFS's default chunker is fixed-size: every block is 262,144 bytes (256 KiB) except the last one, via the DefaultSplitter/SizeSplitter in the boxo/chunker package
• Fixed-size chunking has a known weakness, the byte-shifting problem, where inserting or deleting even one byte near the start of a file shifts every chunk boundary after it, changing almost every block's hash
• IPFS also ships two content-defined chunkers, Rabin fingerprinting and Buzhash, which pick cut points based on the data's own content, so edits only affect the chunks near the edit, not everything downstream of it
• Content-defined chunking trades a small amount of chunk-size predictability (Rabin's actual output only stays within a min/max range, not an exact target) for much better deduplication when files change and get re-added
• Neither choice is visible in AIOZ Pin's SDK; pinFilesToIPFS() uses whatever chunker the underlying IPFS layer defaults to, which is fixed-size unless configured otherwise

The Default: Fixed-Size Chunking

IPFS's default splitter is about as simple as chunking gets: read 262,144 bytes (256 KiB), call that one block, repeat until the file runs out, with the final block being whatever is left over. This is the DefaultSplitter (built on NewSizeSplitter) in IPFS's chunker package, and it is what runs unless you explicitly ask for something else. Fixed-size chunking is fast, predictable, every block except the last is exactly the same size, and simple to reason about.

The Problem It Creates: Byte Shifting

The weakness shows up the moment a file changes rather than getting uploaded fresh. Insert a single byte near the beginning of a large file and re-chunk it with a fixed-size splitter: every chunk boundary after that insertion point shifts by one byte, which means every block after it now contains different bytes than before, which means every one of those blocks hashes differently. A one-byte edit to a 10 MB file can end up looking, block-by-block, like an almost entirely different file, even though 99.99% of the actual content is unchanged. This is the "byte-shifting problem," and it directly undermines deduplication: IPFS can only skip re-storing a block if that exact block already exists somewhere, and fixed-size chunking after an edit produces mostly-new blocks even when the underlying change was tiny.

The Alternative: Content-Defined Chunking

IPFS's chunker package also ships two content-defined chunkers: Rabin fingerprinting and Buzhash. Both work on the same underlying idea: instead of counting a fixed number of bytes and cutting, they run a rolling hash over the data and cut whenever that hash matches a pattern, a boundary condition determined by the content itself rather than a byte counter. Because the cut points are anchored to the data, inserting a byte near the start of a file shifts the position of the next boundary but the content-derived boundaries themselves stay in the same relative places relative to unchanged data further into the file. Most blocks after the edit end up identical to before, and only the block or two immediately around the edit actually changes.

The Rabin implementation in IPFS's chunker takes a target average block size and derives a min and max from it (roughly a third of the average as the floor, one and a half times the average as the ceiling), rather than guaranteeing an exact size like the fixed-size splitter does. That is the real tradeoff: content-defined chunking gives up exact, predictable block sizes in exchange for chunk boundaries that survive small edits.

Why This Matters for Deduplication

Deduplication on IPFS happens purely at the block level: if two different files (or two versions of the same file) happen to produce an identical block, that block is only stored and transferred once, referenced by both. Fixed-size chunking gets this right for files that are genuinely identical byte-for-byte, and gets it almost entirely wrong for files that are mostly the same but shifted by an insertion or deletion somewhere in the middle. Content-defined chunking is specifically the fix for that second case, at the cost of slightly more complex, slightly slower chunking logic, since computing a rolling hash over every byte costs more than counting to 262,144.

What This Means on AIOZ Pin

AIOZ Pin's SDK methods (pinFilesToIPFS(), pinFolderToIPFS()) do not expose a chunker choice; they run on whatever the underlying IPFS layer defaults to, fixed-size, 256 KiB blocks. In practice this means: pinning the exact same file twice deduplicates cleanly (identical bytes produce identical blocks either way), but pinning a lightly-edited version of a file you already pinned, a corrected typo in a document, a metadata tweak, a version bump, will likely produce a mostly new set of blocks rather than reusing most of the original ones. If you are building something where minimizing storage across many versions of similar files actually matters, that is a real, practical consequence of the default chunker, not a hypothetical one.

Frequently Asked Questions

What is IPFS's default chunk size? 262,144 bytes (256 KiB) per block, except the final block of a file, which is whatever is left over. This comes from the DefaultSplitter in IPFS's chunker package.

What is the byte-shifting problem? When fixed-size chunking hits a file that has been edited (a byte inserted or removed), every chunk boundary after the edit point shifts, causing most subsequent blocks to hash differently from before, even though most of the actual content is unchanged.

What is content-defined chunking? An alternative chunking method where cut points are determined by a rolling hash over the data itself rather than a fixed byte count, so most blocks stay identical across small edits to a file instead of shifting entirely.

Does IPFS support content-defined chunking? Yes. IPFS's chunker package includes both Rabin fingerprinting and Buzhash as content-defined alternatives to the default fixed-size splitter.

Which chunker does AIOZ Pin use? AIOZ Pin's SDK does not expose a chunker option; pinning calls use the underlying IPFS layer's default, which is fixed-size, 256 KiB blocks.

Does chunking choice affect a file's CID? Yes, indirectly. Different chunk boundaries produce different blocks, which produce a different Merkle DAG structure, which produces a different root hash, so the same file content chunked two different ways can resolve to two different CIDs.

Related Reading

• How IPFS Really Works: Merkle DAGs, Chunking, and Routing - where chunking fits into the overall path from file to CID
• Anatomy of a CID: Decoding an IPFS Identifier - what the hash of the resulting root block actually encodes
• How AIOZ Pin Actually Keeps Your Files Online: Replication, CIDs, and Proof-of-Storage - how these blocks get replicated once chunked

References

• boxo/chunker package documentation
• go-ipfs-chunker Rabin implementation (archived, superseded by boxo/chunker)