How IPFS Splits Files: Fixed vs. Content-Defined Chunking

IPFS chunking is the step that cuts a file into blocks before they become a Merkle DAG. By default, Kubo cuts fixed 256 KiB blocks. It also ships two content-defined chunkers, Rabin and Buzhash, which pick cut points from the data. That choice decides how many blocks survive an edit.

TL;DR:

• The default chunker is fixed size: every block is 262,144 bytes (256 KiB) except the last. In Kubo that's Import.UnixFSChunker = size-262144.
• Insert one byte near the start of a file, and fixed-size chunking shifts every later boundary. Nearly every block gets a new hash.
• Rabin and Buzhash cut where a rolling hash matches a pattern, so boundaries realign after an edit and most blocks deduplicate.
• You choose with ipfs add --chunker=size-N, rabin-..., or buzhash. Only fixed size promises the same CID for the same data across releases.

Updated Oct 19, 2026: added the ipfs add --chunker options from the Kubo docs, the 1 MiB unixfs-v1-2025 profile, and a byte-shift diagram. The AIOZ Pin section is corrected: our docs don't name the upload chunker.

What is chunking in IPFS?

When you add a file, IPFS doesn't store it as one object. It splits the bytes into chunks, hashes each chunk into a block, and links the blocks under a root node. The root's hash becomes your CID. So a 10 MiB file with the default chunker becomes 40 leaf blocks of 256 KiB each, plus the node that links them.

Chunking happens once, when the file is added. After that, the blocks are what gets stored, deduplicated, and fetched from peers. How IPFS works covers the rest of that path, from blocks to a Merkle DAG to content routing.

The default: fixed-size chunking

The default splitter is about as simple as chunking gets. It reads 262,144 bytes, makes that one block, and repeats until the file runs out. The final block holds whatever is left. In the Go library boxo/chunker, this is DefaultSplitter, built on NewSizeSplitter with DefaultBlockSize = 1024 * 256.

Fixed size has real strengths. It's fast, since there's no hash to compute per byte. Every block but the last has the same size, which makes storage easy to predict. And it's deterministic, which matters for CIDs, as we'll see below.

Kubo also has a newer import profile, unixfs-v1-2025, that uses 1 MiB fixed chunks with CIDv1 and raw leaves. The Kubo config docs recommend it for new projects that need the same CID across IPFS implementations. You turn it on with ipfs config profile apply unixfs-v1-2025.

Why does one edit change every block?

This is the byte-shifting problem. Say you insert a single byte near the start of a large file and add it again. Every boundary after the insert moves by one byte. Each later block now holds slightly different bytes, so each one hashes differently.

So a one-byte edit to a 10 MB file can look, block by block, like a new file. Over 99.99% of the content is the same, yet almost none of it deduplicates. IPFS can only skip storing a block if that exact block already exists, and after a shift, almost none do.

Appending to the end is the friendly case. If you only add bytes at the end, every earlier fixed-size block stays identical, and only the last one or two change.

How does content-defined chunking work?

Content-defined chunkers don't count bytes. They slide a rolling hash over the data and cut wherever the hash matches a pattern. Since the cut points come from the content, an insert only disturbs the boundary near the edit. Further along, the chunker finds the same patterns in the same bytes, and the boundaries line up again. Most later blocks come out identical.

Kubo ships two of these:

• Rabin uses Rabin fingerprinting. Give it one average size and it derives the bounds: a minimum of a third of the average and a maximum of 1.5 times it. Or you set all three yourself. The minimum must be greater than 16 bytes.
• Buzhash uses a cyclic polynomial hash with fixed bounds: blocks are at least 128 KiB and at most 512 KiB. You can't tune it from the command line.

The trade-off is predictability. You get boundaries that survive edits, but block sizes vary within a range, and hashing every byte costs more CPU than counting to 262,144.

Choosing an ipfs add --chunker option

The --chunker flag (short form -s) sets the algorithm for one ipfs add. The Kubo CLI reference lists the formats as size-[bytes], rabin-[min]-[avg]-[max], or buzhash. The boxo parser also accepts plain rabin and rabin-[avg]. These commands follow those docs. We didn't run them here, because this machine has no IPFS binary.

# Default: fixed-size 256 KiB chunks (Import.UnixFSChunker = size-262144)
ipfs add photo.tar

# Fixed size, explicit byte count (1 MiB here)
ipfs add --chunker=size-1048576 photo.tar

# Rabin: one value sets the average, min and max are derived from it
ipfs add --chunker=rabin-262144 photo.tar

# Rabin with explicit min, average, and max
ipfs add --chunker=rabin-131072-262144-524288 photo.tar

# Buzhash, fixed internal bounds
ipfs add --chunker=buzhash photo.tar

# Make a chunker the node default
ipfs config Import.UnixFSChunker buzhash

Two limits apply. A size- value or a Rabin maximum can't exceed 2 MiB minus 256 bytes (2,096,896 bytes), so blocks stay inside the Bitswap 2 MiB block limit. Buzhash has its own 512 KiB cap, so the limit doesn't touch it.

As a rule of thumb, use the default, or the 1 MiB profile, when you want reproducible CIDs and files that rarely change. Try Rabin or Buzhash when you store many versions of large files that change in the middle, such as datasets, disk images, or archives. Test it on your own data before you switch, because the gain depends on how your files change.

How the chunker changes a file's CID

Different boundaries give different blocks, a different DAG, and so a different root hash. The same file added with size-262144 and with buzhash gets two CIDs. That's one reason the same file can get different CIDs in different tools.

The Kubo config docs add one more warning. Only the fixed-size chunker guarantees that the same data always produces the same CID. Rabin and Buzhash may change their internal parameters in a future release. If someone needs to rebuild your CID from the raw file later, fixed size is the safe choice.

How AIOZ Pin handles chunking

Our upload endpoint, POST /api/pinning, and the SDK's pinFilesToIPFS and pinFolderToIPFS take your file and return a CID. They don't expose a chunker option, and the AIOZ Pin docs don't say which chunker the server applies. So treat the chunking of an upload as fixed by us.

If the chunker matters to you, build the DAG yourself and pin it by CID. Add the file on your own Kubo node with the --chunker you want, then hand the CID to AIOZ Pin. You can do that through the Pinning Service API, or through POST /api/pinning/pinByHash. AIOZ Pin then fetches exactly the blocks you made, so the CID stays yours. With the remote service set up as in our Pinning Service API guide, it looks like this (not run here):

ipfs add --chunker=buzhash --cid-version=1 dataset-v2.tar
ipfs pin remote add --service=pin --name=dataset-v2 <CID from the line above>

Keep your node online until the pin shows pinned, because AIOZ Pin has to fetch the blocks from it. Next step: read the AIOZ Pin Pinning Services API docs to set up the remote service.

Frequently Asked Questions

What is the default IPFS chunk size? 262,144 bytes (256 KiB) per block, except the last block of a file. Kubo stores it as Import.UnixFSChunker = size-262144. The unixfs-v1-2025 profile switches to 1 MiB.

What is the byte-shifting problem? With fixed-size chunks, inserting or deleting bytes moves every boundary after the edit. Most later blocks then hash differently, even though most content is unchanged.

What is content-defined chunking? A method that cuts where a rolling hash over the data matches a pattern, instead of at a fixed byte count. Boundaries realign after an edit, so most blocks stay the same.

How do I change the chunker in IPFS? Pass --chunker to ipfs add, for example --chunker=buzhash or --chunker=rabin-262144. To change the default, set Import.UnixFSChunker with ipfs config.

What is the largest chunk size IPFS allows? 2,096,896 bytes (2 MiB minus 256 bytes) for size- and the Rabin maximum. Buzhash is capped at 512 KiB.

Which chunker does AIOZ Pin use? The AIOZ Pin docs don't name it, and the upload API has no chunker option. To control it, add the file on your own node with --chunker and pin the resulting CID.

Related Reading

• How Does IPFS Work? Merkle DAGs, Chunking, and Routing - where chunking fits on the path from file to CID
• Anatomy of a CID - what the hash of the root block encodes
• UnixFS: How IPFS Represents Files and Directories - where chunk sizes get recorded in a file's structure
• IPLD: The Data Model Underneath IPFS - the data model behind the links between blocks
• Why IPFS Gave Different CIDs for the Same File - how chunker and CID settings split CIDs across tools
• How AIOZ Pin keeps files online - what happens to the blocks after you pin them

References

• boxo/chunker package documentation
• boxo/chunker source (parse.go, rabin.go, buzhash.go)
• Kubo CLI reference: ipfs add
• Kubo config docs: Import.UnixFSChunker and profiles
• IPIP-499: UnixFS CID Profiles
• AIOZ Pin Pinning API documentation