How IPFS Shards Large Directories: The HAMT, Not a B-Tree

If you have gone looking for how IPFS handles a folder with thousands of files in it, you have probably run into the term "b-tree" somewhere along the way. That is not what is actually happening. IPFS shards large directories using a HAMT, a Hash Array Mapped Trie, a different structure with a different set of tradeoffs. This covers what a HAMT actually is, why IPFS reaches for it specifically, and what happens inside AIOZ Pin when a folder you pin crosses that threshold.

TL;DR:

• IPFS does not use a B-tree to shard large directories. It uses a HAMT (Hash Array Mapped Trie), a tree of DAG nodes keyed by a hash of each entry's name
• A small directory is a single DAG node listing its entries directly; once it grows past a size threshold, IPFS reshapes it into a HAMT automatically
• IPFS's HAMT uses a table size of 256 per node (not the 32 commonly used in general-purpose HAMT implementations) and hashes names with Murmur3
• Two optimizations keep it efficient: leaf values get inlined into their parent node instead of requiring a separate lookup, and shards containing only one child get collapsed to avoid pointless extra hops
• Pinning a folder on AIOZ Pin through pinFolderToIPFS() triggers this same sharding transparently once the folder is large enough; nothing about the call changes

What a HAMT Actually Is

A Hash Array Mapped Trie is a tree where each node holds a fixed-size table of slots, and which slot an entry lands in is decided by hashing its key and reading off a few bits of that hash per level. Look up an entry by hashing its name, use the first chunk of bits to pick a slot at the root, and either find the entry there or follow a pointer down to a child node keyed by the next chunk of bits. It is a trie in the classical sense (a lookup structure keyed by pieces of an identifier, one level per chunk) built out of hash-derived indices instead of literal character-by-character keys.

This is a different structure from a B-tree. A B-tree balances itself by splitting and merging nodes to keep entries in sorted order and the tree's depth even; a HAMT does not sort anything and does not rebalance, it simply routes by hash, and its depth is bounded by how many bits of hash it takes before entries stop colliding. Both are tree structures used to make lookups faster than scanning a list, but they get there in genuinely different ways, and IPFS's own specs and codebase describe this structure specifically as a HAMT, never as a B-tree.

Why a Directory Needs Sharding at All

A small IPFS directory is stored as one DAG node, essentially a list of (name, CID) pairs for its entries. That works fine until the directory gets big enough that listing every entry in one node stops being practical, at which point IPFS reshapes the directory into a HAMT automatically, without you doing anything differently on the client side. The reshaping is transparent: you still add files to the directory and read them back by name, the sharded structure is an internal representation, not something you interact with directly.

The Specific Shape of IPFS's HAMT

IPFS's implementation (the hamt package inside go-unixfs, and its JavaScript counterpart js-hamt-sharding) makes a few concrete choices worth knowing:

• Table size of 256 per node, rather than the 32 used in many textbook and general-purpose HAMT implementations. A wider table means fewer levels of nesting for a given number of entries, trading a bit more space per node for shallower lookups.
• Murmur3 as the hash function used to derive each entry's position, chosen for speed rather than cryptographic strength; a directory's sharding does not need to resist tampering the way content hashing does, since directory listings are already covered by the Merkle DAG's own hash-of-hashes property.
• Leaf-value inlining: an entry stored directly in a shard node includes its value in that same node rather than pointing to a separate node one hop away, which saves a full fetch at the DAG level for the common case of reading a single file out of a large directory.
• Empty-shard collapsing: if a shard ends up holding only one child, IPFS collapses it rather than leaving a near-empty node in the middle of the tree, keeping the structure from bloating with pointless single-entry hops.

Why This Matters for Lookup Speed

Without sharding, a directory with tens of thousands of entries would either need one enormous DAG node listing every entry (slow to fetch and slow to update, since adding one file means re-hashing and re-transmitting the whole list) or a flat structure that gets linearly slower to search as entries pile up. A HAMT avoids both problems: looking up a single file by name touches only as many nodes as the tree is deep, a handful of hops rather than a scan through every entry, regardless of whether the directory holds a hundred files or a hundred thousand. That is the actual payoff of choosing a hash-keyed tree over a flat list: lookup cost stays roughly constant as the directory grows, instead of degrading with it.

What This Looks Like on AIOZ Pin

AIOZ Pin's Node.js SDK exposes pinFolderToIPFS() for pinning an entire directory in one call. Nothing about calling it changes based on folder size, but what happens underneath does: a folder small enough to list directly stays a flat DAG node, and one that grows past the sharding threshold gets represented as a HAMT automatically, the same mechanism any IPFS implementation uses. This matters practically if you are pinning something like a large NFT collection's asset folder or a dataset with thousands of files: the CID you get back for the folder is still one identifier either way, the sharding is invisible from the API, but it is the reason lookups inside a folder with 50,000 files stay fast instead of degrading linearly.

Frequently Asked Questions

Does IPFS use a B-tree for directories? No. Large IPFS directories are sharded using a HAMT (Hash Array Mapped Trie), a hash-keyed tree structure, not a B-tree.

What triggers a directory to become a HAMT on IPFS? Directory size. A small directory stays a single flat DAG node listing its entries; once it grows past IPFS's internal size threshold, it gets automatically reshaped into a HAMT.

Do I need to do anything differently to pin a large folder on AIOZ Pin? No. pinFolderToIPFS() works the same way regardless of folder size; sharding happens transparently underneath the same API call.

Why does IPFS use a table size of 256 instead of the standard 32? A wider table means fewer levels of nesting are needed for a given number of entries, trading a small amount of extra space per node for shallower, faster lookups.

Is Murmur3 a secure hash function? No, and IPFS does not need it to be for this purpose. Murmur3 is fast, not cryptographically secure, and directory sharding does not need tamper-resistance since the surrounding Merkle DAG already provides that at the content level.

Does sharding change a folder's CID? No. The folder still resolves to one CID either way; the HAMT is purely an internal representation of how that folder's entries are organized on disk and over the network.

Related Reading

• How IPFS Really Works: Merkle DAGs, Chunking, and Routing - the overview this article goes deeper on
• UnixFS: How IPFS Represents Files and Directories - where HAMTShard fits among UnixFS's other node types
• Minting NFTs with Persistent Metadata on AIOZ Pin - a practical case where folder-scale pinning comes up
• AIOZ Pin for Developers: API, SDK, and CLI Explained - where pinFolderToIPFS() fits among AIOZ Pin's other SDK methods

References

• go-unixfs HAMT package documentation
• js-hamt-sharding: JavaScript implementation of hash array mapped tries for use in sharding