How IPFS Shards Large Directories: The HAMT, Not a B-Tree

A HAMT (hash array mapped trie) is the structure IPFS uses to split one large directory into many smaller blocks, and it is not a B-tree. Kubo switches a directory to a HAMT once its listing passes about 256 KiB, and you never call anything different to make that happen.

TL;DR:

• IPFS shards large directories with a HAMT, a tree of shard nodes keyed by a hash of each file name, not with a B-tree
• Kubo converts a basic directory to a HAMT when its estimated size goes over Import.UnixFSHAMTDirectorySizeThreshold, which defaults to 256 KiB
• Each shard node has 256 buckets by default (Import.UnixFSHAMTDirectoryMaxFanout), and names are hashed with murmur3-x64-64
• Sharding changes the directory's CID: the UnixFS spec test vectors show one directory at 262,144 bytes staying basic and one at 262,145 bytes becoming a HAMT
• On AIOZ Pin you pin a folder the same way at any size; the docs don't say which import settings the service applies

Updated Oct 19, 2026: added the Kubo threshold settings, the IPIP-499 boundary test vectors, a lookup diagram, and a corrected answer on whether sharding changes the CID.

What Is a HAMT?

A hash array mapped trie is a tree where each node holds a fixed number of buckets. You hash the key, read a few bits of the hash, and those bits pick the bucket at the current level. If the bucket holds the entry, you are done, and if it holds a pointer to another node, you go down one level and read the next few bits.

In IPFS the key is a file name inside the directory. The UnixFS spec says every current implementation hashes names with murmur3-x64-64. With the default fanout of 256, each level consumes 8 bits of that hash, so the bucket names run from 00 to FF.

A B-tree works differently: it keeps keys sorted and splits or merges nodes to stay balanced. A HAMT never sorts anything and routes by hash, so its depth depends on how many bits it takes before names stop colliding. If you saw "B-tree" in an older answer about IPFS directories, the specs and the Kubo code both call this structure a HAMT.

Why a Directory Needs Sharding

A small IPFS directory is one dag-pb block that lists every entry as a name and a CID. That works until the listing gets big. The Kubo config docs warn that peers on the public swarm tend to ignore blocks over 2 MiB, so one giant listing block can become impossible to fetch.

A single block also makes every change expensive, because adding one file to a 50,000-entry listing means rewriting and re-hashing the whole block. With a HAMT, the same change rewrites one shard plus the path of shards above it.

When Does IPFS Shard a Directory?

In Kubo, the switch is controlled by Import.UnixFSHAMTDirectorySizeThreshold, which affects ipfs add and the daemon. The default is 256 KiB, and the docs add "may change", so check your version before you rely on it. Setting it to 1B forces every directory into a HAMT, which the docs suggest for testing.

The size is an estimate, not the exact block size. By default (Import.UnixFSHAMTDirectorySizeEstimation set to links), Kubo adds up the byte length of each link name plus each CID. The newer block mode measures the full serialized block instead, and the unixfs-v1-2025 import profile turns it on.

Here are the related keys with the defaults the Kubo config docs list. This is a fragment of the config file, not command output:

"Import": {
  "UnixFSHAMTDirectorySizeThreshold": "256KiB",
  "UnixFSHAMTDirectoryMaxFanout": 256,
  "UnixFSHAMTDirectorySizeEstimation": "links",
  "UnixFSDirectoryMaxLinks": 0
}

How many files fit in 256 KiB? By our own arithmetic, not a documented figure, a CIDv0 is 34 bytes in binary, so an entry with a 10-character name costs about 44 bytes. That puts the switch somewhere near 6,000 entries, and longer names reach it sooner. A second trigger, UnixFSDirectoryMaxLinks, can force the switch by entry count, but it is off by default.

The boundary is strict: IPIP-499 says a directory converts only when the estimate is greater than 262,144 bytes. Its test vectors include a directory at exactly 262,144 bytes that stays basic and one at 262,145 bytes that becomes a HAMT.

How a Lookup Walks the Shards

The UnixFS spec walks through a real test vector, a sharded directory with 1,000 files. To find 470.txt, you hash the name with murmur3, and the first 8 bits give bucket 00 at the root. That link points to a sub-shard, because several names landed in the same bucket.

The sub-shard reads the next 8 bits, which give 6E. There it finds a link named 6E470.txt, the bucket prefix followed by the original name. That link points straight at the file's CID, so the lookup touched two shard blocks instead of a list of 1,000 entries.

Here is the root shard as the spec shows it, trimmed:

// Root HAMT shard (bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i)
Data = {
  Type = HAMTShard        // Type = 5
  Data = 0xffffff...      // Bitmap: bits set for populated buckets
  hashType = 0x22         // murmur3-x64-64
  fanout = 256            // 256 buckets (8-bit width)
}
Links = [
  { Hash = bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni, Name = "00" },
  { Hash = bafybeia322onepwqofne3l3ptwltzns52fgapeauhmyynvoojmcvchxptu, Name = "01" },
  // ... more buckets as needed up to "FF"
]

Three details from the spec matter if you read raw blocks. The bitmap in Data tells a reader which buckets exist without scanning every link. Bucket prefixes must be uppercase hex, and empty intermediate shards are typically collapsed on deletion, so the same set of names produces the same tree whatever order you added them in.

Why 256 Buckets Instead of 32

Many textbook HAMTs use 32 buckets per node, while IPFS defaults to 256, and the spec allows any multiple of 8 that is a power of two, up to 1,024. A wider node means fewer levels, so a lookup fetches fewer blocks, but each block is bigger and every change rewrites more bytes.

The spec caps fanout at 1,024 for safety, since larger values could exhaust memory when a node allocates its bucket array, a problem tracked as CVE-2023-23625. You can change the fanout with Import.UnixFSHAMTDirectoryMaxFanout, but a different fanout gives you a different CID for the same files.

Does Sharding Change a Directory's CID?

Yes. A HAMT root is a different block from a basic directory block, so the same files get a different directory CID once sharding kicks in. The two IPIP-499 vectors above show it, since one extra byte of estimated size moves the directory from a basic node to a HAMT root with a new CID.

This is why two tools could hand you different CIDs for the same folder. The IPIP-499 survey found Kubo v0.39 switching at >= the threshold while Helia used >, and early js-ipfs counted entries instead of bytes. IPIP-499 settled on > and named profiles, so you can pin down the settings when you need CIDs to match.

How AIOZ Pin Handles It

You pin a folder on AIOZ Pin the same way at any size. The SDKs page in our docs shows the CLI form, which takes a directory path:

pinning pin --file ../some/where

The Node.js SDK does the same with pinFolderToIPFS(). Its depth option sets how far into subfolders the pin goes: all, 0 for the top folder only, or 1 for one level down. The response returns one root cid with is_dir set to true.

What the docs don't cover yet: which import settings the service uses, meaning CID version, chunk size, HAMT threshold and size estimation. If you need a folder CID to match one you computed locally, pin it by CID instead with the Pinning API's pinByHash, which takes a CID you already have. Start with the AIOZ Pin SDKs page.

Frequently Asked Questions

Does IPFS use a B-tree for directories? No. Large IPFS directories are sharded with a HAMT (hash array mapped trie), while small directories stay a single flat block.

What is the default HAMT threshold in Kubo? The default is 256 KiB, set by Import.UnixFSHAMTDirectorySizeThreshold, and the docs note it may change, so check your version.

Which hash function does an IPFS HAMT use? murmur3-x64-64 (multihash code 0x22). It is fast, not cryptographic, and that is fine: the CIDs inside each shard already secure the content.

Can I turn HAMT sharding off? Yes. Setting Import.UnixFSHAMTDirectorySizeEstimation to disabled keeps directories basic, but a huge directory then risks a block over 2 MiB that public peers may refuse.

Does sharding change a folder's CID? Yes. A sharded directory has a different root block, so its CID differs from the basic form of the same files, which is why IPIP-499 pins the threshold rules down.

Do I need to do anything different to pin a large folder on AIOZ Pin? No. Both pinning pin --file and pinFolderToIPFS() work the same at any folder size, and sharding happens below the API.

Related Reading

• UnixFS: How IPFS Represents Files and Directories - where HAMTShard sits among the other UnixFS node types
• Why IPFS Used to Give You Different CIDs for the Same File - the IPIP-499 profiles that fixed the threshold mismatch
• How IPFS Works: Merkle DAGs, Chunking, and Routing - the overview this article goes deeper on
• IPLD: The Data Model Underneath IPFS's Merkle DAG - the data model a shard node is built from
• AIOZ Pin for Developers: API, SDK, and CLI Explained - where pinFolderToIPFS() fits among the SDK methods
• How to Mint NFTs with Persistent Metadata on AIOZ Pin - a case where folder-scale pinning comes up

References

• UnixFS specification: HAMTDirectory
• IPIP-499: UnixFS CID Profiles
• Kubo config docs: Import.UnixFSHAMTDirectorySizeThreshold
• AIOZ Pin docs: SDKs
• js-hamt-sharding