Back

Blog details

UnixFS: How IPFS Represents Files and Directories

AIOZ Network
7 min readAugust 27, 2026
aioz-pin

UnixFS is the data format IPFS uses to represent files and directories. The raw Merkle DAG only knows hash-linked blocks of bytes. UnixFS adds filenames, folders, symlinks and file sizes on top, so a CID can stand for a whole file or folder.

TL;DR:

  • The Merkle DAG is a generic hash-linked graph of blocks; on its own it has no idea of files, directories, or filenames
  • UnixFS is a protobuf format, encoded with the dag-pb codec, that adds file, directory and symlink meaning to DAG nodes
  • The DataType field has six values: Raw, Directory, File, Metadata, Symlink and HAMTShard, although the spec now marks Raw as deprecated and Metadata as reserved
  • Large files are split into chunks (256 KiB by default in Kubo) and stitched back together through a File node's filesize and blocksizes fields
  • Large directories switch to a HAMTShard layout once they pass a size threshold (256 KiB in Kubo), with a fanout of 256

Updated Oct 29, 2026: tightened the explanation, checked node types and defaults against the current UnixFS spec and Kubo docs, added a stdlib-computed raw block CID, a labelled diagram, and a closing section on what the AIOZ Pin docs say about folders.

Diagram of a UnixFS tree: a Directory node with named links beach.jpg and notes.txt; beach.jpg is a File node with filesize and blocksizes fields pointing to raw leaf chunks of up to 256 KiB; notes.txt fits in one block; a very large directory becomes a HAMTShard with fanout 256 past a 256 KiB threshold
How UnixFS turns a folder into DAG nodes. Defaults shown are Kubo's.

Why Does IPFS Need UnixFS?

The Merkle DAG that underlies all of IPFS is generic on purpose. It has nodes, links and hashes, and it makes no assumption about what the data inside a node means. That is why IPFS can carry many data formats on the same graph, not only files.

The cost of that generality is that the DAG can't answer the questions a filesystem needs. Is this node a file or a folder, what is it called, and how big is it? And if it is split across blocks, in what order do they go back together? UnixFS is the layer that answers them for you, every time you add a file or folder.

What Is UnixFS?

UnixFS is a protocol-buffers format for describing files, directories and symlinks on IPFS. Each UnixFS node is a dag-pb block whose Data field holds a small protobuf message. That message has a required Type field, plus optional fields for raw bytes, file size, block sizes and directory settings such as fanout.

To be precise, UnixFS is one way of reading the Merkle DAG, not a different storage system. The blocks and hashes are the same; a UnixFS-aware tool just knows how to read the extra structure. A tool that doesn't understand UnixFS can still fetch and verify every block, but it can't show you files and folders.

The Six UnixFS Node Types

The DataType enum in the UnixFS spec lists six values. Here is what each one does today:

  • Raw (0): bare data inside a dag-pb node. The spec now marks it deprecated and says to use blocks with the separate raw codec instead.
  • Directory (1): a folder whose links are named entries, which is how myfolder/photo.png resolves by name.
  • File (2): a file, either holding its bytes directly or linking to chunk blocks.
  • Metadata (3): reserved for future use.
  • Symlink (4): a symbolic link, preserved the way a Unix filesystem would keep it.
  • HAMTShard (5): one node of a sharded directory, used when a folder grows too large.

Day to day, you mostly meet File, Directory and HAMTShard nodes, plus raw leaf blocks if you add files with CIDv1. The spec also defines two optional fields from UnixFS 1.5, mode and mtime, for Unix permissions and modification times, and implementations only set them when you opt in.

How a Large File Becomes Many Blocks

Blocks have a practical size ceiling, so any big file you add has to be split. Kubo's default chunker cuts files into fixed 256 KiB pieces, and the post on fixed-size vs content-defined chunking covers the other options.

The File node then links to those chunks, in order, and records two fields. filesize is the total length of the file, and blocksizes lists the size of each child. A UnixFS reader uses blocksizes to rebuild the file in the right order, whichever peers the blocks come from. The spec notes that the same list shows which blocks hold a byte range, so you can serve an HTTP Range request without fetching the whole file.

Very large files need more than one level, because Kubo caps a File node at 174 links by default, so beyond that it builds a tree of File nodes above the leaves. The leaves themselves are often not UnixFS nodes at all. With CIDv1, Kubo turns on raw leaves by default, so each chunk is stored as a plain raw block.

A raw block's CID is easy to compute yourself, and we built one with Python's standard library for the 12 bytes hello world\n. The recipe is the bytes 01 55 12 20 (CIDv1, raw codec, sha2-256, 32-byte length), then the SHA-256 digest, encoded as lowercase base32 with a b prefix:

# Run here with Python's standard library (no IPFS binary)
import hashlib, base64
data = b"hello world\n"
cid = bytes([0x01, 0x55, 0x12, 0x20]) + hashlib.sha256(data).digest()
print("b" + base64.b32encode(cid).decode().lower().rstrip("="))
# bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4

The bafkrei prefix is typical of a CIDv1 raw block with a SHA-256 hash. A dag-pb UnixFS node for the same bytes would have a different CID, which is one reason the same file can end up with different CIDs.

How a Large Directory Becomes a HAMTShard

A plain Directory node stores its entries as one flat list of named links. That works until the folder is so big that the list itself becomes a huge block. Kubo switches a directory to sharding when its estimated size passes 256 KiB by default, and the spec gives 256 KiB to 1 MiB as the common range.

Past that point, the folder becomes a tree of HAMTShard nodes. The fanout field sets how many buckets each level has, and Kubo's default is 256, which uses 8 bits of the name's hash per level. For you, folder/file.txt still resolves the same way. Only the node structure behind it changes, as the post on how IPFS shards large directories explains in detail.

What Does This Mean When You Pin a Folder?

When you pin a folder's root CID recursively, you protect the whole UnixFS tree under it. That includes every File node, every chunk, every subdirectory and every HAMT shard, because each one is reachable by links from the root. The post on unpinning and garbage collection covers what a recursive pin keeps and what garbage collection removes.

It also explains why changing one file in a folder changes the folder's CID. The file's CID changes, so the link in its parent changes, so every directory up to the root gets a new hash. A static website hosted on IPFS is exactly this: a build folder turned into a UnixFS tree, with one root CID per release.

How AIOZ Pin Handles It

AIOZ Pin's docs give you two documented ways to pin a whole folder: the Node.js SDK and the pinning CLI. In the quick start, client.pinFolderToIPFS takes the folder path as sourcePath, and the CLI pins a directory when you pass its path to --file. The docs don't say which chunk size, CID version or sharding threshold the service uses when it builds the tree.

This snippet is copied from the AIOZ Pin quick start and was not run here, since it needs real keys:

// Not run: from the AIOZ Pin quick start (Node.js SDK)
client.pinFolderToIPFS({ sourcePath: './folder', options }).then((result) => {
  console.log(result);
}).catch((err) => {
  console.log(err);
});

The pin records in the Pinning API include an is_dir flag, so you can tell a folder pin from a file pin when you list your pins. Next step: follow the AIOZ Pin quick start to pin your first folder.

Frequently Asked Questions

What is UnixFS in IPFS? It is the format IPFS uses for files and directories. It adds file, directory and symlink meaning on top of the Merkle DAG, which has none of its own.

Does UnixFS change how the Merkle DAG works? No. It is a way of reading the same DAG, using dag-pb blocks with a UnixFS message inside, not a separate storage layer.

What are the UnixFS node types? Raw, Directory, File, Metadata, Symlink and HAMTShard, though the spec marks Raw as deprecated in favor of raw codec blocks, and Metadata as reserved.

How does IPFS put a large file back together? The File node's filesize and blocksizes fields record the total size and each chunk's size in order. A reader uses them to rebuild the file, whichever order the blocks arrive in.

What chunk size does IPFS use? Kubo's default chunker is size-262144, which is 256 KiB per chunk. You can change it per add or in the config.

Is a HAMTShard different from a regular directory? It is the same directory stored a different way: Kubo switches to it once the directory passes about 256 KiB, and lookups by name work the same either way.

References

We only send updates when meaningful changes ship, and you can unsubscribe anytime

Related Content

blog thumbnail

What Is a Multiaddr? Peer IDs and Addresses in IPFS

A multiaddr is a network address that names every protocol it uses, like /ip4/.../tcp/4001/p2p/ and a peer ID. Learn to read one and how IPFS peer IDs work.

7 min readNovember 06, 2026
blog thumbnail

IPFS Bootstrap Nodes and How a New Node Finds Its Peers

IPFS bootstrap nodes are the first peers a new node dials to join the Amino DHT. See how Kubo uses them, who runs them after Shipyard, and how to edit the list.

6 min readNovember 04, 2026
blog thumbnail

ipfs.io Gateway Not Working? What Changed and Who Runs It

The ipfs.io gateway now returns HTTP 429 to scripts and servers and sends browsers to inbrowser.link. See who runs the public IPFS services and what to change.

6 min readNovember 02, 2026
blog thumbnail

IPFS Glossary: Key IPFS and libp2p Terms Explained A to Z

An IPFS glossary of 115 terms from A to Z: CIDs, Bitswap, the DHT, IPNS, gateways and pinning, each defined in plain words and linked to a deeper guide.

21 min readOctober 31, 2026
blog thumbnail

What Is IPFS Provide Sweep? How Kubo Announces Many CIDs

IPFS Provide Sweep is Kubo's provider that batches CIDs by DHT region, not one lookup each. See how it works, when it became default, and how to check it.

6 min readOctober 30, 2026
blog thumbnail

IPFS Dedicated Gateway Free vs Paid Plans on AIOZ Pin

An IPFS dedicated gateway on AIOZ Pin is called a premium gateway. See what the free plan limits, what a paid plan adds, and what happens at zero balance.

6 min readOctober 28, 2026