UnixFS: How IPFS Represents Files and Directories

UnixFS is the format that makes "file" and "directory" meaningful concepts on IPFS at all. The raw Merkle DAG covered elsewhere in this series only knows about hash-linked blocks of bytes, it has no built-in idea of a filename, a folder, file permissions, or where one file ends and another begins. UnixFS is the layer that adds all of that on top, and understanding it explains why a pinned directory behaves the way it does, and why very large directories eventually turn into the HAMT structure covered in an earlier article in this series.

TL;DR:

• The Merkle DAG is a generic hash-linked graph of blocks; it has no native concept of files, directories, or filenames
• UnixFS is a protocol-buffers-based format layered on top of the DAG that adds exactly that: file, directory, and symlink semantics, encoded using dag-pb
• Every UnixFS node carries a DataType field with six possible values: Raw, File, Directory, Metadata, Symlink, and HAMTShard
• Large files get split into chunks and reassembled through a File node's filesize and blocksizes fields; large directories convert into a HAMTShard node using a fanout field, the same HAMT mechanism covered in the previous article
• When AIOZ Pin's pinFolderToIPFS() pins a directory, it is walking and protecting a UnixFS tree, not a flat list of files, which is why pinning a folder recursively protects everything nested inside it automatically

The Merkle DAG Doesn't Know What a File Is

The Merkle DAG structure that underlies all of IPFS is deliberately generic: nodes, links, and hashes, with no assumptions baked in about what the data inside a node actually represents. That generality is a feature, it is what lets IPFS support entirely different data formats (arbitrary IPLD structures, not just files) on the exact same underlying graph. But it also means the raw DAG alone cannot answer basic questions a filesystem needs answered: is this node a file or a folder, what's it called, how big is it, and if it's split across multiple blocks, in what order do they go back together.

UnixFS: A Filesystem Format Layered on Top

UnixFS answers those questions. It is a protocol-buffers-based format for describing files, directories, and symlinks on IPFS, encoded into DAG nodes using the dag-pb codec. Every UnixFS-formatted DAG node carries a small protobuf Data structure with a required Type field, and optional fields for the node's raw byte payload, file size, and directory-specific metadata like fanout width. It is worth being precise about what this is: UnixFS is one interpretation of the generic Merkle DAG, not a different underlying storage mechanism. The DAG doesn't change; a UnixFS-aware client just knows how to read the extra structure layered onto it.

The Six Node Types

The DataType field in UnixFS's Data message defines exactly six kinds of node: Raw (bare, unwrapped data), File, Directory, Metadata, Symlink, and HAMTShard. Everyday content addressing only really touches a handful of these directly. Adding a small file to IPFS typically produces a File node. Adding a folder produces a Directory node whose links are named entries pointing at the files and subfolders inside it, which is also what makes it possible to fetch myfolder/photo.png by name instead of by raw CID. Symlink nodes preserve symbolic links the same way a Unix filesystem would. HAMTShard is the special case covered below.

How a Large File Becomes Multiple Blocks

A single block has a size ceiling, so a file larger than that ceiling can't fit in one node no matter what type it is. UnixFS handles this with two fields on the File node: filesize, the total byte length of the reassembled file, and blocksizes, a repeated list recording the size of each child block in order. Combined with the chunking process covered in this series' dedicated chunking article, this is what lets a large file get split into many blocks on the way in, and reassembled in the exact right order on the way out, regardless of which peers those blocks actually get fetched from.

How a Large Directory Becomes a HAMTShard

A plain Directory node stores its entries as a flat list of named links, which works fine until the directory holds so many entries that the node itself would need to be enormous to list them all. Past that threshold, UnixFS switches representation to a HAMTShard node, using the fanout field to configure how wide the resulting hash-trie splits at each level, the same Hash Array Mapped Trie mechanism, table size 256 in IPFS's implementation, covered in this series' HAMT article. The directory's contents don't change conceptually, folder/file.txt still resolves the same way, but the node structure that gets you there switches from one flat list to a tree of shards once flat storage stops being practical.

Why the Layering Matters

Splitting "generic hash-linked graph" (the DAG) from "this specific graph represents a filesystem" (UnixFS) is what lets IPFS support other IPLD-based data formats on the same underlying transport, routing, and verification machinery without needing filesystem semantics baked into the core protocol. It also means a UnixFS-unaware tool can still fetch and verify raw blocks correctly, it just won't know how to present them as files and folders, since that interpretation lives one layer up.

Where AIOZ Pin Fits In

When AIOZ Pin's pinFolderToIPFS() pins a directory, what actually gets recursively pinned is a UnixFS Directory (or HAMTShard, past the size threshold) node and everything it links to, files, subdirectories, and any nested HAMT shards included. This is the concrete mechanism behind the recursive-pin behavior covered in this series' garbage collection article: a recursive pin on a folder's root node protects the whole UnixFS tree beneath it automatically, because that's exactly what "recursive" means when the thing being pinned is a directory built out of linked UnixFS nodes rather than a single flat file.

Frequently Asked Questions

What is UnixFS in IPFS? A protocol-buffers-based format layered on top of the Merkle DAG that adds file, directory, and symlink semantics, since the raw DAG has no native concept of any of those.

Does UnixFS change how the Merkle DAG works? No. It's an interpretation of the same DAG structure, encoded using the dag-pb codec, not a separate storage layer. A DAG node either has UnixFS-formatted data in it or it doesn't.

What are the six UnixFS node types? Raw, File, Directory, Metadata, Symlink, and HAMTShard, defined by the DataType field on every UnixFS node's Data structure.

How does IPFS know how to reassemble a large file from multiple blocks? The File node's filesize and blocksizes fields record the total size and each chunk's size in order, which is enough to reconstruct the file correctly regardless of fetch order.

Is a HAMTShard a different thing from a regular directory? It's the same conceptual directory, just a different node representation used once a directory has too many entries to list flatly. Lookups still resolve by name either way.

Does pinning a folder on AIOZ Pin protect everything inside it? Yes, as long as it's pinned recursively, which is the default for pinFolderToIPFS(). A recursive pin on a UnixFS directory node protects every file and subdirectory it links to.

Related Reading

• How IPFS Shards Large Directories: Inside the Hash Array Mapped Trie (Not a B-Tree) - the HAMTShard mechanism this article's fanout section builds on
• What Happens When You Unpin a File on IPFS: Garbage Collection Explained - what a recursive pin on a UnixFS tree actually protects
• How IPFS Splits Files Into Blocks: Fixed-Size vs. Content-Defined Chunking - how the chunks behind a File node's blocksizes get created
• How IPFS Really Works: Merkle DAGs, Chunking, and Content Routing Explained - the generic DAG structure UnixFS is layered on top of
• How to Host a Static Website on IPFS (and Why Pin It) - a UnixFS directory tree in practice, the actual thing a static site's build folder becomes

References

• IPFS Docs: File Systems
• go-unixfs unixfs.proto source