How IPFS Really Works: Merkle DAGs, Chunking, and Routing

How does IPFS work, mechanically, once a file leaves your machine? Most explainers stop at "it's decentralized storage" and move on. This one does not. A file added to IPFS gets split into blocks, each block gets hashed into a content identifier, the blocks link together into a Merkle DAG, and two separate protocols handle finding that DAG on the network and pulling its blocks across. This is the first in a short series covering each piece of that path in depth (directory sharding, CID encoding, chunking, and content routing each get their own article); this one is the map of how the pieces fit together.

TL;DR:

• IPFS represents every file as a Merkle DAG: a directed graph of blocks where each block's identifier is a hash of its own contents plus the hashes of everything it links to
• Files get split into blocks before hashing, either fixed-size or content-defined, so a change in one part of a file doesn't reshuffle the identifiers of unrelated blocks
• A file's top-level identifier is called a CID (Content Identifier), and it changes if a single byte anywhere in the file changes
• Finding who has a given CID is a separate job (the DHT) from actually transferring the blocks (Bitswap), and this two-step split is why IPFS retrieval looks different from a single HTTP request
• AIOZ Pin runs on top of this exact mechanism; pinning a file keeps its blocks available for the DHT to find and Bitswap to serve, and a premium gateway skips the discovery step entirely by pointing directly at a known host

The Merkle DAG: One Structure for Every File

Every file, folder, and piece of linked data on IPFS is represented the same way: as a Merkle DAG, a directed acyclic graph where each node's identifier is derived by hashing the node's own data together with the identifiers of every node it points to. That second part is what makes it a Merkle structure rather than a plain hash: a node's identifier depends on its entire subtree, not just its own bytes. Change one leaf anywhere in the graph and every node between that leaf and the root gets a new identifier, because each parent's hash includes its children's hashes.

This has a direct consequence: two nodes with identical content always produce the identical identifier, and no two different pieces of content ever produce the same one (short of a hash collision, which is why IPFS uses cryptographic hash functions). A Merkle DAG is not unique to IPFS, git commit history works the same way, but IPFS applies it to arbitrary files and directories instead of just source code changes.

From File to Blocks: Why Chunking Happens First

Before a file becomes a DAG, it gets split into blocks. A small file might be a single block; a large one gets split into many, each under a size limit (256 KB is the common default), and the blocks link together as children of a root node. Splitting happens for two practical reasons: IPFS moves data block by block over the network, so smaller units mean better parallelism and resumability, and deduplication only works at the block level, if two files share a stretch of identical bytes, they can share the block that stretch hashes to, instead of storing the same data twice.

How the split points get chosen (fixed-size boundaries versus content-defined ones that shift with the data) changes how well that deduplication actually works in practice, especially when a file gets edited rather than uploaded fresh. That tradeoff is deep enough to warrant its own article, next in this series.

The CID: What a Merkle DAG's Root Actually Looks Like

The root of a file's Merkle DAG gets its own identifier: the CID (Content Identifier), the string you actually see and share, like bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi. A CID is not a random ID assigned to your upload, it's derived entirely from the content, specifically from the hash of the file's root block. Upload the same file twice, from two different machines, and you get the same CID both times. Change one byte anywhere in the file, and the CID changes completely, because that change propagates up through every parent hash in the DAG.

A CID packs more into itself than just a hash: a version marker, a codec identifier saying how to interpret the linked content, and the hash itself, algorithm and digest both. Unpacking that structure byte by byte is worth its own article; the short version here is that a CID is a self-describing pointer into the DAG, not an opaque string.

Finding Content: The DHT

Once a DAG exists, some peer on the network needs to have its blocks, and other peers need a way to find that peer. That's the job of IPFS's DHT (Distributed Hash Table), a Kademlia-based system where every peer and every piece of content maps to a location in the same address space, derived by hashing the peer's ID or the content's identifier. A peer that has a block "provides" it by publishing a provider record to the DHT peers closest to that block's address. A peer looking for the block queries the same region of the address space and gets back a list of who's holding it. This is a lookup step, not a data-transfer step, its whole job is answering "who has this," in roughly logarithmic time relative to network size.

Moving Content: Bitswap

Once the DHT has answered who has a block, a second protocol, Bitswap, handles actually getting it. Bitswap works off want-lists: a peer announces which blocks it wants, connected peers respond with whether they have each one, and the ones that do send the block itself. If none of a peer's existing connections have what it needs, Bitswap falls back to querying the DHT to find someone who does. Splitting "who has this" from "send it to me" into two protocols is deliberate: it lets IPFS reuse existing connections for blocks a peer didn't know it needed yet, and lets multiple peers serve pieces of the same file in parallel instead of routing everything through one source.

Where AIOZ Pin Fits Into This Mechanism

None of the above is AIOZ-specific; it's the IPFS protocol itself, the same DAG structure, DHT, and Bitswap that any IPFS node runs. What AIOZ Pin adds is a layer on top: it keeps a file's blocks online and replicated so there's always someone for the DHT to find and Bitswap to serve, instead of a file going dark once the one machine that uploaded it goes offline. AIOZ Pin's premium gateways go one step further and skip the discovery step entirely, a request to a dedicated gateway goes directly to infrastructure that already has the content, rather than running a DHT lookup and Bitswap negotiation on every fetch.

Frequently Asked Questions

How does IPFS work in one sentence? A file gets split into blocks, the blocks link into a Merkle DAG identified by a CID, and separate protocols (a DHT for discovery, Bitswap for transfer) find and move that DAG's blocks across the network.

Is IPFS's data structure a B-tree? No. IPFS's core structure is a Merkle DAG, not a B-tree. One specific case, sharding very large directories, uses a different tree structure called a HAMT (Hash Array Mapped Trie), covered in the next article in this series.

Why does IPFS split files into blocks instead of storing them whole? Blocks enable parallel transfer (multiple peers can serve different blocks of the same file at once) and deduplication (identical blocks across different files only need to be stored once).

What's the difference between the DHT and Bitswap? The DHT answers "who has this content" through a lookup; Bitswap handles the actual exchange of blocks once a peer's been found. They're separate protocols solving separate problems.

Does AIOZ Pin change how the underlying IPFS protocol works? No. AIOZ Pin runs standard IPFS mechanics underneath; what it adds is persistence (pinning keeps content available for the network to find) and premium gateways (a direct route that skips DHT/Bitswap discovery for faster retrieval).

Why does changing one byte in a file change its entire CID? Because a Merkle DAG's hashes propagate upward: changing a leaf block changes that block's hash, which changes its parent's hash, all the way to the root, which is what the CID identifies.

Related Reading

This series has grown into 35 articles across five themes. Rather than one long list, here's the map:

Core structure & encoding - How IPFS Shards Large Directories: The HAMT, Not a B-Tree - Anatomy of a CID: Decoding an IPFS Identifier - How IPFS Splits Files: Fixed vs. Content-Defined Chunking - UnixFS: How IPFS Represents Files and Directories - IPLD: The Data Model Underneath IPFS's Merkle DAG - IPLD Selectors: Fetching Part of a DAG, Not All of It

Networking & discovery - How IPFS Finds and Retrieves Files: DHT and Bitswap - libp2p: The Networking Layer Underneath IPFS - Why IPFS Provider Records Expire and Get Renewed - IPFS Bitswap Ledgers: How Peers Decide Who to Trust - IPFS Private Networks: Swarm Keys Explained - GossipSub: How IPFS's PubSub Layer Actually Works - IPFS Transports: TCP vs. QUIC Explained - Delegated Routing: How Light Clients Skip Running a DHT - What Actually Determines IPFS Retrieval Speed - Types of IPFS Nodes: The Different Roles, Explained - Optimistic Provide: How IPFS Made Publishing 10x Faster

Naming, addressing & access - IPFS Content Addressing vs. Location Addressing - IPNS: Mutable Pointers to Immutable IPFS Content - DNSLink: Pointing a Domain Name at IPFS Content - ipfs:// in the Browser: How It Actually Works - Helia: IPFS's Modern JavaScript Implementation - IPFS Immutability: Why You Can't Edit a File in Place - How to Host a Static Website on IPFS (and Why Pin It)

Trust, verification & portability - How IPFS Verifies Content Without Trusting the Gateway - CAR Files: How IPFS Packages a DAG Into One File - Content Moderation on IPFS: What Actually Happens - Does IPFS Encrypt Your Files? Privacy on IPFS Explained - IPFS vs. Filecoin: How the Two Protocols Actually Relate

Lifecycle & operations - IPFS Garbage Collection: What Unpinning Actually Does - How Multi-Node Pinning Services Replicate Content

Where this connects to AIOZ Pin - What Is IPFS? The Ultimate Guide to Decentralized Storage - the concept-level introduction this whole series goes one layer deeper than - How AIOZ Pin Works: Pinning, Gateways, and NFT Tools - how AIOZ Pin's product layer sits on top of everything above - Explore How IPFS Gateway Works Across Decentralized Networks - how a gateway turns the CID resolution and retrieval this series covers into a plain HTTP response - What Happens to Your Files If You Stop Using AIOZ Pin - the practical, product-specific question this series' more abstract mechanics eventually lead to - Real-World IPFS Case Studies: Audius and LikeCoin - two real, documented architectures built on the mechanics this series covers - IPFS Is Rate-Limiting Public Gateways: What Changes for You - a real, current policy shift directly affecting the gateway mechanics this series covers

References

• IPFS Docs: Merkle DAGs
• IPFS Docs: Content Addressing