What it carries
BitTorrent carries an Internet Archive torrent exactly the way it carries every other load it was built for, and the Internet Archive publishes a torrent beside nearly every item in its collection. The protocol was designed for one shape of problem: a single large file that a great many people want inside the same few hours. Operating system images on release day, archived film and audio collections, game and application patches, and research datasets all have that shape, and all of them move through swarms every day. The protocol has no idea what is inside the file it is moving, which is why the same mechanism serves a public library in San Francisco, a volunteer distribution project, and a games company pushing an update, without any of them altering a line of it.
The condition that makes a swarm the right tool
A swarm is the right tool when one large file is wanted by many people at once, and it is the wrong tool for almost everything else. With a single web server, every extra person who starts downloading is extra cost and extra contention: the same finite outbound capacity is divided into thinner and thinner slices. In a swarm the arithmetic inverts. Each file is cut into pieces of a fixed length, often a quarter or half a megabyte on a medium torrent and larger on a very big one, and each piece carries its own 20 byte SHA-1 hash recorded in the torrent's info dictionary. A peer asks for a piece in 16 KiB blocks, and a peer that has verified one piece can hand that piece to a peer that has not, so every arrival brings capacity with it. The words that describe this, seeder, leecher, piece and info hash, have exact meanings, and the Glossary holds them.
Three properties have to line up before the arithmetic pays. The file has to be large enough that the setup cost is worth paying. The audience has to overlap in time, because a peer only helps while it is connected. And the file has to be identical for everybody, since a swarm distributes one fixed set of bytes and cannot personalize anything. Release day for an operating system meets all three at once, which is why that is the case people meet first.
| What moves | Typical size | Why a swarm suits it |
|---|---|---|
| Operating system image | A few gigabytes | Demand collapses into the hours after an announcement, and every copy is identical |
| Archived collection item | Megabytes to many gigabytes | A non-profit with fixed bandwidth is publishing to an unpredictable audience |
| Game or application patch | Tens or hundreds of megabytes | Every player is told to update in the same window |
| Research or machine learning dataset | Tens of gigabytes to terabytes | The file is too large to serve repeatedly, and the audience already has capable machines |
| Server code going out inside a company | Hundreds of megabytes | One build has to reach thousands of machines in minutes |
The Internet Archive and the torrent beside almost every item
The Internet Archive publishes torrents because it is a non-profit library with finite bandwidth and an audience it cannot predict. It has operated since 1996, it is based in San Francisco, and it holds web captures, books, audio, film, software and concert recordings. When an item is added to the collection, the Archive generates a torrent for that item's files and stores the torrent as one more file inside the item. Access costs nothing to the reader; the organization is funded by donations, grants and digitization work it does for other libraries.
Two details make an Archive torrent behave differently from an abandoned one. The Archive seeds from its own servers, so a swarm with no other peers in it is still a working transfer. And the torrent carries a web seed, an ordinary HTTP location for the same data, which lets a client fall back to pulling a missing piece straight from the Archive's storage when no peer has it. The practical result is that these torrents rarely go dead the way a torrent with a vanished last seeder does.
Operating system images on release day
Operating system images are the clearest case there is. A desktop image of a few gigabytes goes out on a fixed day to an audience that has been waiting for it, and the project publishing it is often funded by donation. Ubuntu, Debian, Linux Mint in its Cinnamon and other editions, Arch Linux and Fedora all publish torrents next to their direct downloads, and the 64 bit image is the one nearly everybody takes. A small image is a different case: an Alpine Linux install medium measured in tens of megabytes rather than gigabytes is cheap enough to serve directly, and the swarm adds coordination without adding much value.
The image page is also where the second half of the ritual lives. A project publishes a checksum for each image, and verifying it is a separate act from anything the protocol does. On a Linux desktop the client itself usually comes from the distribution's own repository rather than from a download page, which is one of the ways clients on Linux behave unlike clients elsewhere.
Patches, updates and code going out to servers
Game and application patches use swarms for the same reason release days do: the publisher tells every installation to update, and the update window is narrow. Blizzard's own updater distributed World of Warcraft patches through a peer to peer distributor for years before the company moved to a delivery system of its own. Inside companies the pattern repeats at a smaller scale and a higher speed. Twitter built and released a tool called Murder that pushed a new build of its code out to a large fleet of servers as a swarm, because a thousand machines pulling from one deployment host is exactly the contention problem the protocol solves.
Nothing about that use is exotic. The machines are peers on a private network, the pieces are verified against the same hashes, and the operators simply wanted the arithmetic of a swarm rather than the arithmetic of a server.
Research data, where the file itself is the problem
Research datasets and machine learning corpora reach a size where ordinary distribution stops working. A sky survey, a genome collection, a text corpus or a set of model weights can run to hundreds of gigabytes or terabytes, and the institution holding it usually cannot pay to serve it repeatedly to everybody who cites the paper. A swarm shifts that cost to the people who already have the storage and the network to handle the file. Per-piece verification matters more here than anywhere else: a corrupted byte in a research archive is not an inconvenience, it is a wrong result, and the client discards and refetches any piece whose hash does not match before it is ever written as final.
What a swarm carries badly
A swarm carries small, private or constantly changing data badly, and knowing that is half of understanding what it is for. A torrent describes one fixed set of bytes: change a single file and the info hash changes, which makes it a different torrent. It also has no notion of privacy, because every peer in a swarm learns the addresses of the peers it talks to. That is the design, not a defect.
- Small files gain nothing, since the coordination costs more than the transfer.
- Files wanted by two or three people gain nothing, since there is no swarm to form.
- Files that keep changing break the model, since each version is a separate torrent.
- Private data is the wrong fit, since membership of a swarm is visible to its members.
- Files nobody has kept simply stop, since a swarm with no complete copy left can never supply the missing pieces.
The peer to peer world has other tools for the cases the protocol handles badly. Syncthing keeps a folder identical across devices that belong to the same person and are known to each other in advance, which is a different problem with a different answer. And the reason a swarm looks the way it does at all is historical: the centrally indexed networks of the Napster era put one company in the middle of every lookup, and everything built afterward was shaped by what happened to that arrangement.