PiecewiseHow a peer to peer transfer actually works

The Internet Archive and its torrents

The Internet Archive

Shelves: a collection that lends rather than sells.

Yes, the Internet Archive is safe to download from in the ordinary sense of the word, and the torrent it publishes beside nearly every item is a plain torrent that any client treats like any other: it is a real non-profit library, it has operated since 1996 from San Francisco, and it puts its own name on everything it holds. The safety question worth asking is not about the Archive and not about the protocol, but about what an individual item is and where it came from. The Archive holds material uploaded by the public alongside material it digitized itself, so a file inside an item deserves the same care as any file from a stranger, while the organization serving it is about as accountable as an organization on the internet gets.

What the Internet Archive is, and who pays for it

The Internet Archive is a digital library that captures web pages and preserves books, audio, film, software and live concert recordings. Its best known service is the Wayback Machine, which stores snapshots of public web pages over time. Reading and downloading cost nothing. The organization is funded by donations, by grants and by digitization work it carries out for other libraries, which is the answer to the recurring question of whether the Archive is free: it is free to the reader and paid for elsewhere.

Why nearly every item also has a torrent

Nearly every item has a torrent because the Archive is a single non-profit with finite outbound bandwidth serving an audience it cannot predict. When an item is created, the Archive generates a torrent describing that item's files and stores the torrent inside the item as one more file. A swarm turns each person downloading a large item into part of the delivery for the next, which is why a library reaches for it rather than buying more capacity.

Two properties make these torrents unusually reliable. The Archive seeds from its own servers, so the swarm has a permanent complete copy in it rather than depending on whoever happened to stay online. And each torrent carries a web seed, an ordinary HTTP location for the same data, which lets a client request a missing piece directly from the Archive's storage when no peer has it. A torrent that would otherwise be dead therefore keeps working.

Opening an Archive torrent in a client

To open an Archive torrent, take the torrent file from the item's own file list and add it to a client such as qBittorrent, Transmission or Deluge, which is the same operation as adding any other torrent. The client reads the info dictionary, derives the 20 byte info hash that identifies the item, shows the file list and the total size, and starts asking the tracker named inside for peers. Nothing about the Archive requires a different client or a different setting, and the choice between programs is a matter of what each one is like to run rather than what it can fetch, which is the subject of the clients section.

Read the file list in the dialog before starting: an item can contain derived formats the Archive produced automatically beside the original upload, so selecting only what is wanted saves a great deal of downloading.

Public domain, open licenses, and everything in between

No, the Internet Archive is not all public domain, and treating it as though it were is the most common misreading of what the collection is. The Archive is a mixed library. Some items are public domain, meaning copyright has expired or never applied. Some carry an open license chosen by the person who uploaded them. Some are made available under arrangements specific to a program or a collection. And public domain is not one global status: the term of copyright differs by country and by the kind of work, so a recording that is out of copyright in one place is not necessarily out of copyright in another. Item pages carry whatever rights statement the uploader or the Archive supplied, and that statement, not the presence of a torrent, is what describes the item.

Where the Archive stands legally

The Internet Archive is a lawfully operating non-profit registered as a library, and the torrents it publishes are a distribution method, not a legal status. Some of its programs have been challenged in court by rightsholders, and those disputes have concerned particular collections and particular ways of lending rather than the protocol or the existence of the torrents. Copyright law differs by country and by case, the situation for any given collection can change, and none of this is advice to any reader about any particular item.

When the Archive will not load, or a torrent will not start

The Archive is one organization running its own infrastructure, so it is a single point of failure in a way a commercial content network is not. When the site is not loading, the cause is usually on that side rather than on the reader's, and there is no alternative mirror of the whole collection to fall back on. A transfer that will not move has its own separate set of causes.

The site is unreachable
Maintenance, heavy load or an attack on the Archive's own infrastructure will take the whole domain offline, including item pages and the web seeds inside torrents.
The transfer is stalled with peers connected
The client is connected but nothing is arriving, which on an Archive torrent usually means the web seed is unavailable and no other peer has the pieces being requested.
The tracker is not answering
The client keeps its existing peers and retries on a schedule, so a tracker that has stopped responding slows discovery of new peers rather than stopping a transfer that is already moving.
Files show as missing
An item can be modified after its torrent was generated, and a torrent describes one fixed set of bytes, so a file that no longer matches will not verify against the piece hashes it was built from.

Terms like web seed, tracker, piece and info hash are set out in the Glossary. It is also worth noticing what the Archive is not doing here: it is publishing one fixed public copy to anyone who wants it, which is a different problem from keeping a private folder identical across a handful of machines, the job Syncthing exists to do. And the reason a library can publish this way at all is that the protocol, first described in 2001, carries no directory and no central index, unlike the Napster generation of file sharing that began in 1999, where the lookup and the company running it were the same thing.

Where to go next