Wayback Machine Alternatives: Which Websites Block the Archive? We Tested 190 Sites

1. Why websites are blocking the Internet Archive

Since late 2025, news publishers have been shutting the Internet Archive's crawlers out of their sites. Their stated worry is AI training: if the Wayback Machine holds a full copy of every article, an AI company could take the articles from the archive instead of from the publisher. In May 2026, Nieman Lab counted 382 news sites limiting the archive's access through robots.txt, 342 of them local outlets, up from 241 sites in January.

The evidence for that worry is thin. Nieman Lab reported in May 2026 that no publisher it contacted had confirmed that an AI company had already scraped its content from the Wayback Machine. Mark Graham, who directs the Wayback Machine, calls the archive "collateral damage caught up in the conflict between AI companies and publishers."

Coverage so far has counted robots.txt files. Nobody had checked what the blocks actually do to the archive, which is what matters to anyone who expects to find an old article there. So we measured it.

2. How we tested 190 websites

We picked 190 large websites in eight groups: US national news (30); US local news (30); international news (30); social and user-generated content platforms (20); ecommerce (20); reviews, travel and jobs (20); government (20); and SaaS platforms (20). For each one we checked three things on November 7 and 8, 2026:

1. The robots.txt file. Does it disallow the whole site for a crawler name that Nieman Lab tracks as the Internet Archive's: archive.org_bot, heritrix (including its named variants), archive-it or special_archiver? We counted rules naming only ia_archiver separately, because Mark Graham told Nieman Lab in May 2026 that the Wayback Machine does not use that name.
2. The homepage. Is there a snapshot from the last 30 days, and does it replay?
3. Pages inside the site. We took up to three article or product links from each site's homepage as the Wayback Machine saved it in early September 2026, so every page had at least two months to be archived. For each one we read every record in the Wayback Machine's index and replayed the copy a visitor would get.

That gave 436 pages on 148 sites. A page counts as having a readable copy when the archive replays the page itself, not an error or a paywall notice. The full method and its limits are in the methodology section, and the site-by-site results are in the downloadable CSV.

3. Which websites block the Wayback Machine

On November 7, 2026, we could read the robots.txt of 162 of the 190 sites. Of those, 29 disallow an Internet Archive crawler, and 28 of them are news sites. The other is the World Health Organization. Ten more sites disallow all crawlers by default with a wildcard rule, which covers the archive along with everyone else: Reddit, X, Facebook, LinkedIn, Pinterest, Yelp, Trustpilot, Reuters, The Wall Street Journal and Netflix.

The 29 sites that block an Internet Archive crawler: al.com, The Atlantic, azcentral.com, The Baltimore Sun, Chicago Tribune, cleveland.com, CNN, Daily Mail, The Denver Post, Detroit Free Press, El Mundo, El País, IndyStar, Le Monde, MassLive, MLive, NBC News, The New York Times, The New Yorker, NJ.com, OregonLive, PennLive, la Repubblica, Der Spiegel, syracuse.com, USA Today, The Washington Post, Wired and the World Health Organization. The crawler named most often is archive.org_bot (27 sites), followed by heritrix (15).

We could not read the robots.txt file of 28 sites, most of which refused our requests. They include the Miami Herald, The Sacramento Bee, The Kansas City Star and The Charlotte Observer. On the eight news sites in this group, only 7 of 19 sampled articles had a readable copy, so the true number of blocking news sites in our sample may be higher.

4. Homepages stay archived, articles do not

If you check a blocking site in the Wayback Machine, it looks fine: 27 of the 29 blocking sites have a homepage snapshot from the last 30 days, and the median blocking site gained 1,551 new homepage snapshots between August 1 and November 7, 2026. The gap only appears one click deeper.

The archive also visits blocked articles far less often. In our sample, a typical page on a site with no block had 57 successful captures by November 2026, and a typical page on a blocking site had 2.

In our November 2026 sample, of the 28 pages on blocking sites with no readable copy, 23 had been visited by the archive and stored as an error response. The other five had no record at all. Sites with no block lost eight pages, all to error or paywall responses: Bloomberg and the South China Morning Post answered with 403 Forbidden, and The Telegraph with 402 Payment Required.

5. Same robots.txt rule, very different results

The averages hide the real pattern. Sites with nearly identical robots.txt rules end up in completely different places:

• Eight Advance Local newspapers (al.com, cleveland.com, MassLive, MLive, NJ.com, OregonLive, PennLive and syracuse.com, all Advance Local brands): 1 readable article out of 24. The archive visited the other 23, and for every one of them it stored a 403 Forbidden response. The most recent homepage snapshot of seven of the eight is dated October 17, 2026, three weeks before our test.
• The six other US local news sites that block (The Baltimore Sun, Chicago Tribune, The Denver Post, Detroit Free Press, azcentral.com and IndyStar): 14 readable articles out of 17.
• US national news that blocks, including The Washington Post, USA Today, The New Yorker and The New York Times: 18 out of 20.
• International news that blocks (Der Spiegel, El País, El Mundo and la Repubblica): 12 out of 12.

Why the robots.txt rule does not decide the outcome

So the robots.txt line correlates with a lower archiving rate, but it does not decide the outcome on its own. Where articles disappear, the record shows why: the site answered the archive with HTTP 403. That is a server refusing the request, which is a separate step from a robots.txt rule. The New York Times told Nieman Lab in January that it was "blocking the Internet Archive's bot," and the one Times interactive in our sample with no readable copy shows the same pattern: 10 visits by the archive, none of which produced a readable copy.

We cannot see when each block was added, and a robots.txt file read in November says nothing about September. What the data does show is that a blocking site can still leave a public, archived trail of its articles, and that the publishers who actually make articles disappear do it at the server.

6. What you see when a page is missing

The Wayback Machine has several ways of saying no, and they mean different things:

• An archived 403, 401 or 402 page. The archive tried, and the site refused or demanded payment. This is what most missing articles in our sample look like. The calendar can still mark these dates as captures, so open the snapshot before you rely on it.
• "The Wayback Machine has not archived that URL." No visit is on record. Five pages on blocking sites and 12 on news sites whose robots.txt we could not read ended here.
• "This URL has been excluded from the Wayback Machine." The page or site has been removed from public view. The Internet Archive explains that this happens because of a robots.txt file or at a site owner's direct request. We saw it on the homepages of Yelp, Booking.com and Quora, not on any news site.
• A search that returns nothing, while the page still opens. The archive's lookup service refused to answer for The Guardian and NBC News, yet both homepages replay, and two Guardian articles from September 2026 that we tested opened normally from their exact addresses. The Guardian told Nieman Lab it had excluded itself from the archive's APIs and filtered its article pages out of the URL listing. If you know the exact URL, try it directly.

Old snapshots of blocking sites survived

One reassurance: the blocks we saw did not erase old snapshots. Of the 29 blocking sites, 28 still have homepage snapshots from 2024, and every one we were able to replay showed the page.

7. Wayback Machine alternatives, by task

If you are looking for sites like the Wayback Machine, our data points to one limit they share: every crawler-based archive depends on the site letting its crawler in. There is no single replacement, because people use the Wayback Machine for different tasks. Pick the tool that fits yours:

Two practical habits work with any of them. Save the page as soon as you know it matters: on the Advance Local sites the archive tried within a day or two of publication, was refused, and never got a readable copy later. And for anything important, check what was saved by opening it, since an archive can hold a 403 page under the right address.