I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least.

That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals.

Cheers.

Recursive scraping:

1) can consume a lot of storage really fast

2) makes IA a bot rather than a user service

3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it.

I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.

https://wiki.archiveteam.org/index.php/ArchiveBot is available if you would like to help save things that are still up.

I never got turned down sending an IRC message "x is going down in a few days" to the AT channels. https://wiki.archiveteam.org/index.php/Archiveteam:IRC

AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.

Pretty much. Your idea of getting coverage could probably work well to make sure IA doesn't "turn into a bot" (though it's practically the same, just without the permanent downloading).

What’s recursive scraping (presumably, following internal links), and what’s broken about it?

Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.

Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache?

I know about the web.archive.org/https://... "hack" but it's probably draining your servers unnecessarily when there are a lot of 301 redirects that weren't archived and on top of that are only able to be validated client side after receiving the whole HTML response.

A REST API would help thirdparty clients to know about this in advance, and the API documentations I found were super unclear in whether something like this exists or not.

Context: I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it.

On top of that I have no idea how to "un-UI" the web archived versions. I know that the wget user agent somehow leads to this, but it's also kind of undocumented how the webserver of IA does this in the background and when exactly the UI is injected and all the URLs are rewritten. Something like maybe a http request header to get the raw actual source would be nice.

> Does the wayback machine offer a JSON/REST API to check for URLs, in the sense of sending a URL and getting back a map of specific crawl datetimes that are available in the cache?

Have you looked at the headers that they send?

  GET /web/20100330210402/https://arxiv.org/abs/0911.1112 HTTP/2
  [...]

  HTTP/2 200 OK
  [...]
  link: <http://arxiv.org/abs/0911.1112>; rel="original",
    <https://web.archive.org/web/timemap/link/http://arxiv.org/abs/0911.1112>;
    rel="timemap"; type="application/link-format",
    <https://web.archive.org/web/http://arxiv.org/abs/0911.1112>;
    rel="timegate",
    <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>;
    rel="first memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
    <https://web.archive.org/web/20100330210402/http://arxiv.org/abs/0911.1112>;
    rel="memento"; datetime="Tue, 30 Mar 2010 21:04:02 GMT",
    <https://web.archive.org/web/20110101203756/http://arxiv.org/abs/0911.1112>;
    rel="next memento"; datetime="Sat, 01 Jan 2011 20:37:56 GMT",
    <https://web.archive.org/web/20211123040625/https://arxiv.org/abs/0911.1112>;
    rel="last memento"; datetime="Tue, 23 Nov 2021 04:06:25 GMT"
  [...]

The Wayback Machine implements RFC 7089. <http://mementoweb.org/guide/quick-intro/>

Regarding your last point, adding "id_" to the end of the timestamp in the URL produces the original downloaded file. Adding "if_" produces the page with rewritten links but without the Wayback Machine header. Additionally, if you look at the HTTP response headers on any version of the archived page, they include copies of the original response headers.

> I'm building a web browser and I'm trying to offer a feature for error cases when the server or URL isn't available anymore, so that users can see the web archived version of it.

Brave browser does that, and I've seen Firefox add-ons that do it. Maybe you can look at their code.

Thank you for your service!

Just wanted to say thank you for all the times you've let me continue researching when the original sources became dead!

Thank you!