What is ia_archiver / Internet Archive?

Preserves pages in the Wayback Machine. It is operated by Internet Archive.

Does ia_archiver / Internet Archive respect robots.txt?

No. ia_archiver / Internet Archive is documented as not governed by robots.txt, so blocking requires a rule at your CDN or web server.

← AI crawler directory

ia_archiver / Internet Archive

Internet ArchiveIgnores robots.txt

Preserves pages in the Wayback Machine. The Archive stopped honouring robots.txt for archival in 2017. Removal requests go through the Archive directly.

User-agent string

Mozilla/5.0 (compatible; archive.org_bot; +http://www.archive.org/details/archive.org_bot)

How to verify it

Reverse DNS to archive.org.

What blocking costs

The Archive stopped honouring robots.txt for archival in 2017. Removal requests go through the Archive directly.

robots.txt rule

# Block ia_archiver / Internet Archive
User-agent: archive.org_bot
Disallow: /

# Allow everything else
User-agent: *
Allow: /

Robots.txt is not enough

ia_archiver / Internet Archive is documented as not governed by robots.txt. Enforce at the CDN or web server, and verify the request genuinely originates from Internet Archive before blocking by string alone.

Generate a full robots.txt

Pick which AI crawlers to allow or block across all 42 agents and copy the file out.

Open the generator →