What is meta-externalfetcher?

Fetches specific URLs on behalf of Meta AI user requests. It is operated by Meta.

Does meta-externalfetcher respect robots.txt?

Yes. Meta states that meta-externalfetcher honours robots.txt directives, so a Disallow rule for the meta-externalfetcher token is sufficient.

AI agent

MetaHonours robots.txt

Fetches specific URLs on behalf of Meta AI user requests. Blocking prevents Meta AI from reading pages users ask about.

meta-externalfetcher/1.1

How to verify it

Meta published IP ranges.

What blocking costs

Blocking prevents Meta AI from reading pages users ask about.

# Block meta-externalfetcher
User-agent: meta-externalfetcher
Disallow: /

# Allow everything else
User-agent: *
Allow: /

Robots.txt is enough

Meta documents meta-externalfetcher as honouring robots.txt. Verify with your logs after deploying — a spoofed agent will keep coming.

Generate a full robots.txt

Pick which AI crawlers to allow or block across all 42 agents and copy the file out.

Other Meta agents