Spider Simulator

See a page as a basic crawler does: status, robots, headings, links and text.

Ready

Page title crawlers read

Enter a URL to start

Status —

Indexing —

Words

—

Links

—

H1 tags

—

URL
Waiting for input
The address you entered
Response
—
Status code and redirects
Crawl rules
—
What robots.txt allows
Index signals
—
noindex and canonical

Crawl report

Enter a page URL above to see the full report: response and redirects, robots rules, canonical, metadata, headings outline, every link and the text crawlers can read.

Fair-use limit: 20 crawls per hour and 100 per day for each visitor IP address. Pages larger than 2 MB are read up to the first 2 MB.