Pick a profile that matches the question
Standard is the normal baseline. Fast works for a small release check. Deep explores more click levels at a gentler pace. The launcher also controls the URL limit, concurrency, sitemap seed, subdomains, and query cleanup.
Watch the stages
Discovery and crawling increase the fetched and queued counters. Persistence writes the collected evidence as a consistent snapshot. Large sites can remain in persistence for several minutes even when the visible fetch count stops moving. The run will either complete or expose a failure state.
Read the crawl log before the reports
The crawl log records duration, response code distribution, and whether the URL allowance stopped discovery. This prevents a partial site sample from being mistaken for full coverage.
Respect ownership and crawl policy
Regular audits follow robots.txt. Verified ownership can unlock the deliberate control needed for an owner-authorized test that ignores robots rules. Major platforms and other unsafe crawl targets cannot be added as projects.