Technical SEO · 4 min read
Robots.txt Basics: Control Crawling Without Blocking Your Site
A robots.txt file gives crawler instructions about which paths they should request. It is not a security feature and it does not guarantee that a page will be excluded from search.
Keep crawl rules narrow and intentional
A small public site usually needs only a simple file that allows important pages and identifies the sitemap. Add disallow rules only when there is a clear crawl-management reason and the exact pattern is understood.
Do not block CSS, scripts, or pages that Google needs to render. A broad rule can prevent crawlers from fetching the website resources used to understand a page.
- Keep directives minimal.
- Do not block resources needed for rendering.
- Review every rule before publishing.
Test the live file
Check that the file returns a successful response with a plain-text content type and that the sitemap line points to the correct canonical host. Search Console's URL Inspection can help diagnose crawl problems, but a robots file is only one possible cause.
Do not use robots.txt to protect private content. Use real access controls for anything that should not be public.
- Check the live response and sitemap host.
- Test important URLs after changing rules.
- Use authentication for private content.
Common questions
Does disallowing a URL remove it from Google's index?
Not necessarily. A blocked URL may still be known from links. Use the correct page-level indexing controls and follow current Google guidance.
Plan your next step
Explore the sample builder, review website packages, or contact Ridgeline Web Co. about project fit.