SEO Robots.txt Basics for New Sites
SEO robots.txt is a plain text file at the root of your site that tells crawlers which paths they may fetch. For a new website, a careful robots.txt protects private areas without hiding the pages you want ranked.
Many beginners copy a strict template and accidentally block CSS, images, or whole sections. Crawlers then see a broken or empty site. Treat robots.txt as a gate with labels, not a trash can for everything you dislike.
What robots.txt can and cannot do
It can reduce crawl on admin folders, staging leftovers, or duplicate parameter paths. It cannot reliably hide a secret page from humans who know the URL, and it is not a substitute for login protection. SEO robots.txt is guidance for bots, not a security vault.
Beginner-safe habits
- Allow your public pages, posts, and important media by default
- Disallow only areas that waste crawl or expose unfinished drafts
- Include a Sitemap: line pointing to your XML sitemap when ready
- Test changes after deploy, not only on localhost
Common beginner mistake
Blocking /wp-admin/ is fine on many WordPress sites, but blocking /wp-content/ or entire themes can break how search engines render pages. If rankings look odd after a robots change, reverse the last edit first.
Keep it boring
A short, readable robots.txt is a technical SEO win. Review it whenever you add a shop, membership area, or staging subdomain. Document why each Disallow exists so future-you does not guess.
Good SEO robots.txt work is quiet: crawlers reach the right URLs, and you stop inventing rules you cannot explain.
Put it into practice
Open your live site at /robots.txt. Confirm important folders are not Disallowed by mistake, and note whether a Sitemap line points to your XML sitemap. Fix accidental blocks before you publish more content.
Write the change, the date, and what you expect to see in search or on the page. Beginners who keep a short SEO notebook learn faster than beginners who only react when rankings drop. One deliberate fix beats five random tweaks.
If the result is unclear after a week, do not scrap the whole site plan. Adjust the single variable you changed, or give the page more time while you improve a different URL. Steady SEO learning compounds when you refuse to thrash. Share the note with a teammate when you have one so the next change stays deliberate too.
- One SEO change, one week, one written expectation
- Check the page on a phone before you call it done
- Prefer clear evidence over hunches when you review results
- Capture a screenshot of titles or reports when you can
Next steps
If you want help running ads and growing your site, check out our marketing packages in the shop. Prefer to talk it through first? Contact us and tell us what you are building.
If you want to become the expert yourself, order SEO Fundamentals: Optimizing for Search Engines. Paperback copies are available worldwide from about $29.99 delivered, in the UK from about ?24.99 delivered, and in Europe typically under ?30 delivered.