A robots.txt file is a plain text file placed at the root of a website that gives instructions to search engine crawlers. It lists which parts of the site they are welcome to visit and which they should skip. Every well-behaved crawler reads it before it starts crawling your pages.
Think of it like a sign at a building’s entrance saying which floors are open to visitors and which are staff-only. The sign does not lock any doors, it just sets expectations, and honest visitors respect it. Google and Bing follow robots.txt by convention, but it is guidance, not a security wall: a page blocked here can still show up in results if it is linked elsewhere. The file often points crawlers to your sitemap so they can find every page you do want indexed. The syntax is short: a User-agent line names the crawler, then Disallow lines list the paths to skip, like an admin folder or a checkout step.
This is also where mistakes get expensive. A single misplaced rule can tell every crawler to ignore the whole site, quietly wiping it out of search. The classic disaster is shipping a staging file that still says Disallow slash, which blocks the entire site on launch day. So treat robots.txt with the same care as a redirect map. Check it after every launch, view it live at the domain root, and never use it to hide private data, since the file itself is public and anyone can read which paths you tried to keep quiet.
At TopDevs we review the robots.txt on every site we ship, so the right pages get crawled and nothing important gets blocked by accident.
