Home / Glossary / Robots.txt Robots.txt A file that tells search engine crawlers which pages to crawl and index. Important for crawl budget management. Explore More SEO Terms Browse our comprehensive SEO glossaryto learn more about search engine optimization, backlinks, and digital marketing terms.