{"id":15785,"date":"2025-11-27T10:00:32","date_gmt":"2025-11-27T10:00:32","guid":{"rendered":"https:\/\/www.wscubetech.com\/blog\/?p=15785"},"modified":"2026-01-20T10:06:34","modified_gmt":"2026-01-20T10:06:34","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/www.wscubetech.com\/blog\/robots-txt\/","title":{"rendered":"What is Robots.txt File & How to Create it? Sample & Examples"},"content":{"rendered":"\n

If you want search engines to crawl your website correctly, the robots txt file becomes one of the first things you must set up. We use the robots txt file in SEO<\/a> to tell Google which pages you want them to access and which ones they should avoid. <\/p>\n\n\n\n

If you have a website, then you need to know that every unnecessary crawl wastes time and crawl budget. That\u2019s exactly where robots.txt helps. <\/p>\n\n\n\n

It controls what search engines can explore, protects private sections like admin folders, and ensures Google reads your site efficiently. A simple text file can completely change how your site appears in search results.<\/p>\n\n\n\n

What is Robots.txt File?<\/h2>\n\n\n\n

A robots.txt file is a plain text file placed in your website\u2019s root folder. It guides search engine bots on which pages they can crawl and which they must ignore. From an SEO point of view, it helps you manage crawl budget, prevent indexing of unwanted URLs, and keep private or duplicate sections out of search results. <\/p>\n\n\n\n

Robots.txt is not a security tool. It only offers instructions to bots. But when you use it correctly, you give search engines a clean path to follow, improving your website\u2019s technical SEO, crawl efficiency, and overall online visibility.<\/p>\n\n\n\n

Robots.txt File Sample<\/h2>\n\n\n\n

This is a sample robots txt file:<\/p>\n\n\n\n

User-agent: *\nDisallow: \/admin\/\nAllow: \/\n\nSitemap: https:\/\/example.com\/sitemap.xml<\/code><\/pre>\n\n\n\n
These are the components of robots txt file with meaning:<\/strong><\/p>\n\n\n\n
Component<\/strong><\/td>Description<\/strong><\/td><\/tr>
User-agent<\/td>Defines which bot the rule applies to (e.g., Googlebot, Bingbot, all bots using *).<\/td><\/tr>
Disallow<\/td>Tells bots which pages or directories they must not crawl.<\/td><\/tr>
Allow<\/td>Specifies exceptions inside a disallowed section that bots can still crawl.<\/td><\/tr>
Sitemap<\/td>Provides the link to the XML sitemap for better crawling and indexing.<\/td><\/tr>
Wildcard (*)<\/td>Used to target multiple URLs or patterns.<\/td><\/tr>
$ Symbol<\/td>Means \u201cend of URL\u201d \u2014 useful for blocking specific file types.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
How Robots.txt File Works?<\/h2>\n\n\n\n
1. Search engine bots visit your domain first<\/h3>\n\n\n\n
Before crawling anything, Googlebot or Bingbot visits yourwebsite.com\/robots.txt. This is the first file they read. If it exists, bots follow the rules; if not, they assume everything is allowed.<\/p>\n\n\n\n
2. Bots identify who the rule is for (User-Agent)<\/h3>\n\n\n\n
The file starts with \u201cUser-agent\u201d. This tells which bot the rule applies to.<\/p>\n\n\n\n
Example: <\/strong><\/p>\n\n\n\n
User-agent: Googlebot <\/em>controls only Google\u2019s main crawler.<\/p>\n\n\n\n
3. Bots read what is blocked (Disallow)<\/h3>\n\n\n\n
The \u201cDisallow\u201d directive tells bots which pages or folders they should not crawl.<\/p>\n\n\n\n
Example: <\/strong><\/p>\n\n\n\n
Disallow: \/admin\/<\/em> blocks all admin pages from crawling.<\/p>\n\n\n\n
4. Bots read what is allowed (Allow)<\/h3>\n\n\n\n
If there are exceptions inside a blocked folder, the \u201cAllow\u201d directive clarifies what bots can crawl.<\/p>\n\n\n\n
Example: <\/strong><\/p>\n\n\n\n
Allow: \/admin\/help-page\/<\/em><\/p>\n\n\n\n
5. Bots follow hints about site structure (Sitemap)<\/h3>\n\n\n\n
A sitemap URL placed in robots.txt helps bots quickly find all important pages to crawl.<\/p>\n\n\n\n
Example: <\/strong><\/p>\n\n\n\n
Sitemap: https:\/\/example.com\/sitemap.xml<\/em><\/p>\n\n\n\n
6. Bots follow rules but are not forced<\/h3>\n\n\n\n
Robots.txt is a guideline, not a law. Good bots follow it; harmful or unknown bots may ignore it. But for SEO, Google and all major crawlers fully respect it.<\/p>\n\n\n\n    \n