Skip to main content

How to Create a robots.txt File Without Blocking the Entire Site

You can create a robots.txt file to provide instructions to search engine crawlers regarding which parts of your website they should or should not index. This guide explains how to create a basic, safe robots.txt file for your website to ensure your...

You can create a robots.txt file to provide instructions to search engine crawlers regarding which parts of your website they should or should not index. This guide explains how to create a basic, safe robots.txt file for your website to ensure your site remains accessible to search engines while excluding specific directories if desired.

Before you begin

You will need:

  • Access to your website's root directory via an FTP client or a web hosting control panel file manager.

  • A plain text editor (such as Notepad, TextEdit, or VS Code).

Create and configure the robots.txt file

  1. Open a plain text editor on your computer.

  2. Create a new file and enter the following basic configuration:

    User-agent: *
    Allow: /

  3. Save the file exactly as robots.txt (ensure the file extension is .txt and not .txt.txt).

  4. Upload the robots.txt file to the root directory of your website. This is typically the public_html or www folder.

  5. Verify the file is accessible by navigating to https://example.com/robots.txt in your web browser.

Warning: A common mistake is using the Disallow directive incorrectly. For example, entering Disallow: / will tell search engines not to crawl any part of your site, which will remove your website from search results.

Confirm the result

  • Navigate to your website's root URL followed by /robots.txt.

  • The browser should display the plain text content you entered in step 2.

  • The site remains accessible to the public, and search engines can still crawl the primary content.

Limitations

  • This procedure only creates a basic "Allow All" file. If you need to block specific folders (such as a /tmp/ or /admin/ directory), you must replace the Allow: / line with specific Disallow rules.

  • Changes to robots.txt may take some time to be processed by search engine crawlers.

  • This file does not prevent a determined user from visiting a page; it only requests that automated crawlers do not index specific areas.

Did this answer your question?