You can create a robots.txt file to provide instructions to search engine crawlers regarding which parts of your website they should or should not index. This guide explains how to create a basic, safe robots.txt file for your website to ensure your site remains accessible to search engines while excluding specific directories if desired.
Before you begin
You will need:
Access to your website's root directory via an FTP client or a web hosting control panel file manager.
A plain text editor (such as Notepad, TextEdit, or VS Code).
Create and configure the robots.txt file
Open a plain text editor on your computer.
Create a new file and enter the following basic configuration:
User-agent: * Allow: /
Save the file exactly as
robots.txt(ensure the file extension is.txtand not.txt.txt).Upload the
robots.txtfile to the root directory of your website. This is typically thepublic_htmlorwwwfolder.Verify the file is accessible by navigating to
https://example.com/robots.txtin your web browser.
Warning: A common mistake is using the Disallow directive incorrectly. For example, entering Disallow: / will tell search engines not to crawl any part of your site, which will remove your website from search results.
Confirm the result
Navigate to your website's root URL followed by
/robots.txt.The browser should display the plain text content you entered in step 2.
The site remains accessible to the public, and search engines can still crawl the primary content.
Limitations
This procedure only creates a basic "Allow All" file. If you need to block specific folders (such as a
/tmp/or/admin/directory), you must replace theAllow: /line with specificDisallowrules.Changes to
robots.txtmay take some time to be processed by search engine crawlers.This file does not prevent a determined user from visiting a page; it only requests that automated crawlers do not index specific areas.
