By GrowthSyntax Editorial  ·  Updated July 2026  ·  8 min read

How to create a robots.txt file? Open a plain text editor, add a few simple rules using “User-agent” and “Disallow,” save the file as robots.txt, and upload it to your site’s root folder. The whole process takes a few minutes once you understand the basic syntax, and no coding experience is required.

5 min

Typical time it takes to create a robots.txt file from scratch

/robots.txt

Exact file name and location search engines look for on your domain

Free

Cost to build one instantly with a robots.txt generator toolTable of Contents

1. What Is a Robots.txt File?
2. Robots.txt Syntax Explained
3. How to Create a Robots.txt File Step by Step
4. A Working Robots.txt Example
5. Where to Upload Your Robots.txt File
6. Common Robots.txt Mistakes to Avoid
7. Robots.txt vs the Meta Robots Tag
8. How to Test Your Robots.txt File
9. Frequently Asked Questions

What Is a Robots.txt File?

A robots.txt file is a plain text file that tells search engine crawlers which pages they can visit. It sits in the root folder of your website and loads before any crawler explores your pages. Every major search engine checks this file first.

The file does not hide pages from people. It only guides automated crawlers, so anyone with a direct link can still open a blocked page. For true privacy, use a password or a noindex tag instead.

Most websites need a robots.txt file for one main reason. It stops crawlers from wasting time on admin pages, duplicate content, or scripts that offer no SEO value. That saved crawl budget goes toward pages that actually matter.

Large sites benefit the most from a well-planned file. Thousands of pages can waste significant crawl budget on filters, tags, or session URLs. A smaller site may only need a handful of simple rules.

Robots.txt Syntax Explained

Robots.txt uses a small set of directives, and each one plays a specific role. Learning how to create a robots.txt file starts with understanding these four terms. The table below breaks each one down.

An asterisk after “User-agent” applies the rule to every crawler. A single forward slash after “Disallow” blocks the entire site, which is rarely what you want. Always double-check that slash before saving the file.

How to Create a Robots.txt File Step by Step

Here is exactly how to create a robots.txt file from scratch, without needing any special software.

1. Open a plain text editor. Notepad, TextEdit, or any code editor works fine. Avoid Word processors, since they can add hidden formatting.
2. Write your rules. Start with “User-agent,” then add “Disallow” or “Allow” lines underneath it for each rule you need.
3. Add your sitemap link. Include a “Sitemap” line pointing to your full sitemap URL, so crawlers can find it quickly.
4. Save the file correctly. Name it exactly “robots.txt,” all lowercase, with no extra extension or capital letters.
5. Upload it to your root folder. The file must sit at yoursite.com/robots.txt, not inside a subfolder.
6. Test the file. Visit the live URL in your browser to confirm it loads and displays your rules correctly.

WordPress users often skip manual uploads entirely. Many SEO plugins generate and edit the file directly from the dashboard. A free online generator works just as well if you prefer a visual tool instead of raw code.

A Working Robots.txt Example

Seeing a full file helps more than reading rules in isolation. Here is a realistic example for a typical blog or small business site.

User-agent: *

Disallow: /wp-admin/

Disallow: /cart/

Disallow: /checkout/

Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

This example blocks crawlers from admin pages, cart pages, and checkout pages, none of which help SEO. The “Allow” line makes an exception for one file that WordPress needs crawlers to reach. Finally, the sitemap line closes out the file.

Notice the blank line before the sitemap entry. That spacing keeps the file easy to read and matches common formatting conventions. Search engines will still parse the file correctly without it, but clean formatting helps you edit it later.

Where to Upload Your Robots.txt File

The file location matters as much as the syntax inside it. Robots.txt must live in your site’s root directory, never inside a folder like /blog/ or /pages/. A file at the wrong path gets ignored completely.

Most hosting providers offer an FTP client or a file manager inside cPanel. Upload the file directly to the folder marked public_html, or whatever your host calls the root directory. WordPress sites often handle this step through a plugin instead.

Quick way to confirm the location. Type yoursite.com/robots.txt directly into your browser. If the file loads as plain text, it sits in the correct place.

Common Robots.txt Mistakes to Avoid

A misplaced slash causes more damage than any other robots.txt mistake. Writing “Disallow: /” instead of a specific path blocks your entire site from search engines. Always review the file before uploading it live.

Blocking CSS or JavaScript files is another common error. Search engines need these files to render your pages properly. Blocking them can hurt how Google understands your layout and content.

Some site owners try to use robots.txt for privacy, which does not work. Sensitive pages need a login, a noindex tag, or password protection instead. It only discourages well-behaved crawlers, and nothing more.

One robots.txt file per domain, not per page. Subdomains need their own separate robots.txt file at their own root, since search engines treat each subdomain independently.

Robots.txt vs the Meta Robots Tag

The robots.txt file controls which pages crawlers can access in the first place. A meta robots tag works differently, controlling whether an already-crawled page gets indexed. Both tools matter, but they solve different problems.

Use robots.txt to save crawl budget on low-value sections like admin folders or filters. Reach for a meta robots tag when you want a specific page crawled but kept out of search results. Combining both gives you full control over crawling and indexing.

A free Meta Tag Generator makes it easy to build the correct meta robots tag for any page. Pairing it with your robots.txt file covers both sides of search engine visibility.

How to Test Your Robots.txt File

Never assume a robots.txt file works just because it uploaded successfully. A single typo can block far more than intended. Testing takes only a minute and prevents costly indexing mistakes.

Start by loading the live URL in your browser to confirm the file displays correctly. Then check Google Search Console, which shows any robots.txt errors it detects. Fix and re-upload the file if any warnings appear.

Re-check the file every time you change your site structure. Renamed folders or new sections can leave old “Disallow” rules pointing at paths that no longer exist. A quick review keeps the file accurate over time.

Keep a backup copy of your robots.txt file somewhere safe. If an update breaks the live version, restoring the backup takes seconds instead of rebuilding every rule.

⚙️ Build Your Robots.txt File Free Skip the manual syntax and use the Robots.txt Generator to create a correctly formatted file in seconds, ready to download and upload straight to your site.

Frequently Asked Questions

How do I create a robots.txt file if I have never coded before?

You do not need coding skills. A plain text editor and a few simple lines are enough, and a free generator tool works too.

Where should I upload the robots.txt file?

It must sit in your site’s root folder, reachable at yoursite.com/robots.txt. Any other location gets ignored by search engines.

Does every website need a robots.txt file?

Not strictly, but most benefit from one. It helps crawlers avoid low-value pages and points them toward your sitemap.

Can a robots.txt file hide pages from Google completely?

Not reliably. Blocked pages can still appear in search results without a snippet. Use a noindex meta tag for guaranteed removal.

What happens if my site has no robots.txt file at all?

Crawlers assume they can access everything. That is fine for small sites, but larger sites often waste crawl budget without one.

Is it safe to edit an existing robots.txt file myself?

Yes, as long as you test changes afterward. Always check the live file and Search Console after any edit to confirm nothing broke.

Final Word: Creating a Robots.txt File the Right Way

Now you know how to create a robots.txt file from a blank editor to a working, tested file. The syntax stays simple even as your site grows more complex.

Double-check every “Disallow” line before you publish, since one wrong slash can block your entire site. Testing the live file afterward takes only a minute and saves real headaches later.

The Robots.txt Generator above builds a correctly formatted file for you instantly, no manual syntax required.