• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Electric Toolbox Blog

The Electric Toolbox Blog

Linux, Apache, Nginx, MySQL, Javascript and PHP articles

  • Applications
  • FCKEditor
  • Apache
  • Windows
  • Contact Us
Home / Disallow the Internet Archive from your website using robots.txt

Disallow the Internet Archive from your website using robots.txt

The Internet Archive runs the "Wayback Machine" at www.archive.org to archive pages from websites maintaining copies of them at several points in time. If you do not wish to have your website archived in this way you can prevent this using a robots.txt file.

robots.txt entry

Simply add the following two lines to your robots.txt file. If you do not already have one, create a text file and save it as robots.txt in the root directory of your website with the following lines.

User-agent: ia_archiver
Disallow: /

This has the effect of preventing the Internet Archiver from archiving your site again in the future, and also removes any existing entries from the archive.

ia_archiver entry in log files

The webserver log files can tell if the archiver has been accessing your website. The following example from an Apache combined log file shows the Internet Archive bot requesting the robots.txt file:

67.202.59.141 - - [28/Aug/2009:13:26:31 +1200] "GET /robots.txt HTTP/1.0" 200 153 "-" 
  "ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)"

I’ve split the above log entry onto two lines for readability.

Check Out These Related posts:

  1. Untar tar.gz
  2. Create a file in Linux
  3. Scp command
  4. What Is a VPN?

Filed Under: Miscellaneous Postings

Primary Sidebar

Categories

  • Apache
  • Applications
  • Article
  • Case Studies
  • Email Servers
  • FCKEditor
  • HTML And CSS
  • Javascript
  • Linux/Unix/BSD
  • Microsoft SQL Server
  • Miscellaneous Postings
  • MySql
  • Networking
  • Nginx Web Server
  • Offsite Articles
  • OSX
  • PHP
  • Quick Tips
  • RFC – Request for Comments
  • SilverStripe
  • VMWare
  • VPN
  • Windows
  • WordPress

Recent Posts

  • Vim Show Line Numbers
  • Add User To Group Linux
  • Chmod 777 Tutorial
  • How to Copy Directory Linux
  • Linux create user

Copyright © 2021. ElectricToolBox. All Rights Reserved.

  • Contact Us
  • Copyright Info
  • Privacy Policy
  • Sitemap