Add a /robots.txt file to both get rid of error messages

(requests to this file resulting in 404 not found) and
to make sure at least some robots don't crawl under
/source/ too much (our cgiweb stuff causes high load).

diff --git a/robots.txt b/robots.txt
new file mode 100644 (file)
index 0000000..5ca672c
--- /dev/null
+++ b/robots.txt
@@ -0,0 +1,8 @@
+##
+##  robots.txt -- Robot Exclusion Standard config file
+##
+
+User-agent: *
+Disallow: /source/cvs/
+Disallow: /source/exp/
+