# Directions for web crawlers. # See http://www.robotstxt.org/wc/norobots.html. User-agent: HTTrack User-agent: puf User-agent: MSIECrawler User-agent: Nutch Disallow: / User-agent: * Disallow: /*/diff/ Disallow: /*/diff2/ Disallow: /*/patch/ Disallow: /*/publish Disallow: /download/ Disallow: /user/ Disallow: /rss/