# Completely disallow WebReaper # hammering website User-agent: WebReaper Disallow: / # Completely disallow MSNBot # hammering website User-agent: msnbot Crawl-Delay: 15 Disallow: /cgi-bin/ Disallow: /css/ Disallow: /error/ Disallow: /help/ Disallow: /img/ Disallow: /js/ Disallow: /manage/ Disallow: /rss/ Disallow: /subsid/ Disallow: /websvc/ Disallow: /zip/ # Completely disallow TurnitinBot # a plagirism detector that is not relevant to site User-agent: TurnitinBot Disallow: / # Rules for all robots not specified above User-agent: * Crawl-Delay: 15 Disallow: /cgi-bin/ Disallow: /css/ Disallow: /error/ Disallow: /help/ Disallow: /img/ Disallow: /js/ Disallow: /manage/ Disallow: /rss/ Disallow: /subsid/ Disallow: /websvc/ Disallow: /zip/