Package logsparser :: Package extras :: Module robots
[frames] | no frames]

Module robots

source code

In this module we define a regular expression used to fetch the most common robots.

Variables
  robots = ['antibot', 'appie', 'architext', 'bingbot', 'bjaalan...
  robot_regex = re.compile(r'(?i)antibot|appie|architext|bingbot...
  __package__ = 'logsparser.extras'
Variables Details

robots

Value:
['antibot',
 'appie',
 'architext',
 'bingbot',
 'bjaaland',
 'digout4u',
 'echo',
 'fast-webcrawler',
...

robot_regex

Value:
re.compile(r'(?i)antibot|appie|architext|bingbot|bjaaland|digout4u|ech\
o|fast-webcrawler|ferret|googlebot|gulliver|harvest|htdig|ia_archiver|\
askjeeves|jennybot|linkwalker|lycos|mercator|moget|muscatferret|myweb|\
netcraft|nomad|petersnews|scooter|slurp|unlost_web_crawler|voila|voyag\
er|webbase|weblayers|wisenutbot|aport|awbot|baiduspider|bobby|boris|bu\
mblebee|cscrawler|daviesbot|exactseek|ezresult|gigabot|gnodspider|grub\
|henrythemiragorobot|holmes|internetseer|justview|linkbot|metager-link\
checker|linkchecker|microsoft_url_control|msiecrawler|nagios|perman|po\
...