URL not recognised:
Full request: robots.txt
Domain: db.rsnz.org
Parameters: URL1: robots.txt
URL2:
URL3:
URL4:
URL5:
URL6:
GET /robots.txt HTTP/1.1
x-cc-id: ccc01-02
Host: db.rsnz.org
User-Agent: CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
Accept: text/html,application/xhtml+xml,text/xml;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: close
Cache-Control: no-cache
Pragma: no-cache
Form variables
Your IP address: 38.107.191.88