Aug 6, Google shows 7,092 pages indexed for the site. Aug 7 (10 pm), changed a site over to HTTPS and redirected all pages. Aug 9, large spike in Google-bot crawler activity. Aug 9 showing 2 pages indexed. Aug 11 showing 899 pages indexed.
It appears the one time rising search engine Cuil is offline and not coming back online. The search engine was released on high hopes to overtake Google. But such an ambitious goal does not come by easily. Cuil came across to many as overly arrogant immediately claiming they there were better than Google and would […]
Just found this “bot” going crazy on a site that is on a dedicated server, took the entire server offline. This bot looks to be new and from China, so be on the lookout. Other reports are saying it is ignoring the robots.txt file, so might not be good enough to deny it there. It […]
Spinn3r is a bot that indexes blog posts. Looks like this in your log files. Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
Anyone know anything about this bot? YebolBot (compatible; Mozilla/5.0; MSIE 7.0; Windows NT 6.0; rv:22.214.171.124; mailTo:[email protected] )