PDA

View Full Version : Why is there a Yahoo and Google spider on the site?


Jamie
12th May 2007, 11:59
Folks just had a look at the users on line and there is a Google and Yahoo 'spider' in there....is this a bad thing??

sorry....IT dunce man here.

Jamie

Kandyman
12th May 2007, 12:12
The quick answer is

Yes there should be there they help with getting the site known

Kandyman
12th May 2007, 12:12
Read here more more info
http://help.yahoo.com/help/us/ysearch/slurp/

Yahoo! Slurp is Yahoo!'s web-indexing robot. The Yahoo! Slurp crawler collects documents from the Web to build a searchable index for search services using
the Yahoo! search engine. These documents are discovered and crawled because other web pages contain links directing to these documents.

As part of the crawling effort, the Yahoo! Slurp crawler will take robots.txt standards into account to ensure we do not crawl and index content from those pages whose content you do not want included in Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, Yahoo! will not read or use the contents of that page. The URL of a protected page may be included in Yahoo! Search Technology as a "thin" document with no text content. Links and reference text from other public web pages provide identifiable information about a URL and may be indexed as part of web search coverage.

GreyGhost
12th May 2007, 12:38
A glut of 195.x.x.x usually means AOL robots are at work too.
This is a major reason why we advise folk not to put personal details in posts, such as telephone numbers and email addresses. These spider bots are relatively harmless and are very useful as Mr. K says to take the site up the search ratings. There are very many less salubrious bots looking specifically for personal info.

JohnDotCom
12th May 2007, 13:27
And in English for the rest of you. They are safe and nothing to worry about.