public marks

PUBLIC MARKS with tag robots.txt

This year

2016

2015

2014

2013

2012

2008

Wikio : le bon (redirection 301) ? La brute (redirection 302) ou le truant (fichier robots.txt frelaté) ?

by -Nicolas-
La multiplication des digg-like, des agrégateurs et autres aspirateurs de contenus nécessite une vigilance accrue quant aux redirections qui affectent les liens pointant vers nos blogs et nos sites web. A cause de Wikio, il faudra non seulement vérifier les redirections, mais aussi les fichiers robots.txt. Magneto !

The Web Robots Pages

by simon_bricolo
The Robots Database lists robot software implementations and operators.

2007

ACAP Launches, Robots.txt 2.0 For Blocking Search Engines?

by kuroyagi
After a year of discussions, ACAP -- Automated Content Access Protocol -- was released today as a sort of robots.txt 2.0 system for telling search engines what they can or can't include in their listings. However, none of the major search engines support ACAP, and its future remains firmly one of "watch and see."

robotstxt.org

by lukeslytalker & 7 others
This is the main source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

辛辣インターフェース評議会 - ポケットはてなは著作権侵害かどうか

by kuroyagi
"ふつう変換系のサービスってrobots.txtいれるよね。(中略)はてなは検索エンジンSPAMで収益を上げる会社ですか?"

ニコニコブックマーク(仮)

by oqdbpo (via)
ニコニコブックマークのuser-agentは nicobot0.1 (+http://www.nicob.jp/?m=default&a=info&p=help) です。 ニコニコブックマークのみに登録させたくないときはrobots.txtに以下のように書いてください。 User-Agent: nicobot Disallow: / <meta>タグによる登録拒否 HTMLに以下のmetaタグを埋め込むことでも登録拒否が可能です。 <meta name="nicob" content="noindex">

Mes meilleurs adresses pour créer un site

by Mario5
Mes références entant que webmestre. Cette page sert aussi à montrer ce que l'on peut faire avec des commandes CSS.

2006

New Robots.txt Syntax Checker: a validator for robots.txt files

by fastclemmy & 7 others
This robots.txt checker is a "validator" that analyzes the syntax of a robots.txt file to see if its format is valid as established by Robot Exclusion Standard (please read the documentation and the tutorial to learn the basics) or if it contains errors.

The Web Robots Pages

by stan & 7 others
Web Robots are programs that traverse the Web automatically. Some people call them Web Wanderers, Crawlers, or Spiders. These pages have further information about these Web Robots.

Active users

dzc
last mark : 28/03/2017 12:17

mfaure
last mark : 28/10/2016 16:37

-Nicolas-
last mark : 19/09/2008 16:01

simon_bricolo
last mark : 10/09/2008 06:59

jmfontaine
last mark : 22/01/2008 10:01

kuroyagi
last mark : 30/11/2007 01:09

lukeslytalker
last mark : 12/10/2007 13:15

krachot
last mark : 15/07/2007 20:10

oqdbpo
last mark : 20/06/2007 12:22

plasticdreams
last mark : 11/02/2007 10:43

Mario5
last mark : 01/02/2007 18:15

fastclemmy
last mark : 11/09/2006 10:06

stan
last mark : 11/09/2006 09:53

martinsam
last mark : 04/09/2006 21:20

trancedelixx
last mark : 05/07/2006 01:39