Back to mnoGoSearch site


mnoGoSearch 3.3.12 reference manual: Full-featured search engine software
PrevNext

Robots

Name

Robots -- defines whether to use robots.txt

indexer.conf

Synopsis

Robots {yes | no}

Description

Robots defines whether to use robots.txt and <META NAME="robots"> exclusions.

Setting Robots to no can be useful when running mnoGoSearch for site validation purposes, as well as when indexing your own Web site.

The default value is yes, for polite crawling purposes.

Scope

Robots can be used multiple times and affects on all following Server and Realm commands until the end of the configuration file, or until the next Robots command.

Examples


Robots no
      

See also

CrawlDelay, Disallow, Index, IndexIf, NoIndexIf, UseSitemap.