Definition
A web crawler automatically discovers and retrieves web pages, often for search indexing or information collection.
Usage and related concepts
Continuous crawlers and user-triggered fetchers behave differently. When measuring agent traffic, distinguish purpose and trigger; a request count alone does not establish that the content is being collected for model training.