In world wide web, Web Crawler is working as
a software agent which helps in web indexing. Web
crawler at times called as spider or internet bot explicitly
operated by many search engines. In Information
Retrieval to colleting an information Web crawler play an
important role. For effective searching and web indexing
is mostly depends on web crawlers. General Purpose,
Distributed and Focused crawling are the types of
crawling techniques of web crawling. This paper
discussed overall survey on web crawlers, its types and
working of web crawlers.
Keywords : Web Crawler, Crawling techniques, WWW, Search engine.