It has two main functions : crawl the WEB to get documents and build a full text database with this documents. The crawler part visit the documents and store intersting information about them locally. It visits the document on a regular basis to make sure that it is still there and updates it if it changes.
