A Web Crawler

Synopsis:

crawl [vbpPad] [url]

Description:

Crawl through the URL space doing various and sundry things. Without any parameters, crawl just returns a list of URL's.

• -v -- Turn on verbose output
• -b -- Find broken links
• -p -- Just parse the files and scream about boo-boos
• -P -- Parse the URL into its components
• -a -- Crawl through the URL space and add words to the Msql database as used by search. For a large site, this can take a while.
• -d -- Unceremoniously delete the entire search database.

Note: You must should specify a fully qualifed URL. Relative paths are taken from the initial path, and "interesting" results may happen if the alignment of the stars is perturbed.

---

© (copyright) 1997 MTCC

Last modified: Fri Apr 25 21:06:53 PDT 1997