Page 1 of 1

Using eValid as a web crawler tool

PostPosted: Thu Aug 15, 2019 7:17 am
by EllieD
Good morning.

Why can't eValid be a basis for web spidering?

I want to index my company's entire website and eValid might be just the tool for the job.

Thanks

Re: Using eValid as a web crawler tool

PostPosted: Fri Aug 16, 2019 12:14 pm
by eValid
EllieD wrote:Good morning.

Why can't eValid be a basis for web spidering?

I want to index my company's entire website and eValid might be just the tool for the job.

Thanks


Interesting question EllieD.

Evidently you want to build some kind of index for a domain, kind of like what the search engines do when their robots visit your pages.

This is possible, but to the fully disclosive, eValid is not efficient enough for this kind of spidering or web crawling.

There are much more efficient ways to generate a searchable table for a website that downloading hundreds or thousands of pages fully, just to extract text and/or to identify static links.

There are many good references on the web that should point you in the right direction.


-- eValid Support