Question: When I put in my URL and click ‘Go’, the program does not crawl my entire website. What’s wrong?
Answer: If our software does not crawl your site, there are a few things to check:
- Make sure your website contains HTML links to all the pages you want our software to crawl. If a particular page (or group of pages) has no links, our crawler won’t be able to find it. You can also give the crawler a hint by specifying one or more Additional Root URLs in the Advanced Project Settings.
- Our crawler uses your website’s domain to determine what’s on-site and what’s off-site content. If your site uses a mix of domain names (for example “www.example.com” and “example.com”, notice the lack of the “www.”) then you should configure a Domain Alias for each domain that is not your Root URL. This can be done in Advanced Project Settings. Click the ‘Wrench’ button to access it.
- Occasionally major HTML errors can cause the crawler to miss some links. (For example a missing ‘>’ at the end of an HTML tag in your document.) Try running some of your pages through an HTML validator to check for errors.
If you are still having difficulty, please contact technical support directly and we would be happy to assist you.