| Issue 54: | Webharvester seems to download/crawl duplicate pages | |
| 1 person starred this issue and may be notified of changes. | Back to list |
Webharvester seems to download/crawl duplicate pages. It happens if the prefix of the same URL differs slightly, like "www." vs. "http://wwww. vs. no prefix at all.
Mar 2, 2010
Project Member
#1
blake.ol...@gmail.com
Status:
Fixed
|