[10876] in Public-Access_Computer_Systems_Forum
HTTP-based Mirror Software
daemon@ATHENA.MIT.EDU (Terry Kuny)
Thu Dec 5 21:23:21 1996
Date: Thu, 05 Dec 1996 14:15:20 -0600 (CST)
From: Terry Kuny <Terry.Kuny@xist.com>
To: Multiple recipients of list PACS-L <PACS-L@UHUPVM1.UH.EDU>
Reply-To: Public-Access Computer Systems Forum <PACS-L@UHUPVM1.UH.EDU>
----------------------------Original message----------------------------
*** HTTP-based Mirror Software ***
Based on a number of responses and hunting
and pecking for different solutions, I have
come up with a list of software that is
suitable for mirroring WWW sites.
My criteria for WWW mirroring software is that it:
1. traverse the site recursively,
2. duplicate the directory structure locally,
3. retrieve only files that are new or where the date has changed,
4. constrain the retrieval to documents located only at the host domain.
I have not been able to test all the software below,
so if anyone has any additions to this or corrections,
could they please let me know.
Thanks to those people who have sent me responses or hints.
Regards,
Terry Kuny
terry.kuny@xist.com
*** Selected WWW (HTTP) Mirror Software ***
These programs have been recommended as
commonly used mirror packages:
1. WebCopy (PERL. UNIX - highly recommended)
URL: http://www.inf.utfsm.cl/~vparada/webcopy.html
Email: vparada@inf.utfsm.cl
2. w3mir (PERL. UNIX, NT - highly recommended)
URL: http://www.ifi.uio.no/~janl/w3mir.html
3. Wget (UNIX)
URL: ftp://gnjilux.cc.fer.hr/pub/unix/util/wget/
URL: http://sunsite.auc.dk/ftp/pub/infosystems/wget/
4. Monster (UNIX, LINUX.)
URL: http://www.neva.ru/monster.list/russian.www.html
Email: diwil@wild.stu.neva.ru
5. WebFetcher
URL: http://www.ontv.com/webfetcher/index.htm
6. WebMirror (PERL.)
URL: ftp://enterprise.ic.gc.ca/pub/perl/CPAN/authors/id/FLEITNER/
7. WebVac (C++. LINUX, HP/UX.)
URL: http://www.federated.com/~tim/webvac.html
8. Grab-A-Site (Win95. example of a program suitable for local hard drive use)
URL: http://www.winsite.com/info/pc/win95/netutil/dgrab3.zip/
The "Web Robots Database" lists a number of other
software implementations which may also function
as mirroring software for web sites.
The "Web Robots Database" is accessible at:
<URL: http://info.webcrawler.com/mak/projects/robots/robots.html>
-------------------------------------------------------------
Mr. Terry Kuny Phone: 819-776-6602
XIST Inc. Email: terry.kuny@xist.com
Global Village Research URL: http://xist.com/kuny/
-------------------------------------------------------------
