Changes in current version - Added support for setting transfer
timeout times through envirnoment variables, and config file
settings (Peter Scott)
- Fixed problem with broker registry not correctly handling
deletion of duplicate objects (Hrvoje Stipetic) - Added
support for relative URLs in client pull inforamtion, and for the
HTML refresh directive. (Hrvoje Stipetic)
- Change the content of the "description" attribute to be the first
255 characters of the objects "body" or "partial-text" attribute
(Peter Valkenburg)
- SGML summariser fixes - * to allow mapping of
Dublin Core directives to attributes * to autmatically
prefix meta generated elements with a custom prefix
* to 'fix' meta generated attributes to only contain
legal SOIF characters (Hrvoje Stipetic)
- Added support for printing total number of items found when in
perpage mode to nph-search (Hermann Straus)
- Altered broker behaviour so that objects with no TTL value are
never expired (Simon Wilkinson)
- Fixed bug in gather which would result in incorrect HELLO
messages
being sent to servers (Dave Beckett) - Fixed broker bug
where duplicate copies of a URL could remain in
the broker in certain circumstances (Hrvoje Stipetic) -
Added reporting of number of objects added to broker output
(Hrvoje Stipetic) - Added broker version information to broker
greeting message (Hrvoje Stipetic) - Fixed bug where
admin password could be written to broker log (Hrvoje Stipetic)
- Removed unnecessary print's in the nph-search script -
Fixed parsing of broker configuration files so that whitespace is
correctly handled in arguments (Peter J. Scott & Hrvoje Stipetic) -
Added -V option to the broker to return the version number
(Hrvoje Stipetic) - Fixed definition of random() in stor_man.c for
better portability (Otis Gospodnetic) - Improved result
ranking in nph-search for occurence of search string in URL and
titles (Bruce R. Lewis) - Fix to LocalMapping code to remove
problem where FD 0 (stdin) could be closed by mistake (Simon
Wilkinson) - Fixes to SGML.sum to handle problems with HTTP-EQUIV
and spaces in META NAME attributes (Hrvoje Stipetic) -
Fix to broker where compression would unlink open files, and
eventually the broker would run out of disk space (Hrvoje Stipetic)
- Fixes to sys_errlist definitions for Linux compatibility (Tim
Riker) - Fixed local file handling so that it doesn't hang if it
tries to access a "special" file (Simon Wilkinson) -
Fixes for glimpse internationalization - ISO_CHAR_SET has to
be not only defined, but be non-zero (Marjan Erzen) -
Added status code logging to liburl, so failed requests are
logged with the reason for failure (Marjan Erzen) - Altered
robots.txt code, so that an empty User-Agent exists every one may
collect data (Marjan Erzen) - Fixed gather.c so that it will
compile with Sun's CC, and so it doesn't redefine existing code
(Marjan Erzen) - Added a new admin command to the Broker - 'parse-
template' which will add a SOIF stream to the broker (Marjan
Erzen) - Added a command line md5 tool (Marjan Erzen)
- Altered Makefiles so that objects can be built outside of the
current source directory when using a make that supports VPATH
- such as gnumake. (Simon Wilkinson) - Added support for
sending Accept: headers to http.c in liburl. Currently sends
Accept: */* (Simon Wilkinson) - Fixed filter.c so both host and
portnumber are made available for
regular expressions (Wesley Alan Wright) - Added a SWISH
based indexing engine, which is considerably more
efficient than Glimpse, if slightly less fully featured
(Simon Wilkinson)
- Changes to nph-search.in so that the truncation warnings aren't
erroneously displayed when the user hasn't supplied a value for
maxobjflag (Allyn Fratkin)
- Added support for META client pull redirections to HTMLurls
(Hrvoje Stipetic)
- Increased limits in HTML.decl (Tim Riker) - Fix for
gather command's uncompressed transfer routine so that it
now transfers blocks correctly in most circumstances (Marjan
Erzen) [This patch went in in pl10 - but the ChangeLog entry was
omitted] - Added N-at-a-time output to the nph-search CGI, so
results can be paged. (David Hoekman) - Added noregex
option to nph-search and to the broker Glimpse interface to allow
the disabling of regular expressions in search patterns (Simon
Wilkinson) - All Harvest binaries should now honour setting of
prefix as being the install location, *when run normally*. Many
still rely upon HARVEST_HOME to point them to the correct
location, but this is provided by the support scripts which run
them (Simon Wilkinson) - Added support for setting defaults in the
query configuration file, so defaults values can be given for all
parameters that are usually passed into the CGI from the Web
search form (Simon Wilkinson) - Added support for sorting query
results by rank to nph-search (Wesley Alan Wright)
- Rewrote most of the Makefiles and altered the configure scripts
so that the build process is more rational, and so that the
targets are now "standardised". In addition a number of problems
with the makefiles have been fixed, and all of the configure
routines use one central cache file - speeding up the build
process (Simon Wilkinson) - Corrected order of adding
files to PATH so that Harvest binaries are now always used in
preference to system ones (Vincent Winczewski) - Replaced
BrokerAdmin.cgi with a perl script which accepts POST requests.
This fixes the potential problem of passwords being sent in GET
requests which were visibile in servers log files (Simon
Wilkinson) - Fixed bug in robots.txt code which caused segmentation
faults when reading incorrect files with empty User-Agent lines
(Simon Wilkinson) - Updated nph-search so that it only sends nph-
headers if invoked as an nph script, and so that it will
automatically decode HTML entities. Removed BrokerQuery.pl (now
symlink to nph-search) (Craig Counterman) - Added a new
perl HTML summariser (Andy Powell)
- Replaced all of the HSR tools with new, fixed versions supplied
by Mic Bowman - Added changes to SGML.sum to make it
compatible with nsgmls, part of SP. Error reports now on by
default (Craig Counterman) - Altered broker/Glimpse/index.c so that
case and word matching are disabled when an errorflag is provided
(Craig Counterman) - Fixed broker/Glimpse/index.c so it compiles on
an SGI (Simon Woods) - Altered http code so it now sends a user
controllable User-Agent and Maintainer address which can be set
in the config file (Hrvoje Stipetic) - Altered httpenum-
breadth so that the count of objects retrieved is accurate
(Hrvoje Stipetic) - Altered summarisers Makefile so that Pdf.sum is
now installed by default (Simon Wilkinson) - Added
-DAGREP_POINTER=1 to glimpse Makefiles - fixes segmentation
faults with picky mallocs (Dan Riley) - Replaced HTMLparse with a
newer version from the Mosaic 2.7b5 which handles comments
correctly (Simon Wilkinson)
- Changed Postscript.sum to a sh script, so stderr doesn't corrupt
the generated SOIF (Simon Wilkinson)
- Added some #includes for AIX compatibility (Simon Wilkinson)
- Default url-filter now blocks VRML files (Craig Counterman)
- Perl scripts which forced /usr/local/harvest changed
(Craig Counterman) - Altered rfc1738 unescaping so it can cope with
badly formed URLs (Simon Wilkinson) - Assorted changes
for compilation under BSDI (Simon Wilkinson) - Fixed configure
files so libraries are included in correct order
for Solaris (Simon Wilkinson) - Fixed SGML.sum so that
all debug output is sent to STDERR so it doesn't corrupt
generated data (Simon Woods) - Fixed undef of SOIF associative
array for compatibilty with Perl 5.004 (Bill Corley) -
Fixed bug where a url wasn't closed in httpenum-breadth (Wolfgang
Klimt) - Made broker more resilient to glimpse bugs (Simon
Wilkinson) - Fixed commenting mistake in glimpse Makefile (Jeffrey
Goldberg)
- Changed broker config so numbers are now indexed by glimpse
- Added missing ftp files ftp.pl and chat2.pl (Simon Wilkinson)
- Fix for Year 2000 logging and file naming problem (logs
should now be produced using 4 digits for the year) (Simon Wilkinson,
suggested by Craig Counterman) - Fixed gatherers so they
don't revisit pages that they've previously found to be broken
(Simon Wilkinson, suggested by David A. Nowitz) - Fixed breadth
first httpenum so it correctly handles META robots control tags
(Simon Wilkinson) - Rewrote Troff.sum in perl (David A. Nowitz)
- Added DOCTYPES and tidied up HTML generally (Craig Counterman)
- Fixed × bug in ×tamp call (Craig Counterman) -
Fixed assorted bugs in nph-search (Craig Counterman)
Changes to v1.5 - Altered some lseek() calls for BSD compatibility
(Martin Hamilton) - Removed optimisation from glimpse configuration
(Simon Wilkinson) - Added -nocol option to broker (Dave Beckett)
- Added virtual host support using the "Host:" header in HTTP/1.1
(Juha Laiho) - Added the HTML 3.2 ISOlat definition
(Simon Wilkinson) - Added support for META Robots tags (Simon
Wilkinson) - Altered broker / glimpse interface so that an attempt
to use regular
expressions on word boundries results on word matching being
turned off, rather than an error message (Simon Wilkinson)
- Fixed core dumps in httpenum when accessing pages requiring
authentication (Simon Wilkinson) - Changed url_retreive
behaviour so original caller gets notified of Redirect. This
fixes a bug which made it possible to index a site and completely
ignore their robots.txt file (Simon Wilkinson) - Added wildcards to
Local-Mapping (Bruce R. Lewis) - Added support to the broker for
terminating connections if the client goes away (Bruce R. Lewis)
- Added depth first gatherer for both HTTP and Gopher enumeration.
(Peter Scott) - Added hooks added to enum, prepurls and
Gatherer script to support user selection of search technique via
the Search variable (Simon Wilkinson) - Bug in depth
first enumerator where a URL first encountered at a depth deeper
than max depth isn't indexed when its seen at a lower depth fixed
(Peter Scott) - Depth counts added to the depth first enumerator
(Peter Scott) - Robots.txt checks added to the depth first HTTP and
Gopher depth first enumerators (Simon Wilkinson) - Added
code to grab URLs from HTML containing frames and client side
image maps (Dean Marino) - Altered breadth first HTTP and Gopher
enumerators so that the server containing the root page is marked
as visited. (Julian Field) - Altered enumerators so URL count is
increased for visited pages only (Ed Knowles) - Altered
HTMLurls so HREFS can now contain new lines (Peter Scott) - Altered
depth first enumerators to reduce number of temporary files
generated (Simon Wilkinson) - Fixed enumerators to avoid buffer
overflows with long URLs (Simon Wilkinson) - Added
MAX_FILTERS env var to URL filter code (Simon Wilkinson) - Fixed
url code to avoid unnecessary generation of symlinks (Bruce R.
Lewis) - Added support for HTTP/1.1 servers to url code (Simon
Wilkinson) - Fixed end of header detection bug in url code (Paul
Johnson) - Fixed temporary file closing bug in url code (Bruce R.
Lewis) - Altered Makefiles so realclean removes config.cache (Simon
Wilkinson) - Altered robots.txt handling so it can cope with errors
in server robots files. Also some performance improvements. (Ed
Knowles) - Added support for user specified User-Agent strings to
the robots.txt code (Simon Wilkinson) - Changed method of
comparing User-Agent string against robots.txt file - now correct
behaviour as per the robots specification (Simon Wilkinson)
- Fixed bug where robots.txt code left file pointer unclosed (Ed
Knowles) - Fixed bug where paths were compared case-
insensitively in the robots.txt code. (Simon Wilkinson) -
Altered Gatherer code so it converts ~ and %7E to %7e in Local-
Mapping strings (Simon Wilkinson) - Added Locale option to Gatherer
(Simon Wilkinson) - Added html32.dtd - the official Wilbur DTD
- Fixed usage of fork in ps2txt wrapper (Peter Scott) -
Fixed Harvest script so the RunGatherd scripts it creates include
correct PATH (Simon Wilkinson) - Fixed Rainbow summariser so it rm
-r's instead of unlinking - Altered files so they will compile
under NetBSD (Martin Hamilton) - Altered configure scripts for
NetBSD support (Dave Beckett) - Upgraded included version of
glimpse to 4.0 - Fixed broker so the port picked for glimpseindex
is less than 30000 (Peter Scott) - Added support for NOT
operator to broker glimpse interface code - Fixed signal handling
bug in glimpseserver (Simon Wilkinson) - Added warning for object
truncated result sets to BrokerQuery.pl.cgi - Added nph-search CGI
for displaying results "as they come" (Bruce R.
Lewis) - Altered query cgis to escape quotes in
$html_query (Simon Wilkinson) - Added explanation of the Log-Key
directive to broker.conf (Dave Beckett) - Fixed bug in
the broker lex file so searches containing an apostrophe will
work Changes to v1.4.pl2:
- Added "robots.txt" support to the gatherer enumeration.
- Added "prefix =" to components/*/Makefile. - Changed
Gopher timeout to 120 seconds. - Changed HTTP-Query byurl pattern
to be any URL with a question mark. - Added
HARVEST_NOT_VISITED_LOG env var to httpenum. - Added support for
Glimpse-based broker to limit the number of matched lines per
object. - Protect single quotes in Gatherer-Name. -
Updated html-mcom.dtd - Changed BrokerQuery.pl to not use a
tmpfile, and to sort the results by number of matched lines.
- Fixed HTTP authentication to work with Netscape server, and
support encoding spaces as RFC1738 escapes. - Fixed
gatherd timeout bug (caused by eliminating the DNS mismatch
warning). - Fixed Carriage-Return substitution in RTF.sum.
- Changed SGML.sum to not do "word wrap" on very large strings.
- Fix HTML-lax.sum to turn carraige returns to spaces. -
Added --body-text option to HTML-lax.sum and in the comments of
HTML.sum. - Fixed SGML.sum to NOT rewrite correct DOCTYPE
declarations. - Changed our log() function to be called Log().
- Changed handling of depth between enum programs. -
Changed broker connect timeout to happen only after some data has
been read. - Added 'Access-Delay' to gatherer.cf. Now adds delay
for LeafNode URLs. - Fixed 'gather' bug when reading
binary data in non-compressed mode. - Configure:
upgraded to v2.7 - BrokerQyery.pl.cgi: protect special characters
in a Broker name. - Essence: Print [L] for URLs where local
mapping succeeds. - Update the Users manual.
Changes between release v1.4.pl1 (November 17, 1995) and v1.4
- Changes to the Gatherer - Fixed NULL BASE URL coredump bug in
HTMLurls - Fixed Gatherer to make Top-Directory set Lib-Directory
value also (like the manual says it does in section 4.6.1).
- Fixed essence and SGML.sum to look in multiple lib dirs. Look
first in Lib-Directory if set, otherwise in
$HARVEST_HOME/lib/gatherer.
- Changes to the Broker - Added <sys/select.h> for compiling on
AIX. - Changed BrokerQuery.pl.cgi to send the query to the broker
before opening the tmpfile. If there is a delay in opening
the tmpfile the broker query could time out. - Fixed
potential coredump in Log_rotate() due to large local array.
##############################################################################
Changes between release v1.4 (November 10, 1995) and v1.3
- Changes to the Gatherer: - Added symbolic link loop detection to
httpenum. - Added a GIF image summarizer (GIFImage.sum), requires
netpbm. The GIFImage type is still in the Essence stoplist by
default. - Added 'C' version of ftpget. - Added ability
to rewrite the SOIF template URL with Essence post-processing.
Could be used to gather file:// URLs and have them exported as
http:// URLs. - Added the ability to specify a program to generate
root/leaf URLs. - Fixed select() timeouts to POSIX semantics.
- Fixed SGML summarizer to give error if input is empty. -
Fixed a Makefile to actually build and install HTML-lax.sum. -
Fixed liburl problem with AFS. Must *copy* files into the
cache-liburl directory. - Fixed News gatherering: If 'newsget.pl'
exits non-zero, close the NNTP server socket. - Fixed
newsget.pl with a major rewrite. - Fixed 'fileenum' to use URLs
and not always return file://hostname/. - Fixed gatherd bug where
child process would remove parent's gatherd.pid file. -
Changed NewsArticle.sum TTL to 7 days by default. - Changed
Essence unnesting to occur in individual directories. - Removed
confusing gatherd DNS mismatch warning message.
- Changes to the Broker: - Added #Restart-Index-Server command to
broker admin command set. - Added error logging and debugging in
Glimpse inline query code. - Fixed select() timeouts to POSIX
semantics. - Fixed Glimpse minor malloc problems. - Fixed
the broker on Linux; needs unbuffered input from gather process.
- Fixed broker query language bug for high-bit (international)
characters. - Changed Broker to allow specifically
setting GlimpseServer_Port again; if not set, port is chosen
randomly. - Changed BrokerAdmin.cgi to use unbuffered output.
- Changed Glimpse macros CLEANUP and RETURN to be functions.
- Changed broker admin/LOG to log FQDN instead of IP address.
- Remove glimpse version ambiguities in Glimpse/index.c. -
Removed getpeername() call in the broker; get address from accept().
- Changes to the Cache: - The cache has been moved to a separate
distribution.
- Miscellaneous Changes - Dont link with -lmalloc on Solaris.
- Fixed User Manual and FAQ inconsistencies.
##############################################################################
Changes between release v1.3 (September 7, 1995) and v1.3.beta:
- Changes to the Broker:
- Added support for auto-validation from the HSR which
includes a description.html file, RunUpdate program
for each new Broker.
- Changes to the Cache:
- Added support to dynamically toggle debug level via USR1 and
USR2.
- Fixed dnsserver parsing numeric addresses.
- Added patches for FreeBSD.
- Changed source_ping to off by default.
- Added optional code for 'local_ip' line in cached.conf.
Addresses given as 'local_ip' will be retrieved directly,
without sending any probe packets.
- Added 'TIMEOUT_DIRECT' as a new kind of entry in
cache_hierarchy.log.
- Changes to the Gatherer:
- Added LMT.gdbm to liburl to keep last-modified-timestamps.
- Added support for using BASE element in HTML enumeration.
- Added support for HTML-3.0 DTD. - Added support for
Netscape DTD. - Added support for HotJava DTD. - Added
old HTML.sum as HTML-lax.sum. - Added MacBinHex as a supported
nested type in essence. - Changed gatherd to die when its data
directory gets removed.
- Fixed bug: repeated HTTP redirected URLs (with help from
glenn@rockie.nsc.com)
- Miscellaneous Changes
- Incorporated fixes for FreeBSD port from ted@oz.plymouth.edu.
- Incorporated fixes for Ultrix port from
dsr@lns598.lns.cornell.edu.
##############################################################################
Changes between release v1.3.beta (August 7, 1995) and v1.2:
- Changes to the Broker:
- Upgraded to Glimpse 3.0.
- Improved and updated WAIS, Inc. support to use version 2.1.1.
- Added support for Verity VDK as backend indexer/searcher.
- Added support for GRASS GIS as spatial database.
- Added support for PLS, Inc. PLWeb as backend indexer/searcher.
- Added IP numbers for incoming requests to log information.
- Added support for displaying individual SOIF attributes via WWW.
- Added 'Uniqify' command to Broker; keeps most current object
of duplicate URLs.
- Added security and name lookup to BrokerQuery.pl.
- Added support for Glimpse inline queries.
- Added error message to report incorrect WWW installation.
- Added some support for Internationalization in the Broker
- Added support for automatic validation by HSR.
- Removed need for 'gzip' in the Broker.
- Changed BrokerQuery.pl to try multiple entries from Brokers.cf
- Changed broker to read queries with a timeout. Very long queries
can get segmented by TCP.
- Fixed bug with matching Description attributes.
- Fixed bug with Glimpse regular expression detection.
- Fixed bug in CreateBroker -- wrong default Gatherer port number.
- Changes to the Cache:
- Added persistent disk storage across cached reboots.
- Added IP-based access control.
- Added setting of the TTL based on URL regular expressions.
- Added more sophisticated setting of the TTL based on HTTP
headers.
- Added more statistics information.
- Added support for logging using the common httpd logfile format.
- Added support for HEAD HTTP request method.
- Added support for user-configurable periodic garbage collection.
- Added support for user-configurable stoplist.
- Added support for WAIS proxy'ing (from Edward Moy, Xerox PARC).
- Added support for quick aborting when client drops connection,
cached stops immediately. Useful for slow network links.
- Added high/low water marks for disk storage.
- Added 'source_ping' to cached.conf.
- Added 'dns_children' to cached.conf.
- Added -z to force a cached to discard (zap) its disk storage.
- Added logging of ftpget.pl failures (exit codes and signals).
- Added Expires timestamp to cache log
- Improved error messages for DNS name lookup failures.
- Improved performance of LRU replacement policy.
- Improved performance for generating statistics.
- Increased listen(2) socket queue size to 50 or max of OS.
- Removed all Tcl code.
- Cleaned memory allocation and management.
- Cleaned up and updated cached.conf.
- Cleaned up debugging output.
- Changed default low watermark to 60%.
- Changed trace mail into cached.conf option.
- Changed algorithm for time estimations using echo ports.
- Changed dnsserver to try gethostbyname(3) again sometimes
- Fixed bugs with URL intepretation.
- Fixed bugs with internal IPcache memory management.
- Fixed bug with DNS lookups on IP numbers.
- Fixed bug with not finding 'dnsserver'.
- Fixed bug with hard timeouts in select loop.
- Fixed bug with some platforms needing strdup().
- Fixed bug with ftpget.pl not including MIME content-type for
unknown filename extensions.
- Fixed bug with ftpget.pl not parsing ls output correctly
(wasn't matching dashes in user/group names).
- Fixed copyright messages in source code.
- Fixed realloc() bug for concurrent object access.
- Fixed bug when neighbors specified and dns_servers != 3.
- Fixed bug with new hash tables when deleting from table as it is
being traversed.
- Fixed various minor bugs.
- Changes to the Gatherer:
- Added ability to pass enumerated URLs through an external
filter program. Allows very specific selection of URLs to
further enumerate.
- Added -background flag to the Gatherer; does export work in bg.
- Added IP-based filtering (regular expressions) in host-filter
- Added Post-processing of summaries to Essence
- Added 'gather' check for 'gzip' before setting compression
option.
- Added username/password support for HTTP retrievals
- Changed gatherer to remove cache-liburl directory after a
successful
gather session.
- Fixed bug: Infinite loops in 'enum' on Invalid URLs
- Fixed bug: HTTP headers not parsed from slow servers
- Improved URL parsing; support for username/password in FTP urls.
- Miscellaneous Changes
- Upgraded autoconf 'configure' scripts to v2.4.
- liburl: better handling of relative URLs.
- liburl retrieval programs abort very large transfers (at 10
Mbytes)
- Fixed bug with subscribing to harvest-users mailing list.
##############################################################################
Changes between release v1.2 (April 3, 1995) and v1.1:
- Changes to the Broker:
- Major performance improvements to the collector interface.
- Added fast, efficient internal Gatherer ID management.
- Added support for clients requesting attributes with #attribute.
- Added support for log file rotation, and terse logging.
- Added support for #operation in query manager interface.
- Cleaned up the log file format.
- Cleaned up the administrative interface.
- Cleaned up the UNIX file system-based storage manager.
- Fixed major bug with WAIS support.
- Fixed file descriptor leaks in glimpseserver when the index
contained files that had since been deleted.
- Fixed bug with overflowing lines from glimpse.
- Fixed bug with hostname initialization.
- Fixed memory leak with the Description-Tag attribute matching.
- Fixed various minor bugs.
- Changes to the Cache:
- Added httpd accelerator support.
- Added IP number logging.
- Added setuid() to a user when cached is run as root.
- Added support for HTTP servers that die abruptly.
- Added client_timeout which places a hard limit on the life
of incoming connections on the ascii port, or on outgoing
HTTP or Gopher clients.
- Cleaner implementation for retrieving FTP URLs via ftpget.pl.
- Tries to write cached.pid file in same directory as cached.conf.
- Changed FTP support to sacrifice correct HTTP headers for
dramatically decreased latency for large FTP objects.
- Fixed ftpget.pl -htmlify to determine directory vs. file
correctly and send HTTP header as soon as possible.
- Fixed rare core dump during HTTP xfers.
- Fixed how the error messages are printed.
- Better support for larger file descriptor tables.
- Debug level 0 and 1 now has timestamp logged.
- Cleaned and updated defaults for cached.conf.
- When run as root and do suid, cached will change current
directory
to its swap directory. Swap directory is pretty sure that
writable
to cached. Just in case, it crashes so it can write core file.
- Minor modification of store error message.
- Remote client connection resets are handled as soft error.
- Strip an extra /r/n from MIME.
- Hierachy log (yet another log, but it's optional).
- Periodically hunts for zombies processes.
- Added more information to the stat interface.
- Cleaned up info data for improved parsability/readability.
- Changes to the Gatherer:
- Added support to follow HTTP redirection pointers.
- Added support for $http_proxy environment variable in liburl.
- Added support for summarizing SGML data.
- Added better support for summarizing TeX data.
- Added support for summarizing RTF and MIF data, using Rainbow
software provided by EBT, which we make available in our new
components distribution
- Added support for summarizing WordPerfect 5.1 data.
- changed HTML summarizing to use SGML summarizer, providing more
easily customizable results
- Added support for local filesystem gatherering for NNTP.
- Improved incremental gatherering support, and integrated the
support into the Essence program (removed dbcheck program).
- Added support for "fake" MD5 generation per SOIF object on
external presentation unnesting streams (exploders) --
permits incremental gathering on data generated by an Exploder.
- Added --memory-efficient to Essence to trade time for memory
efficiency; this help users who have limited with memory
resources
but are dealing with large SOIF objects.
- Added --confirm-host to Essence for explicit host DNS validation.
- Added --max-refresh to Essence to limit refreshing activity.
- RootNode enumerators generate RFC 1738 escaped URLs.
- Improved performance of SOIF parsing.
- Fixed bug in locating gzip in gatherd.
- Fixed bug in the unnesting commands in Essence.
- Fixed bug with HTTP/1.0 requests, now sends encoded URIs for
GETs.
- Fixed ftp.pl for Solaris. Wasn't setting PF_INET correctly.
- Changes to the Replicator:
- Updated with USC's version from 3/15/95
- Changes to the User's Manual:
- Added sections for new plug'n'play components: standard,
SGML, HTML, MIF, RTF, WordPerfect 5.1.
- Updated support policy.
- Added clarification in Local Gatherering section.
- Added clarification in RootNode enumeration section.
- Added clarification on Gatherer/Broker information flow.
- Added clarification for some cached internals.
- Added section on upgrading from v1.1 to v1.2.
- Added discussion about httpd_accel for cached.
- Updated info about software for the replicator section.
- Updated numerous facts to v1.2.
- Reorganized essence/content extraction customization section.
- Added description of SGML summarizing and components distribution
(including Rainbow software for MIF and RTF formats)
- Added more troubleshooting comments to all sections.
- Added more detail to cache and replication sections, including
discussions of httpd-accelerator, CreateReplica, and some of the
performance and failure-mode characteristics of the cache.
- Cleared up inaccuracies and unclarities in Gatherer RootNode
specification section.
- Added notes about user-contributed software.
- Updated support policy.
- Added index entries for all programs in appendicies.
- Other minor changes.
- Miscellaneous changes:
- Reorganized the source tree to support plug'n'play components.
##############################################################################
Changes between release v1.1 (February 17, 1995) and v1.1.beta.v2:
- Changes to the Broker:
- Added a leading protocol version header for the result set.
- Added support for query flags during Broker-to-Broker
collections.
- Added support for limiting the lifetime of glimpse queries.
- Fixed major bugs in Broker-to-Broker collections.
- Fixed major bugs with deleting Registry entries during initial
build.
- Fixed memory leaks and file descriptor mgmt bugs in
glimpseserver.
- Fixed bug with -L in glimpseserver.
- Fixed bug that increased the size of structured glimpse indexes.
- Fixed bugs in the administrative interface and WAIS support.
- Fixed core dump when searching the Registry during collections.
- Fixed display SOIF links flag in BrokerQuery.pl.
- Fixed .cgi pgms, so that httpd kills the cleanly after user
abort.
- Changed glimpseserver and broker so that they will not block
longer than 15 seconds while waiting for an incoming connection.
This prevents SunOS from blindly swapping out the process.
- Optimized so that a full glimpseindex will only happen if more
than 10% of the objects have changed.
- Added some more logging output.
- Fixed various minor bugs.
- Changes to the Cache:
- Added Gopher->HTML support. For mosaic proxy, you'll need to
set gopher_proxy http://cache.server:3128/
instead of
set gopher_proxy gopher://cache.server:3128/
- Fixed bug with HTML-ify FTP directories using ftpget.pl.
- Fixed bug with hierachical problem for refreshing.
- Fixed bogus client error message.
- Improved cached error messages.
- Changes to the Gatherer:
- Generates the 'Description' attribute whenever possible.
- Fixed bug in the expiring of objects from the PRODUCTION
database.
- Fixed bug in httpenum that wasn't cleaning up correctly.
- Fixed newsenum to obey URL-Max limit.
- Improved the Mail summarizer.
- Improved the USENET support, added NewsArticle and NewsGroup.
- Improved gatherd to speed up SEND-UPDATE timestamp computation.
- Improved preparation for the Gatherer's database to be exported.
- Purify'd Essence to remove memory leaks.
- Changes to the User's Manual:
- Updated the section on the Broker's Collection.conf file.
- Updated many minor points.
- Improved HTML version of the manual, by upgrading latex2html pgm.
- Miscellaneous changes:
- Fixed problems with Solaris' socket.ph for Perl programs.
##############################################################################
Changes between release v1.1.beta.v2 (February 3, 1995) and v1.1.beta:
- Changes to the Broker:
- Major performance improvements while doing collections.
- Uses the customizable BrokerQuery.pl for the WWW interface.
- Fixed major bugs in Broker-to-Broker transfers.
- Fixed minor bug in collections that caused necessary indexing.
- Cleaned and improved the information that is logged to
broker.out.
- Changed broker to run cleanly as a daemon by disconnecting from
the controlling terminal.
- glimpseserver now prints its error messages correctly.
- Fixed various minor bugs.
- Changes to the Cache:
- Fixed core dump bug when cached is heavily loaded.
- Improved error messages.
- Changes to the Gatherer:
- Site enumeration filter is based on host:port, and better argv
processing for 'Gatherer' - fixes by "Albert Dvornik"
<bert@MIT.EDU>
- Major performance improvements while preparing databases.
- Fixed Gatherer to change to Top-Directory before running.
- Fixed Gatherer to write dummy index.html files in data/ and tmp/.
- Fixed bug in HTTP enumeration to only extract links from HTML.
- Fixed various minor bugs.
- Changes to the User's Manual:
- Added detailed appendix on Harvest software layout and programs.
- HTML version of the manual now contains the local copy of the
icons.
- Added section on customizing BrokerQuery.pl.
- Fixed example for Filters during RootNode enumeration.
- Added a search interface to the User's Manual using a Broker.
- Updated index.
- Miscellaneous changes:
- Improved log output format to be more readable.
- Added HP-UX port/fixes from Chris Dalton (crd@hplb.hpl.hp.com).
##############################################################################
Changes between release v1.1.beta (January 26, 1995) and v1.0:
- Changes to the Broker:
- Upgraded to Glimpse 2.1 which includes glimpseserver.
- Added faster, more memory-efficient internal Registry lookups.
- Added support for switching the indexing subsystem at run-time.
- Added a statistics generator for the Broker.
- Fixed BrokerQuery.cgi so that the rejection message from the
Broker while its doing indexing works all of the time.
- Fixed Broker bug that would cause the Broker to hang sometimes
on a pclose() after doing a collection with the gather command.
- Immediately denies outside connections during a collection,
indexing, or other administrative operations.
- Improved the HTML result set generated by BrokerQuery.
- Pointers to content summaries in the result set is now an option.
- Changed /brokers to /Harvest/brokers, etc.
- Limit the time that the Glimpse search engine runs for a query.
- Added Query.cgi which can be used to support Broker replicas.
- Added support for minimal bookkeeping from Gatherer.
- Fixed problems with the Broker's cleaning, added compress
Registry.
- Fixed problems with the Broker's updating of objects.
- Fixed BrokerQuery syntax error message to point to
queryhelp.html.
- Fixed BrokerRestart for Replicator interface.
- Fixed WWW interface to work with any document root.
- Fixed various minor bugs.
- Changes to the Cache:
- Fixed serious hierachical cache bug.
- New error messages. HTTP/1.0 compliant.
- Nuke If-Modified-Since to work with Netscape.
- Non-blocking DNS lookup using dnsserver program.
- New config parameter, cache_dns_program.
- Removed Tcl library binaries - have a precompiled version of
Harvest.
- Fixed stat for outgoing message.
- Use multiple directories for on-disk swap storage.
- Changes to the Gatherer:
- Added flexible support for specifying a Gatherer's workload.
- Added support for gatherering through the local file system.
- Added support for USENET URLs.
- Added INFO command to Gatherer for statistics.
- Added support for generating minimial bookkeeping attributes.
- Improved HTTP/1.0 support for MIME headers and Last-Modified
headers.
- Fixed bug with 'gather' that caused 'gunzip' decompression to
fail.
- Made automatic keyword generation, and local disk cache maximum
size a run-time flag.
- Added a SOIF parser in Perl.
- Changed HTML URL extractor from HTML.sum to separate program.
- Fixed Gopher support to have longer read timeout.
- Consolidated GDBM utilities into the 'gdbmutil' program.
- Fixed bug with gatherd leaving zombie children.
- Fixed various minor bugs.
- Changes to the Replicator:
- Replaced with USC's Replicator distribution.
- Changes to the User's Manual:
- Added a new subsection on Extended RootNode Specifications
- Added discussion about new Local-Mapping support
- Fixed various typos and clarified wording in various places
- Fixed some URLs, and added others
- Fixed the discussion on using Glimpse with the Broker.
- Added a new subsection the Perl SOIF library.
- Added more descriptions about various system components (e.g.,
HSR)
- Added more index entries, and clarified some of the existing
entries
- Added a note about realtime Gatherer updates
- Added mention of cache RAM requirements
- Added section on Support Policy and Harvest Team Contact
Information
- Updated copyright/licensing discussion
- Added a section about the binary-only distribution
- Changed section names and content at beginning to make it more
clear and to make more sense with the new installation.
- Reorganized manual by subsystem
- Added troubleshootings sections to each subsystem, and shifted
some stuff into there that had been in other places
- Expanded section on supported platforms and software needed for
running/building Harvest
- Clarified some parts of the ``Querying a Broker'' section
- Added appendix on Directory layout of installed Harvest software
- Updated to reflect new httpd reorg
- Updated default summarizer action list
- Noted that glimpseserver is now part of the system
- Added more discussion to replicator section, including a figure
- Miscellaneous changes:
- Reorganized Harvest's installed directory structure.
- Integrated port to AIX 3.2 and AIX cc by greving@dv.go.dlr.de.
- Integrated port to HP-UX A.09.03 by steff@csc.liv.ac.uk.
- Integrated port to IRIX 5.3 by leclerc@ai.sri.com.
- Integrated port to Linux 1.1.59 by hardy@cs.colorado.edu.
- Integrated port/fixes to HP-UX 09.03 and HP ANSI C compiler
A.09.69
by crd@hplb.hpl.hp.com.
- Changed all Perl scripts to work under Perl 4.x or 5.0.
- Try to use vfork rather than fork to save memory when possible.
- Updated Copyright.
##############################################################################
Changes between release v1.0 (November 7, 1994) and v1.0-beta-1.5:
- Changes to the Broker:
- Upgraded Glimpse from version 1.1 to 2.0.
- Added support for Glimpse 2.0 which allows byte-level indexing,
limiting result set sizes, arbitrary Boolean queries, and more.
- Made case insenstive and word matching the default for Glimpse.
- Improved and updated queryhelp.html and adminhelp.html.
- Added soifhelp.html to the help suite.
- Added a reboot-broker tag to the default broker Makefile.
- Fixed various minor bugs.
- Changes to the Gatherer:
- Better HTTP/1.0 support, sends User-Agent and From fields.
- Fixed a problem with cross-site Gopher RootNode enumeration.
- Fixed bug in HTTP RootNode enumeration.
- Generation of unique, sorted keyword list is optional in
config.h.
- Changed Gatherer program to work around Solaris 2.3 Perl 4.036
bug.
- Fixed various minor bugs.
- Changes to the Cache:
- Added support for the Netscape browser.
- No longer caches /cgi-bin/ URLs.
- Updated the Tcl/Tk/dpwish pointers for the Cache manager.
- Changes to the User's Manual:
- Added an index with over 300 entries.
- Added a new section about Querying a Broker.
- Added a new section about common SOIF attribute names.
- Added a new section on periodic gatherering.
- Added a new section on tuning Glimpse.
- Added a new section on the WWW interface to the Broker.
- Added a new section on integrating new search/indexing subsystems
into the Broker, and give detailed interface description.
- Added more detail to SOIF appendix.
- Improved and updated the Administrating a Broker subsection.
- Added more explanation about manual annotations.
- Folded in content from FAQ.
- Noted particular usefulness of the Essence-Options variable,
e.g., for setting --full-text.
- Added a note to the Customizing the candidate selection step
subsection that it's particularly useful to do section based on
file and URL naming heuristics when gathering remote data,
because it can avoid retrieving lots of data.
- Added a note in the subsection on Running a Gatherer that you can
set MAX_ENUM in src/common/include/config.h, and that a future
release of Harvest we will make it possible to set this limit
more
flexibly. Also noted about the robot guidelines.
- Added an overview about the lib and bin directories for the
Gatherer,
including the defaults and descriptions of each file.
- Showed RunGatherer and RunGatherd scripts and added discussion
of how to use them from cron and /etc/rc.local.
- Added pointer to FAQ on setting up HTTPD in the Broker section.
- Put the logo on the cover page.
- Miscellaneous changes:
- Updated the COPYRIGHT and added it to all appropriate source
files.
- Updated the FAQ, and converted to HTML.
- Fixed BSD compatability bug in src/install.sh.
##############################################################################
Changes between release v1.0-beta-1.5 (October 14, 1994) and v1.0-beta-1.4:
- Added a user manual that is intended to help both novice and advanced
Harvest users better use the system. It covers the following topics:
- Introduction to Harvest (1 page)
- Subsystem Overview (2 pages)
- Getting and Installing the Harvest software (1 page)
- Making Basic Use of Harvest (3 pages)
- Advanced Features of Harvest (5 pages)
- References (1 page)
- Appendix on The Summary Object Interchange Format (SOIF) (3
pages)
- Appendix on Essence Summarizer Actions (1 page)
- Appendix on Gatherer Examples (6 pages)
- Appendix on Broker's Query Manager and Collector Interface (2
pages)
- Changes to the Broker:
- Improved Broker installation, and added the CreateBroker program
that automatically creates and configures a Harvest Broker based
on a brief Question & Answer session with the user.
- Improved the Mosaic interface to be more user-friendly.
- Added support for duplicate removal based on MD5 values.
- Made Query Manager and Administrative interface more extensible.
- Rewrote the Broker registry to improve performance and
readability.
- Added the dumpregistry command to view the Broker's registry.
- Added the test-broker command for simple testing of a Broker.
- Added support for wais-8-b5, freeWAIS, commerical WAIS, and
Nebula.
- Cleaned up the admin.html and query.html files.
- Cleaned up much of the code to make more extensible.
- Fixed bug in the registry garbage collection.
- Fixed major memory leak bugs.
- Fixed various minor bugs.
- Changes to the Cache:
- Started using icp version_id 2 of the protocol.
- Improved support for OSF/1 v2.0 on 64-bit DEC Alphas.
- Added password support for administrative interface.
- Fix bug with FTP "Parent Directory", and cleaned up HTML for
dirs.
- Fixed various major bugs with hierarchial caching.
- Fixed various minor bugs.
- Changes to the Gatherer:
- Added support for generating a sorted, unique keyword attribute,
based on the Descripton, Partial-Text, or Keywords attribute.
- Added an "allow only these types" in the Candidate Selection
step.
- Added stub Exploder type to help users use the unnesting step.
- Gatherer automatically creates a gatherd.cf file if needed.
- Fixed major gatherd bug that caused.
- Fixed various minor bugs and memory leaks.
- Changes to the Replicator:
- Working on instrumenting the code to measure peformance.
- Fixed various bugs.
##############################################################################
ChangeLog,v 1.214 1996/02/01 06:35:49 duane Exp
