Fast LogResolve

NOTE: This project will be moved to SourceForge (fastlogres.sourceforge.net) very soon.

Fastlogresolve is an enhanced replacement for the logresolve utility coming with apache and other similar tools. The main advantages and features are:

• asynchronous DNS requests (requires GNU adns library)

• caching of already resolved entries

• small (and configureable) resource allocation

• support for multiple input files (or reading from stdin)

• optional support for compressed files (input and output) (requires zlib library)

For more detailed information please have a look at the README.

NOTES: (from README)
Unter Linux a problem occurs when writing directly to a compressed file. (fastlogres needs to be compiled with zlib support.) Eventually the adns library runs into a blocking call within libc. The blocking code refers to a __sigprocmask() call, located at ../sysdeps/unix/sysv/linux/sigprocmask.c, line 49. (used libc is libc.so.6, kernel versions are 2.2.12 and 2.2.13.)
Another adns blocking sometimes has occured with Linux when using (including) the resolver 193.101.111.10 or 192.76.144.66 from the UUNET DE network. (Resolver 193.101.111.20 works fine instead.) After the warning 'TCP connection failed: poll/select: exceptional condition detected' the next TCP connect to the above servers eventually results in the same blocking call and code as described above.

If you successfully compiled and tested additional binaries for arbitrary OS, please drop them to me;
I'll add them to the list above.

Statistics / comparision:
I don't include some comparision / measurements into the distribution. So as a hint I include one of the various statistical outputs (header) of my tests here.
With a logbuf of 50 MB (-b 50m), it resolved 1.1 Mio. lines with approx. 81,000 unique hosts in 6:17 minutes. The hardware was an Ultra Sparc 5, 270 MHz CPU running Solaris 2.6, with three resolving DNS servers in /etc/resolv.conf. (This includes reading from and writing to a gzip-compressed file.)

  fastlogres Statistics (elapsed time): 6:17
  Entries: 1138099 (291.2 MBytes)
    With name   : 1
    Resolves    : 1138098
    - Timeout (30) : 153
    - Server Failure (31) : 8
    - Invalid Response (33) : 2
    - Server Failure (60) : 462
    - Inconsistent PTR/A Record (100) : 1952
    - Prohibited CNAME (101) : 3
    - Domain Invalid (102) : 24
    - No such Domain (300) : 12599
    - No Data Available (301) : 16

  Cache buckets   : 29961/32711 (91.6%) used, 2/10 avg/max entries
  Cache size      : 80823
  Cache hits      : 1057275 (1308.1%)

I got comparable results with Linux on a 400 MHz PII (6:13 minutes), so the bottleneck is still the DNS.
This can be speed up by increasing the (default) size of the logbuf (option -b). The default coming with the distribution is 10 MB. In any case you should avoid to cause swapping on the machine, when running parallel with other processes, like an HTTP-Server(s).

5.7.2001, jan@wede.de