Reference

lr_dlf_analyze(1)

analyze the key distribution and key length of a dlf file

NAME

lr_dlf_analyze - analyze the key distribution and key length of a dlf file

SYNOPSIS

lr_dlf_analyze [--period p] superservice

DESCRIPTION

lr_dlf_analyze is a tool for Lire developers. It analyzes the key distribution and key length of the dlf inputs to have a better idea of what is going on.

The way the tool works is that it analyzes a DLF file for fields that are likely to be use as key in group element. (It doesn't make sense to analyze the distribution of int, bytes or time field since they can't be used as discrete key).

It compute the number of different keys for a field in a given period, the average key length, and the total memory used by the keys. It outputs a summary after each period and at a summary at the end.

The period p can be string like, 1w, 2d, 36h, etc.

EXAMPLE

Running on a 500,000 records dlf:

 [francis@Arendt tests]$ lr_dlf_analyze -p 16w www <combined.dlf > stats
 all all UNSET lr_dlf_analyze info started with -p 16w www
 all all UNSET lr_dlf_analyze info processing 5000th records...
 all all UNSET lr_dlf_analyze info processing 10000th records...
 ...
 all all UNSET lr_dlf_analyze info processing 565000th records...
 all all UNSET lr_dlf_analyze info processed 568688 records
 all all UNSET lr_dlf_analyze info memory stats: vsize=7644K rss=6232K majflt=407
 all all UNSET lr_dlf_analyze info elapsed time in seconds real=87 user=71.29 system=1.25
 all all UNSET lr_dlf_analyze info stopped


 [francis@Arendt tests] cat stats
 Period: 2000-11-01 - 2001-02-21
 DLF records: 69558
 Field                Type         Key %    Key Size   Avg Key Length
 client_host          hostname      1.12 %     13.61k      17.97
 client_domain        hostname      0.78 %      5.27k       9.89
 who                  string        0.00 %          1       1.00
 http_action          string        0.01 %         24       4.00
 requested_page       url           4.28 %     97.17k      33.39
 requested_page_ext   string        0.05 %        164       4.82
 requested_file       filename      2.04 %     47.34k      34.24
 http_protocol        string        0.00 %         17       5.67
 referer              string        1.66 %     61.93k      54.76
 useragent            string        1.02 %     20.31k      29.41
 gzip_result          string        0.00 %         13      13.00


 Period: 2001-02-21 - 2001-06-13
 DLF records: 239613
 Field                Type         Key %    Key Size   Avg Key Length
 client_host          hostname      1.60 %     78.47k      20.92
 client_domain        hostname      0.95 %     22.05k       9.91
 who                  string        0.00 %          1       1.00
 http_action          string        0.00 %         20       4.00
 requested_page       url           2.79 %    239.61k      36.74
 requested_page_ext   string        0.02 %        497       8.42
 requested_file       filename      1.35 %    121.63k      38.37
 http_protocol        string        0.00 %         22       5.50
 referer              string        1.35 %    199.91k      63.10
 useragent            string        0.47 %     54.71k      50.20
 gzip_result          string        0.00 %         13      13.00


 Period: 2001-06-13 - 2001-09-02
 DLF records: 259516
 Field                Type         Key %    Key Size   Avg Key Length
 client_host          hostname      2.35 %    121.29k      20.38
 client_domain        hostname      1.45 %     36.96k      10.06
 who                  string        0.00 %          1       1.00
 http_action          string        0.00 %      1.07k     109.80
 requested_page       url           1.85 %    185.11k      39.38
 requested_page_ext   string        0.06 %      2.65k      17.52
 requested_file       filename      0.93 %     93.90k      39.77
 http_protocol        string        0.00 %         24       6.00
 referer              string        1.13 %    187.05k      65.57
 useragent            string        0.52 %     66.51k      50.79


 gzip_result          string        0.00 %         13      13.00


 ===============
 Sumarry
 Period: 2000-11-01 - 2001-09-02
 DLF records: 568688
 Field                Type         Key %    Key Size   Avg Key Length
 client_host          hostname      1.88 %    213.38k      20.40
 client_domain        hostname      1.16 %     64.28k      10.00
 who                  string        0.00 %          3       1.00
 http_action          string        0.00 %      1.12k      54.38
 requested_page       url           2.54 %    521.88k      36.93
 requested_page_ext   string        0.04 %      3.30k      13.61
 requested_file       filename      1.24 %    262.87k      38.02
 http_protocol        string        0.00 %         63       5.73
 referer              string        1.29 %    448.89k      62.77
 useragent            string        0.56 %    141.53k      45.80
 gzip_result          string        0.00 %         39      13.00

SEE ALSO

This post on the LogReport development list: http://www.nlnet.nl/projects/logreport/hypermail/logreport/development/0335.html

VERSION

$Id: lr_dlf_analyze.in,v 1.3 2001/10/18 19:13:17 flacoste Exp $

COPYRIGHT

Copyright (C) 2001 Stichting LogReport Foundation [email protected]

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program (see COPYING); if not, check with http://www.gnu.org/copyleft/gpl.html or write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111, USA.

AUTHOR

Francis J. Lacoste <[email protected]>