cblfb

Combined Log bot filtering

cblfb uses a bot filter pattern file to filter out bot request from HTTP logs in Combined Log Format.

Each pattern in the file is matched against the HTTP User-Agent field in the log file.

Install

On a Linux or BSD machine, in a terminal, run:

mkdir -p ~/src
cd ~/src
wget -qO - https://adi.onl/cblfb/cblfb@1.0.0.tgz | tar -xzvf -
cd cblfb
make install PREFIX=$HOME

Uninstall

cd ~/src/cblfb
make uninstall PREFIX=$HOME

Usage

Set the BOTSFILE variable to a file containing bot patterns, we recommend Project Counter's official list of user agents that are regarded as robots/spiders. COUNTER_Robots_list.txt is the plain text version.

wget https://raw.githubusercontent.com/atmire/COUNTER-Robots/master/generated/COUNTER_Robots_list.txt
BOTSFILE=COUNTER_Robots_list.txt cblfb /var/www/logs/access.log

We recommend setting the BOTSFILE variable in your $HOME/.profile file.

mkdir -p ~/share/misc
cd ~/share/misc
wget https://raw.githubusercontent.com/atmire/COUNTER-Robots/master/generated/COUNTER_Robots_list.txt
echo "export BOTSFILE=$HOME/share/misc/COUNTER_Robots_list.txt" >> ~/.profile

Files

Download

cblfb@1.0.0.tgz

Changelog

[1.0.0] - 2022-03-04

Added

Combined Log bot filtering