Combined Log bot filtering
cblfb uses a bot filter pattern file to filter out bot request from
HTTP logs in
Combined Log Format.
Each pattern in the file is matched against the
in the log file.
On a Linux or BSD machine, in a terminal, run:
mkdir -p ~/src cd ~/src wget -qO - https://firstname.lastname@example.org | tar -xzvf - cd cblfb make install PREFIX=$HOME
cd ~/src/cblfb make uninstall PREFIX=$HOME
BOTSFILE variable to a file containing bot patterns, we
recommend Project Counter's
official list of user agents that are regarded as robots/spiders. COUNTER_Robots_list.txt is the plain text version.
wget https://raw.githubusercontent.com/atmire/COUNTER-Robots/master/generated/COUNTER_Robots_list.txt BOTSFILE=COUNTER_Robots_list.txt cblfb /var/www/logs/access.log
We recommend setting the
BOTSFILE variable in your
mkdir -p ~/share/misc cd ~/share/misc wget https://raw.githubusercontent.com/atmire/COUNTER-Robots/master/generated/COUNTER_Robots_list.txt echo "export BOTSFILE=$HOME/share/misc/COUNTER_Robots_list.txt" >> ~/.profile
[1.0.0] - 2022-03-04
- Combined Log bot filtering