Security tool that monitors log files for suspicious activity (like repeated failed login attempts) and automatically blocks offending IP addresses for a specified period of time. It can monitor logs from various services like SSH and web servers, making it an effective defense against brute force attacks.
Setup
We setup fail2ban on Dockerhost to
- Detect suspicious ssh behaviors (not too relevant as ssh isn’t exposed to the WAN, but this is enabled by default on Debian so we might as well configure it)
- Bans through
iptables
- Bans through
- Detect suspicious web behavior through Nginx Proxy Manager (NPM)
- Bans at the Cloudflare level, through their API (iptables would be useful as public traffic is only only going through Docker network layers)
- All bans are reported to Telegram
Fail2ban can similarly be setup on other hosts for SSH/etc as needed. Note: TrueNAS has its own failure reporting system (System > Alert settings > System).
Setup NPM
The Cloudfare connecting IP (header CF-Connecting-IP) needs to be saved in the logs.
This just adds “[CF-conn-IP $http_cf_connecting_ip]” to the default NPM log format:
sudo -u dockeruser tee -a /srv/docker/volumes/npm/data/nginx/custom/http_top.conf
log_format default_plus_cf '[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [CF-conn-IP $http_cf_connecting_ip] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"';
Then, in NPM’s UI > Proxy hosts > “Edit” relevant hosts > Advanced, add:
location / {
access_log /data/logs/proxy-host-X_access.log default_plus_cf;
}
where X is the host number.
Setup Cloudflare
We need a Firewall Access Rules token for the cloudflare-firewall action.
Go to CF Dashboard > My Profile > API Tokens > Create Token > Create Custom Token:
- Token name: Fail2Ban
- Permissions: Account, Account Firewall Access Rules, Edit
- Other fields can be left as default or as applicable.
- Summary > Create
Setup Telegram
For the telegram-notif action.
See online how to get the Bot Token and the Chat Id.
Setup Fail2Ban
sudo apt update && sodo apt install fail2ban jq
/etc/fail2ban/jail.local
[DEFAULT]
bantime = 2h
findtime = 10m
maxretry = 3
action = cloudflare-firewall
telegram-notif
# Ignore all direct wireguard-to-npm access
ignoreip = 127.0.0.1/32 192.168.1.0/24 10.100.0.3/32
# sshd is activated by default in jail.d/defaults-debian.conf
[sshd]
backend = systemd
action = iptables
telegram-notif
# NPM 404 errors (potential scanning)
[npm-404]
enabled = true
logpath = /srv/docker/volumes/npm/data/logs/proxy-host-*_access.log
filter = npm-404
# Less strict for 404s
maxretry = 10
# HTTP forbidden access
[npm-403]
enabled = true
logpath = /srv/docker/volumes/npm/data/logs/proxy-host-*_access.log
filter = npm-403
# NPM forbidden access
[npm-forbidden]
enabled = true
logpath = /srv/docker/volumes/npm/data/logs/proxy-host-*_error.log
filter = npm-forbidden
# NPM protocol errors (SSL etc)
[npm-protocol]
enabled = true
logpath = /srv/docker/volumes/npm/data/logs/fallback_access.log
filter = npm-protocol
/etc/fail2ban/filter.d/npm-404.conf
# For NPM's 404 errors in proxy-host-X_access.log
[Definition]
# If CF-conn-IP is present and not "-", use that, otherwise use Client IP
failregex = 404 - .* \[Client [^\]]+\] \[CF-conn-IP <HOST>\]
404 - .* \[Client <HOST>\] (\[CF-conn-IP -\]|\[Length)
datepattern = \[b/H:S \+0100\]
/etc/fail2ban/filter.d/npm-403.conf
# For NPM's 403 errors in proxy-host-X_access.log
[Definition]
# If CF-conn-IP is present and not "-", use that, otherwise use Client IP
failregex = 403 - .* \[Client [^\]]+\] \[CF-conn-IP <HOST>\]
403 - .* \[Client <HOST>\] (\[CF-conn-IP -\]|\[Length)
datepattern = \[b/H:S \+0100\]
/etc/fail2ban/filter.d/npm-forbidden.conf
# For NPM's "access forbidden by rule, client: x.x.x.x" in proxy-host-X_error.log
[Definition]
failregex = access forbidden by rule, client: <HOST>,
datepattern = m/H:S
/etc/fail2ban/filter.d/npm-protocol.conf:
# For 400/444/maybe-others errors from NPM's fallback_access.log
[Definition]
failregex = \] 4[0-9][0-9] - .* \[Client [^\]]+\] \[CF-conn-IP <HOST>\]
\] 4[0-9][0-9] - .* \[Client <HOST>\] (\[CF-conn-IP -\]|\[Length)
datepattern = \[b/H:S \+0100\]
/etc/fail2ban/action.d/telegram-notif.conf: (chmod 600)
[Definition]
actionban = curl -X POST https://api.telegram.org/bot<token>/sendMessage \
-d chat_id=<chat_id> \
-d parse_mode=HTML \
-d "text=🚫 Banned <ip> to jail '<name>' -- <matches>"
actionunban = curl -X POST https://api.telegram.org/bot<token>/sendMessage \
-d chat_id=<chat_id> \
-d "text=✅ Unbanned <ip> from jail '<name>'"
[Init]
token = TELEGRAM_TOKEN
chat_id = TELEGRAM_CHATID
/etc/fail2ban/action.d/cloudflare-firewall.conf: (chmod 600)
[Definition]
actionban = curl -s -X POST <_cf_headers> <_cf_url> \
-d '{"mode":"block","configuration":{"target":"ip","value":"<ip>"},"notes":"Fail2Ban - jail <name>"}'
actionunban = id=$(curl -s <_cf_headers> "<_cf_url>?mode=block&configuration_target=ip&configuration_value=<ip>" \
| jq -r .result[0].id)
if [ "$id" = "null" ]; then echo "<name>: id for <ip> cannot be found"; exit 0; fi;
curl -s -o /dev/null -X DELETE <_cf_headers> "<_cf_url>/$id"
[Init]
_cf_url = "https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules"
_cf_headers = -H "Authorization: Bearer CF_FIREWALL_API_TOKEN" -H "Content-Type: application/json"
Replace TELEGRAM_TOKEN, TELEGRAM_CHATID and CF_FIREWALL_API_TOKEN above.
Double-check timezone in datepattern expressions are correct.
If local IPs should be ignored at the filter level instead of at the jail one (jail.local), use for example the following expression is npm-404.conf, etc:
ignoreregex = \[Client (?:127\.0\.0\.1|192\.168\.1\.\d+|172\.20\.0\.3)\]
Useful commands
Restart and see startup logs:
sudo systemctl restart fail2ban && sleep 2 && sudo systemctl status fail2ban | cat
Manual ban/unban an IP for filter sshd (will trigger associated actions):
sudo fail2ban-client set sshd banip 66.66.66.66
sudo fail2ban-client set sshd unbanip 66.66.66.66
Parse a (test or real) log file with filter npm-404 and see failed/ignored/missed lines:
fail2ban-regex /tmp/test-404.log /etc/fail2ban/filter.d/npm-404.conf
Tests
Verify IP bans work in Cloudflare
Go to CF Dashboard > click on a domain > Security > WAF > Tools > IP Access Rules.
sudo fail2ban-client set npm-404 banip 66.66.66.66 for example should create an entry “66.66.66.66 / Fail2Ban - jail npm-404”.
Log files used to test NPM rules with fail2ban-regex
File: /tmp/test-403.log
[31/Dec/2024:13:26:07 +0000] - - 403 - GET https transmission.one137.dev "/favicon.ico" [Client 172.18.0.1] [Length 111] [Gzip 1.35] [Sent-to ct-wire
guard-client] "Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Mobile/15E148 Safari/604.
1" "-"
[06/Jan/2025:09:20:30 +0100] - - 403 - HEAD https one137.dev "/test" [Client 192.168.1.66] [CF-conn-IP -] [Length 0] [Gzip -] [Sent-to STATIC_FILES]
"curl/8.7.1" "-"
[06/Jan/2025:09:22:03 +0100] - - 403 - GET https one137.dev "/test" [Client 172.21.0.2] [CF-conn-IP 197.130.148.156] [Length 111] [Gzip 1.35] [Sent-t
o STATIC_FILES] "Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Mobile/15E148 Safari/60
4.1" "-"
File: /tmp/test-404.log
[04/Jan/2025:21:27:41 +0000] - - 404 - GET https one137.dev "/abc" [Client 172.21.0.2] [CF-conn-IP 197.130.148.156] [Length 111] [Gzip 1.35] [Sent-to
STATIC_FILES] "Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0 Mobile/15E148 Safari/604
.1" "-"
[02/Jan/2025:22:46:04 +0000] - 404 404 - GET https portainer.one137.dev "/api/endpoints/2" [Client 192.168.1.66] [Length 168] [Gzip -] [Sent-to ct-po
rtainer] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "https://portainer.o
ne137.dev/"
[04/Jan/2025:21:25:57 +0000] - - 404 - GET https one137.dev "/abc" [Client 192.168.1.66] [CF-conn-IP -] [Length 172] [Gzip 3.21] [Sent-to STATIC_FILE
S] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-"
File: /tmp/test-forbidden.log
2025/01/02 22:46:54 [error] 254#254: *24053 access forbidden by rule, client: 172.21.0.2, server: one137.dev, request: "GET / HTTP/2.0", host: "one13
7.dev"
2025/01/02 22:46:54 [error] 254#254: *24053 access forbidden by rule, client: 172.20.0.3, server: one137.dev, request: "GET / HTTP/2.0", host: "one13
7.dev"
File: /tmp/test-protocol.log
[04/Jan/2025:23:40:05 +0100] 400 - - http localhost "-" [Client 190.168.1.66] [Length 154] [Gzip -] "-" "-"
[04/Jan/2025:23:40:05 +0100] 400 - - http localhost "-" [Client 190.168.1.66] [Length 154] [Gzip -] "-" "-"
[04/Jan/2025:23:39:57 +0100] 444 - GET https localhost "/" [Client 192.168.1.66] [Length 0] [Gzip -] "-" "-"
[04/Jan/2025:23:39:57 +0100] 444 - GET https localhost "/" [Client 192.168.1.66] [Length 0] [Gzip -] "-" "-"
Generate 403 errors
If no real 403 route is configured, add in an NPM proxy host:
location /test-403 {
return 403;
access_log /data/logs/proxy-host-1_access.log default_plus_cf;
}
Generate protocol errors
400: Bad Request (malformed requests, invalid TLS) errors can be generated using
openssl s_client -connect one137.dev:443 -no_tls1_2 -no_tls1_3 -quiet
C03DC746F87F0000:error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available:ssl/statem/statem_lib.c:153
444: Connection Closed (suspicious TLS behavior) errors with
echo -n "GET / HTTP/1.0\r\n\r\n" | openssl s_client -connect one137.dev:443 -quiet
C03DC746F87F0000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:693