How To Resolve DNS Server Abuse
One day, I noticed that my CentOS system log file (which is located at /var/log/messages) was filled with thousands of messages that looked like this:
16-Dec-2019 08:04:24.451 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
16-Dec-2019 08:04:24.452 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
16-Dec-2019 08:04:24.453 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
16-Dec-2019 08:04:24.454 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
After a little bit of research, I determined that this was a consequence of my Domain Name Server (DNS) server process, named, was being abused. Anonymous clients around the internet were using my DNS server to resolve IP addresses for any domain, not just the ones that I was responsible for.
This is a holdover from when the Internet was a kinder and gentler place. In the old days, a client could connect to any DNS server they wanted to resolve a name to an IP address, because name servers were complicated and rare and bandwidth was relatively scarce. But, for decades now, Google has maintained two very high-quality and well-publicized DNS servers at 8.8.8.8 and 8.8..4.4, so there's no need for anyone to ask little old me to be resolving a name beyond the domains for which I am responsible.
In this more cynical and hard-edged Internet age, open DNS servers like mine are being abused to power various nefarious schemes, including spam and denial of service attacks.
So I decided to do something about this situation, and use the fail2ban package to monitor DNS and block clients who made too many requests within a given period of time.
How To Set Up DNS for Abuse Prevention
The first system to make adjustments to is named, the DNS service offered by bind.
Clearing Up Some Naming Convention Confustion
Unfortunately, DNS is a bit of a confusing situation with bunch of different names involved. It's a long and old story, but the quick version is this:
DNS - the name resolution standard defined in a document called an Request For Comment (RFC)
bind - the software package that implemented the standard outlined in the RFC
named - the name of the running service that offers the services outlined in the RFC
(re)Configuring named
The configuration file for named is located at /etc/named.conf. It contains a lot of information, but for this exercise we are only going to look at a small amount of settings
1) Turn off named recursion
2) Segregate named logging to a dedicated logfile
How To Turn off named recursion
Recursion in named is the settng that enables anonymous clients to resolve any name using your server. These days, this is not an advisable setting to turn on, and it is easy to disable:
Here's what my named.conf looked like when I was done:
/*
- If your recursive DNS server has a public IP address, you MUST enable access
control to limit queries to your legitimate users. Failing to do so will
cause your server to become part of large scale DNS amplification
attacks. Implementing BCP38 within your network would greatly
reduce this attack surface
*/
#
# GL 2018-02-02 Uncomment ONE of the following
#
recursion no;
# recursion yes;
How To Segregate named Logging To A Dedicated Logfile
The next thing to do is configure named so it logs its messages to a segregated logfile, which makes system monitoring easier in some ways, because the logfile isn't flooded with named related messaging.
Thankfully, there's some instructions embedded in the fail2ban package that help out with this task. Here they are:
# Fail2Ban filter file for named (bind9).
#
# This filter blocks attacks against named (bind9) however it requires special
# configuration on bind.
#
# By default, logging is off with bind9 installation.
#
# You will need something like this in your named.conf to provide proper logging.
#
# logging {
# channel security_file {
# file "/var/log/named/security.log" versions 3 size 30m;
# severity dynamic;
# print-time yes;
# };
# category security {
# security_file;
# };
# };
Let's look at the existing named logging situation:
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
};
};
So, what's in the data/named.run file?
16-Dec-2019 08:04:24.451 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
16-Dec-2019 08:04:24.452 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
16-Dec-2019 08:04:24.453 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
16-Dec-2019 08:04:24.454 client: query (cache) '236.31.230.157.in-addr.arpa/PTR/IN' denied
So it looks like this file also contains messages related to named. Let's go ahead and implement the changes required by fail2ban.
Here's what my named.conf file looked like when I was done:
/*
#
# GL 2019-12-16 commented out as an experiment
#
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
};
};
*/
logging {
channel security_file {
file "/var/log/named/security.log" versions 3 size 30m;
severity dynamic;
print-time yes;
};
category security {
security_file;
};
};
SELINUX / CentOS Peculiarities
Because SELINUX is a more stringent security setting than most Linux distributions, a couple of other things need to be done to get this solution working.
Who is DNS?
One of the things that needs to be determined is the security context of the DNS server. This is established in a couple of places, but the easiest way I know is to scan the /etc/passwd file, which contains a list of all of the users in the system, including services.
And indeed, this line appeared in /etc/passwd:
named:x:25:25:Named:/var/named:/sbin/nologin
Now we know that there is a named user context on this machine. Next thing is to check the user context of the running service. This can be done with the ps command:
# ps ax | grep "named"
5907 ? Ssl 1:12 /usr/sbin/named -u named -c /etc/named.conf
This output confirms that the username named is being used by DNS service on this machine via the -u parameter (-u named).
Directory Work
OK, with our detective work done, we can go ahead and do the following:
- Create a named subdirectory in the /var/log directory.
- Assign ownership of the named subdirectory to the user context of the DNS server process (named).
- Change the security context of the named subdirectory to that of a named log
- Allow other processes (like fail2ban) permission to enter that directory
Here's how I did it:
# mkdir named
# chown named:named named
# chcon system_u:object_r:named_log_t:s0 named
# chmod 755 named
Here's the result of those actions:
drwxr-xr-x. named named system_u:object_r:named_log_t:s0 named
Those commands created a directory called named owned (and therefore manipulatable) by the user named (the user context of the DNS server) in the security context of a named log, that may be read and entered into by other processes, like fail2ban.
File Work
Now that the named subdirectory has been prepared, a file needs to be created in it for the named service to write to, and for fail2ban to scan to determine who to ban.
Here's how I did it:
# cd named
# touch security.log
# chown named:named security.log
# chcon system_u:object_r:named_log_t:s0 security.log
# chmod 644 security.log
Here's the result of those actions:
-rw-r--r--. named named system_u:object_r:named_log_t:s0 security.log
Those commands created a file called security.log owned (and therefore manipulatable) by the user named (the user context of the DNS server) in the security context of a named log, that may also be read by other processes like fail2ban.
How To (re)Start The name Service
With all of the preliminary work done, it's time to restart the named service and see if everything was set up properly.
Here's what you will see if there's a configuration problem:
# service named restart
Redirecting to /bin/systemctl restart named.service
Job for named.service failed because the control process exited with error code. See "systemctl status named.service" and "journalctl -xe" for details.
Here's what you will see if everything is OK:
# service named restart
Redirecting to /bin/systemctl restart named.service
#
Here's what you will see if there's a configuration problem:
# service named restart
Redirecting to /bin/systemctl restart named.service
Job for named.service failed because the control process exited with error code. See "systemctl status named.service" and "journalctl -xe" for details.
Here's what you will see if everything is OK:
# service named restart
Redirecting to /bin/systemctl restart named.service
#
How To Check The named Server Is Running
Checking to make sure that the named server is running is pretty easy with the service <service_name> status command:
# service named status
Redirecting to /bin/systemctl status named.service
● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-12-16 09:43:19 HKT; 5min ago
Process: 10192 ExecStop=/bin/sh -c /usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 10208 ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF} $OPTIONS (code=exited, status=0/SUCCESS)
Process: 10204 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)
Main PID: 10209 (named)
CGroup: /system.slice/named.service
└─10209 /usr/sbin/named -u named -c /etc/named.conf
Now that we have named running correctly and its messaging related to DNS abusers segregated into a dedicated logfile (/var/log/named/security.log), we can configure fail2ban to scan that logfile and start banning DNS abusers!
Redirecting to /bin/systemctl status named.service
● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-12-16 09:43:19 HKT; 5min ago
Process: 10192 ExecStop=/bin/sh -c /usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 10208 ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF} $OPTIONS (code=exited, status=0/SUCCESS)
Process: 10204 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)
Main PID: 10209 (named)
CGroup: /system.slice/named.service
└─10209 /usr/sbin/named -u named -c /etc/named.conf
How To Configure fail2ban To Prevent DNS Server Abuse
Now that we have named running correctly and its messaging related to DNS abusers segregated into a dedicated logfile (/var/log/named/security.log), we can configure fail2ban to scan that logfile and start banning DNS abusers!
The first place to visit is /etc/fail2ban/jail.local, which is where local fail2ban policies reside. Here's what I found:
# !!! WARNING !!!
# Since UDP is a connection-less protocol, spoofing of IP and imitation of illegal
# actions is way too simple. Thus enabling of this filter might provide an easy
# way to implementing a DoS against a chosen victim.
#
# See: http://nion.modprobe.de/blog/archives/690-fail2ban-+-dns-fail.html
#
# Please DO NOT USE this jail unless you know what you are doing.
#
# IMPORTANT: see filter.d/named-refused for instructions to enable logging
#
# This jail blocks UDP traffic for DNS requests.
#
# [named-refused-udp]
# enabled = true
# filter = named-refused
# port = domain,953
# protocol = udp
# logpath = /var/log/named/security.log
# IMPORTANT: see filter.d/named-refused for instructions to enable logging
# This jail blocks TCP traffic for DNS requests.
#
# GL 2019-12-15 Enabled the following
#
[named-refused]
enabled = true
port = domain,953
logpath = /var/log/named/security.log
#
# GL 2019-12-15 Enabled the following
#
[named-refused-udp]
enabled = true
#
# GL 2019-12-15 Enabled the following
#
[named-refused-tcp]
enabled = true
How To Resolve fail2ban SELINUX Issues
Dec 16 10:02:17 vm setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the file disable. For complete SELinux messages run: sealert -l 4f0b9ddb-62c0-4568-a19a-ada24825e993
# sealert -l 4f0b9ddb-62c0-4568-a19a-ada24825e993
SELinux is preventing /usr/bin/python2.7 from read access on the file disable.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that python2.7 should be allowed read access on the disable file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do allow this access for now by executing:
# ausearch -c 'fail2ban-server' --raw | audit2allow -M my-fail2banserver
# semodule -i my-fail2banserver.pp
Additional Information:
Source Context system_u:system_r:fail2ban_t:s0
Target Context system_u:object_r:sysfs_t:s0
Target Objects disable [ file ]
Source fail2ban-server
Source Path /usr/bin/python2.7
Port <Unknown>
Host vm.yougrow.net
Source RPM Packages python-2.7.5-86.el7.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-252.el7_7.6.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name vm.yougrow.net
Platform Linux vm.yougrow.net 3.10.0-1062.9.1.el7.x86_64 #1
SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64
Alert Count 33
First Seen 2019-12-15 10:24:17 HKT
Last Seen 2019-12-16 10:02:14 HKT
Local ID 4f0b9ddb-62c0-4568-a19a-ada24825e993
Raw Audit Messages
type=AVC msg=audit(1576461734.872:303365): avc: denied { read } for pid=12096 comm="fail2ban-server" name="disable" dev="sysfs" ino=2085 scontext=system_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1576461734.872:303365): arch=x86_64 syscall=open success=no exit=EACCES a0=7fcb234bf110 a1=80000 a2=1b6 a3=24 items=0 ppid=1 pid=12096 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=fail2ban-server exe=/usr/bin/python2.7 subj=system_u:system_r:fail2ban_t:s0 key=(null)
Hash: fail2ban-server,fail2ban_t,sysfs_t,file,read
Implementing the policies:
# ausearch -c 'fail2ban-server' --raw | audit2allow -M my-fail2banserver
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i my-fail2banserver.pp
# semodule -i my-fail2banserver.pp
#
Behind the scenes, this was going in in /var/log/messages:
Dec 16 10:04:57 vm systemd: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 12436 (find)
Dec 16 10:04:57 vm systemd: Mounting Arbitrary Executable File Formats File System...
Dec 16 10:04:57 vm systemd: Mounted Arbitrary Executable File Formats File System.
Dec 16 10:37:54 vm kernel: SELinux: Converting 2406 SID table entries...
Dec 16 10:37:57 vm dbus[953]: [system] Reloaded configuration
Checking fail2ban again now..
DID NOT RESOLVE ISSUE
DID NOT RESOLVE ISSUE
Rotating The New named Security Log
How To List The Contents Of fail2ban Jails
#!/bin/bash
###########################################################################################
#
# fail2ban-jails.bash
#
# This program lists all of the active jails in the FAIL2BAN system.
#
# GL 2019-12-04 fail2ban-jails.bash
#
###########################################################################################
JAILS=`fail2ban-client status | grep "Jail list" | sed -E 's/^[^:]+:[ \t]+//' | sed 's/,//g'`
for JAIL in $JAILS
do
fail2ban-client status $JAIL
done
