most common reasons why syslog won't work

Most Common Reasons Why Syslog Won’t Work 2022

There have been numerous of times when I installed a Ubuntu or Red Hat machine with Syslog all to find out later that Syslog just won’t work.

Many times I encountered that Syslog will not write any log files from remote sources.

This can be very frustrating because often it is hard to troubleshoot.
I’ll try to sort you out, let’s begin troubleshooting syslog.

First things first.

For the sake of this blog post I’m running Red Hat.
However, if you run Fedora, Ubuntu or another version, that’s fine.
The concepts apply to most distributions.

  1. First, check if RSyslog is running, the obvious.
Copy Code
[dominic@localhost ~]$ sudo systemctl status rsyslog
[sudo] password for dominic: 
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-08-20 18:03:31 CEST; 11min ago
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 3055 (rsyslogd)
      Tasks: 5 (limit: 23284)
     Memory: 3.0M
        CPU: 44ms
     CGroup: /system.slice/rsyslog.service
             └─3055 /usr/sbin/rsyslogd -n

Aug 20 18:03:31 localhost.localdomain systemd[1]: Starting System Logging Service...
Aug 20 18:03:31 localhost.localdomain systemd[1]: Started System Logging Service.

If it’s not running, start it with: sudo systemctl start rsyslog
If that won’t start it either, triple check your /etc/rsyslog.conf file.
There must a mistake in there.
Also the messages you’ll find in the lower end of your screen will tell you where the problem could be.

If you can’t find it, reinstall rsyslog in order to get a clean config file and go from there.
Check if it’s running and incrementally adjust the config file and reload the Syslog deamon each time you make an adjustment.

2. Check if Syslog packet’s are coming in.

Start TCP dump and check if (UDP) packets are coming in.
When that is verified you can continue.
If you troubleshoot ghosts you will be busy sometime.

Copy Code
[dominic@localhost ~]$ sudo tcpdump udp port 514
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes

18:19:27.953184 IP Monster-FGT60D.domain_not_set.invalid.6a44 > localhost.localdomain.syslog: SYSLOG local7.notice, length: 708
18:19:30.924493 IP Monster-FGT60D.domain_not_set.invalid.6a44 > localhost.localdomain.syslog: SYSLOG local7.notice, length: 708
18:19:33.869063 IP Monster-FGT60D.domain_not_set.invalid.6a44 > localhost.localdomain.syslog: SYSLOG local7.notice, length: 708
18:19:34.499210 IP Monster-FGT60D.domain_not_set.invalid.6a44 > localhost.localdomain.syslog: SYSLOG local7.notice, length: 665
18:19:34.500527 IP Monster-FGT60D.domain_not_set.invalid.6a44 > localhost.localdomain.syslog: SYSLOG local7.notice, length: 663
18:19:36.669901 IP Monster-FGT60D.domain_not_set.invalid.6a44 > localhost.localdomain.syslog: SYSLOG local7.notice, length: 663
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

3. Check your firewall(s)!

One thing that kept me busy is checking the firewalls.
Yes, multiple.
To be exact, UFW, IPtables and firewalld.

Verify that all firewalls allow the traffic of shut them down.
Only shut your firewalls down for testing purposes only!

Copy Code
####### IPTables

[dominic@localhost ~]$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

####### Firewalld

[dominic@localhost ~]$ sudo systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-08-20 17:35:38 CEST; 47min ago
       Docs: man:firewalld(1)
   Main PID: 1004 (firewalld)
      Tasks: 2 (limit: 23284)
     Memory: 40.0M
        CPU: 455ms
     CGroup: /system.slice/firewalld.service
             └─1004 /usr/bin/python3 -s /usr/sbin/firewalld --nofork --nopid

Aug 20 17:35:38 localhost systemd[1]: Starting firewalld - dynamic firewall daemon...
Aug 20 17:35:38 localhost systemd[1]: Started firewalld - dynamic firewall daemon.

####### Stopping Firewalld

[dominic@localhost ~]$ sudo systemctl stop firewalld

[dominic@localhost log]$ ufw
bash: ufw: command not found...

Clearly UFW is not installed.
IPtables is configured to allow everything.
Firewalld is this case was causing problems because it was dropping packets.

You should configure one firewall and restrict network access via that firewall.
Never leave a host unprotected!

4. RPF, or Reverse Path Forwarding issues

If you have a server with multiple NIC cards you can enter a situation in where your server drops packets because of the RPF feature.
To fully understand RPF, you must have some network knowledge, and for the sake of this discussion I won’t go in there.

But to be brief, if your server routes packet A with destination X, yet it receives a packet with source X on a different interface than your server will route the packet too, it will discard the packet, considering it spoofed or unsafe.

How to check if RPF is turned on?

Copy Code
$ sysctl -a 2>/dev/null | grep "\.rp_filter"
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.enp0s31f6.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.tun0.rp_filter = 0
net.ipv4.conf.virbr0.rp_filter = 0
net.ipv4.conf.virbr0-nic.rp_filter = 0
net.ipv4.conf.virbr1.rp_filter = 0
net.ipv4.conf.virbr1-nic.rp_filter = 0
net.ipv4.conf.wlp58s0.rp_filter = 0

Now, if net.ipv4.conf.all.rp_filter = 1 that means it will discard the packet if the frame is coming from a unexpected interface.

Let’s turn it off.

Copy Code
sysctl -w "net.ipv4.conf.all.rp_filter=0"

5. Permissions

Sometimes it’s just plain old permissions.
However, I don’t encounter that many times.

But in order to troubleshoot why Syslog is not writing the logging from remote sources you can change the permissions for the sake of testing.

Copy Code
[dominic@localhost log]$ ls -alrt syslog
-rwxrwxrwx. 1 dominic root 8266777 Aug 20 18:41 syslog
[dominic@localhost log]$ sudo chmod 777 syslog
[sudo] password for dominic: 
[dominic@localhost log]$ ls -alrt syslog
-rwxrwxrwx. 1 dominic root 8271912 Aug 20 18:42 syslog
[dominic@localhost log]$

Don’t forget to change the permission after testing!

6. Check if Syslog is listening on your NIC.

Doublecheck if your server is listening to inbound Syslog messages.
If this command doesn’t show any results it will never work.
Still, even if this output shows it is listening, ingress Syslog traffic can still be filtered by the firewall.
Just keep that mind.

Copy Code
[dominic@localhost log]$ sudo netstat -tuapen | grep 514
[sudo] password for dominic: 
udp        0      0 0.0.0.0:514             0.0.0.0:*                           0          36479      3055/rsyslogd       
udp6       0      0 :::514                  :::*                                0          36480      3055/rsyslogd       
[dominic@localhost log]$

If this article helped you, make sure to leave a comment.
A comment is always appreciated 👌

If you are still stuck, let me know below and maybe I can help you out.