When to restart services after package updating.
When updating Linux packages with a package manager it is occasionally necessary to identify services running, having file(s) open that have been unlinked from the directory tree, i.e. deleted.
This is most commonly caused by a process having a shared library(.so) file open that has been updated to a newer version.
Consider the case of the recent Openssl Heartbleed vulnerability. http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-0160
Updating Openssl via your package manager without restarting running services on your system relying on “libssl.so.*”, will most likely result in the system still being vulnerable to attacks.
In this post I’ll demonstrate a couple of ways to identify running services with open and unlinked files and automatically restarting them.
Finally I will share my application/script "needy-restart", for automatically identifying and restarting these services.
The following should work as is, on
RHEL, Oracle Linux, SL, CentOS Linux distributions.
For other Linux distributions, minor adjustments should be expected.
Here’s from a recent system update
Part of the “yum-utils” package has a python script to output processes that needs restarting, called “needs-restarting”
It is very useful for getting info helping determining what processes will need to be restarted.
This is what the script outputs on this system:
# needs-restarting
1869 : /sbin/dhclient-1-q-cf/etc/dhcp/dhclient-eth0.conf-lf/var/lib/dhclient/dhclient-eth0.leases-pf/var/run/dhclient-eth0.pideth0
2125 : /usr/bin/mimedefang-multiplexor-p/var/spool/MIMEDefang/mimedefang-multiplexor.pid-m2-x10-y0-Udefang-b600-l-s/var/spool/MIMEDefang/mimedefang-multiplexor.sock
2214 : crond
Unfortunately the script will not restart the services for us, so if we want that functionality we have to roll our own script.
In the following we are only interested in matching and restarting these services:
2125 : /usr/bin/mimedefang-multiplexor-p/var/spool/MIMEDefang/mimedefang-multiplexor.pid-m2-x10-y0-Udefang-b600-l-s/var/spool/MIMEDefang/mimedefang-multiplexor.sock
2214 : crond
The following should be restarted and verified by doing a system reboot.
1869 : /sbin/dhclient-1-q-cf/etc/dhcp/dhclient-eth0.conf-lf/var/lib/dhclient/dhclient-eth0.leases-pf/var/run/dhclient-eth0.pideth0
# If you’re feeling brave, issuing “service network restart” will do, but is *not* recommended on a production system without scheduled downtime.
Let’s make our own script.
Check for processes with open files that has been unlinked/deleted using “lsof”
We want to identify “DEL” or “deleted” in the 5th column of the output from lsof and filter out “/dev/zero” & “async I/O (AIO)”
# lsof |grep "DEL\|deleted" | grep -v "/dev/zero\|\[aio\]"
dhclient 1869 root txt REG 202,0 572256 18270 /sbin/dhclient (deleted)
dovecot 2006 root 122u REG 202,0 0 41056 /var/run/dovecot/login-master-notifye2e4a04bb9b2d719 (deleted)
dovecot 2006 root 125u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
mimedefan 2125 defang DEL REG 202,0 24600 /lib64/libfreebl3.so
crond 2214 root DEL REG 202,0 24600 /lib64/libfreebl3.so
imap-logi 17666 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 17671 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 17708 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18376 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18383 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18385 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18387 dovenull 4u REG 202,0 0 42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
We are interested in the two services having an unlinked shared library file open. In this case:
mimedefan 2125 defang DEL REG 202,0 24600 /lib64/libfreebl3.so
crond 2214 root DEL REG 202,0 24600 /lib64/libfreebl3.so
Let’s check what package was updated.
# yum whatprovides /lib64/libfreebl3.so |grep -B3 -A3 "installed"
nss-softokn-freebl-3.14.3-19.el6_6.x86_64 : Freebl library for the Network
: Security Services
Repo : installed
Matched from:
Other : Provides-match: /lib64/libfreebl3.so
So the package “nss-softokn-freebl-3.14.3-19.el66.x8664” was updated and two services need to be restarted to use the new shared library provided by this package.
But in the interest of coding a script to identifying and restarting these services, we’ll wait.
Let’s tighten the output so we can use it in a script.
# lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && !$seen{$F[0]}++)'
mimedefan <--- OUTPUT
crond <--- OUTPUT
Here we still filter out “/dev/zero” & “async I/O (AIO)” with the grep command, pipe the output to perl and print the first column if column “4” matches “DEL” or “deleted”. The last part is a perl emulation of “sort -uk1,1” showing only unique first column matches.
Much cleaner output and more suitable for use in a script.
Unfortunately the output can not be used directly to wrap in a for loop to restart, as “mimedefan” service restart is done like this: “service mimedefang restart”
Let’s grab the running service names from /var/lock/subsys/ and compare them to our previous output.
# proc=$(lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && ! $seen{$F[0]}++)') ; \
if ! [ -z "$proc" ]; then ls /var/lock/subsys/ |grep "$proc"; else echo "No service needs restart."; fi
crond <--- OUTPUT
mimedefang <--- OUTPUT
Now we have process names we can use to restart with the service command.
Let’s test it by issuing “service <servicename> status” on these processes first.
# proc=$(lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && ! $seen{$F[0]}++)') ; \
if ! [ -z "$proc" ]; then ls /var/lock/subsys/ |grep "$proc"| xargs -I{} service {} status; else echo "No service needs restart."; fi
crond (pid 2214) is running... <--- OUTPUT
mimedefang (pid 2142) is running... <--- OUTPUT
mimedefang-multiplexor (pid 2125) is running... <--- OUTPUT
0/10 .......... 0 <--- OUTPUT
Now we have a working one-liner that clearly is begging to be put in a script.
Let’s construct a bash script to identify and restart service(s) if needed after updating of packages.
needy-restart script
Below is the finished script.
#!/bin/sh
# Version 0.6
# needy-restart: Identifying and restarting running services
# with open files unlinked/deleted from the directory tree.
#
# Copyright (C) 2015 Lars Bjaerris <lars@bjaerris.com>
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# -------------------------------------------------------------------
proc=$(lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && ! $seen{$F[0]}++)')
fini=0
while [ "$#" -gt 0 ]
do
case "$1" in
-r)
if ! [ -z "$proc" ] # Check that we're not grep'ing with an empty string
then
echo "Restarting needy restart(s):"; ls /var/lock/subsys/ |grep "$proc" |xargs -I{} service {} restart
else
echo "No needy restart(s)"
fini=1 #
fi
;;& # Continue with the last check of "-v" unless $fini -eq 1.
-v|-r)
if [ "$fini" -eq 1 ]
then
exit
elif ! [ -z "$proc" ]
then
echo "Checking for needy restart(s):"; ls /var/lock/subsys/ |grep "$proc"
echo "If processes are stil listed below after running "needy-restart -r","
echo "you should evaluate them and consider if a reboot is necessary."
echo "-------------------------------------------------------------------------------"
lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && !$seen{$F[0]}++)'
else
echo "No needy restart(s)"
fi
;;
-*)
echo >&2 "usage: $0 [-v] [-r] -v: Check for needy restart(s) or -r: Restart needy restart(s)"
exit 1;;
*)
echo >&2 "usage: $0 [-v] [-r] -v: Check for needy restart(s) or -r: Restart needy restart(s)"
break;; # terminate while loop
esac
shift
done
Let's test it
# vi needy-restart
Paste!
Set the execute bit.
#chmod +x needy-restart
Execute.
# ./needy-restart -v
Checking for needy-restart(s): <--- OUTPUT
crond <--- OUTPUT
mimedefang <--- OUTPUT
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
Give the restart service(s) option.
# ./needy-restart -r
Restarting needy-restart(s):
Stopping crond: [ OK ]
Starting crond: [ OK ]
Shutting down mimedefang: [ OK ]
Shutting down mimedefang-multiplexor: [ OK ]
Waiting for daemons to exit
Checking filter syntax: OK
Starting mimedefang-multiplexor: [ OK ]
Starting mimedefang: [ OK ]
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
Final check.
# ./needy-restart -v
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
EDIT:
Here is an example using "needy-restart after updating: glibc-* & kernel-headers.
# ./needy-restart -r
Restarting needy-restart(s):
Stopping auditd: [ OK ]
Starting auditd: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Stopping Dovecot Imap: [ OK ]
Starting Dovecot Imap: [ OK ]
Stopping crond: [ OK ]
Starting crond: [ OK ]
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
Stopping nginx: [ OK ]
Starting nginx: [ OK ]
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
Stopping php-fpm: [ OK ]
Starting php-fpm: [ OK ]
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
Stopping fail2ban: [ OK ]
Starting fail2ban: [ OK ]
Checking for needy-restart(s):
auditd
sshd
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
init
udevd
dhclient
auditd
portreser
mimedefan
agetty
mingetty
npm
node
sshd
bash
pickup
master
qmgr
tlsmgr
In this case we have to reboot the server.
After Reboot testing
# ./needy-restart -v
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
Hope you enjoyed it!
Lars Bjaerris