<![CDATA[bjaerris.com]]>https://bjaerris.com/Ghost 0.6Thu, 01 Jan 2026 19:46:37 GMT60<![CDATA[Rename Multiple Filenames With Different Date Formats Using Bash And Regular Expressions (Regex) Captures/Substitutions]]>A common way of batch renaming filenames on the command-line, is to use the "mv" & "sed" commands with a "for loop", or using "find" with the exec option.
Here I'll show a couple of examples utilizing both options.

Here is a typical rename example converting uppercase "JPG" filename extensions

]]>https://bjaerris.com/rename-multiple-filenames-with-different-date-formats-using-bash-and-regular-expressions-regex-capturessubstitutions-2/860e0698-3dde-4eee-b63f-f0d44e426e56Sun, 05 Apr 2015 12:14:41 GMTA common way of batch renaming filenames on the command-line, is to use the "mv" & "sed" commands with a "for loop", or using "find" with the exec option.
Here I'll show a couple of examples utilizing both options.

Here is a typical rename example converting uppercase "JPG" filename extensions to lowercase "jpeg" using "mv", "sed" and command substitution.

Create test folder and populate it with a couple of test files.

$ mkdir test_folder && cd test_folder/
$ touch {1..3}.JPG
$ ls -1
1.JPG
2.JPG
3.JPG

Batch rename the files with "JPG" filename extension to "jpeg" filename extension, using sed.

$ for i in *.JPG; do mv "$i" "$(echo "$i" | sed 's/^\(.*\)\.JPG$/\1\.jpeg/')"; done
$ ls -1
1.jpeg
2.jpeg
3.jpeg

For a more elegant and faster solution using Bash parameter expansion/substitution.

$ for i in *.jpeg; do mv "$i" "${i%.jpeg}.jpg"; done
$ ls -1
1.jpg
2.jpg
3.jpg

This works well for a single folder containing files we want to rename.
If we have several nested folders with files we want to rename, using the find command and its exec option is a good choice.

This time we'll add another step and move every file we rename into a folder called "jpeg_folder"
Let's clear out our test folder, create subfolders and populate these with test files.

$ rm *
$ mkdir -p Subfolder.{1..5}
$ ls -1
Subfolder.1
Subfolder.2
Subfolder.3
Subfolder.4
Subfolder.5

Populate the folders with unique filenames and create the "jpeg_folder"

$ COUNT=1 ; for i in $(ls -1) ; do touch ./$i/$COUNT.{1..5}.JPG && ((COUNT++));done
$ mkdir jpeg_folder

That gives us the the following test scenario to work with

$ ls -1R
Subfolder.1
Subfolder.2
Subfolder.3
Subfolder.4
Subfolder.5
jpeg_folder

./Subfolder.1:
1.1.JPG
1.2.JPG
1.3.JPG
1.4.JPG
1.5.JPG

./Subfolder.2:
2.1.JPG
2.2.JPG
2.3.JPG
2.4.JPG
2.5.JPG

./Subfolder.3:
3.1.JPG
3.2.JPG
3.3.JPG
3.4.JPG
3.5.JPG

./Subfolder.4:
4.1.JPG
4.2.JPG
4.3.JPG
4.4.JPG
4.5.JPG

./Subfolder.5:
5.1.JPG
5.2.JPG
5.3.JPG
5.4.JPG
5.5.JPG

./jpeg_folder:

Let's use the "find" command to recursively find all files with a ".JPG" file extension and rename them with a ".jpeg" file extension, while also moving them into the "jpeg_folder"

$ find * -name "*.JPG" -exec sh -c 'mv  "$0" "jpeg_folder/$(basename "${0%.JPG}.jpeg")"' {} \;
$ ls -1R
Subfolder.1
Subfolder.2
Subfolder.3
Subfolder.4
Subfolder.5
jpeg_folder

./Subfolder.1:

./Subfolder.2:

./Subfolder.3:

./Subfolder.4:

./Subfolder.5:

./jpeg_folder:
1.1.jpeg
1.2.jpeg
1.3.jpeg
1.4.jpeg
1.5.jpeg
2.1.jpeg
2.2.jpeg
2.3.jpeg
2.4.jpeg
2.5.jpeg
3.1.jpeg
3.2.jpeg
3.3.jpeg
3.4.jpeg
3.5.jpeg
4.1.jpeg
4.2.jpeg
4.3.jpeg
4.4.jpeg
4.5.jpeg
5.1.jpeg
5.2.jpeg
5.3.jpeg
5.4.jpeg
5.5.jpeg

As you can see from the previous example, the "find" command can be a very effective tool when paired with a bit Bash command substitution and parameter expansion/substitution.

Batch Renaming Files With Different Date Formats

That brings us to files containing different date formats in the filename.
We usually have two common formats, depending on which side of the Atlantic we're on.
Below are those, formatted with the "date" command
So for the "13th of March 2015":

# MDY date format (Month, Day, Year)
$ date +"%m-%d-%Y"
03-13-2015

# DMY date format (Day, Month, Year)
$ date +"%d-%m-%Y"
13-03-2015

Let's convert files with either of these two date formats to your choice of format.
Again let's clear our test folder and populate it with a few test files in which the MDY date format is included in the filename.

$ rm -rf *
$ touch somefilename-03-{13..15}-2015.txt  
$ ls -1
somefilename-03-13-2015.txt
somefilename-03-14-2015.txt
somefilename-03-15-2015.txt

Let's convert those files to DMY format using sed in a "for loop"

$ for i in *.txt; do mv "$i" "$(echo $i|sed -E 's/(somefilename)\-([0-9]{2})\-([0-9]{2})\-([0-9]{4})(\.txt)/\1-\3-\2-\4\5/')"; done
$ ls -1
somefilename-13-03-2015.txt
somefilename-14-03-2015.txt
somefilename-15-03-2015.txt

For a faster version, converting it back to MDY format without the use of command substitution, sed or "piping", using Bash regular expressions.

$ for i in *.txt; do if [[ ${i} =~ (somefilename)-([0-9]{2})-([0-9]{2})-([0-9]{4})(\.txt) ]] ; then mv "$i" "${BASH_REMATCH[1]}-${BASH_REMATCH[3]}-${BASH_REMATCH[2]}-${BASH_REMATCH[4]}${BASH_REMATCH[5]}"; fi; done
$ ls -1
ls -1
somefilename-03-13-2015.txt
somefilename-03-14-2015.txt
somefilename-03-15-2015.txt

We can then convert back and forth between either format by running the same command again.

$ for i in *.txt; do if [[ ${i} =~ (somefilename)-([0-9]{2})-([0-9]{2})-([0-9]{4})(\.txt) ]] ; then mv "$i" "${BASH_REMATCH[1]}-${BASH_REMATCH[3]}-${BASH_REMATCH[2]}-${BASH_REMATCH[4]}${BASH_REMATCH[5]}"; fi; done
$ ls -1
somefilename-13-03-2015.txt
somefilename-14-03-2015.txt
somefilename-15-03-2015.txt

Voila! Hope you enjoyed it!
Lars Bjaerris

]]><![CDATA[Testing Destination Mailserver With Telnet, Netcat And Bash Scripting SMTP/ESMTP]]>Testing email delivery against a destination/receiving mailserver, using telnet, netcat (ncat, nc) and bash scripting.

After installing a border/destination/incoming MTA, or for that matter any "public" facing server, testing connectivity and functionality/operation from the viewpoint of a remote user/client should be your number one priority.

]]>https://bjaerris.com/testing-destination-mailserver-with-telnet-netcat-and-bash-scripting-smtpesmtp/65d48475-104c-433a-ab62-95a7a5422dc2Thu, 05 Mar 2015 14:09:31 GMTTesting email delivery against a destination/receiving mailserver, using telnet, netcat (ncat, nc) and bash scripting.

After installing a border/destination/incoming MTA, or for that matter any "public" facing server, testing connectivity and functionality/operation from the viewpoint of a remote user/client should be your number one priority.
Most TCP/IP servers can be tested with use of commonly available command-line tools such as "nc" and "telnet".

For the sake of this post I will demonstrate SMTP communication against border MTA(s), with the use of both telnet, nc (netcat) and a bash shell script.

Generally the purpose of a border/destination/incoming MTA is to perform checks against a set of rules during the SMTP conversation/connection, before handing over the email to a local delivery agent or relay to another MTA for further delivery.

Checks usually performed on a destination MTA:

  • Connecting IP address reputation checks (DNSBL lookups and return codes).
  • Scan for viruses or other malicious payloads.
  • Decide if the message is UBE (Spam).
  • Check if destination domain is one we accept email or relay for.
  • Check if user in the domain or local user is valid for delivery.

One thing to keep in mind when configuring a destination MTA; once a destination MTA accepts an email with a 2.x.x SMTP code (Success) that MTA is responsible for further delivery of that message.
Email messages containing viruses, spam etc. are almost always sent with spoofed/fake return email addresses, so ending/closing the SMTP conversation with the connecting MTA and later trying to bounce an email with a NDN (Non-Delivery Notification), will in the majority of cases end up as email backscatter. Also the risk of a malicious sender using your MTA to bounce Spam/Virus emails to other systems should be considered.
Degradation of your ip/domain reputation on DNSBLs and entry in http://www.backscatterer.org would likely be the eventual outcome of such a configuration.

Make sure to always test your MTA(s) for proper rejection to invalid users and domains during the SMTP conversation, making sure you do not allow a configuration that accepts and then bounces emails.

Let's test a good configuration

Let us test against the MX server handing email for my domain "bjaerris.com"
I have set up a valid email address for this purpose: "valid.email@bjaerris.com".

Let's get an MX record for the domain:

$ dig +short MX bjaerris.com
10 gw4.node25.com.

Let's connect to "gw4.node25.com" and talk some SMTP:

$ echo $HOSTNAME                               <---- Command
gw3.node25.com
$ telnet gw4.node25.com 25                     <---- Command
Trying 2a01:7e00::f03c:91ff:fe50:fb1a...
Connected to gw4.node25.com.
Escape character is '^]'.
220 gw4.node25.com ESMTP Postfix
HELO gw3.node25.com                            <---- Command
250 gw4.node25.com
MAIL FROM: <smtp.test@gw3.node25.com>          <---- Command
250 2.1.0 Ok
RCPT TO: <invalid@bjaerris.com>                <---- Command
550 5.1.1 <invalid@bjaerris.com>: Recipient address rejected: User unknown in virtual mailbox table
RCPT TO: <test@example.com>                    <---- Command
554 5.7.1 <test@example.com>: Relay access denied
RCPT TO: <valid.email@bjaerris.com>            <---- Command
250 2.1.5 Ok
DATA                                           <---- Command
354 End data with <CR><LF>.<CR><LF>
To: valid.email@bjaerris.com                   <---- Data
Subject: Test message                          <---- Data
Hello!                                         <---- Data
.                                              <---- "Data" (Will be interpreted as end of message by the SMTP server)
250 2.0.0 Ok: queued as F3B40498E
QUIT                                           <---- Command
221 2.0.0 Bye
Connection closed by foreign host.

To summarize the above "good configuration", testing for invalid user, relaying, and valid user.

RCPT TO: <invalid@bjaerris.com

"550 5.1.1 invalid@bjaerris.com: Recipient address rejected: User unknown in virtual mailbox table"

RCPT TO: <test@example.com>

"554 5.7.1 test@example.com: Relay access denied"

"RCPT TO: <valid.email@bjaerris.com>"

"250 2.1.5 Ok"

.

"250 2.0.0 Ok: queued as F3B40498E"

Testing with netcat (ncat, nc)

We could use Netcat the same way as the previous telnet example, but for this demonstration we're going to prepare a file with the input and redirect to stdin of the netcat process.
First let's create a file with the commands/data we want to send to the SMTP server.

$ cat > SMTP_TALK <<EOF
> HELO gw3.node25.com
> MAIL FROM:<smtp.test@gw3.node25.com>
> RCPT TO:<invalid@bjaerris.com>
> RCPT TO:<test@example.com>
> RCPT TO:<valid.email@bjaerris.com>
> DATA
> From: [SMTP TEST] <smtp.test@gw3.node25.com>
> To: <valid.email@bjaerris.com>
> Subject: Test message.
> Hello!
> .
> QUIT
> EOF

Use netcat to connect to the MX server and redirect output from our newly created file to stdin of the netcat process.

# Note on some systems Netcat is called "ncat"

$ nc gw4.node25.com 25 < SMTP_TALK
220 gw4.node25.com ESMTP Postfix
250 gw4.node25.com
250 2.1.0 Ok
550 5.1.1 <invalid@bjaerris.com>: Recipient address rejected: User unknown in virtual mailbox table
554 5.7.1 <test@example.com>: Relay access denied
250 2.1.5 Ok
354 End data with <CR><LF>.<CR><LF>
250 2.0.0 Ok: queued as 0BEF949CC
221 2.0.0 Bye

Having the commands ready in a file makes things a bit easier if testing more than one server.

Let's make a bash script for more permanent SMTP/ESMTP testing purposes

#!/bin/sh
# smtp_test.sh
# Lars Bjaerris <lars at bjaerris.com>
# Version 0.2

# Argument checks
if [[ ( $# == "--help" ||  $# == "-h") || ! ($# -eq 1 && $1 =~ ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}$) ]]; then
    echo -e "\nThis script takes an email address as argument." 
    echo -e "Usage:\n$0 user@example.com \n" 
    exit 1
fi
# Define variables
LOCAL_HOSTNAME=$(uname -n)
EMAIL_ADDR="$1"
REMOTE_LOCAL_ADDR="${EMAIL_ADDR%@*}"
REMOTE_DOMAIN="${EMAIL_ADDR#*@}"
REMOTE_MX=$(dig +short $REMOTE_DOMAIN MX |head -n1 |cut -d ' ' -f 2)
SMTP_PORT="25"
SENDER="smtp.test@$LOCAL_HOSTNAME"
MESSAGE="Hello!"
INVALID_RECIPIENT="invalid@$REMOTE_DOMAIN"
RELAY_TEST="test@example.com"
DATE=$(date '+%a, %d %b %Y %H:%M:%S %z')

# Check if an MX record was returned for the domain
echo 
[ -z "$REMOTE_MX" ] && echo "Failed to get MX record from \"$REMOTE_DOMAIN\"" && exit 1 || echo "Got MX:\"$REMOTE_MX\" for DOMAIN:\"$REMOTE_DOMAIN\""

# Populate array with SMTP commands to run
SMTP_commands=( \
    "HELO $LOCAL_HOSTNAME" \
    "MAIL FROM: <$SENDER>" \
    "RCPT TO: <$INVALID_RECIPIENT>" \
    "RCPT TO: <$RELAY_TEST>" \
    "RCPT TO: <$EMAIL_ADDR>" \
    "DATA" \
    "To: <$EMAIL_ADDR>\r\nFrom: [SMTP TEST] <$SENDER>\r\nSubject: Test Message.\r\nDate: $DATE\r\n$MESSAGE\r\n." \
    "quit"
)
# Define sending function
email_smtp () {
    echo "Trying connect to MX:\"$REMOTE_MX\""
    echo
    exec 3<>/dev/tcp/$REMOTE_MX/$SMTP_PORT
    read -u 3 reply
    echo "$reply"
    # Set internal field separator for "SMTP_commands" array looping
    IFS=""
    # Loop over "SMTP_commands" array
    for i in ${SMTP_commands[@]}
    do
        echo "----Sending Data:"
        echo "$i"
        echo -en "$i\r\n" >&3
        read -u 3 reply
        echo "----Server Reply:"
        echo "$reply"
    done
}
#Call senderfunction
email_smtp

Let's test it!

$ ./smtp_test.sh valid.email@bjaerris.com

Got MX:"gw4.node25.com." for DOMAIN:"bjaerris.com"
Trying connect to MX:"gw4.node25.com."

220 gw4.node25.com ESMTP Postfix
----Sending Data:
HELO gw3.node25.com
----Server Reply:
250 gw4.node25.com
----Sending Data:
MAIL FROM: <smtp.test@gw3.node25.com>
----Server Reply:
250 2.1.0 Ok
----Sending Data:
RCPT TO: <invalid@bjaerris.com>
----Server Reply:
550 5.1.1 <invalid@bjaerris.com>: Recipient address rejected: User unknown in virtual mailbox table
----Sending Data:
RCPT TO: <test@example.com>
----Server Reply:
554 5.7.1 <test@example.com>: Relay access denied
----Sending Data:
RCPT TO: <valid.email@bjaerris.com>
----Server Reply:
250 2.1.5 Ok
----Sending Data:
DATA
----Server Reply:
354 End data with <CR><LF>.<CR><LF>
----Sending Data:
To: <valid.email@bjaerris.com>\r\nFrom: [SMTP TEST] <smtp.test@gw3.node25.com>\r\nSubject: Test message.\r\nDate: Thu, 05 Mar 2015 14:05:10 +0000\r\nHello!\r\n.
----Server Reply:
250 2.0.0 Ok: queued as F0EBD498E
----Sending Data:
quit
----Server Reply:
221 2.0.0 Bye

Voila!

Hope you enjoyed it!
Lars Bjaerris

]]><![CDATA[Parsing Plaintext Logfiles On The Commandline Using Perl.]]>Perl, "the Swiss Army chainsaw of scripting languages"

Using Perl, regular expressions and sort to parse and format a plaintext log-file.
And finally comma separate the output (CSV format) for import into e.g. Excel.
The approach here can easily be modified to parse other plaintext files in the same

]]>https://bjaerris.com/parsing-plaintext-logfiles-on-the-commandline-using-perl/65fd22d5-ae92-4b52-9a90-175d4e297e8fMon, 02 Feb 2015 14:01:27 GMTPerl, "the Swiss Army chainsaw of scripting languages"

Using Perl, regular expressions and sort to parse and format a plaintext log-file.
And finally comma separate the output (CSV format) for import into e.g. Excel.
The approach here can easily be modified to parse other plaintext files in the same manner.

Here is a snippet of an Nginx access.log file example we are going to use for parsing:

$ cat ./access.log

216.218.206.66 - - [01/Feb/2015:05:35:11 +0000] "GET / HTTP/1.1" 403 162 "-" "-"
199.180.112.34 - - [01/Feb/2015:05:59:48 +0000] "GET //MyAdmin/scripts/setup.php HTTP/1.1" 404 56 "-" "-"
199.180.112.34 - - [01/Feb/2015:05:59:48 +0000] "GET //phpMyAdmin/scripts/setup.php HTTP/1.1" 404 56 "-" "-"
199.180.112.34 - - [01/Feb/2015:05:59:48 +0000] "GET //pma/scripts/setup.php HTTP/1.1" 404 56 "-" "-"
199.180.112.34 - - [01/Feb/2015:05:59:48 +0000] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" 404 56 "-" "-"
199.180.112.34 - - [01/Feb/2015:05:59:48 +0000] "GET //myadmin/scripts/setup.php HTTP/1.1" 404 56 "-" "-"
199.180.112.34 - - [01/Feb/2015:05:59:48 +0000] "GET /muieblackcat HTTP/1.1" 404 162 "-" "-"
111.251.50.236 - - [01/Feb/2015:07:17:47 +0000] "CONNECT mx0.mail2000.com.tw:25 HTTP/1.0" 400 166 "-" "-"
23.239.196.71 - - [01/Feb/2015:07:45:48 +0000] "GET / HTTP/1.1" 301 178 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)"
69.171.237.116 - - [01/Feb/2015:11:06:01 +0000] "GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1" 301 178 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
69.171.237.116 - - [01/Feb/2015:11:06:02 +0000] "GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1" 200 57525 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
77.37.231.208 - - [01/Feb/2015:11:26:03 +0000] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0"
173.252.110.115 - - [01/Feb/2015:11:31:20 +0000] "GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1" 301 178 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
173.252.110.119 - - [01/Feb/2015:11:31:21 +0000] "GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1" 200 57525 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
61.240.144.66 - - [01/Feb/2015:12:42:47 +0000] "GET / HTTP/1.0" 200 612 "-" "masscan/1.0 (https://github.com/robertdavidgraham/masscan)"
108.166.85.126 - - [01/Feb/2015:16:05:43 +0000] "GET /admin/config.php HTTP/1.0" 499 0 "-" "-"
23.23.38.251 - - [01/Feb/2015:16:24:00 +0000] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB6 (.NET CLR 3.5.30729)"
23.23.38.251 - - [01/Feb/2015:16:24:01 +0000] "GET / HTTP/1.1" 200 7105 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB6 (.NET CLR 3.5.30729)"
185.5.51.50 - - [01/Feb/2015:16:56:48 +0000] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:50 +0000] "GET / HTTP/1.1" 200 7105 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:50 +0000] "GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1" 200 9977 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:51 +0000] "GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1" 200 39386 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:51 +0000] "GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1" 200 2698 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:51 +0000] "GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1" 200 3075 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:51 +0000] "GET /content/images/2015/01/lars.jpg HTTP/1.1" 200 47097 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:56:51 +0000] "GET /assets/fonts/casper-icons.woff HTTP/1.1" 200 2260 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
185.5.51.50 - - [01/Feb/2015:16:57:20 +0000] "GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1" 200 6642 "https://bjaerris.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53"
207.34.25.76 - - [01/Feb/2015:18:01:13 +0000] "GET /robots.txt HTTP/1.1" 301 178 "-" "R6_CommentReader(www.radian6.com/crawler)"
71.11.195.254 - - [01/Feb/2015:21:37:39 +0000] "GET /tmUnblock.cgi HTTP/1.1" 400 166 "-" "-"
175.44.8.98 - - [01/Feb/2015:22:12:06 +0000] "POST /ghost_linux_init_script/ HTTP/1.1" 301 178 "-" "-"
78.133.20.10 - - [01/Feb/2015:22:42:33 +0000] "GET / HTTP/1.1" 200 72 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
78.133.20.10 - - [01/Feb/2015:22:42:33 +0000] "GET /favicon.ico HTTP/1.1" 404 162 "https://gw4.node25.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
31.202.241.242 - - [02/Feb/2015:01:15:13 +0000] "GET / HTTP/1.0" 301 178 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
207.145.97.131 - - [02/Feb/2015:04:00:25 +0000] "GET / HTTP/1.1" 400 166 "-" "-"
207.145.97.131 - - [02/Feb/2015:04:00:26 +0000] "GET //Net_work.xml HTTP/1.1" 400 166 "-" "-"
198.20.69.74 - - [02/Feb/2015:06:18:10 +0000] "GET / HTTP/1.1" 403 162 "-" "-"
198.20.69.74 - - [02/Feb/2015:06:18:10 +0000] "GET /robots.txt HTTP/1.1" 403 162 "-" "-"
198.20.69.74 - - [02/Feb/2015:06:18:12 +0000] "" 400 0 "-" "-"
198.20.69.74 - - [02/Feb/2015:06:18:13 +0000] "" 400 0 "-" "-"
198.20.69.74 - - [02/Feb/2015:06:18:13 +0000] "" 400 0 "-" "-"
198.20.69.74 - - [02/Feb/2015:06:18:17 +0000] "quit" 400 166 "-" "-"
54.89.61.205 - - [02/Feb/2015:08:25:02 +0000] "GET /robots.txt HTTP/1.1" 200 48 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.197.168.201 - - [02/Feb/2015:08:25:02 +0000] "GET /robots.txt HTTP/1.1" 200 48 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.89.61.205 - - [02/Feb/2015:08:25:02 +0000] "GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1" 200 6638 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.197.168.201 - - [02/Feb/2015:08:25:02 +0000] "GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1" 200 5682 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.160.249.160 - - [02/Feb/2015:08:25:03 +0000] "GET /robots.txt HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.160.249.160 - - [02/Feb/2015:08:25:03 +0000] "GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1" 200 5682 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.82.74.230 - - [02/Feb/2015:08:25:03 +0000] "GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)"
54.82.74.230 - - [02/Feb/2015:08:25:04 +0000] "GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1" 200 57525 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.6; +http://flipboard.com/browserproxy)"
54.160.249.160 - - [02/Feb/2015:08:25:08 +0000] "GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1" 200 5682 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
186.15.3.50 - - [02/Feb/2015:11:13:54 +0000] "GET /tmUnblock.cgi HTTP/1.1" 400 166 "-" "-"
176.58.116.39 - - [02/Feb/2015:12:01:46 +0000] "GET / HTTP/1.1" 200 7107 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:46 +0000] "GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:47 +0000] "GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:47 +0000] "GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:47 +0000] "GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:48 +0000] "GET /content/images/2015/01/lars.jpg HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:54 +0000] "GET / HTTP/1.1" 200 7107 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:54 +0000] "GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:55 +0000] "GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:55 +0000] "GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:55 +0000] "GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
176.58.116.39 - - [02/Feb/2015:12:01:55 +0000] "GET /content/images/2015/01/lars.jpg HTTP/1.1" 304 0 "https://bjaerris.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"

Let's say we want to:

  1. Extract the IP address, date, request, URN and HTTP Status Code.
  2. Ignore duplicates from the same day.

Here one might think that "awk" & "sort" would do, but because of the date format with "hour:minute:seconds", ignoring duplicates with these, eventually ends up convoluted:

$ awk '{print $1, $4, $6, $7}' access.log |sort -u

108.166.85.126 [01/Feb/2015:16:05:43 "GET /admin/config.php
111.251.50.236 [01/Feb/2015:07:17:47 "CONNECT mx0.mail2000.com.tw:25
173.252.110.115 [01/Feb/2015:11:31:20 "GET /content/images/2015/01/ghost_login_owner.png
173.252.110.119 [01/Feb/2015:11:31:21 "GET /content/images/2015/01/ghost_login_owner.png
175.44.8.98 [01/Feb/2015:22:12:06 "POST /ghost_linux_init_script/
176.58.116.39 [02/Feb/2015:12:01:46 "GET /
176.58.116.39 [02/Feb/2015:12:01:46 "GET /assets/css/screen.css?v=cfc2d462d6
176.58.116.39 [02/Feb/2015:12:01:47 "GET /assets/js/index.js?v=cfc2d462d6
176.58.116.39 [02/Feb/2015:12:01:47 "GET /assets/js/jquery.fitvids.js?v=cfc2d462d6
176.58.116.39 [02/Feb/2015:12:01:47 "GET /public/jquery.min.js?v=cfc2d462d6
176.58.116.39 [02/Feb/2015:12:01:48 "GET /content/images/2015/01/lars.jpg
176.58.116.39 [02/Feb/2015:12:01:54 "GET /                                              <---Duplicate
176.58.116.39 [02/Feb/2015:12:01:54 "GET /assets/css/screen.css?v=cfc2d462d6            <---Duplicate
176.58.116.39 [02/Feb/2015:12:01:55 "GET /assets/js/index.js?v=cfc2d462d6               <---Duplicate
176.58.116.39 [02/Feb/2015:12:01:55 "GET /assets/js/jquery.fitvids.js?v=cfc2d462d6      <---Duplicate
176.58.116.39 [02/Feb/2015:12:01:55 "GET /content/images/2015/01/lars.jpg               <---Duplicate
176.58.116.39 [02/Feb/2015:12:01:55 "GET /public/jquery.min.js?v=cfc2d462d6             <---Duplicate
185.5.51.50 [01/Feb/2015:16:56:48 "GET /
185.5.51.50 [01/Feb/2015:16:56:50 "GET /                                                <---Duplicate
185.5.51.50 [01/Feb/2015:16:56:50 "GET /assets/css/screen.css?v=cfc2d462d6
185.5.51.50 [01/Feb/2015:16:56:51 "GET /assets/fonts/casper-icons.woff
185.5.51.50 [01/Feb/2015:16:56:51 "GET /assets/js/index.js?v=cfc2d462d6
185.5.51.50 [01/Feb/2015:16:56:51 "GET /assets/js/jquery.fitvids.js?v=cfc2d462d6
185.5.51.50 [01/Feb/2015:16:56:51 "GET /content/images/2015/01/lars.jpg
185.5.51.50 [01/Feb/2015:16:56:51 "GET /public/jquery.min.js?v=cfc2d462d6
185.5.51.50 [01/Feb/2015:16:57:20 "GET /identifying-services-needing-restart-after-updating-linux-packages/
186.15.3.50 [02/Feb/2015:11:13:54 "GET /tmUnblock.cgi
198.20.69.74 [02/Feb/2015:06:18:10 "GET /
198.20.69.74 [02/Feb/2015:06:18:10 "GET /robots.txt
198.20.69.74 [02/Feb/2015:06:18:12 "" 400
198.20.69.74 [02/Feb/2015:06:18:13 "" 400                                               <---Duplicate
198.20.69.74 [02/Feb/2015:06:18:17 "quit" 400
199.180.112.34 [01/Feb/2015:05:59:48 "GET /muieblackcat
199.180.112.34 [01/Feb/2015:05:59:48 "GET //myadmin/scripts/setup.php
199.180.112.34 [01/Feb/2015:05:59:48 "GET //MyAdmin/scripts/setup.php
199.180.112.34 [01/Feb/2015:05:59:48 "GET //phpmyadmin/scripts/setup.php
199.180.112.34 [01/Feb/2015:05:59:48 "GET //phpMyAdmin/scripts/setup.php
199.180.112.34 [01/Feb/2015:05:59:48 "GET //pma/scripts/setup.php
207.145.97.131 [02/Feb/2015:04:00:25 "GET /
207.145.97.131 [02/Feb/2015:04:00:26 "GET //Net_work.xml
207.34.25.76 [01/Feb/2015:18:01:13 "GET /robots.txt
216.218.206.66 [01/Feb/2015:05:35:11 "GET /
23.23.38.251 [01/Feb/2015:16:24:00 "GET /
23.23.38.251 [01/Feb/2015:16:24:01 "GET /                                               <---Duplicate
23.239.196.71 [01/Feb/2015:07:45:48 "GET / 
31.202.241.242 [02/Feb/2015:01:15:13 "GET /
54.160.249.160 [02/Feb/2015:08:25:03 "GET /host-your-own-blog-with-ghost-nginx-on-linux/
54.160.249.160 [02/Feb/2015:08:25:03 "GET /robots.txt
54.160.249.160 [02/Feb/2015:08:25:08 "GET /host-your-own-blog-with-ghost-nginx-on-linux/
54.197.168.201 [02/Feb/2015:08:25:02 "GET /host-your-own-blog-with-ghost-nginx-on-linux/
54.197.168.201 [02/Feb/2015:08:25:02 "GET /robots.txt
54.82.74.230 [02/Feb/2015:08:25:03 "GET /content/images/2015/01/ghost_login_owner.png
54.82.74.230 [02/Feb/2015:08:25:04 "GET /content/images/2015/01/ghost_login_owner.png   <---Duplicate
54.89.61.205 [02/Feb/2015:08:25:02 "GET /identifying-services-needing-restart-after-updating-linux-packages/
54.89.61.205 [02/Feb/2015:08:25:02 "GET /robots.txt
61.240.144.66 [01/Feb/2015:12:42:47 "GET /
69.171.237.116 [01/Feb/2015:11:06:01 "GET /content/images/2015/01/ghost_login_owner.png
69.171.237.116 [01/Feb/2015:11:06:02 "GET /content/images/2015/01/ghost_login_owner.png <---Duplicate
71.11.195.254 [01/Feb/2015:21:37:39 "GET /tmUnblock.cgi
77.37.231.208 [01/Feb/2015:11:26:03 "GET /
78.133.20.10 [01/Feb/2015:22:42:33 "GET /
78.133.20.10 [01/Feb/2015:22:42:33 "GET /favicon.ico

Clearly another approach is needed if we're to easily ignore duplicates from the same day.
Removing the "hour:minute:seconds" part of date stamp from the output to "sort -u" will fix that.
So let's get rid of the duplicate lines, this time using Perl and regular expressions to parse the log lines.

Perl, the right tool for the job.

$ perl -lane 'print "$1 $2 $3 $4 " if /^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\-\s\-\s\[([^:]+)[^"]+"([^"]+)"\s(\d{3})/' ./access.log

216.218.206.66 01/Feb/2015 GET / HTTP/1.1 403 
199.180.112.34 01/Feb/2015 GET //MyAdmin/scripts/setup.php HTTP/1.1 404 
199.180.112.34 01/Feb/2015 GET //phpMyAdmin/scripts/setup.php HTTP/1.1 404 
199.180.112.34 01/Feb/2015 GET //pma/scripts/setup.php HTTP/1.1 404 
199.180.112.34 01/Feb/2015 GET //phpmyadmin/scripts/setup.php HTTP/1.1 404 
199.180.112.34 01/Feb/2015 GET //myadmin/scripts/setup.php HTTP/1.1 404 
199.180.112.34 01/Feb/2015 GET /muieblackcat HTTP/1.1 404 
111.251.50.236 01/Feb/2015 CONNECT mx0.mail2000.com.tw:25 HTTP/1.0 400 
23.239.196.71 01/Feb/2015 GET / HTTP/1.1 301 
69.171.237.116 01/Feb/2015 GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1 301 
69.171.237.116 01/Feb/2015 GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1 200 
77.37.231.208 01/Feb/2015 GET / HTTP/1.1 301 
173.252.110.115 01/Feb/2015 GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1 301 
173.252.110.119 01/Feb/2015 GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1 200 
61.240.144.66 01/Feb/2015 GET / HTTP/1.0 200 
108.166.85.126 01/Feb/2015 GET /admin/config.php HTTP/1.0 499 
23.23.38.251 01/Feb/2015 GET / HTTP/1.1 301 
23.23.38.251 01/Feb/2015 GET / HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET / HTTP/1.1 301 
185.5.51.50 01/Feb/2015 GET / HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /content/images/2015/01/lars.jpg HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /assets/fonts/casper-icons.woff HTTP/1.1 200 
185.5.51.50 01/Feb/2015 GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1 200 
207.34.25.76 01/Feb/2015 GET /robots.txt HTTP/1.1 301 
71.11.195.254 01/Feb/2015 GET /tmUnblock.cgi HTTP/1.1 400 
175.44.8.98 01/Feb/2015 POST /ghost_linux_init_script/ HTTP/1.1 301 
78.133.20.10 01/Feb/2015 GET / HTTP/1.1 200 
78.133.20.10 01/Feb/2015 GET /favicon.ico HTTP/1.1 404 
31.202.241.242 02/Feb/2015 GET / HTTP/1.0 301 
207.145.97.131 02/Feb/2015 GET / HTTP/1.1 400 
207.145.97.131 02/Feb/2015 GET //Net_work.xml HTTP/1.1 400 
198.20.69.74 02/Feb/2015 GET / HTTP/1.1 403 
198.20.69.74 02/Feb/2015 GET /robots.txt HTTP/1.1 403 
198.20.69.74 02/Feb/2015 quit 400 
54.89.61.205 02/Feb/2015 GET /robots.txt HTTP/1.1 200 
54.197.168.201 02/Feb/2015 GET /robots.txt HTTP/1.1 200 
54.89.61.205 02/Feb/2015 GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1 200 
54.197.168.201 02/Feb/2015 GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1 200 
54.160.249.160 02/Feb/2015 GET /robots.txt HTTP/1.1 301 
54.160.249.160 02/Feb/2015 GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1 200 
54.82.74.230 02/Feb/2015 GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1 301 
54.82.74.230 02/Feb/2015 GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1 200 
54.160.249.160 02/Feb/2015 GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1 200 
186.15.3.50 02/Feb/2015 GET /tmUnblock.cgi HTTP/1.1 400 
176.58.116.39 02/Feb/2015 GET / HTTP/1.1 200 
176.58.116.39 02/Feb/2015 GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /content/images/2015/01/lars.jpg HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET / HTTP/1.1 200 
176.58.116.39 02/Feb/2015 GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1 304 
176.58.116.39 02/Feb/2015 GET /content/images/2015/01/lars.jpg HTTP/1.1 304

Before we go further, let me explain the regular expression used in the previous command.

^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\-\s\-\s\[([^:]+)[^"]+("[^"]*")\s(\d{3})

^            # Match start of string.
###
# 1st Capturing group.
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) 
\d{1,3})  # Match a digit [0-9] Between 1 and 3 times (greedy).
\.        # Match the character "." literally.
###
\s           # Match any white space character [\r\n\t\f].
\-           # Match the character "-" literally.
\s           # Match any white space character [\r\n\t\f].
\-           # Match the character "-" literally.
\s           # Match any white space character [\r\n\t\f]
\[           # Match the character "[" literally.
###
# 2nd Capturing group.
([^:]+)
[^:]+        # Match anything not a ":", one or more times (greedy).
###
[^"]+        # Match anything not a """, one or more times (greedy).
###
# 3rd Capturing group.
("[^"]*")
"            # Match a single character """ literally.
[^"]*        # Match anything not a """, zero or more times (greedy).
"            # Match a single character """ literally.
###
\s           # Match any white space character [\r\n\t\f].
###
# 4th Capturing group.
(\d{3})
\d{3}        # match a digit [0-9] Exactly 3 times.

For more on regular expressions: http://www.regular-expressions.info/tutorial.html

Let's make it "pretty".

Let's move the HTTP status code to the 3rd column and use the printf function to line up the columns to make the output more legible.

$ perl -lane 'printf ("%-16s %-12s %-4s %s\n", $1, $2, $4, $3) if /^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\-\s\-\s\[([^:]+)[^"]+"([^"]+)"\s(\d{3})/' ./access.log

216.218.206.66   01/Feb/2015  403  GET / HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //MyAdmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //phpMyAdmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //pma/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //phpmyadmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //myadmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET /muieblackcat HTTP/1.1
111.251.50.236   01/Feb/2015  400  CONNECT mx0.mail2000.com.tw:25 HTTP/1.0
23.239.196.71    01/Feb/2015  301  GET / HTTP/1.1
69.171.237.116   01/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
69.171.237.116   01/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
77.37.231.208    01/Feb/2015  301  GET / HTTP/1.1
173.252.110.115  01/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
173.252.110.119  01/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
61.240.144.66    01/Feb/2015  200  GET / HTTP/1.0
108.166.85.126   01/Feb/2015  499  GET /admin/config.php HTTP/1.0
23.23.38.251     01/Feb/2015  301  GET / HTTP/1.1
23.23.38.251     01/Feb/2015  200  GET / HTTP/1.1
185.5.51.50      01/Feb/2015  301  GET / HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET / HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /content/images/2015/01/lars.jpg HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/fonts/casper-icons.woff HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1
207.34.25.76     01/Feb/2015  301  GET /robots.txt HTTP/1.1
71.11.195.254    01/Feb/2015  400  GET /tmUnblock.cgi HTTP/1.1
175.44.8.98      01/Feb/2015  301  POST /ghost_linux_init_script/ HTTP/1.1
78.133.20.10     01/Feb/2015  200  GET / HTTP/1.1
78.133.20.10     01/Feb/2015  404  GET /favicon.ico HTTP/1.1
31.202.241.242   02/Feb/2015  301  GET / HTTP/1.0
207.145.97.131   02/Feb/2015  400  GET / HTTP/1.1
207.145.97.131   02/Feb/2015  400  GET //Net_work.xml HTTP/1.1
198.20.69.74     02/Feb/2015  403  GET / HTTP/1.1
198.20.69.74     02/Feb/2015  403  GET /robots.txt HTTP/1.1
198.20.69.74     02/Feb/2015  400  quit
54.89.61.205     02/Feb/2015  200  GET /robots.txt HTTP/1.1
54.197.168.201   02/Feb/2015  200  GET /robots.txt HTTP/1.1
54.89.61.205     02/Feb/2015  200  GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1
54.197.168.201   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
54.160.249.160   02/Feb/2015  301  GET /robots.txt HTTP/1.1
54.160.249.160   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
54.82.74.230     02/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
54.82.74.230     02/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
54.160.249.160   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
186.15.3.50      02/Feb/2015  400  GET /tmUnblock.cgi HTTP/1.1
176.58.116.39    02/Feb/2015  200  GET / HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /content/images/2015/01/lars.jpg HTTP/1.1
176.58.116.39    02/Feb/2015  200  GET / HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /content/images/2015/01/lars.jpg HTTP/1.1

Much better!

Removing duplicate lines.

$ perl -lane 'printf ("%-16s %-12s %-4s %s\n", $1, $2, $4, $3) if /^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\-\s\-\s\[([^:]+)[^"]+"([^"]+)"\s(\d{3})/' ./access.log |sort -u

108.166.85.126   01/Feb/2015  499  GET /admin/config.php HTTP/1.0
111.251.50.236   01/Feb/2015  400  CONNECT mx0.mail2000.com.tw:25 HTTP/1.0
173.252.110.115  01/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
173.252.110.119  01/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
175.44.8.98      01/Feb/2015  301  POST /ghost_linux_init_script/ HTTP/1.1
176.58.116.39    02/Feb/2015  200  GET / HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /content/images/2015/01/lars.jpg HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/fonts/casper-icons.woff HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /content/images/2015/01/lars.jpg HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET / HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  301  GET / HTTP/1.1
186.15.3.50      02/Feb/2015  400  GET /tmUnblock.cgi HTTP/1.1
198.20.69.74     02/Feb/2015  400  quit
198.20.69.74     02/Feb/2015  403  GET / HTTP/1.1
198.20.69.74     02/Feb/2015  403  GET /robots.txt HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET /muieblackcat HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //myadmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //MyAdmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //phpmyadmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //phpMyAdmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //pma/scripts/setup.php HTTP/1.1
207.145.97.131   02/Feb/2015  400  GET / HTTP/1.1
207.145.97.131   02/Feb/2015  400  GET //Net_work.xml HTTP/1.1
207.34.25.76     01/Feb/2015  301  GET /robots.txt HTTP/1.1
216.218.206.66   01/Feb/2015  403  GET / HTTP/1.1
23.23.38.251     01/Feb/2015  200  GET / HTTP/1.1
23.23.38.251     01/Feb/2015  301  GET / HTTP/1.1
23.239.196.71    01/Feb/2015  301  GET / HTTP/1.1
31.202.241.242   02/Feb/2015  301  GET / HTTP/1.0
54.160.249.160   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
54.160.249.160   02/Feb/2015  301  GET /robots.txt HTTP/1.1
54.197.168.201   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
54.197.168.201   02/Feb/2015  200  GET /robots.txt HTTP/1.1
54.82.74.230     02/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
54.82.74.230     02/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
54.89.61.205     02/Feb/2015  200  GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1
54.89.61.205     02/Feb/2015  200  GET /robots.txt HTTP/1.1
61.240.144.66    01/Feb/2015  200  GET / HTTP/1.0
69.171.237.116   01/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
69.171.237.116   01/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
71.11.195.254    01/Feb/2015  400  GET /tmUnblock.cgi HTTP/1.1
77.37.231.208    01/Feb/2015  301  GET / HTTP/1.1
78.133.20.10     01/Feb/2015  200  GET / HTTP/1.1
78.133.20.10     01/Feb/2015  404  GET /favicon.ico HTTP/1.1

Want to sort it based on URN/file-path?

$ perl -lane 'printf ("%-16s %-12s %-4s %s\n", $1, $2, $4, $3) if /^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\-\s\-\s\[([^:]+)[^"]+"([^"]+)"\s(\d{3})/' ./access.log |sort -u|sort -k5,5

198.20.69.74     02/Feb/2015  400  quit
176.58.116.39    02/Feb/2015  200  GET / HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET / HTTP/1.1
185.5.51.50      01/Feb/2015  301  GET / HTTP/1.1
198.20.69.74     02/Feb/2015  403  GET / HTTP/1.1
207.145.97.131   02/Feb/2015  400  GET / HTTP/1.1
216.218.206.66   01/Feb/2015  403  GET / HTTP/1.1
23.23.38.251     01/Feb/2015  200  GET / HTTP/1.1
23.23.38.251     01/Feb/2015  301  GET / HTTP/1.1
23.239.196.71    01/Feb/2015  301  GET / HTTP/1.1
31.202.241.242   02/Feb/2015  301  GET / HTTP/1.0
61.240.144.66    01/Feb/2015  200  GET / HTTP/1.0
77.37.231.208    01/Feb/2015  301  GET / HTTP/1.1
78.133.20.10     01/Feb/2015  200  GET / HTTP/1.1
108.166.85.126   01/Feb/2015  499  GET /admin/config.php HTTP/1.0
176.58.116.39    02/Feb/2015  304  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/css/screen.css?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/fonts/casper-icons.woff HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/js/index.js?v=cfc2d462d6 HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /assets/js/jquery.fitvids.js?v=cfc2d462d6 HTTP/1.1
173.252.110.115  01/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
173.252.110.119  01/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
54.82.74.230     02/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
54.82.74.230     02/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
69.171.237.116   01/Feb/2015  200  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
69.171.237.116   01/Feb/2015  301  GET /content/images/2015/01/ghost_login_owner.png HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /content/images/2015/01/lars.jpg HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /content/images/2015/01/lars.jpg HTTP/1.1
78.133.20.10     01/Feb/2015  404  GET /favicon.ico HTTP/1.1
175.44.8.98      01/Feb/2015  301  POST /ghost_linux_init_script/ HTTP/1.1
54.160.249.160   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
54.197.168.201   02/Feb/2015  200  GET /host-your-own-blog-with-ghost-nginx-on-linux/ HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1
54.89.61.205     02/Feb/2015  200  GET /identifying-services-needing-restart-after-updating-linux-packages/ HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET /muieblackcat HTTP/1.1
111.251.50.236   01/Feb/2015  400  CONNECT mx0.mail2000.com.tw:25 HTTP/1.0
199.180.112.34   01/Feb/2015  404  GET //myadmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //MyAdmin/scripts/setup.php HTTP/1.1
207.145.97.131   02/Feb/2015  400  GET //Net_work.xml HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //phpmyadmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //phpMyAdmin/scripts/setup.php HTTP/1.1
199.180.112.34   01/Feb/2015  404  GET //pma/scripts/setup.php HTTP/1.1
176.58.116.39    02/Feb/2015  304  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
185.5.51.50      01/Feb/2015  200  GET /public/jquery.min.js?v=cfc2d462d6 HTTP/1.1
198.20.69.74     02/Feb/2015  403  GET /robots.txt HTTP/1.1
207.34.25.76     01/Feb/2015  301  GET /robots.txt HTTP/1.1
54.160.249.160   02/Feb/2015  301  GET /robots.txt HTTP/1.1
54.197.168.201   02/Feb/2015  200  GET /robots.txt HTTP/1.1
54.89.61.205     02/Feb/2015  200  GET /robots.txt HTTP/1.1
186.15.3.50      02/Feb/2015  400  GET /tmUnblock.cgi HTTP/1.1
71.11.195.254    01/Feb/2015  400  GET /tmUnblock.cgi HTTP/1.1

Output in CSV format, for import into a spreadsheet application like Excel.

$ perl -lane 'printf "$1,$2,$4,$3\n" if /^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s\-\s\-\s\[([^:]+)[^"]+"([^"]+)"\s(\d{3})/' ./access.log |sort -u > access.log.csv

That's it!

Hope you enjoyed it!

Lars Bjaerris

]]><![CDATA[Needy-Restart; Service Restart After Linux Package Update]]>When to restart services after package updating.

When updating Linux packages with a package manager it is occasionally necessary to identify services running, having file(s) open that have been unlinked from the directory tree, i.e. deleted.
This is most commonly caused by a process having a shared library(

]]>https://bjaerris.com/identifying-services-needing-restart-after-updating-linux-packages/848c2b56-5ca3-454c-9a74-84cad15a20d7Wed, 28 Jan 2015 11:47:12 GMTWhen to restart services after package updating.

When updating Linux packages with a package manager it is occasionally necessary to identify services running, having file(s) open that have been unlinked from the directory tree, i.e. deleted.
This is most commonly caused by a process having a shared library(.so) file open that has been updated to a newer version.

Consider the case of the recent Openssl Heartbleed vulnerability. http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-0160
Updating Openssl via your package manager without restarting running services on your system relying on “libssl.so.*”, will most likely result in the system still being vulnerable to attacks.

In this post I’ll demonstrate a couple of ways to identify running services with open and unlinked files and automatically restarting them.

Finally I will share my application/script "needy-restart", for automatically identifying and restarting these services.

The following should work as is, on
RHEL, Oracle Linux, SL, CentOS Linux distributions.

For other Linux distributions, minor adjustments should be expected.

Here’s from a recent system update

Part of the “yum-utils” package has a python script to output processes that needs restarting, called “needs-restarting”
It is very useful for getting info helping determining what processes will need to be restarted.

This is what the script outputs on this system:

# needs-restarting 
1869 : /sbin/dhclient-1-q-cf/etc/dhcp/dhclient-eth0.conf-lf/var/lib/dhclient/dhclient-eth0.leases-pf/var/run/dhclient-eth0.pideth0
2125 : /usr/bin/mimedefang-multiplexor-p/var/spool/MIMEDefang/mimedefang-multiplexor.pid-m2-x10-y0-Udefang-b600-l-s/var/spool/MIMEDefang/mimedefang-multiplexor.sock
2214 : crond

Unfortunately the script will not restart the services for us, so if we want that functionality we have to roll our own script.

In the following we are only interested in matching and restarting these services:

2125 : /usr/bin/mimedefang-multiplexor-p/var/spool/MIMEDefang/mimedefang-multiplexor.pid-m2-x10-y0-Udefang-b600-l-s/var/spool/MIMEDefang/mimedefang-multiplexor.sock 
2214 : crond

The following should be restarted and verified by doing a system reboot.

1869 : /sbin/dhclient-1-q-cf/etc/dhcp/dhclient-eth0.conf-lf/var/lib/dhclient/dhclient-eth0.leases-pf/var/run/dhclient-eth0.pideth0

# If you’re feeling brave, issuing “service network restart” will do, but is *not* recommended on a production system without scheduled downtime.

Let’s make our own script.

Check for processes with open files that has been unlinked/deleted using “lsof”

We want to identify “DEL” or “deleted” in the 5th column of the output from lsof and filter out “/dev/zero” & “async I/O (AIO)”

# lsof |grep "DEL\|deleted" | grep -v "/dev/zero\|\[aio\]"

dhclient   1869       root  txt       REG              202,0   572256      18270 /sbin/dhclient (deleted)
dovecot    2006       root  122u      REG              202,0        0      41056 /var/run/dovecot/login-master-notifye2e4a04bb9b2d719 (deleted)
dovecot    2006       root  125u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
mimedefan  2125     defang  DEL       REG              202,0               24600 /lib64/libfreebl3.so
crond      2214       root  DEL       REG              202,0               24600 /lib64/libfreebl3.so
imap-logi 17666   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 17671   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 17708   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18376   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18383   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18385   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)
imap-logi 18387   dovenull    4u      REG              202,0        0      42553 /var/run/dovecot/login-master-notify4804d9d068f2d611 (deleted)

We are interested in the two services having an unlinked shared library file open. In this case:

mimedefan  2125     defang  DEL       REG              202,0               24600 /lib64/libfreebl3.so
crond      2214       root  DEL       REG              202,0               24600 /lib64/libfreebl3.so

Let’s check what package was updated.

# yum whatprovides /lib64/libfreebl3.so |grep -B3 -A3 "installed"

nss-softokn-freebl-3.14.3-19.el6_6.x86_64 : Freebl library for the Network
                                          : Security Services
Repo        : installed
Matched from:
Other       : Provides-match: /lib64/libfreebl3.so

So the package “nss-softokn-freebl-3.14.3-19.el66.x8664” was updated and two services need to be restarted to use the new shared library provided by this package.

But in the interest of coding a script to identifying and restarting these services, we’ll wait.

Let’s tighten the output so we can use it in a script.

# lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && !$seen{$F[0]}++)'
mimedefan        <--- OUTPUT
crond            <--- OUTPUT

Here we still filter out “/dev/zero” & “async I/O (AIO)” with the grep command, pipe the output to perl and print the first column if column “4” matches “DEL” or “deleted”. The last part is a perl emulation of “sort -uk1,1” showing only unique first column matches.

Much cleaner output and more suitable for use in a script.
Unfortunately the output can not be used directly to wrap in a for loop to restart, as “mimedefan” service restart is done like this: “service mimedefang restart”

Let’s grab the running service names from /var/lock/subsys/ and compare them to our previous output.

# proc=$(lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && ! $seen{$F[0]}++)') ; \
if ! [ -z "$proc" ]; then ls /var/lock/subsys/ |grep "$proc"; else echo "No service needs restart."; fi

crond             <--- OUTPUT
mimedefang        <--- OUTPUT

Now we have process names we can use to restart with the service command.
Let’s test it by issuing “service <servicename> status” on these processes first.

# proc=$(lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && ! $seen{$F[0]}++)') ; \
 if ! [ -z "$proc" ]; then ls /var/lock/subsys/ |grep "$proc"| xargs -I{} service {} status; else echo "No service needs restart."; fi

crond (pid  2214) is running...                     <--- OUTPUT
mimedefang (pid  2142) is running...                <--- OUTPUT
mimedefang-multiplexor (pid  2125) is running...    <--- OUTPUT
  0/10 .......... 0                                 <--- OUTPUT

Now we have a working one-liner that clearly is begging to be put in a script.
Let’s construct a bash script to identify and restart service(s) if needed after updating of packages.

needy-restart script

Below is the finished script.

#!/bin/sh
# Version 0.6
# needy-restart: Identifying and restarting running services
# with open files unlinked/deleted from the directory tree.
#
# Copyright (C) 2015 Lars Bjaerris <lars@bjaerris.com>
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
# -------------------------------------------------------------------

proc=$(lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && ! $seen{$F[0]}++)')
fini=0
while [ "$#" -gt 0 ]
do
    case "$1" in
        -r)
           if ! [ -z "$proc" ] # Check that we're not grep'ing with an empty string
           then
               echo "Restarting needy restart(s):"; ls /var/lock/subsys/ |grep "$proc" |xargs -I{} service {} restart
           else    
               echo "No needy restart(s)"
               fini=1 #
           fi
           ;;&    # Continue with the last check of "-v" unless $fini -eq 1.
     -v|-r)
           if [ "$fini" -eq 1 ]
             then
             exit
           elif ! [ -z "$proc" ] 
           then
               echo "Checking for needy restart(s):"; ls /var/lock/subsys/ |grep "$proc"
               echo "If processes are stil listed below after running "needy-restart -r","
               echo "you should evaluate them and consider if a reboot is necessary."
               echo "-------------------------------------------------------------------------------"
               lsof |grep -v "/dev/zero\|\[aio\]" | perl -lane 'print $F[0] if ($F[3] =~ /DEL|deleted/ && !$seen{$F[0]}++)'
           else 
               echo "No needy restart(s)"
           fi
           ;;
    -*)
            echo >&2 "usage: $0 [-v] [-r] -v: Check for needy restart(s) or -r: Restart needy restart(s)"
        exit 1;;
     *)  
            echo >&2 "usage: $0 [-v] [-r] -v: Check for needy restart(s) or -r: Restart needy restart(s)"
            break;;    # terminate while loop
    esac
    shift
done

Let's test it

# vi needy-restart

Paste!

Set the execute bit.

#chmod +x needy-restart

Execute.

# ./needy-restart -v
Checking for needy-restart(s):    <--- OUTPUT
crond                             <--- OUTPUT
mimedefang                        <--- OUTPUT
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------

Give the restart service(s) option.

# ./needy-restart -r
Restarting needy-restart(s):
Stopping crond: [  OK  ]
Starting crond: [  OK  ]
Shutting down mimedefang: [  OK  ]
Shutting down mimedefang-multiplexor: [  OK  ]
Waiting for daemons to exit
Checking filter syntax: OK
Starting mimedefang-multiplexor: [  OK  ]
Starting mimedefang: [  OK  ]
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------

Final check.

# ./needy-restart -v
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
EDIT:

Here is an example using "needy-restart after updating: glibc-* & kernel-headers.

# ./needy-restart -r
Restarting needy-restart(s):
Stopping auditd: [  OK  ]
Starting auditd: [  OK  ]
Shutting down system logger: [  OK  ]
Starting system logger: [  OK  ]
Stopping Dovecot Imap: [  OK  ]
Starting Dovecot Imap: [  OK  ]
Stopping crond: [  OK  ]
Starting crond: [  OK  ]
Stopping sshd: [  OK  ]
Starting sshd: [  OK  ]
Stopping nginx: [  OK  ]
Starting nginx: [  OK  ]
Stopping mysqld:  [  OK  ]
Starting mysqld:  [  OK  ]
Stopping mysqld:  [  OK  ]
Starting mysqld:  [  OK  ]
Stopping php-fpm: [  OK  ]
Starting php-fpm: [  OK  ]
Shutting down ntpd: [  OK  ]
Starting ntpd: [  OK  ]
Stopping fail2ban: [  OK  ]
Starting fail2ban: [  OK  ]
Checking for needy-restart(s):
auditd
sshd
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------
init
udevd
dhclient
auditd
portreser
mimedefan
agetty
mingetty
npm
node
sshd
bash
pickup
master
qmgr
tlsmgr

In this case we have to reboot the server.

After Reboot testing

# ./needy-restart -v
Checking for needy-restart(s):
If processes are stil listed below after running needy-restart -r,
you should evaluate them and consider if a reboot is necessary.
-------------------------------------------------------------------------------

Hope you enjoyed it!
Lars Bjaerris

]]><![CDATA[Ghost Blog Sys-V init Linux Startup Script]]>https://bjaerris.com/ghost_linux_init_script/63ad3734-f110-4490-8d40-3b19b23fca97Sun, 25 Jan 2015 11:39:22 GMTInstalling nodejs init script for Ghost Blog.

If you followed my guide on: Hosting your own blog with Ghost & Nginx on Linux

The following Linux init script should work as is, on
RHEL, Oracle Linux, SL, CentOS Linux distributions.

For other Linux distributions, minor adjustments should be expected.

Installing Ghost Blog init script

$ sudo vi /etc/init.d/ghostblog

^Add the below script to the file and save.

Ghost Blog Linux init script:

#!/bin/sh
#
# ghostblog   Startup script for nodejs Ghost Blog
#
# Copyright (C) 2015 Lars Bjaerris lars<at>bjaerris.com
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
# -------------------------------------------------------------------
# chkconfig: 2345 80 10
#
### BEGIN INIT INFO
# Provides: ghostblog
# Required-Start: $local_fs $remote_fs $network
# Required-Stop: $local_fs $remote_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: start and stop ghostblog
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

user="ghostblog"
prog="ghostblog"
exec="/usr/bin/npm"
run_env="NODE_ENV=production"
option="start"
daemon_dir="/home/ghostblog/ghost/"
piddir="/home/$user/ghost/run/"
pidfile="$piddir$prog.pid"
prockill='node index'

[[ -d $piddir ]] || su $user -c " mkdir -p $piddir"

[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$prog

lockfile=/var/lock/subsys/$prog

start() {
    [ -x $exec ] || exit 5
    echo -n $"Starting $prog: "
    su $user -c "cd $daemon_dir; export $run_env; nohup $exec $option > /dev/null 2>&1 &"  
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile && sleep 2;echo $(/usr/bin/pgrep -u ghostblog -f 'node index') > $pidfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    pkill -u $user -f "$prockill"
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile && rm -f $pidfile 
    return $retval
}

restart() {
    stop
    start
}

reload() {
    restart
}

force_reload() {
    restart
}

rh_status() {
status $prockill
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?

-

 $ sudo chmod 0750 /etc/init.d/ghostblog

^Make it executable and set permissions.

$ sudo chkconfig --add /etc/init.d/ghostblog

^Add the init scipt to services

$ sudo chkconfig ghostblog on

^Set the startup flag to on.

-

Testing:

$ sudo service ghostblog start
Starting ghostblog: 
$ sudo service ghostblog status
node (pid 24909) is running...
$ sudo service ghostblog stop
Stopping ghostblog: 
$ sudo service ghostblog start
Starting ghostblog: 
$ sudo service ghostblog status
node (pid 24963) is running...
$

ghostblog is configured to start and stop as a service.

Hope you enjoyed it!
Lars Bjaerris

]]><![CDATA[Host Your Own Blog With Ghost & Nginx On Linux]]>https://bjaerris.com/host-your-own-blog-with-ghost-nginx-on-linux/f2b01ba5-444d-4907-b454-73d49e3132f4Fri, 23 Jan 2015 16:54:04 GMTSelf hosting your Ghost blog on Linux.

This Blog runs on the excellent Ghost blog platform.
Official website: http://ghost.org/

This is a Setup Guide/How-to, getting the Ghost Blog platform up and running on a RHEL/Oracle Linux/CentOS/Scientific Linux server.
(For other Linux flavors, minor adjustments to this guide should be expected.)

We will serve it thru a Reverse Proxy using another excellent open source application; the NGINX Web-Server/Reverse-Proxy/Load-Balancer/Http-Cache.

Prerequisites
  1. Linux server. (RHEL, Scientific Linux, Oracle Linux, CentOS)
  2. System user account with sudoers privilege or root access.
  3. Domain name with DNS A record pointing to the public IP of the server.
  4. A console or if remote, terminal and SSH.
  5. If you have "2." I'll presume you're comfortable with the commandline and your editor of choice. :)
Initial Installation

Let's add the blog system user/process owner and install the blog software, along with the required software dependencies.

 $ sudo yum list installed epel-release || yum -y install epel-release
 $ sudo yum -y install nodejs npm
 $ sudo useradd ghostblog
 $ sudo su - ghostblog

Above commands explained:

  1. Check if we have the "Extra Packages for Enterprise Linux repository" (epel-release) and if not, install it.
  2. Install "nodejs", "npm" and dependencies.
  3. Add a system user that will own the core blog files and processes.
  4. Switch to our new system user and environment.
Ghost blog software installation
$ curl -LO https://ghost.org/zip/ghost-latest.zip 
$ unzip -uo ghost-latest.zip -d ghost && rm -f ghost-latest.zip
$ cd /home/ghostblog/ghost/
$ npm install --production
$ nohup npm start > ghost.log 2>&1 &

Above commands explained:

  1. Get the latest Ghost source files.
  2. Unzip the files into the "ghost" directory (Create it if it doesn’t exist) and remove the zip file if the previous command succeeded.
  3. Change into directory "/home/ghostblog/ghost/".
  4. Install the Ghost platform.
  5. Start it and detach it from current terminal. (We will construct a proper SysV-init script later for proper system and application startup/shutdown.)

The last command will generate output similar to this:

[ghostblog@gw4 ghost]$ nohup npm start > ghost.log 2>&1 &

[1] 10430

Issue the "jobs" command to check that we are up and running. 

$ jobs

Above command explained:

  1. List running and suspended jobs in the current bash environment.

Output:

[1]+ Running nohup npm start > ghost.log 2>&1 &

[ghostblog@gw4 ghost]$

Ghost configuration

Configure "/home/ghostblog/ghost/config.js"

$ vi /home/ghostblog/ghost/config.js

Change "url:" to your domain:

// When running Ghost in the wild, use the production environment
// Configure your URL and mail settings here
production: {
    url: 'http://example.com', // #<--- Your domain here

And here:

    // ### Development **(default)**
development: {
    // The url to use when providing links to the site, E.g. in RSS and email.
    // Change this to your Ghost blogs published URL.
    url: 'http://example.com', // #<--- Your domain here

Restart Ghost into production environment:

$ pkill -u ghostblog -f 'node index'
$ NODE_ENV=production nohup npm start > ghost.log 2>&1 &

Above commands explained:

  1. Kill process owned by "ghostblog" matching "node index"
  2. Start Ghost in prodution environment, background and detach from terminal.
Nginx Installation

Installation and Configuration of Nginx as a reverse proxy, to the Ghost Blog locally running instance.

$ exit
$ sudo yum list installed nginx || yum -y install nginx
$ sudo /etc/init.d/nginx start || chkconfig nginx on
$ sudo vi /etc/nginx/nginx.conf

Example nginx.conf file

user  nginx;
worker_processes  2;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log off;
    sendfile        on;
    keepalive_timeout  65;

    include /etc/nginx/conf.d/*.conf;
}

Creating the Ghost Blog Nginx configuration file

$ sudo vi /etc/nginx/conf.d/ghostblog.conf

Add the following and change to your domain and ip address marked with "#<---"

# Reverse proxy server
upstream ghost  {
    server 127.0.0.1:2368;
}

server {

       listen       80;
       server_name  example.com www.example.com; #<--- Your domain goes here.

       ## send all traffic to the back-end
       location / {
            proxy_pass        http://ghost;
            proxy_set_header Host      $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_redirect    off;
            proxy_hide_header X-Powered-By;

            location ~* \.(html?|css|jpg|gif|ico|js|woff)$ {
                    proxy_cache_path /tmp/cache levels=1:2 keys_zone=cache:60m max_size=1G;
                    proxy_cache          cache;
                    proxy_cache_key      $host$uri$is_args$args;
                    proxy_cache_valid    200 301 302 30m;
                    proxy_cache_valid    404 1m;
                    expires              30m;
                    ### proxy-buffers ###
                    proxy_buffering         on; # Has to be on for cache to work
                    proxy_buffer_size       8k;
                    proxy_buffers           256 8k;
                    proxy_busy_buffers_size    64k; proxy_temp_file_write_size 64k;
                    proxy_pass  http://ghost;
            }

            # Temporary access control below until we have set up an owner/admin for the Ghost Blog.
            allow 127.0.0.1; # <--- Your public server ip address goes here.
            deny    all;
       }
}
Test Nginx Configuration and Reload
$ sudo nginx -t && /etc/init.d/nginx reload

Above command explained:

  1. Test nginx configuration and reload if ok.

Output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful

Reloading nginx:

Creating the Blog Owner (You!)

Start your favorite browser and connect to your new blog on:

http://example.com/ghost/ #<-- Substitute for your own domain.

Output: (Hopefully! ;) Ghost Initial Login

Removing the access restriction.

Once you are ready to let the world read your blog, you will have to edit the "/etc/nginx/nginx.conf" once more.

$ sudo vi /etc/nginx/conf.d/ghostblog.conf

Find this part again:

# Get your ip address from Google: Search for "my ip"
 allow 127.0.0.1; #<--- Your ip address from above search goes here.
 deny    all;

Change it to:

# Get your ip address from Google: Search for "my ip"
# allow 127.0.0.1; #<--- Your ip address from above search goes here.
# deny    all;

Test and reload the Nginx configuration:

$ sudo nginx -t && /etc/init.d/nginx reload

Above command explained:

  1. Test nginx configuration and reload if OK.
Your blog is ready and world readable.

Next: Installing a Ghost Blog Sys-V init Linux Startup Script

Hope you enjoyed it!
Lars Bjaerris

]]><![CDATA[What's This?]]>https://bjaerris.com/lars_bjaerris/fc79acc9-c9f6-4839-a669-0ba2cc8b8174Thu, 22 Jan 2015 07:10:00 GMTLinux guides, tips and tricks.

This is a collection of Linux/*nix Engineering how-to's, tips and tricks.

I hope you find them useful!
Lars Bjaerris - System Engineer

]]>