IP Networking: Execute if Process is Missing
Posted on April 21, 2021 — 9 Minutes Read
For any network reliability or performance evaluation it is often necessary to keep certain commands or tools running during the course of the test for a round-the-clock and at times round-the-week measurement. Some network evaluation tools such as iPerf have an option to have part of the instance specifically the server side in the case of iPerf run as a daemon. The problem is though if for reason unknown the process exits or is terminated prematurely, there is no way for the process to reinitiate itself for it is no longer in progress. A time based scheduler, for which nothing compares to the tried and trust cron, may be instructed to initiate the process at some predefined interval during course of the task, that may nonetheless create multiples instances of the process and produce undesired results. A dilemma this may seem, solving this with a Bash script takes merely a few lines.
The rest of the code is shared on Github for reference and further development.
Writing a Bash script
There are many tools or scripting languages that one can use to automate tasks on a server and Bash is one of them. It is one of the most widely-used scripting languages and is the default login shell for Linux. When you connect to your Linux server by Secure Shell (SSH, RFC 4523), you are already using Bash.
Use the nano text editor to open a new file and start writing the script.
$ nano
First line to add the script shall be to inform the executer that this is a Bash script and is to be interpreted with Bash. This line is elegantly called a shebang.
#!/bin/bash
Thereafter goes the script.
#!/bin/bash
if ! ps aux | grep -v grep | grep -v $0 | grep "$*" 2>&1 > /dev/null; then
$*
fi
exit
Control + o to save the script. Enter the a name for the new file e.g. daemonc, with a full path if it is to be saved in a location other than the current working directory e.g. /app/daemonc/daemonc.sh. Press enter to save.
Control + x to exit nano.
Inverse If Conditional
Neatly the script does the check in a single inverse if statement with the if conditional being a series of pipelines that performs the checks one after another. The first of which executes the ps command to print a snapshot of the current processes and passes the print to the next pipe. The following pipes call upon grep to search and filter the ps output for the argument supplied to the script. If grep finds a matching pattern, the inverse if conditional will be false and the script will exit gracefully. If otherwise grep finds no match i.e. the argument supplied is missing in the current processes, the inverse if conditional will be true and the script will proceed to execute the rest of the argument supplied. Any leftover output from the ps command will be written to /dev/null for discard.
if ! ps aux | grep -v grep | grep -v $0 | grep "$*" 2>&1 > /dev/null; then
...
fi
Testing and Troubleshooting
Now the script can be tested.
First thing is to do is to provide the script with execute permission.
chmod +x *full-path-to-script*
$ chmod +x /app/daemonc/daemonc.sh
To execute the script, simply call it in your Bash shell with a command and its corresponding arguments. For example, you can feed the command iperf -s -D to the script to have it execute iPerf in server mode as a daemon if it is not founded in the current processes. Before doing this, check that no iperf is in session with an if conditional that executes the ps command matching for iperf, and kill it with the killall command if it is found.
$ if [[ $(ps aux | grep -v grep | grep iperf) ]]; then killall iperf; fi
Now execute the script with iPerf server as a daemon.
$ /app/daemonc/daemonc.sh iperf -s -D
You should see an output similar to the below if all goes well.
Running Iperf Server as a daemon
This means the script finds that iPerf is missing from the current processes and as such executes it accordingly. You can verify that the iperf server is running as a daemon by executing iperf in client mode to itself at the loopback IP address that is 127.0.0.1.
$ iperf -c 127.0.0.1
You should see an output similar to the below if the iperf server daemon is working as expected.
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ 3] local 127.0.0.1 port 44678 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.6 GBytes 9.09 Gbits/sec
With an iperf server in process, if you call again the script, you will find that nothing is returned for the script now sees an iperf server in session and exits gracefully.
Now if you call the ps command filtering for iperf again.
$ ps aux | grep -v grep | grep iperf
You should find an iperf server in session.
root 3141700 0.0 0.0 153592 208 ? Ssl 03:45 0:00 iperf -s -D
Be sure to clean up by killing the iperf server process with the kill command before moving on.
kill *process-id*
$ kill 3141700
Below are some of the common errors.
Bash: permission denied
-bash: full-path-to-script: Permission denied
Please make sure that your script has execute permission. To add execute permission.
chmod +x *full-path-to-script*
$ chmod +x /app/daemonc/daemonc.sh
Automating a Bash Script
Now that the script works as expected, it can be used to automate tasks such as starting Ping or iPerf and stopping them at some specific times as appropriate together with cron.
Network Latency Evaluation
Recapping what was discussed, without this script, tasks can still be automated in a way with cron, in the sense that cron can be instructed to start and stop a task some specific times. The problem is again if for reason unknown the process exits or is terminated prematurely, there is no way for the process to reinitiate itself for it is no longer in progress. If cron is coded to initiate the process at some predefined interval instead during the course of task, multiples instances of the process will be created which could produce undesired consequences. Solving this problem is precisely where this simple and elegant script stands
For example if you are to ping a certain host e.g. the Cloudflare DNS server at 1.1.1.1 for an estimation of its (RTT) during the course of a 7-day evaluation from April 23rd to 30th, with the probe set to once every 100 millisecond (ms) and its output in Comma Separated Values (CSV), you will need for starter a cron job to start the ping at the begining of the evaluation. Now since the output should be in CSV which is more computable than the default unstructured text, you could take advantage of yet another neat Bash script that was described in details here to start the ping and filter its output for you. Since neither ping nor the output-filtering Bash script has any embedded intelligence to trigger any action such as to reinitiate itself if it exits or is terminated prematurely, to ensure ping stays in process during the course of the task, you could run this Bash script instead to start the out-filtering script for you which in turn runs ping at it core if it is not already running. Since this Bash script will not reinitiate another instance of the output-filtering script and nor ping if it is still in session, this Bash script could be called at some predefined interval during the course of the task e.g. every minute to ensure everything is working as it should without distorting the results without multiple instances of the out-filtering script and of ping.
Scheduling a Bash Script
Two sets of cron jobs will as such be needed. The first will initiate this Bash script every minute run the output-filtering Bash script and in turn, ping. Another will be set to start the first job at the beginning of the evaluation and stop it at the end.
To edit the crontab which is short for cron table.
$ crontab -e
You may be asked which text editor should be used to edit the crontab, select nano or your preferred one.
Add these new schedules at the end of the crontab.
# * * * * * /app/daemonc/daemonc.sh /app/pingc/pingc.sh -i0.1 1.1.1.1 >> /app/ping-mb-1.1.1.1.csv
0 0 23 4 * crontab -l | perl -nle 's/^#\s*([0-9*])/$1/;print' | crontab
# 0 0 30 4 * killall pingc
# 0 0 30 4 * crontab -l | perl -nle 's/^([^#])/# $1/;print' | crontab
The first line is the first set of cron job which in essence initiates this Bash script every minute with the output redirected to a local file. The first part of the line i.e. * * * * * denotes when and at what interval this task should be executed which in this case is once every minute. The remaining part of the line tells cron which script to execute at the defined interval. The hash character (#) at the start of the line informs cron that this line is a comment and do not execute it just yet for the start time has not come, and it falls on the shoulder of the second set of cron jobs to start and subsequently stop the first set of cron when the times come by uncommenting or commenting it.
The second to the forth lines will do exactly that with the second line running a series of pipelines to uncomment any line that starts with a numeric or a wildcard (*) after the hash in the cron table on April 23rd at 00:00:00 as per the system time. This will uncomment the first, third and forth lines of the cron schedule and put the ping in session with a verification every minute until 7 days later at the end of the task on April 30th. At 00:00:00 on which date the third and forth lines are scheduled to kill the output-filtering Bash script in the current processes and use a series of pipelines to comment all the lines that do not start with a hash in the cron table in order to stop the first set of cron.
When done, control + x to exit, and y and enter to save.
Thoughts
This is just one example of using a simple Bash script to execute and automate commands. There are plenty of other fascinating tasks that can be done and with Bash. Explore and be amazed.