INDEX

---

1. Introduction

---

Have you ever needed to reboot a whole server nest after upgrading their OS? Or maybe extract information from all of them for our inventory? Those are typical tasks that are usually done in every company, and we have been forced to deal with them, with all the time we spend on that.

Doing multiple operations can become a tedious task, especially if we need to do it for a lot of servers.

In this web we like to optimize “time consuming” tasks, so we are going to show you how to work with these type of tasks for hundreds of Linux Servers. Let's go!

 

2. Prerrequisites

---

For this lab we will need:

• A Linux host (it can be a virtual machine, or even a Windows Subsystem for Linux(WSL)).
• We will need to install "sshpass"
• Some Bash scripting knowledge

 

2.1. How to install SSHPass?

We will need the default package manager” for the distribution you're running. Some examples are:

Centos: yum install sshpass

Ubuntu: sudo apt-get install sshpass

Fedora: dnf install sshpass

NOTES: In case of Centos OS, it will be needed to previously install "epel-release" repository. You can install it through “yum install epel-release” command.

 

3. What are we going to do?

---

These are the main steps to do to every server:

1. Check that server is "reachable" (if can be pinged)
2. Connect to server
3. Run the tasks on server

 

This is the flow diagram that we're going to do.

 

4. Steps

---

4.1. Create files

First step will be to create the files that we will need for this script:

 

hosts.txt: We will store all the servers address that we want to connect to.

[salvatutiempo@localhost temp]$ nano hosts.txt
1.1.1.1
2.2.2.2
3.3.3.3
4.4.4.4
5.5.5.5

 

pass.txt: We will store the password that will be need to connect to every server, and the one that our script will use to automate this task.

[salvatutiempo@localhost temp]$ nano pass.txt
PASS_SERVIDOR

stt_auto_tasks.sh: This will be the file where all the code will be. We will need to provide execute permissions , using the following comand:

chmod +x stt_autotasks.sh

[salvatutiempo@localhost temp]$ sudo touch stt_auto_tasks.sh
[salvatutiempo@localhost temp]$ sudo chmod +x stt_auto_tasks.sh
[salvatutiempo@localhost temp]$ ls -l stt_auto_tasks.sh
-rwxr-xr-x 1 root root 0 Dec 11 10:44 stt_auto_tasks.sh

 

We will check every step that the script will have.

 

4.2. Variables

We will define the variables that will be needed later

SSH="sshpass -e ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null"

HOSTS="hosts.txt"
USER="username"
SSHPASS=$(cat pass.txt)
export SSHPASS

 

SSH: This variable will have part of the command that we will use to connect to servers, and will be explained in the following steps.

HOSTS: This variable has the text file path with all the destination servers.

USER: This variable will have the username that we will need to connect to servers.

SSHPASS: It will read the password that we stored and it will be used to connect to servers

export SSHPASS: This line of code it will allow SSHPASS variable to be used as global variable

 

4.3. Functions

For the repetitive tasks, we will create some functions that will be run in our Main code. The functions needed will be:

• pingable function: It will allow us to check if a host is reachable (if it's pingable).
• actions_remote function: It will have all the tasks that we need to run on destination hosts, once we are connected to them.

 

“pingable" function

This function will led us know if we can reach to a host. We will use "ping " command for that, with some extra options.

pingable() {
     echo $(ping -c1 $1 | grep -c error)
}

-c1: we restrict the command to only send one (ICMP) packet

$1: This is our input value for our function. We will expect to receive a host address where to "ping".

| grep -c error: From the results of the "ping" command, we will count those lines where "error" string appears.  We will only receive this string when the host is unreachable.

# EXAMPLE OF A "REACHABLE" HOST

[salvatutiempo@localhost temp]$ ping -c1 192.168.56.105
PING 192.168.56.105 (192.168.56.105) 56(84) bytes of data.
64 bytes from 192.168.56.105: icmp_seq=1 ttl=64 time=0.179 ms

--- 192.168.56.105 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.179/0.179/0.179/0.000 ms


# EXAMPLE OF A NON "REACHABLE" HOST

[salvatutiempo@localhost temp]$ ping -c1 192.168.56.111
PING 192.168.56.111 (192.168.56.111) 56(84) bytes of data.
From 192.168.56.104 icmp_seq=1 Destination Host Unreachable

--- 192.168.56.111 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

 

“actions_remote" function

This function will define the tasks that we need to run on destination hosts. In this example, we will try to extract some hardware information from servers, and it will be exported. 

actions_remote() {
     HOSTNAME=$(hostname)
     IPS=$(hostname -I)
     DISTRI=$(hostnamectl | grep Operating | cut -d: -f2)
     CPUUSE=$(top -b -n 1 | grep Cpu | cut -d, -f 1 | cut -d: -f2)
     MEMFREE=$(free -b | grep ^Mem| tr -s " "| cut -d" " -f 4)
     ROOTFREE=$(df / --output=pcent | sed 1d)
     echo "${HOSTNAME},${IPS},${DISTRI},${CPUUSE},${MEMFREE},${ROOTFREE}"
}

 

These are the fields that we will export:

HOSTNAME: Server hostname

IPS: Network IP addresses from every network card

DISTRI: Host Linux distribution

CPUUSE: CPU load

MEMFREE: Available RAM memory (in bytes)

ROOTFREE: Available free space in root partition (/)

All the information is stored in variables, and exported in the last line of the function.

 

4.4. Main Code

Once we have defined the functions, we will explain our main code step by step.

 

Run Servers

In this first step, we will create a "for" loop that will read all our destination hosts addresses. Every line from that file will be stored in a temporary variable “$host”.

for hosts in $(cat "$HOSTS")
do
     # CODE
done

We will define the rest of the code inside that loop

 

Check if it's reachable

We will check if host if "reachable" by running our "pingable" function, and checking results

if [ "$PINGABLE" == "0" ]
then
     # IF_CORRECT code
else
     echo "HOST $host is UNREACHABLE"
fi

 

Connect to server

Once we have confirmed that we can reach the host, we will try to connect to it

In the first part of connection command, we will use SSH protocol. For that, we could use "ssh" command, but we should manually set password for every host that the script will connect to.

So, to automate that process, we will use “sshpass” app.

In order to run the command, we will use some extra options.

SSH="sshpass -e ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null"

 

sshpass -e ssh: This option will run sshpass command using SSHPASS variable as the password to connect to host (-e option). We should have previously set that variable.

SSHPASS=$(cat pass.txt)

 

StrictHostKeychecking: This option will allow the command to accept new connections automatically while connecting to hosts, avoiding the prompt that usually shows.

[user@localhost ~]$ ssh user@192.168.56.105
The authenticity of host '192.168.56.105 (192.168.56.105)' can´t be established.
ECDSA key fingerprint is SHA256:PalS7dsCr0izjrKtK41ED1DKl2X+AHbRPYgcXJfye60.
ECDSA key fingerprint is MD5:26:a2:db:97:30:cc:da:01:cc:8c:07:e8:d0:e6:92:51.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '1.1.1.1' (ECDSA) to the list of known hosts.

 

UserKnownHostsFile: This option will specify the file where connection "keys" will be stored.

In this case, using "/dev/null" as destination, when doing SSH previous check to connect to, system will believe it has never connected before, so it will not generate an "invalid key" error.

 

Run tasks

We will define actions to run once we connect to host.

$SSH "${USER}@${host}" "$(declare -f actions_remote); actions_remote"

 

${USER}@${host}: EIn this section we define host address (host), as well as username that we will use to connect to (USER).

$(declare -f actions_remote); actions_remote: In this section we will include tasks that will be run on host. In our case:

• We declare in destination host the function that we previously defined with all the tasks to run ($(declare -f actions_remote))
• We will run the function (actions_remote)

 

If I run admin tasks (sudo)?

Maybe we need to run tasks that require admin permissions. A typical example could be to reboot multiple servers. 

actions_remote() {
     sudo reboot
}

 

In this case, we should make some changes on the last command line:

$SSH "${USER}@${host}" "echo '${SSHPASS}' | sudo --stdin bash -c '$(declare -f actions_remote); actions_remote'"

 

echo ${SSHPASS} |:  This command will allow us to include our destination host SSHPASS variable. We should have previously set that variable with the user access password.

 

sudo --stdin bash -c: this command will allow us to run the commands that will be run later with admin tasks.

It basically tries to run admin commands (sudo --stdin), receiving use password as input (through SSHPASS variables), and command to run is the shell command (bash -c).

 

$(declare -f actions_remote); actions_remote: In this last section, it will declare the functions that we will need to run on destination host, and run that function.

 

5. Extras

---

5.1. Read Only Permissions

Regarding the file where have stored our password to connect to server, it's a common thing to have more permissions that needed. This can make other users could read it.

[salvatutiempo@localhost temp]$ ls -l pass.txt
-rw-r--r-- 1 root root 5 Dec  8 09:54 pass.txt

 

In this example, our password file has:

• Read and write permissions from the owner of the file (root user)
• Read permissions from every user that belongs to root group
• Read permissions from every other user that does not belong to the previously defined steps

 

We should restrict that access, so that only the user that created can have those permissions. For that, we will run the following command, that will apply those restrictions.

sudo chmod 400 pass.txt

After running, we can check that permissions have changed:

[salvatutiempo@localhost temp]$ sudo chmod 400 pass.txt

[salvatutiempo@localhost temp]$ ls -l pass.txt
-r-------- 1 root root 5 Dec  8 09:54 pass.txt

 

5.2. Encrypt password

In this lab we used a password and stored it in a plan file in order to connect to destination hosts. 

In order to be safer we can encrypt that password, and only decrypt it in code, in order to be used only when needed. In that encryption, a "passphrase" will be asked. That passphrase will be the key to decrypt our file.

For that:

1. Let's suppose that our password stored in “pass.txt” file. We will use GPG to encrypt the file, using the following command:

$ gpg -c pass.txt

2. It will ask us to set a “passphrase” 

lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x Enter passphrase                                    x
x                                                     x
x                                                     x
x Passphrase ________________________________________ x
x                                                     x
x       <OK>                             <Cancel>     x
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj

 

That will generate a new file with the same name as the original file, but with a new extension "gpg" (in our case, "pass.txt.gpg").

3. We will remove our plain fie "pass.txt"

$ sudo rm -rf pass.txt

4. In order to be even safer, we will hide our file. For that, we will just add a dot "." at the beginning og the file name (in our example, ".pass.txt.gpg"). It will only be send using “ls -a” command.

5. We will also store our "passphrase" in our system in order to automatically decrypt the password file. In this example, we will store it in ".passphrase.txt" file (with a "." at the beginning to hide it).

6. We will restrict our "passphrase" permissions so that can be only be read by the owner of the file

chmod 400 .passphrase.txt

 

7. In order to decrypt the file, as we used SSHPASS variable to read from it, we will make that:

• instead of reading “pass.txt” plain file

SSHPASS=$(cat pass.txt)

 

• decrypt our new encrypted file “.pass.txt.gpg” using our “passphrase”

SSHPASS=$(gpg --passphrase-file .passphrase.txt -d .pass.txt.gpg)

 

6. I want the code

---

You can find the whole code in the following repository that I have created:

Whole Code