SSH Jump Server

SSH Jump Server
ssh-jump-server-concept

Introduction     

In recent times, there is an increasing need for organizations to give employees access to their IT facilities due to the ongoing Covid restrictions ( such as work from home )  in place and in other cases grant access to external parties like clients, vendors  who wants to troubleshoot and fix issues with the IT Infrastructure remotely.
 
More so, is the need for multiple manage SSH access to the company’s Linux servers, Routers, Switches, while meeting regulatory and security compliance.
 
This need led to the emergence of the SSH Jump Server concept.
It is a secure intermediary server where all your system administrators would login in first via SSH before getting to access the remote devices such as Linux instance, Routers, Switches etc. The purpose of having the SSH Jump server is to improve security and consolidate SSH user activities to a single point hence better security and accountability. SSH  Jump server is also known by the name SSH Jump Box, SSH Jump Host & SSH Gateway.

What is SSH Jump Server and how does it work?                               

An SSH Jump Server is simply a single, hardened server that you “jump” through in order to access other servers or devices on the inner network.
Sometimes called a SSH Jump host , or SSH Jump server or  ssh bastion host or a relay host, it’s simply a server that all of your users can log into and use as a relay server to connect to other Linux servers, Routers, Switches and more. Therefore, a jump server is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones.
 
In other words, it is an intermediary host or an SSH gateway to a remote network, through which a connection can be made to another host in a dissimilar security zone, for example a demilitarized zone (DMZ2). In short it is intended to breach the gap between two security zones. This is done with the purpose of establishing a gateway to access something inside of the security zone, from the DMZ
 
The SSH Jump Box bridges two dissimilar security zones and offers controlled and monitored access between them.
 
For users accessing your secure network over the internet, the jump host provides a highly secured and monitored environment especially when it spans a private network and a DMZ with servers providing services to users on the internet.
 
Furthermore, a classic scenario is connecting from your desktop or laptop from inside your company’s internal network, which is highly secured with firewalls to a DMZ. In order to easily manage a server in a DMZ, you may access it via a jump host.
 
Therefore, a jump host is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones. An example would be a high security zone inside a corporation. The policy guide states that this zone cannot be accessed directly from a normal user zone. Hence, in a DMZ off the firewall protecting this zone you have a jump host.
 
Connections are permitted to the ssh jump host from the user zone, and access to the secure zone are permitted from the jump host.
 
More often, there is a separate authentication method for the jump host fortified with multi factor authentication, Single Sign On ( SSO ) , Radius  & more. 


How to Configure an SSH Jump Server

  • Using OpenSSH
    A basic ssh jump server with limited features and functionalities  can be configured using OpenSSH packages  that available by default on most Linux distributions. In the example below, we will just use the basic ssh command line to proxy a ssh connection to the remote server via a intermediate jump server.
ssh -J jump_machine remote_machine

If the -J option is not available use the -W option to pivot the connection through an intermediate bastion host.

ssh -o ProxyCommand="ssh -W %h:%p bastion.gateway.org"  remote.server.org

With the OpenSSH 7.3,  the easiest way to pass through hop through intermediate one or more jump hosts is  using the ProxyJump directive ssh_config

Host remote server
HostName 192.168.0.177
ProxyJump [email protected]:22
User devops

Multiple jump hosts can be chained as well

Host remote server
HostName 192.168.0.177
ProxyJump [email protected]:22, [email protected]:22
User devops

 

Do refer the article SSH Proxy and SSH JumpHost  for configuring  a basic jump server that is very limited in feature and functionality when compared to the modern day ssh  jump host solutions.

  • Using Ezeelogin SSH Jump server
     Ezeelogin is a much more powerful and advanced SSH Jump host software solution and  can be deployed quickly.  It has powerful features that  makes managing hundreds of Linux devices and granting ssh access to these device a piece of cake.  Do refer the article to  configure a ssh jump server quickly on your premise or on cloud.                                                    

Why do you need a SSH Jump server solution to manage ssh access? 

The OpenSSH based jump server is clearly not enough to meet the modern day requirements  of an IT enterprise. The challenges for  the enterprise are constantly changing and dynamic . On day , it could be from maintaining security, granting ssh access to the users to designated server and that too for particular time and on another day it could be the security compliances that needs to be met at the time of a Linux servers infrastructure audit.
The modern day  SSH Jump host solutions are designed to address the challenges faced by an IT enterprise when it comes to  security and to meet various security compliances like PCI DSS, NIST, ISO 27001 and more.
 
The modern day ssh jump server  software has the  following features and more.
  • Identity and Access management (IAM)
  • Privileged Access management (PAM),
  • Role Based Access Control to delegate access to Linux servers and Network devices.
  • Two factor authentication methods  like Google Authenticator, DUO Security 2FA, & Yubikey in SSH.
  • Integrates with Windows Active Directory, OpenLDAP, Redhat IDM.
  • Supports SAML for  Single Sign On.
  • Support RADIUS Authentication to access network devices such as Routers and Switches
  • Password Manager
  • SSH key rotation, 
  • Automated root password management

OpenSSH proxy or jump server cannot perform all these advanced activities.

Limitations of normal openssh jump servers:-

  1. The OpenSSH jump servers hold the ssh-certs in plain text format. Since the jump server is a trusted device, any intruder with sufficient privileges can jeopardize the entire network.
  2.  An OpenSSH jump server does not have any mechanism to detect user impersonation, i.e., using another one’s login to access the server. The system will let you in if you have valid login credentials. 
  3.  Routine activities such as user or server addition, removal, setting privileges, or security management takes a lot of system administration time.
  4. Logging the user activities on the remote server from the jumpserver is impossible without placing an agent software on the server.
  5.  Jump server management is through the command-line interface. If you are not a competent Linux administrator, you must hire someone, increasing operational expenses.
  6.  Migrating or upgrading a jump server is a tedious process with a high probability of downtime.
  7. Integrating with other authentication systems or exporting multiple users from systems such as Active Directory or LDAP are not supported.
  8.  Many companies adopt OpenSSH jump server implementation to avoid the costs of purchasing custom-made ssh jump server solutions. But the inherent limitations of ssh jump server cost them dearly by denting their development possibilities. 
  9.  The lack of fail-over systems makes the entire network inaccessible in the event of jump server failure.

Only an ssh jump server solution that moves abreast with the technological and industry demands alone can ensure you a smooth growth trajectory. Ezeelogin, with its innovative features, robust security, and exceptional user experience, helps companies worldwide to scale new heights with absolute confidence.

Jump server best practices

To ensure maximum protection of your ssh jump server, you should focus on the process of server security hardening. In simple terms, that means applying a combination of basic and advanced security measures to address vulnerabilities in your bastion host server and operating system to boost overall server security.

Find how to secure your ssh jump server

 

Jump server comparison:

OpenSSH Vs. Ezeelogin

OpenSSH Jump Server
Ezeelogin Jump Server

Only password or cert-based authentication

Supports 2FA – Google Authenticator, Yubikey, and Duo

Stores SSH Keys in plain text format 

The encrypted keys are stored in databases

Uses default shells. No customization is possible.

Uses custom shell – ezsh

Can’t restrict command execution

Command-Line Guard restricts the user from executing dangerous commands.

Login works only for command line

Both command line and WebSSH login works

Needs professional Linux administration skills

Only a few mouse clicks are required. No server admin expertise is essential.

Command execution on multiple servers requires separate ssh logins.

Parallel shell enables simultaneous execution of commands across multiple servers.

User activity session recording is possible only through agent software.

No agent software is required for ssh session logging.

Password Reset, Rotation, and login sharing are quite clumsy processes.

Password management is automated.

Single point of failure – you can’t access your servers if the jump server is down.

Master-slave architecture to avoid the single point failure

Achieving security compliance is hard.

Can easily fulfill security compliance requirements

Permits access only for system users.

Login for LDAP and Active Directory users is possible.

CONCLUSION

IT Enterprises that use a Ezeelogin SSH Jump Server solution in improving security of their critical IT asset and in meeting various mandatory security compliances  (which would otherwise prove very costly in case of a breach),  are more likely to succeed due to the improved operational efficiency, digital security, hence more successful business for the company’s end customers. Ezeelogin jump server helps the organizations worldwide to unleash their growth potential without any limitations and helps them grow bigger and bigger without any server management worries.

References

SSH Proxy and SSH JumpHost

How to Record Linux Terminal/SSH Sessions?

What’s the Purpose of Recording SSH Sessions?

After the covid 19 pandemic, there are a lot of system administrators working from home and using
remote software for managing and configuring servers remotely. Most system administrators
choose the SSH protocol for remote administration and management of Linux-based servers. In
the enterprise environment, a lot of users are working on the same server via SSH. This may lead
to internal data leaks or other threats either intentionally or unintentionally. This is where the SSH
session recording comes into the picture. Recording SSH sessions for administrators and users
have always been a demand for security and knowledge-sharing purposes.
SSH session recording allows you to track everything that the user runs on the terminal and play
the recorded session later for auditing purposes.
It will help you to find out abuse to reduce the risk of suspicious activity on the server before they
result in data breaches. In simple words, SSH session recording helps you to identify what
happened when, where, and by whom.

What are the different methods to record SSH sessions?

There are several methods to record SSH sessions in Linux-based distributions. Some of them are
listed below:

  1. Bash history command
  2. Script command
  3. Ezeelogin

How to record SSH sessions using the bash history command?

The history command is a Linux command-line utility that allows you to track all commands
executed by the user in a Linux terminal. It is a very useful tool for system administrators to audit
all commands with the date and time executed in the terminal session.

How to configure history command settings?
By default, all executed commands are stored in the .bash_history file located in each user’s
HOME directory. You can also define the number of command stores in the history file.
There are two options to configure:
HISTFILESIZE – Allows you to define a number of commands kept in the history file
HISTSIZE – Allows you to define the number of commands loaded from the memory.
You can configure the above options by editing the .bashrc file:

         nano ~/.bashrc
Find both the HISTSIZE and HISTFILESIZE parameters and change it with your required values:
        HISTSIZE=10000
        HISTFILESIZE=50000
Save and close the .bashrc file

Also, bash saves all executed commands in the history file at the end of each session and
overwrites the existing history file. You can change the default setting by editing the .bashrc file:

       nano ~/.bashrc
    Add the following line:
       shopt -s histappend
    Save and close the file.

By default, bash adds all commands to your history file after the end of the session. If you want to
add the executed command immediately, edit the .bashrc file:

         nano ~/.bashrc
     Add the following lines:
        export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"
Save and close the file then run the following command to apply the changes:
        source ~/.bashrc

How to Use the History Command?

In this section, we will show you how to use the history command to track and see the previously
executed command. Let’s run the history command without any argument:

          :~$ history

This will show you all previously executed commands saved in the history file:

How to display last 2 executed lines using history command?

To display the last 2 executed commands, run the following command:

     :~$ history 2

This command will show you the last two executed commands:

How to search particular command using history?

You can also use the search pattern to filter the specific command from the history file.

:~$ history | grep apt

This command will find commands that match the pattern apt:

How to delete specific command from history command?

If you want to delete the specific command from the history file use the -d option. For example, delete the command number 4011 from the history list, and run the following command:

       history -d 4011

How to clear all the history?

To clear all history, run the following command:

    history -c

Is it possible to disable recording of particular command using history?

Yes, to disable recording the executed command, run the following command:

       set +o history

How to re-enable recording in bash history?

You can also re-enable it with the following command:

         set -o history

2. Record SSH Sessions Using the Script Command

The script is a Linux command line utility that allows users to track and record all commands executed in terminal sessions. You can also play the recorded session later via the command line interface.

Steps to Install Script Utility

By default, the Script tool comes pre-installed on all major Linux distributions. You can also install it by yourself if it is not available.

a. How to install script utility on Ubuntu and Debian-based operating systems?

Execute the following command
     apt-get install util-linux -y

b. How to install script utility on RHEL, CentOS and Fedora-based operating systems

     dnf install util-linux -y

Record SSH Session Using the Script Command

By default, the script command record and saves all executed commands in a file called typescript in your current working directory.

Let’s see how to record SSH Sessions using the script command :

  To start recording SSH session just execute the command script.
                  :~$ script
               Script started, file is typescript
  This will start the recording. Now, let's run some commands on the terminal               
                 :~$ pwd
                 :~$ who
                 :~$ whoami
                 :~$ free -m
                 :~$ ls
                 :~$ echo testing
Now, stop the recording and exit from the script session using the following command:
                 :~$ exit
You should see the following output:
               Script done, file is typescript
You can now use the cat command to see the content of the typescript file:
              cat typescript
You should see all your executed command

Record Session History in Custom File using script command

The script command can record and store session history in a typescript file. You can also define your own file to save the session history. To save the session history in a custom file called sessionhistory.txt, run the following command:

       script session-history.txt

You can exit from the script session using the following command:

      exit

You can also use the -a option to append the session history to the existing file.

    script -a session-history.txt

The script command also allows you to record session history with timing information. You can achieve this using the —timing option.

Record Session History with Timing Information

Let’s run the script command and capture the session history with timing information:

      script --timing=timing-info.txt session-history.txt
Next, run some commands on the terminal:

Play Recorded Terminal Session

The scriptreplay is a Linux command-line utility that allows you to replay the recorded terminal session.

You can replay the recorded session by specifying the session and timing log file:

   scriptreplay --timing=timing-info.txt session-history.txt
You should see the recorded session on the following screen:

3. How to Record SSH Sessions Established via a Jump Server

In this section, we will show you how to record SSH sessions of users accessing remote servers using a self-hosted Jump server and Ezeelogin.

How to Record SSH Sessions of Users Accessing Remote Servers via Self-Hosted SSH Jump Server

A Jump server is a central server where all users can access all servers hosted on the private network from a public network. A Jump server can minimize the chance of a potential attack.

In this section, we will show you how to configure the Jump server on a Linux machine. We will then record and track all SSH sessions of users who are accessing the remote server via the Jump server.

First, create a directory to store all recorded log files and give necessary permission:

mkdir /var/log/jump
chmod -R 777 /var/log/jump

Next, edit the SSH configuration file and modify some default parameters:

nano /etc/ssh/sshd_config
Change the following lines:

AllowTcpForwarding no
X11Forwarding no

Then, add the following line at the end of the file:

ForceCommand /usr/bin/jump/shell

Save and close the file then create a custom OpenSSH script that runs when any user login to the Jump server via SSH:

mkdir /usr/bin/jump

nano /usr/bin/jump/shell

Add the following code:

if [[ -z $SSH_ORIGINAL_COMMAND ]]; then

# The format of log files is /var/log/jump/YYYY-MM-DD_HH-MM-SS_user

LOG_FILE="`date --date="today" "+%Y-%m-%d_%H-%M-%S"`_`whoami`"
LOG_DIR="/var/log/jump/"

# Print a welcome message

echo ""
echo "NOTE: This SSH session will be recorded"
echo "AUDIT KEY: $LOG_FILE"
echo ""

# I suffix the log file name with a random string. I explain why
# later on.
SUFFIX=`mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`
# Wrap an interactive shell into "script" to record the SSH session

script -qf --timing=$LOG_DIR$LOG_FILE$SUFFIX.time $LOG_DIR$LOG_FILE$SUFFIX.data --command=/bin/bash
else
echo "This jump supports interactive sessions only. Do not supply a command"
exit 1
fi

Save and close the file then give executable permission to the script:

chmod a+x /usr/bin/jump/shell

Next, restart the SSH service to apply the configuration changes:

service sshd restart

Next, create a new user called jumpuser1 for which you want to record all terminal session activities.

adduser jumpuser1

At this point, the Jump server is configured to record and track all users’ activity that is connecting to a remote server via the Jump server.

Next, go to your local machine and log in to your Jump server via SSH.

ssh [email protected]

Once you are logged in, you should see a message saying that your SSH terminal session will be recorded:

Next, log in to the remote server from the Jump server using the following command:

ssh [email protected]

After the successful login, run the following commands one by one on the terminal:

pwd
date
uptime
df -h
free -m
ls
whoami

Next, exit from the remote server with the following command:

exit

Your SSH session is now recorded and save the recorded log files in the /var/log/jump/ directory. You can check the generated log files using the following command:

ls -l /var/log/jump/

You should see both files in the following output:

total 8
-rw-rw-r-- 1 jumpuser1 jumpuser1 2509 Nov 17 09:16 2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.data
-rw-rw-r-- 1 jumpuser1 jumpuser1 1352 Nov 17 09:16 2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.time

You can now use the cat command to view the recorded session logs:

cat /var/log/jump/2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.data

You should see all command history that is executed on the remote server in the following screen:

You can also replay the recorded SSH session using the scriptreplay command:

cd /var/log/jump/
scriptreplay --timing=2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.time 2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.data

How to Record SSH Sessions of Users Accessing Remote Servers via Ezeelogin?

The SSH-based Jump server is not ideal for modern IT infrastructure requirements due to its limitation. Ezeelogin is a secure and web-based Bastion host software tool that allows you to set up your own Jump server on a Linux machine. It provides a simple and user-friendly web interface where you can monitor and see all users’ terminal session activity.

Refer complete tutorial on record the SSH session via Ezeelogin.

Comparison Chart Between History , Script and Record SSH Session in Ezeelogin

FeatureHistory CommandScript CommandRecord SSH Session In Ezeelogin
Log ExportNoNoYes
Correct Login TimeNoNoYes
SearchableYesYesYes
Timestamps of SSH LogsNoNoYes
Automatic Truncation of Logs Based of Size

NoNoYes
Input RecordingYesYesYes
Output RecordingNoYesYes
Delete HistoryYesYesYes
Search For a Particular TimeNoYesYes
Live StreamingNoNoYes
View Active or Ongoing SessionsNoNoYes

Related articles

Complete tutorial on record SSH Sessions in Ezeelogin