In recent times, there is an increasing need for organizations to give employees access to their IT facilities due to the ongoing Covid restrictions ( such as work from home ) in place and in other cases grant access to external parties like clients, vendors who wants to troubleshoot and fix issues with the IT Infrastructure remotely.
More so, is the need for multiple manage SSH access to the company’s Linux servers, Routers, Switches, while meeting regulatory and security compliance.
This need led to the emergence of the SSH Jump Server concept.
It is a secure intermediary server where all your system administrators would login in first via SSH before getting to access the remote devices such as Linux instance, Routers, Switches etc. The purpose of having the SSH Jump server is to improve security and consolidate SSH user activities to a single point hence better security and accountability. SSH Jump server is also known by the name SSH Jump Box, SSH Jump Host & SSH Gateway.
What is SSH Jump Server and how does it work?
An SSH Jump Server is simply a single, hardened server that you “jump” through in order to access other servers or devices on the inner network.
Sometimes called a SSH Jump host , or SSH Jump server or ssh bastion host or a relay host, it’s simply a server that all of your users can log into and use as a relay server to connect to other Linux servers, Routers, Switches and more. Therefore, a jump server is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones.
In other words, it is an intermediary host or an SSH gateway to a remote network, through which a connection can be made to another host in a dissimilar security zone, for example a demilitarized zone (DMZ2). In short it is intended to breach the gap between two security zones. This is done with the purpose of establishing a gateway to access something inside of the security zone, from the DMZ
The SSH Jump Box bridges two dissimilar security zones and offers controlled and monitored access between them.
For users accessing your secure network over the internet, the jump host provides a highly secured and monitored environment especially when it spans a private network and a DMZ with servers providing services to users on the internet.
Furthermore, a classic scenario is connecting from your desktop or laptop from inside your company’s internal network, which is highly secured with firewalls to a DMZ. In order to easily manage a server in a DMZ, you may access it via a jump host.
Therefore, a jump host is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones. An example would be a high security zone inside a corporation. The policy guide states that this zone cannot be accessed directly from a normal user zone. Hence, in a DMZ off the firewall protecting this zone you have a jump host.
Connections are permitted to the ssh jump host from the user zone, and access to the secure zone are permitted from the jump host.
More often, there is a separate authentication method for the jump host fortified with multi factor authentication, Single Sign On ( SSO ) , Radius & more.
How to Configure an SSH Jump Server
Using OpenSSH A basic ssh jump server with limited features and functionalities can be configured using OpenSSH packages that available by default on most Linux distributions. In the example below, we will just use the basic ssh command line to proxy a ssh connection to the remote server via a intermediate jump server.
ssh -J jump_machine remote_machine
If the -J option is not available use the -W option to pivot the connection through an intermediate bastion host.
Do refer the article SSH Proxy and SSH JumpHost for configuring a basic jump server that is very limited in feature and functionality when compared to the modern day ssh jump host solutions.
Using Ezeelogin SSH Jump server Ezeelogin is a much more powerful and advanced SSH Jump host software solution and can be deployed quickly. It has powerful features that makes managing hundreds of Linux devices and granting ssh access to these device a piece of cake. Do refer the article to configure a ssh jump server quickly on your premise or on cloud.
Why do you need a SSH Jump server solution to manage ssh access?
The OpenSSH based jump server is clearly not enough to meet the modern day requirements of an IT enterprise. The challenges for the enterprise are constantly changing and dynamic . On day , it could be from maintaining security, granting ssh access to the users to designated server and that too for particular time and on another day it could be the security compliances that needs to be met at the time of a Linux servers infrastructure audit.
The modern day SSH Jump host solutions are designed to address the challenges faced by an IT enterprise when it comes to security and to meet various security compliances like PCI DSS, NIST, ISO 27001 and more.
The modern day ssh jump server software has the following features and more.
Identity and Access management (IAM)
Privileged Access management (PAM),
Role Based Access Control to delegate access to Linux servers and Network devices.
Two factor authentication methods like Google Authenticator, DUO Security 2FA, & Yubikey in SSH.
Integrates with Windows Active Directory, OpenLDAP, Redhat IDM.
Supports SAML for Single Sign On.
Support RADIUS Authentication to access network devices such as Routers and Switches
SSH key rotation,
Automated root password management
OpenSSH proxy or jump server cannot perform all these advanced activities.
Limitations of normal openssh jump servers:-
The OpenSSH jump servers hold the ssh-certs in plain text format. Since the jump server is a trusted device, any intruder with sufficient privileges can jeopardize the entire network.
An OpenSSH jump server does not have any mechanism to detect user impersonation, i.e., using another one’s login to access the server. The system will let you in if you have valid login credentials.
Routine activities such as user or server addition, removal, setting privileges, or security management takes a lot of system administration time.
Logging the user activities on the remote server from the jumpserver is impossible without placing an agent software on the server.
Jump server management is through the command-line interface. If you are not a competent Linux administrator, you must hire someone, increasing operational expenses.
Migrating or upgrading a jump server is a tedious process with a high probability of downtime.
Integrating with other authentication systems or exporting multiple users from systems such as Active Directory or LDAP are not supported.
Many companies adopt OpenSSH jump server implementation to avoid the costs of purchasing custom-made ssh jump server solutions. But the inherent limitations of ssh jump server cost them dearly by denting their development possibilities.
The lack of fail-over systems makes the entire network inaccessible in the event of jump server failure.
Only an ssh jump server solution that moves abreast with the technological and industry demands alone can ensure you a smooth growth trajectory. Ezeelogin, with its innovative features, robust security, and exceptional user experience, helps companies worldwide to scale new heights with absolute confidence.
Jump server best practices
To ensure maximum protection of your ssh jump server, you should focus on the process of server security hardening. In simple terms, that means applying a combination of basic and advanced security measures to address vulnerabilities in your bastion host server and operating system to boost overall server security.
Supports 2FA – Google Authenticator, Yubikey, and Duo
Stores SSH Keys in plain text format
The encrypted keys are stored in databases
Uses default shells. No customization is possible.
Uses custom shell – ezsh
Can’t restrict command execution
Command-Line Guard restricts the user from executing dangerous commands.
Login works only for command line
Both command line and WebSSH login works
Needs professional Linux administration skills
Only a few mouse clicks are required. No server admin expertise is essential.
Command execution on multiple servers requires separate ssh logins.
Parallel shell enables simultaneous execution of commands across multiple servers.
User activity session recording is possible only through agent software.
No agent software is required for ssh session logging.
Password Reset, Rotation, and login sharing are quite clumsy processes.
Password management is automated.
Single point of failure – you can’t access your servers if the jump server is down.
Master-slave architecture to avoid the single point failure
Achieving security compliance is hard.
Can easily fulfill security compliance requirements
Permits access only for system users.
Login for LDAP and Active Directory users is possible.
IT Enterprises that use a Ezeelogin SSH Jump Server solution in improving security of their critical IT asset and in meeting various mandatory security compliances (which would otherwise prove very costly in case of a breach), are more likely to succeed due to the improved operational efficiency, digital security, hence more successful business for the company’s end customers. Ezeelogin jump server helps the organizations worldwide to unleash their growth potential without any limitations and helps them grow bigger and bigger without any server management worries.
After the covid 19 pandemic, there are a lot of system administrators working from home and using remote software for managing and configuring servers remotely. Most system administrators choose the SSH protocol for remote administration and management of Linux-based servers. In the enterprise environment, a lot of users are working on the same server via SSH. This may lead to internal data leaks or other threats either intentionally or unintentionally. This is where the SSH session recording comes into the picture. Recording SSH sessions for administrators and users have always been a demand for security and knowledge-sharing purposes. SSH session recording allows you to track everything that the user runs on the terminal and play the recorded session later for auditing purposes. It will help you to find out abuse to reduce the risk of suspicious activity on the server before they result in data breaches. In simple words, SSH session recording helps you to identify what happened when, where, and by whom.
What are the different methods to record SSH sessions?
There are several methods to record SSH sessions in Linux-based distributions. Some of them are listed below:
Bash history command
How to record SSH sessions using the bash history command?
The history command is a Linux command-line utility that allows you to track all commands executed by the user in a Linux terminal. It is a very useful tool for system administrators to audit all commands with the date and time executed in the terminal session.
How to configure history command settings? By default, all executed commands are stored in the .bash_history file located in each user’s HOME directory. You can also define the number of command stores in the history file. There are two options to configure: HISTFILESIZE – Allows you to define a number of commands kept in the history file HISTSIZE – Allows you to define the number of commands loaded from the memory. You can configure the above options by editing the .bashrc file:
Find both the HISTSIZE and HISTFILESIZE parameters and change it with your required values:
Save and close the .bashrc file
Also, bash saves all executed commands in the history file at the end of each session and overwrites the existing history file. You can change the default setting by editing the .bashrc file:
Add the following line:
shopt -s histappend
Save and close the file.
By default, bash adds all commands to your history file after the end of the session. If you want to add the executed command immediately, edit the .bashrc file:
Add the following lines:
export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"
Save and close the file then run the following command to apply the changes:
How to Use the History Command?
In this section, we will show you how to use the history command to track and see the previously executed command. Let’s run the history command without any argument:
This will show you all previously executed commands saved in the history file:
How to display last 2 executed lines using history command?
To display the last 2 executed commands, run the following command:
:~$ history 2
This command will show you the last two executed commands:
How to search particular command using history?
You can also use the search pattern to filter the specific command from the history file.
:~$ history | grep apt
This command will find commands that match the pattern apt:
How to delete specific command from history command?
If you want to delete the specific command from the history file use the -d option. For example, delete the command number 4011 from the history list, and run the following command:
history -d 4011
How to clear all the history?
To clear all history, run the following command:
Is it possible to disable recording of particular command using history?
Yes, to disable recording the executed command, run the following command:
set +o history
How to re-enable recording in bash history?
You can also re-enable it with the following command:
set -o history
2. Record SSH Sessions Using the Script Command
The script is a Linux command line utility that allows users to track and record all commands executed in terminal sessions. You can also play the recorded session later via the command line interface.
Steps to Install Script Utility
By default, the Script tool comes pre-installed on all major Linux distributions. You can also install it by yourself if it is not available.
a. How to install script utility on Ubuntu and Debian-based operating systems?
Execute the following command
apt-get install util-linux -y
b. How to install script utility on RHEL, CentOS and Fedora-based operating systems
dnf install util-linux -y
Record SSH Session Using the Script Command
By default, the script command record and saves all executed commands in a file called typescript in your current working directory.
Let’s see how to record SSH Sessions using the script command :
To start recording SSH session just execute the command script.
Script started, file is typescript
This will start the recording. Now, let's run some commands on the terminal
:~$ free -m
:~$ echo testing
Now, stop the recording and exit from the script session using the following command:
You should see the following output:
Script done, file is typescript
You can now use the cat command to see the content of the typescript file:
You should see all your executed command
Record Session History in Custom File using script command
The script command can record and store session history in a typescript file. You can also define your own file to save the session history. To save the session history in a custom file called session–history.txt, run the following command:
You can exit from the script session using the following command:
You can also use the -a option to append the session history to the existing file.
script -a session-history.txt
The script command also allows you to record session history with timing information. You can achieve this using the —timing option.
Record Session History with Timing Information
Let’s run the script command and capture the session history with timing information:
script --timing=timing-info.txt session-history.txt
Next, run some commands on the terminal:
Play Recorded Terminal Session
The scriptreplay is a Linux command-line utility that allows you to replay the recorded terminal session.
You can replay the recorded session by specifying the session and timing log file:
scriptreplay --timing=timing-info.txt session-history.txt
You should see the recorded session on the following screen:
3. How to Record SSH Sessions Established via a Jump Server
In this section, we will show you how to record SSH sessions of users accessing remote servers using a self-hosted Jump server and Ezeelogin.
How to Record SSH Sessions of Users Accessing Remote Servers via Self-Hosted SSH Jump Server
A Jump server is a central server where all users can access all servers hosted on the private network from a public network. A Jump server can minimize the chance of a potential attack.
In this section, we will show you how to configure the Jump server on a Linux machine. We will then record and track all SSH sessions of users who are accessing the remote server via the Jump server.
First, create a directory to store all recorded log files and give necessary permission:
chmod -R 777 /var/log/jump
Next, edit the SSH configuration file and modify some default parameters:
Change the following lines:
Then, add the following line at the end of the file:
Save and close the file then create a custom OpenSSH script that runs when any user login to the Jump server via SSH:
Add the following code:
if [[ -z $SSH_ORIGINAL_COMMAND ]]; then
# The format of log files is /var/log/jump/YYYY-MM-DD_HH-MM-SS_user
LOG_FILE="`date --date="today" "+%Y-%m-%d_%H-%M-%S"`_`whoami`"
# Print a welcome message
echo "NOTE: This SSH session will be recorded"
echo "AUDIT KEY: $LOG_FILE"
# I suffix the log file name with a random string. I explain why
# later on.
SUFFIX=`mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`
# Wrap an interactive shell into "script" to record the SSH session
script -qf --timing=$LOG_DIR$LOG_FILE$SUFFIX.time $LOG_DIR$LOG_FILE$SUFFIX.data --command=/bin/bash
echo "This jump supports interactive sessions only. Do not supply a command"
Save and close the file then give executable permission to the script:
chmod a+x /usr/bin/jump/shell
Next, restart the SSH service to apply the configuration changes:
service sshd restart
Next, create a new user called jumpuser1 for which you want to record all terminal session activities.
At this point, the Jump server is configured to record and track all users’ activity that is connecting to a remote server via the Jump server.
Next, go to your local machine and log in to your Jump server via SSH.
You should see all command history that is executed on the remote server in the following screen:
You can also replay the recorded SSH session using the scriptreplay command:
scriptreplay --timing=2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.time 2022-11-17_09-15-05_jumpuser1_GgGSQLnHGRx0wojUjPnsks865ggl4lSS.data
How to Record SSH Sessions of Users Accessing Remote Servers via Ezeelogin?
The SSH-basedJump server is not ideal for modern IT infrastructure requirements due to its limitation. Ezeelogin is a secure and web-based Bastion host software tool that allows you to set up your own Jump server on a Linux machine. It provides a simple and user-friendly web interface where you can monitor and see all users’ terminal session activity.