SSH Jump Server

SSH Jump Server
ssh-jump-server-concept

Introduction     

In recent times, there is an increasing need for organizations to give employees access to their IT facilities due to the ongoing Covid restrictions ( such as work from home )  in place and in other cases grant access to external parties like clients, vendors  who wants to troubleshoot and fix issues with the IT Infrastructure remotely.
 
More so, is the need for multiple manage SSH access to the company’s Linux servers, Routers, Switches, while meeting regulatory and security compliance.
 
This need led to the emergence of the SSH Jump Server concept.
It is a secure intermediary server where all your system administrators would login in first via SSH before getting to access the remote devices such as Linux instance, Routers, Switches etc. The purpose of having the SSH Jump server is to improve security and consolidate SSH user activities to a single point hence better security and accountability. SSH  Jump server is also known by the name SSH Jump Box, SSH Jump Host & SSH Gateway.

What is SSH Jump Server and how does it work?                               

An SSH Jump Server is simply a single, hardened server that you “jump” through in order to access other servers or devices on the inner network.
Sometimes called a SSH Jump host , or SSH Jump server or  ssh bastion host or a relay host, it’s simply a server that all of your users can log into and use as a relay server to connect to other Linux servers, Routers, Switches and more. Therefore, a jump server is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones.
 
In other words, it is an intermediary host or an SSH gateway to a remote network, through which a connection can be made to another host in a dissimilar security zone, for example a demilitarized zone (DMZ2). In short it is intended to breach the gap between two security zones. This is done with the purpose of establishing a gateway to access something inside of the security zone, from the DMZ
 
The SSH Jump Box bridges two dissimilar security zones and offers controlled and monitored access between them.
 
For users accessing your secure network over the internet, the jump host provides a highly secured and monitored environment especially when it spans a private network and a DMZ with servers providing services to users on the internet.
 
Furthermore, a classic scenario is connecting from your desktop or laptop from inside your company’s internal network, which is highly secured with firewalls to a DMZ. In order to easily manage a server in a DMZ, you may access it via a jump host.
 
Therefore, a jump host is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones. An example would be a high security zone inside a corporation. The policy guide states that this zone cannot be accessed directly from a normal user zone. Hence, in a DMZ off the firewall protecting this zone you have a jump host.
 
Connections are permitted to the ssh jump host from the user zone, and access to the secure zone are permitted from the jump host.
 
More often, there is a separate authentication method for the jump host fortified with multi factor authentication, Single Sign On ( SSO ) , Radius  & more. 


How to Configure an SSH Jump Server

  • Using OpenSSH
    A basic ssh jump server with limited features and functionalities  can be configured using OpenSSH packages  that available by default on most Linux distributions. In the example below, we will just use the basic ssh command line to proxy a ssh connection to the remote server via a intermediate jump server.
ssh -J jump_machine remote_machine

If the -J option is not available use the -W option to pivot the connection through an intermediate bastion host.

ssh -o ProxyCommand="ssh -W %h:%p bastion.gateway.org"  remote.server.org

With the OpenSSH 7.3,  the easiest way to pass through hop through intermediate one or more jump hosts is  using the ProxyJump directive ssh_config

Host remote server
HostName 192.168.0.177
ProxyJump [email protected]:22
User devops

Multiple jump hosts can be chained as well

Host remote server
HostName 192.168.0.177
ProxyJump [email protected]:22, [email protected]:22
User devops

 

Do refer the article SSH Proxy and SSH JumpHost  for configuring  a basic jump server that is very limited in feature and functionality when compared to the modern day ssh  jump host solutions.

  • Using Ezeelogin SSH Jump server
     Ezeelogin is a much more powerful and advanced SSH Jump host software solution and  can be deployed quickly.  It has powerful features that  makes managing hundreds of Linux devices and granting ssh access to these device a piece of cake.  Do refer the article to  configure a ssh jump server quickly on your premise or on cloud.                                                    

Why do you need a SSH Jump server solution to manage ssh access? 

The OpenSSH based jump server is clearly not enough to meet the modern day requirements  of an IT enterprise. The challenges for  the enterprise are constantly changing and dynamic . On day , it could be from maintaining security, granting ssh access to the users to designated server and that too for particular time and on another day it could be the security compliances that needs to be met at the time of a Linux servers infrastructure audit.
The modern day  SSH Jump host solutions are designed to address the challenges faced by an IT enterprise when it comes to  security and to meet various security compliances like PCI DSS, NIST, ISO 27001 and more.
 
The modern day ssh jump server  software has the  following features and more.
  • Identity and Access management (IAM)
  • Privileged Access management (PAM),
  • Role Based Access Control to delegate access to Linux servers and Network devices.
  • Two factor authentication methods  like Google Authenticator, DUO Security 2FA, & Yubikey in SSH.
  • Integrates with Windows Active Directory, OpenLDAP, Redhat IDM.
  • Supports SAML for  Single Sign On.
  • Support RADIUS Authentication to access network devices such as Routers and Switches
  • Password Manager
  • SSH key rotation, 
  • Automated root password management

OpenSSH proxy or jump server cannot perform all these advanced activities.

Limitations of normal openssh jump servers:-

  1. The OpenSSH jump servers hold the ssh-certs in plain text format. Since the jump server is a trusted device, any intruder with sufficient privileges can jeopardize the entire network.
  2.  An OpenSSH jump server does not have any mechanism to detect user impersonation, i.e., using another one’s login to access the server. The system will let you in if you have valid login credentials. 
  3.  Routine activities such as user or server addition, removal, setting privileges, or security management takes a lot of system administration time.
  4. Logging the user activities on the remote server from the jumpserver is impossible without placing an agent software on the server.
  5.  Jump server management is through the command-line interface. If you are not a competent Linux administrator, you must hire someone, increasing operational expenses.
  6.  Migrating or upgrading a jump server is a tedious process with a high probability of downtime.
  7. Integrating with other authentication systems or exporting multiple users from systems such as Active Directory or LDAP are not supported.
  8.  Many companies adopt OpenSSH jump server implementation to avoid the costs of purchasing custom-made ssh jump server solutions. But the inherent limitations of ssh jump server cost them dearly by denting their development possibilities. 
  9.  The lack of fail-over systems makes the entire network inaccessible in the event of jump server failure.

Only an ssh jump server solution that moves abreast with the technological and industry demands alone can ensure you a smooth growth trajectory. Ezeelogin, with its innovative features, robust security, and exceptional user experience, helps companies worldwide to scale new heights with absolute confidence.

Jump server best practices

To ensure maximum protection of your ssh jump server, you should focus on the process of server security hardening. In simple terms, that means applying a combination of basic and advanced security measures to address vulnerabilities in your bastion host server and operating system to boost overall server security.

Find how to secure your ssh jump server

 

Jump server comparison:

OpenSSH Vs. Ezeelogin

OpenSSH Jump Server
Ezeelogin Jump Server

Only password or cert-based authentication

Supports 2FA – Google Authenticator, Yubikey, and Duo

Stores SSH Keys in plain text format 

The encrypted keys are stored in databases

Uses default shells. No customization is possible.

Uses custom shell – ezsh

Can’t restrict command execution

Command-Line Guard restricts the user from executing dangerous commands.

Login works only for command line

Both command line and WebSSH login works

Needs professional Linux administration skills

Only a few mouse clicks are required. No server admin expertise is essential.

Command execution on multiple servers requires separate ssh logins.

Parallel shell enables simultaneous execution of commands across multiple servers.

User activity session recording is possible only through agent software.

No agent software is required for ssh session logging.

Password Reset, Rotation, and login sharing are quite clumsy processes.

Password management is automated.

Single point of failure – you can’t access your servers if the jump server is down.

Master-slave architecture to avoid the single point failure

Achieving security compliance is hard.

Can easily fulfill security compliance requirements

Permits access only for system users.

Login for LDAP and Active Directory users is possible.

CONCLUSION

IT Enterprises that use a Ezeelogin SSH Jump Server solution in improving security of their critical IT asset and in meeting various mandatory security compliances  (which would otherwise prove very costly in case of a breach),  are more likely to succeed due to the improved operational efficiency, digital security, hence more successful business for the company’s end customers. Ezeelogin jump server helps the organizations worldwide to unleash their growth potential without any limitations and helps them grow bigger and bigger without any server management worries.

References

SSH Proxy and SSH JumpHost

Tutorial on SSH Agent

What is an SSH Agent?

The SSH agent is a key manager for SSH service, which allow us to authenticate remote Linux system interactively. It stores the user’s SSH key and passphrases in process memory. It is one type of single sign-on (SSO). It uses stored SSH keys and passphrases to authenticate to the remote server without typing a password or passphrase again.

In this tutorial on SSH Agent, we will explain how to set up an SSH-Agent and use it to authenticate a remote Linux system.

How to generate SSH Key pair?

Before starting, you will need to create a SSH key pair ( create public and private SSH Key pair ) to authenticate the remote Linux system without using a password.

Let’s run the ssh-keygen command to generate a public and private SSH key:

ssh-keygen -t rsa

Define your SSH key path and passphrase as shown below:

sshkey

As you can see, both public and private keys are created and stored in the  ~/.ssh directory. 

Now, you will need to add the public key on the remote Linux server for password-less authentication. 

You can use the ssh-copy-id command followed by the remote Linux server IP address to add the SSH key on the remote Linux server:

ssh-copy-id [email protected]

You will need to provide the root password to add the public key on the remote server:

add sshkey

At this point, the SSH key is added to the remote server. You can now proceed to set up an SSH agent.

How to setup an SSH-Agent

Before setting up this, you will need to start the agent.

How to start an ssh agent?

To start, first open your command-line terminal and run the following command:

eval $(ssh-agent)

The above command will start the ssh-agent service in a daemon mode:

Agent pid 7019

[email protected]:#$ eval "$(ssh-agent -s)"
Agent pid 7019

Next, add the SSH key passphrase which you have defined during the ssh key pair generation process to the SSH agent using the ssh-add command:

ssh-add

You will also need to provide your SSH key passphrase to add it to the SSH agent as shown below:

ssh-add

Note: The SSH Key passphrases will automatically be removed from the system memory when you log out from your terminal session.

You can also define the specific time limit to remember the SSH passphrase. You can define it in seconds, minutes, or hours.

How to set time limit in ssh agent?

To set a time limit of 600 seconds, run:

ssh-add -t 600

To set a time limit of 50 minutes, run:

ssh-add -t 50m

To set a time limit of 1 hour, run:

ssh-add -t 1h

Now, verify the SSH passphrase cached by ssh-agent using the following command:

ssh-add -l

If everything is configured properly, you should see the following output:

2048 SHA256:qENeCHiuTkho5/oEfrr49MlFc+VKQiaXfYq47bnMCJU /home/vyom/.ssh/id_rsa (RSA)

How to list Keys added to ssh agent ?

You can also see all public key parameters using the following command:

ssh-add -L

You should get the following output:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVOoidnUeRzBJ0ni4Tbr3T1n+aTP/MM6LyRI1t7MqIh426yjBroY+BYG3NEel5dAhKtB++ZEujp3dU+Mq7YydzCGzKj4Iv2qQ5iTdUIZyk9uOmH5bR0bq7YeIP9UOptniKoliaLfRQpPliCTXXBFZtEiKROJ6W1NHgsXNOT/JNtLgg7GN5NtRPpPZ0agwXPAMSeYExEBrfoz8pVEoRlhqMxnk9IsFnSMCREN4LNMRxY4gk9qf6ydi08t/pcusxaT7hf7vj5+6jJzC2jMzqIUzMNt7sqaqWOtSqk5tMaopEIIiJ4imCfRtczvaxy9BHb5prFJu/ONC3LxMSPl3vVkcL /home/vyom/.ssh/id_rsa

How to stop or disable ssh agent ?

You can use the below command to stop.

[email protected]:~$ eval "$(ssh-agent -k)"
Agent pid 182 killed

How to remove keys from ssh agent?

You can remove already added keys from using the command

ssh-add -D

How to Use ssh agent for Authentication on Linux

At this point, the passphrase of your key is added to the agent. Now, you don’t need to provide an SSH passphrase when connecting to the remote server.

Let’s use the SSH command followed by remote server IP to connect to the remote Linux system:

ssh [email protected]

You should log in to the remote Linux system without any passphrase prompt:

Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 5.4.0-110-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

174 updates can be applied immediately.
125 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable

New release '20.04.4 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Your Hardware Enablement Stack (HWE) is supported until April 2023.
You have mail.
[email protected]:~# 

You can also use the SSH Agent with SCP and SFTP commands to transfer files to and from the remote Linux server.

If you want to delete all cached passphrases, run the following command:

ssh-add -D

The – D argument supported by the command helps eliminate all the added keys from the client. However, the lowercase version of the argument, i.e., -d allows you to select individual keys.

What is ssh agent Forwarding?

SSH agent forwarding allows you to use the keys on your local computer to transparently authenticate the server and simplify the management of credentials on remote hosts.

In order to create a secure connection between a server and a remote system, generally SSH keys, i.e., the Secure Shell keys are used. It is a network protocol that provides a better authentication process as well as encrypted data communications when two systems try to connect over an open network. 

SSH keys use a client-server model to connect a Secure Shell client application with an SSH server. They have public and private key pairs to authenticate the hosts.

SSH Public keys are the username that can be shared with anybody, whereas the private SSH keys are the passwords that are saved by the user locally on the computer and must not be shared with everyone or outside the organization’s network.

Once the ssh keys are generated, to verify and create a secure connection, the SSH requests for a passphrase that helps encrypt and decrypt the private keys. In simple terms, every time you want to use the private key, it sends you a request to enter your passphrase, which can annoy anyone. 

Hence, to manage this process and make it less annoying, SSH uses an agent. The role of the SSH agent is to store the decrypted private key in memory. It means you will not have to enter your passphrase each time until the active user session times out. Once the agent unlocks it, you gain the access to log into your servers securely without entering your passphrase again and again.

However, ssh-agent Forwarding is a deeper version, i.e., it is a tool that authentication agents use to forward authentication information securely to an intermediate service.

Purpose of SSH agent forwarding

The main purpose of agent forwarding is to provide single sign-on services and simplify the management of credentials on remote hosts.

The most common use case for SSH Agent Forwarding is when you want to access a private git repository on a remote server.

Let us say you want the remote server to pull a specific code from Github. Well! In this case, the server generally cross-checks the id_rsa files to get a solution, but sometimes it forwards the query to the local machine.

There are two ways to independently forward the SSH agent on a local machine. One is to try using a real command in a terminal, and the other is to use SSH config to forward SSH Agent.

The local machine, in return, answers the query with a response without including the private key. It does not matter to the Github or main server how you fetched the answer, it just sees the answer and allows the connection.

How to Configure SSH Agent Forwarding

First, you will need to configure an SSH agent forwarding for a specified remote server in your local system.

To do so, create a new configuration file on your local system:

nano /home/vyom/.ssh/config

Add the following lines:

Host remote-server-ip
  ForwardAgent yes

Save the SSH configuration file after you finish editing.

If you don’t want to create a configuration file, then you can use the -A flag to enable the agent forwarding:

ssh -A [email protected]

Next, you will also need to enable the agent forwarding on the remote server machine.

Now, log in to the remote server and edit the SSH configuration file using the nano editor to enable agent forwarding:

nano /etc/ssh/sshd_config

Scroll down the page and change the following line:

AllowAgentForwarding yes

Save the file after editing, then restart the SSH service:

systemctl restart ssh

After setting up agent forwarding, let’s check whether the agent forwarding works or not.

To test it, connect to the remote server from the client machine:

ssh [email protected]

After the successful authentication, you can SSH to other remote servers using your local system’s Public key.

ssh [email protected]

If the connection is successful, that means you are successfully authenticated with the SSH key on your local machine.

Security Risks for Using Agent Forwarding

Like there are advantages to using SSH Agent, there are risks too. If you by any chance leave your terminal unattended, then any person who has physical access to the terminal can appeal as an authority and authenticate with the SSH server.

In simple terms, when you use agent forwarding to jump between SSH servers without copying a private key, there are high chances that any person with root access to the jump server can invoke the agent and misuse authentication.

Thus, it is necessary to follow best practices to tighten the agent usage and reduce the risk of getting compromised.

How can we use SSH Agent safely?

Here are a few ways through which you can minimize the risk of getting SSH agents getting compromised:

    1. Configure the Timeout Parameter

We recommend you use the ssh-add -t (timeout) argument with the ssh-add command to identify with a private key. Every time you run an ssh-add command, you get a request for a passphrase to decrypt the private key. Once provided, it is stored by the SSH agent in the memory throughout the active user session.

This feature, at the same time, also poses a risk. The longer timeout means you are giving enough time to an outsider to get physical access and misuse the SSH keys. However, by configuring the timeout parameter, you can determine the time frame up to which the mechanism will run and collapse. Once the sessions time out, you will need to provide the passphrase again for the next session.

    2. Eliminate All Unused Keys

Unused private keys can expose your data or create unauthorized SSH sessions on other devices. In order to remove the unused keys from the SSH agent, all you need to do is use the ssh-add -D command.

The – D argument supported by the command helps eliminate all the added keys from the client. However, the lowercase version of the argument, i.e., -d allows you to select individual keys.

    3. Make Sure to Always Exist Sessions

To create an SSH connection, the passphrase is needed during the initial SSH sessions. Once provided, you won’t be asked for the same in subsequent sessions. But leaving the SSH agent active with keys open is not a good idea.

The minute you think an SSH agent is not a requirement, use the eval “$(ssh-agent -k)” command to kill the ssh-agent or .logout or.bash_logout files for automatic session closure.

    4. Try ProxyJump rather than Agent Forwarding

First and foremost, be cautious and avoid forwarding SSH agents to random machines not worthy of your trust. Secondly, if you use the ProxyJump feature, you can jump to several servers without agent forwarding. It is a safe option and has been available since OpenSSH version 7.3.