setup a ssh bastion host

ssh-jump-host
ssh bastion host concept

Introduction     

In recent times, there is an increasing need for organizations to give employees access to their IT facilities due to the ongoing Covid restrictions ( such as work from home )  in place and in other cases grant access to external parties like clients, vendors  who wants to troubleshoot and fix issues with the IT Infrastructure remotely.
 
More so, is the need for multiple manage SSH access to the company’s Linux servers, Routers, Switches, while meeting regulatory and security compliance.
 
This need led to the emergence of the SSH Bastion host concept.
It is a secure intermediary server where all your system administrators would login in first via SSH before getting to access the remote devices such as Linux instance, Routers, Switches etc. The purpose of having the SSH bastion host  is to improve security and consolidate SSH user activities to a single point hence better security and accountability. SSH  Jump bastion host is also known by the name SSH Jump Box, SSH Jump Host & SSH Gateway.

SSH Bastion Host explained.                                 

An SSH Bastion host is simply a single, hardened server that you “jump” through in order to access other servers or devices on the inner network.
Sometimes called a SSH Jump host , or SSH Jump server or  ssh gateway or a relay host, it’s simply a server that all of your users can log into and use as a relay server to connect to other Linux servers, Routers, Switches and more. Therefore, a jump server is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones.
 
In other words, it is an intermediary host or an SSH gateway to a remote network, through which a connection can be made to another host in a dissimilar security zone, for example a demilitarized zone (DMZ2). In short it is intended to breach the gap between two security zones. This is done with the purpose of establishing a gateway to access something inside of the security zone, from the DMZ
 
The SSH Bastion host bridges two dissimilar security zones and offers controlled and monitored access between them.
 
For users accessing your secure network over the internet, the Bastion host provides a highly secured and monitored environment especially when it spans a private network and a DMZ with servers providing services to users on the internet.
 
Furthermore, a classic scenario is connecting from your desktop or laptop from inside your company’s internal network, which is highly secured with firewalls to a DMZ. In order to easily manage a server in a DMZ, you may access it via a bastion host.
 
Therefore, a bastion host is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones. An example would be a high security zone inside a corporation. The policy guide states that this zone cannot be accessed directly from a normal user zone. Hence, in a DMZ off the firewall protecting this zone you have a jump host.
 
Connections are permitted to the ssh bastion host from the user zone, and access to the secure zone are permitted from the bastion host.
 
More often, there is a separate authentication method for the bastion host fortified with multi factor authentication, Single Sign On ( SSO ) , Radius  & more. 

Configure a SSH Bastion Host Solution
  • Using OpenSSH
    A basic ssh bastion host server with limited features and functionalities  can be configured using OpenSSH packages  that available by default on most Linux distributions. In the example below, we will just use the basic ssh command line to proxy a ssh connection to the remote server via a intermediate jump server.
ssh -J jump_machine remote_machine

If the -J option is not available use the -W option to pivot the connection through an intermediate bastion host.

ssh -o ProxyCommand="ssh -W %h:%p bastion.gateway.org"  remote.server.org

 

With the OpenSSH 7.3,  the easiest way to pass through hop through intermediate one or more jump hosts is  using the ProxyJump directive ssh_config

 

Host remote server
HostName 192.168.0.177
ProxyJump admin@jump-server.org:22
User devops

 

Multiple bastion hosts can be chained as well

 

Host remote server
HostName 192.168.0.177
ProxyJump admin@jump-server.org:22, admin@jump-server2.org:22
User devops

 

Do refer the article SSH Proxy and SSH Bastion Host  for configuring  a basic bastion host server that is very limited in feature and functionality when compared to the modern day ssh  jump host solutions.

  • Using Ezeelogin
     Ezeelogin is a much more powerful and advanced SSH bastion host software solution  and  can be deployed quickly on a Linux server.  It has powerful features that  makes managing hundreds of Linux devices and granting ssh access to these device a piece of cake.  Do refer the article to  setup and configure a ssh bastion host  quickly on your premise or on cloud.                                                    

Why do you need a SSH Bastion Host solution to manage ssh access? 

The OpenSSH based bastion host server is clearly not enough to meet the modern day requirements  of an IT enterprise. The challenges for  the enterprise are constantly changing and dynamic . On day , it could be from maintaining security, granting ssh access to the users to designated server and that too for particular time and on another day it could be the security compliances that needs to be met at the time of a Linux servers infrastructure audit.
The modern day  SSH  bastion host solutions are designed to address the challenges faced by an IT enterprise when it comes to  security and to meet various security compliances like PCI DSS, NIST, ISO 27001 and more.
 
The modern day ssh bastion host  software such as Ezeelogin has the  following features  and more.
  • Identity and Access management (IAM)
  • Privileged Access management (PAM),
  • Role Based Access Control to delegate access to Linux servers and Network devices.
  • Two factor authentication methods  like Google Authenticator, DUO Security 2FA, & Yubikey in SSH.
  • Integrates with Windows Active Directory, OpenLDAP, Redhat IDM.
  • Supports SAML for  Single Sign On.
  • Support RADIUS Authentication to access network devices such as Routers and Switches
  • Password Manager
  • SSH key rotation, 
  • Automated root password management

CONCLUSION

IT Enterprises that use a SSH Bastion host solution in improving security of their critical IT asset and in meeting various mandatory security compliances  (which would otherwise prove very costly in case of a breach),  are more likely to succeed due to the improved operational efficiency, digital security, hence more successful business for the company’s end customers.

Setup and Configure a SSH Jump Server

ssh-jump-host
ssh-jump-server-concept

Introduction     

In recent times, there is an increasing need for organizations to give employees access to their IT facilities due to the ongoing Covid restrictions ( such as work from home )  in place and in other cases grant access to external parties like clients, vendors  who wants to troubleshoot and fix issues with the IT Infrastructure remotely.
 
More so, is the need for multiple manage SSH access to the company’s Linux servers, Routers, Switches, while meeting regulatory and security compliance.
 
This need led to the emergence of the SSH Jump Server concept.
It is a secure intermediary server where all your system administrators would login in first via SSH before getting to access the remote devices such as Linux instance, Routers, Switches etc. The purpose of having the SSH Jump server is to improve security and consolidate SSH user activities to a single point hence better security and accountability. SSH  Jump server is also known by the name SSH Jump Box, SSH Jump Host & SSH Gateway.

SSH Jump Server explained.                                 

An SSH Jump Server is simply a single, hardened server that you “jump” through in order to access other servers or devices on the inner network.
Sometimes called a SSH Jump host , or SSH Jump server or  bastion or a relay host, it’s simply a server that all of your users can log into and use as a relay server to connect to other Linux servers, Routers, Switches and more. Therefore, a jump server is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones.
 
In other words, it is an intermediary host or an SSH gateway to a remote network, through which a connection can be made to another host in a dissimilar security zone, for example a demilitarized zone (DMZ2). In short it is intended to breach the gap between two security zones. This is done with the purpose of establishing a gateway to access something inside of the security zone, from the DMZ
 
The SSH Jump Box bridges two dissimilar security zones and offers controlled and monitored access between them.
 
For users accessing your secure network over the internet, the jump host provides a highly secured and monitored environment especially when it spans a private network and a DMZ with servers providing services to users on the internet.
 
Furthermore, a classic scenario is connecting from your desktop or laptop from inside your company’s internal network, which is highly secured with firewalls to a DMZ. In order to easily manage a server in a DMZ, you may access it via a jump host.
 
Therefore, a jump host is a server inside a secure zone, which can be accessed from a less secure zone. It is then possible to jump from this host to greater security zones. An example would be a high security zone inside a corporation. The policy guide states that this zone cannot be accessed directly from a normal user zone. Hence, in a DMZ off the firewall protecting this zone you have a jump host.
 
Connections are permitted to the ssh jump host from the user zone, and access to the secure zone are permitted from the jump host.
 
More often, there is a separate authentication method for the jump host fortified with multi factor authentication, Single Sign On ( SSO ) , Radius  & more. 

Configure a SSH Jump Server Solution
  • Using OpenSSH
    A basic ssh jump server with limited features and functionalities  can be configured using OpenSSH packages  that available by default on most Linux distributions. In the example below, we will just use the basic ssh command line to proxy a ssh connection to the remote server via a intermediate jump server.
ssh -J jump_machine remote_machine

If the -J option is not available use the -W option to pivot the connection through an intermediate bastion host.

ssh -o ProxyCommand="ssh -W %h:%p bastion.gateway.org"  remote.server.org

With the OpenSSH 7.3,  the easiest way to pass through hop through intermediate one or more jump hosts is  using the ProxyJump directive ssh_config

Host remote server
HostName 192.168.0.177
ProxyJump admin@jump-server.org:22
User devops

Multiple jump hosts can be chained as well

Host remote server
HostName 192.168.0.177
ProxyJump admin@jump-server.org:22, admin@jump-server2.org:22
User devops

 

Do refer the article SSH Proxy and SSH JumpHost  for configuring  a basic jump server that is very limited in feature and functionality when compared to the modern day ssh  jump host solutions.

  • Using Ezeelogin
     Ezeelogin is a much more powerful and advanced SSH Jump host software solution and  can be deployed quickly.  It has powerful features that  makes managing hundreds of Linux devices and granting ssh access to these device a piece of cake.  Do refer the article to  setup and configure a ssh jump server quickly on your premise or on cloud.                                                    

Why do you need a SSH Jump server solution to manage ssh access? 

The OpenSSH based jump server is clearly not enough to meet the modern day requirements  of an IT enterprise. The challenges for  the enterprise are constantly changing and dynamic . On day , it could be from maintaining security, granting ssh access to the users to designated server and that too for particular time and on another day it could be the security compliances that needs to be met at the time of a Linux servers infrastructure audit.
The modern day  SSH Jump host solutions are designed to address the challenges faced by an IT enterprise when it comes to  security and to meet various security compliances like PCI DSS, NIST, ISO 27001 and more.
 
The modern day ssh jump server  software has the  following features  and more.
  • Identity and Access management (IAM)
  • Privileged Access management (PAM),
  • Role Based Access Control to delegate access to Linux servers and Network devices.
  • Two factor authentication methods  like Google Authenticator, DUO Security 2FA, & Yubikey in SSH.
  • Integrates with Windows Active Directory, OpenLDAP, Redhat IDM.
  • Supports SAML for  Single Sign On.
  • Support RADIUS Authentication to access network devices such as Routers and Switches
  • Password Manager
  • SSH key rotation, 
  • Automated root password management

CONCLUSION

IT Enterprises that use a SSH Jump Server solution in improving security of their critical IT asset and in meeting various mandatory security compliances  (which would otherwise prove very costly in case of a breach),  are more likely to succeed due to the improved operational efficiency, digital security, hence more successful business for the company’s end customers.

References

SSH Proxy and SSH JumpHost

SSH Key Management

 

ssh key management
ssh key management

SSH key based authentication  is the default method that any Linux admin would choose for  granting ssh access to  Linux servers or Linux cloud instances. SSH key based  authentication is the preferred method since its far more secure and more popular than password based authentication. The two commonly used   keys for authentication are RSA and DSA and RSA Keys are preferred since DSA is known to be vulnerable.

Its good to have SSH Key based authentication but imagine the amount of management that has to be done if you have 100 employees and have 1000 Linux instances and how do you  grant access to the employees to the thousand Linux servers or Linux cloud instances? How do you add the ssh keys to grant access ?  How do you remove the ssh keys when the employee leaves the company? How do you rotate the keys?

You would have to manually add the keys to all the Linux servers and cloud  instances to grant access and when an employee leaves, you would have to ensure that the employees public keys are removed from all your servers failing which it becomes a serious security issue.

Ezeelogin helps you address the following issues

  1. SSH  Key rotation
  2. Centralised ssh key management
  3. Helps to reduce the overhead that comes with managing the ssh keys to almost zero.

 

Parallel Shell

Parallel shell – Run commands on multiple Linux servers or Cloud instances   simultaneously

If you are in charge of large server farms, cluster of Linux nodes for high performance computing, or cryptocurrency mining farms then parallel shell would easily let you manage multiple Linux servers or Cloud instances easily and quickly.

Parallel shell is built into the backend shell of the Ezeelogin SSH Jump host. You can work with it as you would work on a normal bash shell and the command would be simultaneously executed on multiple Linux servers.

Imagine, that you are the security engineer in charge for fleet of linux server or aws instances. One fine morning as you are going through your daily job routine you are notified of a critical kernel vulnerability.  As a responsible security officer, you do not want to postpone patching the kernels for the next day  as the longer the delay to patch the Kernel, greater the possibility of a security breach.

In such critical scenarios, the parallel shell feature could be extremely useful as you can compile kernel  on one thousand machine at the time of compiling a kernel for one server.

This feature is a god send for many , however with great powers come great responsibilities.  The image shows a command being executed in using parallel shell.

parallel shell
parallel shell- Multiple Linux server management

Some of the benefits of this feature are

  1. Improve productivity of your system administrators and devops engineers
  2. Improve the efficiency with which Linux nodes or cloud or aws instances are managed.
  3. Easily execute command simultaneously on group of servers. There are no hard configurations to be done.
  4. Better and faster management of Linux server and cloud instances.
  5. Easily copy files across group of servers
  6. Server orchestration would be very easy if you have many Linux instances.
  7. There is no need to install agents on remote machines
  8. Faster setup
  9. Delivery faster services
  10. Runs on OpenSSH

SSH Two Factor Authentication

ssh jumpbox with duo two factor authentication
SSH gateway and Jumphost with DUO 2FA o

 

SSH JumpHost and SSH Gateway Ezeelogin supports DUO Security two factor authentication ( 2FA ) which means that anyone having a smartphone these days can easily use it for the second layer of authentication. With DUO, you dont have to type in complex strings or numbers, just tap on the smartphone screen and you are securely authenticated easily. No extra devices like RSA Keys or security token generating devices has to be carried since you already have a smartphone with you to authenticate into your SSH Gateway.

OpenSSH 7.0 disables DSA keys by default


The road ahead was never bright for DSA keys and the writing was clear
on the wall. Ezeelogin SSH gateway will be dropping DSA keys and would 
be using ONLY RSA keys in future releases. 
Starting with the 7.0 release of OpenSSH, support for ssh-dss keys has
been disabled by  default at runtime due to their inherit weakness.If
you rely on these key types,you will have to take corrective action or 
risk being locked out. Your best option is to generate new keys 
using strong algos such as rsa or ecdsa or ed25519.RSA keys will give
you the greatest portability with other clients/servers while ed25519
will get you the best security with OpenSSH.(but requires recent versions of
client & server).

If you are stuck with DSA keys, you can re-enable support locally by updating
 your sshd_config and ~/.ssh/config files with lines like so: 
     PubkeyAcceptedKeyTypes=+ssh-dss

Be aware though that eventually OpenSSH will drop support for DSA keys entirely, 
so this is only a stop gap solution.

More details can be found on 
OpenSSH's website: http://www.openssh.com/legacy.html

Automated root password management on Linux servers

Automatic root password management
Automatic root password management

 

Boss wants you to enable password based authentication on hundred  Linux server, he wants you to  set 30 plus character strong password on each server, share the root passwords with  developers ,  change the root passwords again once the developers logs out of the servers at the end of the day, also your boss want you to reset the root password on all the Linux server on a daily basis  as he is paranoid  when it comes to security.

Well without eating your boss alive and instead to get a promotion, here is the magic wand, use the Ezeelogin root password management feature and you will  be able to meet all his requirement and if not even better. Being a Linux system administrator you know for fact that Key based authentication are exponentially stronger even if your passwords are 100 characters long but for some unearthly reasons you need to have password based authentication enabled on your hundred Linux servers.

 

jump server password view
jump server password view

Here are the key issues that Ezeelogin root password management features addresses.

  • Automatically set and reset and strong root passwords up to 32 characters long in a click on hundreds or thousands of Linux servers
  • Schedule periodic reset of root password across all your linux servers in a click
  • Reset root passwords on all your Linux server in a click.

 

 

 

 

Record SSH Session

record ssh sessions
Record SSH Session

Record ssh sessions on Linux server, Amazon EC2  instances for Security Compliance

It is mandatory to record ssh sessions to be PCI DSS Complaint when system administrator, system engineers or devops engineers login via ssh into your Linux machines or cloud instances, . It is also a must to maintain a log of all ssh activities on your Linux servers. This is tedious task for any Linux system administrator as deploying a improvised solution has often turned to be useless in the hour of need.

Why you should record ssh session of your staff /employees on a bastion host ?

Imagine that you have 100 Linux servers or cloud instances. You have 100  users  having access to these server via SSH with some system admins having privileged access or root access. If a user deletes a critical files or a database record which leads to a serious downtime and million of dollar lost in revenue,  following are some of the questions that you would have to answer as the security officer of your Linux infrastructure

  • Which user did it ? How will you find out who is responsible when you have so many of your employees accessing your servers?
  • How did it happen? When did it happen?
  • What is the extent of damage?
  • How will you prevent this in the future?
  • Is it possible to track  ssh server activities of employees?
  • Is it possible to do a forensic investigation when somebody does something bad like opening a backdoor?
  • How to monitor a staff in ssh in real time?
  • Has any Linux  server been breached?
  • How  to ensure that your employees password typed in on STDIN are not recorded as required by security compliances ?
    Enabling the SSH Session recording feature in the Ezeelogin SSH jump server would help you achieve this very quickly and easily so you have a complete record of what was done on your server at any point of time by which jump server user. This is very useful for forensic ssh log audits or for maintaining an audit trail for pci compliance.
    The solution records every ssh activity.  It includes the user input and the output with timestamps.  There is provision to search through the recorded ssh logs as well.

Bastion host – How to secure and harden the ssh server on it?

  1. Enable a firewall and by default block all  IP access to the SSH Port and enable only your staff ips or dynamic ip ranges that you trust.
  2. Disable direct root login. Its always better to login as a non privileged user first and the switch to the root user. This is the norm if you are looking for PCI DSS Compliance.Edit /etc/sshd/sshd_config
    PermitRootLogin noEzeelogin SSH Gateway has a feature called ‘AUTO SU or SUDO’ which would automatically does the switching part  so you would not waste your time retrieving password of the ‘admin’ user and then entering the root password.
  3. Disable password based authentication and enable only Key based authentication in the  sshd configuration file. I would rate this as the most important of all.

    PasswordAuthentication no

  4. Enable Key based authentication. RSA is know to be more secure than DSA keys.

    RSAAuthentication yes

    PubkeyAuthentication yes

    allowtcpforwarding no

  5. Change the sshd default listening port from 22 to something like 22656 since its hard to guess and attackers would have to scan.Use custom SSH Port and Listening IPs.
    Port 22656
    ListenAddress 192.168.5.6.123
  6. Configure a VPN and having your server behind a VPN is good idea. This would really improve the security and harden the server.