• Santhi A
• June 14, 2023

Troubleshooting SSH : Common issues and how to fix them

Introduction
Secure Shell (SSH) is a network protocol that allows us to secure remote access to servers. A network protocol called Secure Shell (SSH) is used to connect and interact securely with distant servers or computers and creates a secure channel over an unsecured network. Users may securely access and run commands on remote machines thanks to SSH because it is widely used by Linux administrators and IT professionals, which offers a more security over the network. System administrators, developers, and other professionals who need to access remote systems frequently utilise SSH. But when using SSH, users frequently run into problems, from connection errors to failed authentication. In this article, we will talk about typical SSH problems and how to fix them. Despite its reliability and security SSH can encounter a few hitches that can make it challenging to use effectively. In this article we will take an in depth look at the most common issues people face while using SSH and how to troubleshoot them

Common Issues with SSH

1. Connection Refused/Error

Encountering SSH Connection Refused/Error is a frequent issue while utilizing SSH. This message usually appears when the client fails to establish a connection with the SSH server. Multiple factors can cause this error, such as incorrect server address, blocked ports, or firewall settings. To troubleshoot and fix SSH Connection Refused/Error, various measures can be taken. The initial step is to verify the SSH daemon status on the server device. You can execute this by running the subsequent command:

1. For CentOS/RHEL:

          sudo service ssh status          

2. For Ubuntu:

sudo systemctl status ssh

If the SSH service/daemon is not running, you can start it executing the following command:

For CentOS/RHEL:                                                                             sudo service ssh start

For Ubuntu:
              sudo systemctl enable ssh

If the SSH daemon is running, the next step is to make sure that the server is listening on the correct port or not. The default port for SSH service is 22, but this can be changed in the SSH configuration file.

To check the port number, execute the following command:

CentOS/RHEL:
sudo netstat -tulpn | grep sshd
Ubuntu:
sudo netstat -tulpn | grep sshd

Check the server’s firewall rules to allow incoming SSH connections:

CentOS/RHEL:
sudo firewall-cmd --list-services
Ubuntu:
sudo ufw status

Below is example error message with complete path for SSH error logs:

SSH Error: ssh: connect to host example.com port 22: Connection refused

The SSH error logs can be found at /var/log/auth.log on Ubuntu, and /var/log/secure
on RHEL/CentOS.

This command should be displaying the port number on which the SSH service/daemon is listening. If the port number is different from the default port 22, you need to specify the correct port number when connecting to the SSH server.

There is another chance If the SSH daemon is running and listening on the correct port, the next step is to check the firewall settings on the server machine. The firewall might be blocking incoming SSH requests, which can cause the Connection Refused/Error message. To resolve this, you will need to add a rule to the firewall to allow incoming SSH connections on the correct port. When using SSH, it is common to encounter the issue of SSH Connection Refused/Error. To resolve this problem, it is recommended that you verify the status of the SSH service/daemon, the port number, and the firewall settings on the server machine. You can diagnose and resolve SSH Connection Refused/Error and establish a secure connection to your SSH server by following these steps.

2. Authentication Failures :

Another frequent problem while using SSH is failures in SSH authentication. When the user cannot be authenticated by the SSH server, this problem occurs. Numerous factors, such as incorrect login information, expired passwords, improper permissions on the user’s home directory, or other problems with SSH key authentication, can lead to authentication failures.

There are numerous actions you can take to diagnose and fix SSH Authentication Failures. First, make sure you are logging in with the correct credentials. Make sure the password and user name are accurate, and make sure the caps lock key is not turned on. Make that your SSH key is authorized to access the server if you’re using SSH key authentication.

If your attempts at authentication continue to fail, check if your password has expired or if you are using an outdated password. You can reset your password using the passwd command on the server:

sudo passwd

You may need to generate a new key pair and configure the server to use it, if you are using SSH key authentication. You can generate a new key pair using the ssh-keygen command and then use the ssh-copy-id command to copy the public key to the server.

        sudo ssh-keygen -t rsa     
        sudo ssh-copy-id user@server

Verify that your home directory’s permission settings on server are not causing SSH authentication failures, as the ability of a user logging in depends upon setting proper permissions for their corresponding home directory. This command sets permissions to 700:

sudo chmod 700 ~/.ssh

In brief, there are various reasons for the occurrence of SSH authentication failures. Reviewing your login information such as credentials expiry date and making sure that your ssh key authentication is working properly should help resolve this issue. Establishing a secure connection to your SSH server is easy by simply following these steps for diagnosing and resolving any Authentication Failure.
Below example error message with complete path for SSH error logs:
sudo SSH Error: Permission denied (publickey,password)

The SSH error logs can be found at /var/log/auth.log on Ubuntu, and /var/log/secure
on RHEL/CentOS.

3. SSH Freezing or Hanging

SSH can cause problems for its users when it becomes frozen or hangs, as SSH sessions that freeze or
become non-responsive make it impossible to transmit/receive data. There are several reasons why your SSH might freeze or hang as well as issues with your firewall configuration and network connectivity troubles along with potential troubles on the server/client side.

When dealing with SSH freezes or hangs there are various troubleshooting steps that one can perform, nd it is recommended to begin by verifying the network connectivity in order to check for any issues with the connection. Running a ping command is an easy way to test if your connectivity is functioning:

sudo ping <server_ip_address>

If you are unable to ping the server, then there might be network connectivity issues that you need
to address first.

Afterward, you should check for an open SSH port by examining the settings on your firewall and to use SSH, you should check if you have opened up the correct default port which is usually 22 in your firewall. Ensure that you have opened up the correct TCP/IP ports if the SSH protocol uses a different port from its default.

To reboot SSH services on the server you must first confirm that your firewall settings are appropriate, then this can be done by running the systemctl command.

CentOS/RHEL:

sudo service sshd restart
Ubuntu:
sudo systemctl restart sshd

To resolve problems with Hanging and Freezing in an SSH connection, you might need to consider updating the SSH Client and/or Server, so check to see if there are any updates available and update
as needed.

Extending the duration of the server’s TCP keep alive time could be a possible solution to maintain a
constant connection with clients this setting makes sure that keep alive messages are regularly sent.

How to increase KeepAlive time in ssh configuration file?

You can Increase keeps alive time by inserting these directives into your SSH configuration file:

TCPKeepAlive yes
ClientAliveInterval 30
ClientAliveCountMax 99999

There are several factors responsible for freezing or hanging in SSH including misconfiguration of firewalls as well as issues with either the client-side or server-side causing interruptions, so to fix this issue check first whether there are any problems with network connectivity or if there is a Firewall setting blocking access between systems. If that doesn’t work, try updating both SSH Client & Server software and then increase Keep Alive Time on server’s end, so you could diagnose the cause of the issue responsible for SSH freezing or hanging issues.

Below is the example of error message for reference:

SSH Error: packet_write_wait: Connection to <server> port 22: Broken pipe

4. Permission Denied

Connecting with an SSH server can be frustrating if you are seeing the common error message of ‘Permission denied’ but access denied reaching the server happens when a user lacks sufficient permissions. This error can happen due to different reasons ranging from incorrect login information and improper permission structure to server misconfiguration.

To troubleshoot SSH Permission Denied, follow the steps below:

1. Confirm your access details: Ensure that both username and password provided are valid then. In situations of doubt regarding login credentials and permission levels it is best advised to either change the old forgotten passwords or open a brand-new authorized user profile.
2. Audit file permissions: Confirm that the SSH key files are authorized with correct user restrictions. The private key must not accessible by anyone except you whereas public one ought to uploaded on authorized keys files of server.

sudo chmod 600 ~/.ssh/id_rsa
sudo chmod 644 ~/.ssh/id_rsa.pub

3. Inspect Server Configuration File: If there’s still problem regarding SSH Permissions being denied after ensuring all prerequisite configurations, then inspect thoroughly for any error in the configuration files on both ends and make necessary adjustments. Authentication process requires particular attention when configuring system settings including specific details like permitted user list, authorization key path within specified location and mode of access through agreed upon method.

nano /etc/ssh/sshd_config

4. Restart SSH service: Once you have made changes to the SSH configuration file, restart the SSH
   service to apply the changes.
   CentOS/RHEL:
   sudo service sshd restart
   Ubuntu:
   sudo systemctl restart sshd
5. Check firewall settings: Finally, ensure that your firewall settings are configured correctly. The SSH
   port (default 22) should be open and accessible from your local machine.
   CentOS/RHEL:
   firewall-cmd --zone=public --permanent --add-service=ssh Ubuntu

Ubuntu

sudo ufw allow ssh

In conclusion, SSH Permission Denied can occur as a result of invalid credentials, issues with file permissions, or server configuration problems. Verify your username and password, check the server configuration, file permissions, restart the SSH service, and check your firewall settings to resolve this problem. You can diagnose and fix SSH Permission Denied by following these instructions, and you can also create a safe and authorized connection to your SSH server. You may refer below example of error message:

SSH Error: Permission denied, please try again

5. SSH Keys Issues

Secure connections play a crucial role in the SSH protocol with its reliance on utilizing SSH keys, so to prevent prolonged server unavailability for users struggling with SSH key problems it is essential to quickly diagnose and fix these problems. Identify prevalent issues with SSH keys and find out how to fix them through these troubleshooting steps:

Invalid Permissions on SSH Keys:
Connection errors may arise in case of incorrectly configured permission settings of your SSH keys for your SSH server/node. The correct configuration of SSH key permissions is important to check. The solution to this issue involves assigning 600 as the permissions for the private key and 644 for the public key.

sudo chmod 600 ~/.ssh/id_rsa
sudo chmod 644 ~/.ssh/id_rsa.pub

Mismatched SSH Keys:
A declined connection occurs when there is no matching between the client’s private key and server’s public keys, so it is important that there is a correspondence between the public and private keys of the server and client. With ssh-keygen command generating a new key pair is possible and subsequently uploading the public key to server is recommended.

Incorrect SSH Agent Configuration:

If your SSH agent is not configured correctly, you might encounter issues when trying to connect to your server. To fix this issue, make sure that the SSH agent is running, and that your SSH keys are added to the agent. Use the following commands:

sudo eval `ssh-agent -s
sudo ssh-add ~/.ssh/id_rsa

Example error message with complete path for SSH error logs:

SSH Error: Permission denied (publickey)

Disabled SSH Keys:
If your server is configured to disable SSH keys, you may not be able to connect using your SSH keys.
To fix this issue, you need to enable SSH key authentication on the server. Open the /etc/ssh/sshd_config file and ensure that the following lines are not commented out:

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

Revoked SSH Keys:
You won’t be able to connect to the server using your SSH key if it has been revoked. You must create a new SSH key pair and upload the public key to the server to resolve this problem. Encountering errors related to revoked SSH keys can disrupt your SSH connections and prevent you from accessing remote servers. Revocation typically occurs when SSH keys are compromised, lost, or no longer authorized for use. This section will guide you through diagnosing and resolving the issue of revoked SSH keys.

How to diagnose Revoked SSH key Issue?

If you are experiencing authentication failures due to revoked SSH keys, follow these steps to diagnose the problem:

1. Check the SSH error logs:
   RHEL/CentOS:

sudo vim /var/log/secure

Ubuntu:
sudo vim /var/log/auth.log

2. Look for error messages indicating revoked keys, such as:

• “Server refused our key”
• “Permission denied (publickey)”

If you find such error messages, it indicates that the SSH keys you are using have been revoked.

How to fix revoked SSH key issue?
To resolve the revoked SSH keys issue, follow these steps:

1. Generate a new SSH key pair:
   RHEL/CentOS/Ubuntu:
   sudo ssh-keygen -t rsa -b 4096
2. Store the new SSH key pair in the appropriate location:
   RHEL/CentOS/Ubuntu:
   • The new SSH key pair will be saved in the ~/.ssh/ directory by default. You
     can specify a different location if desired.
3. Update the authorized keys file on the remote server:
   RHEL/CentOS/Ubuntu:
   ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote_server
   This command will copy the new public key to the authorized keys file on the remote server.
4. Test the new SSH key pair by attempting to connect to the remote server:
   RHEL/CentOS/Ubuntu:

ssh user@remote_server
If the connection is successful, it means the revoked SSH keys have been replaced with the new keys.
By generating and using a new SSH key pair, you can regain access to the remote server and overcome the revoked SSH keys error.

In conclusion, SSH key issues can prevent users from accessing their servers. Common SSH key issues include keys with invalid permissions, keys that don’t match, keys that are disabled or revoked, and keys that are configured incorrectly for SSH agents. Set the proper permissions, create a new key pair, confirm that the SSH agent is running, enable SSH key authentication, and upload the public key to the server to resolve these problems.

Best Practices to Avoid Common SSH Issues

To avoid common SSH issues, there are some best practices that you can follow:

1. Keep software updated
   SSH client and server software should be kept up to date to avoid security vulnerabilities, compatibility issues and performance issues. Regular updates provide bug fixes, security patches and new features that improve stability.
2. Limit SSH access to trusted hosts
   Limiting SSH access to trusted hosts can reduce the risk of security breaches. You can configure your SSH server to allow connections only from specific IP addresses or networks using firewall rules or SSH configuration settings.
3. Use SSH keys for authentication
   SSH keys are more secure than passwords, as they use a public-private key pair. The private key is stored on the client machine, and the public key is stored on the server. When connecting to the server, the server authenticates you by checking whether your public key matches its own private key, which was generated from your password or username.
4. Use strong passwords
   When using passwords for authentication, use strong (complex) passwords that are difficult to guess or crack. Avoid common passwords or dictionary words, and use a combination of letters, numbers, and symbols.
5. Disable SSH root login
   In many cases, SSH is configured to allow root login by default. This can be a security risk because attackers may use brute force attacks or other methods to guess your root password and gain full access to the system. It is best to disable root login and create a separate user account that has limited privileges.
6. Enable two-factor authentication
   Two-factor authentication can provide an extra layer of security to your SSH connections by requiring users to provide two types of authentications, such as a password and a token. This can prevent unauthorized access even if the password is compromised.
7. Monitor SSH logs
   Monitoring SSH logs can help detect and prevent security breaches. They contain information about login attempts, successful and failed connections, and other events. Log analysis tools can be used to monitor SSH logs and alert you of any suspicious activity.

Conclusion
SSH is an important tool for remote system management, but it can be challenging to figure out how to use it. To ensure secure and reliable connections to your systems, it is important to follow best practices and use updated security measures. Maintaining a secure and stable SSH environment can require various precautions, such as updating software, limiting access via SSH, and monitoring logs.

Author : Vijay Kakade