Reverse Tunnels

This page describes how I set up a reverse tunnel so that I can access my private GNU/linux(ubuntu) computer from anywhere on the Internet. I document it here to be helpful to me, but you might also find it helpful.

There are three computers involved, here named: mercury, the private computer I want to remotely access; hbelabs.com, a public server (possibly rented); and moon, the computer from which I want to access the first one, mercury.

To set this up, you should [initially] have physical access (in the same room) to mercury. Of course, moon is your mobile computer (or the computer at the library or school) that you have with you; while hbelabs is in some data center, and we may never get to see that one.

Both moon and mercury must be able to ssh into hbelabs, and we should have sudo privilege on mercury and hbelabs).

The strategy begins by creating a user called "autossh" on hbelabs and mercury, and giving mercury passwordless access to hbelabs. The purpose of this user account is to establish the reverse tunnel: It is started on mercury when the computer mercury turns on (and its network is up and running), and this tunnel connects to hbelabs. Then we can from moon use that reverse tunnel to access mercury through hbelabs.

The program "autossh" is used since it is able to maintain the ssh connection to hbelabs, even if there are network changes in the mercury-hbelabs connection or if mercury or hbelabs goes through a reboot, for example. (The username "autosssh" and the program "autossh" are two different things.)

Create User autossh

On both the hbelabs and mercury, as your normal (non-root) user with sudo access:

The first command creates the user and the second and third commands show how to log in and out as that user. Note that user autossh does not actually have a password and the normal login is disabled, but the account can be accessed as shown using the "sudo su" command but we have to specify the shell.

Details: The "-m" says, create the home directory structure, "/home/autossh/", and the "-u 499" gives him a user ID that might not show up on the usual ubuntu login screen, but also doesn't interfere with the usual numbering of ordinary user IDs, which start near 1000. The "-s /usr/sbin/nologin" disables normal logins (from the GUI, for example), but (as the second command shows) a normal user can log into it.

Passwordless ssh Access

On mercury, login to the autossh account and enter the command

The defaults might be OK (which means responding with a <Enter>). Do not enter a password. You must also never share the file "/home/autossh/.ssh/id_rsa" with anyone.

The public key file can be shared. In fact, you want to put it on the public server (here, hbelabs). But, since we can't use the command "ssh-copy-id", since we have disabled logins, we must use some other way.

This is what I do. On mercury, I login as autossh and cat the file (/home/autossh/.ssh/id_rsa.pub) and copy its contents to your clipboard buffer (use the mouse to select the text, right-click, choose "copy").

That entire second row (which will wrap on your terminal), is what gets copied. Of course, you will use your own public key.

Next, ssh into your public server (hbelabs) and log in to the autossh account. As autossh, edit the file "/home/autossh/.ssh/authorized_keys" and paste that public key into it. That key occupies only one line in the file, but will wrap making it look like several lines.

If you don't do this next step, it may not work (since there may be a prompt with no-one to respond). Log off the public server (hbelabs) and from the autossh/mercury account, attempt to log into the public server, hbelabs as autossh (e.g., $ ssh autossh@hbelabs.com, and $ ssh autossh@45.79.107.26). It will ask you to confirm the host for known_hosts. Accept. Try logging in using both the FQDN and the IP address ... just to be sure those entries are in the known_hosts file. Doing that will display some stuff about the server (indicating it did login), but it will bump you right back to your private computer. That is OK.

Security Caution: If you use a rented public server, never reverse this process: Never create a passwordless access from the public server (hbelabs) to your private computer (mercury). The level of trust on the public server is assumed to be zero. An administrator who works there, or someone with a court order, or some Russian agent, can illegally but very easily access your PC if you do. Assume ALL information on the public server is not private.

Create the reverse tunnel

When you are able to log into the server from the client autossh account without any prompts, we are ready to create the simple script file and put it in a place that ubuntu runs when it is booting up.

My goal is to be able to sit at some computer on the internet and type

$ ssh -p20202 bob@hbelabs.com

to access my private client computer, mercury, even if mercury is sitting at the lightdm login prompt halfway around the world.

The port number "20202" is how I uniquely identify mercury from the outside. That is a valid high (5-digits) port number, which I choose to be easy to remember, and which isn't used by something else.

Reverse tunnel - starting by hand: First, make sure we can set up the reverse tunnel by hand. While at mercury, as your normal user, enter this

$ sudo su -s /bin/sh autossh -c "/usr/bin/autossh -N -R :20202:localhost:22 autossh@hbelabs.com"

... on one line. (The program autossh might not be installed. To install, on mercury, I use $ sudo apt install autossh.)

One thing I use to check to see it worked is to log into the server (hbelabs, as normal user) and check using

$ netstat -an |grep -i listen|grep 20202

or variations thereof. If nothing displays, then there is no reverse tunnel.

Timeouts: You may get autossh to work right, at first, but it might stop working if idle for even a few minutes. To fix that timeout problem, you have to edit the file "/etc/ssh/sshd_config" on the computer you want to remotely access. I actually change both mercury and hbelabs this way. Make sure these features (TCPKeepAlive, ClientAliveInterval, ClientAliveCountMax) are uncommented (not prefixed with #) and set accordingly. There are other settings in that file, don't touch anything else. I'm not sure if placement in the file is important.

I'm sure there is a way to restart sshd without rebooting, but a reboot will ensure the settings are enabled on that next reboot --- but make sure you don't make any typos on the remote: It might not start the sshd properly and you might lose the ability to log in.

Firewall (iptables, ufw): In order to actually make it work, I have to open up the port 20202 on the public server, so, while still logged into the server, I simply use (Uncomplicated FireWall),

$ sudo ufw allow 20202
$ sudo ufw status
$ sudo ufw enable

the third command of which makes sure it is actually enabled.

Startup script: Once I figure out and solve all the reasons it might not be working, it is time to set it up to start automatically before the lightdm login prompt, but after we have some network configuration.

As the normal sudoer user on mercury, create the file "/etc/network/if-up.d/reversetunnel",

Note that that is the same as the manual entry above. We don't need the "-f" switch on autossh, for example. (The second command is me trying to set up the ssvnc server.)

Make sure the file is executable and owned by root with

$ sudo chmod a+x reversetunnel
$ sudo chown root:root reversetunnel
$ ls -al reversetunnel
-rwxr-xr-x 1 root root 212 June 27 16:26 reversetunnel

Pudding: The proof of the pudding is in the tasting. Reboot all computers (and without using the GUI to log into mercury, leave it at the login prompt), and try to access from a normal user on moon to bob on mercury using

$ ssh -p20202 bob@hbelabs.com

This can work even if moon is Windows computer (using putty, something useful to put on Windows) or an android phone (using JuiceSSH, for example). [Since Apple's OS is based on FreeBSD, it already should have a terminal program with ssh installed, but iOS is more challenging since you may have to jailbreak it to put this useful stuff on it.]

You may also want to wait a few hours "idle" and try it again to see how stable it is.

Other Conerns: Sometimes (maybe if you have several network cards installed and configured) this does not execute at the right time nor does it actually execute, which can be frustrating. Rather than fight with configuring $IFACE conditionals in the script or the NetworkManager configuration challenges, if you configure the interface with "/etc/network/interfaces", you might be able to put the script in "/etc/networking/" (so ubuntu won't get too confused and you have a chance of finding the script later) and you can add a tab-prefixed line calling the script (as shown in this "interfaces" fragment):

In the script ("reversetunnel"), you can also see the second command that sets up a reverse tunnel to the vnc server. For the VNC tunnel, neither port 15926 (on the server hbelabs.com) nor 5900 (on the client mercury) need be opened up for public access. The VNC will go through ssh port 22 (on the server), which must be open.

I reboot the client mercury, and the server, a few times just to be sure it works automatically.

Now, if I managed to set up x11vnc properly, I can sit at that computer somewhere out on the internet and also type,

$ ssvncviewer -via hbelabs.com localhost:15926

and view the wonderful lightdm login prompt ... and login to its gui and try to break some things.

NOTE: The sole purpose of autossh (the user account) is to use passwordless login from mercury to the public server so it can establish the reverse tunnel. I use my ordinary account to actually use ssh or vnc. autossh, the program, tries to keep the ssh connections alive.

Updated August 2017. rsjones7 at yahoo dot com. hbelabs.com