Bastion Hopping with SSH and ScaleFT

January 24, 2018

by Robert Chiniquy


Working at ScaleFT means constantly seeing new ways that people use SSH. When I talk to users, or prospective users, I get to hear all about how they secure their environments and how they structure network access into and between their servers. SSH bastions are frequently a part of this picture.

A bastion can be any host, typically a Linux host, which is available to inbound connections from the public internet, and can initiate outbound network connections into a private network. By existing in a DMZ, bastions enable the target hosts behind them to reside in private networks, which prohibit inbound network access except for that originating from the bastion or DMZ. Most users use bastions with SSH port-forwarding to enable transparent, authenticated access to the hosts in the private network.

These commonalities aside, bastions are rarely set up or used in the same way. Every organization has a set of different production architectures because they think about things differently. Different teams of engineers with different needs, different tools, and different mindsets, building and maintaining their systems under different constraints, and through years of changing conditions, naturally end up producing very different results.

In this wide variety of outcomes, you frequently find unfortunate consequences for bastion architecture, leading to poor security posture and user experiences, such as:

  • Private keys kept on the bastion
  • Shared users used on the bastion
  • Shared deployment (or other) tools on the bastion
  • ssh-agent enabled for bastion hopping
  • Users being forced through SSO/MFA hoops multiple times, once per SSH hop
  • Lateral movement becoming habitual for authorized users in the environment to avoid round trips through the bastion
  • Long-lived screen sessions or tunnels are left in place to avoid bastion hops
  • A bastion being configured to MITM all SSH traffic for auditing
  • A complex, hand-maintained, per-region SSH client configuration being required for bastion hops, which vary across the team and makes onboarding new teammates awkward

One of our values at ScaleFT is to do our best to support our users where they are, with the decisions and tools they’ve already selected. This means treating SSH bastions as an SSH feature, parameterizing and centralizing the associated configurations, and seamlessly integrating it into our users’ daily workflows. We’ve taken what we think of as the best practices and behaviors, and made them work transparently with SSH on the client and the server side.

If any of the problems listed above sounds familiar to you, you might be surprised how much ScaleFT’s bastion support could improve your team’s experience working with your production environment.

Bastion hopping in action

Setup

The commands in this post are being run against an environment created by this terraform module.

If you have an AWS account, feel free to try this out yourself. A moderate amount of setup may be required and is documented in the README. If you’ve already tried out ScaleFT, you’ve probably already done most of this setup.

Among the recommended setup steps is configuring ScaleFT to work with your SSH client using OpenSSH ProxyCommand. ProxyCommand is nice, not only because it enables seamless use of ScaleFT with SSH, but because it also enables using ScaleFT with all the tools that use SSH as a transport, such as scp, git, or even Ansible. For this reason, ScaleFT’s bastion features are most convenient and impressive when combined with ProxyCommand.

This terraform module will spin up two servers, ubuntu-bastion and ubuntu-target. The ubuntu-target server will be automatically configured to use the ubuntu-bastion server as an SSH bastion. This configuration is done in a userdata script which you can find on Github.

In ScaleFT, bastions are configured as attributes of target nodes. This means that users are able to just directly use tools referring only to the names of target nodes, and ScaleFT can work backwards from the target, through each individual bastion hop, and return a complete connection chain to your SSH client.

Users don’t even need to know a bastion is there. This also means you can easily change a bastion, or add new bastion hops, and your users will likely never even notice.

Using your usual SSH client

If you have configured ProxyCommand, you will now be able do a bastion hop with just one command: ssh ubuntu-target

demo > ssh ubuntu-target Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-1020-aws x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud 88 packages can be updated. 0 updates are security updates. *** System restart required *** Last login: Mon Jan 22 16:31:13 2018 from 172.31.6.7 alice@ip-172-31-25-10:~$

Otherwise, for now you can use the command sft ssh ubuntu-target. This will use the same methods to open an interactive SSH session with the remote machine, and also does provide features like port forwarding, but in the long run you’re going to want to configure ProxyCommand so you can use everything else SSH has to offer.

Once your session is open on the target server, check out the output of last | head. You’ll see that your incoming SSH connection came from a private IP address in the same AWS VPC as your target host. It did not originate from whatever home or office wifi you are trying this out from.

To see the difference, you can directly ssh ubuntu-bastion and check last, and then you will see both your prior bastion hop, and your current session, with the full connection flow chain made clear.

The original ssh command you issued took you through a bastion hop, having done a few ScaleFT API interactions in the background to request a few ephemeral certificates, and I bet you didn’t notice any differences between that and a direct SSH connection.

You can also try using scp to transfer some files to ubuntu-target; that will use the bastion and ScaleFT credentials automatically as well.

The ProxyCommand configuration

The ProxyCommand configuration that you’re using looks something like this:

$ cat ~/.ssh/config Match exec "/usr/local/bin/sft resolve -q %h" ProxyCommand "/usr/local/bin/sft" proxycommand %h UserKnownHostsFile "/Users/alice/Library/Application Support/ScaleFT/proxycommand_known_hosts"

Let’s step through these lines and see what each one does.

Match exec

When you run $ ssh ubuntu-target, your SSH client encounters the “Match exec” directive in your SSH config file. This is a conditional block. The directive causes your SSH client to briefly fork the command in quotes as a child process, substituting in “ubuntu-target” for the %h, to determine if the entire configuration block applies to this ssh command.

This allows our local sft binary, which is installed by the same installer that installs our GUI application, to issue a metadata query to the server inventory maintained in the ScaleFT Platform to see if there is a server known by that name which you have been granted access to. If the server name resolves, your SSH client knows to apply the remaining configurations in the block to your SSH command. You can read more about resolution in ScaleFT in our docs.

If you are not currently logged in, these resolution queries will silently fail to resolve. Without an authenticated session, nothing will be returned from the server inventory query in the Match Exec block. If you haven’t been granted access to a server, it will also not be resolved here. To use ProxyCommand with ScaleFT, you must have an active authenticated session already open. You can get one with the command sft login.

However, because this server is managed by ScaleFT, you are logged in, and you should have granted yourself access, this command will exit 0, which is how your SSH client knows to apply the subsequent configuration directives.

ProxyCommand

OpenSSH ProxyCommand allows ScaleFT to both use ephemeral client certificates with SSH as well as to configure the SSH client based upon configurations in our host inventory. As described before, the ScaleFT Platform will figure out what bastion hops are required. These hops are used by the SSH client in the form of Host configuration blocks with names like SCALEFT-SYNTHESIZED-HOP-0. These dynamic configurations take the place of what, for an experienced operator, might be a massive personal SSH configuration file.

Here’s what a dynamic SSH configuration with one bastion hop might look like (if you were able to see inside your SSH client):

Host SCALEFT-SYNTHESIZED-HOP-0 ForwardAgent no ForwardX11 no HostName 203.0.113.20 PreferredAuthentications publickey Protocol 2 ServerAliveInterval 60 StrictHostKeyChecking yes UseRoaming no User alice Host SCALEFT-SYNTHESIZED-HOP-1 ForwardAgent no ForwardX11 no HostName 172.31.38.20 PreferredAuthentications publickey Protocol 2 ServerAliveInterval 60 StrictHostKeyChecking yes UseRoaming no User alice

All of these configurations are supplied to your SSH client by ScaleFT. This config block is mostly helpful in illustrating the flow of connections.

First, your SSH client opens a mutually authenticated and encrypted connection to the first-hop SSH server (SCALEFT-SYNTHESIZED-HOP-0).

Then, using a native port-forwarding channel to the bastion sshd, your SSH client opens an end-to-end encrypted and mutually authenticated connection to the target SSH server (SCALEFT-SYNTHESIZED-HOP-1).

Each hop in the full connection uses time-limited client certificates, and is individually dynamically authenticated, authorized, and audited.

These configurations are only used for the scope of a single SSH connection. There are a few helpful configurations in there, such as disabling SSH Agent for each host individually.

The PreferredAuthentications publickey line configures the SSH client to use client certificate authentication for that host (in your sshd logs, these logins will be identified by the string RSA_CERT, but in the client configurations, it is under publickey).

The username for each hop is independently configured, and the username is pinned in the certificate as well so the remote SSHD process can verify that only the correct user account is accessed.

UserKnownHostsFile

This SSH configuration directive allows the ScaleFT client to provide the correct host keys for each hop from the server inventory to your SSH client at connection time. Users no longer are asked if they trust hosts on first connection, so “trust on first use” is no longer an everyday part of life with SSH. ScaleFT provides this solution so that SSH host keys will finally provide the security benefits they were always intended to, without the usual inconvenience for the user (and associated headache for automation and tools).

So what?

So you can easily and securely use ScaleFT with SSH bastions. Is there anything more to say?

Consider that, since user access patterns across bastions are star-shaped, meaning that all connections emanate out from the bastion, and users do not move laterally between target nodes, you can easily deploy new network restrictions to prevent traffic on port 22 between target nodes, making lateral movement by an attacker much more difficult.

Also, notice that the two nodes in this deployment shared a VPC, and the connection to ubuntu-target traveled over that VPC. Because ScaleFT has an agent on each server, we are able to integrate with the Amazon EC2 Instance Metadata service to pull certain items of configuration data from AWS. This includes the VPC ID and the associated interface, allowing the ScaleFT Platform to route the final hop over the private network which is shared by both hosts.

You can see the metadata we pull for your sample host from your client with a command like sft list-servers -l hostname=ubuntu-target -o json. This is just an example of the features we build using cloud-specific metadata services to provide awesome convenience and security for our users.

I hope you’ve enjoyed this tour of bastion hopping with ScaleFT! If you did spin up some servers to follow along with this post, now might be a good time to run terraform destroy.

If you haven’t gotten started with ScaleFT yet, now is a good time! To get started yourself, sign up for a free trial, or get in touch with us directly to learn more.


SSH Bastions

Subscribe to the ScaleFT Newsletter to get the latest and greatest delivered straight to your inbox.

Sick of VPNs?

All of our plans start with a 30 day free trial. No credit card required. See our flexible Pricing Plans.

Start a Free Trial Request a Demo