Working at ScaleFT means constantly seeing new ways that people use SSH. When I talk to users, or prospective users, I get to hear all about how they secure their environments and how they structure network access into and between their servers. SSH bastions are frequently a part of this picture.
A bastion can be any host, typically a Linux host, which is available to inbound connections from the public internet, and can initiate outbound network connections into a private network. By existing in a DMZ, bastions enable the target hosts behind them to reside in private networks, which prohibit inbound network access except for that originating from the bastion or DMZ. Most users use bastions with SSH port-forwarding to enable transparent, authenticated access to the hosts in the private network.
These commonalities aside, bastions are rarely set up or used in the same way. Every organization has a set of different production architectures because they think about things differently. Different teams of engineers with different needs, different tools, and different mindsets, building and maintaining their systems under different constraints, and through years of changing conditions, naturally end up producing very different results.
In this wide variety of outcomes, you frequently find unfortunate consequences for bastion architecture, leading to poor security posture and user experiences, such as:
- Private keys kept on the bastion
- Shared users used on the bastion
- Shared deployment (or other) tools on the bastion
ssh-agentenabled for bastion hopping
- Users being forced through SSO/MFA hoops multiple times, once per SSH hop
- Lateral movement becoming habitual for authorized users in the environment to avoid round trips through the bastion
- Long-lived screen sessions or tunnels are left in place to avoid bastion hops
- A bastion being configured to MITM all SSH traffic for auditing
- A complex, hand-maintained, per-region SSH client configuration being required for bastion hops, which vary across the team and makes onboarding new teammates awkward
One of our values at ScaleFT is to do our best to support our users where they are, with the decisions and tools they’ve already selected. This means treating SSH bastions as an SSH feature, parameterizing and centralizing the associated configurations, and seamlessly integrating it into our users’ daily workflows. We’ve taken what we think of as the best practices and behaviors, and made them work transparently with SSH on the client and the server side.
If any of the problems listed above sounds familiar to you, you might be surprised how much ScaleFT’s bastion support could improve your team’s experience working with your production environment.
Bastion hopping in action
The commands in this post are being run against an environment created by this terraform module.
If you have an AWS account, feel free to try this out yourself. A moderate amount of setup may be required and is documented in the README. If you’ve already tried out ScaleFT, you’ve probably already done most of this setup.
Among the recommended setup steps is configuring ScaleFT to work with your SSH client using OpenSSH ProxyCommand. ProxyCommand is nice, not only because it enables seamless use of ScaleFT with SSH, but because it also enables using ScaleFT with all the tools that use SSH as a transport, such as
git, or even Ansible. For this reason, ScaleFT’s bastion features are most convenient and impressive when combined with ProxyCommand.
This terraform module will spin up two servers,
ubuntu-target server will be automatically configured to use the
ubuntu-bastion server as an SSH bastion. This configuration is done in a userdata script which you can find on Github.
In ScaleFT, bastions are configured as attributes of target nodes. This means that users are able to just directly use tools referring only to the names of target nodes, and ScaleFT can work backwards from the target, through each individual bastion hop, and return a complete connection chain to your SSH client.
Users don’t even need to know a bastion is there. This also means you can easily change a bastion, or add new bastion hops, and your users will likely never even notice.
Using your usual SSH client
If you have configured ProxyCommand, you will now be able do a bastion hop with just one command:
Otherwise, for now you can use the command
sft ssh ubuntu-target. This will use the same methods to open an interactive SSH session with the remote machine, and also does provide features like port forwarding, but in the long run you’re going to want to configure ProxyCommand so you can use everything else SSH has to offer.
Once your session is open on the target server, check out the output of
last | head. You’ll see that your incoming SSH connection came from a private IP address in the same AWS VPC as your target host. It did not originate from whatever home or office wifi you are trying this out from.
To see the difference, you can directly
ssh ubuntu-bastion and check
last, and then you will see both your prior bastion hop, and your current session, with the full connection flow chain made clear.
ssh command you issued took you through a bastion hop, having done a few ScaleFT API interactions in the background to request a few ephemeral certificates, and I bet you didn’t notice any differences between that and a direct SSH connection.
You can also try using
scp to transfer some files to
ubuntu-target; that will use the bastion and ScaleFT credentials automatically as well.
The ProxyCommand configuration
The ProxyCommand configuration that you’re using looks something like this:
Let’s step through these lines and see what each one does.
When you run
$ ssh ubuntu-target, your SSH client encounters the “Match exec” directive in your SSH config file. This is a conditional block. The directive causes your SSH client to briefly fork the command in quotes as a child process, substituting in “ubuntu-target” for the
%h, to determine if the entire configuration block applies to this
This allows our local
sft binary, which is installed by the same installer that installs our GUI application, to issue a metadata query to the server inventory maintained in the ScaleFT Platform to see if there is a server known by that name which you have been granted access to. If the server name resolves, your SSH client knows to apply the remaining configurations in the block to your SSH command. You can read more about resolution in ScaleFT in our docs.
If you are not currently logged in, these resolution queries will silently fail to resolve. Without an authenticated session, nothing will be returned from the server inventory query in the
Match Exec block. If you haven’t been granted access to a server, it will also not be resolved here. To use ProxyCommand with ScaleFT, you must have an active authenticated session already open. You can get one with the command
However, because this server is managed by ScaleFT, you are logged in, and you should have granted yourself access, this command will exit
0, which is how your SSH client knows to apply the subsequent configuration directives.
OpenSSH ProxyCommand allows ScaleFT to both use ephemeral client certificates with SSH as well as to configure the SSH client based upon configurations in our host inventory. As described before, the ScaleFT Platform will figure out what bastion hops are required. These hops are used by the SSH client in the form of Host configuration blocks with names like SCALEFT-SYNTHESIZED-HOP-0. These dynamic configurations take the place of what, for an experienced operator, might be a massive personal SSH configuration file.
Here’s what a dynamic SSH configuration with one bastion hop might look like (if you were able to see inside your SSH client):
All of these configurations are supplied to your SSH client by ScaleFT. This config block is mostly helpful in illustrating the flow of connections.
First, your SSH client opens a mutually authenticated and encrypted connection to the first-hop SSH server (
Then, using a native port-forwarding channel to the bastion
sshd, your SSH client opens an end-to-end encrypted and mutually authenticated connection to the target SSH server (
Each hop in the full connection uses time-limited client certificates, and is individually dynamically authenticated, authorized, and audited.
These configurations are only used for the scope of a single SSH connection. There are a few helpful configurations in there, such as disabling SSH Agent for each host individually.
PreferredAuthentications publickey line configures the SSH client to use client certificate authentication for that host (in your
sshd logs, these logins will be identified by the string
RSA_CERT, but in the client configurations, it is under
The username for each hop is independently configured, and the username is pinned in the certificate as well so the remote SSHD process can verify that only the correct user account is accessed.
This SSH configuration directive allows the ScaleFT client to provide the correct host keys for each hop from the server inventory to your SSH client at connection time. Users no longer are asked if they trust hosts on first connection, so “trust on first use” is no longer an everyday part of life with SSH. ScaleFT provides this solution so that SSH host keys will finally provide the security benefits they were always intended to, without the usual inconvenience for the user (and associated headache for automation and tools).
So you can easily and securely use ScaleFT with SSH bastions. Is there anything more to say?
Consider that, since user access patterns across bastions are star-shaped, meaning that all connections emanate out from the bastion, and users do not move laterally between target nodes, you can easily deploy new network restrictions to prevent traffic on port 22 between target nodes, making lateral movement by an attacker much more difficult.
Also, notice that the two nodes in this deployment shared a VPC, and the connection to
ubuntu-target traveled over that VPC. Because ScaleFT has an agent on each server, we are able to integrate with the Amazon EC2 Instance Metadata service to pull certain items of configuration data from AWS. This includes the VPC ID and the associated interface, allowing the ScaleFT Platform to route the final hop over the private network which is shared by both hosts.
You can see the metadata we pull for your sample host from your client with a command like
sft list-servers -l hostname=ubuntu-target -o json. This is just an example of the features we build using cloud-specific metadata services to provide awesome convenience and security for our users.
I hope you’ve enjoyed this tour of bastion hopping with ScaleFT! If you did spin up some servers to follow along with this post, now might be a good time to run