I was tired enough of losing connections to work systems I was working on when network topology changes, or my laptop was moved, or it went to sleep, or I moved to a new computer (e.g. I'm at home) that I wrote a simple script to jump all my ssh connections through a VM at work, but with the extra step that the connection from the jump VM happens in a tmux that's named based on the desired host, and with options to reconnect to an existing session if it exists.
With the script named "go", Here's what that allows in practice:
go foo.bar - Connects to host foo.bar
go foo.bar - Second connection to host foo.bar that uses same session, so keystrokes show in both, even if they originate from separate locations, like home and work.
go foo.bar 2 - Additional param is added to session name, so you get a new connection to foo.bar.
go -list - Lists all connection sessions, and only connection sessions, because there's a special prefix to distinguish them from normal tmux sessions that might exist on the VM.
go -restore - Spawn a new terminal for all open connections. Useful for getting all terminals back after the network drops, or you reboot, or you're on your home computer instead of work, etc.
Currently this is implemented in a batch script on windows with some ugly hacks to make it work well with what PuTTY's command line options support (commands for the remote host need to be in a file you specify...), and it's pretty ugly, but I'll share if if anyone is interested. It would be much easier in bash with openssh (it's even possible OpenSSH supports enough features to do this in the ssh config).
Because convincing my org to install mosh-server on a few hundred systems is a non-starter. What I have is a small shim that sits on top of SSH the provides the interesting bits I need at no additional requirement per server. It works for any server running SSH, because it is SSH, so it already works with our key distribution system and expected way of managing servers. Not only that, SSH is well known and extremely well vetted security wise. Mosh has some nice features, but it doesn't really fit the criteria in this situation.
I do have my scrollback buffer. The tmux session is on the VM, and within that session is another ssh connection to the target system. If the target system and the VM are disconnected (very unlikely without either the target or VM restarting) then sure, I might lose my scrollback (since tmux is execed with the SSH command, when it exits the session will end), but in the much more common scenario that my side loses connectivity to the VM (or I change locations), the VM still has an active connection going on in a tmux session that I'm joining.
ET looks great for a lot of things, but not necessarily this environment, which is a few hundred systems administered by multiple people, with extremely high stability and security requirements. Honestly, all the extra stuff ET and MOSH does is to give you that extra 1-2% of features to make it seamless, but at the expense of separate protocols and new software, so you don't have to expend new hardware (or in this case, virtualized hardware.
Connectivity problems almost always come from the last mile, whether that's you moving to make the last mile somewhere else or your wifi or home connection having a problem. A VM at Digital Ocean, or in my case the highly redundant and available VMware cluster at work, is much less likely to have any sort of problems, as are the servers that are generally being connected to (and it those ARE having problems, you can't rely on sessions to them being kept anyway).
For 99% of the cases, you can get by easily by just SSHing to a highly available VM, starting a tmux session for the desired connection, and within that session SSHing to the desired system. Jumping through other systems with SSH is so common that OpenSSH has features built in to support it, even transparently (where your config can just make it automatic for a class of systems). In fact, I bet there's a way to get the OpenSSH Proxying SSH server to keep the session open to reconnect to from the client if it's only the client side that had a problem, so it doesn't even require the little script I have. It's actually on my todo list to figure out the windows included OpenSSH agent stuff and see how well the new Windows Terminal works as an SSH terminal, but I haven't gotten around to it (or just use the WSL stuff, but I haven't seen much need for it yet, I'm happy to do most my dev work in vim on a dev server).
With the script named "go", Here's what that allows in practice:
go foo.bar - Connects to host foo.bar
go foo.bar - Second connection to host foo.bar that uses same session, so keystrokes show in both, even if they originate from separate locations, like home and work.
go foo.bar 2 - Additional param is added to session name, so you get a new connection to foo.bar.
go -list - Lists all connection sessions, and only connection sessions, because there's a special prefix to distinguish them from normal tmux sessions that might exist on the VM.
go -restore - Spawn a new terminal for all open connections. Useful for getting all terminals back after the network drops, or you reboot, or you're on your home computer instead of work, etc.
Currently this is implemented in a batch script on windows with some ugly hacks to make it work well with what PuTTY's command line options support (commands for the remote host need to be in a file you specify...), and it's pretty ugly, but I'll share if if anyone is interested. It would be much easier in bash with openssh (it's even possible OpenSSH supports enough features to do this in the ssh config).