See this, for example: https://groups.google.com/g/opensshunixdev/c/FVv_bK16ADM/m/R...
It boilds down to using a Linux-specific API, though it's really BSD that is lacking support for a standard (RFC 5014).
Well, for different reasons, but you have similar issues with IPv6 as well. If your client uses temporary addresses (most likely since they're enabled by default on most OS), OpenSSH will pick one of them over the stable address and when they're rotated the connection breaks.
For some reason, OpenSSH devs refuse to fix this issue, so I have to patch it myself:
--- a/sshconnect.c
+++ b/sshconnect.c
@@ -26,6 +26,7 @@
#include <net/if.h>
#include <netinet/in.h>
#include <arpa/inet.h>
+#include <linux/ipv6.h>
#include <ctype.h>
#include <errno.h>
@@ -370,6 +371,11 @@ ssh_create_socket(struct addrinfo *ai)
if (options.ip_qos_interactive != INT_MAX)
set_sock_tos(sock, options.ip_qos_interactive);
+ if (ai->ai_family == AF_INET6 && options.bind_address == NULL) {
+ int val = IPV6_PREFER_SRC_PUBLIC;
+ setsockopt(sock, IPPROTO_IPV6, IPV6_ADDR_PREFERENCES, &val, sizeof(val));
+ }
+
/* Bind the socket to an alternative local IP address */
if (options.bind_address == NULL && options.bind_interface == NULL)
return sock;