This post was drafted with AI, based on a real production troubleshooting session but I was too lazy to write it down myself. Details have been generalized/anonymized, but the investigation, root cause, and fix are real.
TL;DR
SSH to a newly hardened Windows domain died with a bare Connection reset by peer — no error, no password prompt, nothing. Firewalls, VPN, and IPv6 were all innocent; a full packet capture showed the connection dying the instant authentication began, right after a clean key exchange.
The real cause: our SSH server used a Kerberos mechanism called S4U2Self to check group membership before the client even sent a password — a passwordless “give me a token for this user” trick. Recent AD hardening (delegation restrictions on sensitive accounts) makes the domain controller flatly refuse that request, and the SSH server responded to the refusal by silently killing the connection instead of falling back to a normal password check.
Fix: stop restricting SSH access by group (which needs that passwordless group-membership check) and restrict by username instead (a plain string match that never touches Kerberos delegation). One line changed in the server config, no AD changes needed — though the username-matching syntax had its own trap worth knowing about (details below).
Lesson: a hard reset with zero application-level error almost always means something crashed or was deliberately killed server-side — go looking there first, and don’t assume “verbose logging” is actually landing where you’re looking for it.
The symptom
We stood up a new, more heavily hardened Windows domain for test automation, migrated our SSH-based tooling over, and hit a wall immediately: every SSH connection attempt against the new machines died with Connection reset by peer right in the middle of the handshake. Not a timeout, not “access denied” — a hard TCP reset, with no SSH-level error message at all.
The frustrating part: the exact same client, the exact same automation account, the exact same SSH server software worked flawlessly against our older, less-hardened domain. Something about the hardening had broken SSH — but nothing in the setup looked obviously wrong.
First instincts, and why they were wrong
The obvious first guesses were all network-shaped: firewall rules, VPN routing, maybe an IPv6 vs IPv4 quirk. We’d actually been burned by a firewall-profile issue on this same domain before (a rule scoped to the wrong network profile), so it was a reasonable place to start.
But the packet trace told a different story. A full ssh -vvv capture showed the TCP handshake completing, the SSH version banners exchanging, and a complete, successful key exchange — algorithm negotiation, Diffie-Hellman, new keys installed on both sides. The connection only died the instant the client sent its first authentication request. That ruled out routing and firewalls outright: you don’t get a clean key exchange over a broken network path.
Narrowing it down: isolate one variable at a time
With network eliminated, the only responsible way forward was to change exactly one thing at a time and see what moved the needle.
Remote vs. local. We reproduced the exact same reset connecting from the server to itself over loopback. That ruled out anything about the client machine, DNS, or the path between two hosts — the bug was entirely inside this one machine’s SSH server process.
Domain account vs. local account. We tried a purely local, non-domain account instead of our usual domain service account, and it also failed to connect. Worth being precise here rather than papering over it: a local account resolves its group membership straight out of the local SAM database — no AD round-trip, no Kerberos, no S4U2Self involved at all. So if you try this yourself and get a clean, named failure (something like a logon-type-not-granted error) instead of a raw reset, that’s not evidence you’ve found a different bug — it’s actually consistent with the same root mechanism, just failing on a different, non-Kerberos code path along the way. Either way, this test did its job: it ruled out a whole category of theories about Active Directory group-membership resolution being broken for domain principals specifically, since local accounts don’t depend on AD at all.
Service vs. foreground process. We stopped the SSH service and ran the server binary directly in a console, in debug mode, as ourselves. That connection succeeded and reached a password prompt — which was actually a red herring, as it turned out later: it succeeded only because a login to “yourself” takes a shortcut in the code and reuses the already-existing process token instead of doing real authentication. Testing with a different account against that same foreground process gave an explicit, clean error message (“unable to generate a token — not running as system”) — informative, but also not the real bug, since the production server runs as the system account and doesn’t hit that particular check.
Each of these tests was individually inconclusive, but together they did their job: they eliminated the network, eliminated AD-account-type as the sole variable, and told us the crash lived specifically in how the actual running service authenticated a different account.
Getting real diagnostic data was its own battle
Windows’ Event Viewer is genuinely painful to extract detail from — no easy copy/paste, and worse, we discovered that even with the SSH daemon’s most verbose logging level enabled, the Windows Event Log for this service only ever recorded a single bland “connection accepted” line per attempt. None of the detailed protocol-level trace you’d expect ever made it into the Event Log at all, service-side.
The fix was to stop looking in Event Viewer entirely and instead point the server at a plain debug log file directly (most SSH server implementations on Windows support redirecting verbose output to a file via a command-line flag, separate from whatever the Windows Service normally does). That was the turning point — it let us capture the real service’s own debug output, in its real execution context, for the very connection that was failing.
Getting to that point took a couple of wrong turns worth naming, since none of this was obvious from the documentation:
- Setting the config’s log level to maximum verbosity wasn’t enough on its own. The server’s own config file supports a “log everything” setting, and we had it on the whole time — it just didn’t matter, because the destination (the Windows Event Log channel) simply doesn’t carry that level of detail for this service, regardless of what the config asks for. Cranking up verbosity is useless if you’re reading it back from a channel that discards most of it.
- Running the server binary manually in the foreground, in its own debug mode, looked like the obvious workaround — and it was actively misleading. Debug mode refuses to run as a background service (by design, it only handles one connection and exits), so it has to be launched as yourself in a console. That’s the trap: logging in as the same account the process is running as takes an internal shortcut and skips real authentication entirely, which is why our first foreground test “succeeded” for no meaningful reason. Testing with a genuinely different account from that same console instead threw an explicit, different error about the process not running with sufficient privilege — informative, but still not the production failure, because the real service runs with a different, more privileged identity than a console you’re sitting at.
- Even reproducing that privileged identity by hand (via a tool that lets you open a console as the system account) didn’t line up with production either. It failed, but earlier and differently than the real service did — a strong sign that a manually-launched “as the system account” console isn’t quite the same execution context as a properly service-managed process, even when the account name matches.
- What actually worked: leave the config’s verbosity setting in place, but change how the service itself is launched so its output goes to a plain file instead of the Event Log — appending a “log to this file” flag directly onto the service’s own startup command (not the foreground-debug flag, which would break the service’s ability to handle more than one connection). That’s a one-line change to how the service is registered, doesn’t touch its actual authentication behavior at all, and — critically — preserves the exact real production context while finally producing full detail.
- One practical gotcha: the log file can end up locked while the service is actively running, so reading it cleanly sometimes meant briefly stopping the service first. Minor, but worth expecting rather than being confused by.
The smoking gun
Buried in that log, right at the point of failure, were two lines that explained everything:
generate_s4u_user_token: LsaLogonUser() failed. User '<account>' Status: 0xC000006D SubStatus 0. unable to generate token for user <account>
0xC000006D is STATUS_LOGON_FAILURE — but the interesting part is which API call produced it: not a normal password check, but something called an S4U2Self request.
Fastest way to confirm you’re hitting this: the status code alone isn’t enough — the substatus is what actually tells you whether this is a delegation refusal versus something else entirely (a typo’d account, a disabled account, etc.). If you’re chasing something similar, correlate the same event on the domain controller side (Event ID 4625 logs both the status and substatus together) rather than relying on the client symptom alone. A
0xC000006Dwith substatus0pointed us at a generic logon failure during token generation; other substatus values (e.g.0xC0000064,STATUS_NO_SUCH_USER) would point at a different root cause even though the top-level status code looks identical — don’t stop reading at the status code.
S4U2Self (“Service for User to Self”) is a Kerberos extension that lets a service ask the domain controller, “give me a token for this user,” without needing that user’s password. It’s how our SSH server was checking group membership for an access-control directive (an “only allow these groups to log in” rule) before the client had even sent a password — there’s no credential to check yet at that point, so the server uses this passwordless trick just to peek at what groups the account belongs to.
If that passwordless check fails, the server doesn’t fall back to “well, let’s just ask for the password and find out” — it treats the account as unresolvable and kills the connection outright, with no explanation sent back to the client. That’s the hard reset we’d been chasing the entire time, and it had nothing to do with the actual password, the network, or even (directly) whether the account was local or domain.
Why hardening broke this specifically
S4U2Self is fundamentally a delegation mechanism — a service vouching for a user without the user being present to authenticate themselves. Several well-known, entirely reasonable Active Directory hardening practices explicitly restrict exactly this kind of delegation on sensitive accounts:
- Marking an account as “This account is sensitive and cannot be delegated” (a single checkbox/attribute on the account)
- Membership in the Protected Users security group, which blanket-blocks delegation, legacy authentication protocols, and credential caching for its members
- Authentication Policy Silos that explicitly restrict where and how an account can be used for delegation
Any one of these — all textbook hardening moves for admin or service accounts — will cause the domain controller to flatly refuse an S4U2Self request for that account, which is exactly the failure we captured. The old, less-hardened domain simply never had these protections in place, so the same SSH server code had always worked there without anyone noticing it depended on S4U2Self at all.
If you’re hitting something similar, those three are the concrete, actionable things to go check on the affected account first — “some AD hardening broke it” isn’t a diagnosis you can act on, but “is this account marked non-delegable, in Protected Users, or covered by an auth policy silo” is a five-minute check.
A tempting shortcut that isn’t worth taking
Early in the investigation, before any of this was understood, the obvious-looking fix on the table was adding the affected accounts (or the servers themselves) to the legacy “Pre-Windows 2000 Compatible Access” built-in group. It’s an old group that grants broad read access to user and group attributes across the directory — which, mechanically, is exactly the kind of access S4U2Self-style group-membership resolution needs. It probably would have made the symptom disappear.
It’s also exactly the kind of fix that undoes the hardening you just did. That group exists for backward compatibility with pre-Active-Directory systems, and it works by granting broad directory-read rights to principals like Authenticated Users (in some configurations, effectively Anonymous Logon too) — the same kind of wide-open enumeration capability that modern AD hardening baselines specifically try to close off. Using it here would have “fixed” this one symptom by quietly reopening a much bigger door than the one it was patching, on a domain that had just been hardened specifically to close doors like it. Security correctly pushed back on this, and that pushback is what forced the deeper investigation that found the real, narrow cause.
It’s a useful gut-check for this kind of problem in general: if the fix you’re reaching for undoes the security property you were trying to add in the first place, it’s not a fix — it’s a rollback wearing a disguise. The fix that actually survived (below) doesn’t touch AD permissions at all, which is the property you want: a fix on a hardened environment should be at least as narrow as the problem it’s solving, not broader.
The fix (and a subtle trap in it)
The access-control directive that triggered the passwordless group check could be replaced with a username-based restriction instead of a group-based one. Matching against a username is a plain string/glob comparison — it never needs to resolve group membership, never touches Kerberos delegation, and so never triggers S4U2Self at all. Once we made that change, connections sailed straight through to a real password prompt with no reset.
There’s a genuinely nasty trap hiding in that fix, though. On this domain, the account name the SSH server actually authenticates against is the fully-qualified form (user@domain), not the bare short name. Username-restriction directives in most SSH server configs treat an @ inside the restriction pattern itself as a special separator, meaning “this user, but only from this specific host” — so writing the restriction as user@domain doesn’t do what it looks like it does; it gets misparsed as a host restriction and silently fails to match the real login:
# Looks right, silently wrong — the @ gets parsed as a host restriction, # not as part of the username, so this never matches the real login: AllowUsers svc-account@corp.example # Works — no @ in the pattern, so the whole string is matched as a # plain username glob: AllowUsers svc-account*
The working form needed to avoid an @ in the pattern entirely (a trailing wildcard did the trick), matching the account name as a plain prefix instead.
One caveat worth stating plainly: this trades group-based scalability for a flat, manually-maintained list. That’s a fine trade for a handful of automation accounts, but it’s not a permanent architectural answer — if the allowed-list keeps growing, the actual fix is resolving the delegation restriction on the AD side (or restructuring which accounts need SSH access at all) rather than leaning on AllowUsers indefinitely as a workaround for a problem it doesn’t actually solve.
Takeaways
A few things from this debugging session feel worth carrying forward to the next mystery like it:
- A hard TCP reset with zero application-level error is a huge clue on its own. It usually means something crashed or explicitly killed the connection server-side, rather than a graceful “access denied” — which should immediately shift attention away from credentials-are-wrong theories and toward “what is the server doing internally right before it would normally respond.”
- Change one variable at a time, even when each individual test feels inconclusive. Loopback vs. remote, local account vs. domain account, foreground process vs. service — none of those tests alone explained the bug, but together they fenced in exactly where it had to be.
- Don’t trust that your logging is actually capturing what you think it is. We had the server’s most verbose logging level enabled the entire time and were still missing the actual failure, because the log destination we were checking simply didn’t carry that level of detail. Always sanity-check that a “verbose” log actually contains verbose content for a known event before trusting its absence of evidence.
- Access-control conveniences (like “allow this group”) can have expensive, non-obvious implementation costs. A directive that reads as a simple authorization check triggered a full passwordless Kerberos delegation request under the hood — a mechanism most engineers configuring that directive would have no reason to know existed, let alone that it was sensitive to unrelated security hardening applied months later by a different team.
- Security hardening and infrastructure tooling need to be validated together, not in isolation. Both the SSH server configuration and the account hardening were individually correct and reasonable. The failure only existed at their intersection, and nothing about either change in isolation would have surfaced it during review.
