# Stop Sequence Order: Why Disable systemd BEFORE Killing the Manual Launcher

## The trap

On a Linux box where both a `hermes-gateway.service` systemd unit and a manually-launched `hermes gateway run` (from `bash -c 'cd /root && ~/.hermes/venv/bin/hermes gateway run 2>&1 &'`) are present, **the order of teardown matters**. If you kill the manual launcher first, the systemd unit's `Restart=always` + `RestartSec=10` brings a fresh one back in 10 seconds. If a manual launcher is holding the lock, systemd's respawned process fails fast on the lock and goes into an `activating (auto-restart)` loop with the counter climbing (saw 537+ in one real session).

## The correct order (POSIX / Linux)

```bash
# 1. Stop systemd from respawning anything. Do this FIRST.
sudo systemctl disable --now hermes-gateway.service

# 2. Verify the unit is actually down (it was 'enabled + active' before this).
#    Both of these should match the commented expected output.
systemctl is-enabled hermes-gateway.service  # disabled
systemctl is-active hermes-gateway.service   # inactive

# 3. Find and kill the manual launcher.
#    The manual launcher is a bash parent (PID 1 of the gateway) + python child.
#    Kill the bash parent first or the python will respawn inside 1s.
ps -ef | grep "hermes.*gateway" | grep -v grep
#    Output looks like:
#    root  1994597  1  0 04:30 ?  00:00:00 bash -c 'cd /root && ~/.hermes/venv/bin/hermes gateway run 2>&1 &'
#    root  1994598  1994597  0 04:30 ?  00:00:12 /root/.hermes/venv/bin/python3 /root/.hermes/venv/bin/hermes gateway run
#
#    Kill the bash launcher AND the python gateway. Both. The python PID is
#    what holds the bot-token lock; the bash PID is the wrapper that respawns
#    the python if the python dies.
sudo kill -9 1994597 1994598

# 4. Kill the dashboard on 9119 if it was a stray (NOT the TUI's own chat
#    backend, which has --session-key as a child flag).
sudo kill -9 <dashboard-pid>

# 5. Confirm clean. Should print 'clean' (or be empty).
ps -ef | grep -E "hermes (gateway|serve)" | grep -v grep || echo "clean"

# 6. Confirm no port 9119 listener.
ss -tlnp | grep 9119 || echo "port 9119 free"
```

## The user-systemd subtlety

`hermes gateway run` (the manual launcher) often forks its own user-level systemd (`/usr/lib/systemd/systemd --user`, PPID 1) to supervise itself. If you do:

```bash
sudo systemctl disable --now hermes-gateway.service
```

and get:

```
Failed to disable unit: Unit file hermes-gateway.service does not exist.
```

…then the user-systemd is supervising via an in-memory transient unit, not a saved unit file. You can't disable what doesn't exist. **Kill the bash launcher directly** — `kill -9` on the bash PID is what actually stops the respawn loop. After that, the user-systemd supervisor notices the process group is gone and tears itself down.

## Real session transcript (abbreviated)

```
# First attempt — wrong order, kills manual launcher first:
$ kill -9 1994598   # python only, bash still alive
$ sleep 2
$ ps -ef | grep hermes | grep -v grep
root  1994597  1  0 04:30 ?  00:00:00 bash -c '...hermes gateway run 2>&1 &'
root  2001234  1994597  0 04:30 ?  00:00:00 /root/.hermes/venv/bin/python3 /root/.hermes/venv/bin/hermes gateway run
# New python PID 2001234 — bash respawned it in <1s.

# Second attempt — right order, kill the bash parent:
$ kill -9 1994597 2001234
$ sleep 2
$ ps -ef | grep -E "hermes (gateway|serve)" | grep -v grep || echo "clean"
clean
```

## Verify after restart

When the user clicks Start in the Web UI, expect a one-time Telegram polling conflict (1/5) as the previous Telegram long-poll expires. It's not a real problem; it self-resolves within ~2 minutes. If the conflict keeps retrying past attempt 5, then something else is polling the same bot token — investigate before continuing.

## What NOT to do

- **Do not reboot the VPS mid-fix.** The Ubuntu "*** System restart required ***" banner is a nag, not a stop signal. Rebooting while systemd is mid-restart-loop will let the loop resume and you'll be back where you started.
- **Do not try to `hermes gateway stop` from inside a TUI session hosted on the same gateway.** The CLI blocks this with "cannot stop the gateway from inside the gateway process." This is correct behavior — SIGTERM would propagate to the very shell you're typing in.
- **Do not assume the 5 `Hermes.exe` processes on Windows are gateway processes.** On a Windows box with Hermes Desktop installed, the expected baseline is 5 `Hermes.exe` with identical `StartTime` and Path under `apps/desktop/release/win-unpacked/`. Those are the Desktop app's renderer/GPU/helpers — leave them alone. See `operating-hermes-gateway` SKILL.md "Windows diagnostic checklist" for the `Path`-based filter that distinguishes them from a real gateway process.