Skip to main content

An agent went offline

You opened your inbox and Retrievy told you a Windows or Docker agent stopped reporting. This page is the runbook: what the email means, how to confirm the outage in the app, the four causes that account for most of these alerts, and how to fix each one without rebuilding the host from scratch.

Before you start​

  • The Create, rotate, and revoke agent tokens permission on your role (under the Settings: Agent Tokens group in Settings → Roles & Permissions). The agent detail panel, the Run Scan button, and the Revoke Agent button only render with it.
  • Shell access to the host that runs the agent. Windows: a local admin PowerShell. Docker: a shell that can run docker commands against the host's daemon.
  • The original install token is fine to leave in place. You only need a new one if you decide to reinstall.

Quick triage​

Start in the app. Open Fleet then Agents in the left navigation.

Look at the agent row for the host named in the email:

  1. The status pill next to the hostname should read Offline in amber. If it reads Online, the agent recovered on its own between the alert and now. You can skip to the Read the alert email section so you know which all-clear to expect.
  2. The right side of the row shows a relative timestamp like Last seen 2 hours ago. Note that value. It tells you how long the outage has been running and where to start in the host logs.
  3. Click the row to open the detail panel. On the Overview tab, the connectivity card reads Not responding for a flagged outage. Open the Security & Auth tab and check the Token Lifecycle card: if Revoked At carries a timestamp, the agent is dead by design and the alert is the trailing edge of a revoke. Stop here and either restore from a fresh install or ignore.

The status badge flips to Offline after 20 minutes without a heartbeat. The first email lands after 1 hour offline. So if the email arrived this minute, the agent has been quiet for at least an hour. See Business rules catalog for the full timing table.

Read the alert email​

Retrievy sends three different emails for the agent lifecycle. The subject line tells you which one you are looking at:

Subject lineWhat it meansWhat you do
[Retrievy] Agent Offline — <hostname> (<agent name>)The first alert. The agent has been silent for more than 1 hour and this is the first time Retrievy is paging you about this outage.Triage now. The steps below.
[Retrievy] Daily Reminder — Agent <hostname> Still OfflineThe agent is still offline 24 hours after the previous email. One follow-up per day until the agent comes back or you revoke it.If you have not started yet, start now. The agent has been down at least a day.
[Retrievy] All Clear — Agent <hostname> is Back OnlineThe agent resumed heartbeats after an alert.No action. The outage is closed. The email shows the restored time and how long the agent was offline.

The body of every email shows the agent name, the hostname, and the timestamps rendered in your workspace timezone (set under Account then Workspace Settings). The agent name is the descriptive label you gave the install token (something like dc01-london). The hostname is what the host reports about itself at first contact.

If you got the Daily Reminder and you have already decided the host is permanently retired, revoke the agent so the reminders stop. See Revoke and reinstall below.

Common causes and fixes​

Four root causes cover almost every alert. Walk them in order. The first one you cannot rule out is usually the cause.

1. The service or container stopped​

Most often the host is up and reachable but the agent process itself is not running. A reboot, a Windows Update, a docker compose down, an OOM kill on the host, or a manual stop someone forgot about.

Windows. From an admin PowerShell on the host:

# Confirm the service exists and check its state
Get-Service RetrievyAgent

If the Status column reads Stopped, start it:

Start-Service RetrievyAgent

Wait two minutes, then refresh the Agents page in Retrievy. The badge flips back to Online on the next successful heartbeat (heartbeats run on a 10-minute cadence, so a freshly started service often lands within one to two minutes).

If Get-Service reports The service name RetrievyAgent does not exist, the install was removed (uninstaller ran, GPO cleanup script, image rebuild). The agent record in Retrievy is orphaned. Revoke it and reinstall. See Revoke and reinstall below.

Docker. From a shell on the Docker host:

# List the container regardless of state
docker ps -a --filter name=retrievy-agent

If the STATUS column reads Exited, restart it:

docker start retrievy-agent

If the container is missing entirely (the row is gone), the container was removed. Recreate it with the docker run line on the token reveal screen, or with your Compose file. See Install the agent with Docker.

If the container is Up but the badge still reads Offline, the process is running and crashing in a loop, or it is running but cannot reach Retrievy. Tail the logs and move to the next cause:

docker logs --tail 100 retrievy-agent

2. Outbound network blocked​

The agent talks outbound HTTPS on port 443 to your workspace URL (<your-workspace>.retrievy.com). It opens no inbound ports. If a firewall change, a new corporate proxy, a DNS misconfiguration, or a TLS-intercepting middlebox sits between the agent and Retrievy, every heartbeat fails silently from the agent's perspective and the row goes Offline.

Windows. Test connectivity from the agent's host:

# Confirm TCP reachability on 443
Test-NetConnection <your-workspace>.retrievy.com -Port 443

# Confirm TLS works end to end (no MITM)
Invoke-WebRequest https://<your-workspace>.retrievy.com -UseBasicParsing -Method Head

TcpTestSucceeded : True proves the route is open. The Invoke-WebRequest call proves the TLS chain is trusted by the host. If your network re-signs HTTPS traffic for inspection, the agent rejects the re-signed certificate by design and stays offline. Add an exception for *.retrievy.com to bypass TLS inspection, or import the inspection CA into the LocalMachine Root store (treat this as a security decision).

Docker. Test from inside the container so you exercise the same egress path the agent uses:

# TCP reachability and TLS handshake in one shot
docker exec retrievy-agent curl -I https://<your-workspace>.retrievy.com

Any non-2xx response, a hang, or a TLS error means the egress path is broken. Compare with the same command from the host shell: if the host succeeds but the container fails, the container's network mode or its DNS configuration is the problem.

3. Token revoked or rotated incorrectly​

This one is rarer but worth ruling out quickly. If someone clicked Revoke Agent on the agent's detail panel, the host self-destructs on its next successful contact. The Retrievy side already counts the row as Offline.

In the agent's detail panel, open the Security & Auth tab and read the Token Lifecycle card:

  • Revoked At carries a timestamp: the agent was revoked deliberately. Either the agent has not reached Retrievy yet to receive its self-destruct signal (still showing in the list), or revoke was the cause of the outage. Reinstall is the recovery path.
  • Install Token reads Consumed and Session Token reads Active: the agent registered successfully and held a working session. The token is not your problem. Move to the next cause.
  • Install Token reads Pending and Session Token reads Not issued: the agent never completed first registration. Install tokens expire 60 minutes after generation. Generate a new token from Settings then Agents then New Token and rerun the install. See Windows install or Docker install.
Do not rotate the workspace URL

The agent's session token is bound to the workspace it registered against. If you move workspaces, change subdomains, or restore a backup into a different tenant ID, every existing agent fails authentication and goes Offline. There is no in-place fix. Revoke and reinstall with a fresh token against the new workspace.

4. Host clock skew​

The agent signs every heartbeat with a short-lived signing window. If the host clock drifts more than five minutes from real time, the signature falls outside the accepted window and Retrievy rejects every request. The agent looks healthy locally and the host looks reachable, but the row in Retrievy stays Offline.

Windows. Confirm the time service is running and force a sync:

# Confirm time service health
Get-Service w32time

# Force an immediate sync against the configured NTP source
w32tm /resync /force

# Confirm the offset is within a few seconds
w32tm /stripchart /computer:time.windows.com /samples:3 /dataonly

The /stripchart output prints the offset in seconds. Anything under one second is fine. Sub-minute drift is not the cause. Drift above one minute usually points at a stopped w32time service, a virtualised host with broken time integration, or an air-gapped network without an internal NTP server.

Docker. Container clocks inherit the host clock. Fix the host:

# Linux hosts: confirm chrony or systemd-timesyncd is active
timedatectl status

# Force a sync
sudo chronyc -a makestep   # if chrony is installed
# or
sudo systemctl restart systemd-timesyncd

A typical symptom of clock skew is rapid heartbeat retries in the container log (401, signature verification failed, or request outside signing window). Tail the log and look for those strings before you spend time elsewhere.

Force a heartbeat​

If you fixed the cause and do not want to wait up to ten minutes for the next scheduled heartbeat, you can prompt the agent to retry now without a full service restart.

Windows. Pause and resume the service. The agent treats resume as a "wake up and check in" signal:

# Forces the agent to re-evaluate state and send a heartbeat
Suspend-Service RetrievyAgent
Resume-Service RetrievyAgent

If you would rather see a clean restart, use:

Restart-Service RetrievyAgent

The badge updates within one to two minutes of the next successful heartbeat. Refresh the Agents page or watch the Overview tab of the agent detail panel: the connectivity card swaps from Not responding to Actively reporting and the relative timestamp updates.

Docker. Send SIGHUP to the running container so it re-reads its state and heartbeats immediately:

docker kill --signal=HUP retrievy-agent

If SIGHUP is not honoured (older builds), restart the container:

docker restart retrievy-agent

You will see the new heartbeat in the agent log within a few seconds:

docker logs --tail 20 -f retrievy-agent

Revoke and reinstall​

Sometimes in-place repair is the wrong call. Pick reinstall when:

  • The host was rebuilt, restored from a snapshot, or imaged from a different machine. Its identity no longer matches what Retrievy recorded.
  • The state volume or the registry key holding the session token was wiped (a stray docker volume rm, a registry cleanup script, a Windows reset).
  • You moved the host to a new workspace. Session tokens do not cross workspaces.
  • The Windows service or the Docker container exists but cannot complete a single heartbeat after the four causes above are ruled out. The local install is corrupt.

Reinstall in three steps:

  1. Revoke the orphaned agent in Retrievy. Open Fleet then Agents, click the row, open Security & Auth, scroll to Danger Zone and click Revoke Agent. The confirmation modal has a five-second cooldown. See Manage your agent fleet for what revoke does on the host.
  2. Uninstall the old agent on the host. Windows: msiexec /x RetrievyAgent-v[version].msi /qn or remove via Apps and Features, then delete HKLM\SOFTWARE\Retrievy\Agent for a fully clean state. Docker: docker stop retrievy-agent && docker rm retrievy-agent && docker volume rm retrievy-agent-state.
  3. Generate a fresh install token and reinstall. See Install the agent on Windows or Install the agent with Docker. The new install token is one-time-use and expires in 60 minutes.
Revoke is permanent for that install

Revoke kills the agent on its next contact. The host wipes its binaries, its state, and its RSA keys. You cannot un-revoke. If the agent was proxying FortiGate devices, those devices fall back to No agent assigned and you will need to re-enter their passwords on the next agent.

After the reinstall completes, the badge flips to Online with a fresh last-seen timestamp. The all-clear email does not fire for a reinstalled agent (it is a new record, not a recovered one). The old offline alerts in your inbox are still valid history.