> ## Documentation Index
> Fetch the complete documentation index at: https://vms.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Health and recovery reference

> What happens to a sandbox when its host stops, when its kernel crashes, or when its guest agent does not answer: health, errorReason, lastReboot, and the recover operation.

This reference gives what happens to a sandbox when its host stops, when its kernel crashes, or when its guest agent does not answer.

Make my agent handle host failures and crashes.

Read this reference. Make my code handle a sandbox in the error state with errorReason host\_lost or boot\_failed.
Watch for the sandbox.degraded and sandbox.recovered webhook events, and show the health field.
Do not ask me to share my API key.

## When the host stops

The host of each sandbox reports to the platform every 2 seconds. When a host does not report for 30 seconds:

1. Each `running` or `stopping` sandbox on the host moves to `error`, with `errorReason` set to `host_lost`. Billing stops. You get the `sandbox.error` event.
2. A sandbox that was `creating` or `starting` for a create moves to `failed`. Its create operation fails with the code `host_lost`.
3. A `stopped` sandbox stays `stopped`. Its files were saved at the stop.

### Automatic recovery

If the latest snapshot of a `running` sandbox is less than 2 minutes old, the sandbox moves from `error` to `starting` at once, with a `recover` operation. The sandbox starts on another host, and gets back its saved files, as a [resume from the saved files](https://vms.dev/docs/sandboxes/stop-and-resume.md#resume-from-the-saved-files) does.

* When the files are back, the sandbox is `running` and the operation is `succeeded`. You get the `sandbox.running` event.
* The processes and the memory of the sandbox do not come back. The files that changed after the latest snapshot do not come back.
* The sandbox gets the IPv6 address of its new host.
* A sandbox with `pinImage: true` starts on its pinned base image if a host with room has it. Else it starts on the latest base image.
* If no other host has room for 60 seconds, the sandbox stays in `error`, and the operation fails with the code `no_capacity`.
* If the files do not come back, the sandbox is `error` with `errorReason` set to `restore_failed`.

`snapshotVerifiedAt` shows the time of the latest snapshot. See [Saved files](https://vms.dev/docs/reference/saved-files.md).

### When the host comes back

* A sandbox that moved to another host does not run again on the old host. The old copy is deleted.
* A sandbox that stayed in `error` with `host_lost` runs again on the old host with its local disk. It moves to `starting`, then to `running`. You get the `sandbox.running` event.

You can also resume a sandbox in `error` with `PATCH /v1/sandboxes/{id}` and `"state": "running"`. It then starts from its saved files on any host with room.

## When the kernel crashes

When the kernel of a sandbox crashes, the sandbox starts again on the same host with its local disk, within 15 seconds.

* The sandbox stays `running`. Its files stay. Its processes and its memory do not.
* `lastReboot` shows `{ "reason": "crash", "at": "<time>" }`.
* If the sandbox does not start again, it moves to `error`, with `errorReason` set to `boot_failed`.

## Health

`health` shows if the guest agent of a `running` sandbox answers. It is `null` when the sandbox is not `running`.

| Value | Meaning |
| - | - |
| `ok` | The guest agent answers. |
| `degraded` | The guest agent did not answer for 60 seconds, but the virtual machine runs. The sandbox keeps running, but commands, files, and snapshots can fail. |
| `unreachable` | The guest agent did not answer for 60 seconds, and the cause is on our side: the virtual machine does not run, or the host does not answer. |

A sandbox with `health: degraded` is billed, because a program in the sandbox can block the guest agent. A sandbox with `health: unreachable` is not billed. When the guest agent answers again, `health` is `ok`.

| Event | When |
| - | - |
| `sandbox.degraded` | `health` changes from `ok` to `degraded` or `unreachable`. |
| `sandbox.recovered` | `health` changes back to `ok`. |

The guest agent listens on port 7911. A firewall in the sandbox that blocks this port makes the sandbox `degraded`. The sandbox is still billed, and a stop cannot save its files.

Read the [changelog.](https://vms.dev/docs/changelog.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.