Skip to main content
This reference gives what happens to a sandbox when its host stops, when its kernel crashes, or when its guest agent does not answer.

Make my agent handle host failures and crashes.

When the host stops

The host of each sandbox reports to the platform every 2 seconds. When a host does not report for 30 seconds:
  1. Each running or stopping sandbox on the host moves to error, with errorReason set to host_lost. Billing stops. You get the sandbox.error event.
  2. A sandbox that was creating or starting for a create moves to failed. Its create operation fails with the code host_lost.
  3. A stopped sandbox stays stopped. Its files were saved at the stop.

Automatic recovery

If the latest snapshot of a running sandbox is less than 2 minutes old, the sandbox moves from error to starting at once, with a recover operation. The sandbox starts on another host, and gets back its saved files, as a resume from the saved files does.
  • When the files are back, the sandbox is running and the operation is succeeded. You get the sandbox.running event.
  • The processes and the memory of the sandbox do not come back. The files that changed after the latest snapshot do not come back.
  • The sandbox gets the IPv6 address of its new host.
  • A sandbox with pinImage: true starts on its pinned base image if a host with room has it. Else it starts on the latest base image.
  • If no other host has room for 60 seconds, the sandbox stays in error, and the operation fails with the code no_capacity.
  • If the files do not come back, the sandbox is error with errorReason set to restore_failed.
snapshotVerifiedAt shows the time of the latest snapshot. See Saved files.

When the host comes back

  • A sandbox that moved to another host does not run again on the old host. The old copy is deleted.
  • A sandbox that stayed in error with host_lost runs again on the old host with its local disk. It moves to starting, then to running. You get the sandbox.running event.
You can also resume a sandbox in error with PATCH /v1/sandboxes/{id} and "state": "running". It then starts from its saved files on any host with room.

When the kernel crashes

When the kernel of a sandbox crashes, the sandbox starts again on the same host with its local disk, within 15 seconds.
  • The sandbox stays running. Its files stay. Its processes and its memory do not.
  • lastReboot shows { "reason": "crash", "at": "<time>" }.
  • If the sandbox does not start again, it moves to error, with errorReason set to boot_failed.

Health

health shows if the guest agent of a running sandbox answers. It is null when the sandbox is not running. A sandbox with health: degraded is billed, because a program in the sandbox can block the guest agent. A sandbox with health: unreachable is not billed. When the guest agent answers again, health is ok. The guest agent listens on port 7911. A firewall in the sandbox that blocks this port makes the sandbox degraded. The sandbox is still billed, and a stop cannot save its files.