Make my agent handle host failures and crashes.
When the host stops
The host of each sandbox reports to the platform every 2 seconds. When a host does not report for 30 seconds:- Each
runningorstoppingsandbox on the host moves toerror, witherrorReasonset tohost_lost. Billing stops. You get thesandbox.errorevent. - A sandbox that was
creatingorstartingfor a create moves tofailed. Its create operation fails with the codehost_lost. - A
stoppedsandbox staysstopped. Its files were saved at the stop.
Automatic recovery
If the latest snapshot of arunning sandbox is less than 2 minutes old, the sandbox moves from error to starting at once, with a recover operation. The sandbox starts on another host, and gets back its saved files, as a resume from the saved files does.
- When the files are back, the sandbox is
runningand the operation issucceeded. You get thesandbox.runningevent. - The processes and the memory of the sandbox do not come back. The files that changed after the latest snapshot do not come back.
- The sandbox gets the IPv6 address of its new host.
- A sandbox with
pinImage: truestarts on its pinned base image if a host with room has it. Else it starts on the latest base image. - If no other host has room for 60 seconds, the sandbox stays in
error, and the operation fails with the codeno_capacity. - If the files do not come back, the sandbox is
errorwitherrorReasonset torestore_failed.
snapshotVerifiedAt shows the time of the latest snapshot. See Saved files.
When the host comes back
- A sandbox that moved to another host does not run again on the old host. The old copy is deleted.
- A sandbox that stayed in
errorwithhost_lostruns again on the old host with its local disk. It moves tostarting, then torunning. You get thesandbox.runningevent.
error with PATCH /v1/sandboxes/{id} and "state": "running". It then starts from its saved files on any host with room.
When the kernel crashes
When the kernel of a sandbox crashes, the sandbox starts again on the same host with its local disk, within 15 seconds.- The sandbox stays
running. Its files stay. Its processes and its memory do not. lastRebootshows{ "reason": "crash", "at": "<time>" }.- If the sandbox does not start again, it moves to
error, witherrorReasonset toboot_failed.
Health
health shows if the guest agent of a running sandbox answers. It is null when the sandbox is not running.
A sandbox with
health: degraded is billed, because a program in the sandbox can block the guest agent. A sandbox with health: unreachable is not billed. When the guest agent answers again, health is ok.
The guest agent listens on port 7911. A firewall in the sandbox that blocks this port makes the sandbox
degraded. The sandbox is still billed, and a stop cannot save its files.