Instance lifecycle
An instance is always in exactly one of eight states. Which state it is in decides three things you care about: whether you can connect to it, whether it is billing you, and which actions are available.
The state machine
The normal path is creating → running → stopping → stopped → starting → running. Teardown is always destroying → destroyed, and you can begin it from any state except destroyed itself — including while the instance is still creating, starting or stopping. The dotted arrows are the platform putting an instance back where it was when the host did not do what was asked; they are covered below.
Asking for a move the current state does not allow — starting an instance that is already running, stopping one that is still creating — is rejected, and the instance stays where it was.
Every state
| State | Transitional | Billing | What it means |
|---|---|---|---|
creating | yes | none | The slice has been reserved and your workload is being brought up for the first time. |
starting | yes | none | A stopped instance is coming back on the same machine, with the same disk. |
running | no | GPU, per second | The workload is up. SSH, JupyterLab and your published ports are reachable. |
stopping | yes | none | The container is being shut down. Your disk is being kept. |
stopped | no | disk, per GB per hour | Nothing is executing. The disk is kept; the GPU slot is still held for you, unless the platform released the GPUs when the balance ran out. |
destroying | yes | none | The container and the disk are being torn down. |
destroyed | no | none | Final. The disk is gone and the offer is back on the marketplace. |
error | no | none | The workload could not be brought up, failed, lost its container while stopped, or finished on its own. See below. |
Transitional states are the four the platform is actively working through; the console shows them with a pulsing badge and refreshes more often while one is in flight. They resolve on their own — there is nothing to retry, though you can destroy the instance instead of waiting.
Non-transitional states are stable. An instance sits in running, stopped, destroyed or error until you, the billing system, or the host's own report moves it.
What causes each transition
| Transition | Trigger |
|---|---|
→ creating | You deploy: POST /v1/instances with an offer, a template and a disk size. |
creating/starting → running | The workload reported itself up. The GPU meter starts here, not at deploy. |
creating → error | The workload could not be brought up. |
running → stopping | You stop it, or your balance reaches ¥0. That stop for an empty balance releases the GPUs where the machine supports it. |
stopping → stopped | The shutdown finished. The disk meter starts here. |
stopping → running | The host did not confirm the stop within 30 minutes. The container is still up, so the GPU meter goes back on and the stop keeps being retried. Its error_code reads stop_timed_out. |
stopped → starting | You start it. Requires a balance that covers one hour of everything the organization runs, this instance included. |
starting → stopped | The host refused the resume or could not be reached, or did not confirm it within 30 minutes (start_timed_out); for an instance whose GPUs were released, those GPUs may also be in use by someone else or out of service on that machine (gpus_unavailable). The instance is back to stopped, with its disk, and you can start it again. |
running → error | The workload failed, the host lost it, or a job ran to completion on its own (workload_completed). The GPU is released. |
stopped (and an instance stopping or starting) → error | Its container disappeared from the machine while the instance was stopped (workload_gone). There is nothing left to start, and storage is no longer charged. |
error → running or stopped | The platform found the workload healthy after all — running, or held with its disk — and put the instance back in that state. |
any state but destroyed → destroying | You destroy it. Auto-destroy does the same to stopped and errored instances once the balance has stayed down long enough. |
destroying → destroyed | Teardown finished. The disk is deleted and the offer is released. |
Two of those triggers are not yours. When the organization's balance hits zero, running instances are stopped automatically; where the machine supports it, their GPUs and those still held by stopped instances are released, and the disks are kept. If the balance then stays at or below zero for 24 hours and is at or below −¥35, stopped and errored instances are destroyed. Running instances are never destroyed directly — they are stopped first. Both rules are covered in Auto-stop and auto-destroy.
Deploying and starting both require the balance to cover one hour of the organization's whole burn — the GPU and disk of every instance that is running, starting or being created, the disk of every stopped one, and the GPU and disk of the instance you are deploying or starting — not merely to be above zero; short of that the request is refused with 402 INSUFFICIENT_BALANCE. Stopping and destroying never do — you can always shut something down, whatever your balance is.
Billing follows the state
Charges are settled at the rate of the state the instance was actually in, and every state change settles what was owed before the change takes effect. The GPU meter runs only in running. The disk meter runs only in stopped. The four transitional states, error and destroyed are free.
This is why a deployment that takes two minutes to come up is not billed for those two minutes, and why a stopped instance still produces a charge every minute in your ledger.
What "error" means
error means the workload is not running and the platform is not holding it for you: the image failed to pull, the container refused to start, the host stopped reporting during launch, the workload failed or was lost while it ran, its container disappeared while the instance was stopped, or a job ran to completion on its own. The instance page says which under What went wrong, and the API reports it as the instance's error_code and error_message. See Why an instance failed.
Four things follow from that:
- You are not billed. The meter is off in
error, and nothing accrues while the instance sits there. - It is not retried. There is no automatic relaunch. The one way out other than destroying is a recovery: if the platform later finds the workload running, or held with its disk, it moves the instance back to
runningorstopped. - It can be cleaned up for you. An instance that errored before it ever ran, or whose workload is known to have ended (
workload_completedandworkload_goneincluded), is destroyed automatically about an hour after it enterederror, disk included. An instance that had run and whose workload may still be held is not — it waits for the platform to find out what happened to it, or for you. - Destroy is the only action you can take. You cannot start, stop or repair an errored instance. Destroy it, fix the cause, and deploy again.
The usual causes of a launch failure are in the template rather than the hardware: an image name or tag that does not exist, or a private registry whose credentials are wrong. An ssh or jupyter template whose image is not built on the Superheat base image cannot come up either, because those modes depend on the base image's entrypoint.
A job that ran to completion — an args-mode template whose command exits — lands in error with error_code workload_completed. Its GPU is released straight away and it is not billed; destroy it once you have what you need from it.
An instance that never reached running has no logs to read, so debug from the template rather than from the instance. Troubleshooting works through the specific failures.
It costs nothing. Once the platform has seen its workload end — failed, lost or finished — its GPUs go back on the market straight away. A deploy that failed before its workload ever reached the host still holds the slice until the instance is destroyed, by you or automatically about an hour later. Either way it counts toward the organization's 25-instance limit until it is destroyed.
Where you see states
The Instances page lists everything except destroyed instances. Through the API, GET /v1/instances?include_destroyed=true brings them back.
Each instance page shows the current state as a badge, only the actions the current state allows — Start on a stopped instance, Stop on a running one, Destroy on every state but destroyed — and, once the workload is up, the connect details on its Connect tab. The API returns the same list as allowed_actions on the instance.