Troubleshooting
Symptom, cause, fix
| Symptom | Cause | Fix |
|---|---|---|
Connection refused on SSH | You used port 22, or the instance is not running yet | Copy the command from the Connect tab; wait for the running badge |
Connection refused on a service you started | The port was never published at launch | Forward it over SSH, or declare it on the template and redeploy |
Permission denied (publickey) | No key was attached at deploy, or your client offered a different key | Redeploy with a key attached; or point at the right key with -i and IdentitiesOnly=yes |
REMOTE HOST IDENTIFICATION HAS CHANGED | A recycled host and port now belongs to a new instance | ssh-keygen -R "[host]:port", then connect again |
Instance sits in creating | A large image is still being pulled | Watch the Logs tab; if it lands in error, destroy and redeploy |
Instance is in error | The workload failed to start, failed or was lost while running, or finished on its own. What went wrong on the instance page says which, as does the API's error_code — see Why an instance failed | Read the logs, destroy it, deploy again — it cannot be started or stopped |
JupyterLab returns 403 Forbidden | The request carried no token, or a stale one | Return to the console and use Open JupyterLab again |
A template's web UI or API answers 401 | The port requires the instance access token, and the request carried neither the token nor the cookie the Open button sets | Open it with the Open button, or send Authorization: Bearer <access token> from scripts. See The instance access token |
| Browser warns the certificate is untrusted | On the direct address, JupyterLab serves HTTPS with a self-signed certificate | Expected; continue past the warning. See The certificate warning |
No space left on device | The disk you chose at deploy is full | Free space, or redeploy on a larger disk — disks cannot be resized |
| Instance stopped on its own | The organization's balance reached ¥0 | Add credits, then start it again |
Start returns 409 CAPACITY_UNAVAILABLE | The instance's GPUs were released when the balance ran out (an auto-stop, or your own stop released later), and those GPUs are now in use by someone else or out of service on that machine | Start it again later; the instance stays stopped with its disk intact, but nothing can reach the disk until a start succeeds. If it keeps failing, destroy it (the disk is deleted with it) and deploy again |
Instance went back to stopped after a start | The host refused the resume, is not connected, did not confirm it within 30 minutes, or its released GPUs are in use by someone else or out of service; the console says which | Start it again later. If START_REFUSED keeps coming back, destroy the instance and deploy again |
Instance went back to running after a stop | The host did not confirm the stop within 30 minutes; the container is still up and billing. The instance page shows The last action did not complete, and the API's error_code reads stop_timed_out | Nothing — the stop is retried on its own. Destroy the instance if you do not need it |
A stopped instance turned into error with error_code workload_gone | Its container disappeared from the machine while it was stopped | There is nothing left to start: destroy it and deploy again. It no longer bills |
| Instance disappeared entirely | The balance stayed at or below ¥0 for 24 hours and reached −¥35, and stopped instances were destroyed | The disk is gone; add credits and deploy again |
409 INVALID_STATE from the API | The instance is mid-transition | Wait for the transitional state to land, then retry |
402 INSUFFICIENT_BALANCE on deploy or start | The balance does not cover one hour of the instance's GPU and disk plus an hour of the organization's other instances; a positive balance can still fall short | An admin adds credits; members cannot |
409 INSTANCE_LIMIT_REACHED on deploy | The organization already has 25 instances that are not destroyed — stopped and errored ones count | Destroy instances you no longer need, including any in error |
Connection refused
Two different problems wear the same message.
If it is SSH itself, you are almost certainly on the wrong port. The instance's sshd is published on a host port assigned at launch, never 22, and the number can change after a stop and start. Copy the command fresh from the Connect tab.
If it is a service you started inside the instance, the port is not published. Ports are fixed when the instance launches, so starting a server afterwards exposes nothing. Forward it through the SSH session you already have:
ssh -N -L 8000:localhost:8000 -p 40123 root@203.0.113.14
Also confirm the process is bound to 0.0.0.0 and not 127.0.0.1 — unless the port is behind the access proxy, where the app belongs on 127.0.0.1. See Ports and the Open button.
Permission denied (publickey)
Password authentication is disabled, so this always means the key your client offered is not the one in authorized_keys.
ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes -p 40123 root@203.0.113.14
If that still fails, the instance most likely has no key at all — the SSH key selector on the Deploy page was left on No key (web terminal only). There is no way to attach a key to a running instance. Add the key under SSH Keys, then deploy again. See Add a key.
Stuck in creating
creating normally completes in well under a minute. What extends it is the image pull: a multi-gigabyte image on a host that has never seen it takes as long as it takes. The Logs tab shows the pull line, then image ready, creating container.
If the workload never comes up, the instance moves to error on its own. It will not sit in creating forever.
An instance in error
error means the workload is not running and the platform is not holding it for you. Most often the container failed to start, and the logs are the only place the reason lives, so read them before you clean up:
- An image or tag that does not exist, or a private image whose registry credentials are wrong.
- A template pinned to an image that does not match the launch mode. The
sshandjupytermodes need an image built from the Superheat base image. - A workload that never became ready inside the platform's start window.
An args-mode command that exits on its own also lands the instance in error, with the API's error_code reading workload_completed: the job is done, its GPU is released and it no longer bills.
A failing onstart is not one of the causes. The base image runs it in a subshell, writes the failure to /var/log/superheat/onstart.log, and leaves the container up so you can connect and fix it.
From error the only action available is Destroy. Fix the template, then deploy again. See Read the logs.
JupyterLab returns 403
The notebook server accepts one credential, a token minted by the platform at deploy. A 403 means the browser sent the wrong one or none at all, which usually means the tab came from a bookmark or a copied address rather than from the console.
Go back to the instance in the console and press Open JupyterLab. That button carries the token the platform minted for the instance; a hand-built address does not. See Open JupyterLab.
Out of disk
df -h /
du -sh /workspace/* | sort -h | tail
The usual culprits are accumulated checkpoints and package caches:
rm -rf ~/.cache/pip ~/.cache/huggingface
The disk cannot be grown. If the job genuinely needs more room, copy what you have off the instance, destroy it, and deploy again with a larger disk_gb. See Disk and storage.
The instance stopped by itself
Running instances are stopped automatically when the organization's balance reaches ¥0. Nothing is lost — the disk is kept, and the instance starts again once the balance covers an hour of it and of the organization's other instances. Storage charges continue against the empty balance while it sits stopped, so it keeps sinking.
Where the machine supports it, that stop also releases the GPUs back to the marketplace. A later start has to take back the same GPUs; if they are in use by someone else or out of service, the start fails — with 409 CAPACITY_UNAVAILABLE, or by going back to stopped with The last action did not complete on the instance page — and the instance stays stopped with its disk intact. Try again later. See After the GPUs are released.
If the balance stays at or below ¥0 for 24 hours and reaches −¥35, stopped instances are destroyed and their disks go with them. Check the balance and the ledger on the billing page, and see Auto-stop and auto-destroy.
Below ¥0 your data is still there — for at least 24 hours, and until the balance reaches −¥35. Once both have passed, stopped instances are destroyed, and there is no recovery.