Smithery is now a part of Arcade.dev! Read more in our announcement here
Execute Vast.ai incident response procedures with triage, mitigation, and postmortem. Use when responding to Vast.ai-related outages, investigating errors, or running post-incident reviews for...
Protect people and credentials first, then preserve recoverable data, stop uncontrolled spend, and restore service from an external checkpoint or last-known-good template. Provider states imply different actions and must not be collapsed into retry.
Confirm account/team context, affected IDs, current state, balance, and whether a credential or data exposure is involved.
Outbid/stopped may retain disk and storage charges; exited is workload failure; scheduling may await a reclaimed GPU; unknown/offline is host heartbeat loss; low credit threatens availability and data.
Copy reachable artifacts and logs, record template/image identity, and verify the newest external checkpoint before any destroy.
Stop new creates, revoke compromised keys, and use stop or destroy only under the incident's data-versus-cost decision.
Resume from the verified checkpoint on a compliant different offer, or roll Serverless back to the last-known-good template.
Confirm service and output, destroy superseded resources, audit charges, rotate temporary access, and document the gap and prevention action.
Use an incident key with the minimum temporary permissions and a short revocation deadline. Keep billing-write, team administration, workload storage, and control-plane authority separated.
Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.
Return severity, IDs, state class, last good checkpoint/template, containment, restored target, data gap, spend impact, and closed resources.
After a host goes offline, the team avoids blind restarts, restores the last external checkpoint on a different verified offer, validates output, and retains the original instance ID for support and billing reconciliation.
| Failure | Response |
|---|---|
| No external checkpoint exists | State the recovery gap explicitly and attempt data salvage only if the host becomes reachable. |
| Balance is zero or negative | Escalate immediately; instances stop and resources may be at risk of deletion. |
| Destruction would erase the only data | Require the incident commander and data owner to make the containment decision. |
| Replacement also fails | Stop churn, compare common image/data/config factors, and escalate with redacted evidence. |
