Data Import Workflow Skill
Use this skill when working on or running the data import pipeline.
Scope
- Running the importer to populate or refresh data
- Extending the importer to handle new fields or types
- Debugging import failures
Running the Importer
| Command |
Use Case |
task importer:start |
Fetch fresh data from tarkov.dev API and update the database. |
task importer:start:use-cache |
Use locally cached JSON files (faster, avoids API rate limits). |
task importer:start:cache-only |
Fetch from API and update local cache, but skip database writes. |
Import Behavior
- Purge: Existing weapons, mods, trader offers, and optimum builds are deleted.
- Fetch: Data is retrieved from tarkov.dev GraphQL API (or local cache).
- Transform: External schema is mapped to internal
models structs.
- Persist: Data is written to the database within a transaction.
- Invalidate:
optimum_builds are purged since they depend on the imported data.
Extending the Importer
When adding new fields or types:
- Update GraphQL Query: Modify
internal/tarkovdev/schemas/queries.graphql.
- Regenerate Client: Run
task tarkovdev:regenerate.
- Update Models: Add new fields to structs in
internal/models/.
- Update Mapping: Adjust conversion logic in
internal/importers/.
- Update SQL: Modify
Upsert functions in internal/models/ to include new columns.
Troubleshooting
- Rate Limiting: Use
--use-cache for repeated runs during development.
- Schema Mismatch: Run
task tarkovdev:get-schema then task tarkovdev:regenerate.
- Connection Issues: Verify
POSTGRES_HOST in .env and ensure the database is running.
