[!NOTE] This skill operates under the universal guidelines in /.agent/storage/docs/. All structures must align with database.md guidelines.

🎯 Directives

1. Immutability: Treat source data as immutable; always transform into new structures.
2. Schema Enforcement: Validate data at every step of the pipeline.
3. Storage Efficiency: Use appropriate serialization (Parquet/JSON/Avro) based on access patterns.

🛠️ Workflows

1. ETL Pipeline Development

• Trigger: pipeline_update
• Steps:
  1. Extract: Script data ingestion from sources.
  2. Transform: Apply cleaning and normalization logic.
  3. Load: Write to the target storage according to standards.

2. Storage Optimization

• Trigger: storage_audit
• Steps:
  1. Analyze: Identify large tables or slow aggregations.
  2. Execute: Implement indexing, partitioning, or compression.

📚 Knowledge Base

• Database Guidelines
• Tech Stack Rules