Discover, use, or subclass existing Dagster integration components (dbt, Looker, PowerBI, Fivetran, etc.)...
This skill helps you discover and work with existing Dagster integration components from the 70+ available integrations. It guides you through finding the right component, using it directly, or subclassing it to add custom functionality.
Key distinction: This skill differentiates between configuration-file-based components (like dbt and Sling) that read from local files, and API-based components (like Fivetran, PowerBI, Looker) that call external services. Configuration-based components should be used directly without demo_mode, while API-based components benefit from demo_mode implementation.
The documentation for subclassing components can be found here: https://docs.dagster.io/guides/build/components/creating-new-components/subclassing-components
When invoked, this skill will:
uv run dg docs integrations --json or browsing https://docs.dagster.io/integrations/librariesexecute() methodget_additional_scope()uv run dg check defsuv run dg list defsBefore running this skill, ensure:
uv is installed (check with uv --version)If you discover the integration package exists but has NO Component class:
uv run python -c "import dagster_<integration>; print([x for x in dir(dagster_<integration>) if 'Component' in x])"[], the integration doesn't have a Component classcreate-custom-dagster-component skill insteadThis skill is ONLY for integrations that have existing Component classes to subclass.
Help the user find the right integration component:
Run discovery command:
uv run dg docs integrations --json
Alternative: Browse https://docs.dagster.io/integrations/libraries for visual list
Common integrations include:
dagster_dbt.DbtProjectComponent - dbt projectsdagster_fivetran.FivetranComponent - Fivetran syncsdagster_sling.SlingReplicationCollectionComponent - Sling replicationsdagster_powerbi.PowerBIWorkspaceComponent - PowerBI workspacesdagster_looker.LookerComponent - Looker instancesdagster_airbyte.AirbyteComponent - Airbyte connectionsdagster_databricks.DatabricksComponent - Databricks workflowsdagster_snowflake.SnowflakeComponent - Snowflake resourcesIMPORTANT: First determine if this is a configuration-file-based or API-based component:
These components read from local configuration files and do NOT require external API credentials:
DbtProjectComponent - Reads from dbt project files (dbt_project.yml, SQL/Python models)SlingReplicationCollectionComponent - Reads from replication YAML filesFor configuration-file-based components:
These components call out to external services and require API credentials:
FivetranComponent - Calls Fivetran APIPowerBIWorkspaceComponent - Calls PowerBI APILookerComponent - Calls Looker APIAirbyteComponent - Calls Airbyte APICensusComponent - Calls Census APIFor API-based components:
Based on the component type, choose:
If using directly, skip to Step 7 to create the component instance YAML.
Install the required Dagster integration package:
uv add dagster-<integration-name>
Examples:
uv add dagster-dbtuv add dagster-slinguv add dagster-powerbiUse dg scaffold defs to create the component instance directory:
uv run dg scaffold defs <package>.<ComponentClass> <instance_name>
Example:
uv run dg scaffold defs dagster_sling.SlingReplicationCollectionComponent my_sling_sync
This creates a directory structure like:
defs/
my_sling_sync/
defs.yaml
# Other config files as needed
Create a component.py file in the component instance directory.
IMPORTANT: Only add demo_mode for API-based components!
When extending integration components, you must understand whether they use dataclass or Pydantic BaseModel patterns, as this determines how to add custom fields like demo_mode.
First, check the parent component's implementation:
# Check if it's a dataclass
uv run python -c "from <package> import <Component>; import dataclasses; print(dataclasses.is_dataclass(<Component>))"
Most Dagster integration components (like SlingReplicationCollectionComponent, DbtProjectComponent, FivetranComponent) use dataclass with the Resolvable interface.
This is the most common pattern for Dagster integration components:
from dataclasses import dataclass
import dagster as dg
from <integration_package> import <BaseComponentClass>
@dataclass
class Custom<ComponentName>(BaseComponentClass):
"""Customized component with demo mode support."""
# New field - will automatically appear in YAML schema via Resolvable
demo_mode: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
"""Build definitions, using demo mode if enabled.
Note: The parent class fields (like API credentials) are still set from YAML,
but when demo_mode is True, we bypass the parent's build_defs() method
and return mocked assets instead, so those credentials are never used.
"""
if self.demo_mode:
# Return mock assets for demo mode - parent credentials are ignored
return self._build_demo_defs(context)
else:
# Use real integration with actual credentials from parent fields
return super().build_defs(context)
def _build_demo_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
"""Build demo mode definitions with mocked assets."""
@dg.asset(
key=dg.AssetKey(["mock_asset"]),
kinds={"integration_name"}, # IMPORTANT: Add the integration kind
)
def mock_asset(context: dg.AssetExecutionContext):
context.log.info("Demo mode: simulating asset execution")
return {"status": "demo_mode"}
return dg.Definitions(assets=[mock_asset])
Key points for dataclass components:
@dataclass decorator on your subclassfield(default_factory=...) for mutable defaults (lists, dicts)Resolvable interface (inherited from parent) handles YAML schema generationExample with multiple custom fields:
from dataclasses import dataclass, field
import dagster as dg
from dagster_sling import SlingReplicationCollectionComponent
@dataclass
class CustomSlingComponent(SlingReplicationCollectionComponent):
"""Extended Sling component with additional configuration."""
# New fields - all will appear in YAML schema
demo_mode: bool = False
enable_notifications: bool = False
notification_channel: str = "slack"
custom_tags: list[str] = field(default_factory=list)
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
if self.demo_mode:
return self._build_demo_defs(context)
return super().build_defs(context)
Some components may use Pydantic BaseModel. In these cases, inherit from both the parent component and dg.Model:
import dagster as dg
from <integration_package> import <BaseComponentClass>
class Custom<ComponentName>(BaseComponentClass, dg.Model):
"""Customized component with demo mode support."""
# New field - will appear in YAML schema
demo_mode: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
if self.demo_mode:
return self._build_demo_defs(context)
return super().build_defs(context)
For older components that override execute() instead of build_defs():
from dataclasses import dataclass
from dagster import AssetExecutionContext
from <integration_package> import <BaseComponentClass>
from collections.abc import Iterator
from typing import Any
@dataclass
class Custom<ComponentName>(BaseComponentClass):
"""Customized component with demo mode support."""
demo_mode: bool = False
def execute(
self,
context: AssetExecutionContext,
**kwargs: Any,
) -> Iterator:
"""Custom execution logic with demo mode support."""
if self.demo_mode:
context.log.info("Running in demo mode with mocked data")
yield from self._execute_demo_mode(context, **kwargs)
else:
context.log.info("Running with real integration")
yield from super().execute(context, **kwargs)
def _execute_demo_mode(
self,
context: AssetExecutionContext,
**kwargs: Any,
) -> Iterator:
"""Demo mode implementation."""
from dagster import Output
context.log.info("Simulating integration execution locally")
yield Output(value=None, output_name="result")
Key customization points:
build_defs() or execute() - Check demo_mode and return mock dataget_additional_scope() for YAML templatingExample with custom templating:
from dataclasses import dataclass
from collections.abc import Mapping
from typing import Any
import dagster as dg
from <integration_package> import <BaseComponentClass>
@dataclass
class Custom<ComponentName>(BaseComponentClass):
demo_mode: bool = False
@classmethod
def get_additional_scope(cls) -> Mapping[str, Any]:
"""Add custom YAML templating functions."""
def _custom_cron(cron_schedule: str) -> dg.AutomationCondition:
return (
dg.AutomationCondition.on_cron(cron_schedule)
& ~dg.AutomationCondition.in_progress()
)
return {"custom_cron": _custom_cron}
When to do this: If other Dagster components in your pipeline will depend on assets from this component, override get_asset_spec() to generate asset keys that match downstream expectations.
This applies when:
depsBy default, integration components generate asset keys in their own structure. For example:
["fivetran", "raw", "customers"]["sling", "replications", "sync_name", "table"]["analytics", "marts", "customer_360"]The problem: Downstream components may expect different key structures, leading to broken dependencies or requiring per-asset configuration with meta.dagster.asset_key.
The solution: Override get_asset_spec() in the upstream component to generate keys that downstream components naturally reference.
def get_asset_spec(self, props) -> dg.AssetSpec:
"""Override to generate asset keys matching downstream component expectations.
This eliminates the need for meta.dagster.asset_key configuration in downstream
components by aligning keys at the source.
"""
base_spec = super().get_asset_spec(props)
original_key = base_spec.key.path
# Customize key structure for your pipeline
# Example: Flatten nested keys for easier consumption
custom_key = dg.AssetKey([...]) # Your key transformation logic
return base_spec.replace_attributes(key=custom_key)
Problem: Fivetran creates ["fivetran", "raw", "customers"], but dbt expects ["fivetran_raw", "customers"]
Solution:
from dagster_fivetran import FivetranAccountComponent
from dagster_fivetran.translator import FivetranConnectorTableProps
import dagster as dg
class CustomFivetranComponent(FivetranAccountComponent):
def get_asset_spec(self, props: FivetranConnectorTableProps) -> dg.AssetSpec:
"""Flatten asset keys for dbt compatibility."""
base_spec = super().get_asset_spec(props)
original_key = base_spec.key.path
# Flatten: ["fivetran", "raw", "table"] -> ["fivetran_raw", "table"]
if len(original_key) >= 2:
flattened_key = dg.AssetKey(["fivetran_raw", "_".join(original_key[1:])])
else:
flattened_key = dg.AssetKey(["fivetran_raw", original_key[-1]])
return base_spec.replace_attributes(key=flattened_key)
Result: dbt sources work automatically without meta.dagster configuration:
# sources.yml - references work naturally now
sources:
- name: fivetran_raw
tables:
- name: customers # Matches ["fivetran_raw", "customers"]
Problem: Sling creates ["sling", "replications", "sync_name", "table"], but custom processors expect ["raw", "table"]
Solution:
from dagster_sling import SlingReplicationCollectionComponent
import dagster as dg
class CustomSlingComponent(SlingReplicationCollectionComponent):
def get_asset_spec(self, props) -> dg.AssetSpec:
"""Simplify keys for downstream consumption."""
base_spec = super().get_asset_spec(props)
original_key = base_spec.key.path
# Simplify: [..., "table"] -> ["raw", "table"]
simplified_key = dg.AssetKey(["raw", original_key[-1]])
return base_spec.replace_attributes(key=simplified_key)
Result: Custom assets reference naturally:
@dg.asset(deps=[dg.AssetKey(["raw", "customers"])])
def process_customers(context): ...
After implementing get_asset_spec(), verify dependencies are correct:
# Check asset keys and dependencies
uv run dg list defs --json | uv run python -c "
import sys, json
assets = json.load(sys.stdin)['assets']
print('\\n'.join([f\"{a['key']}: deps={a.get('deps', [])}\" for a in assets]))
"
What to look for:
deps array["fivetran", "raw", "table"] and ["fivetran_raw", "table"])get_asset_spec() override applies to all assets the component createsmeta.dagster.asset_key in every downstream referenceUpdate defs.yaml to reference your custom subclass. Your new fields (like demo_mode) will be available in the YAML schema.
type: <project_name>.defs.<instance_name>.component.Custom<ComponentName>
attributes:
# Your custom fields
demo_mode: true
# Parent class required fields
<parent_field>: <value>
Example for Sling (Configuration-based - NO demo_mode):
Sling reads from local replication YAML files, so no demo_mode is needed:
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
replications:
- path: replication.yaml
Example for dbt (Configuration-based - NO demo_mode):
dbt reads from local project files, so no demo_mode is needed:
type: dagster_dbt.DbtProjectComponent
attributes:
project:
project_dir: analytics_dbt
Example with dummy credentials for demo mode (API-based components only):
For API-based components, you MUST provide dummy values to satisfy parent class schema validation. These values remain in the YAML but are ignored when demo_mode is true:
Example for Looker:
type: my_project.defs.looker_dashboards.component.CustomLookerComponent
attributes:
demo_mode: true
# Dummy Looker credentials - required for schema validation but ignored in demo mode
looker_resource:
base_url: "https://demo.looker.com"
client_id: "demo_client_id"
client_secret: "demo_client_secret"
Example for PowerBI:
type: my_project.defs.powerbi_workspace.component.CustomPowerBIComponent
attributes:
demo_mode: true
# Dummy PowerBI credentials - required for schema validation but ignored in demo mode
powerbi_resource:
client_id: "demo_client_id"
client_secret: "demo_client_secret"
tenant_id: "demo_tenant_id"
workspace_id: "demo_workspace_id"
Example for Fivetran:
type: my_project.defs.fivetran_sync.component.CustomFivetranComponent
attributes:
demo_mode: true
# Dummy Fivetran credentials - required for schema validation but ignored in demo mode
fivetran_resource:
api_key: "demo_api_key"
api_secret: "demo_api_secret"
connector_id: "demo_connector_id"
Important Notes:
To switch to real mode:
demo_mode: falseFill in the required attributes for the integration component. Consult the component's documentation:
Common attributes include:
Run validation commands to ensure everything works:
# Check that definitions load without errors
uv run dg check defs
# List all assets to verify they were created
uv run dg list defs
Verify that:
CRITICAL: Verify Asset Key Alignment
Check that asset dependencies are correct by running:
uv run dg list defs --json | uv run python -c "
import sys, json
data = json.load(sys.stdin)
assets = data.get('assets', [])
print('Asset Dependencies:\n')
for asset in assets:
key = asset.get('key', 'unknown')
deps = asset.get('deps', [])
if deps:
print(f'{key}')
for dep in deps:
print(f' ← {dep}')
else:
print(f'{key} (no dependencies)')
print()
"
What to verify:
deps array["category", "name"])Common Issues and Fixes:
dbt models not depending on upstream sources:
{{ source('source_name', 'table') }}select * from {{ source('source_name', 'table') }} references in SQL (even if just in a CTE that's not used for demo mode)Asset keys don't match downstream expectations:
["fivetran", "connector", "schema", "table"] but dbt expects ["fivetran_raw", "table"]get_asset_spec() to flatten keys (see Step 5.5 for details)Reverse ETL components can't find dbt models:
deps=[AssetKey(["model_name"])] explicitlyIf you implemented demo mode:
demo_mode: true in the component YAMLuv run dg check defs to verify it works locallyTesting commands:
# Check definitions load
uv run dg check defs
# List available assets
uv run dg list defs
# Materialize a specific asset in demo mode
uv run dg materialize <asset_key>
When creating demo mode assets, ALWAYS add the kinds parameter to properly categorize the asset by its integration type. This helps with:
Example with kinds:
@dg.asset(
key=dg.AssetKey(["fivetran", "salesforce_sync"]),
description="Demo Fivetran sync of Salesforce data",
kinds={"fivetran"}, # ← REQUIRED: Add the integration kind
)
def fivetran_salesforce_sync(context: dg.AssetExecutionContext):
context.log.info("Demo mode: Simulating Fivetran sync")
return {"status": "success", "mode": "demo"}
Common integration kinds:
kinds={"fivetran"} for Fivetran assetskinds={"dbt"} for dbt assetskinds={"census"} for Census assetskinds={"sling"} for Sling assetskinds={"powerbi"} for PowerBI assetskinds={"looker"} for Looker assetskinds={"airbyte"} for Airbyte assetsYou can verify kinds are showing correctly by running:
uv run dg list defs
The "Kinds" column should show the integration type for each asset.
IMPORTANT: DbtProjectComponent is configuration-file-based and reads from local dbt project files. DO NOT implement demo_mode - instead, just load the dbt project directly.
Use the component directly:
# defs.yaml for dbt component
type: dagster_dbt.DbtProjectComponent
attributes:
project:
project_dir: my_dbt_project # Path to local dbt project
Only subclass if you need custom behavior (NOT for demo mode):
from dataclasses import dataclass
from dagster_dbt import DbtProjectComponent
import dagster as dg
@dataclass
class CustomDbtComponent(DbtProjectComponent):
"""Custom dbt component with additional features."""
# Add custom fields if needed (NOT demo_mode)
enable_custom_logging: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
# Add custom behavior here
defs = super().build_defs(context)
if self.enable_custom_logging:
# Custom logging logic
pass
return defs
The dbt component naturally works with local project files without requiring external API calls.
IMPORTANT: SlingReplicationCollectionComponent is configuration-file-based and reads from local replication YAML files. DO NOT implement demo_mode - instead, just load the replication configuration.
Use the component directly:
# defs.yaml for Sling component
type: dagster_sling.SlingReplicationCollectionComponent
attributes:
replications:
- path: replication.yaml # Path to local replication config
Only subclass if you need custom behavior (NOT for demo mode):
from dataclasses import dataclass
from dagster_sling import SlingReplicationCollectionComponent
import dagster as dg
@dataclass
class CustomSlingComponent(SlingReplicationCollectionComponent):
"""Custom Sling component with additional features."""
# Add custom fields if needed (NOT demo_mode)
enable_custom_validation: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
# Add custom behavior here
defs = super().build_defs(context)
if self.enable_custom_validation:
# Custom validation logic
pass
return defs
The Sling component naturally works with local replication files without requiring external API calls.
Fivetran is API-based and requires credentials. Implement demo_mode to mock API calls:
from dataclasses import dataclass
from dagster_fivetran import FivetranComponent
import dagster as dg
@dataclass
class CustomFivetranComponent(FivetranComponent):
"""Fivetran component with demo mode support."""
demo_mode: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
if self.demo_mode:
return self._build_demo_defs(context)
return super().build_defs(context)
def _build_demo_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
@dg.asset(
key=dg.AssetKey(["fivetran", "demo_sync"]),
kinds={"fivetran"},
)
def fivetran_demo_sync(context: dg.AssetExecutionContext):
context.log.info("Demo mode: Simulating Fivetran sync")
return {"status": "demo_success", "records_synced": 1000}
return dg.Definitions(assets=[fivetran_demo_sync])
PowerBI is API-based and requires credentials. Implement demo_mode to mock API calls:
from dataclasses import dataclass
from dagster_powerbi import PowerBIWorkspaceComponent
import dagster as dg
@dataclass
class CustomPowerBIComponent(PowerBIWorkspaceComponent):
"""PowerBI component with demo mode support."""
demo_mode: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
if self.demo_mode:
return self._build_demo_defs(context)
return super().build_defs(context)
def _build_demo_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
@dg.asset(
key=dg.AssetKey(["powerbi", "demo_dashboard"]),
kinds={"powerbi"},
)
def powerbi_demo_dashboard(context: dg.AssetExecutionContext):
context.log.info("Demo mode: Using cached PowerBI metadata")
return {"dashboards": ["Sales Dashboard", "Marketing Dashboard"]}
return dg.Definitions(assets=[powerbi_demo_dashboard])
The component is complete when:
execute() method (if applicable)uv run dg check defs passes without errorsuv run dg list defs shows all expected assets from the integrationProblem: Schema validation errors for missing required fields
Cause: The parent component requires certain fields, but you didn't provide them in YAML.
Solution: Provide dummy values for required fields. Keep them UNCOMMENTED - they're needed for schema validation but will be ignored in demo mode:
attributes:
demo_mode: true
# Dummy credentials - MUST be present and uncommented for schema validation
looker_resource:
base_url: "https://demo.looker.com"
client_id: "demo_client_id"
client_secret: "demo_client_secret"
If you see import errors, ensure:
uv add dagster-<integration>from dagster_<name> import ...If the component can't be found:
defs.yaml matches your Python pathcomponent.py is in the correct directoryIf dg check defs fails:
model_config = ConfigDict(extra="allow") is set in your Python classAfter completion, inform the user:
uv run dg docs integrations --json