Security Verifiers Development

Testing, linting, formatting, and development workflow for the Security Verifiers codebase.

Quick Start

For a scoped review with dependencies already installed:

make review-check REVIEW_PATHS="<relevant source paths>" REVIEW_TESTS="<relevant tests>"

This uses non-fixing checks and selected tests without dependency sync. Install dependencies only when missing and needed for the task. Full setup/check below is for an intentionally broad workflow; make check and make lint can run fixing hooks over unrelated files.

# Full setup (creates venv, installs all deps)
make setup
source .venv/bin/activate

# Full run (lint + format + test); fixing hooks may modify unrelated files
make check

Testing

Run All Tests

make test

Test Specific Environment

make test-env E=network-logs
make test-env E=config-verification
make test-env E=code-vulnerability
make test-env E=phishing-detection
make test-env E=redteam-attack
make test-env E=redteam-defense

Environment Shortcuts

make e1  # test network-logs
make e2  # test config-verification
make e3  # test code-vulnerability
make e4  # test phishing-detection
make e5  # test redteam-attack
make e6  # test redteam-defense

Test Utils Package

make test-utils

Test with Coverage

make test-cov

Run Single Test

uv run pytest environments/sv-env-network-logs/sv_env_network_logs_test.py::TestNetworkLogParser::test_extracts_label_and_confidence -q

Linting and Formatting

Full lint workflow (may fix files)

make lint

Fix Linting Issues

make lint-fix

Format Code

make format

Quick Fix (lint + format)

make quick-fix

Pre-commit Hooks

# Install and run hooks
make pre-commit

This runs:

• Ruff linting
• Ruff formatting
• Trailing whitespace removal
• End-of-file fixes

CI Checks

Run the same checks as CI:

make ci

This runs:

• ruff check . --exit-non-zero-on-fix
• pytest -q --tb=short

Environment Setup

Full Setup

make setup

Creates .venv, installs all environments and dev tools.

Manual Steps

make venv           # Create virtual environment
make install        # Install all environments
make install-dev    # Install dev tools (pytest, ruff, etc.)

Install Security Tools (Ubuntu)

For E2 config-verification, install pinned tool versions:

make install-linux  # Installs kube-linter, opa, semgrep
make check-tools    # Verify versions match ci/versions.txt

Project Structure

security-verifiers/
├── environments/          # Environment packages
│   ├── sv-env-network-logs/
│   ├── sv-env-config-verification/
│   └── ...
├── sv_shared/             # Shared utilities (security-verifiers-utils)
├── scripts/               # Build, eval, data scripts
├── outputs/               # Evaluation outputs
└── skills/                # Agent skills

Common Workflows

Before Committing

Use scoped review-check for touched behavior. Use make check or the applicable CI gate when broad validation is required, understanding that fixing hooks may change files. Inspect the diff before staging; do not include unrelated formatting.

After Changing Environment Code

make test-env E=network-logs  # test specific env
make lint                      # check linting

After Changing Shared Code

make test-utils
make test  # run all tests to check for regressions

Cleaning Up

make clean           # Build artifacts and caches
make clean-outputs   # Eval outputs (preserves logs)
make clean-logs      # Log files only
make clean-all       # Everything including venv

File Watch (requires entr)

# Install: brew install entr (macOS) or apt install entr (Ubuntu)
make watch

Automatically runs tests when Python files change.

Environment Info

make info       # Show environment status
make list-envs  # List environment names

Troubleshooting

venv issues: make clean-all && make setup Import errors: Ensure source .venv/bin/activate Tool version mismatch: make check-tools then make install-linux Pre-commit fails: make lint-fix && make format