RAM Consumption Verification

[Created by Claude: ab7e3fa0-6c2e-49c0-a55f-21779f70b64a 2026-01-25]

Purpose

Verify that a process's self-reported RAM consumption is accurate by:

1. Measuring total system RAM with htop
2. Killing the target process(es)
3. Measuring total system RAM again
4. Comparing the difference

Prerequisites

• htop installed
• tmux installed
• Agent must know their suffix(5) (last 5 chars of Agent ID)

---

⚠️ CRITICAL: Always Clean Up

You MUST clean up the tmux session after the test completes.

The tmux session is ONLY for running htop temporarily. Failure to clean up leaves orphaned sessions.

---

Step-by-Step Procedure

Step 1: Create tmux session with htop

Use your suffix(5) to name the session:

tmux new-session -d -s ram-test-{suffix5} && \
tmux send-keys -t ram-test-{suffix5} 'htop' Enter

Example for agent with suffix 0b64a:

tmux new-session -d -s ram-test-0b64a && \
tmux send-keys -t ram-test-0b64a 'htop' Enter

Step 2: Wait and capture BEFORE measurement

sleep 2 && tmux capture-pane -t ram-test-{suffix5} -p | grep "^  Mem"

Record this value (e.g., 53.8G/128G)

Step 3: Kill target process(es)

Use pkill -9 for forceful termination:

pkill -9 -f "{process-pattern}"

Examples:

# Kill ingestors
pkill -9 -f "ingest-to-centralized-db"

# Kill a specific python script
pkill -9 -f "my_script.py"

# Kill by PID
kill -9 {pid1} {pid2}

Step 4: Wait and capture AFTER measurement

sleep 1 && tmux capture-pane -t ram-test-{suffix5} -p | grep "^  Mem"

Record this value (e.g., 43.2G/128G)

Step 5: Verify processes are dead

pgrep -f "{process-pattern}" || echo "Processes confirmed dead"

Step 6: Calculate and report

RAM Before: 53.8 GB
RAM After:  43.2 GB
Difference: 10.6 GB (this is the actual RAM used by killed processes)

Step 7: ⚠️ CLEANUP (MANDATORY)

Always run this step, even if earlier steps failed:

tmux kill-session -t ram-test-{suffix5}

---

Complete Example Script

For an agent with suffix e274f verifying ingestor RAM:

# Step 1: Create session
tmux new-session -d -s ram-test-e274f && \
tmux send-keys -t ram-test-e274f 'htop' Enter

# Step 2: Measure BEFORE
sleep 2 && tmux capture-pane -t ram-test-e274f -p | grep "^  Mem"
# Output: Mem[||||||||||||||||||||||||||||||||||||53.8G/128G]

# Step 3: Kill processes
pkill -9 -f "ingest-to-centralized-db"

# Step 4: Measure AFTER
sleep 1 && tmux capture-pane -t ram-test-e274f -p | grep "^  Mem"
# Output: Mem[||||||||||||||||||||||||||||||||||||43.2G/128G]

# Step 5: Verify dead
pgrep -f "ingest-to-centralized-db" || echo "Ingestors confirmed dead"

# Step 6: Report
echo "RAM dropped from 53.8GB to 43.2GB = 10.6GB used by ingestors"

# Step 7: CLEANUP
tmux kill-session -t ram-test-e274f

---

Interpreting Results

Scenario	RAM Change	Interpretation
Large drop (GBs)	-10 GB	Process was using significant RAM
Small drop (MBs)	-50 MB	Process was using minimal RAM
No change / slight increase	±0.5 GB	Process RAM was negligible; fluctuation is noise

---

Why htop Instead of ps/top?

• ps aux RSS values can be misleading (shared memory, copy-on-write)
• htop shows real-time system-wide memory that the kernel reports
• Measuring before/after gives ground truth of actual memory freed

---

Troubleshooting

htop not showing in capture

# Increase sleep time
sleep 3 && tmux capture-pane -t ram-test-{suffix5} -p | grep "^  Mem"

Session already exists

# Kill existing and recreate
tmux kill-session -t ram-test-{suffix5} 2>/dev/null
tmux new-session -d -s ram-test-{suffix5}

Forgot to cleanup

# List all ram-test sessions
tmux ls | grep ram-test

# Kill all ram-test sessions
tmux ls | grep ram-test | cut -d: -f1 | xargs -I{} tmux kill-session -t {}