MikoPBX Log Analyzing

Efficiently analyze logs inside MikoPBX Docker container to diagnose issues, monitor processes, and track system behavior.

What This Skill Does

• Intelligently searches logs based on problem context
• Identifies relevant log files for each issue type
• Filters noise to show only relevant entries
• Correlates logs across multiple files
• Tracks worker processes and their status
• Provides actionable insights from log analysis
• Works with both host-mounted logs and docker exec for flexibility

Host Log Access (Faster)

For local development, logs are mounted on the host at:

/Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/

Advantages of host access:

• ✅ Faster (no docker exec overhead)
• ✅ Can use Read tool directly
• ✅ Can use Grep tool with full power
• ✅ Better for large log files

When to use host path:

• Analyzing large log files
• Complex grep patterns
• Multiple log file analysis
• Performance-critical operations

When to Use This Skill

Use this skill when:

• User reports an error or issue
• Need to diagnose API problems
• Tracking specific worker processes
• Investigating system behavior
• User asks "check logs" or "what's in the logs?"
• After tests fail (to find root cause)
• Monitoring real-time system activity

Quick Start

Step 1: Choose Access Method

Option A: Host Path (Recommended for analysis)

# Direct access to logs (faster)
HOST_LOG_PATH="/Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log"

# Use Grep tool
# Pattern: errors in system log
# Path: $HOST_LOG_PATH/system/messages

# Or use Read tool for specific log files

Option B: Docker Exec (Needed for process checks)

# Find MikoPBX container (needed for worker checks)
CONTAINER_ID=$(docker ps | grep mikopbx | awk '{print $1}' | head -1)

Quick Decision Guide:

• Log file analysis → Use host path with Grep/Read tools
• Process/worker checks → Use docker exec with Bash
• Real-time monitoring → Use docker exec with tail -f

Step 2: Determine Log Context

Based on issue type, select appropriate logs:

API Issues:

• System messages (WorkerApiCommands)
• PHP error log
• Nginx logs

Call Issues:

• Asterisk messages/error/verbose
• Security log

Database Issues:

• System messages
• Debug query logs (if enabled)

Worker Issues:

• System messages (filter by worker name)
• Process status

Step 3: Execute Analysis

Use targeted search patterns (see search-patterns.md):

# Recent errors
docker exec $CONTAINER_ID tail -500 /storage/usbdisk1/mikopbx/log/system/messages | grep -i error

# Specific worker
docker exec $CONTAINER_ID tail -300 /storage/usbdisk1/mikopbx/log/system/messages | grep WorkerApiCommands

# Real-time monitoring
docker exec $CONTAINER_ID tail -f /storage/usbdisk1/mikopbx/log/system/messages

Top 5 Common Verification Patterns

Pattern 1: Recent System Errors

When to use: First step in diagnosing any issue

Method A: Host Path (Recommended)

# Use Grep tool
# Pattern: error
# Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/system/messages
# Options: -i (case insensitive), -C 2 (2 lines context)

Method B: Docker Exec

# Get last 500 lines, filter errors
docker exec $CONTAINER_ID tail -500 /storage/usbdisk1/mikopbx/log/system/messages | grep -i error

What to look for:

• Error timestamps (correlation)
• Error frequency (isolated vs recurring)
• Component name (which part failed)
• PID (which process)

Common findings:

• PHP fatal errors
• Database constraint violations
• Worker crashes
• Memory exhaustion

---

Pattern 2: API Request Debugging

When to use: API request fails or returns unexpected result

Method A: Host Path (Recommended)

# Use Grep tool for WorkerApiCommands activity
# Pattern: WorkerApiCommands
# Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/system/messages

# Use Read tool for PHP errors (recent)
# Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/php/error.log
# Offset: calculate from file size
# Limit: 50

# Use Grep for nginx access
# Pattern: POST /pbxcore/api
# Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/nginx/access.log

Method B: Docker Exec

# Check WorkerApiCommands activity
docker exec $CONTAINER_ID tail -200 /storage/usbdisk1/mikopbx/log/system/messages | grep WorkerApiCommands

# Check PHP errors
docker exec $CONTAINER_ID tail -50 /storage/usbdisk1/mikopbx/log/php/error.log

# Check nginx access log
docker exec $CONTAINER_ID tail -100 /storage/usbdisk1/mikopbx/log/nginx/access.log | grep "POST /pbxcore/api"

What to look for:

1. Request received by nginx
2. Request queued to Redis
3. WorkerApiCommands processing
4. PHP errors during execution
5. Response status code

Expected flow:

[06:52:10] nginx: POST /pbxcore/api/v3/extensions
[06:52:11] WorkerApiCommands: Processing request
[06:52:12] WorkerApiCommands: Executing SaveRecordAction
[06:52:13] WorkerApiCommands: Response sent (200 OK)

See analysis-scenarios.md for detailed example.

---

Pattern 3: Worker Process Monitoring

When to use: Worker not responding, system seems stuck

# Check if worker is running
docker exec $CONTAINER_ID ps aux | grep WorkerApiCommands

# Check for crashes
docker exec $CONTAINER_ID tail -500 /storage/usbdisk1/mikopbx/log/system/messages | grep -E "WorkerApiCommands|terminated|crash"

# Count instances
docker exec $CONTAINER_ID ps aux | grep WorkerApiCommands | grep -v grep | wc -l

Expected: WorkerApiCommands should have 3 instances

What to look for:

• Process terminated messages
• Orphaned process warnings
• Fatal PHP errors
• Last activity before crash

Common issues:

• Worker crashed (0 instances)
• Too many workers (restart loop)
• Zombie processes

See worker-processes.md for complete worker reference.

---

Pattern 4: Database Error Investigation

When to use: Constraint violations, lock errors, data inconsistency

# Check for database errors
docker exec $CONTAINER_ID tail -200 /storage/usbdisk1/mikopbx/log/system/messages | grep -iE "database|sqlite|constraint"

# Check PHP PDO errors
docker exec $CONTAINER_ID tail -100 /storage/usbdisk1/mikopbx/log/php/error.log | grep -iE "database|pdo|sql"

Common errors:

• UNIQUE constraint failed: Extensions.number - Duplicate data
• FOREIGN KEY constraint failed - Invalid reference
• Database is locked - Concurrent access
• Database disk image is malformed - Corruption

Investigation steps:

1. Identify which query failed
2. Check input data for duplicates/invalids
3. Verify foreign key references exist
4. Check for long-running transactions

See analysis-scenarios.md for detailed troubleshooting.

---

Pattern 5: Real-Time Activity Monitoring

When to use: Need to see what's happening right now

# Follow system log
docker exec $CONTAINER_ID tail -f /storage/usbdisk1/mikopbx/log/system/messages

# Follow with error filter
docker exec $CONTAINER_ID tail -f /storage/usbdisk1/mikopbx/log/system/messages | grep -i error

# Follow specific worker
docker exec $CONTAINER_ID tail -f /storage/usbdisk1/mikopbx/log/system/messages | grep WorkerApiCommands

Use cases:

• Monitor API request processing
• Watch worker activity
• See errors as they occur
• Debug timing issues

Pro tip: Press Ctrl+C to stop monitoring when done.

---

Critical Log Files

Host Paths (Fast Access)

# Log base directory on host
HOST_LOG_BASE="/Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log"

# PRIMARY: Main system log
${HOST_LOG_BASE}/system/messages

# PHP errors
${HOST_LOG_BASE}/php/error.log

# Asterisk errors
${HOST_LOG_BASE}/asterisk/error

# Web server errors
${HOST_LOG_BASE}/nginx/error.log

# Fail2ban logs
${HOST_LOG_BASE}/fail2ban/fail2ban.log

Container Paths (For docker exec)

# PRIMARY: Main system log
/storage/usbdisk1/mikopbx/log/system/messages

# PHP errors
/storage/usbdisk1/mikopbx/log/php/error.log

# Asterisk errors
/storage/usbdisk1/mikopbx/log/asterisk/error

# Web server errors
/storage/usbdisk1/mikopbx/log/nginx/error.log

Quick Access Commands

Using Host Path (Faster)

# Check recent errors in system log
# Use Grep tool:
#   Pattern: error
#   Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/system/messages
#   Options: -i -C 2

# Read PHP error log
# Use Read tool:
#   Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/php/error.log

# Search for specific pattern
# Use Grep tool:
#   Pattern: WorkerApiCommands
#   Path: /Users/nb/PhpstormProjects/mikopbx/dev_docker/tmp/projects/main/storage/usbdisk1/mikopbx/log/system/messages

Using Docker Exec (For processes)

# Find container
docker ps | grep mikopbx

# Tail system log
docker exec <id> tail -100 /storage/usbdisk1/mikopbx/log/system/messages

# Tail errors only
docker exec <id> tail -500 /storage/usbdisk1/mikopbx/log/system/messages | grep -i error

# Check PHP errors
docker exec <id> tail -50 /storage/usbdisk1/mikopbx/log/php/error.log

# Check workers (MUST use docker exec)
docker exec <id> ps aux | grep Worker

Output Format

Always structure log analysis output clearly:

📋 Log Analysis: <Component/Issue>
==================================

🔍 Search Criteria:
   • Log file: /path/to/log
   • Time range: HH:MM - HH:MM
   • Pattern: <search pattern>
   • Lines analyzed: N

📊 Summary:
   • Total lines: N
   • Errors: N
   • Warnings: N

❌ Errors Found (N):

1. [HH:MM:SS] Error Type
   Location: File.php:line
   Message: Error description
   Process: WorkerName[PID]

2. [HH:MM:SS] Error Type
   ...

⚠️  Warnings Found (top 5):

1. [HH:MM:SS] Warning description
2. ...

🔄 Process Status:

WorkerApiCommands:
   ✅ Running (3 instances)
   PIDs: 1637, 1639, 1641

💡 Analysis:

<interpretation of findings>

🎯 Recommendations:

1. Fix X in File.php:line
2. Check Y in database
3. Monitor Z

Additional Resources

Complete References

• Log Structure Reference - All log files, formats, and locations
• Worker Processes Reference - Complete worker and service list
• Search Patterns Reference - All grep patterns and search techniques

Practical Examples

• Analysis Scenarios - 6 common troubleshooting scenarios with step-by-step analysis

Quick References

Log Locations:

/storage/usbdisk1/mikopbx/log/
├── system/messages      # Main system log
├── php/error.log        # PHP errors
├── asterisk/error       # Asterisk errors
├── nginx/error.log      # Web errors
└── fail2ban/fail2ban.log # Security

Common Workers:

• WorkerApiCommands (3 instances) - API processing
• WorkerModelsEvents (1) - DB change events
• WorkerCdr (1) - CDR processing
• WorkerNotifyByEmail (1) - Email sender

Best Practices

Efficient Analysis

1. Start broad, then narrow

   • Check system log first (overview)
   • Filter for errors (specific issues)
   • Check component logs (details)

2. Use tail with limits

   • ✅ tail -500 | grep (fast)
   • ❌ cat | grep (slow for large files)

3. Filter aggressively

   • Use specific patterns
   • Combine with grep -E for OR logic
   • Use grep -v to exclude noise

4. Check timestamps

   • Correlate events by time
   • Reconstruct event sequence
   • Find related errors

5. Monitor processes

   • Verify services are running
   • Check worker counts
   • Look for zombies/orphans

6. Look for patterns

   • Single error = isolated
   • Repeated errors = systemic
   • Use uniq -c to count occurrences

7. Save findings

   • Document errors
   • Note timestamps
   • Record context

Performance Considerations

• Use tail -N instead of cat for large files
• Pipe through head to limit output
• Check file sizes before reading entire files (ls -lh)
• Use -f for real-time monitoring only when needed
• Kill monitoring processes when done (Ctrl+C)

Troubleshooting Tips

If logs are empty:

• Check if logging is enabled
• Verify log file permissions
• Check if rsyslog is running

If container command fails:

• Verify container is running: docker ps
• Check container name/ID
• Try with docker exec -it for interactive mode

If can't find pattern:

• Try case-insensitive: -i
• Try broader pattern: -E "pattern1|pattern2"
• Increase tail count
• Check related log files

Integration with Other Skills

With mikopbx-docker-restart-tester:

1. Restart container
2. Use this skill to monitor startup
3. Check for errors during initialization

With mikopbx-api-test-generator:

1. Generate and run tests
2. Use this skill to analyze test failures
3. Find root cause in logs

With mikopbx-endpoint-validator:

1. Validator finds issue
2. Use this skill to see actual runtime errors
3. Correlate validation findings with logs

Common Error Patterns

PHP Errors

• Fatal error: - PHP crashed, script stopped
• Undefined variable: - Variable not initialized
• Call to undefined function: - Missing function
• UNIQUE constraint failed: - Database duplicate

Worker Errors

• Process terminated - Worker crashed
• Orphaned process - Worker parent died
• Failed to connect - Service unavailable
• Timeout exceeded - Operation too slow

Asterisk Errors

• Could not create an object - Config error
• Error parsing allow= - Codec config issue
• Authentication failed - Wrong credentials
• No route to destination - Routing problem

System Errors

• Out of memory - Memory exhausted
• Cannot allocate memory - Resources low
• Too many open files - File descriptor limit
• Connection refused - Service not running

Success Criteria

Log analysis is successful when:

• ✅ Root cause identified
• ✅ Actionable information extracted
• ✅ Relevant context provided
• ✅ Next steps clear
• ✅ No unnecessary noise in output
• ✅ Findings well-structured and readable