OSINT Platform - Quick Reference & Troubleshooting

Quick Start Commands

Local Development Setup

# Clone repository
git clone https://github.com/yourusername/osint-platform.git
cd osint-platform
 
# Create environment file
cp .env.example .env
 
# Start all services
docker-compose up -d
 
# Check status
docker-compose ps
 
# View logs
docker-compose logs -f backend

Hetzner Cloud Deployment

# SSH into server
ssh -i ~/.ssh/hetzner-osint root@<server-ip>
 
# Navigate to project
cd osint-platform
 
# Pull latest code
git pull origin main
 
# Rebuild images
docker-compose build --no-cache
 
# Deploy
docker-compose up -d
 
# Restart specific service
docker-compose restart backend

API Endpoints

Endpoint	Method	Purpose
`/health`	GET	Health check
`/ingest`	POST	Ingest OSINT data
`/entities/ips`	GET	Retrieve IP entities
`/entities/domains`	GET	Retrieve domain entities
`/entities/geospatial`	GET	Get GeoJSON for map
`/search`	GET	Advanced search
`/export`	POST	Export data as CSV/JSON

Common Issues & Solutions

1. Docker Network Issues

# Inspect network
docker network inspect osint-platform_osint-network
 
# Restart network
docker-compose down
docker network prune
docker-compose up -d

2. Database Connection Errors

# Check database status
docker-compose logs postgres
 
# Rebuild database
docker-compose exec postgres psql -U osint_user -d osint_db -c "SELECT 1"
 
# Recreate database
docker-compose down
docker volume rm osint-platform_postgres_data
docker-compose up -d postgres

3. API Timeout Issues

# Increase timeouts
# In nginx.conf:
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
 
# Reload Nginx
docker-compose exec nginx nginx -s reload

4. Certificate Issues

# Check certificate status
ls -la /etc/letsencrypt/live/your-domain.com/
 
# Force renewal
certbot renew --force-renewal -v
 
# Validate Nginx config
docker-compose exec nginx nginx -t
 
# Reload Nginx
docker-compose exec nginx nginx -s reload

5. Memory/CPU Issues

# Check resource usage
docker stats
 
# Limit container resources in docker-compose.yml:
services:
  backend:
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 1G

6. Redis Connection Issues

# Test Redis connection
docker-compose exec redis redis-cli ping
 
# Monitor Redis operations
docker-compose exec redis redis-cli MONITOR
 
# Clear Redis cache
docker-compose exec redis redis-cli FLUSHALL

Performance Tuning

Database Optimization

-- Analyze tables
VACUUM ANALYZE;
 
-- Reindex
REINDEX DATABASE osint_db;
 
-- Check slow queries
SELECT query, calls, mean_exec_time FROM pg_stat_statements 
ORDER BY mean_exec_time DESC LIMIT 10;

Nginx Caching

# Add to nginx.conf
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=api_cache:10m max_size=1g inactive=60m;
 
location /api/ {
    proxy_cache api_cache;
    proxy_cache_valid 200 10m;
    proxy_cache_key "$scheme$request_method$host$request_uri";
    add_header X-Cache-Status $upstream_cache_status;
}

Redis Connection Pooling

# In backend configuration
from redis import ConnectionPool, Redis
 
pool = ConnectionPool.from_url(
    redis_url,
    max_connections=50,
    decode_responses=True
)
redis_client = Redis(connection_pool=pool)

Monitoring & Logging

View Logs

# Backend logs
docker-compose logs -f --tail=100 backend
 
# Database logs
docker-compose logs -f postgres
 
# Nginx logs
docker-compose logs -f nginx
 
# Combined logs
docker-compose logs -f

Setup Log Rotation

# Create logrotate config
sudo cat > /etc/logrotate.d/docker-osint << EOF
/var/lib/docker/containers/*/*.log {
  rotate 10
  daily
  compress
  delaycompress
  missingok
  maxage 7
}
EOF

Data Management

Export Data

# Export to CSV
curl -X POST http://localhost:8000/export \
  -H "Content-Type: application/json" \
  -d '{"format": "csv", "entity_type": "ip"}' \
  -o osint_data.csv
 
# Export to JSON
curl -X POST http://localhost:8000/export \
  -H "Content-Type: application/json" \
  -d '{"format": "json", "entity_type": "all"}' \
  -o osint_data.json

Database Backup

# Manual backup
docker exec osint-db pg_dump -U osint_user -d osint_db | gzip > backup.sql.gz
 
# Restore from backup
zcat backup.sql.gz | docker exec -i osint-db psql -U osint_user -d osint_db
 
# Upload to S3
aws s3 cp backup.sql.gz s3://your-bucket/osint/backup_$(date +%Y%m%d).sql.gz

Integration with OSINT Sources

Shodan API Setup

import shodan
 
api = shodan.Shodan('YOUR_API_KEY')
 
# Search Shodan
results = api.search('apache')
 
# Get IP info
ip_info = api.host('8.8.8.8')
 
# Stream results
try:
    for banner in api.search_cursor('http.title:"admin"'):
        print(banner)
except shodan.APIError as e:
    print(f"Error: {e}")

MaxMind GeoIP Setup

from geoip2.database import Reader
 
# Download GeoLite2 database from MaxMind
# https://www.maxmind.com/en/geolite2/geolite2-free-geolocation-data
 
reader = Reader('/path/to/GeoLite2-City.mmdb')
 
response = reader.city('8.8.8.8')
print(f"Country: {response.country.iso_code}")
print(f"City: {response.city.name}")
print(f"Lat: {response.location.latitude}")
print(f"Lon: {response.location.longitude}")

Domain WHOIS Integration

import whois
 
domain_info = whois.whois('example.com')
print(f"Registrar: {domain_info.registrar}")
print(f"Created: {domain_info.creation_date}")
print(f"Expires: {domain_info.expiration_date}")
print(f"Nameservers: {domain_info.name_servers}")

Security Hardening

SSH Key Configuration

# On local machine
ssh-keygen -t ed25519 -f ~/.ssh/osint-hetzner -C "osint@hetzner"
 
# Add to authorized_keys on server
ssh-copy-id -i ~/.ssh/osint-hetzner.pub root@<server-ip>
 
# Add to ~/.ssh/config
cat >> ~/.ssh/config << EOF
Host osint-hetzner
    HostName <server-ip>
    User root
    IdentityFile ~/.ssh/osint-hetzner
EOF
 
# Connect
ssh osint-hetzner

Firewall Configuration

# UFW setup
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable
 
# Check status
ufw status
 
# Advanced: Rate limiting
ufw limit 22/tcp

API Key Management

# Use environment variables
import os
from dotenv import load_dotenv
 
load_dotenv()
 
SHODAN_API_KEY = os.getenv('SHODAN_API_KEY')
MAXMIND_LICENSE = os.getenv('MAXMIND_LICENSE')
 
# Never commit .env to git
# Add to .gitignore
echo ".env" >> .gitignore
echo ".env.local" >> .gitignore

Maintenance Tasks

Weekly

# Update Docker images
docker-compose pull
docker-compose up -d
 
# Check certificate status
certbot certificates
 
# Review logs
docker-compose logs | tail -100

Monthly

# Full system update
apt-get update && apt-get upgrade -y
 
# Optimize database
docker-compose exec postgres psql -U osint_user -d osint_db -c "VACUUM ANALYZE;"
 
# Verify backups
ls -lah /backups/osint/ | head -10
 
# Review security logs
grep "error\|failed" /var/log/auth.log | tail -20

Quarterly

# Backup configuration
tar -czf osint-config-backup.tar.gz .env nginx/ backend/ frontend/
 
# Upload to S3
aws s3 cp osint-config-backup.tar.gz s3://your-bucket/config-backups/
 
# Review and update documentation
# Update security policies
# Audit user access

CLI Tools & Utilities

Docker Compose Cheat Sheet

# Start services
docker-compose up -d
 
# Stop services
docker-compose down
 
# View running containers
docker-compose ps
 
# Execute command in container
docker-compose exec backend bash
 
# View resource usage
docker stats
 
# Rebuild images
docker-compose build --no-cache
 
# Update specific service
docker-compose up -d --no-deps --build backend
 
# View environment variables
docker-compose exec backend env
 
# Access database shell
docker-compose exec postgres psql -U osint_user -d osint_db

Useful Shortcuts

# Create alias for docker-compose
alias dc='docker-compose'
 
# View last 50 lines of logs
dc logs --tail=50
 
# Watch logs in real-time
dc logs -f --tail=100
 
# Restart all services
dc restart
 
# Remove unused volumes
docker volume prune
 
# Remove unused images
docker image prune -a

Performance Metrics

CPU Usage Baseline

Backend: 5-10% idle
Frontend: <5% idle
Database: 10-15% idle
Nginx: <2% idle

Memory Usage Baseline

Backend: 200-300 MB
Frontend: 100-150 MB
Database: 300-500 MB (varies with data size)
Redis: 50-100 MB

Response Time Targets

API endpoints: <200ms
Map loading: <500ms
Search queries: <1s
Data export: <5s (depending on size)

Advanced Troubleshooting

Check Port Conflicts

# List ports in use
netstat -tuln | grep LISTEN
 
# Kill process using port
lsof -i :8000
kill -9 <PID>

Docker System Prune

# Remove unused containers, networks, and images
docker system prune -a
 
# Remove unused volumes
docker volume prune
 
# Full cleanup (with volumes)
docker system prune -a --volumes

Rebuild from Scratch

# Stop all services
docker-compose down
 
# Remove all volumes
docker-compose down -v
 
# Remove images
docker-compose down --rmi all
 
# Rebuild and start
docker-compose up -d --build

Reference Links

Docker Documentation: https://docs.docker.com/
FastAPI: https://fastapi.tiangolo.com/
PostgreSQL: https://www.postgresql.org/docs/
PostGIS: https://postgis.net/docs/
Nginx: https://nginx.org/en/docs/
Let’s Encrypt: https://letsencrypt.org/docs/
Hetzner Cloud: https://docs.hetzner.cloud/
React: https://react.dev/
MapLibre GL JS: https://maplibre.org/maplibre-gl-js/docs/

My Knowledge Base

Explorer

OSINT Platform - Quick Reference & Troubleshooting

OSINT Platform - Quick Reference & Troubleshooting

Quick Start Commands

Local Development Setup

Hetzner Cloud Deployment

API Endpoints

Common Issues & Solutions

1. Docker Network Issues

2. Database Connection Errors

3. API Timeout Issues

4. Certificate Issues

5. Memory/CPU Issues

6. Redis Connection Issues

Performance Tuning

Database Optimization

Nginx Caching

Redis Connection Pooling

Monitoring & Logging

View Logs

Setup Log Rotation

Data Management

Export Data

Database Backup

Integration with OSINT Sources

Shodan API Setup

MaxMind GeoIP Setup

Domain WHOIS Integration

Security Hardening

SSH Key Configuration

Firewall Configuration

API Key Management

Maintenance Tasks

Weekly

Monthly

Quarterly

CLI Tools & Utilities

Docker Compose Cheat Sheet

Useful Shortcuts

Performance Metrics

CPU Usage Baseline

Memory Usage Baseline

Response Time Targets

Advanced Troubleshooting

Check Port Conflicts

Docker System Prune

Rebuild from Scratch

Reference Links

Backlinks