My Knowledge Base

Complete OSINT Data Ingestion Platform - Hetzner Deployment Guide

16 min read

Complete OSINT Data Ingestion Platform - Hetzner Deployment Guide

Executive Overview

This guide provides a production-ready architecture for deploying a comprehensive OSINT data ingestion platform on Hetzner Cloud. The system ingests data from multiple sources (Shodan, IP geolocation services, domain registrars, web hosting providers), stores it in a spatial database, and provides stunning real-time visualizations through an interactive dashboard.

Technology Stack:

  • Backend: Python FastAPI + Node.js Express (async task queue)
  • Database: PostgreSQL with PostGIS extension (geospatial queries)
  • Message Queue: Redis (task queue, caching)
  • Frontend: React with MapLibre GL, D3.js, TimelineJS
  • Containerization: Docker + Docker Compose
  • Reverse Proxy: Nginx
  • SSL/TLS: Let’s Encrypt via Certbot
  • Deployment: Hetzner Cloud with automatic backups

Part 1: Architecture Overview

System Components

┌─────────────────────────────────────────────────────┐
│                    FRONTEND (React)                  │
│  MapLibre Maps | Timeline | Network Graph | Search   │
└────────────────────┬────────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────────┐
│              Nginx Reverse Proxy                      │
│         (SSL/TLS Termination, Load Balancing)        │
└────────────────────┬────────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────────┐
│         Docker Network (internal)                    │
├─────────────────────────────────────────────────────┤
│ ┌──────────────────┐  ┌──────────────────────────┐  │
│ │  FastAPI Backend │  │  Express.js Web Server  │  │
│ │ (Data Ingestion) │  │  (Frontend Serving)     │  │
│ │ (Port 8000)      │  │  (Port 3000)            │  │
│ └──────────────────┘  └──────────────────────────┘  │
│                                                      │
│ ┌──────────────────┐  ┌──────────────────────────┐  │
│ │  Redis Server    │  │  Celery Worker Pool      │  │
│ │  (Port 6379)     │  │  (Background Tasks)      │  │
│ │  (Caching, Queue)│  │  (Data Enrichment)       │  │
│ └──────────────────┘  └──────────────────────────┘  │
│                                                      │
│ ┌──────────────────────────────────────────────────┐│
│ │     PostgreSQL + PostGIS Extension               ││
│ │     (Port 5432, internal only)                   ││
│ │     Geospatial Data Storage & Queries            ││
│ └──────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────┘

External APIs:
├── Shodan API (for live intelligence)
├── IP Geolocation Services (MaxMind GeoLite2, IPInfo.io)
├── Domain WHOIS Services (Team Cymru, ICANN)
└── Web Hosting Detection APIs (BuiltWith, whatRuns)

Data Flow

  1. Ingestion Phase: User submits OSINT data (IP, domain, URL) via API or web form
  2. Enrichment Phase: Celery workers fetch geolocation, WHOIS, DNS records asynchronously
  3. Normalization Phase: Data normalized into unified schema and stored in PostgreSQL
  4. Spatial Indexing: PostGIS creates spatial indexes for geographic queries
  5. Visualization Phase: Frontend queries API and displays on maps, timelines, network graphs
  6. Real-time Updates: WebSockets push live updates to connected clients

Part 2: Prerequisites & Setup

Hetzner Account Setup

  1. Create Hetzner Cloud Account

    • Sign up at https://console.hetzner.cloud/
    • Create new project (recommended: “OSINT-Platform”)
    • Generate API token: Console → Security → API Tokens

  2. Domain Setup

    • Point your domain’s nameservers to Hetzner DNS (or use external DNS)
    • If using Hetzner DNS: Add domain in DNS Console
    • Create A records pointing to your server’s IP

  3. SSH Key Setup

    • Generate SSH key locally: ssh-keygen -t ed25519 -f hetzner-osint -C "osint@hetzner"
    • Add public key to Hetzner: Console → Security → SSH Keys

Local Prerequisites

  • Docker Desktop installed (Windows/Mac) or Docker Engine (Linux)
  • Docker Compose v2.0+
  • Git installed
  • Text editor or IDE
  • Terminal/Command Prompt
  • GitHub account (for version control)

Part 3: Complete Docker Compose Setup

Create a docker-compose.yml file in your project root:

version: '3.8'
 
services:
  # PostgreSQL with PostGIS
  postgres:
    image: postgis/postgis:16-3.4
    container_name: osint-db
    environment:
      POSTGRES_DB: osint_db
      POSTGRES_USER: osint_user
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
      POSTGRES_INITDB_ARGS: "-c shared_preload_libraries=pg_stat_statements"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-db.sql:/docker-entrypoint-initdb.d/init.sql
    ports:
      - "5432:5432"
    networks:
      - osint-network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U osint_user -d osint_db"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped
 
  # Redis for caching and task queue
  redis:
    image: redis:7-alpine
    container_name: osint-redis
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD:-changeme}
    volumes:
      - redis_data:/data
    ports:
      - "6379:6379"
    networks:
      - osint-network
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped
 
  # FastAPI Backend
  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    container_name: osint-backend
    environment:
      DATABASE_URL: postgresql://osint_user:${POSTGRES_PASSWORD:-changeme}@postgres:5432/osint_db
      REDIS_URL: redis://:${REDIS_PASSWORD:-changeme}@redis:6379/0
      SHODAN_API_KEY: ${SHODAN_API_KEY}
      FASTAPI_ENV: production
      SECRET_KEY: ${SECRET_KEY}
    volumes:
      - ./backend:/app
    ports:
      - "8000:8000"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - osint-network
    command: uvicorn main:app --host 0.0.0.0 --port 8000 --reload
    restart: unless-stopped
 
  # Celery Worker
  celery:
    build:
      context: ./backend
      dockerfile: Dockerfile
    container_name: osint-celery
    environment:
      DATABASE_URL: postgresql://osint_user:${POSTGRES_PASSWORD:-changeme}@postgres:5432/osint_db
      REDIS_URL: redis://:${REDIS_PASSWORD:-changeme}@redis:6379/0
      SHODAN_API_KEY: ${SHODAN_API_KEY}
    volumes:
      - ./backend:/app
    depends_on:
      - postgres
      - redis
    networks:
      - osint-network
    command: celery -A tasks worker --loglevel=info --concurrency=4
    restart: unless-stopped
 
  # Express Frontend Server
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    container_name: osint-frontend
    environment:
      REACT_APP_API_URL: http://localhost:8000
      NODE_ENV: production
    ports:
      - "3000:3000"
    depends_on:
      - backend
    networks:
      - osint-network
    restart: unless-stopped
 
  # Nginx Reverse Proxy
  nginx:
    image: nginx:alpine
    container_name: osint-nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/conf.d:/etc/nginx/conf.d:ro
      - ./certbot/conf:/etc/letsencrypt:ro
      - ./certbot/www:/var/www/certbot:ro
      - ./public:/usr/share/nginx/html:ro
    depends_on:
      - backend
      - frontend
    networks:
      - osint-network
    restart: unless-stopped
 
  # Certbot for SSL certificates
  certbot:
    image: certbot/certbot
    container_name: osint-certbot
    volumes:
      - ./certbot/conf:/etc/letsencrypt
      - ./certbot/www:/var/www/certbot
    entrypoint: /bin/sh -c 'trap exit TERM; while :; do certbot renew --quiet; sleep 12h & wait $${!}; done;'
    networks:
      - osint-network
    restart: unless-stopped
 
volumes:
  postgres_data:
  redis_data:
 
networks:
  osint-network:
    driver: bridge

Environment Variables (.env file)

Create a .env file in your project root:

# Database
POSTGRES_PASSWORD=your_secure_password_here
POSTGRES_DB=osint_db
 
# Redis
REDIS_PASSWORD=your_redis_secure_password
 
# Shodan API
SHODAN_API_KEY=your_shodan_api_key_here
 
# Application Secrets
SECRET_KEY=your_very_long_random_secret_key_generate_with_openssl_rand_hex
 
# Domain Configuration
DOMAIN=your-domain.com
EMAIL=your-email@example.com
 
# Hetzner DNS (if using Hetzner DNS)
HETZNER_DNS_TOKEN=your_hetzner_dns_api_token
 
# FastAPI Environment
FASTAPI_ENV=production

Generate secure secrets:

# Generate SECRET_KEY
python3 -c "import secrets; print(secrets.token_urlsafe(32))"
 
# Generate POSTGRES_PASSWORD
openssl rand -base64 32
 
# Generate REDIS_PASSWORD
openssl rand -base64 24

Part 4: Backend Implementation

4.1 FastAPI Backend Structure

Create backend/main.py:

from fastapi import FastAPI, HTTPException, Depends, BackgroundTasks
from fastapi.middleware.cors import CORSMiddleware
from sqlmodel import SQLModel, create_engine, Session, select
from sqlalchemy.pool import StaticPool
from pydantic import BaseModel
from typing import List, Optional
import os
from datetime import datetime
 
# Initialize FastAPI app
app = FastAPI(
    title="OSINT Intelligence Platform",
    description="Multi-source data ingestion and visualization",
    version="1.0.0"
)
 
# CORS Configuration
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],  # Configure for production
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)
 
# Database connection
DATABASE_URL = os.getenv(
    "DATABASE_URL",
    "postgresql://osint_user:changeme@localhost:5432/osint_db"
)
 
engine = create_engine(
    DATABASE_URL,
    echo=False,
    future=True,
)
 
def get_session():
    with Session(engine) as session:
        yield session
 
# Data Models
class IPEntity(SQLModel, table=True):
    __tablename__ = "ip_entities"
    id: Optional[int] = None
    ip_address: str
    latitude: float
    longitude: float
    city: str
    country: str
    asn: str
    isp: str
    threat_level: int
    source: str
    discovered_at: datetime
    ports: Optional[List[int]] = None
    services: Optional[List[str]] = None
 
class DomainEntity(SQLModel, table=True):
    __tablename__ = "domain_entities"
    id: Optional[int] = None
    domain: str
    registrar: str
    registration_date: datetime
    expiry_date: datetime
    nameservers: Optional[List[str]] = None
    source: str
    discovered_at: datetime
 
class IngestRequest(BaseModel):
    data_type: str  # "ip", "domain", "url", "email"
    value: str
    source: str  # "shodan", "ip_geolocation", "domain_registrar", etc.
    metadata: Optional[dict] = None
 
class IngestResponse(BaseModel):
    status: str
    message: str
    entity_id: Optional[int] = None
 
# API Endpoints
@app.get("/health")
async def health_check():
    """Health check endpoint"""
    return {"status": "healthy", "service": "osint-platform"}
 
@app.post("/ingest", response_model=IngestResponse)
async def ingest_data(
    request: IngestRequest,
    background_tasks: BackgroundTasks,
    session: Session = Depends(get_session)
):
    """
    Ingest OSINT data from various sources.
    Triggers background enrichment tasks.
    """
    try:
        if request.data_type == "ip":
            # Add task to queue for enrichment
            background_tasks.add_task(
                enrich_ip_data,
                request.value,
                request.source
            )
            return IngestResponse(
                status="accepted",
                message=f"IP {request.value} queued for enrichment"
            )
        elif request.data_type == "domain":
            background_tasks.add_task(
                enrich_domain_data,
                request.value,
                request.source
            )
            return IngestResponse(
                status="accepted",
                message=f"Domain {request.value} queued for enrichment"
            )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
 
@app.get("/entities/ips")
async def get_ips(
    limit: int = 100,
    offset: int = 0,
    session: Session = Depends(get_session)
):
    """Retrieve all IP entities with pagination"""
    statement = select(IPEntity).offset(offset).limit(limit)
    entities = session.exec(statement).all()
    return {"total": len(entities), "entities": entities}
 
@app.get("/entities/domains")
async def get_domains(
    limit: int = 100,
    offset: int = 0,
    session: Session = Depends(get_session)
):
    """Retrieve all domain entities with pagination"""
    statement = select(DomainEntity).offset(offset).limit(limit)
    entities = session.exec(statement).all()
    return {"total": len(entities), "entities": entities}
 
@app.get("/entities/geospatial")
async def get_geospatial_data(session: Session = Depends(get_session)):
    """Return geospatial data for map visualization"""
    ips = session.exec(select(IPEntity)).all()
    return {
        "type": "FeatureCollection",
        "features": [
            {
                "type": "Feature",
                "geometry": {
                    "type": "Point",
                    "coordinates": [ip.longitude, ip.latitude]
                },
                "properties": {
                    "ip": ip.ip_address,
                    "city": ip.city,
                    "country": ip.country,
                    "asn": ip.asn,
                    "threat_level": ip.threat_level
                }
            }
            for ip in ips
        ]
    }
 
# Background enrichment functions
async def enrich_ip_data(ip: str, source: str):
    """Placeholder for IP enrichment logic"""
    # In production, integrate with:
    # - MaxMind GeoLite2 for geolocation
    # - Shodan API for port/service data
    # - Team Cymru for ASN/ISP info
    pass
 
async def enrich_domain_data(domain: str, source: str):
    """Placeholder for domain enrichment logic"""
    # In production, integrate with:
    # - WHOIS services
    # - DNS resolution
    # - Certificate transparency logs
    pass
 
if __name__ == "__main__":
    # Create tables
    SQLModel.metadata.create_all(engine)
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

4.2 Celery Task Configuration

Create backend/tasks.py:

from celery import Celery
import os
 
redis_url = os.getenv("REDIS_URL", "redis://localhost:6379/0")
 
celery_app = Celery(
    "osint_tasks",
    broker=redis_url,
    backend=redis_url
)
 
celery_app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='UTC',
    enable_utc=True,
    task_track_started=True,
    task_time_limit=30 * 60,  # 30 minutes
)
 
@celery_app.task(name="tasks.enrich_ip")
def enrich_ip(ip_address: str):
    """Enrich IP data with geolocation and threat intelligence"""
    # Integration with external APIs
    pass
 
@celery_app.task(name="tasks.enrich_domain")
def enrich_domain(domain: str):
    """Enrich domain data with WHOIS and DNS records"""
    # Integration with external APIs
    pass

4.3 Dockerfile for Backend

Create backend/Dockerfile:

FROM python:3.11-slim
 
WORKDIR /app
 
RUN apt-get update && apt-get install -y \
    gcc \
    postgresql-client \
    && rm -rf /var/lib/apt/lists/*
 
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
 
COPY . .
 
EXPOSE 8000
 
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Create backend/requirements.txt:

fastapi==0.104.1
uvicorn[standard]==0.24.0
sqlmodel==0.0.14
sqlalchemy[postgresql]==2.0.23
psycopg[binary]==3.17.0
celery[redis]==5.3.4
redis==5.0.1
pydantic==2.5.0
pydantic-settings==2.1.0
python-multipart==0.0.6
aiofiles==23.2.1
requests==2.31.0

Part 5: Frontend Setup

5.1 React Application Structure

Create frontend/src/App.jsx:

import React, { useState, useEffect } from 'react';
import Dashboard from './components/Dashboard';
import MapView from './components/MapView';
import TimelineView from './components/TimelineView';
import NetworkGraph from './components/NetworkGraph';
import DataIngest from './components/DataIngest';
import './App.css';
 
export default function App() {
  const [currentView, setCurrentView] = useState('dashboard');
  const [data, setData] = useState(null);
 
  useEffect(() => {
    fetchData();
  }, []);
 
  const fetchData = async () => {
    try {
      const response = await fetch(`${process.env.REACT_APP_API_URL}/entities/geospatial`);
      const result = await response.json();
      setData(result);
    } catch (error) {
      console.error('Error fetching data:', error);
    }
  };
 
  const renderView = () => {
    switch (currentView) {
      case 'dashboard':
        return <Dashboard data={data} />;
      case 'map':
        return <MapView data={data} />;
      case 'timeline':
        return <TimelineView data={data} />;
      case 'network':
        return <NetworkGraph data={data} />;
      case 'ingest':
        return <DataIngest onSuccess={fetchData} />;
      default:
        return <Dashboard data={data} />;
    }
  };
 
  return (
    <div className="app">
      <nav className="navbar">
        <h1>OSINT Intelligence Platform</h1>
        <ul className="nav-links">
          <li><button onClick={() => setCurrentView('dashboard')}>Dashboard</button></li>
          <li><button onClick={() => setCurrentView('map')}>Map</button></li>
          <li><button onClick={() => setCurrentView('timeline')}>Timeline</button></li>
          <li><button onClick={() => setCurrentView('network')}>Network</button></li>
          <li><button onClick={() => setCurrentView('ingest')}>Ingest Data</button></li>
        </ul>
      </nav>
      <main className="main-content">
        {renderView()}
      </main>
    </div>
  );
}

5.2 Dockerfile for Frontend

Create frontend/Dockerfile:

FROM node:20-alpine as builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
 
FROM node:20-alpine
WORKDIR /app
RUN npm install -g serve
COPY --from=builder /app/build ./build
EXPOSE 3000
CMD ["serve", "-s", "build", "-l", "3000"]

Create frontend/package.json:

{
  "name": "osint-frontend",
  "version": "1.0.0",
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "maplibre-gl": "^4.0.0",
    "d3": "^7.8.5",
    "axios": "^1.6.0"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  }
}

Part 6: Nginx Configuration

Create nginx/nginx.conf:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
 
events {
    worker_connections 1024;
}
 
http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
 
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log /var/log/nginx/access.log main;
 
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    client_max_body_size 20M;
 
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml text/javascript
               application/json application/javascript application/xml+rss
               application/rss+xml font/truetype font/opentype
               application/vnd.ms-fontobject image/svg+xml;
 
    include /etc/nginx/conf.d/*.conf;
}

Create nginx/conf.d/default.conf:

upstream backend {
    server backend:8000;
}
 
upstream frontend {
    server frontend:3000;
}
 
# HTTP to HTTPS redirect
server {
    listen 80;
    server_name _;
    
    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
    
    location / {
        return 301 https://$host$request_uri;
    }
}
 
# HTTPS server
server {
    listen 443 ssl http2;
    server_name your-domain.com;
 
    ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
 
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
 
    # Frontend
    location / {
        proxy_pass http://frontend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
 
    # Backend API
    location /api/ {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
 
    # WebSocket support
    location /ws/ {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
}

Part 7: Database Initialization

Create init-db.sql:

-- Enable PostGIS extension
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
 
-- Create schema for OSINT data
CREATE SCHEMA IF NOT EXISTS osint;
 
-- IP Entities table with geospatial support
CREATE TABLE IF NOT EXISTS osint.ip_entities (
    id SERIAL PRIMARY KEY,
    ip_address INET UNIQUE NOT NULL,
    location GEOGRAPHY(POINT, 4326),
    city VARCHAR(255),
    country VARCHAR(255),
    country_code CHAR(2),
    asn VARCHAR(20),
    isp VARCHAR(255),
    threat_level SMALLINT CHECK (threat_level >= 0 AND threat_level <= 10),
    source VARCHAR(100),
    discovered_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    ports INTEGER[],
    services VARCHAR(255)[],
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
 
-- Domain Entities table
CREATE TABLE IF NOT EXISTS osint.domain_entities (
    id SERIAL PRIMARY KEY,
    domain VARCHAR(255) UNIQUE NOT NULL,
    registrar VARCHAR(255),
    registration_date DATE,
    expiry_date DATE,
    nameservers VARCHAR(255)[],
    source VARCHAR(100),
    discovered_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
 
-- Relationships between entities
CREATE TABLE IF NOT EXISTS osint.relationships (
    id SERIAL PRIMARY KEY,
    source_type VARCHAR(50),
    source_id INTEGER,
    target_type VARCHAR(50),
    target_id INTEGER,
    relationship_type VARCHAR(100),
    strength DECIMAL(3,2) CHECK (strength >= 0 AND strength <= 1),
    discovered_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
 
-- Create indexes for performance
CREATE INDEX idx_ip_entities_location ON osint.ip_entities USING GIST(location);
CREATE INDEX idx_ip_entities_threat ON osint.ip_entities(threat_level);
CREATE INDEX idx_ip_entities_source ON osint.ip_entities(source);
CREATE INDEX idx_domain_entities_source ON osint.domain_entities(source);
CREATE INDEX idx_relationships_source ON osint.relationships(source_type, source_id);

Part 8: Hetzner Cloud Deployment

8.1 Create Hetzner Cloud Server

# Using Hetzner CLI
hcloud server create \
  --name osint-platform \
  --type cpx31 \
  --image docker-ce \
  --location fsn1 \
  --ssh-key your-ssh-key-name

Or via console:

  1. Go to Hetzner Cloud Console
  2. Create Server with Ubuntu 24.04 LTS
  3. Select CPX21 or higher (recommended: CPX31 for production)
  4. Select Docker-CE app
  5. Attach SSH key
  6. Create server

8.2 Initial Server Configuration

SSH into your server:

ssh -i ~/.ssh/hetzner-osint root@<server-ip>

Update system and install tools:

apt-get update && apt-get upgrade -y
apt-get install -y git curl wget htop tmux
 
# Install Docker Compose (already included with docker-ce app)
docker --version
docker-compose --version

8.3 Clone Repository and Deploy

# Clone your repository
git clone https://github.com/yourusername/osint-platform.git
cd osint-platform
 
# Copy and edit environment file
cp .env.example .env
# Edit .env with secure passwords and API keys
nano .env
 
# Create necessary directories
mkdir -p certbot/conf certbot/www nginx/conf.d
 
# Start all services
docker-compose up -d
 
# Verify services
docker-compose ps
docker-compose logs -f backend

Part 9: SSL Certificate Setup (Let’s Encrypt)

9.1 Initial Certificate Acquisition

# Install Certbot
apt-get install -y certbot python3-certbot-dns-hetzner
 
# Obtain certificate
certbot certonly \
  --dns-hetzner \
  --dns-hetzner-credentials ~/.hetzner_dns_credentials \
  --email your-email@example.com \
  --agree-tos \
  -d your-domain.com \
  -d www.your-domain.com

9.2 Certbot Credentials File

Create ~/.hetzner_dns_credentials:

dns_hetzner_api_token = YOUR_HETZNER_DNS_API_TOKEN
chmod 600 ~/.hetzner_dns_credentials

9.3 Auto-Renewal Configuration

Create systemd timer for certificate renewal:

# Create renewal timer
systemctl enable certbot.timer
systemctl start certbot.timer
 
# Verify
systemctl list-timers certbot.timer

Part 10: Data Backup and Recovery

10.1 Automated Database Backups

Create backup script scripts/backup.sh:

#!/bin/bash
 
BACKUP_DIR="/backups/osint"
DB_CONTAINER="osint-db"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
BACKUP_FILE="$BACKUP_DIR/osint_backup_$TIMESTAMP.sql.gz"
 
mkdir -p $BACKUP_DIR
 
# Backup PostgreSQL
docker exec $DB_CONTAINER pg_dump \
  -U osint_user \
  -d osint_db | gzip > $BACKUP_FILE
 
echo "Backup completed: $BACKUP_FILE"
 
# Keep only last 30 days of backups
find $BACKUP_DIR -name "osint_backup_*.sql.gz" -mtime +30 -delete
 
# Upload to S3 (optional)
# aws s3 cp $BACKUP_FILE s3://your-backup-bucket/osint/

10.2 Cron Job for Automated Backups

crontab -e
 
# Add this line for daily backups at 2 AM
0 2 * * * /path/to/scripts/backup.sh

10.3 Restore from Backup

# Restore from backup file
zcat /backups/osint/osint_backup_20251103_120000.sql.gz | \
  docker exec -i osint-db psql -U osint_user -d osint_db

Part 11: Monitoring and Maintenance

11.1 Docker Container Health Monitoring

# Check container status
docker-compose ps
 
# View logs
docker-compose logs -f --tail=100 backend
docker-compose logs -f --tail=100 frontend
 
# Resource usage
docker stats
 
# Database connection pool
docker exec osint-db psql -U osint_user -d osint_db -c "SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;"

11.2 Performance Optimization

Create scripts/optimize.sh:

#!/bin/bash
 
# Optimize PostgreSQL
docker exec osint-db psql -U osint_user -d osint_db -c "VACUUM ANALYZE;"
 
# Check index health
docker exec osint-db psql -U osint_user -d osint_db -c "REINDEX DATABASE osint_db;"
 
# Clear Redis cache (optional)
docker exec osint-redis redis-cli FLUSHDB

Part 12: Scaling Considerations

12.1 Horizontal Scaling

For higher traffic, use Docker Swarm or Kubernetes:

# Multiple replicas in docker-compose
services:
  backend:
    deploy:
      replicas: 3
  
  celery:
    deploy:
      replicas: 5

12.2 Database Optimization

  • Enable table partitioning for large datasets
  • Implement read replicas for query load
  • Use connection pooling (PgBouncer)
  • Archive old data to separate storage

12.3 Caching Strategy

Implement Redis caching for:

  • Geospatial query results
  • API responses
  • Enrichment data
  • Session management

Part 13: OSINT Data Sources Integration

13.1 Shodan API Integration

import shodan
 
def integrate_shodan(api_key: str, query: str):
    api = shodan.Shodan(api_key)
    try:
        results = api.search(query)
        return results
    except shodan.APIError as e:
        print(f"Error: {e}")

13.2 IP Geolocation (MaxMind GeoLite2)

from geolite2 import geolite2
 
def geolocate_ip(ip: str):
    match = geolite2.reader().get(ip)
    if match:
        return {
            "country": match["country"]["names"]["en"],
            "city": match.get("city", {}).get("names", {}).get("en"),
            "latitude": match["location"]["latitude"],
            "longitude": match["location"]["longitude"]
        }

13.3 Domain WHOIS Lookup

import whois
 
def lookup_domain(domain: str):
    try:
        w = whois.whois(domain)
        return {
            "registrar": w.registrar,
            "creation_date": w.creation_date,
            "expiration_date": w.expiration_date,
            "name_servers": w.name_servers
        }
    except Exception as e:
        print(f"Error: {e}")

Part 14: Security Best Practices

14.1 Network Security

# Install UFW firewall
apt-get install -y ufw
 
# Configure firewall rules
ufw allow 22/tcp      # SSH
ufw allow 80/tcp      # HTTP
ufw allow 443/tcp     # HTTPS
ufw enable

14.2 API Authentication

Implement JWT tokens for API access:

from jose import JWTError, jwt
from fastapi import Depends, Security
from fastapi.security import HTTPBearer, HTTPAuthCredentials
 
security = HTTPBearer()
 
def verify_token(credentials: HTTPAuthCredentials = Security(security)):
    token = credentials.credentials
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
        return payload
    except JWTError:
        raise HTTPException(status_code=401, detail="Invalid token")

14.3 Database Security

-- Restrict user permissions
REVOKE ALL PRIVILEGES ON DATABASE osint_db FROM osint_user;
GRANT CONNECT ON DATABASE osint_db TO osint_user;
GRANT USAGE ON SCHEMA osint TO osint_user;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA osint TO osint_user;
 
-- Enable SSL for connections

Part 15: Complete Deployment Checklist

  • Domain registered and DNS configured
  • SSH keys generated and added to Hetzner
  • Hetzner Cloud server created with Docker
  • Repository cloned on server
  • .env file created with secure values
  • SSL certificates generated via Let’s Encrypt
  • Docker services started and verified
  • Database initialized and schema created
  • API endpoints tested
  • Frontend accessible via domain
  • Backup strategy implemented
  • Firewall configured
  • Monitoring setup in place
  • External API integrations configured
  • Load balancing configured for production traffic

Conclusion

You now have a complete, production-ready OSINT data ingestion platform deployed on Hetzner Cloud. The system is designed to scale, includes proper security measures, automated backups, and integrates with multiple OSINT data sources for comprehensive intelligence gathering and visualization.

For updates and additional features, maintain version control with Git and follow CI/CD best practices.

Backlinks