← 목록으로

Show HN: ShadowDiff – Shadow testing and SQL diffing for AI-migrated back ends

Show HN: ShadowDiff – Shadow testing and SQL diffing for AI-migrated back ends

요약

Dual-Traffic Shadow Testing Engine for AI-Refactored Legacy Code | Real-time traffic teeing, semantic diffing & automated LLM regression test generation - vishalrajofficial/ShadowDiff

본문

🌗 ShadowDiff

Dual-Traffic Shadow Testing Engine for AI-Refactored Legacy Code

License: MIT Go Version Envoy Proxy Fly.io Live Demo

Eliminating the 5% risk that stops enterprise CTOs from deploying AI-modernized backends to production.
🌐 Interactive Live Cloud Sandbox: https://shadowdiff.fly.dev (Self-driving dual-traffic simulator active)

🎬 Live Terminal Walkthrough (vhs)

ShadowDiff Terminal Walkthrough

⚡ The Problem

Enterprises are actively modernizing mission-critical legacy backends (Java 7/8 Spring monoliths, Python 2 Django, PHP 5.6 backends) into modern cloud-native services (Go, TypeScript/Node, FastAPI) using AI code migration engines.

While AI generates ~95% of refactored code accurately, the remaining 5% of hidden edge cases make production deployment an unacceptable risk:

  • Subtle Serialization Discrepancies: Jackson omitting empty lists vs Go serializing null, datetime timezone truncations, float precision drift.
  • Implicit Framework Quirks: Legacy servlet filter quirks, undocumented fallback headers, HTTP 200 responses with embedded error payloads.
  • Database State Regressions: Silent differences in database transaction boundaries, missing audit writes, or N+1 query loop explosions.

💡 The Solution: ShadowDiff

ShadowDiff is a high-performance network proxy and analysis engine that duplicates real staging or production HTTP traffic, sending identical payloads to both the legacy service and the AI-refactored service simultaneously.

 [ Live Inbound Traffic ]
 │
 ▼
 ┌─────────────────────────────┐
 │ Envoy Gateway / Mesh │
 │ (request_mirror_policies) │
 └──────────────┬──────────────┘
 │
 ┌───────────────────┴───────────────────┐
 ▼ ▼
 ┌───────────────────┐ ┌───────────────────┐
 │ Legacy Service │ │ ShadowDiff Core │
 │ (Target) │ │ (Analysis Engine)│
 └─────────┬─────────┘ └─────────┬─────────┘
 │ │
 │ (Live Response to Client) ▼
 │ ┌───────────────────┐
 │ │ AI-Refactored │
 │ │ Candidate (S) │
 │ └─────────┬─────────┘
 │ │
 └───────────────────┬───────────────────┘
 ▼
 ┌─────────────────────────────┐
 │ 3-Way Diff & SQL Comparator │
 │ - P1 ⊕ P2 Noise Filter │
 │ - SQL Mutation Interceptor │
 │ - N+1 Query Loop Detector │
 └──────────────┬──────────────┘
 │
 (Pure Regressions Isolated)
 ▼
 ┌─────────────────────────────┐
 │ Automated AI Test Generator │
 │ - Powered by Google Gemini │
 │ - Synthesizes Go/Jest Tests │
 │ - Root Cause Code Fixes │
 └─────────────────────────────┘

🚀 Key Architectural Capabilities

1. ⚡ Network-Level Envoy Traffic Mirroring

ShadowDiff integrates directly with Envoy Proxy and Istio Service Mesh using request_mirror_policies. Traffic is cloned asynchronously at line rate in the network layer with zero application latency penalty.

2. 🎯 3-Way Differential Noise Cancellation (Diffy Model)

Automatically filters out non-deterministic runtime noise (timestamps, UUIDs, session nonces) by evaluating twin baseline instances: $$\mathcal{N} = P_1 \oplus P_2$$ $$\mathcal{D} = (P_1 \oplus S) \setminus \mathcal{N}$$ Only verified contract regressions reach developer triage.

3. 💾 Database Mutation & SQL Side-Effect Diffing

Captures and correlates SQL statements executed by both services for each HTTP trace:

  • Missing Writes: Flags when legacy wrote to audit_logs or ledger tables that modern code omitted.
  • Transaction Leaks: Warns if legacy used BEGIN...COMMIT and modern ran autocommit statements.
  • N+1 Query Explosions: Detects when modern ORMs loop and execute $\ge 3\times$ more queries than legacy.

4. 🤖 Google Gemini Regression Test Synthesis

Every isolated edge case is analyzed by Google Gemini (gemini-2.0-flash), which writes a turnkey integration test (Go httptest or Vitest) and diagnoses the code fix.

5. 🐙 Autonomous Closed-Loop GitHub PR Bot

Eliminates manual triaging by directly integrating with the GitHub API:

  • Automatic Branching: Spins up shadowdiff/fix-<trace_id> from the repository's default branch.
  • Test File Commitment: Automatically commits the synthesized regression test directly to the repository (e.g. tests/regression/test_drift_<trace_id>.go).
  • Rich Pull Request Generation: Opens a detailed PR with side-by-side JSON diff tables, SQL mutation waterfall analysis, N+1 query warnings, and local verification commands.
  • 1-Click Triage in Console: Developers can inspect the incident in the web console and click "🚀 Open GitHub PR" to immediately open the PR on GitHub.

6. 🚦 CI/CD "Shadow Gating" CLI (shadowdiff gate)

A native deployment blocker that evaluates traffic quality during pre-release soak tests or integration pipelines:

  • Automated Evaluation: Assesses semantic parity, database mutations, and N+1 query loops over a specified evaluation window.
  • Zero False Failure: Honors Diffy noise cancellation ($\mathcal{N} = P_1 \oplus P_2$), only failing on genuine regressions.
  • Pipeline Integration: Returns standard exit codes (0 for PASS, 1 for FAIL/violation, 2 for error) with high-impact ANSI colored terminal summaries and optional JSON format.
# Evaluate quality gate in your CI/CD pipeline:
shadowdiff gate \
 -dashboard-url=http://localhost:8083 \
 -duration=15s \
 -min-parity=100.0 \
 -min-requests=50 \
 -max-drifts=0 \
 -allow-n-plus-one=false \
 -allow-db-drift=false

7. 🛡️ Ephemeral Copy-on-Write (CoW) Database Sandboxing

Allows safe shadow mirroring of mutating HTTP methods (POST, PUT, DELETE, PATCH) with zero database pollution:

  • Transactional Savepoint Rollbacks: Injects boundary headers to ensure shadow writes execute within an isolated savepoint and automatically roll back.
  • Ephemeral Schema Sandboxing: Dynamically creates disposable schemas per test run and drops them upon completion.
  • Serverless CoW Branching: Leverages Neon / Supabase serverless branching to clone database state instantly via API for mutating candidate runs.

8. 🔒 Enterprise PII & Sensitive Data Redactor

Automated zero-latency privacy sanitization across all request payloads, responses, SQL queries, and diff snapshots:

  • Credit Cards (PAN): Full regex & Luhn-validated masking ([REDACTED_CREDIT_CARD]).
  • Social Security & National IDs: Automated pattern redaction ([REDACTED_SSN]).
  • Emails & Phone Numbers: Standard RFC & E.164 patterns ([REDACTED_EMAIL], [REDACTED_PHONE]).
  • API Keys & JWTs: Detects and redacts Bearer tokens, JWT headers/signatures, and private keys.
  • Sensitive JSON Keys: Automatically scrubs passwords, CVVs, pins, secrets, and auth tokens while preserving valid JSON syntax.
  • Zero Leakage to AI: Guarantees raw customer data NEVER reaches external LLM APIs, generated test files, or public dashboards.

9. 🦙 Local & Air-Gapped LLM Synthesizer (Ollama / vLLM)

For regulated enterprise environments (finance, healthcare, defense) where cloud APIs cannot be reached:

  • Native Ollama Integration: Seamless support for local models (llama3.3, deepseek-r1:8b, qwen2.5-coder) via -ai-provider=ollama.
  • OpenAI-Compatible Fallback: Native compatibility with vLLM, LocalAI, and LM Studio.
  • Zero External Telemetry: 100% on-premises execution with zero cloud network traffic.

🚦 GitHub Actions CI/CD Quality Gating Workflow

Incorporate ShadowDiff directly into your automated deployment pipeline (.github/workflows/shadowdiff.yml):

name: Dual-Traffic Shadow Gating
on: [pull_request, push]

jobs:
 shadow-gate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: actions/setup-go@v5
 with:
 go-version: '1.22'

 - name: Build & Start Services
 run: |
 go build -o bin/shadowdiff ./cmd/shadowdiff/main.go
 ./bin/legacy-service -port=:8081 &
 ./bin/refactored-service -port=:8082 &
 ./bin/shadowdiff -proxy-port=:8080 -primary-url=http://localhost:8081 -shadow-url=http://localhost:8082 -sandbox-mode=rollback &
 sleep 2

 - name: Replay Traffic
 run: |
 for i in {1..25}; do
 curl -s "http://localhost:8080/api/users?id=$i" > /dev/null
 curl -s -X POST "http://localhost:8080/api/orders" -d '{"amount":99.50}' > /dev/null
 done

 - name: Run ShadowDiff Quality Gate
 run: |
 ./bin/shadowdiff gate \
 -dashboard-url=http://localhost:8083 \
 -min-parity=100.0 \
 -min-requests=20 \
 -allow-n-plus-one=false \
 -allow-db-drift=false

🛠️ Quickstart with Envoy & Docker Compose

Launch the complete 5-container service mesh testbed (Envoy Gateway + ShadowDiff + Legacy Target + Twin Replica + AI Candidate):

# 1. Clone the repository
git clone https://github.com/vishalrajofficial/ShadowDiff.git
cd ShadowDiff

# 2. Launch the Envoy Service Mesh topology
docker compose -f docker-compose.envoy.yml up -d

Send Ingress Traffic Through Envoy (Port 10000):

# Send test traffic to the Envoy Gateway (mirrored to ShadowDiff automatically):
curl http://localhost:10000/api/users/1
curl -X POST http://localhost:10000/api/orders

Open the telemetry dashboard at http://localhost:8083 to inspect the live 3-way diffs, SQL mutation timeline, and Gemini-generated tests!

☸️ Kubernetes & Istio Service Mesh Deployment

For production Kubernetes clusters running Istio or native Envoy, deploy the included EnvoyFilter to mirror traffic dynamically:

kubectl apply -f deploy/k8s/envoy-shadow-sidecar.yaml

See docs/ARCHITECTURE.md for full architectural specifications.

📄 License

This project is licensed under the MIT License.

← 목록으로