Webhook Failures Troubleshooting Guide
Webhook Failures Troubleshooting Guide
This guide helps diagnose why webhooks are not working. Follow the systematic debug process to identify and fix issues quickly.
Quick Diagnosis
Webhook not received at all? β See Webhook Not Delivered Webhook received but signature fails? β See Signature Validation Webhook times out? β See Timeout Issues Webhook processed but business logic wrong? β See Logic Errors Duplicate webhooks? β See Duplicate Webhooks
Systematic Debug Process
Step 1: Verify External Service Configuration
PactaPay Dashboard Check:
- Login to https://dashboard.pactapay.com
- Navigate to Webhooks section
- Verify endpoint configured:
- URL:
https://YOUR_SUBDOMAIN.ngrok-free.app/api/commerce/webhooks/pactapay - Events:
payment.completed,payment.failed,payment.refunded - Status: β Active (not paused)
- URL:
Test Webhook:
# In PactaPay dashboard, click "Send test event"# Check delivery status:# - β
200: Success# - β οΈ 4xx/5xx: Error (check logs)# - β±οΈ Timeout: Server not respondingDelivery Logs:
- Dashboard β Webhooks β Click endpoint β Delivery logs
- Look for recent attempts and response codes
- Failed deliveries show error details (DNS, connection, timeout)
Step 2: Verify ngrok Tunnel
Check tunnel running:
# Should show ngrok processps aux | grep ngrok
# If not running, start itngrok http 8000Check tunnel URL:
# Visit ngrok web UIopen http://127.0.0.1:4040
# Or get URL programmaticallycurl http://127.0.0.1:4040/api/tunnels | jq '.tunnels[0].public_url'Match URL in PactaPay:
- PactaPay webhook URL:
https://abc123.ngrok-free.app/api/commerce/webhooks/pactapay - ngrok tunnel URL:
https://abc123.ngrok-free.app - β Should match exactly (ngrok URLs change on restart)
β οΈ Important: ngrok free URLs are not persistent. Every restart generates a new URL that must be updated in PactaPay dashboard.
Step 3: Verify Local Server
Server running:
# Check processps aux | grep "python.*manage.py"
# Or for uvicornps aux | grep uvicorn
# Check portlsof -i :8000
# Test health endpointcurl http://localhost:8000/health# Should return: {"status": "ok"}Endpoint exists:
# Check route registered in OpenAPI docscurl http://localhost:8000/docs | grep "webhooks"
# Test endpoint directly (local)curl -X POST http://localhost:8000/api/commerce/webhooks/pactapay \ -H "Content-Type: application/json" \ -d '{"event":"test"}'# Should return: 200 or 401 (signature validation)Step 4: Check ngrok Request Inspector
Open Inspector:
open http://127.0.0.1:4040/inspect/httpLook for incoming requests:
- β If you see requests β Tunnel working, check response status
- β If no requests β Webhook not reaching ngrok (check PactaPay config)
Inspect request details:
- URL: Should match your endpoint path exactly
- Headers: Check
X-Pactapay-Signaturepresent - Body: Should be valid JSON
- Response: Check status code
200= Success401= Signature validation failed500= Server error- Timeout = Handler exceeded 10 seconds
Replay requests:
- Click on a past request
- Click Replay to resend identical webhook
- Edit request body/headers if needed for testing
Step 5: Check Server Logs
View logs:
# Dockerdocker logs backend -f
# Localtail -f logs/app.log
# Look for:# - "Webhook received" (confirms request arrived)# - "Invalid signature" (signature validation failed)# - "Exception" or "ERROR" (handler crashed)Common log patterns:
Pattern 1: No logs
# No webhook-related logs at allβ Problem: Request not reaching serverβ Check: ngrok tunnel, endpoint path, server runningPattern 2: Signature validation fails
ERROR: Invalid webhook signatureβ Problem: Signature mismatchβ Check: PACTAPAY_WEBHOOK_SECRET matches dashboardPattern 3: Exception in handler
ERROR: Exception in webhook handler: KeyError: 'payment_id'β Problem: Business logic crashedβ Check: Handler code, database connection, missing fieldsCommon Issues & Solutions
Issue 1: Webhook Not Delivered
Symptoms:
- No requests in ngrok inspector
- No logs in server
- PactaPay shows βDelivery failedβ
Diagnosis:
# Check PactaPay delivery logs# Dashboard β Webhooks β Click on endpoint β Delivery logs# Common errors:# - DNS resolution failed# - Connection refused# - Connection timeoutSolutions:
A. ngrok URL changed
# Get current URLcurl http://127.0.0.1:4040/api/tunnels | jq '.tunnels[0].public_url'
# Update in PactaPay dashboard# Webhooks β Edit endpoint β Update URLB. ngrok tunnel closed
# Restart tunnelpkill ngrokngrok http 8000
# Copy new URL and update in PactaPay dashboardC. Firewall blocking
# Check firewall status (macOS)sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
# Allow ngrok through firewallsudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/local/bin/ngrokD. Server not reachable
# Verify server responds to ngrokcurl https://YOUR_SUBDOMAIN.ngrok-free.app/health
# If timeout or connection refused:# - Check server is running (lsof -i :8000)# - Check ngrok is forwarding to correct port# - Restart both server and ngrokIssue 2: Signature Validation Fails
Symptoms:
- Webhook arrives (seen in ngrok inspector)
- Server returns
401 Unauthorized - Logs show βInvalid signatureβ or βSignature verification failedβ
Diagnosis:
# Check webhook secret configuredecho $PACTAPAY_WEBHOOK_SECRET
# Check .env filecat .env | grep WEBHOOK_SECRET
# Verify matches PactaPay dashboard# Dashboard β Webhooks β Click endpoint β "Signing secret"Solutions:
A. Secret mismatch
# Copy secret from PactaPay dashboard# Update .envecho "PACTAPAY_WEBHOOK_SECRET=whsec_xyz789" >> .env
# Restart server to reload environmentdocker-compose restart backendB. Secret not loaded
# Verify environment variable loading in your appfrom dotenv import load_dotenvimport os
load_dotenv()
# Debug: Print secret (remove in production!)print(f"Webhook secret: {os.getenv('PACTAPAY_WEBHOOK_SECRET')}")# Should NOT be NoneC. Signature algorithm wrong
# PactaPay uses HMAC-SHA256import hmacimport hashlib
def verify_signature(payload: str, signature: str, secret: str) -> bool: """Verify webhook signature using HMAC-SHA256""" expected = hmac.new( secret.encode(), payload.encode(), hashlib.sha256 ).hexdigest()
# Compare with constant-time comparison to prevent timing attacks return hmac.compare_digest(signature, f"sha256={expected}")
# β Common mistake: Using MD5 or SHA1 instead of SHA256D. Payload modified before validation
# β WRONG: Parsing JSON before validation@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): data = await request.json() # This modifies payload! signature = request.headers.get("X-Pactapay-Signature") verify_signature(request.body, signature, secret) # FAILS!
# β
CORRECT: Use raw body for validation@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): raw_body = await request.body() signature = request.headers.get("X-Pactapay-Signature")
# Verify signature on raw bytes verify_signature(raw_body.decode(), signature, secret)
# Parse JSON only after verification data = json.loads(raw_body)E. Signature header format mismatch
# Check exact header format# PactaPay sends: X-Pactapay-Signature: sha256=abc123...# Stripe sends: Stripe-Signature: t=timestamp,v1=signature
# Extract signature correctly:signature_header = request.headers.get("X-Pactapay-Signature")
# If format is "sha256=abc123", extract the hash partif signature_header.startswith("sha256="): signature = signature_header.split("=", 1)[1]else: signature = signature_headerIssue 3: Webhook Timeout
Symptoms:
- PactaPay shows βTimeout after 10sβ
- ngrok shows request started but no response
- Server logs show request received but no βCompletedβ log
Cause: Webhook handlers must respond within 10 seconds. Long-running operations cause timeouts.
Diagnosis:
# Add timing logs to your handler# logger.info("Webhook handler started")# ... processing ...# logger.info("Webhook handler completed")
# If "completed" never appears β Handler is hangingSolutions:
A. Long-running operation in handler
# β WRONG: Blocking operation in handler@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): payment = process_payment(request) send_confirmation_email(payment) # Takes 5 seconds! generate_pdf_invoice(payment) # Takes 3 seconds! return {"received": True} # Timeout!
# β
CORRECT: Use background tasksfrom fastapi import BackgroundTasks
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request, bg: BackgroundTasks): payment = process_payment(request) # Fast operation
# Queue slow operations as background tasks bg.add_task(send_confirmation_email, payment) bg.add_task(generate_pdf_invoice, payment)
return {"received": True} # Return immediately (under 1s)B. Database deadlock or slow query
# Check for long-running database queriespsql $DATABASE_URL -c "SELECT * FROM pg_stat_activity WHERE state = 'active' AND query_start < NOW() - INTERVAL '5 seconds';"
# Kill deadlocked queriespsql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'active' AND query_start < NOW() - INTERVAL '10 seconds';"# Add query timeout to database connectionsfrom sqlalchemy import create_engine
engine = create_engine( DATABASE_URL, connect_args={ "options": "-c statement_timeout=5000" # 5 second timeout })C. External API call timeout
# Add timeout to external HTTP callsimport httpx
async def fetch_payment_details(payment_id: str): async with httpx.AsyncClient(timeout=3.0) as client: # 3 second timeout try: response = await client.get(f"https://api.pactapay.com/payments/{payment_id}") return response.json() except httpx.TimeoutException: logger.error(f"Payment API timeout for {payment_id}") return None # Handle gracefullyD. Synchronous operation causing blocking
# β WRONG: Synchronous blocking operationimport time
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): time.sleep(15) # Blocks async event loop! return {"received": True}
# β
CORRECT: Use async operationsimport asyncio
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): await asyncio.sleep(1) # Non-blocking return {"received": True}Issue 4: Duplicate Webhooks
Symptoms:
- Payment processed twice
- Emails sent multiple times
- Multiple database records for same event
- Multiple logs for same webhook ID
Cause: PactaPay retries failed webhooks (up to 3 times with exponential backoff) if your server:
- Returns non-2xx status code
- Times out
- Has connection error
Solutions:
A. Idempotency checks with in-memory storage
# Simple in-memory deduplication (single process only)processed_webhooks = set()
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): data = await request.json() webhook_id = data['id'] # PactaPay sends unique 'id' field
# Check if already processed if webhook_id in processed_webhooks: logger.info(f"Duplicate webhook ignored: {webhook_id}") return {"received": True, "status": "duplicate"}
# Process webhook process_payment(data)
# Mark as processed processed_webhooks.add(webhook_id) return {"received": True, "status": "processed"}B. Database-backed idempotency (production-ready)
# Create table for webhook tracking# CREATE TABLE processed_webhooks (# webhook_id VARCHAR(255) PRIMARY KEY,# processed_at TIMESTAMP DEFAULT NOW()# );
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request, db: Session = Depends(get_db)): data = await request.json() webhook_id = data['id']
# Check if webhook already processed (atomic) existing = db.query(ProcessedWebhook).filter_by(webhook_id=webhook_id).first() if existing: logger.info(f"Duplicate webhook: {webhook_id} (processed at {existing.processed_at})") return {"received": True, "status": "duplicate"}
# Process webhook try: process_payment(data)
# Record as processed db.add(ProcessedWebhook(webhook_id=webhook_id)) db.commit()
return {"received": True, "status": "processed"} except Exception as e: db.rollback() logger.error(f"Webhook processing failed: {e}") raiseC. Database unique constraints
# Ensure payment_id is unique to prevent duplicatesfrom sqlalchemy import Column, String, Integer, UniqueConstraint
class Payment(Base): __tablename__ = "payments"
id = Column(Integer, primary_key=True) payment_id = Column(String, unique=True, index=True, nullable=False) # Unique constraint amount = Column(Integer, nullable=False) status = Column(String, nullable=False)
# On duplicate insert, catch exceptionfrom sqlalchemy.exc import IntegrityError
try: payment = Payment(payment_id=data['payment_id'], amount=data['amount']) db.add(payment) db.commit()except IntegrityError: db.rollback() logger.warning(f"Duplicate payment prevented: {data['payment_id']}") return {"received": True, "status": "duplicate"}D. Redis-based idempotency (distributed systems)
import redis
redis_client = redis.Redis(host='localhost', port=6379, db=0)
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): data = await request.json() webhook_id = data['id']
# Try to set webhook ID with expiration (24 hours) # Returns True only if key doesn't exist (NX flag) is_new = redis_client.set( f"webhook:{webhook_id}", "processed", ex=86400, # Expire after 24 hours nx=True # Only set if not exists )
if not is_new: logger.info(f"Duplicate webhook: {webhook_id}") return {"received": True, "status": "duplicate"}
# Process webhook process_payment(data) return {"received": True, "status": "processed"}Issue 5: Business Logic Errors
Symptoms:
- Webhook processed successfully (200 OK)
- Signature validation passes
- But payment status incorrect or data missing
Solutions:
A. Missing or null fields
# Add defensive field validation@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): data = await request.json()
# Validate required fields required_fields = ['id', 'event', 'payment_id', 'amount'] missing = [f for f in required_fields if f not in data]
if missing: logger.error(f"Webhook missing fields: {missing}") return {"received": True, "error": f"Missing fields: {missing}"}
# Validate types if not isinstance(data['amount'], int) or data['amount'] <= 0: logger.error(f"Invalid amount: {data['amount']}") return {"received": True, "error": "Invalid amount"}
# Process only after validation process_payment(data)B. Event type handling
# Handle all event types explicitly@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): data = await request.json() event_type = data.get('event')
if event_type == 'payment.completed': handle_payment_completed(data) elif event_type == 'payment.failed': handle_payment_failed(data) elif event_type == 'payment.refunded': handle_payment_refunded(data) else: logger.warning(f"Unknown event type: {event_type}") return {"received": True, "warning": f"Unknown event: {event_type}"}
return {"received": True, "status": "processed"}Testing Webhooks Locally
Manual Test (No Signature)
# Send test webhook to local servercurl -X POST http://localhost:8000/api/commerce/webhooks/pactapay \ -H "Content-Type: application/json" \ -H "X-Pactapay-Signature: sha256=test_signature" \ -d '{ "id": "evt_test123", "event": "payment.completed", "payment_id": "pay_test123", "amount": 5000, "currency": "USD" }'With Valid Signature (Python)
#!/usr/bin/env python3"""Generate valid webhook signature for testing"""import hmacimport hashlibimport json
# Your webhook payloadpayload = { "id": "evt_test123", "event": "payment.completed", "payment_id": "pay_test123", "amount": 5000}
# Serialize to JSON (no spaces, sorted keys for consistency)payload_str = json.dumps(payload, separators=(',', ':'), sort_keys=True)
# Your webhook secret from .envsecret = "whsec_your_secret_here"
# Compute HMAC-SHA256 signaturesignature = hmac.new( secret.encode(), payload_str.encode(), hashlib.sha256).hexdigest()
print(f"Payload: {payload_str}")print(f"Signature: sha256={signature}")print(f"\nCurl command:")print(f"""curl -X POST http://localhost:8000/api/commerce/webhooks/pactapay \\ -H "Content-Type: application/json" \\ -H "X-Pactapay-Signature: sha256={signature}" \\ -d '{payload_str}'""")With Valid Signature (Bash)
#!/bin/bashPAYLOAD='{"id":"evt_test","event":"payment.completed","payment_id":"pay_test"}'SECRET="whsec_your_secret"
# Compute HMAC-SHA256 signatureSIGNATURE=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $2}')
echo "X-Pactapay-Signature: sha256=$SIGNATURE"echo ""echo "Curl command:"echo "curl -X POST http://localhost:8000/api/commerce/webhooks/pactapay \\"echo " -H 'Content-Type: application/json' \\"echo " -H 'X-Pactapay-Signature: sha256=$SIGNATURE' \\"echo " -d '$PAYLOAD'"Using ngrok Inspector
- Open http://127.0.0.1:4040/inspect/http
- Find a past webhook request
- Click Replay to resend identical request
- Edit request body/headers if needed
- Click Replay again to test
Advantages:
- Uses real PactaPay webhook format
- Includes correct signature
- Can modify payload for testing edge cases
Security Best Practices
1. Always Verify Signatures
# NEVER skip signature validation, even in development@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): # β NEVER DO THIS # if os.getenv("ENV") == "development": # # Skip validation in dev # pass
# β
ALWAYS validate raw_body = await request.body() signature = request.headers.get("X-Pactapay-Signature") secret = os.getenv("PACTAPAY_WEBHOOK_SECRET")
if not verify_signature(raw_body.decode(), signature, secret): raise HTTPException(status_code=401, detail="Invalid signature")
# Process webhook...2. Use Constant-Time Comparison
import hmac
# β
CORRECT: Prevents timing attacksdef verify_signature(payload: str, signature: str, secret: str) -> bool: expected = compute_signature(payload, secret) return hmac.compare_digest(signature, expected)
# β WRONG: Vulnerable to timing attacksdef verify_signature_unsafe(payload: str, signature: str, secret: str) -> bool: expected = compute_signature(payload, secret) return signature == expected # Leaks timing information!3. Validate Timestamp (Replay Attack Prevention)
import timefrom datetime import datetime, timedelta
@router.post("/webhooks/pactapay")async def handle_webhook(request: Request): data = await request.json()
# Check timestamp if provided if 'timestamp' in data: webhook_time = datetime.fromisoformat(data['timestamp']) age = datetime.utcnow() - webhook_time
# Reject webhooks older than 5 minutes if age > timedelta(minutes=5): logger.warning(f"Webhook too old: {age}") raise HTTPException(status_code=400, detail="Webhook expired")4. Rate Limiting
from fastapi_limiter import FastAPILimiterfrom fastapi_limiter.depends import RateLimiter
@router.post( "/webhooks/pactapay", dependencies=[Depends(RateLimiter(times=100, seconds=60))] # 100 req/min)async def handle_webhook(request: Request): # Process webhook... passDebug Checklist
When webhook fails, verify:
- β PactaPay webhook endpoint configured correctly
- β
ngrok tunnel running (
ps aux | grep ngrok) - β ngrok URL matches PactaPay config
- β
Local server running (
lsof -i :8000) - β
Endpoint exists (
curl http://localhost:8000/docs) - β ngrok shows incoming request (http://127.0.0.1:4040)
- β Server logs show request received
- β Signature validation passes
- β Handler returns within 10 seconds
- β Response status 200
Automated Debug Script
Save as webhook_debug.sh:
#!/bin/bash# Automated webhook debugging script
echo "=== Webhook Debug Report ==="echo ""echo "Generated: $(date)"echo ""
echo "1. ngrok tunnel status:"if pgrep -f ngrok > /dev/null; then echo " β
ngrok is running" TUNNEL_URL=$(curl -s http://127.0.0.1:4040/api/tunnels | jq -r '.tunnels[0].public_url // "Not available"') echo " URL: $TUNNEL_URL"else echo " β ngrok is NOT running"fiecho ""
echo "2. Local server status:"if lsof -i :8000 > /dev/null 2>&1; then echo " β
Server is running on port 8000" HEALTH=$(curl -s http://localhost:8000/health || echo "Health check failed") echo " Health: $HEALTH"else echo " β No server running on port 8000"fiecho ""
echo "3. Environment variables:"if [ -f .env ]; then echo " β
.env file exists" if grep -q "PACTAPAY_WEBHOOK_SECRET" .env; then echo " β
PACTAPAY_WEBHOOK_SECRET configured" else echo " β PACTAPAY_WEBHOOK_SECRET not found in .env" fielse echo " β οΈ .env file not found"fiecho ""
echo "4. Recent server logs (last 20 lines):"if docker ps | grep -q backend; then docker logs backend 2>&1 | tail -20else echo " β οΈ Docker container 'backend' not running"fiecho ""
echo "5. ngrok recent requests (last 3):"curl -s http://127.0.0.1:4040/api/requests/http | jq -r '.requests[0:3] | .[] | "\(.StartTime) \(.Method) \(.Uri) -> \(.Response.StatusCode)"' 2>/dev/null || echo " β οΈ No requests in ngrok inspector"echo ""
echo "=== Debug report complete ==="Run with:
chmod +x webhook_debug.sh./webhook_debug.shGetting Help
If webhook still fails after debugging:
-
Collect diagnostics:
Terminal window ./webhook_debug.sh > webhook-debug.log -
Export ngrok request:
- Open http://127.0.0.1:4040/inspect/http
- Right-click on failed request
- Select βExport as cURLβ
- Save to file
-
Collect server logs:
Terminal window docker logs backend > backend.log 2>&1 -
Create GitHub issue with:
webhook-debug.log- cURL export from ngrok
backend.log- Description of expected vs actual behavior
Related Documentation
- Tunnel Configuration Guide
- Commerce API Reference
- Common Errors Troubleshooting
- Webhook Security Best Practices
Appendix: Webhook Flow Diagram
ββββββββββββββββ PactaPay ββ Service βββββββββ¬βββββββ β β POST /api/commerce/webhooks/pactapay β Headers: X-Pactapay-Signature β Body: {"event": "payment.completed", ...} β βΌββββββββββββββββ ngrok ββ Tunnel β https://abc123.ngrok-free.appββββββββ¬βββββββ β β Forwards to localhost:8000 β βΌββββββββββββββββ FastAPI ββ Server β localhost:8000ββββββββ¬βββββββ β β 1. Read raw request body β 2. Verify HMAC-SHA256 signature β 3. Parse JSON payload β βΌββββββββββββββββ Webhook ββ Handler βββββββββ¬βββββββ β β 4. Validate event type β 5. Check idempotency β 6. Update database β 7. Queue background tasks β βΌββββββββββββββββ Response ββ 200 OK β {"received": true}βββββββββββββββLast Updated: February 2026 Version: 1.0 Maintainer: Seed & Source Team