Cryptographic Verification for Trusted Bots

Not all bots are malicious. Search engines, legitimate crawlers, and trusted partners need reliable ways to prove their identity to your systems. But how can you differentiate a legitimate Google crawler from a sophisticated scraper that’s spoofing its user agent?

WebDecoy’s Cryptographic Verification Feature answers this question using industry-standard protocols that allow bots to cryptographically prove their legitimacy—without requiring usernames and passwords.

The Problem: Bots That Claim to Be Legitimate

Traditional bot detection relies on pattern matching:

• Does the user agent match a known bot signature?
• Does the IP reverse-DNS to a legitimate domain?
• Does the behavior match known bot patterns?

But these signals can be spoofed. A sophisticated attacker can:

• Spoof a Google user agent
• Route traffic through legitimate IP ranges
• Mimic human-like behavior patterns
• Use residential proxies to bypass reputation checks

How do you know if a request from Googlebot is actually from Google, or from an attacker impersonating Google?

That’s where cryptographic verification comes in.

How Cryptographic Verification Works

Instead of trusting metadata (which can be forged), WebDecoy verifies cryptographic signatures and tokens that prove a bot’s identity mathematically.

RFC 9421: HTTP Message Signatures

A bot can sign its request using cryptographic algorithms. Your server then verifies the signature using the bot’s public key.

How it works:

Bot creates a request:
├── Adds HTTP headers: Signature, Signature-Input, Signature-Agent
├── Computes a cryptographic signature using its private key
└── Sends the signed request

Your server receives the request:
├── Extracts the signature from headers
├── Reconstructs what was signed (following RFC 9421 spec)
├── Verifies the signature using the bot's public key
└── If it matches → request is genuinely from that bot

Example signed request:

curl -X GET https://example.com/api \
  -H "Signature: sig1=:base64encodedSignature:; key=bot-key-2024-01" \
  -H "Signature-Input: (@method @authority @path);alg=Ed25519" \
  -H "Signature-Agent: trusted-bot-v1"

Key benefit: Even if an attacker intercepts the request and modifies it, the signature will no longer match. They’d need the bot’s private key to create a valid signature, and that’s cryptographically infeasible.

RFC 9576: Privacy Pass Tokens

Privacy Pass tokens let bots prove legitimacy without revealing identity—useful when you want to trust a bot without knowing exactly who it is.

How it works:

Token Issuer (e.g., a bot network) creates a token:
├── Encodes bot identity, expiration, rate limits
├── Signs the token with their private key
└── Issues tokens to legitimate bots in their network

Bot includes token in request:
├── Adds Authorization header: "PrivacyPass <token>"
└── Adds Token-Issuer header

Your server verifies the token:
├── Checks if the issuer is trusted
├── Validates the token hasn't expired
├── Verifies rate limit compliance
└── If all checks pass → bot is legitimate

Example token request:

curl -X GET https://example.com/api \
  -H "Authorization: PrivacyPass eyJhbGci..." \
  -H "Token-Issuer: privacy-pass-provider.example.com"

Rate limiting built-in: Tokens support usage quotas. A token might be valid for 100 requests per hour. WebDecoy tracks usage and enforces limits automatically.

Trust Score System

Every cryptographic verification produces a trust score (0-100%):

Signature Verification Scores

• 100% - Valid Ed25519 signature from whitelisted agent (highest confidence)
• 50% - Valid signature but agent not yet whitelisted
• 0% - No signature provided
• -100% - Signature verification failed (tampering detected)

Token Verification Scores

• 100% - Valid token from trusted issuer, not expired, rate limit OK
• 50% - Valid token but issuer not yet whitelisted
• 0% - No token provided
• -100% - Token verification failed (expired, invalid, rate limit exceeded)

Combined Trust Score

When both signature and token are present, scores are averaged for a comprehensive trust assessment.

Real-World Scenarios

Scenario 1: Google’s Crawlers

Google wants to crawl your site, but you want to verify it’s actually Google, not an imposter:

Traditional approach:

• Check user agent for “Googlebot” ✓ (easily spoofed)
• Check IP reverse-DNS → google.com ✓ (can be spoofed with BGP hijacking)
• Problem: An attacker could spoof both

Cryptographic approach:

• Google signs requests with their private key ✓
• You verify using Google’s public key ✓
• Only Google can create valid signatures (they own the private key)
• Result: 100% confidence it’s Google

Scenario 2: Partner API Consumer

A partner bot needs to access your API thousands of times per day:

Traditional approach:

• Rate limiting by IP? (Partner rotates IPs)
• Rate limiting by user agent? (Easy to spoof)
• API key? (Have to implement auth yourself)

Cryptographic approach:

• Issue a Privacy Pass token to the partner with a quota of 10,000 requests/day
• Token includes expiration (30 days) and rate limit
• Bot includes token in each request
• Server verifies token and enforces quota automatically
• Result: Legitimate partner gets reliable access; quotas enforced cryptographically

Scenario 3: Detecting Impersonation

An attacker tries to impersonate a legitimate bot:

Attack: Spoof user agent, IP, and behavioral patterns Detection:

• Attacker includes fake “Signature” header
• Server tries to verify signature using bot’s public key
• Verification fails (attacker doesn’t have the private key)
• Result: Detection fails with -100% trust score; request is blocked

WebDecoy’s Implementation

Where It Works

Cryptographic verification applies to the honeypot/decoy layer—the endpoints and interactions WebDecoy monitors. It helps you:

• Distinguish legitimate bots from malicious ones in your honeypot data
• Track which trusted bots are accessing your site
• Enforce rate limits on bot access
• Build allowlists of cryptographically-verified bots

Detection Enrichment Pipeline

When a request hits your honeypot, WebDecoy’s enrichment pipeline processes it in 4 steps:

1. Reverse DNS lookup      (Is the IP's domain legitimate?)
2. AbuseIPDB enrichment    (Is this IP known to be malicious?)
3. Cryptographic verification (Can this bot prove identity?)
4. Store enriched detection (Save all data for analysis)

Cryptographic verification is step 3—a non-blocking async operation that runs in the background.

Storage and Display

All verification results are stored with the detection:

{
  "cryptographic_verification": {
    "signature": {
      "agent": "googlebot-v3",
      "algorithm": "Ed25519",
      "verified": true,
      "key_id": "google-public-key-2024"
    },
    "token": {
      "issuer": "partner-api.example.com",
      "expires_at": "2025-12-31T23:59:59Z",
      "verified": true,
      "rate_limit_used": 45,
      "rate_limit": 100
    },
    "trust_score": 0.95
  }
}

In your detection dashboard, you can see:

• ✓ Verified signatures with agent names and algorithms
• ✓ Token status with expiration and rate limit usage
• ✓ Combined trust scores with color-coded confidence levels
• ✓ Verification errors with details if verification failed

Why Standards Matter

WebDecoy implements IETF-standard protocols:

• RFC 9421: HTTP Message Signatures - Industry-standard format for signing HTTP requests
• RFC 9576: Privacy Pass Protocol - Privacy-preserving token standard used by Cloudflare, Apple, and others
• RFC 9578: ARC Extension - Rate limiting capabilities for tokens

Using standards means:

1. Interoperability - Bots built by anyone following the standard can verify with your system
2. Security review - Standards are vetted by cryptography experts
3. Future-proof - As the standard evolves, implementations update to match
4. No lock-in - You’re not dependent on proprietary formats

Getting Started: Whitelisting Trusted Bots

To trust a bot, you add its public key to your whitelist:

Trusted Agents (Signature Verification):
├── googlebot-v3: "ed25519 public key (base64)"
├── bingbot-v2: "ed25519 public key (base64)"
└── partner-crawler-v1: "ed25519 public key (base64)"

Trusted Issuers (Token Verification):
├── privacy-pass-provider.example.com
└── partner-api.example.com

WebDecoy validates every request against these whitelists. Verified bots show high trust scores; unverified or failed verifications show low scores.

Benefits for Your Security

1. Reduce False Positives

Previously, legitimate bots might get caught in your detection net because they looked suspicious. Now, if they’re cryptographically verified, you know they’re safe—even if their behavior is unusual.

2. Selective Allowlisting

Instead of allowing all of Google’s IP range (which could include malicious actors using Google’s infrastructure), you allow only Google’s specifically-signed requests.

3. Rate Limiting Without Friction

Token-based rate limiting is automatic and cryptographically enforced. No manual IP allowlisting needed.

4. Privacy-Preserving Trust

Privacy Pass tokens let you verify a bot is legitimate without knowing exactly which bot it is. Perfect for bot networks and federated systems.

5. Audit Trail

Every cryptographic verification is logged with details:

• Which agent/issuer was verified
• What algorithm was used
• Whether verification succeeded or failed
• Timestamp and context

Limitations and Future Work

Current limitations:

• Ed25519 signatures are fully implemented; RSA and ECDSA are planned
• Whitelists are currently in-memory (will be moved to database for hot updates)
• No automatic certificate rotation detection yet

Coming soon:

• Database-backed whitelists for dynamic updates without restart
• UI dashboard for managing trusted agents and issuers
• Automatic key rotation detection
• Additional Privacy Pass extensions

Security Best Practices

1. Protect public keys - Even though they’re public, rotate them regularly and distribute securely
2. Validate token expiration - Always check that tokens haven’t expired
3. Enforce rate limits - Use token quotas to prevent abuse
4. Audit whitelist changes - Track who adds/removes trusted agents
5. Monitor verification failures - Repeated verification failures might indicate an attack

Conclusion

Cryptographic verification transforms bot detection from pattern-matching guesswork into mathematical proof. A verified bot has proven its identity using cryptography—there’s no ambiguity.

For legitimate bots, search engines, and trusted partners, this means:

• Reliable, friction-free access to your site
• No false blocks from aggressive detection systems
• Rate limiting that’s fair and transparent
• Privacy-preserving verification when needed

For your security team, this means:

• Distinguishing legitimate bots from impostors
• Enforcing policies with cryptographic certainty
• Reducing false positives and noise
• Building trusted relationships with bot operators

Ready to implement cryptographic bot verification?

• WebDecoy’s verification features are available in our latest release
• Works with RFC 9421 and RFC 9576 standard protocols
• No proprietary formats or lock-in required
• Integrates seamlessly with your existing honeypot infrastructure

WebDecoy. Verify bots. Trust data.

Share this post

Like this post? Share it with your friends!

• Share on Facebook, opens in a new tab
• Share on Twitter, opens in a new tab
• Share on LinkedIn, opens in a new tab