When it comes to scaling VPN deployments, particularly dynamic ones like DMVPN and FlexVPN, traditional pre-shared keys (PSKs) quickly become impractical and insecure. Enter PKI (Public Key Infrastructure) and certificate-based authentication, which offers stronger security, easier management, and simplified key rotation.

Why Use PKI Instead of Pre-Shared Keys?

The use of rsa-sig in IKE (ISAKMP) policies allows for certificate-based authentication between peers. This is especially valuable in large-scale environments, like DMVPN deployments, where managing dozens or hundreds of static pre-shared keys is cumbersome and insecure.

An all-too-common insecure shortcut is to use a wildcard key like:

crypto isakmp key password123 address 0.0.0.0

This opens up your router to connection attempts from anyone. Instead, using certificates tied to trusted Certificate Authorities (CAs) ensures mutual authentication and better control over who can initiate VPN sessions.

How PKI-Based VPN Authentication Works

Each router is configured with an RSA key pair, consisting of a private and public key. During setup, the two routers exchange public keys, which are then used for encrypting and decrypting authentication messages. The sending router encrypts using the recipient’s public key, and the recipient decrypts with its private key.

To secure this initial public key exchange and avoid spoofing, both routers trust a common Certificate Authority (CA). Each router must obtain a certificate from the CA using a Certificate Signing Request (CSR). This certificate includes metadata such as the subject name, expiration date, and the public key.

The certificate format typically includes:

  • A unique serial number
  • Distinguished Name (DN) of the subject (Common Name, Organization, Country)
  • Issuer DN (the CA)
  • Extended Key Usage (EKU) to define the purpose (e.g., VPN, server auth)
  • Validity period
  • The subject’s public key
  • A CA-issued digital signature binding it all together

When authenticating, routers exchange certificates and validate them using:

  • CRL or OCSP for revocation status
  • Signature check to verify CA authenticity
  • Validity date check (beware of mismatched clocks)
  • Common Name / SAN match to confirm identity

The Authentication Exchange (Main Mode Message 5)

When routers connect, one sends a certificate embedded in Main Mode message #5. This message contains an ID payload (usually a Fully Qualified Domain Name or Distinguished Name) and a hash encrypted with the initiator’s private key. The entire message is encrypted with a derived session key (SKEYID_e).

Here’s how the responder processes the message:

  1. Receives MM#5 and decrypts it using SKEYID_e
  2. Extracts and validates the certificate (checks revocation, signature, and validity)
  3. Retrieves the public key from the cert
  4. Uses the public key to decrypt the sender’s hash
  5. Verifies the hash and confirms the identity

This chain of trust ensures that both endpoints are authenticated securely using their issued certificates.

PKI Configuration and Requirements

On the configuration side, here’s the core command to enable certificate-based IKE authentication:

crypto isakmp policy 20
authentication rsa-sig

Each router must have:

  • Its own public and private key (Pub_Key_R, Priv_Key_R)
  • A valid certificate issued by a trusted CA (CA_Cert)

Setting Up a Router as a Certificate Authority (IOS CA)

For environments without a centralized PKI, Cisco allows routers to act as Certificate Authorities via IOS CA. Routers can enroll directly with another router acting as a CA. This enables smaller deployments to benefit from certificate-based authentication without external PKI infrastructure.

Conclusion

Using PKI for VPN authentication isn’t just about improved security; it’s also about long-term manageability. With certificate-based authentication, rotating keys becomes as easy as revoking and reissuing certs. It also provides better logging, traceability, and resilience against man-in-the-middle attacks compared to pre-shared keys. For dynamic environments like DMVPN or when scaling site-to-site VPNs, certificate-based VPNs with rsa-sig should be the gold standard.