From Network Security Wiki


  • The VPN security model provides:
Confidentiality : Encryption
 - Symmetric (secret) key. Same key is used for encryption and decryption. Widely used for bulk data encryption. Key management becomes a huge issue.
 - Asymmectric (public & private) key : private key is used for encryption, mathematically related public key is used for decryption. Mostly used only to transmit session keys
Authentication : HMAC "Hashed Method Authentication Code"
Integrity : Hashing
  • Phase 1 negotiates 5 things:
Hashing, Authentication, Group(DH 1,2,5), Lifetime, Encryption

Phase 1.5


Phase 2 Quick Mode

        This section needs verification or testing!
  • Which phase has what?
Phase 1 Phase 2
DH Algorithm PFS (new DH, new Nonce)
Nonce calc 4 Proposals
4 Proposals Phase 1 Gateway Name
IKE version 1 or 2 Replay Protection
Remote Gateway IP address/Peer ID/Dialup user/Dialup Group Transport mode
Preshared Key Bind to Tunnel Interface/Zone
Outgoing Interface Proxy-ID Check
NAT-Traversal DSCP Markings (only if Bind to Tunnel Interface enabled)
Peer Status Detection
HeartBeat VPN Group
DPD VPN Monitor/Rekey
  • How to capture VPN traffic? What port/protocol numbers do you see in packets of VPN Traffic?
Packet type Filter settings
Phase 1 & 2 negotiations UDP Port 500
NAT Traversal UDP Port 4500 (IPSec over UDP) (NAT-T Draft 2)

TCP Port 10000 (Cisco proprietary)

ESP Protocol no 50
AH Protocol no 51
GRE Protocol no 47
  • What is ISAKMP?
        This section is under construction.
  • What is IPSEC?

It combines three main protocols to form a cohesive security framework:

  • What is IKE?
Provides a framework for negotiating security parameters and establishing authenticated keys.

IKE's responsibilities in the IPsec protocol are:

* Negotiating protocol parameters
* Exchanging public keys
* Authenticating both sides
* Managing keys after the exchange

IKE solves the problems of manual and unscalable IPsec implementation by automating the entire key-exchange process. This is one of IPsec's critical requirements. IKE, like IPsec, is also a combination of three different protocols:

* SKEME- Provides a mechanism for using public key encryption for authentication purposes.
* Oakley- Provides a mode-based mechanism for arriving at an encryption key between two IPsec peers.
* ISAKMP- Defines the architecture for message exchange, including packet formats and state transitions between two IPsec peers.

IKE combines these three protocols in one framework to provide IPsec these facilities

  • What is difference between ESP & AH?


Protocol no 50
Header can be modified, because all payload is encapsulated (important for NAT)


No Encryption
Protocol no 51
Cannot pass through NAT in transport mode, because the original header would be modified.
Faster than ESP, but not as secure (no encryption)
  • What is XAuth?
XAuth allows another layer of authentication for VPN between a Remote Client and a Juniper Firewall VPN device. 
Allows verifying auth to the firewall device's local auth database using a Radius, Secure ID, or LDAP server. 
Groups may be used to combine the dial-up users or use individual dial-up users. 
Grouping cannot be used if using Secure-ID or an LDAP server.
Occurs after Phase 1 and before Phase 2.
Starts from 4th Packet(after 3 packets of Phase 1).
Available in both Main & Aggressive mode.



  • Hash Message Authentication Code
  • This is the only place in the communication where a PSK is used
  • Sender adds hashed Pre Shared Key to data
  • Receiver appends same hashed key to data before sending back
  • Hashes are compared
  • Key is never transmitted along with the data


        This complete sub-section needs reformatting and simplification
  • Traditionally, IPSec does not work when traversing across a device doing NAT.
  • NAT-T is an IKE phase 1 algorithm.
  • Both VPN end-points must support NAT-T.
  • NAT-T IPSec peers first detect if there is a NAT device between them.
  • This is done in the key negotiation process
  • It requires additional messages to be sent.
  • With pre-shared keys, the network behind the NAT device needs to initiate the IPSec negotiations.
  • The responder (non-NAT device), checks if the remote identity matches one of the locally configured interface addresses and IKE UDP port number (responder NAT check).
  • The responder also checks if the local identity matches the source IP address and UDP port in the receiving packet (initiator NAT check).
  • If both of these two conditions are not met, the responder then knows NAT-T is required.
  • After two IPSec peers agree that NAT-T is needed, IPSec packets between them will be encapsulated by one new and two extra headers such that even if the IP packet on the wire is altered by NAT device(s), there is enough information in the extra headers and SPI to recover the original IPSec packet.
  • Original IPsec packet
[Original outer IP header][ESP header][Original inner IP header][IP data][ESP trailer][ESP auth]
  • NAT-T encapsulated IPsec packet
[New IP header][UDP header][NAT-T header][ESP header][Original inner IP header][IP data][ESP trailer][ESP auth]
  • The new IP header is the same as the original IPSec outer header except the protocol is changed from AH/ESP to UDP.
  • The UDP header has a source port of 500 and the destination port equals the public IKE port of the peer.
  • This makes the peer think this is an IKE packet.
  • NAT-T works only if the NAT devices along the route between the two IPSec gateways maintain the same address mapping since the last IKE negotiation.
  • The keepalive is sent to keep the NAT devices from removing the mapping entries.

  • Minimum requirements for NAT Traversal (NAT-T):
   One side of the Virtual Private Network (VPN) must NOT be behind a Network Address Translation (NAT) device.
   The non-NAT side must have a static IP address. 
   If the non-NAT side has a dynamically assigned IP address like DHCP or PPPoE, the NAT-Traversal VPN will not work.
   Both sides must support NAT-Traversal.
   The VPN device behind the NAT device must initiate the IPSec negotiations.
   For pre-shared key IKE VPN, both sides must be negotiating Aggressive Mode.
   The NAT device in front of the Juniper Firewall device must have IPSec Pass-through feature disabled.
  • The difference between ScreenOS 5.0 implementation and ScreenOS 5.1 and later is that NAT-T draft 2 can use UDP port 4500 for IKE negotiation when NAT is detected.
  • NAT-T draft 2 changes the UDP IKE SRC and DST port numbers from 500 to 4500.
  • This UDP header, using port 4500, is found between the outer IP and the ESP or AH header, thus changing the IPSec protocol field from 50 (ESP) or 51 (AH) to 17 (for UDP).
  • The IKE packet would appear as:
[IP Header] [UDP Header] [ISAKMP Header] [Payload]
  • The [ UDP Header], after NAT-T is detected, is broken out as:
[Src port 4500] [Dst port 4500] [Length] [Checksum] [Non-esp marker (0000)] [Payload] 
  • The non-ESP marker is 4 bytes of zero (0000), is added to the UDP segment to distinguish an encapsulated ISAKMP packet from an encapsulated ESP packet which does not have a marker.
  • The Firewall sends two MD-5 hashes in the Vendor ID payload during the first two exchanges of Phase 1 negotiations.
  • One hash for "draft 0" the other for "draft 2":
MD-5 hash of "draft-ietf-ipsec-nat-t-ike-00" 	 4485152d 18b6bbcd 0be8a846 9579ddcc
MD-5 hash of "draft-ietf-ipsec-nat-t-ike-02" 	 90cb8091 3ebb696e 086381b5 ec427b1f
  • Both VPN peers must send and receive at least one of these in order for NAT-T setup to continue.
  • If hashes for both drafts are sent and received, the Firewall will use NAT-T draft 2 implementation.
  • Screen OS 5.1 and later supports NAT-T based on draft-ietf-ipsec-nat-t-ike-02.txt and draft-ietf-ipsec-udp-encaps-02.txt, as well as version 0 of these drafts.

NAT Discovery

        This section needs re-formatting
Src-IP|Src-Port<===>Dst-IP|Dst-Port              #used to detect Natting at local side in VPN

Source: rfc3947.txt

  • The NAT-Traversal capability of the remote host is determined by an exchange of vendor ID payloads.
  • In the first two messages of Phase 1, the vendor id payload for this specification MUST be sent if supported (and it MUST be received by both sides) for the NAT-Traversal probe to continue.
  • The content of the payload is the MD5 hash of RFC 3947
  • The exact content in hex for the payload is
  • The NAT-D payload not only detects the presence of NAT between the two IKE peers, but also detects where the NAT is.
  • The location of the NAT device is important, as the keepalives have to initiate from the peer "behind" the NAT.
  • To detect NAT between the two hosts, we have to detect whether the IP address or the port changes along the path.
  • This is done by sending the hashes of the IP addresses and ports of both IKE peers from each end to the other.
  • If both ends calculate those hashes and get same result, they know there is no NAT between.
  • If the hashes do not match, somebody has translated the address or port.
  • This means that we have to do NAT-Traversal to get IPsec packets through.
  • If the sender of the packet does not know his own IP address (in case of multiple interfaces, and the implementation does not know which IP address is used to route the packet out), the sender can include multiple local hashes to the packet (as separate NAT-D payloads).
  • In this case, NAT is detected if and only if none of the hashes match.
  • The hashes are sent as a series of NAT-D (NAT discovery) payloads.
  • Each payload contains one hash, so in case of multiple hashes, multiple NAT-D payloads are sent.
  • In the normal case there are only two NAT-D payloads.
  • The NAT-D payloads are included in the third and fourth packets of Main Mode, and in the second and third packets in the Aggressive Mode.
  • The format of the NAT-D packet is
       1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
     | Next Payload  | RESERVED      | Payload length                |
     ~                 HASH of the address and port                  ~
  • The payload type for the NAT discovery payload is 20.
  • The HASH is calculated as follows:
  HASH = HASH(CKY-I | CKY-R | IP | Port)
  • The first NAT-D payload contains the remote end's IP address and port (i.e., the destination address of the UDP packet).
  • The remaining NAT-D payloads contain possible local-end IP addresses and ports (i.e., all possible source addresses of the UDP packet).
  • If there is no NAT between the peers, the first NAT-D payload received should match one of the local NAT-D payloads (i.e., the local NAT-D payloads this host is sending out), and one of the other NAT-D payloads must match the remote end's IP address and port.
  • If the first check fails (i.e., first NAT-D payload does not match any of the local IP addresses and ports), it means that there is dynamic NAT between the peers, and this end should start sending keepalives as defined in the [RFC3948] (this end is behind the NAT).
  • The CKY-I and CKY-R are the initiator and responder cookies.
  • They are added to the hash to make pre-computation attacks for the IP address and port impossible.
  • The following example is of a Phase 1 exchange using NAT-Traversal in Main Mode (authentication with signatures):
  Initiator                           Responder
  ------------                        ------------
  HDR, SA, VID -->
                                      <-- HDR, SA, VID
  HDR, KE, Ni, NAT-D, NAT-D -->
                                      <-- HDR, KE, Nr, NAT-D, NAT-D
  HDR*#, IDii, [CERT, ] SIG_I -->
                                      <-- HDR*#, IDir, [CERT, ], SIG_R

  • The following example is of Phase 1 exchange using NAT-Traversal in Aggressive Mode (authentication with signatures):
  Initiator                           Responder
  ------------                        ------------
  HDR, SA, KE, Ni, IDii, VID -->
                                      <-- HDR, SA, KE, Nr, IDir,
                                              [CERT, ], VID, NAT-D,
                                              NAT-D, SIG_R
  HDR*#, [CERT, ], NAT-D, NAT-D,
      SIG_I -->

The # sign indicates that those packets are sent to the changed port if NAT is detected.

  • NAT Traversal on Cisco Routers:
NAT Traversal PCAP


  • Next Hop Tunnel Binding helps to bind multiple IPsec VPN tunnels to a single tunnel interface.
  • To link a destination to one of the VPN tunnels bound to the same tunnel interface, two tables are used: Route Table and NHTB table.
  • The Firewall maps the Next-Hop Gateway IP address specified in the Route Table entry to a particular VPN tunnel specified in the NHTB table.
  • The Firewall uses the NHTB table to differentiate each tunnel.
  • With this method, a single tunnel interface can support many VPN tunnels.

NHTB Configuration:

SSG320(M)-> get config | i nhtb
 set interface tunnel.1 nhtb vpn "ssg20 vpn"
 set interface tunnel.1 nhtb vpn "ssg350 vpn"

Verify NHTB Configuration:

SSG320(M)-> get int tunnel.1
Interface tunnel.1(VSI):
  description tunnel.1
  number 20, if_info 655208, if_index 1, mode nat
  link ready
  vsys Root, zone Trust, vr trust-vr, vsd 0
  *ip  unnumbered, source interface ethernet1/1
  *manage ip
  bound vpn: 
    ssg20 vpn
    ns5gt vpn
  Next-Hop Tunnel Binding table
  Flag Status Next-Hop(IP)    tunnel-id  VPN
   S    D 0x00000001 ssg20 vpn
   S    U 0x00000003 ns5gt vpn

VPN Monitor

  • Juniper Proprietary
  • Uses ICMP echo requests which are sent within the tunnel(encrypted).
  • The VPN monitor is generally used to detect and report if the VPN is down.
  • When you enable VPN monitoring for a specific tunnel, the security device sends ICMP echo requests (or pings) via the tunnel at specified intervals (configured in seconds) to monitor network connectivity through the tunnel.
  • If the VPN Monitor detects that a virtual tunnel is down, it marks it as down; which brings all the associated routes down.
  • By default, the VPN monitor interval is 10 seconds and the VPN monitor threshold is 10.
  • So if the VPN monitor fails, it will take 100 seconds to bring the tunnel interface down.

To verify VPN monitor settings:

SSG320(M)-> get vpnmonitor
 Vpn monitor interval : 10(seconds)
 Vpn monitor threshold: 10

To change the VPN monitor and threshold values:

SSG320(M)-> set vpnmonitor interval 3
SSG320(M)-> set vpnmonitor threshold 2
Source Interface & Destination IP
  • VPN monitoring in ScreenOS uses ping between the public IPs but sent within the tunnel (that is, encrypted).
  • The problem is these pings do not match the proxy-id of the VPN.
  • A ScreenOS device will accept and respond, but a non-ScreenOS device will discard them.
  • Therefore we define Source Interface and Destination IP for a Non-ScreenOS device.
  • When enabling optimization, existing traffic via the VPN is used for the monitoring packet, instead of using the VPN monitor ping, which would normally be sent.
  • When Optimized is selected, firewall accepts incoming traffic through the VPN tunnel as a substitute for ICMP echo replies. *If there is both incoming and outgoing traffic through the VPN tunnel, the device suppresses VPN monitoring pings.
  • Rekey keeps the VPN SA active, even if there is no other VPN traffic.
  • When the key lifetime for a Phase 1 or Phase 2 SA is about to expire, the rekey option renews the key, resets the key lifetime, and keeps the SA active.
  • This is similar to the IKE heartbeat rekey; with the exception that it uses the VPN Monitor mechanism.
  • The Rekey option cannot be used alone; it has to be used with VPN monitor.
  • If you enable the rekey option, along with the VPN monitor, without optimization, the security device starts to immediately send ICMP echo, when the tunnel configuration is complete and continues to send them indefinitely.
  • The echo requests trigger an attempt to initiate IKE negotiations to establish a VPN tunnel, until the state of VPN monitoring for the tunnel is up.
  • The security device then uses the pings for VPN monitoring purposes.
  • If you do not enable Rekey with the VPN monitor, then the VPN will no longer send echo requests,when the VPN tunnel is marked as down by the VPN monitor.
  • But, when the Rekey option is enabled, the security device continues to send echo requests to its peer at defined intervals; which triggers attempts to reinitiate IKE Phase 2 negotiations and Phase 1 negotiations, if necessary, until it succeeds.
  • The firewall re-activates Phase 2 SA, generates a new key, and re-establishes the tunnel.
  • A message is generated in the event log, which states that a successful rekey operation has occurred.
  • Rekey is used to ensure that a tunnel is always up to monitor devices at the remote site or allow dynamic routing protocols.

  • VPN monitoring allows the SRX to send ICMP traffic either to the peer gateway, or to another destination on the other end of the tunnel (such as a server), along with specifying the source IP address of the ICMP traffic.
  • If the ICMP traffic fails, the VPN is considered down.
  • VPN monitoring is Juniper propriety and it works only with ScreenOS and Juniper devices(it's simply a ping across the tunnel, so any device on the far side that will repsond to pings can be used as the destination IP.
  • One caveat is to make sure that whatever you use as your source is within any relevant proxy-ID pairs, if applicable.); works on top of Phase 2 SA; it guarantees the data path.
  • If VPN Monitor detects a tunnel down, it can mark the tunnel interface as down, bringing all associated routes down (or indicating to the dynamic routing protocol to drop routes, similar to pulling a cable and bringing the layer 2 link state down on a physical interface).
  • ICMP is used as a probe for VPN Monitor for better interoperability with other vendors.


  • DPD is a method used by devices to verify the current existence and availability of peer devices.
  • The device sends encrypted IKE Phase 1 message (R-U-THERE) to peers
  • The Peer device sends DPD acknowledgements (R-U-THERE-ACK).
  • DPD doesn’t mean that the underlying VPN is up and running
  • It means just that the peer is up and responding.
  • DPD is used where Dynamic Routing Protocol is not used.
  • OSPF, etc can automatically detect a failure & choose another path without the need for DPD.

IKE Heartbeat

  • This feature can only be used exclusively between two NetScreen devices.
  • Both ends of the tunnel must support IKE heartbeats to be used.
  • IKE heartbeats are mainly used for VPN groups which are used in policy-based tunnels for tunnel failover.
  • When the Heartbeat Hello option is enabled, the NetScreen sends an IKE Heartbeat Hello to the other gateway.
  • If the NetScreen does not receive the hello packet, it will assume there is a problem with the tunnel, and it will tear down its Phase 1 and Phase 2 proposals.
  • The Reconnect feature will keep a VPN tunnel up, even after the phase 2 lifetime has expired.
  • VPN monitor & Heartbeat, both accomplish the same thing, to detect a tunnel outage & will bring up the tunnel automatically when there is no traffic.

Tunnel READY state

Without proper configuration, the state of tunnel interfaces changes to READY state instead of DOWN state.

  • If route-based VPN is configured without VPN Monitor, when the associated VPN tunnel goes down, the tunnel interface will be in READY state, as there is no mechanism to identify the correct state.
  • If the route-based VPN is configured with only VPN Monitor, during the initial boot up process, the system will put the tunnel interface in the READY state. Traffic initiation will change the state if VPN comes UP.
  • If a cable is physically pulled out of the interface that the tunnel interface is bound to, the tunnel interface state will change to a Down state.
  • If using NSRP, during initial boot up process, the system will have the tunnel interface in ready state "R" on the master unit, and inactive on the backup unit.
  • If route-based VPN is configured with VPN Monitor and Rekey option this option will only have either UP or DOWN state.

When VPN Monitor and rekey are enabled, if the VPN goes down, tunnel interface linked with that route will change to the Down state, and hence the route associated with the tunnel interface will be inactive.


  • The two peers generate a 8-byte pseudo-random number that is used for anticlogging purposes.
  • These cookies are based on a unique identifier for each peer (src and destination IP addresses) and therefore protect against replay attacks.
  • The ISAKMP RFC states that the method of creating the cookie is implementation-dependent but suggests performing a hash of the IP source and destination address, the UDP source and destination ports, a locally generated random value, time, and date.
  • The cookie becomes a unique identifier for the rest of the messages that are exchanged in IKE negotiation.
  • Initiator or Responder cookie:[3]
CKY-I or CKY-R = md5{(src_ip, dest_ip), random number, time, and date}

Replay Protection

  • A replay attack is when a hacker intercepts packets and uses them later to flood the system causing denial of service attacks (DoS) or to gain access to the internal network.
  • The 'Enable Replay Protection' enables the ScreenOS device to check every IPSec packet to see if it has already been received.
  • ScreenOS looks at the sequence number in the ESP header and if the packets arrive out of a specified sequence range, the packets are dropped.
VPNs > AutoIKE > Advanced > check "Replay Protection"
set vpn myvpn ike no-anti-replay

Diffie-Hellman Algorithm

  • DH Algorithms allow the two peers to agree on a shared secret, to generate keying material for subsequent use, without knowing any secrets beforehand.
  • The preshared key is already defined on the two peers, the DH secret is used along with preshared key to authenticate the two peers to each other.
DH Algorithm
There exists a DH public value = Xa There exists another DH public value = Xb
Xa = ga mod p Xb = gb mod p
a is a private secret known only to the initiator b is a private secret known only to the responder
g is the generator p is a large prime number
Initiator Secret = (Xb)a mod p = (Xa)b mod p = Responder Secret = gab
  • Both pears generate a shared secret known only to the two of them by simply exchanging the values Xa and Xb.
  • In order to calculate the DH secret between the two peers, the two peers calculate the DH public values and send them to each other.
  • In addition, a value known as a nonce is also generated and exchanged.
  • A nonce is a very large random number generated using certain mathematical techniques.
  • It is used in later calculations of the keying material.
  • As soon as the DH public values have been calculated, the two peers also independently calculate the nonces.


  • Initiator can "suggest" to use PFS to the responder during 1st message of quick mode.
  • If the responder agrees to do PFS, it continues the quick mode exchange.
  • Otherwise, it returns "Attributes Not Supported," and the initiator can continue without PFS if it is so configured.
  • PFS forces the peers to generate a new DH secret during the quick mode exchange.
  • Encryption keys used to encrypt data are generated using this new DH secret.
  • This ensures added secrecy.
  • If the original DH secret gets compromised, this new secret can ensure privacy for the data.
  • Negotiations for the new DH are carried out using encrypted traffic, this new DH secret has an added element of secrecy.
        This section needs verification or testing!
* It is not possible to negotiate the DH group as it was offered and accepted in main mode as quick mode has only one exchange in which the DH exchange must take place.
* Therefore, the PFS DH group configured on both ends must match.
  • This is done in exactly the same manner as was done in phase 1 of IKE.
  • If no PFS is used, session key is same for 8 hours.
  • If PFS is used, new session key is calculated every hour.


  • What are the different phases in VPN?
Phase 1: Peers negotiate a secure, authenticated channel with which to communicate.
Phase 2: Security associations are negotiated on behalf of IPsec services.

Aim of Phase 1:

  1. Agreeing on parameters to authenticate two peers
  2. Encrypt parts of main mode and all Phase 2 (Nothing from aggressive mode is encrypted if used).
  3. Generate keys to generate keying material for actual encryption of data.

All the information negotiated is stored as IKE or ISAKMP Security Association (SA). Any two IPsec peers have only one ISAKMP security association between them.

Encryption is optional in Phase 1 of aggressive mode.

According to RFC2409(Page 29):

"While the last roundtrip of Main Mode (and optionally the last message of
Aggressive Mode) is encrypted it is not, strictly speaking, authenticated."

Aim of Phase 2:

  1. Creating SA used to encrypt (in the case of ESP) the data.
  2. In the case of PFS, quick mode is responsible for redoing the Diffie-Hellman (DH) exchange.

Packet Exchange

Phase 1 Main Mode Packets/Negotiations

Note: Thick grey border means the packet is encrypted & authenticated. Orange border means Encryption is optional.

Message 1

Cookie I, SA Proposal List

Message 2
Cookie I, Cookie R, SA Proposal Accept
Message 3

DH Public Value A, Nonce I(Random #)

Message 4
DH Public Value B, Nonce R(Random #)
Message 5

ID I, ID Hash I)

Message 6
ID R, ID Hash R

  • Identity payload(ID) - Contains information about the identity of the initiator. This is in the form of the initiator's IP address or Host name.
  • Hash payload - Used for authentication purposes. The responder calculates the same hash on its end. If the two hashes come out to be the same, authentication is said to have taken place.
Hash_I(or R) = PRF(Preshared Key, SKEYID, Cookie_I. Cookie_R, SA Payload, Proposals + Transforms, ID)
  • 5th & 6th packet payloads are encrypted using SKEYID_E.
  • Based on Nonce, DH secret calculated, and preshared keys, three sets of keys are generated which are used to authenticate two peers to each other as well as to encrypt subsequent IKE exchanges.
  • These keys come out to be the same on both ends due to the nature of the DH exchange and the rest of the elements used to generate the keys.
  • These keys are called session keys (SKEYs):
SKEYID_d - This key is used to calculate subsequent IPsec keying material.
SKEYID_a - This key is used to provide data integrity and authentication to subsequent IKE messages.
SKEYID_e - This key is used to encrypt subsequent IKE messages.
PRF= Pseudo Random Function based on Negotiated Hash.

SKEYID = PRF(Pre-shared Key, Ni | Nr)
SKEYID_d = PRF(SKEYID, gab | Cookie-I | Cookie-R | 0)
SKEYID_a = PRF(SKEYID, SKEYID_d | gab | Cookie-I | Cookie-R | 1)
SKEYID_e = PRF(SKEYID, SKEYID_a | gab | Cookie-I | Cookie-R | 2)
  • When both ends have the keys generated, rest of the IKE exchange occur confidentially using encryption done using SKEYID_e.
  • Note that for the keys to be generated correctly, each peer must find the corresponding preshared key for its peer.
  • A number of preshared keys might be configured on both the peers for each of the many peers they communicate with.
  • However, the ID payload identifying the peer's IP address or the host name does not arrive until 5th & 6th messages are exchanged.
  • Therefore, each peer must find the preshared key for its peer by using the source IP address from which it is receiving the ISAKMP packets.
  • The reason for sending the ID payload later is to hide it by using encryption afforded by the keys that have been generated in this step.
  • This method pose a problem in aggressive mode.

Phase 1 Aggressive Mode Packets/Negotiations

Message 1

Identification I, SA Proposal List, DH Public Value A, Nonce I

Message 2
Identification R, SA Proposal Accept, DH Public Value B,

Nonce R, Identification Hash R

Message 3

Identification Hash I

Encryption is optional for 3rd packet as per RFC2409(Page 29).
But Juniper firewalls generally encrypt 3rd packet.

Phase 2 Quick Mode Packets/Negotiations

Message 1

Hash using Phase 1 information, Message ID,
SA Proposal List, Nonce I, DH Public Key I, Proxy-ID

Message 2
Hash using Phase 1 information,Message ID,SA Proposal List Accept,

Nonce R, DH Public Key I, Proxy-ID

Message 3

Hash using Phase 1 information,
Message ID, Nonce I, Nonce R,

Route vs Policy based

  • Reasons to use a Route-based VPN[2]
  1. NAT is required
  2. Overlapping Subnets/IP Addresses
  3. Hub-and-spoke VPN topology
  4. Redundant VPNs
  5. A Dynamic Routing Protocol is running across the VPN
  6. Need to access multiple subnets across the VPN
  7. Network functions are separated from policy functions
  8. Changes can be made to a permit or deny policy on one end of a tunnel without effecting the other end of the tunnel
  • Reasons to use a Policy-based VPN
  1. The remote VPN device is a non-Juniper device
  2. Need to access only one subnet across the VPN

Route-Based VPNs Policy-Based VPNs
A policy does not point to a VPN tunnel. A tunnel along with source, destination, application, action, form a tunnel policy that permits VPN traffic.
The policy points to a destination address. A tunnel policy points to a tunnel.
Maximum no of VPNs is limited by the number of route entries or tunnel interfaces that the device supports, whichever number is lower. Maximum no of VPNs is limited by the number of policies that the device supports.
Use it to conserve tunnel resources while setting granular restrictions on VPN traffic. Although can create numerous tunnel policies referencing the same VPN tunnel, each tunnel policy pair creates an individual IPSec SA which counts as an individual VPN tunnel.
You can configure dozens of policies to regulate traffic flowing through a single VPN tunnel between two sites and only one IPsec SA is at work. You can also set the action as deny. Action must be permit and must include a tunnel.
Supports exchange of dynamic routing information through VPN tunnels, can enable an instance of OSPF on tunnel interface that is bound to a VPN. The exchange of dynamic routing information is not supported.
Can be used for hub-and-spoke topologies. Cannot be used for hub-and-spoke topologies.
A policy does not point to a VPN tunnel. If a tunnel does not connect large networks with dynamic routing protocols and there is no need to conserve tunnels or define various policies to filter traffic through tunnel, a policy-based tunnel is best choice.
Do not support remote-access (dial-up) VPNs. They are required for remote-access (dial-up) VPNs.
They might not work correctly with third-party vendors. They might be required if the third party requires separate SAs for each remote subnet.
When performing a route lookup, Firewall finds a route via a tunnel interface, which is bound to a VPN tunnel. Consider tunnel as a means for delivering traffic and consider the policy as a method for either permitting or denying traffic. Consider a tunnel as an element in the construction of a policy.
It supports NAT for tunnel interfaces. It cannot be used if NAT is required for tunneled traffic.

AutoConnect VPN

Source: fir3net.com

  • AC-VPN works with a hub and spoke setup.
  • Once static VPNs have been configured between all the spokes and the hubs, AC-VPN and NHRP (Next Hop Routing Protocol) is configured on each spoke and the hub.
  • When traffic is initiated between 2 spokes the traffic is passed via the hub while a dynamic tunnel is established between the 2 spokes.
  • Once this tunnel is established the traffic is passed between the 2 spokes and the previous tunnel terminated.
  • The Hub in the hub and spoke network is classed as the "Next Hop Server" (NHS) and the spoke is referred to as the "Next Hop Client".
  • Messages are then exchanged between the client and the server using NBMA (Non Broadcast Multi Access) messages.
  • By default there are 7 types of NBMA messages along with 2 more that are added by the Netscreen.
  • These are :
Registration Request - Once a static VPN becomes active between the NHC and the NHS this message is sent containing Client information (such as routing, subnet masks etc)
Registration Reply - The NHS can ACK or NAK (not Acknowledged) a registration request.
Resolution Request, Resolution Reply - Added by Screen OS these messages ensure that the client cache of the NHS is update to date.
Purge Request, Purge Reply - This allows the NHC cache of the NHS to be purged in the event of a NHC being shutdown.
Error Indication - Logs a NHRP error conditions
  • To support AC-VPN ScreenOS adds the following message pair:-
Resolution-set, Resolution-ack
When a static tunnel is established all the information required for each spoke to set up a tunnel between themselves is sent.
AC-VPN Restrictions
All VPN tunnels configured toward the hub must be route based.
Automatic key management in phase 1 must be in aggressive mode.
The authentication method must be self-signed certificate and generic PKI.
All spokes must be connected to a single zone on the hub.
Configuring NHRP in multiple instances of virtual routers is supported only on the NHS.
General Steps
Configuration on the Hub
Create a static gateway and VPN.
Create static tunnels to the spokes and bind the VPNs to the tunnels.
Create an AC-VPN gateway profile.
Create an AC-VPN VPN profile.
Enable NHRP on the virtual router.
Select the ACVPN-Profile for NHRP.
Enable NHRP on the tunnel interface.
Configure routing.
Configuration on Each Spoke
The hub includes the following:
Create a static tunnel to the hub.
Create a gateway.
Create a VPN gateway.
Create an ACVPN-Dynamic gateway.
Create ACVPN-Dynamic VPN
Enable NHRP on the virtual router
Configure the NHS IP address
Configure the local cache.
Enable NHRP on the tunnel interface.
Configure routing.

Dynamic Multipoint VPN

        This section is under construction.

Digital Signatures

        This section is under construction.


  • Only Un-numbered Tunnel needs Interface Binding; Fixed IP Tunnel needs IP address.
  • PFS will run DH algorith again in Phase 2. It is more secure.
  • Deny in Crypto ACL means "Dont encrypt that Traffic".
  • Policy Precedence in Cisco ASA VPN:
Dymnamic Access Policy
User Profile
Group in User Profile
Group in Conn Profile
  • To Check fragmentation in VPN:
get sa stat
  • Which command is used to set VPN MSS value?

set flow tcp-mss 1350 command is applicable to only VPN traffic i.e. it can be used to change the MSS value for the SYN packet of the TCP handshake within the Tunnel

  • How to see VPN error in event log:
get event type 536
  • If VPN with Monitoring & Optimized enabled is Flapping:
  • Check if all VPNs are flapping ==> If yes, Check Network & device errors
  • Confirm if the ISP is not blocking ESP traffic.
  • Is the remote side a non-Juniper Firewall or remote VPN configured to block ICMP Echo Requests ==> If yes, reconfigure VPN Monitor to use the Source interface and Destination IP options.
If rekey is enabled, SA will continually rekey at the vpnmonitor threshold timeout.

VPN Troubleshooting Approach

  • Is this a Site-to-Site or Dial up VPN?
  • Are there any IKE Phase 1 or 2 messages on the Initiator or Responder VPN Firewall?

Initiator Side Error - Phase 1: Retransmission limit has been reached

No Error on Responder side

- Check if you can ping Responder side Gateway IP?
- If yes, Check if the gateway IP in Phase 1 config same as Remote side public IP?
- Check if IKE gateway's outgoing interface match the route to the destination? 
- Check if ISP or any intermediate router blocks traffic? Perform snoop on both sides at the same time to verify.

Error on Responder side

- Phase 1 : retransmission limit has been reached - this most likely indicates a proposal mismatch in Phase 1. 
   Because Phase 1 has not been completed yet, no response is sent back to the originator, which eventually reports that the retransmission limit has been reached
- Phase 1 : Rejected proposals from peer. Negotiations failed
- Phase 2 : Received notify message fro DIO <1> <14> <NO_PROPOSAL_CHOSEN>
- Phase 1 : Rejected proposals from peer. Negotiations failed
- Proposal list configured in phase 1 and/or phase 2 does not match on both gateways. Check PFS setting in Phase2 as well!
- If you have troubles with the proposals, you can configure the Juniper to accept all proposals : set ike accept-all-proposals
- Preshared Key mismatch : Correct the PSK and try again
- No route information : both gateways needs to be able to contact each other, perhaps routing is not set up correctly.

Rejected initial Phase 1 packet from unrecognized peer gateway

Possible Causes:

- Initiator uses dynamic IP, but tunnel is still in main mode.
- Peer gateway addresses don’t match
- Peer ID misconfigured  (on gateway with dynamic IP, you need to set the local ID, on gateway with fixed IP, you need to set the peer ID. Both ID strings must match)
- Outgoing interface is incorrect

Phase 2: No policy exists for the proxy ID received: local ID <…..> remote ID <……>

- Address book errors (proxy ID) : Address book entries do not match on both gateways - Check subnet masks

P2 Attributes not supported

- This mostly happens between a Juniper and non-Juniper device (such as Nortel), and is caused because some vendor specific settings are enabled on the non-Juniper device (such as compression, vendor ID, etc)

Invalid cookie

## 2007-11-27 22:15:51 : IKE<> Received notify message for DOI <1> <4> <INVALID-COOKIE>.
- This indicates that the gateway (peer) IP addresses do not match at both sides

Replay protection is not configured on both ends

Other misconfigurations

- If one side is set to route based VPN and the other side to policy based VPN, then you’ll have to set proxy ID’s on the route based VPN as well.
   By default, the route based VPN proxy ID’s are set to zero.
- If the tunnel interface is in the same zone as the source network object, then you need to set up intrazone blocking in order to be able to create a policy, or leave it turned off if you want all traffic to pass through anyway.
   That having said, since a policy based VPN uses a policy, you need to turn on blocking if you want to trigger the VPN.
   As soon as you turn it on, the default global policy will kick in, denying all traffic. 
   So don’t forget to create your rules for intrazone traffic.
        This section is under construction.
  • Are there any IKE Phase 1 or 2 error messages for this VPN Tunnel in the Event Logs?
  • Is the VPN Tunnel's SA Active? Is VPN up but not passing Traffic?
  • Proposal Mismatch
  • Preshared key Mismatch
  • No route information
Route information is need for the device to forward IKE traffic to the peer gateway. Remote gateway should be reachable.
  • Proxy-ID
Mismatched IDs will cause a failure during IKE phase 2
  • Policy Based NAT
Unnumbered tunnel interfaces can NOT be the egress interface when policy-based NAT is configured
  • Tunnel interface bound to a different zone than the ingress/egress interface of the data
Policies must be configured to allow data from one security zone to another
  • Intra Zone Blocking
If both the ingress/egress interface and the tunnel interface are bound to the same security zone—check that the Intra-zone block is set to off
  • Incorrect Routing

  1. www.convergys.com
  2. 2.0 2.1 www.bureauveritas.co.in
  3. www.accenture.com

{{#widget:DISQUS |id=networkm |uniqid=VPN |url=https://aman.awiki.org/wiki/VPN }}