IPsec VPN FAQs

What's the difference between tunnel and transport mode?

When using tunnel mode, you can think of the payload packet as being completely encased in another packet. In addition, IPsec can allow or deny packets access to the tunnel depending on policies. When using transport mode, communication is limited between two hosts. Only one IP header is present, with the rest of the packet being encrypted. Unless you have very specific needs, you'll most likely want to use tunnel mode.

What devices can I connect to with Untangle's IPsec VPN?

We have currently verified that IPsec VPN can successfully connect to other Untangle boxes and pfSense. We have user-submitted settings for other devices below, but please be aware Untangle Support cannot debug tunnels between Untangle and a 3rd party device. We only support IPsec tunnels between two Untangle boxes.

If I install Untangle behind a NAT device, what do I need to forward to Untangle for IPsec VPN to connect?

You will need to forward ESP, AH, and UDP port 500 from the public IP to the Untangle server. You may also need to enable NAT traversal. It is recommended to give Untangle a public IP if you want to set up IPsec tunnels.

Can I use IPsec on a server that uses DHCP to get its external address?

It is generally recommended to use IPsec VPN only on Untangle servers configured with static IPs. However, technically it can work with DHCP, but you will need to reconfigure the tunnel whenever the IP address actually changes. On some ISPs this is rare and servers will often have the same IP for months. On other ISPs IPs change daily.

Does IPsec traffic go through other Untangle applications?

Yes and Maybe. IPsec tunnel traffic and traffic from L2TP and Xauth clients will pass through all the other apps just like any other LAN traffic. However, if you want IPsec tunnel traffic to bypass scanning by other applications you can add a bypass rule.

Note: In versions prior to 11.2, the default was to bypass all IPsec tunnel traffic (but not L2TP or Xauth). You may still have a bypass rule in place to Bypass all IPsec traffic which will cause the traffic to not be scanned by other apps.

How do I connect IPsec between Untangle and my IPsec Device?

IPsec on Untangle should work with any compatible endpoint, but unfortunately Untangle doesn't have the resources to test against all known IPSec devices. Untangle recommends documenting the Phase1/Phase2 settings of the 3rd party IPSec device then matching those settings on Untangle, which cab be entered under the Manual Configuration available in all tunnel configurations. Untangle support has successfully deployed IPSec connections to various models from the following 3rd party manufacturers.

Cisco Endian eSoft Firebox Fortinet Juniper M0n0wall pfSense Sonicwall Watchguard and many others....

How do I connect IPsec between Untangle and pfSense?

These settings have been verified by Untangle Support:

Untangle Settings:

• Enable: (check if you want the tunnel up)
• Description: (whatever you want)
• Connection Type: Tunnel
• Auto Mode: Start
• Interface: (pick your interface)
• External IP: (will be automatically set from the WAN you choose)
• Remote IP: The WAN IP of the pfSense box
• Local Network: The LAN of the Untangle box (eg 192.168.1.0/24)
• Local IP: Untangle's LAN IP (eg 192.168.1.1)
• Remote Network: The LAN of the pfSense box (eg 192.168.2.0/24)
• PFS: Checked
• Shared Secret: (must match the Pre-Shared Key secret on the pfSense box)

pfSense Settings:

Phase 1:

• Disabled: (unchecked)
• Interface: WAN
• Remote Gateway: Untangle's WAN IP
• Description: (whatever you want)
• Authentication: Mutual PSK
• Negotiation Mode: main
• My identifier: My IP address
• Peer identifier: Peer IP address
• Pre-Shared Key: (must match the Shared Secret on the pfSense box)
• Policy Generation: Default
• Proposal: Default
• Encryption algorithm: 3DES
• Hash algorithm: SHA1
• DH key group: 2
• Lifetime: 28800
• NAT Traversal: Enable
• Dead Peer Detection: (checked)

Phase 2:

• Disabled (unchecked)
• Mode: Tunnel
• Local Network: LAN Subnet
• Remote Network: Network, Address: The LAN of the Untangle box (eg 192.168.1.0/24)
• Description: (whatever you want)
• Protocol: ESP
• Encryption algorithm: check AES, 128 bits
• Hash algorithm: check SHA1
• PFS key group: 2
• Lifetime: 28800

How can I connect IPsec from Untangle to M0n0wall?

These settings have not been verified by Untangle Support (thanks random person):

• Local subnet : M0n0wall LAN subnet
• Remote subnet: x.x.x.0 / xx (fill in your Remote Untangle's subnet address and netmask with .0 on the end)
• Remote gateway: <Remote Untangle's External IP address>

Phase 1:

• Negotiation mode : main
• Encryption algorithm : 3DES
• Hash algorithm : SHA1
• DH key group : 2 = 1024 bit
• Authentication method : Pre-shared key

Phase 2:

• Protocol : ESP
• Encryption algorithm : 3DES
• Hash algorithm : SHA1
• PFS key group : 2 = 1024 bit

How can I connect IPsec from Untangle to Cisco RV series?

These settings have not been verified by Untangle Support (thanks jcoffin):

• Keying Mode : IKE with Pre-shared key
• Phase1 DH Group : Group 2
• Phase1 Encryption : 3DES
• Phase1 Authentication : SHA1
• Phase1 SA Life Time : 86400 seconds
• Perfect Forward Secrecy : checked
• Phase2 DH Group : Group 2
• Phase2 Encryption : 3DES
• Phase2 Authentication : SHA1
• Phase2 SA Life Time 3600 seconds
• Pre-shared Key : <same as on UT>
• Advanced (all unchecked except)
  • AH Hash Algorithm  : SHA1

How can I connect IPsec from Untangle to Endian?

These settings have not been verified by Untangle Support (thanks aboyce):

• Remote host / IP : Public IP of the Untangle server
• Local Subnet : Endian LAN subnet
• Remote Subnet : x.x.x.0 / xx (fill in your Remote Untangle's subnet address and netmask with .0 on the end)
• Local ID : Public IP of the Endian server
• Remote ID : Public IP of the Untangle server
• Dead Peer Detection : Restart
• Pre-shared Key : <same as on UT>

• Advanced settings:
• IKE encryption AES (128 bit) and 3DES
• IKE integrity : SHA and MD5
• IKE Group DH group 5 (1536 bits) and DH group 2 (1024 bits)
• IKE lifetime 1 hours

• ESP encryption AES (128 bit) and 3DES
• ESP integrity SHA1 and MD5
• ESP key life 8 hours
• IKE Aggresive Mode Allowed : Off
• Perfect Forward Secrecy (PFS) : On
• Negotiate Payload : Off

How can I connect IPsec from Untangle to a Cisco 870 series?

These settings have not been verified by Untangle Support (thanks djoey1982):

• On the Untangle:
• Connection Type: Tunnel
• Auto Mode: Start
• Interface: External
• External IP: (The external IP address of this server)
• Remote IP: (The public IP address of the remote IPsec gateway)
• Local Network: (The private network attached to the local side of the tunnel)
• Local IP: (The IP address of this server on the local private network)
• Remote Network: (The private network attached to the remote side of the tunnel)
• Perfect Forward Secrecy (PFS) : unchecked
• Shared Secret : <same as Cisco>

• Link to Cisco 870 Settings: Cisco 870 Settings

How can I connect IPsec from Untangle to a Watchguard Firebox X10/X20?

These settings have not been verified by Untangle Support (thanks snecklifter!)

• Credential Method: Shared Key
• Main Mode, IP Address

Phase 1:

• SHA1-HMAC
• 3DES-CBC
• Neg expires in 0kb, 8 hours
• DH group 2
• Enabled DPD (Note that this is important, IKE Keep alive is proprietary and does not work)

Phase 2:

• SHA1-HMAC
• AES 256
• Untick TOS for IPSEC
• Enable PFS
• key expiry in 128000kb, 24 hours

How can I connect IPsec from Untangle to an eSoft InstaGate?

The default InstaGate and Untangle settings can be used to create a connection.

• Network: Local Network to Remote Network
• Key Management: Automatic (Shared Secret)

IKE Settings (Phase 1):

• 24 hours, 0 KB
• Strict PFS disabled
• Aggressive Mode disabled
• High Security

IPSec Settings (Phase 2):

• 1 hours, 0 KB
• PFS Group 2 (DH)
• High Security

How can I connect IPsec from Untangle to a Sonicwall?

Sonicwall Configuration is listed below.

General:

• Authentication Method: IKE using Pre-shared Secret
• IPsec Primary Gateway Name or Address: WAN IP on Untnagle
• Enter Shared Secret

Network:

• Local Networks: Choose local network from list, select predefined network.
• Destination Networks: Choose local network from list, select predefined network.

Proposals:

IKE (Phase 1) Proposal

• Exchange: Main Mode
• DH Group: Group 2
• Encryption: 3DES
• Authentication: SHA1
• Life Time: 28800

Ipsec (Phase 2) Proposal

• Protocol: ESP
• Encryption: AES-128
• Authentication: SHA1
• Enable Perfect Forward Secrecy: Enabled (checked)
• DH Group: Group 2
• Life Time: 28800

Advanced:

• Enable Keep Alive: Enabled (checked)