Turns out that something (possibly CiscoAnyconnect) has unloaded racoon on startup. To fix (with Anyconnect uninstalled) do the following:
launchctl load -w /System/Library/LaunchDaemons/com.apple.racoon.plist
You may also start racoon manually by:
sudo /usr/sbin/racoon
I've had some success in Mac OSX Leopard 10.5.8. For my setup, I have a Mac Mini behind a Verizon FiOS Actiontec router. I was using Android phone to connect.
At first, it worked fine internally (Phone on the same Wifi), but would fail when connecting externally (Phone on data connection). In the end, it is working exactly the opposite.
I don't have a user account for the "VPN User" because this method just created a single user/password for the connection. I am not "logged in" to Mac, but could remote desktop with a "real user" after connected.
I used version 2.4b of iVPN to configure the settings, and here are some end results:
Forwarded ports
UDP Any->1701, UDP Any->500
Android Settings
Name: YourConnectionName (e.g. Mac Server)
Type: L2TP/IPSec PSK
Server address: hostname.no-ip.org
L2TP secret: (not used)
IPSec identifier: (not used)
IPSec pre-shared key: **YourSharedSecret**
When connecting
Username: auser
Password: challenge
/etc/ppp/user.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Password</key>
<string>challenge</string>
<key>User</key>
<string>auser</string>
</dict>
</plist>
/etc/ppp/chap-secrets
auser * challenge *
/Library/Preferences/SystemConfiguration/com.apple.RemoteAccessServers.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>ActiveServers</key>
<array>
<string>com.apple.ppp.l2tp</string>
</array>
<key>Servers</key>
<dict>
<key>com.apple.ppp.l2tp</key>
<dict>
<key>DNS</key>
<dict>
<key>OfferedSearchDomains</key>
<array/>
<key>OfferedServerAddresses</key>
<array>
<string>208.67.222.222</string>
<string>208.67.220.220</string>
</array>
</dict>
<key>IPv4</key>
<dict>
<key>ConfigMethod</key>
<string>Manual</string>
<key>DestAddressRanges</key>
<array>
<string>192.168.10.101</string>
<string>192.168.10.200</string>
</array>
<key>OfferedRouteAddresses</key>
<array>
<string>192.168.10.100</string>
</array>
<key>OfferedRouteMasks</key>
<array>
<string>255.255.255.0</string>
</array>
<key>OfferedRouteTypes</key>
<array>
<string>Private</string>
</array>
</dict>
<key>Interface</key>
<dict>
<key>SubType</key>
<string>L2TP</string>
<key>Type</key>
<string>PPP</string>
</dict>
<key>L2TP</key>
<dict>
<key>IPSecSharedSecret</key>
<string>**YourSharedSecret**</string>
<key>Transport</key>
<string>IPSec</string>
</dict>
<key>PPP</key>
<dict>
<key>AuthenticatorProtocol</key>
<array>
<string>MSCHAP2</string>
</array>
<key>LCPEchoEnabled</key>
<integer>1</integer>
<key>LCPEchoFailure</key>
<integer>5</integer>
<key>LCPEchoInterval</key>
<integer>60</integer>
<key>Logfile</key>
<string>/var/log/ppp/vpnd.log</string>
<key>VerboseLogging</key>
<integer>1</integer>
</dict>
<key>Server</key>
<dict>
<key>Logfile</key>
<string>/var/log/ppp/vpnd.log</string>
<key>MaximumSessions</key>
<integer>128</integer>
<key>VerboseLogging</key>
<integer>1</integer>
</dict>
</dict>
</dict>
</dict>
</plist>
/etc/racoon/remote/anonymous.conf
remote anonymous {
doi ipsec_doi;
situation identity_only;
exchange_mode main;
verify_identifier off;
shared_secret use "**YourSharedSecret**";
nonce_size 16;
nat_traversal_multi_user on;
initial_contact on;
support_mip6 on;
proposal_check claim;
proposal {
authentication_method pre_shared_key;
hash_algorithm sha1;
encryption_algorithm 3des;
lifetime time 3600 sec;
dh_group 2;
}
}
sainfo anonymous {
encryption_algorithm aes, 3des;
authentication_algorithm hmac_sha1, hmac_md5;
compression_algorithm deflate;
lifetime time 3600 sec;
}
You might have to touch /var/log/ppp/vpnd.log
and if you're not using iVPN, it looks like (from a ps -ax
) the server is started with vpnd -i com.apple.ppp.l2tp
. After changing settings and PSKs, I also racoonctl flush-sa ipsec
.
Best Answer
Trouble getting Windows to connect to an L2TP VPN
Firstly, if the VPN server is behind a NAT and the VPN client is behind a NAT this could cause a problem because apparently "by default Windows does not support IPSec network address translation (NAT) Traversal (NAT-T) security associations to servers that are located behind a NAT device", and this applies to Windows 10 still as well.
The advice given by Microsoft "if you have to put a server behind a NAT device and then use an IPsec NAT-T environment, you can enable communication by changing a registry value on the VPN client computer and the VPN server."
PowerShell (Suggested Fix)
Note: You must run this in an admin elevated PowerShell session.
Important: You must restart the machine(s) you apply this to before it's effective.
PowerShell (Remove Fix)
Note: You must run this in an admin elevated PowerShell session.
Important: You must restart the machine(s) you apply this to before it's effective.
If it is not a double NAT issue then . . .
It seems perhaps in some configurations port
1701
is used over TCP and UDP both and not just UDP only. Adjust your rule to allow the TCP port1701
through as well and see if that fixes the problem.Furthermore, ensure your Windows Firewall "allow" rule(s) for the applicable TCP and UDP ports (and any correlated VPN client software exe's, etc.) from the
Advanced
tab has allprivate
,domain
, andpublic
profiles checked.Upon further research it's not super clear to me if some of this applies to the client side rather than the VPN server side for L2TP but some advice suggests actually allowing UDP port
50
.There's another post that also talks about the
ESP (value 50) <- Used by IPSec data path
and others that refer to the ports used by IPSec protocols and ports.Further Troubleshooting
To troubleshoot further, consider running Wireshark with the Windows Firewall disabled and make the successfully VPN connection and save that trace. Then with the Windows Firewall enabled, run a new trace, attempt a VPN connection, and save that trace.
Now you can look over both successful and unsuccessful L2TP VPN connection traces, filter, and see at the packet level what is really going on to determine what further you may need to allow through the Windows Firewall.
Supporting Resources
Understanding Firewall Profiles
Layer 2 Tunneling Protocol