OpenVPN Bridge under VMware ESXi
So, as these things go, I spent 2 days looking at my OpenVPN config files and bridging setup before finding the solution to my problem was elsewhere.
A little background: I created a new OpenVPN VM using the Debian Squeeze net install CD, configured it to match what was already working on a physical Windows XP box, but only had limited success. I was able to connect to the VPN, ping the OpenVPN server on the network, but couldn’t connect to anything else. Trying to ping another server from the VPN client, and running tcpdump on that other server showed that it was receiving ICMP requests and replying, but they were not making it back to the VPN client. I tried a hundred different ways of creating the bridge on the OpenVPN server, but nothing worked. Finally, good old Google found the answer. The ESXi virtual switch drops promiscous packets by default. To fix it, open the vSphere Client, click on the ESXi host on the left side, click on the “Configuration” tab on the right, click “Networking” in the Hardware box, click on “Properties…” at the top-right of your “Virtual Switch: vSwitch#” graphic. Now on the “Tools” tab of this popup window, select the “vSwitch” and click the “Edit…” button. In this popup, click on the “Security” tab and change “Promiscuous Mode” from “Reject” to “Accept”. Click “OK” then “Close” and you should be all set.
Other than that, it’s really pretty basic – do an install with nothing but SSH, then add bridge-utils and openvpn. Also add vim and locate for convenience if you like.
# apt-get install bridge-utils openvpn vim locate
I set up the bridge manually instead of using the scripts that come with OpenVPN. Here is my /etc/network/interfaces:
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
pre-up /usr/sbin/openvpn --mktun --dev tap0
pre-up /sbin/ifconfig eth0 0.0.0.0 promisc up
pre-up /sbin/ifconfig tap0 0.0.0.0 promisc up
pre-up /usr/sbin/brctl addbr br0
pre-up /usr/sbin/brctl stp br0 off
pre-up /usr/sbin/brctl setfd br0 0
pre-up /usr/sbin/brctl addif br0 eth0
pre-up /usr/sbin/brctl addif br0 tap0
address 10.100.1.18
netmask 255.255.255.0
network 10.100.1.0
broadcast 10.100.1.255
gateway 10.100.1.1
Configure OpenVPN according to the instructions at openvpn.net. Here’s my server.conf:
local 10.100.1.18
port 1194
management 10.100.1.18 5555 /etc/openvpn/mgmt_passwd.txt
proto tcp
dev tap0
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
dh /etc/openvpn/keys/dh1024.pem
server-bridge 10.100.1.18 255.255.255.0 10.100.1.30 10.100.1.39
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 10.100.1.2"
push "dhcp-option WINS 10.100.1.2"
client-to-client
keepalive 10 120
comp-lzo
max-clients 10
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3
The management line is optional, but it allows connections to the management console so you can see who is connected etc. There is a pretty nice GUI tool here you can use from a Windows box on the network if you want.
Other than those settings, all you have to do is follow the instructions for creating the keys using the easy-rsa scripts and you should be fine…
Thank your very much for this great article! I’ve been trying to solve this for 1.5 days. After doing the magic in vSphere it finally works. Thanks again!
Many thanks Jeremy. Worked great. Steps are just slightly different for esxi 5.1 UP2 for the vSwitch, but still the same fix.
Thank you for the suggestion.
Unfortunately enabling promiscious mode on the vswitch enables other vms to sniff all transferred packets. Its a major security flaw
see vmware knowledge base:
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1002934
So if you don’t trust your VMs, put them on another switch…
Awesome tip. Thanks much!!!
Thanks a lot pointing this out! I would never found out.
Master has given Dobby the solution. Dobby is free!
lol
Thanks! It worked..
I strugging with same problem for the last two day. I will try it and let u know guys.
So Jeremy,
This is Chris we worked together at Schooner if you recall. You just saved me sooooo much headscratching, I was having this very issue and would have been hard-pressed to find the solution. I owe you a beer 🙂
Hi Chris – I think you probably worked with the “MySQL” Jeremy Cole and not me, but I’m glad I helped, and I’ll take that beer any time!
You’ve ended my two days of desperation.
Many thanks! 😀
Please,
I can’t put to work my Openvpn ESXi bridge to work.
Could someone show me it’s ifconfig output, so I could figure what is wrong in my config ?
Or could someone help me ?
I’m developer, not network specialist. But I need this bridge to make some tests.
I installed Openvpn Access Serveur Virtual Appliance in my ESXi box.
Then my /etc/network/interfaces :
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
address 200.53.114.43
netmask 255.255.255.240
gateway 200.53.114.33
#Private Network
allow-hotplug eth1
iface eth1 inet static
address 172.1.50.9
netmask 255.255.255.240
# gateway x.x.x.x
My ESXi Openvpn : public IP : 200.53.114.43, and LAN : 172.1.50.9.
I put VPN mode to : Layer 2, but no bridge name.
I need to bridge from home to LAN 172.1.50.0 network, so I put in Advanced VPN -> Server Config Directive the line :
ifconfig-pool 172.1.50.12 172.1.50.12 255.255.255.240
When I connect from home, I get the ip : 172.1.50.12, and the right route print entry, but I can’t ping Openvpn ESXi bridge LAN ip (eth1) 172.1.50.9, neither ips from LAN computers.
In my Openvpn ESXi bridge, the ifconfig give me :
as0t0 Link encap:Ethernet HWaddr fe:ff:ff:65:1f:00
inet6 addr: fe80::fcff:ffff:fe65:1f00/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:1223 errors:0 dropped:0 overruns:0 frame:0
TX packets:1138 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:200
RX bytes:118971 (116.1 KiB) TX bytes:122625 (119.7 KiB)
asbr0 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:74
inet addr:200.53.114.43 Bcast:200.53.114.47 Mask:255.255.255.240
inet6 addr: fe80::20c:29ff:feae:ba74/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4861 errors:0 dropped:0 overruns:0 frame:0
TX packets:2774 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:541219 (528.5 KiB) TX bytes:498187 (486.5 KiB)
eth0 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:74
inet6 addr: fe80::20c:29ff:feae:ba74/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:41472 errors:0 dropped:0 overruns:0 frame:0
TX packets:4003 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22860963 (21.8 MiB) TX bytes:617626 (603.1 KiB)
eth1 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:7e
inet addr:172.1.50.9 Bcast:172.1.50.15 Mask:255.255.255.240
inet6 addr: fe80::20c:29ff:feae:ba7e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17364 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1438884 (1.3 MiB) TX bytes:468 (468.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:31 errors:0 dropped:0 overruns:0 frame:0
TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:13074 (12.7 KiB) TX bytes:13074 (12.7 KiB)
Is it normal that asbr0 get ip 200.53.114.43 ?
If I keep mode to : Layer 2, but but put bridge name asbr0, ifconfig give me :
eth0 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:74
inet addr:200.53.114.43 Bcast:200.53.114.47 Mask:255.255.255.240
inet6 addr: fe80::20c:29ff:feae:ba74/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:42295 errors:0 dropped:0 overruns:0 frame:0
TX packets:5015 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:23005868 (21.9 MiB) TX bytes:820754 (801.5 KiB)
eth1 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:7e
inet addr:172.1.50.9 Bcast:172.1.50.15 Mask:255.255.255.240
inet6 addr: fe80::20c:29ff:feae:ba7e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17631 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1460972 (1.3 MiB) TX bytes:468 (468.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:31 errors:0 dropped:0 overruns:0 frame:0
TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:13074 (12.7 KiB) TX bytes:13074 (12.7 KiB)
But it’s the same, from home I get 172.1.50.12 but can’t ping 172.1.50.x nodes.
I spent some days without success. I suppose I don’t have all the knowledge to fully understand the problem.
I’m teacher for public school and need this bridge for my students to learn some programing…
Please if someonr can help.
Thanks a lot.
Vincent MARECHAL
And yet another satisfied reader. I only spent 3 days hunting down this issue, tried several vpn solutions attempting to simply bridge taps with eths in Linux VMs on an ESXi 4.1 host. The second (or maybe 2 seconds later) I flipped promiscous to accept on the vswitch that was dropping everything, ICMP from remote network segment started reaching into the internal network.
Thanks!
You. Rock.
many thanks for this post.
it saved me a lot of time
thanks a lot!
I wish I could say I found this page before I spent around 4 hours trying to solve why I couldn’t ping my internal LAN. I was trying tcpdump, running debugs on my Cisco router and was getting nowhere.
As i’d ran this config perfectly fine on a physical box before, my efforts turned towards VMWare and I was digging around and ticket the promiscious box and voila it bloody worked!!
I just wished I had found this page 4 hours ago! Thanks for confirming it though 🙂
Well at least it only took you 4 hours… I spent 2 days on it. :/
Wow, THANK YOU, saved me a lot of time ……
Thanks for this great blog entry. I ran into exact the same problem as you, i wanted to move our OpenVPN from physical to our new ESXi Cluster. The old VPN runs under Debian but i tried Ubuntu for the new one. When it didn´t work and i could not make out the problem, i supected Ubuntu to show strange behaviour. So i switched to Debian but the same here, despite of exactly matching configs. Today i used tcpdump to locate the Problem, and i suspected our firewall to be missconfigured. I dont know how but suddenly there was the word promisc in my mind which meaning i didnt know. After searching for that term i thought that i forgot to activate promisc-mode on my tap device but no it´s set. Then your post appeared and i nearly dropped from my chair 🙂
Thank you 1000 times!
seriously our esxi was working just fine with openvpn and everything.
but then we needed more hardware (ram and an added hdd) and from then on everything went kablooey.
as i didn’t set up the server in the first place, i wasn’t even aware that “promiscuous mode” could become a problem. so from the addition of the hardware, as i said, everything went kablooey. after 2 weeks of just thinking “well, guess we made a serious error in the code for our product”, i tried to corner the problem.
local machine => check
other machines on the network => check
vm’s on those machines => check
vm’s on the esxi => NO check
so i tried to work out why… and after some hours asked big daddy google.
found your blog.
switched the DAMNED promiscuous mode to accept.
AND EVERYTHING WORKS.
thank you SO much man!
Hey man, thanks for the clear instructions. Even though I *knew* it was a promiscuous mode problem, even finding out how to fix the problem is not a trivial search. Great post.
Dude…
I usually don’t leave comment’s on people blogs as they don’t help me out this much! I have spent almost 4 days trying to figure out why the traffic wasn’t flowing through, knowing full well that I have set up 20-30 OpenVPN devices on hardware alone in the past and NEVER had this much trouble. Literally within a second of changing promiscuous mode to accept everything worked! THANK YOU!!!!!!!!!
That’s exactly how I felt when I figured it out… Yay!!!
I win, i spend just 1 day and found that problem ! :))
Thanks!! YOU SAVED MY LIFE !! I was desperated with this issue :))