OpenVPN Bridge under VMware ESXi

So, as these things go, I spent 2 days looking at my OpenVPN config files and bridging setup before finding the solution to my problem was elsewhere.

A little background: I created a new OpenVPN VM using the Debian Squeeze net install CD, configured it to match what was already working on a physical Windows XP box, but only had limited success. I was able to connect to the VPN, ping the OpenVPN server on the network, but couldn’t connect to anything else. Trying to ping another server from the VPN client, and running tcpdump on that other server showed that it was receiving ICMP requests and replying, but they were not making it back to the VPN client. I tried a hundred different ways of creating the bridge on the OpenVPN server, but nothing worked. Finally, good old Google found the answer. The ESXi virtual switch drops promiscous packets by default. To fix it, open the vSphere Client, click on the ESXi host on the left side, click on the “Configuration” tab on the right, click “Networking” in the Hardware box, click on “Properties…” at the top-right of your “Virtual Switch: vSwitch#” graphic. Now on the “Tools” tab of this popup window, select the “vSwitch” and click the “Edit…” button. In this popup, click on the “Security” tab and change “Promiscuous Mode” from “Reject” to “Accept”. Click “OK” then “Close” and you should be all set.

Other than that, it’s really pretty basic – do an install with nothing but SSH, then add bridge-utils and openvpn. Also add vim and locate for convenience if you like.

# apt-get install bridge-utils openvpn vim locate

I set up the bridge manually instead of using the scripts that come with OpenVPN. Here is my /etc/network/interfaces:

auto lo
iface lo inet loopback
auto br0
iface br0 inet static
pre-up /usr/sbin/openvpn --mktun --dev tap0
pre-up /sbin/ifconfig eth0 0.0.0.0 promisc up
pre-up /sbin/ifconfig tap0 0.0.0.0 promisc up
pre-up /usr/sbin/brctl addbr br0
pre-up /usr/sbin/brctl stp br0 off
pre-up /usr/sbin/brctl setfd br0 0
pre-up /usr/sbin/brctl addif br0 eth0
pre-up /usr/sbin/brctl addif br0 tap0
address 10.100.1.18
netmask 255.255.255.0
network 10.100.1.0
broadcast 10.100.1.255
gateway 10.100.1.1

Configure OpenVPN according to the instructions at openvpn.net. Here’s my server.conf:

local 10.100.1.18
port 1194
management 10.100.1.18 5555 /etc/openvpn/mgmt_passwd.txt
proto tcp
dev tap0
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
dh /etc/openvpn/keys/dh1024.pem
server-bridge 10.100.1.18 255.255.255.0 10.100.1.30 10.100.1.39
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 10.100.1.2"
push "dhcp-option WINS 10.100.1.2"
client-to-client
keepalive 10 120
comp-lzo
max-clients 10
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3

The management line is optional, but it allows connections to the management console so you can see who is connected etc. There is a pretty nice GUI tool here you can use from a Windows box on the network if you want.

Other than those settings, all you have to do is follow the instructions for creating the keys using the easy-rsa scripts and you should be fine…

31 replies
  1. martin
    martin says:

    Thank your very much for this great article! I’ve been trying to solve this for 1.5 days. After doing the magic in vSphere it finally works. Thanks again!

    Reply
  2. fustipisti
    fustipisti says:

    Many thanks Jeremy. Worked great. Steps are just slightly different for esxi 5.1 UP2 for the vSwitch, but still the same fix.

    Reply
  3. Chris Lahti
    Chris Lahti says:

    So Jeremy,

    This is Chris we worked together at Schooner if you recall. You just saved me sooooo much headscratching, I was having this very issue and would have been hard-pressed to find the solution. I owe you a beer 🙂

    Reply
  4. Vincent MARECHAL
    Vincent MARECHAL says:

    Please,

    I can’t put to work my Openvpn ESXi bridge to work.

    Could someone show me it’s ifconfig output, so I could figure what is wrong in my config ?

    Or could someone help me ?
    I’m developer, not network specialist. But I need this bridge to make some tests.

    I installed Openvpn Access Serveur Virtual Appliance in my ESXi box.
    Then my /etc/network/interfaces :
    auto lo
    iface lo inet loopback

    # The primary network interface
    allow-hotplug eth0
    iface eth0 inet static
    address 200.53.114.43
    netmask 255.255.255.240
    gateway 200.53.114.33

    #Private Network
    allow-hotplug eth1
    iface eth1 inet static
    address 172.1.50.9
    netmask 255.255.255.240
    # gateway x.x.x.x

    My ESXi Openvpn : public IP : 200.53.114.43, and LAN : 172.1.50.9.
    I put VPN mode to : Layer 2, but no bridge name.

    I need to bridge from home to LAN 172.1.50.0 network, so I put in Advanced VPN -> Server Config Directive the line :
    ifconfig-pool 172.1.50.12 172.1.50.12 255.255.255.240

    When I connect from home, I get the ip : 172.1.50.12, and the right route print entry, but I can’t ping Openvpn ESXi bridge LAN ip (eth1) 172.1.50.9, neither ips from LAN computers.

    In my Openvpn ESXi bridge, the ifconfig give me :
    as0t0 Link encap:Ethernet HWaddr fe:ff:ff:65:1f:00
    inet6 addr: fe80::fcff:ffff:fe65:1f00/64 Scope:Link
    UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
    RX packets:1223 errors:0 dropped:0 overruns:0 frame:0
    TX packets:1138 errors:0 dropped:1 overruns:0 carrier:0
    collisions:0 txqueuelen:200
    RX bytes:118971 (116.1 KiB) TX bytes:122625 (119.7 KiB)

    asbr0 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:74
    inet addr:200.53.114.43 Bcast:200.53.114.47 Mask:255.255.255.240
    inet6 addr: fe80::20c:29ff:feae:ba74/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:4861 errors:0 dropped:0 overruns:0 frame:0
    TX packets:2774 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:541219 (528.5 KiB) TX bytes:498187 (486.5 KiB)

    eth0 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:74
    inet6 addr: fe80::20c:29ff:feae:ba74/64 Scope:Link
    UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
    RX packets:41472 errors:0 dropped:0 overruns:0 frame:0
    TX packets:4003 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:22860963 (21.8 MiB) TX bytes:617626 (603.1 KiB)

    eth1 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:7e
    inet addr:172.1.50.9 Bcast:172.1.50.15 Mask:255.255.255.240
    inet6 addr: fe80::20c:29ff:feae:ba7e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:17364 errors:0 dropped:0 overruns:0 frame:0
    TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:1438884 (1.3 MiB) TX bytes:468 (468.0 B)

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:31 errors:0 dropped:0 overruns:0 frame:0
    TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:13074 (12.7 KiB) TX bytes:13074 (12.7 KiB)

    Is it normal that asbr0 get ip 200.53.114.43 ?

    If I keep mode to : Layer 2, but but put bridge name asbr0, ifconfig give me :
    eth0 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:74
    inet addr:200.53.114.43 Bcast:200.53.114.47 Mask:255.255.255.240
    inet6 addr: fe80::20c:29ff:feae:ba74/64 Scope:Link
    UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
    RX packets:42295 errors:0 dropped:0 overruns:0 frame:0
    TX packets:5015 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:23005868 (21.9 MiB) TX bytes:820754 (801.5 KiB)

    eth1 Link encap:Ethernet HWaddr 00:0c:29:ae:ba:7e
    inet addr:172.1.50.9 Bcast:172.1.50.15 Mask:255.255.255.240
    inet6 addr: fe80::20c:29ff:feae:ba7e/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:17631 errors:0 dropped:0 overruns:0 frame:0
    TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:1460972 (1.3 MiB) TX bytes:468 (468.0 B)

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:31 errors:0 dropped:0 overruns:0 frame:0
    TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:13074 (12.7 KiB) TX bytes:13074 (12.7 KiB)

    But it’s the same, from home I get 172.1.50.12 but can’t ping 172.1.50.x nodes.

    I spent some days without success. I suppose I don’t have all the knowledge to fully understand the problem.
    I’m teacher for public school and need this bridge for my students to learn some programing…

    Please if someonr can help.

    Thanks a lot.

    Vincent MARECHAL

    Reply
  5. Trent B
    Trent B says:

    And yet another satisfied reader. I only spent 3 days hunting down this issue, tried several vpn solutions attempting to simply bridge taps with eths in Linux VMs on an ESXi 4.1 host. The second (or maybe 2 seconds later) I flipped promiscous to accept on the vswitch that was dropping everything, ICMP from remote network segment started reaching into the internal network.
    Thanks!

    Reply
  6. Adam
    Adam says:

    I wish I could say I found this page before I spent around 4 hours trying to solve why I couldn’t ping my internal LAN. I was trying tcpdump, running debugs on my Cisco router and was getting nowhere.

    As i’d ran this config perfectly fine on a physical box before, my efforts turned towards VMWare and I was digging around and ticket the promiscious box and voila it bloody worked!!

    I just wished I had found this page 4 hours ago! Thanks for confirming it though 🙂

    Reply
  7. Andi
    Andi says:

    Thanks for this great blog entry. I ran into exact the same problem as you, i wanted to move our OpenVPN from physical to our new ESXi Cluster. The old VPN runs under Debian but i tried Ubuntu for the new one. When it didn´t work and i could not make out the problem, i supected Ubuntu to show strange behaviour. So i switched to Debian but the same here, despite of exactly matching configs. Today i used tcpdump to locate the Problem, and i suspected our firewall to be missconfigured. I dont know how but suddenly there was the word promisc in my mind which meaning i didnt know. After searching for that term i thought that i forgot to activate promisc-mode on my tap device but no it´s set. Then your post appeared and i nearly dropped from my chair 🙂

    Thank you 1000 times!

    Reply
  8. Ewigan
    Ewigan says:

    seriously our esxi was working just fine with openvpn and everything.
    but then we needed more hardware (ram and an added hdd) and from then on everything went kablooey.
    as i didn’t set up the server in the first place, i wasn’t even aware that “promiscuous mode” could become a problem. so from the addition of the hardware, as i said, everything went kablooey. after 2 weeks of just thinking “well, guess we made a serious error in the code for our product”, i tried to corner the problem.
    local machine => check
    other machines on the network => check
    vm’s on those machines => check
    vm’s on the esxi => NO check
    so i tried to work out why… and after some hours asked big daddy google.
    found your blog.
    switched the DAMNED promiscuous mode to accept.
    AND EVERYTHING WORKS.

    thank you SO much man!

    Reply
  9. Jason
    Jason says:

    Hey man, thanks for the clear instructions. Even though I *knew* it was a promiscuous mode problem, even finding out how to fix the problem is not a trivial search. Great post.

    Reply
  10. DAvid
    DAvid says:

    Dude…
    I usually don’t leave comment’s on people blogs as they don’t help me out this much! I have spent almost 4 days trying to figure out why the traffic wasn’t flowing through, knowing full well that I have set up 20-30 OpenVPN devices on hardware alone in the past and NEVER had this much trouble. Literally within a second of changing promiscuous mode to accept everything worked! THANK YOU!!!!!!!!!

    Reply

Trackbacks & Pingbacks

  1. […] some digging, this post pointed me to the probable solution. The Virtual Switch that sat in ESX between the router and the […]

  2. […] which I failed to mention, which is that this server lives on a VMWare virtual machine. Thanks to this blog post, I discovered that “the ESXi virtual switch drops promiscuous packets by default,” and […]

  3. […] article on how to configure ESXi for this to work, because apparently, in its default configuration, ESXi can cause some problems getting this to […]

  4. […] much Google searching, which turned up issues from promiscuous mode needing to be enabled (didn’t do anything for […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *