Ray Van Dolson's Blog

Pontifications from smoggy Southern California

Posts Tagged ‘apache

RHEL’s Apache and LDAP Referrals

with 4 comments

After upgrading to RHEL 5.7, one of my Apache servers which protects content using mod_ldap authenticating against Active Directory stopped working. Error logs showed the following:

auth_ldap authenticate: user xxxx authentication failed; URI /viewvc/ 
[ldap_search_ext_s() for user failed][Operations error]


This usually means there is a problem with referrals, but OpenLDAP was properly configured to not follow them (/etc/openldap.ldap.conf w/ REFERRALS off).

Turns out that buriedin the RHEL 5.7 release notes, this gem can be found. A brand new configuration directive called LDAPChaseReferrals. Apparently, when referrals are provided in a result-set, mod_ldap by default does not bind to them using the provided credentials. The new version of Apache in RHEL 5.7 corrects that and provides this new directive to enable or disable the feature.

However, I couldn’t find any documentation for it upstream, and after following the bugzilla link in the release notes realized that the Apache project had actually gone a different route and called their directive LDAPReferrals in addition to adding another directive to limit the number of hops that would be followed.

End result is that it appears Red Hat has added in a one-off and undocumented configuration directive. 🙂 Likely unintentionally, but this definitely led me to some confusion.

Opened a new bug to see about getting this resolved.

Written by rayvd

August 1, 2011 at 2:21 pm

Posted in Systems Administration, Technology

Tagged with , ,

Debugging Juniper VPN

leave a comment »

In the process of trying to get OpenJDK + IcedTea to allow me to connect to my work’s Juniper based VPN (see this), I really needed to take a peek at what the applets were transmitting to the VPN site.

Wireshark and Apache to the Rescue!

Wireshark has a nifty feature where it is able to decrypt SSL, so it seemed like a no-brainer to make use of it. Unfortunately, snagging the private key out of the VPN server at work wasn’t super straightforward (really not sure where it’s stored — the only thing we could easily extract without too much digging where we shouldn’t be was the certificate file which contained no private key). Bummer.

My next thought was to try and use stunnel listening on port 80 on my local machine and retransmitting to my VPN server’s port 443. This almost worked, but unfortunately there were too many 3xx redirects that had https embedded in them which kept fouling me up.

With stunnel not quite doing what I needed, I turned to Apache and reverse proxying. With Apache as my local SSL endpoint, I could have control of the certificate and private key and decrypt the SSL session between my browser and
Apache! Apache would then forward the packets on in a completely new SSL session to my company’s VPN server. Here’s how I got it going:

  1. First, let’s make sure we have a clean environment on the client side:

    client% rm -rf ~/.icedteaplugin
    client% rm -rf ~/.juniper_networks
  2. Set up a virtual host on your Apache server (I used a separate physical machine than my local machine to avoid DNS issues):

       # This should be the DNS name of your work VPN
       ServerName workvpn.domain.com:443
       SSLEngine on
       SSLProtocol all -SSLv2
       # We set this explicitly for Wireshark's benefit.
       SSLCipherSuite RSA:+MEDIUM:+LOW
       SSLCertificateFile /etc/pki/tls/certs/localhost.crt
       # This is the key you'll need to decrypt packets.
       SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
       <Files ~ "\.(cgi|shtml|phtml|php3?)$">
         SSLOptions +StdEnvVars
       <Directory "/var/www/cgi-bin">
         SSLOptions +StdEnvVars
       SetEnvIf User-Agent ".*MSIE.*" \
                 nokeepalive ssl-unclean-shutdown \
                 downgrade-1.0 force-response-1.0
       SSLProxyEngine on
       ProxyRequests off
       SSLProxyVerify none
       ProxyPreserveHost on
       <Location />
         # Again, DNS names for your work VPN.
         ProxyPass https://workvpn.domain.com/
         ProxyPassReverse https://workvpn.domain.com/
  3. Copy the /etc/pki/tls/private/localhost.key file from your Apache server to your home directory on your local machine. We’ll need it later to decrypt packets.

    server# cd /etc/pki/tls/private
    server# scp localhost.key rayvd@client:~/
  4. Next, on my local machine (, I added an entry to my /etc/hosts file for workvpn.domain.com:    workvpn.domain.com
  5. Staying in the DNS world, I also found that while Firefox seems to make use of the /etc/hosts file for lookups, it didn’t seem that Java applets always did. I wasn’t clear if the Java plugin subsystem used its own DNS cache (tried disabling Firefox’s DNS cache) or some other method, but, in the end I modified my own local DNS cache server to return the entry for workvpn.domain.com. My DNS server is BIND, and I added a “workvpn.domain.com” zone as follows:

    zone "workvpn.domain.com" {
      type master;
      file "workvpn.domain.com";

    I then created the zone file with the following contents:

    $ORIGIN .
    $TTL 86400
    workvpn.domain.com     IN      SOA     ns.bludgeon.org. hostmaster.bludgeon.org. (
                                    2h )
                    IN      NS      ns.bludgeon.org.
                    IN      A

    Reload your nameserver and test via dig that workvpn.domain.com resolves to
    your Apache server’s IP now.

  6. Finally we should be ready to capture some packets. On your local machine, prepare tcpdump to look for packets going to your Apache server:

    client# tcpdump -i eth0 -n -s 0 -w /tmp/vpn.cap host and port 443
  7. Next, jump over to Firefox, browse to your company’s VPN site, log in and start the NetworkConnect client software. You can keep your eye on the tcpdump counter as it captures packets. If you’re using Sun’s JRE, the client will be retrieved successfully and then fail to fully connect since it attempts to talk to your VPN server via port 4500 which we haven’t bothered to forward.
  8. Once you’ve captured some packets, stop the tcpdump and load the resulting /tmp/vpn.cap into Wireshark. Use the SSL protocol filter with options set to something like the following:,443,http,/home/rayvd/localhost.key

    This should be enough to decrypt the HTTPS packets and, with luck, you should see the GET requests and other information you’re interested in!

  9. When you’re ready to re-run the configuration, it’s usually a good idea to clear out any Java related caches:

    client% rm -rf ~/.icedteaplugin
    client% rm -rf ~/.juniper_networks

By the way, after examining output from both the Sun plugin and IcedTea’s, the hunch that Sun’s set cookie information while IcedTea did not, turned out to be correct.


  • To make discerning between the Firefox requests and the Java HTTP requests easier, you might want to configure Firefox to use SSLv3 only. The Java applet will insist on using TLSv1 which will make it easier to spot the Firefox vs Java packets in the Wireshark output.
  • In the Apache configuration, I initially tried disabling everything but SSLv3. I found this worked fine for Firefox, but the Java client insists on using TLSv1 and errors out if it’s not available.
  • When I initially tried this without using the custom BIND zone file, I could capture packets just fine from Firefox, but the Java client would fire up and magically begin communicating directly to my company’s VPN server via HTTPS and completely bypassing my proxy. It either apparently uses its own DNS resolution method that bypasses /etc/hosts or something was getting cached.
  • I had some hits and misses with getting Wireshark to decrypt packets. At times it seemed a little voodoo’ish as to whether it would work or not. I believe perhaps that my global Apache config file’s SSL options were overriding the options I created in my Virtual Host above and were triggering non-RSA key exchange algorithms which Wireshark can’t decode. Just to be on the safe side I edited my /etc/httpd/conf.d/ssl.conf file (CentOS 5) and made the SSL settings there match the ones in my Virtual Host.

Written by rayvd

February 23, 2009 at 8:41 pm