Ray Van Dolson's Blog

Pontifications from smoggy Southern California

My First Greasemonkey Script

leave a comment »

My first Greasemonkey script compels MarkMail‘s message view pane to use a fixed width font instead of a variable width font. Much nicer.

Written by rayvd

October 25, 2009 at 9:03 pm

File Locks on Solaris 10

with 6 comments

In the process of troubleshooting a file locking issue on a Samba/NFS server, I needed to be able to take a look at the locks on a Solaris 10 system. In Linux this is fairly straightforward to do with the lslk command or by taking a peek at /proc/locks. No such luck on Solaris.

Fortunately, I stumbled across this excellent reference and was introduced to the Solaris Modular Debugger (mdb).

The ::lminfo command gave me almost exactly what I needed, except, as Chris mentions in his wiki entry, the path information is truncated. You can easily cycle through and print only the path out, but then you’re missing the rest of the information which is awfully nice to see.

> ::lminfo
ADDR             TP FLAG    PID COMM             VNODE            PATH
600114a7040      WR 0021    315 ypbind           60012161080      /var/yp/binding/xpr
6001203ea00      WR 0021    315 ypbind           60012160080      /var/yp/binding/xpr
6001203e800      WR 0021    315 ypbind           600115b0140      /var/yp/binding/xpr
60011452700      WR 0001    558 mdmonitord       600131da100      /etc/lvm/.mdmonitor
6001203eb00      WR 0021    315 ypbind           60012160180      /var/yp/binding/xpr
60010a90e80      WR 0001    505 automountd       60013024180      /etc/svc/volatile/f

I couldn’t figure out a good way to convince ::print to display multiple lock_descriptor_t members — and format it as nicely as ::lminfo did. I was about to write an external parser in awk or python to hack this together, when Jonathan Adams of Sun suggested that an mdb module could be created to accomplish just what I was after.

After some trials and tribulations getting this going, I was able to create a ::lminfo2 module that not only displays the pathname of the locked file sans truncation, but also spits out the whence, start and length information for ranged locks! Sample output:

# echo "::load /home/rayvd/src/mdb/sparcv9/lminfo2.so; ::lminfo2" | mdb -k
ADDR             TP FLAG    PID COMM             VNODE            WHENCE    START     LEN       PATH
600114a7040      WR 0021    315 ypbind           60012161080      1         0         1         /var/yp/binding/xprt.udp.2
6001203ea00      WR 0021    315 ypbind           60012160080      1         0         1         /var/yp/binding/xprt.ticlts.2
6001203e800      WR 0021    315 ypbind           600115b0140      1         0         1         /var/yp/binding/xprt.ticotsord.3
60011452700      WR 0001    558 mdmonitord       600131da100      0         0         0         /etc/lvm/.mdmonitord.lock
6001203eb00      WR 0021    315 ypbind           60012160180      1         0         1         /var/yp/binding/xprt.ticlts.3
60010a90e80      WR 0001    505 automountd       60013024180      0         0         0         /etc/svc/volatile/filesystem-autofs.lock

The main challenge I encountered was dealing with the mdb_printf and mdb_snprintf commands. Both are “smart” in that they automatically truncate lines at the end of the terminal.

To build the module, you need a C compiler, the SUNWmdbdm package, and also, a header file (mdb_ks.h) from the mdb sources (available in OpenSolaris) to gain access to some internal mdb functions not exposed by mdb_modapi.h.

The module, and some basic instructions on building are available here. Feedback welcome.

Written by rayvd

October 25, 2009 at 8:16 pm

Posted in Systems Administration, Technology

Tagged with ,

Tricks with find

leave a comment »

I was attempting to find the newest C file in a tree of files I’d checked out from CVS. find to the rescue:

$ find . -name '*.c' -printf '%-50p %-15T@ %T+\n' | sort -k2
./ssl/main.c                                       1039831674      2002-12-13+18:07:54
./ssl/lex.yy.c                                     1039831674      2002-12-13+18:07:54
./ssl/ssl_enum.c                                   1039831678      2002-12-13+18:07:58
./ssl/y.tab.c                                      1039831683      2002-12-13+18:08:03
./common/lib/debug.c                               1039831686      2002-12-13+18:08:06
./common/lib/r_list.c                              1039831689      2002-12-13+18:08:09
./common/lib/r_time.c                              1039831689      2002-12-13+18:08:09
./common/lib/r_errors.c                            1039831689      2002-12-13+18:08:09
./common/lib/r_replace.c                           1039831689      2002-12-13+18:08:09
./common/lib/r_assoc_test.c                        1039831689      2002-12-13+18:08:09
./common/lib/threads/pthreads/pthread.c            1039831690      2002-12-13+18:08:10
./base/debug.c                                     1039831693      2002-12-13+18:08:13
./base/common.c                                    1039831693      2002-12-13+18:08:13
./base/proto_mod.c                                 1039831694      2002-12-13+18:08:14
./base/print_utils.c                               1039831694      2002-12-13+18:08:14
./base/tcpconn.c                                   1041533083      2003-01-02+10:44:43
./null/null_analyze.c                              1041533086      2003-01-02+10:44:46
./ssl/ssl_analyze.c                                1041533087      2003-01-02+10:44:47
./ssl/ciphersuites.c                               1051291844      2003-04-25+10:30:44
./ssl/ssl_rec.c                                    1051291846      2003-04-25+10:30:46
./common/lib/r_data.c                              1166728933      2006-12-21+11:22:13
./common/lib/r_assoc.c                             1166728933      2006-12-21+11:22:13
./common/lib/r_bitfield.c                          1166728933      2006-12-21+11:22:13
./ssl/sslprint.c                                   1166728991      2006-12-21+11:23:11
./ssl/ssl.enums.c                                  1166728991      2006-12-21+11:23:11
./ssl/sslxprint.c                                  1166728991      2006-12-21+11:23:11
./base/network.c                                   1166729027      2006-12-21+11:23:47
./base/tcppack.c                                   1166729027      2006-12-21+11:23:47
./base/pcap-snoop.c                                1166729027      2006-12-21+11:23:47
./ssl/ssldecode.c                                  1247069555      2009-07-08+09:12:35

Written by rayvd

July 8, 2009 at 8:35 pm

Posted in Systems Administration, Technology

Tagged with ,

Recording Streaming Automatically

leave a comment »

I wanted to record a streamed radio show automatically once a week at a certain time, for a certain duration. Enter mplayer, cron and a simple shell script:

# show.sh
URL="mmsh://url"
DATE=$(date +%Y%m%d)
OUTFILE="$HOME/Music/Show/show-$DATE.asf"

[ -f "$OUTFILE" ] && rm -f "$OUTFILE"

mplayer -dumpstream -dumpfile $OUTFILE $URL &
PID=$!
sleep 10800
kill $PID

Then add a cron entry as follows:

0 7 * * Sun $HOME/bin/show.sh

This will record the specified stream for three hours, every Sunday at 7am local time.

Written by rayvd

June 23, 2009 at 8:38 pm

Debugging Juniper VPN

leave a comment »

In the process of trying to get OpenJDK + IcedTea to allow me to connect to my work’s Juniper based VPN (see this), I really needed to take a peek at what the applets were transmitting to the VPN site.

Wireshark and Apache to the Rescue!

Wireshark has a nifty feature where it is able to decrypt SSL, so it seemed like a no-brainer to make use of it. Unfortunately, snagging the private key out of the VPN server at work wasn’t super straightforward (really not sure where it’s stored — the only thing we could easily extract without too much digging where we shouldn’t be was the certificate file which contained no private key). Bummer.

My next thought was to try and use stunnel listening on port 80 on my local machine and retransmitting to my VPN server’s port 443. This almost worked, but unfortunately there were too many 3xx redirects that had https embedded in them which kept fouling me up.

With stunnel not quite doing what I needed, I turned to Apache and reverse proxying. With Apache as my local SSL endpoint, I could have control of the certificate and private key and decrypt the SSL session between my browser and
Apache! Apache would then forward the packets on in a completely new SSL session to my company’s VPN server. Here’s how I got it going:

  1. First, let’s make sure we have a clean environment on the client side:

    client% rm -rf ~/.icedteaplugin
    client% rm -rf ~/.juniper_networks
    
  2. Set up a virtual host on your Apache server (I used a separate physical machine than my local machine to avoid DNS issues):

     <VirtualHost 192.168.20.3:443>
       # This should be the DNS name of your work VPN
       ServerName workvpn.domain.com:443
       SSLEngine on
       SSLProtocol all -SSLv2
       # We set this explicitly for Wireshark's benefit.
       SSLCipherSuite RSA:+MEDIUM:+LOW
       SSLCertificateFile /etc/pki/tls/certs/localhost.crt
       # This is the key you'll need to decrypt packets.
       SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
    
       <Files ~ "\.(cgi|shtml|phtml|php3?)$">
         SSLOptions +StdEnvVars
       </Files>
       <Directory "/var/www/cgi-bin">
         SSLOptions +StdEnvVars
       
       SetEnvIf User-Agent ".*MSIE.*" \
                 nokeepalive ssl-unclean-shutdown \
                 downgrade-1.0 force-response-1.0
    
       SSLProxyEngine on
       ProxyRequests off
       SSLProxyVerify none
       ProxyPreserveHost on
    
       <Location />
         # Again, DNS names for your work VPN.
         ProxyPass https://workvpn.domain.com/
         ProxyPassReverse https://workvpn.domain.com/
       </Location>
     </VirtualHost>
    
  3. Copy the /etc/pki/tls/private/localhost.key file from your Apache server to your home directory on your local machine. We’ll need it later to decrypt packets.

    server# cd /etc/pki/tls/private
    server# scp localhost.key rayvd@client:~/
    
  4. Next, on my local machine (192.168.20.101), I added an entry to my /etc/hosts file for workvpn.domain.com:

     192.168.20.3    workvpn.domain.com
    
  5. Staying in the DNS world, I also found that while Firefox seems to make use of the /etc/hosts file for lookups, it didn’t seem that Java applets always did. I wasn’t clear if the Java plugin subsystem used its own DNS cache (tried disabling Firefox’s DNS cache) or some other method, but, in the end I modified my own local DNS cache server to return the 192.168.20.3 entry for workvpn.domain.com. My DNS server is BIND, and I added a “workvpn.domain.com” zone as follows:

    zone "workvpn.domain.com" {
      type master;
      file "workvpn.domain.com";
    };
    

    I then created the zone file with the following contents:

    $ORIGIN .
    $TTL 86400
    workvpn.domain.com     IN      SOA     ns.bludgeon.org. hostmaster.bludgeon.org. (
                                    2009022201
                                    2h
                                    30m
                                    6w
                                    2h )
                    IN      NS      ns.bludgeon.org.
                    IN      A       192.168.20.3
    

    Reload your nameserver and test via dig that workvpn.domain.com resolves to
    your Apache server’s IP now.

  6. Finally we should be ready to capture some packets. On your local machine, prepare tcpdump to look for packets going to your Apache server:

    client# tcpdump -i eth0 -n -s 0 -w /tmp/vpn.cap host 192.168.20.3 and port 443
    
  7. Next, jump over to Firefox, browse to your company’s VPN site, log in and start the NetworkConnect client software. You can keep your eye on the tcpdump counter as it captures packets. If you’re using Sun’s JRE, the client will be retrieved successfully and then fail to fully connect since it attempts to talk to your VPN server via port 4500 which we haven’t bothered to forward.
  8. Once you’ve captured some packets, stop the tcpdump and load the resulting /tmp/vpn.cap into Wireshark. Use the SSL protocol filter with options set to something like the following:

    192.168.20.3,443,http,/home/rayvd/localhost.key
    

    This should be enough to decrypt the HTTPS packets and, with luck, you should see the GET requests and other information you’re interested in!

  9. When you’re ready to re-run the configuration, it’s usually a good idea to clear out any Java related caches:

    client% rm -rf ~/.icedteaplugin
    client% rm -rf ~/.juniper_networks
    

By the way, after examining output from both the Sun plugin and IcedTea’s, the hunch that Sun’s set cookie information while IcedTea did not, turned out to be correct.

Notes

  • To make discerning between the Firefox requests and the Java HTTP requests easier, you might want to configure Firefox to use SSLv3 only. The Java applet will insist on using TLSv1 which will make it easier to spot the Firefox vs Java packets in the Wireshark output.
  • In the Apache configuration, I initially tried disabling everything but SSLv3. I found this worked fine for Firefox, but the Java client insists on using TLSv1 and errors out if it’s not available.
  • When I initially tried this without using the custom BIND zone file, I could capture packets just fine from Firefox, but the Java client would fire up and magically begin communicating directly to my company’s VPN server via HTTPS and completely bypassing my proxy. It either apparently uses its own DNS resolution method that bypasses /etc/hosts or something was getting cached.
  • I had some hits and misses with getting Wireshark to decrypt packets. At times it seemed a little voodoo’ish as to whether it would work or not. I believe perhaps that my global Apache config file’s SSL options were overriding the options I created in my Virtual Host above and were triggering non-RSA key exchange algorithms which Wireshark can’t decode. Just to be on the safe side I edited my /etc/httpd/conf.d/ssl.conf file (CentOS 5) and made the SSL settings there match the ones in my Virtual Host.

Written by rayvd

February 23, 2009 at 8:41 pm