Notes on a Proxy for Securely Tunnelling NFS through Firewalls

Introduction

As my résumé indicates, a few years back, I designed and partially built an NFS proxy server to permit tunneling NFS traffic safely from behind a corporate firewall to machines in the firewall's DMZ. Recently, someone out on the net stumbled across a reference to this project, and asked me about it. After rummaging through my memory and recreating all the gory details, it occurred to me to capture them in this document.

Note: I never completed this work, and I no longer have a copy of the software I was building. This means I can offer you no actual software to download and peruse. Sorry.

The Gory Details

In late 1996, the new media company for which I worked owned a number of music-related web sites. From behind the firewall, a small army of editors and graphics gurus maintained and enhanced the content. This content needed to be staged periodically to the live systems running in our DMZ. Our initial infrastructure relied on FTP uploads from quasi-secure machines, but this proved unwieldy for a number of reasons:

1. Our content developers used Mac and Windows machines, while our servers ran UNIX. At a time when network address translation software wasn't widely available, our firewall used a SOCKS server as a general-purpose proxy mechanism. Finding SOCKSified FTP clients for Mac and Windows platforms was difficult. The firewall also used the Firewall Toolkit (FWTK) ftp-gw proxy, but training our content management folks to configure their FTP clients appropriately was an ongoing chore.
2. The content developers were accustomed to an environment where security didn't exist. They preferred to log into the live servers and updating content as needed; they found our security measures to be cumbersome and annoying. We wanted to implement a solution that was more transparent to them, without compromising security.

Since our content developers had ready access to off-the-shelf NFS client software, I suggested implementing an NFS proxy, a customized NFS server that could run on a machine behind the firewall but read and write files to the web servers in the DMZ using a secure protocol.

I started with the Linux user-mode NFS server. The general idea was to replace only the file-access portion of the NFS server's back-end; instead of accessing local files, the back-end would transfer the file operations to a server process on the DMZ machine, using a cryptographically secure TCP protocol akin to Bill Cheswick's cget, cput and stage routines. You can find a Postscript copy of Cheswick's original Usenix paper on these routines at http://www.cheswick.com/ches/papers/crio.ps.

I couldn't actually use Cheswick's software, because Bell Labs had not yet decided to make it publicly available, and it wasn't likely to be released for commercial use in any case. (The software is available now, though only for non-commercial use. See http://www.cheswick.com/ches/software/software.html.) However, Cheswick (a friend for some 20 years) had sent me an early draft of his paper, and the information it contained was very helpful in clarifying my approach. A quote from his abstract describes how the software works:

Cget, cput and stage are three simple programs that implement authenticated or encrypted file transfers on the Internet. Cget and cput read and write files to a remote hots, and stage ensures that a remote directory accurately mirrors a local master directory.

These routines use private key cryptography for authentication and privacy between pairs of secured hosts. They are simple, paranoid UNIX tools that can be used to support systems that operate in a hostile environment.

I figured I could put together something roughly equivalent to Cheswick's software (though probably less comprehensive, at least initially) for use with the modified NFS server.

A description of how the NFS server might have been used is the best way to clarify what I was trying to do. For ease of explanation, let's assume the existence of three server machines:

safe	A machine on the internal network, behind the firewall. Safe runs the modified NFS server.
fw	The firewall. In our setup, this machine was the bastion host, one of several components of our custom-built firewall. Fw sits between the machines on the secured inside network and the more vulnerable machines in the DMZ.
www	A DMZ-resident web server. Www runs the custom server to which the modified NFS server connects.

This simple scenario illustrates how the components work together.

• A user wishing to access a roped-off file system on www mounts, via NFS, a directory on safe. The mount operation fires up the modified NFS server. The NFS server immediately makes a cryptographically secure TCP connection to the server running on www; the TCP connection is held open as long as the user has the file system mounted.
• The NFS server on safe initiates a cryptographically secure TCP connection with a server on www. The server on www runs in a jail (minimally, in a chrooted area).
• When the user writes data to a file, the NFS server copies the written data and the file ID across the TCP connection to www, where the server updates the file. When the user reads data from a file, the NFS server uses the secure connection to the server on www to obtain the data to satisfy the read. Tunnelling through fw was to be accomplished via SOCKS. (Nowadays, I'd used network address translation, instead.)

Running a purely user-mode NFS server had some potential performance implications, of course. I knew it'd be slower than a kernel-based implementation, but I didn't really care, since:

• It was bound to be more convenient to use than SOCKSified FTP, which is what we were forcing our unhappy Mac and Windows users to put up with.
• It was a simpler and more maintainable solution than modifying the operating system's kernel-based NFS implementation. Not only is it more difficult to debug kernel code, but modifying the kernel can destabilize the operating system. The risks are much higher, and those risks weren't outweighed by corresponding benefits. Plus, I didn't want to worry about constantly upgrading custom kernel code to account for changes in the operating system. A standalone user-mode program would be far easier to maintain and keep isolated.
• The NFS server was never going to be used for high-demand file system access. No one would be putting his home directory on the web server, for instance. It was primarily intended to permit users to upload and synchronize content between local machines and the live system. Convenience of use, reliability, and security were the top priorities. It didn't have to be blindingly fast; it only had to be fast enough.

The use of (tunneled) TCP, instead of UDP, between safe and www simplified the implementation; I could let TCP handle any retries and packet drops. Using TCP also simplified the deployment, since it was much easier to tunnel TCP through our SOCKS-based firewall. Of course, maintaining open connections between the modified NFS server on safe and the file server process on www could have resulted in wasted network resources; however, given the proposed usage profile, these considerations didn't worry me too much. If the NFS server tended to use too many open TCP connections, we'd just beef up the memory in the various systems and reconfigure the various operating system kernels to permit more open TCP connections. It was more important to keep the implementation as simple as possible, given the need to deploy the solution in a security-sensitive environment.

The planned phases of implementation were

1. Port the Linux-based NFS server to BSD. Our back-end server was running BSD/OS, a BSD-based platform. Porting the software was more work than I initially suspected, since the server made use of some Linux-specific system calls that had clearly been put in the kernel solely for the purpose of supporting the user-mode NFS server (e.g., setfsuid(2)).
2. Determine what cryptographic protocol(s) to use between the NFS server program and the web server file access process.
3. Write the file access server to be run on the DMZ machine, as well as the associated client-side routines that would let the NFS server communicate with it.
4. Replace the NFS server's file access code with calls to the routines written in the previous step.
5. Test thoroughly. During this phase, I planned to run a series of benchmarks to capture file access times using the modified NFS server, comparing those times to similar benchmarks for (a) an unmodified user-mode NFS server, and (b) a standard kernel-based NFS server.

I managed to finish Step 1 before being sidetracked onto more pressing fire drills. I ended up leaving the company before I could get back to the project. I have confidence that this approach would have worked and would have provided us with a relatively simple but flexible solution to our problem.

$Id: nfs-proxy.html 6964 2007-08-12 15:24:39Z bmc $