How I exported AUFS via NFS
Use case
The goal is to create a SaaS application out of one of popular CMS systems. I want several thousands installations, I want bulk updates, I want a single code base and I want no modifications to core source code.
AUFS looked like a perfect solution. Among many of its possible configurations, one allows to have a base read-only directory which is mounted with an overlay. This works as following: let's assume that base directory is named /base and the overlay (which is called branch in AUFS slang) is called /br1. When you mount /base to /br1 mountpoint using aufs filesystem, all files and directories existing in /base will appear in /br1. All changes you make in /base, also appear in /br1 in real time. All changes made to /br1 are saved only locally in /br1.
One example that comes to my mind is a LCD screen (/base) on which you place transparent film (/br1). You draw on the film and anything you have drawn remains, but on the empty space, you see what the LCD screen shows.
In my use case, I put my dream CMS install in /base and mount it to /br1, /br2 and so on - one branch per user. He is allowed to install tempaltes, upload images and all these actions only modify his home directory - the AUFS branch. However, whenever I install any updates to the CMS software, all the users automatically get the modified files. As a side effect, less disk space is also used, however this was not the goal.
AUFS
In order to use AUFS, you need to build a kernel module. This process is rather described on their website and rather straightforward (unless you need to use some heavily modified kernel image, which was my case). Options regarding AUFS appear under file systems in make menuconfig. Be sure to tune maximum number of branches and check "NFS-exportable AUFS".
"Show whiteouts" is an interesting, however poorly-named option. Whiteouts in AUFS slang are files with names like .wh..wh.aufs and this is the real place where differences between base dir and overlay are kept. They are owned by root and not accessible to user. While showing them might seem a ridiculous idea, you may want to consider this when you want to replicate AUFS branches among servers without unmounting them.
When you build the module, you can mount the directories using the following command: mount -t aufs -o br:/br1=rw:/base=rr none /br1
AUFS+NFS
When you export AUFS branch as a separate share, everything works great. When you try to export a directory containing multiple shares, things start to get complicated. In theory, it is possible to mount it using NFSv3 - Junjiro Okajima, author of AUFS suggested using nohide and crossmnt options in /etc/exports. In practice, I failed to export AUFS shares on NFSv3.
I however succeeded using NFSv4 (tip: in case you have a recent 2.6 kernel and v4 does not work or mounting hangs - upgrade nfs-utils!). My /etc/exports looked like this:
/mnt/data 0.0.0.0/0.0.0.0(rw,fsid=0,insecure,no_subtree_check,sync,crossmnt) /mnt/data/test/br1 0.0.0.0/0.0.0.0(rw,sync,fsid=999) /mnt/data/test/br2 0.0.0.0/0.0.0.0(rw,sync,fsid=999)
The crossmnt option is crucial for the process to succeed and tells NFS that other filesystem are mounted under this directory, and NFS should export them.
Unfortunately, you have to add a separate entry for each mount exported - you then re-export them using exportfs -r. Whenever a branch is added, you also need to re-mount the share at NFS client (mount -o remount).
Also, when exporting via NFS, the AUFS mount should have xino=/tmp/somefile option attached, with the file different for different share.
On the client side, mounting the share is as easy as mount -t nfsv4 host:/ /mnt/point. Viola, you have your AUFS exported.
This was a very brief howto, for details and reasons why it's done this way and not other, look at aufs and nfs manual pages, as well as aufs-users mailing list archives.
Thanks
Thanks for Junjiro Okajima, the AUFS author for responding me very fast on the aufs-users mailing list (and for writing AUFS, of course).