NFS-Ganesha is a user-space NFS-server that is available in Fedora. It contains several plugins (FSAL, File System Abstraction Layer) for supporting different storage backends. Some of the more interesting are:
- VFS: a normal mounted filesystem
- GLUSTER: libgfapi based access to a Gluster Volume
- CEPH: libcephfs based access to the filesystem on a Ceph Cluster
Setting up a basic NFS-Ganesha server
Exporting a mounted filesystem is pretty simple. Unfortunately this failed for me when running with the standard nfs-ganesha packages on a minimal Fedora 20 installation. The following changes were needed to make NFS-Ganesha work for a basic export:
- install
rpcbindand make thenfs-ganesha.servicedepend on it - copy
/etc/dbus-1/system.d/org.ganesha.nfsd.conffrom the sources - createa
/etc/sysconfig/nfs-ganeshaenvironment file
When these initial things have been taken care of, a configuration file needs to be created. The default configuration file mentioned in the environment file is /etc/ganesha.nfsd.conf. The sources of nfs-ganesha contain some examples, the vfs.confis quite usable as a starting point. After copying the example and modifying the paths to something more suitable, starting the NFS-server should work:
# systemctl start nfs-ganesha
In case something failed, there should be a note about it in /var/log/ganesha.log.
Exporting the Ceph filesystem with NFS-Ganesha
This assumes you have a working Ceph Cluster which includes several MON, OSD and one or more MDS daemons. The FSAL_CEPH from NFS-Ganesha uses libcephfs which seems to be the same as the ceph-fuse package for Fedora. the easiest way to make sure that the Ceph filesystem is functional, is to try and mount it with ceph-fuse.
The minimal requirements to get a Ceph client system to access the Ceph Cluster, seem to be a /etc/ceph/ceph.conf with a [global]section and a suitable keyring. Creating the ceph.conf on the Fedora system that was done the ceph-deploy:
$ ceph-deploy config push $NFS_SERVER
In my setup I scp'd the /etc/ceph/ceph.client.admin.keyring from one of my Ceph servers to the $NFS_SERVER. There probably are better ways to create/distribute a keyring, but I'm new to Ceph and this worked sufficiently for my testing.
When the above configuration was done, it was possible to mount the Ceph filesystem on the Ceph client that is becoming the NFS-server. This command worked without issues:
# ceph-fuse /mnt
# echo 'Hello Ceph!' > /mnt/README
# umount /mnt
The first write to the Ceph filesystem took a while. This is likely due to the initial work the MDS and OSD daemons need to do (like creating pools for the Ceph filesystem).
After confirming that the Ceph Cluster and Filesystem work, the configuration for NFS-Ganesha can just be taken from the sources and saved as /etc/ganesha.nfsd.conf. With this configuration, and restarting the nfs-ganesha.service, the NFS-export becomes available:
# showmount -e $NFS_SERVER
Export list for $NFS_SERVER:
/ (everyone)
NFSv4 uses a 'pseudo root' as mentioned in the configuration file. This means that mounting the export over NFSv4 results in a virtual directory structure:
# mount -t nfs $NFS_SERVER:/ /mnt
# find /mnt
/mnt
/mnt/nfsv4
/mnt/nfsv4/pseudofs
/mnt/nfsv4/pseudofs/ceph
/mnt/nfsv4/pseudofs/ceph/README
Reading and writing to the mountpoint under /mnt/nfsv4/pseudofs/ceph works fine, as long as the usual permissions allow that. By default NFS-Ganesha enabled 'root squashing', so the 'root' user may not do a lot on the export. Disabling this security measure can be done by placing this option in the export section:
Squash = no_root_squash;
Restart the nfs-ganesha.service after modifying /etc/ganesha.nfsd.conf and writing files as 'root' should work too now.
Future Work
For me, this was a short "let's try it out" while learning about Ceph. At the moment, I have no intention on working on the FSAL_CEPH for NFS-Ganesha. My main interest of this experiment with exporting a Ceph filesystem though NFS-Ganesha on a plain Fedora 20 installation, was to learn about usability of a new NFS-Ganesha configuration/deployment. In order to improve the user experience with NFS-Ganesha, I'll try and fix some of the issues I run into. Progress can be followed in Bug 1144799.
In future, I will mainly use NFS-Ganesha for accessing Gluster Volumes. My colleague Soumya posted a nice explanation on how to download, build and run NFS-Ganesha with support for Gluster. We will be working on improving the out-of-the-box support in Fedora while stabilizing the FSAL_GLUSTER in the upstream NFS-Ganeasha project.