A while ago I put together a post detailing the installation and configuration of 2 hosts running glusterfs, which was then presented as CIFS based storage. http://jonarcher.info/2014/06/windows-cifs-fileshares-using-glusterfs-ctdb-highly-available-data/ This post gained a bit of interest through the comments and social networks, one of the comments I got was from John Mark Walker suggesting I look at the …read more
The post Gluster, CIFS, ZFS – kind of part 2 appeared first on Jon Archer.
NFS-Ganesha is a user-space NFS-server that is available in Fedora. It contains several plugins (FSAL, File System Abstraction Layer) for supporting different storage backends. Some of the more interesting are:
Exporting a mounted filesystem is pretty simple. Unfortunately this failed for me when running with the standard nfs-ganesha packages on a minimal Fedora 20 installation. The following changes were needed to make NFS-Ganesha work for a basic export:
When these initial things have been taken care of, a configuration file needs to be created. The default configuration file mentioned in the environment file is /etc/ganesha.nfsd.conf. The sources of nfs-ganesha contain some examples, the vfs.confis quite usable as a starting point. After copying the example and modifying the paths to something more suitable, starting the NFS-server should work:
# systemctl start nfs-ganesha
In case something failed, there should be a note about it in /var/log/ganesha.log.
This assumes you have a working Ceph Cluster which includes several MON, OSD and one or more MDS daemons. The FSAL_CEPH from NFS-Ganesha uses libcephfs which seems to be the same as the ceph-fuse package for Fedora. the easiest way to make sure that the Ceph filesystem is functional, is to try and mount it with ceph-fuse.
The minimal requirements to get a Ceph client system to access the Ceph Cluster, seem to be a /etc/ceph/ceph.conf with a [global]section and a suitable keyring. Creating the ceph.conf on the Fedora system that was done the ceph-deploy:
$ ceph-deploy config push $NFS_SERVER
In my setup I scp‘d the /etc/ceph/ceph.client.admin.keyring from one of my Ceph servers to the $NFS_SERVER. There probably are better ways to create/distribute a keyring, but I’m new to Ceph and this worked sufficiently for my testing.
When the above configuration was done, it was possible to mount the Ceph filesystem on the Ceph client that is becoming the NFS-server. This command worked without issues:
# ceph-fuse /mnt
# echo 'Hello Ceph!' > /mnt/README
# umount /mnt
The first write to the Ceph filesystem took a while. This is likely due to the initial work the MDS and OSD daemons need to do (like creating pools for the Ceph filesystem).
After confirming that the Ceph Cluster and Filesystem work, the configuration for NFS-Ganesha can just be taken from the sources and saved as /etc/ganesha.nfsd.conf. With this configuration, and restarting the nfs-ganesha.service, the NFS-export becomes available:
# showmount -e $NFS_SERVER
Export list for $NFS_SERVER:
/ (everyone)
NFSv4 uses a ‘pseudo root’ as mentioned in the configuration file. This means that mounting the export over NFSv4 results in a virtual directory structure:
# mount -t nfs $NFS_SERVER:/ /mnt
# find /mnt
/mnt
/mnt/nfsv4
/mnt/nfsv4/pseudofs
/mnt/nfsv4/pseudofs/ceph
/mnt/nfsv4/pseudofs/ceph/README
Reading and writing to the mountpoint under /mnt/nfsv4/pseudofs/ceph works fine, as long as the usual permissions allow that. By default NFS-Ganesha enabled ‘root squashing’, so the ‘root’ user may not do a lot on the export. Disabling this security measure can be done by placing this option in the export section:
Squash = no_root_squash;
Restart the nfs-ganesha.service after modifying /etc/ganesha.nfsd.conf and writing files as ‘root’ should work too now.
For me, this was a short “let’s try it out” while learning about Ceph. At the moment, I have no intention on working on the FSAL_CEPH for NFS-Ganesha. My main interest of this experiment with exporting a Ceph filesystem though NFS-Ganesha on a plain Fedora 20 installation, was to learn about usability of a new NFS-Ganesha configuration/deployment. In order to improve the user experience with NFS-Ganesha, I’ll try and fix some of the issues I run into. Progress can be followed in Bug 1144799.
In future, I will mainly use NFS-Ganesha for accessing Gluster Volumes. My colleague Soumya posted a nice explanation on how to download, build and run NFS-Ganesha with support for Gluster. We will be working on improving the out-of-the-box support in Fedora while stabilizing the FSAL_GLUSTER in the upstream NFS-Ganeasha project.
RPMs for 3.4.6beta1 are available for testing at http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.6beta1/ Above repo contains RPMs for CentOS 5, 6, 7 and Fedora 19, 20, 21, 22. Packages for other platforms/distributions will be available once they are build. If you find any issue … Continue reading →
If you’re a reader of my code or of this blog, it’s no secret that I hack on a lot of puppet and vagrant. Recently I’ve fooled around with a bit of docker, too. I realized that the vagrant, environments … Continue reading →
Over the past few years, there was an enormous increase in the number of user-space filesystems being developed and deployed. But one of the common challenges which all those filesystems’ users had to face was that there was a huge performance hit when their filesystems were exported via kernel-NFS (well-known and widely used network protocol).To […]
Seagate has just publicly announced 8TB HDD’s in a 3.5″ form factor. I decided to do some rough calculations to understand the density a bit better… Note: I have decided to ignore the distinction between Terabytes (TB) and Tebibytes (TiB), since … Continue reading →
I decided to try the upgrade process from EL 6 to 7 on the servers I used in my previous blog post “Windows (CIFS) fileshares using GlusterFS and CTDB for Highly available data” Following the instructions here I found the process fairly painless. However there were 1 or two little niggles which caused various issues which …read more
The post Upgrade CentOS 6 to 7 with Upgrade Tools appeared first on Jon Archer.
Scenario: You are operating a busy GlusterFS cluster and for whatever reason the volume data gets corrupted. Luckily, you have been backing up the underlying bricks so you are able to restore the bricks to a usable state, but now…
Ovirt is an open source tool used to create/manage gluster nodes through an easy to use web interface. This document is to cover how gluster can be used with ovirt. Want to manage gluster nodes with ease using ovirt ? Create your own ovirt by following these simple steps. Machine Requirements : Fedora19 with 4GB […]
A small blog on how to put Ovirt inside a docker. Install docker on your system. Get an account in docker. pull a base image from docker which ovirt supports. For example : Fedora and centos. Let us install ovirt on centos, by pulling centos base image from docker. Instructions to follow: docker run -i -t centos […]
You want to learn scala. And you want to learn spark. And you’ve heard of SBT. Where do you start?There are alot of different idioms for developing spark apps. One possibility is to use Ipython and Pyspark, which I’ve writ…
GlusterFS 3.5.2beta1 has just been released. This is the first beta to allow users to verify the fixes for the bugs that were reported. See the bug reports below for more details on how to test and confirm the fix (or not). This is a bugfix only releas…
When I’m enjoying the sun/wind/rain on the balcony, I tend to use my XO-1.75 for duties where most people would use a tablet. Reading/writing emails, browsing the internet, bug triaging or writing small fixes, release notes and all can be done fine on …
How to install GlusterFS with a replicated volume over 2 nodes on Ubuntu 14.04
In this tutorial I will explain GlusterFS configuration in Ubuntu 14.04. GlusterFS is an open source distributed file system which provides easy replication over multip…
A few quick notes on adding new packages to RHEL 7.Nice stuff about RHEL 7Docker is now a first class citizen. You can read about how RHEL is moving to support Docker here.Exciting stuff in the EPELsThere’s docker, gluster, and loads of other goo…
One time someone asked me why I liked build tools so much. Here is why.The answer is : because nobody else does. Its like the same reason why my wife is passionate about what kind of crib the kids get. Its because I’m not. You see th…
Simple recipe for spinning up VMs on Libvirt from zerosu -c “yum install @virtualization”This sets up all the virtualization entries. I never remember to use it, but more about why you should use it here.Create disk for a VM and attach itqemu-img…
James (who happens to be a coworker of mine now) recently posted some vagrant on libvirt tutorials. Ironically, I was (I think) one of the original dudes who prodded him to post about vagrant (specifically to demonstrate his puppet-gluster …
This tutorial will walk through the setup and configuration of GlusterFS and CTDB to provide highly available file storage via CIFS. GlusterFS is used to replicate data between multiple servers. CTDB provides highly available CIFS/Samba functionality. Prerequisites: 2 servers (virtual or physical) with RHEL 6 or derivative (CentOS, Scientific Linux). When installing create a partition …read more
The post Windows (CIFS) fileshares using GlusterFS and CTDB for Highly available data appeared first on Jon Archer.
Recently another Docker image for Gluster was released. More people getting Gluster images into Docker is fantastic news, but because Docker is essentially brand spanking new, few of us are experts. With that in mind, explaining WHY this is great news might be helpful.
In the simplest terms, Docker (also referred to as containers), is a way to deploy machine images. Big deal, we already have kickstart and virtualization, who needs another way to deploy? Anyone who obsesses about near-instant provisioning with absolutely minimal resource consumption, that’s who. In other words, pretty much all of us. An example from my test machine shows that launching the container takes a whopping two seconds:
# docker ps
CONTAINER ID IMAGE COMMAND CREATED \
77b24fbf053a humble/fed20-gluster:latest /usr/bin/supervisord 18 seconds ago \
STATUS PORTS NAMES
Up 16 seconds 22/tcp fedora20-gluster
Another benefit is the footprint. The Docker images I have seen typically are less than 200MB to download. The image here is larger because it includes a fair amount of space to play with Gluster. Unless you have specific needs, you don’t have to build them from scratch as there is a global registry of every major flavor of Linux out there, and many with specific applications baked right in (as is our case with the Gluster image). You can search for other images at the Docker Hub Registry, or from the command line: docker search <string>
.
Maybe you already have rock-solid deploy scripts to your favorite cloud provider, where provisioning times are typically quite fast. That is great, but also comes with a price. Deploying with Docker is free, fast, and because, it uses as few resources as possible—and even those can be tuned, although that is outside the scope of this article—hearing of people deploying hundreds of images running simultaneously to a single machine is not uncommon. You might be able to get a few hundred VMs running on a single box, but typically the performance of the machines is minimal, and the cost of the hardware can be hard to swallow. So Docker, in a nutshell, gives us a great new way to test Gluster.
Before getting started, you must first do a few things. Start by installing the docker-io package (e.g., on my Fedora 20 machine, yum install docker-io
). There also is an older package called docker
, which apparently was for system tray replacement for KDE or Gnome. Also having that package is okay, I supposed, but it won’t help for our needs here.
After installing Docker, start the service:
# systemctl start docker.service
Next, follow the steps outlined in Part 1 of our Run Gluster Containers Using Docker instructions on the Gluster blog.
After you have deployed, ssh to the container IP address. (If you are using the humble/fed20-gluster image, the password is redhat).
To create a test volume from within the container, follow the usual steps:
mkdir -p /export/gluster
gluster volume create gv0 172.17.0.2:/gluster/export
(changing the IP to whatever is appropriate)gluster volume create gv0 172.17.0.2:/export/gluster force
(we use the force command here because we are doing something unsupported, which is fine for our testbed)gluster volume start gv0
Simplicity at its finest. After you have done the initial download, deploying another instance is as simple as:
#docker run --privileged -d --name <new instance name> -i -t 764261ddfd16
Getting another volume set up in the new instance took me less than three minutes, running all the commands manually. That’s very useful, for example, when you want to test similar volumes with different settings. Currently the only drawback is that the image by itself is limited to one node. Again, this is perfect for basic test scenarios.
The dockit project is working on a solution that allows multiple containers to communicate with each other, which is something I am eager to test. But for now, we can do basic work to “fake” creating distributed or distributed-replicated volumes.
First, create several folders within the /exports directory:
# mkdir -p /export/{1..6}
# ls /export/
1 2 3 4 5 6
Although these are not separate filesystems, we will use a special tool called “the power of imagination!” to pretend that they are. As we progress, you will see that functionally the directories will work the same.
To create a pure distributed volume:
# gluster volume create gv-dht 172.17.0.2:/export/{1..2}
And to add another volume with distributed-replicated capabilities:
# gluster volume create gv-afr 172.17.0.2:/export/{3..6}
# gluster volume info
Volume Name: gv0
Type: Distribute
Volume ID: 097eb563-e5db-4b38-8531-bc27a8e2de3d
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 172.17.0.2:/export/gluster
Volume Name: gv-dht
Type: Distribute
Volume ID: 0abe6387-bfe5-4a10-9226-3ce8e7e5f051
Status: Created
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 172.17.0.2:/export/1
Brick2: 172.17.0.2:/export/2
Volume Name: gv-afr
Type: Distribute
Volume ID: cdabdef6-bb57-4c7d-ae87-0db4ccb3fc13
Status: Created
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 172.17.0.2:/export/3
Brick2: 172.17.0.2:/export/4
Brick3: 172.17.0.2:/export/5
Brick4: 172.17.0.2:/export/6
Here we see all three volumes that we have created so far. Mount them the same as always:
# mount -t glusterfs 172.17.0.2:/gv-afr /mnt/gluster
If you prefer, make three mount points, one for each volume.
Keep in mind that the total amount of space in the instance is around 8GB, which should be plenty for simple tests; however, that 8GB is shared between all the volumes. Remember when we used the force
option to do something we weren’t supposed to? And that’s it — happy testing!
This article originally appeared on
community.redhat.com.
Follow the community on Twitter at
@redhatopen, and find us on
Facebook and
Google+.