On Tue, Jun 24, 2014 at 03:15:58AM -0700, Gluster Build System wrote:> > > SRC: http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.5.1.tar.gz> > This release is made off jenkins-release-73Many thanks to everyone how tested the…
Earlier this year, R.I.Pienaar released his brilliant data in modules hack, a few months ago, I got the chance to start implementing it in Puppet-Gluster, and today I have found the time to blog about it. What is it? R.I.’s … Continue reading →
Reposting the email to the Gluster Users and Developers mailinglists.On Sat, 24 May, 2014 at 11:34:36PM -0700, Gluster Build System wrote:> > SRC: http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.5.1beta.tar.gzThis beta release is i…
oVirt’s Hosted Engine feature, introduced in the project’s 3.4 release, enables the open source virtualization system to host its own management server, which means one fewer required machine, and more self-sufficiency for your oVirt installation.
While a self-sufficient oVirt installation has been achievable for some time using the project’s “All-in-One” method of running an oVirt virtualization host and management server together on one machine, the Hosted Engine feature allows multiple machines to partake in the hosting duties, eliminating any one host as a single point of failure.
The Hosted Engine feature relies on NFS storage to house the management VM. Running an NFS server on one of our virtualization hosts would make that host a new single point of failure, which means we need either to tap an external NFS filer (the approach I took in the walkthrough I posted here recently) or we need to figure out how to make our oVirt hosts serve up their own, replicated NFS storage.
In this post, I’m going to walk through that latter option – setting up a pair of CentOS 6 machines to serve as oVirt virtualization hosts that together provide the NFS storage required for the Hosted Engine feature, using Gluster for this replicated storage and for NFS and CTDB to provide a virtual IP address mount point for the storage.
NOTE: People have been running into some issues with converged Gluster + oVirt setups like the one I describe here.
It’s important to use CTDB, or something like it, to provide for automated IP failover. While it may seem reasonable to simply use “localhost” as the NFS mount point for the hosted engine storage, and rely on Gluster to handle the replication between the servers, this ends up not working reliably.
In my own lab, I’m running a setup like the one below, but with three machines, each serving as virt+storage hosts, with replica 3 Gluster volumes to ensure that a 51% quorum is maintained when one of the machines is down for maintenance.
My planned outages, where (as described below) I first stop the ctdb service on the to-be-shutdown machine, thereby prompting another node to pick up the job, have run smoothly.
I recently tested with an unplanned outage, where I pulled the plug (stopped via power management) on the machine hosting my Gluster NFS storage. Here the handoff left much to be desired – it took 18 minutes for oVirt engine to become fully available again. However, it took the only other VM I had running at the time (which wasn’t on the downed machine) never went offline, as is the norm for oVirt engine outages.
The prerequisites are the same as for the Up and Running with oVirt 3.4 walkthrough, with the addition of a healthy-sized disk or partition to use for our Gluster volumes. The hosted engine VM will require 20GB, and you’ll want to have plenty of storage space left over for the VMs you’ll create and manage with oVirt.
For networking, you can get away with a single network adapter, but for best results, you’ll want three: one for the CTDB heartbeat, one for Gluster traffic, and one for oVirt management traffic and everything else. No matter how you arrange your networking, your two hosts will need to be able to reach other on your network(s). If need be, edit /etc/hosts
on both of your machines to establish the right ip address / host name mappings.
NOTE: Unless I indicate otherwise, you’ll need to perform the steps that follow on both of your machines.
We need a partition to store our Gluster bricks. For simplicity, I’m using a single XFS partition, and my Gluster bricks will be directories within this partition.
You’ll need to carry out these steps on both of your oVirt/Gluster machines, as each will be sharing in the storage duties.
I’m using CentOS 6 hosts for this walkthrough, which is missing one of my favorite utilities from Fedora, system-storage-manager. I packaged up ssm for CentOS using Fedora’s nifty Copr service:
cd /etc/yum.repos.d/ && curl -O http://copr.fedoraproject.org/coprs/jasonbrooks/system-storage-manager/repo/epel-6-x86_64/jasonbrooks-system-storage-manager-epel-6-x86_64.repo && yum install -y http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm && yum install -y system-storage-manager python-argparse
I use ssm to add a new disk to my LVM pool and to create an XFS partition for Gluster (you can do it however you want):
ssm list (take note of your pool name)
ssm add -p $YOUR_POOL_NAME $YOUR_DEVICE_NAME
ssm create -p $YOUR_POOL_NAME --fstype xfs -n gluster
Next, modify your /etc/fstab
to add the new partition:
mkdir /gluster
blkid $YOUR_NEW_VOLUME_NAME (something like /dev/pool/gluster)
Edit your /etc/fstab
and add the line:
UUID=$YOUR_UUID /gluster xfs defaults 1 2
mount -a
Edit /etc/sysconfig/iptables
to include the rules you’ll need for Gluster, oVirt and CTDB:
# oVirt/Gluster firewall configuration
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
# vdsm
-A INPUT -p tcp --dport 54321 -j ACCEPT
# SSH
-A INPUT -p tcp --dport 22 -j ACCEPT
# snmp
-A INPUT -p udp --dport 161 -j ACCEPT
# libvirt tls
-A INPUT -p tcp --dport 16514 -j ACCEPT
# guest consoles
-A INPUT -p tcp -m multiport --dports 5900:6923 -j ACCEPT
# migration
-A INPUT -p tcp -m multiport --dports 49152:49216 -j ACCEPT
# glusterd
-A INPUT -p tcp -m tcp --dport 24007 -j ACCEPT
# portmapper
-A INPUT -p udp -m udp --dport 111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38465 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38466 -j ACCEPT
# nfs
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 38467 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
# status
-A INPUT -p tcp -m tcp --dport 39543 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 55863 -j ACCEPT
# nlockmgr
-A INPUT -p tcp -m tcp --dport 38468 -j ACCEPT
-A INPUT -p udp -m udp --dport 963 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 965 -j ACCEPT
# ctdbd
-A INPUT -p tcp -m tcp --dport 4379 -j ACCEPT
# Ports for gluster volume bricks (default 100 ports)
-A INPUT -p tcp -m tcp --dport 24009:24108 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 50152:50251 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 34865:34867 -j ACCEPT
# Reject any other input traffic
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m physdev ! --physdev-is-bridged -j REJECT --reject-with icmp-host-prohibited
COMMIT
NOTE: During the oVirt Hosted Engine install process, which we’ll get to shortly, the installer script will ask if you want it to configure your iptables. You should answer no, but if you answer yes on accident, the installer will saved a backup copy of the previous configuration as something like /etc/sysconfig/iptables.$DATE
and you can just copy that back over. Keep an eye on this, because your pair of machines will have to communicate with each other for Gluster, for CTDB, for NFS, etc.
Restart your iptables service:
service iptables restart
yum localinstall -y http://resources.ovirt.org/releases/ovirt-release.noarch.rpm && yum install -y glusterfs-server
Edit /etc/glusterfs/glusterd.vol
, uncomment the line option base-port 49152
and change the value 49152
to 50152
. This change works around a conflict between the ports used by libvirt for live migration, and the ports Gluster uses for its bricks.
Now start the Gluster service and configure it to auto-start after subsequent reboots:
service glusterd start && chkconfig glusterd on
Now, we’ll probe our second machine from our first, combining them into a single Gluster trusted pool. Unlike many of the other commands in this walkthrough, it’s only necessary to run this on one of your two machines. If you’re using a separate network for Gluster traffic, you must use your machine’s address on that network for this command:
gluster peer probe $YOUR_OTHER_MACHINE
First, we’ll create a “meta” volume for the clusted file system that CTDB requires for its own needs:
gluster volume create meta replica 2 $YOUR_FIRST_HOST:/gluster/meta0 $YOUR_SECOND_HOST:/gluster/meta1
Then, we’ll start that volume:
gluster volume start meta
chkconfig rpcbind on
service rpcbind start
yum install -y nfs-utils ctdb
Set up conf files for ctdb:
mkdir -p /mnt/lock
mount -t glusterfs localhost:/meta /mnt/lock
Edit /mnt/lock/ctdb
:
CTDB_PUBLIC_ADDRESSES=/mnt/lock/public_addresses
CTDB_NODES=/etc/ctdb/nodes
# Only when using Samba. Unnecessary for NFS.
CTDB_MANAGES_SAMBA=no
# some tunables
CTDB_SET_DeterministicIPs=1
CTDB_SET_RecoveryBanPeriod=120
CTDB_SET_KeepaliveInterval=5
CTDB_SET_KeepaliveLimit=5
CTDB_SET_MonitorInterval=15
CTDB_RECOVERY_LOCK=/mnt/lock/reclock
Edit /mnt/lock/nodes
to include the list of CTDB interconnect/heartbeat IPs. For our two-node install there’ll be two of these. For more info on CTDB configuration, see Configuring CTDB.
Next, edit /mnt/lock/public_addresses
to include the list of virtual addresses to be hosted between the two machines (we only need one), and the network range, and the nic we’re using to host this virtual address:
XX.XX.XX.XX/24 eth0
chkconfig rpcbind on
service rpcbind start
yum install -y nfs-utils ctdb
mkdir -p /mnt/lock
mount -t glusterfs localhost:/meta /mnt/lock
Now, on both hosts, we’ll point our CTDB configuration files at the files we’ve created in the shared meta
volume:
mv /etc/sysconfig/ctdb /etc/sysconfig/ctdb.orig
ln -s /mnt/lock/ctdb /etc/sysconfig/ctdb
ln -s /mnt/lock/nodes /etc/ctdb/nodes
ln -s /mnt/lock/public_addresses /etc/ctdb/public_addresses
mount -t glusterfs localhost:/meta /mnt/lock && service ctdb start
gluster volume create engine replica 2 $YOUR_FIRST_MACHINE:/gluster/engine0 $YOUR_OTHER_MACHINE:/gluster/engine1
gluster volume set engine storage.owner-uid 36 && gluster volume set engine storage.owner-gid 36
gluster volume start engine
Create a file named `/var/lib/glusterd/groups/virt’ and paste in the lines below. This provides a “virt” group with settings optimized for VM storage. I’ve left off two quorum-related options present in the original group definition. These quorum settings help prevent split-brain, but will cause VMs hosted on Gluster volumes with the settings applied to pause when one of our two machines goes offline.
quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
Next, we’ll add our new engine volume to this virt group:
gluster volume set engine group virt
While we’re at it, let’s create a third Gluster volume, for our regular VM storage in oVirt:
gluster volume create data replica 2 $YOUR_FIRST_MACHINE:/gluster/data0 $YOUR_OTHER_MACHINE:/gluster/data1
gluster volume set data storage.owner-uid 36 && gluster volume set data storage.owner-gid 36
gluster volume set storage group virt
gluster volume start data
With our Gluster-provided NFS storage for the oVirt engine VM arranged, we can proceed with the Hosted Engine installation. See the “Installing oVirt with Hosted Engine” heading in the Up & Running walkthrough and follow the steps there.
As you do, keep these things in mind:
hosted-engine --deploy
command on your first machine, run ‘service ctdb stop’ to ensure that the shared virtual IP address is hosted from your other machine. This will ensure that the ovirtmgmt
network bridge will be created correctly. When it comes time to install your second host (under the heading “Installing a Second Host”), run service ctdb start
on your first host and then service ctdb stop
on your second host, and then start CTDB back up on the second host when the install is done./mnt/lock/public_addresses
) or a host name that resolves to this address. Using this virtual IP ensures that when one machine goes down, the other will automatically pick up the hosting duties for the NFS share.$YOUR_VIRTUAL_IP:data
to use the “data” Gluster volume we configured above for your VM storage.To take down one of the machines, while enabling VM workloads and the management server to continue running on the other machine, follow these steps:
service ctdb stop
to ensure that the second node takes on the virtual IP / NFS hosting chores.When it’s time to boot your maintained host back up, follow these steps on that machine:
mount -t glusterfs localhost:/meta /mnt/lock
service ctdb start
Commands that will come in handy:
service ctdb status
gluster volume heal $YOURVOL statistics
tail -f /var/log/ovirt-hosted-engine-ha/*
I hope that this walkthrough helps you get up and running with Gluster-based storage on your oVirt installation. The Hosted Engine feature is relatively new, and I’m still fully wrapping my brain around it. The biggest issue to watch out for with this configuration is getting your Gluster volumes into a split-brain state. On IRC, I’m jbrooks, ping me in the #ovirt room on OFTC, or the #gluster room on Freenode, or write a comment below.
If you’re interested in getting involved with oVirt or Gluster, you can find all the mailing list, issue tracker, source repository, and wiki information you need on the oVirt or Gluster project sites.
Finally, be sure to follow us on Twitter at @redhatopen for news on oVirt and other open source projects in the Red Hat world.
This article originally appeared on
community.redhat.com.
Follow the community on Twitter at
@redhatopen, and find us on
Facebook and
Google+.
Vagrant has become the de facto tool for devops. Faster iterations, clean environments, and less overhead. This isn’t an article about why you should use Vagrant. This is an article about how to get up and running with Vagrant on … Continue reading →
This is a quick trick for making working with git submodules more magic. One day you might find that using git submodules is needed for your project. It’s probably not necessary for everyday hacking, but if you’re glue-ing things together, … Continue reading →
We are pleased to announce that GlusterFS 3.5 is now available. The latest release includes several long-awaited features such as improved logging, file snapshotting, on-wire compression, and at-rest encryption.
You can download GlusterFS 3.5 now.
#…
Here are the topics this blog is going to cover. Samba Server Samba VFS Libgfapi GlusterFS VFS plugin for Samba and libgfapi Without GlusterFS VFS plugin FUSE mount vs VFS plugin About Samba Server: Samba server runs on Unix and … Continue reading →
Deploying an open source enterprise cloud just got a little bit easier yesterday with the release of the newest version of the OpenStack platform: Icehouse. To quote an email from OpenStack rele…
Setting up an application server in the cloud isn’t that hard if you’re familiar with the tools and your application’s requirements. But what if you needed to do it dozens or hundreds of times, …
GlusterFS 3.5 has not been released yet, but that should happen hopefully anytime soon (currently in beta). The RPM-packaging in this version has changed a little, and now offers a glusterfs-cli
package. This package mainly contains the gluster
commandline interface (and pulls in any dependencies).
On of the very useful things that is now made possible, is to list the available volumes on Gluster Storge Servers. This similar functionality is used by the /etc/auto.net
script to list NFS-exports that are available for mounting. The auto.net
script is by default enabled after installing and starting autofs
:
# yum install autofs
# systemctl enable autofs.service
# systemctl start autofs.service
Checking, and mounting NFS-exports is made as easy as:
$ ls /net/nfs-server.example.net
archive media mock_cache olpc
$ ls /net/nfs-server.example.net/mock_cache/fedora-rawhide-armhfp/
yum_cache
Making this functionality available for Gluster Volumes is simple, just follow these steps:
install the gluster
command
# yum install glusterfs-cli
save the file below as /etc/auto.glfs
#!/bin/bash
# /etc/auto.glfs -- based on /etc/auto.net
#
# This file must be executable to work! chmod 755!
#
# Look at what a host is exporting to determine what we can mount.
# This is very simple, but it appears to work surprisingly well
#
key="$1"
# add "nosymlink" here if you want to suppress symlinking local filesystems
# add "nonstrict" to make it OK for some filesystems to not mount
opts="-fstype=glusterfs,nodev,nosuid"
for P in /usr/local/bin /usr/local/sbin /usr/bin /usr/sbin /bin /sbin
do
if [ -x ${P}/gluster ]
then
GLUSTER_CLI=${P}/gluster
break
fi
done
[ -x ${GLUSTER_CLI} ] || exit 1
${GLUSTER_CLI} --remote-host="${key}" volume list | \
awk -v key="$key" -v opts="$opts" -- '
BEGIN { ORS=""; first=1 }
{ if (first) { print opts; first=0 }; print " \\\n\t/" $1, key ":/" $1 }
END { if (!first) print "\n"; else exit 1 }' | \
sed 's/#/\\#/g'
make the script executable
# chmod 0755 /etc/auto.glfs
add an automount point to the autofs configuration
# echo /glfs /etc/auto.glfs > /etc/auto.master.d/glfs.autofs
reload the autofs configuration
# systemctl reload autofs.service
After this, autofs
should have created a new /glfs
directory. The directory itself is empty, but a ls /glfs/gluster.example.net
will show all the available volumes on the gluster.example.net server. These volumes can now be accessed through the autofs mountpoint. When the volumes are not used anymore, autofs will automatically unmount them after a timeout.
My petstore is huge !Thanks everyone for your help so far on BigPetStore! We’ve made alot of progress this last couple of months. The open stack sahara community has adopted it for there RHS demo: Thanks matt for your hard work getting it u…
What is libgfapi ? User-space library for accessing data in GlusterFS Filesystem-like API Runs in application process no FUSE, no copies, no context switches …but same volfiles, translators, etc. Could be used for Apache/nginx modules, MPI I/O (maybe), Ganesha, etc….
How an open source project is governed can matter just as much as the features it supports, the speed at which it runs, or the code that underlies it. Some open source projects have what we migh…
When reading a recent article by Red Hat CEO Jim Whitehurst, I was struck by a comparison made between OpenStack and the interstate highway system. The article in Wall Street and Technology…
I’ve been afraid of RPM and package maintaining [1] for years, but thanks to Kaleb Keithley, I have finally made some RPM’s that weren’t generated from a high level tool. Now that I have the boilerplate done, it’s a relatively … Continue reading →
Puppet is missing a number of much-needed features. That’s the bad news. The good news is that I’ve been able to write some of these as modules that don’t need to change the Puppet core! This is an article about … Continue reading →
Calling all storage, cloud, and big data developers and devops practitioners: The Gluster Community invites you to attend the Data Liberate open source storage hackathon and mixer on Sunday, April 13, in San Francisco.
Highlights of the event include:
#Agenda
Are you a beginner? Cool! Experts will be on hand leading sessions throughout the day to help you get started.
This article originally appeared on
community.redhat.com.
Follow the community on Twitter at
@redhatopen, and find us on
Facebook and
Google+.
Last week, the New York Times Magazine posted a fascinating article about the dichotomy of technology found in IT today: the push and pull between the young, entrepreneurial hackers who are looking for the next big thing and the older, more experienced engineers who have seen it all.
This article was interesting to me for several reasons, not the least of which was the fact that I am approaching the half-century mark in the next few years and I work with a lot of younger folks who don’t quite see the work we do in the same way. I don’t see that as a particularly bad situation; seeing and learning different perspectives is one of the things I like to do.
Another big reason for the interest in this article was the conclusion… that in all of the rush to get the latest and greatest Next Big Thing out there, a lot of shortcuts are being taken and technology quality suffers as a result.
“But the churn feels more problematic now, in part because it deprives the new guard as well as the old — and by extension, it deprives us all. In pursuing the latest and the coolest, young engineers ignore opportunities in less-sexy areas of tech such as semiconductors, data storage, and networking, the products that form the foundation on which all of Web 2.0 rests. Without a good router to provide reliable Wi-Fi, your Dropbox file-sharing application is not going to sync; without Nvidia’s graphics processing unit, your BuzzFeed GIF is not going to make anyone laugh. The talent — and there’s a ton of it— flowing into Silicon Valley cares little about improving these infrastructural elements. What they care about is coming up with more web apps.”
Actually this is not a new phenomenon: People are always more interested in building or using the latest shiny automobile than fundamentally improving the roads on which that car has to drive. But as anyone who suffered through this Winter in North America knows, making sure the roads are in good shape is not something that should be a passing concern.
Infrastructure, by definition, is not sexy. If I build a better router or submit a nice patch to the Linux kernel, my efforts will be appreciated, but no one is likely to acquire my company for billions of dollars. But if I build a new app with vague social media potential and no revenue to speak of, well, then ka-ching!
But flash and cash doesn’t drive every one. Red Hat, for instance, has done a pretty good job for itself working on and supporting Linux, which is, at the end of the day, infrastructure. Moreover, the company has fostered the development of a host of other infrastructure technologies, such as [OpenStack] (http://openstack.redhat.com/Main_Page), [Gluster] (http://www.gluster.org/), and [oVirt] (http://www.ovirt.org/Home), all by working with those respective communities in an open source way.
Working on infrastructure isn’t going to turn a lot of heads, but it’s the groundwork on which all of those entrepreneurs (and small businesses and enterprises and governments…) can get their ideas created and their work done. It’s decent work, our parents’ generation might say; something to be proud of.
Innovation still happens: Networks can get faster, roads smoother, pipes stronger, operating systems better. The team at my own project, oVirt, has done some nice things with virtual datacenter management that will knock the socks off of anyone looking for managing their own virtual machines, something that will be noticed even more when release 3.4 comes out soon. Being in infrastructure doesn’t mean throwing innovation out the window.
Does oVirt get the same attention as the flashy start up in Silicon Valley? Nope, but that’s not the point. Sometimes the point is to get things done and make it better for those who do.
This article originally appeared on
community.redhat.com.
Follow the community on Twitter at
@redhatopen, and find us on
Facebook and
Google+.
this is howw u doooo it …….The maven release:prepare plugin can be very powerful if you follow maven guidelines, but some projects choose not to follow maven guidelines.I’m not sure why anyone in the universe would EVER use maven without following …