I wanted to take a moment and share all the things that are going on in the Gluster Community. It really has been an amazing year, and we’re only halfway through. Here’s a recap for those of you watching from home: Launched the Gluster Community Forge in early May – http://forge.gluster.org/ as of now, there …Read more

Big Top normalizes your hadoop distributions by unified building, packaging, and integration testing of the latest distributions of tools in the hadoop ecosystem.  That way you know that your various ecosystem components play nice together.  …

Apache BigTop smoke tests can confirm all your ecosystem components… Butyou never really want to test EVERYTHING.0) When you start , make sure you have logging configured to TRACE for easy debugging: Logging of bigtop tests is largely done by th…

Adding multiple disks in the image above (IDE Disk 2,3,4) with virt-install (above — the view of external disks from VMM)I’ve recently been creating VM’s from OTHER PEOPLES VMs.  This means that I’ve had to play around with external storage.&nbs…

I have generic users that are logged in automatically since they’re not allowed user prefs anyway, and it would be more of a hindrance in their environment.
I use puppet to manage a staging directory that has the home directory as I want it to be, and …

Dev Code that uses that hadoop needs to be customized.  If you have control over your dev tests, you can simply useConfiguration.addDefaultResource(“/path/to/core-site.xml”);This will load the conf file after loading other default resources.In gen…

The people at JetBrains have given the GlusterFS project an “unlimited # of users” license for their cross-platform PyCharm Python IDE: http://www.jetbrains.com/pycharm/download/ Everyone has their own preferences with programming IDE’s, and I’ve used several over the years, generally based on Eclipse.  However, I happen to like PyCharm at the moment, so applied for JetBrain’s “free …Read more

One of the best things about OpenStack development is the rich testing and review frameworks around it.  The most important of which are tempest and devstack.  In this post I will discuss how I recently squashed a particularly painful bug (read: I make dumb mistakes) in my latest patch to Glance. The Bug My patch […]

I went into this expecting problems. Fedora 19 ships with GlusterFS 3.4.0 beta and I’m using GlusterFS 3.3 in production. I expected that I would have to downgrade my Fedora packages so I could use my volumes. I expected problems.
What I didn’t expect …

{% img right http://www.hastexo.com/system/files/imagecache/sidebar/20120221105324808-f2df3ea3e3aeab8_250_0.png %} As I promised on Twitter, this is how I automate a GlusterFS deployment. I’m making a few assumptions here:

• I am using CentOS 6, so should work on RHEL 6 and Scientific Linux 6 too. Others may work, but YMMV.
  • As I use XFS, RHEL users will need the Scalable Storage option. Ext4 will work, but XFS is recommended.
• That you have a way of automating your base OS installation. My personal preference is to use Razor.
• You have a system with at least a complete spare disk dedicated to a GlusterFS brick. That is the best way to run GlusterFS anyway.
• You have 2 nodes and want to replicate the data
• You have a simple setup with only a single network, because I am being lazy. As a proof-of concept this is fine. Modifying this for second network is quite easy, just change the IP address in you use.

{% img https://docs.google.com/drawings/d/1XA7GH3a4BL1uszFXrSsZjysi59Iinh-0RmhqdDbt7QQ/pub?w=673&h=315 ‘simple gluster architecture’ %}

The diagram above shows the basic layout of what to start from in terms of hardware. In terms of software, you just need a basic CentOS 6 install and to have Puppet working.

I use a pair of Puppet modules (both in the Forge): thias/glusterfs and puppetlabs/lvm. The GlusterFS module CAN do the LVM config, but that strikes me as not the best idea. The UNIX philosophy of “do one job well” holds up for Puppet modules as well. You will also need my yumrepos module.

Clone those 3 modules into your modules directory:


cd /etc/puppet/
git clone git://github.com/chriscowley/puppet-yumrepos.git modules/yumrepos
puppet module install puppetlabs/lvm --version 0.1.2
puppet module install thias/glusterfs --version 0.0.3


I have specified the versions as that is what was the latest at the time of writing. You should be able to take the latest as well, but comment with any differences if any. That gives the core of what you need so you can now move on to you nodes.pp.

“`
class basenode {
class { ‘yumrepos’: }
class { ‘yumrepos::epel’: }
}

class glusternode {
class { ‘basenode’: }
class { ‘yumrepos::gluster’: }

volume_group { “vg0”:

ensure => present,
physical_volumes => "/dev/sdb",
require => Physical_volume["/dev/sdb"]

}
physical_volume { “/dev/sdb”:

ensure => present

}
logical_volume { “gv0”:

ensure => present,
require => Volume_group['vg0'],
volume_group => "vg0",
size => "7G",

}
file { [ ‘/export’, ‘/export/gv0’]:

seltype => 'usr_t',
ensure => directory,

}
package { ‘xfsprogs’: ensure => installed
}
filesystem { “/dev/vg0/gv0”:

ensure => present,
fs_type => "xfs",
options => "-i size=512",
require => [Package['xfsprogs'], Logical_volume['gv0'] ],

}

mount { ‘/export/gv0’:

device => '/dev/vg0/gv0',
fstype => 'xfs',
options => 'defaults',
ensure => mounted,
require => [ Filesystem['/dev/vg0/gv0'], File['/export/gv0'] ],

}
class { ‘glusterfs::server’:

peers => $::hostname ? {
  'gluster1' => '192.168.1.38', # Note these are the IPs of the other nodes
  'gluster2' => '192.168.1.84',
},

}
glusterfs::volume { ‘gv0’:

create_options => 'replica 2 192.168.1.38:/export/gv0 192.168.1.84:/export/gv0',
require => Mount['/export/gv0'],

}
}

node ‘gluster1’ {
include glusternode
file { ‘/var/www’: ensure => directory }
glusterfs::mount { ‘/var/www’:

device => $::hostname ? {
  'gluster1' => '192.168.1.84:/gv0',
}

}
}

node ‘gluster2’ {
include glusternode
file { ‘/var/www’: ensure => directory }
glusterfs::mount { ‘/var/www’:

device => $::hostname ? {
  'gluster2' => '192.168.1.38:/gv0',
}

}
}
“`

What does all that do? Starting from the top:

• The basenode class does all your basic configuration across all your hosts. Mine actually does a lot more, but these are the relevant parts.
• The glusternode class is shared between all your GlusterFS nodes. This is where all your Server configuration is.
• Configures LVM
  • Defines the Volume Group “vg0” with the Physical Volume /dev/sdb
  • Creates a Logical Volume “gv0” for GlusterFS use and make it 7GB
• Configures the file system
  • Creates the directory /export/gv0
  • Formats the LV created previously with XFS (installs the package if necessary)
  • Mounts the LV at /export/gv0

This is now all ready for the GlusterFS module to do its stuff. All this happens in those last two sections.

• The class glusterfs::Server sets up the peering between the two hosts. This will actually generate a errors, but do not worry. This because gluster1 successfully peers with gluster2. As a result gluster2 fails to peer with gluster1 as they are already peered.
• Now glusterfs::volume creates a replicated volume, having first ensured that the LV is mounted correctly.
• All this is then included in the node declarations for gluster1 and gluster2.

All that creates the server very nicely. It will need a few passes to get everything in place, while giving a few red herring errors. It should would however, all the errors are there in the README for the GlusterFS module in PuppetForge, so do not panic.

A multi-petabyte scale-out storage system is pretty useless if the data cannot be read by anything. So lets use those nodes and mount the volume. This could also be a separate node (but once again I am being lazy) the process will be exactly the same.

• Create a mount point for it ( `file {‘/var/www’: ensure => directory }
• Define your glusterfs::mount using any of the hosts in the cluster.

Voila, that should all pull together and give you a fully automated GlusterFS set up. The sort of scale that GlusterFS can reach makes this sort of automation absolutely essential in my opinion. This should be relatively easy to convert to Chef or Ansible, whatever takes your fancy. I have just used Puppet because of my familiarity with it.

This is only one way of doing this, and I make no claims to being the most adept Puppet user in the world. All I hope to achieve is that someone finds this useful. Courteous comments welcome.

I’ve been a little slow in making release announcements, so here’s some news: I’ve just released the third stage of my puppet-ipa module. At the moment it now supports installation, managing of hosts, and managing of services. It integrates with … Continue reading →

UPDATE: We’re extending testing until 00:00 UTC on Tuesday, June 25. We want to give everyone a chance to get their RDMA clusters set up. It’s that time again – we want to test the GlusterFS 3.4 beta before we … Continue reading →

UPDATE: We’re extending testing until 00:00 UTC on Tuesday, June 25. We want to give everyone a chance to get their RDMA clusters set up.

It’s that time again – we want to test the GlusterFS 3.4 beta before we unleash it on the world. Like our last test fest, we want you to put the latest GlusterFS beta through real-world usage scenarios that will show you how it compares to previous releases.

This time around, we want to focus this round of testing on Infiniband and RDMA hardware.

For a description of how to do this, see the GlusterFest page on the community wiki. Run the tests, report the results, and report any bugs you found as a result.

As an added bonus, use the gluster-users mailing list as another outlet for your testing. After reporting the results on the GlusterFest page, report them on the list, too, and other users can confirm – or counter – your results.

Find a new bug that is confirmed by the Gluster QE team, and I’ll send you a free t-shirt (see image below).

Testing is underway now, and wraps up at 00:00 UTC on Saturday, June 22 (aka 5pm PT/8pm ET on Friday, June 21).

Last month, over 80 users and developers gathered at Intel’s Shanghai China Campus for a two-day workshop centered on oVirt, the Open Virtualization management platform. Jackson He, General Manager of Intel Asia and Pacific R&D Ltd. and Intel Software and … Continue reading →

Last month, over 80 users and developers gathered at Intel’s Shanghai China Campus for a two-day workshop centered on oVirt, the Open Virtualization management platform.

Jackson He, General Manager of Intel Asia and Pacific R&D Ltd. and Intel Software and Services Group PRC, provided the opening keynote, in which he spoke to a mostly local audience about Intel’s growth in China and continued commitment to open source software including such projects as oVirt, OpenStack, KVM and Hadoop. Intel’s continued commitment to Open Source virtualization was further demonstrated throughout the Workshop with great presentations by Gang Wei and Dongxiao Xu.

With three tracks spread across two days, this was the first workshop that also featured a day long Gluster Operations Track. This track, lead by John Mark Walker, Community Lead for Gluster, allowed for not only introductions and examples of leveraging GlusterFS storage solutions with oVirt, but also more advanced discussions, including a talk on developing with libgfapi, the GlusterFS translator framework, presented by Vijay Bellur, a Senior Principal Software Engineer at Red Hat.

In conjunction with the Gluster track on the first day of the workshop was the primary oVirt Operations track. With Red Hat presentations ranging from an introduction to oVirt to getting into the weeds of Troubleshooting, oVirt attendees were exposed to all levels of operational use cases and deployment tips. Presentations from IBM engineers Shu Ming and Mark Wu provided solid operational discussions covering oVirt testing in a nested virtualization configuration and outlining IBM’s commitment to and planned development objectives for oVirt.

The second day was all about oVirt developers. Of particular interest to attendees was a presentation by Zhengsheng Zhou of IBM discussing work done to support oVirt on Ubuntu. A key highlight of this event is the continued growth and interest around open virtualization solutions, with oVirt serving a foundational role. The interest in making oVirt available to other platforms is greatly encouraging, and we’re excited to see the community grow to include new platforms.

Also on day two, Doron Fediuck of Red Hat presented on oVirt SLAs, enforced by MoM, the Memory Overcommitment Manager. This presentation also provided a roadmap moving forward on this and other key features. Great discussions on this and most of the presentations allowed for developers to get engaged and focus in on where to help moving forward.

Presentations from the event are now available on the oVirt website.

With over 80 attendees representing Intel, IBM, Red Hat as well as the greater oVirt and Gluster communities, we’re pleased that this workshop was a success.

Scouring through large amounts of source code using just “grep” is soooo 1990.  With git, pipes, and scripting languages, you can combine the convenience of grep with simpler and more flexible parsing tools.   Here’s a couple of ways to …

In response to some discussion in the gluster community, I am releasing my puppet-lsi module. It’s quite simple, but it is very useful for rebuilding machines. It could do a lot more, but I wanted to depend on the proprietary … Continue reading →

Today at Red Hat Summit, Jon Masters, Red Hat’s chief ARM architect, demonstrated GlusterFS replicated on two ARM 64 servers, streaming a video. This marks the first successful demo of a distributed filesystem running on ARM 64. Video and podcast to come soon.    

Somebody today asked if GlusterFS could be made as fast as a local filesystem. My answer just came out without a ton of thought behind it, but I found it rather profound.
Comparing (any) clustered filesystem to a local filesystem is like comparing appl…

Setting up HBase can be tricky because of the intermediate states of processes which may be running.  Here are some important configs I found + an idempotent install script, reproducible hbase deployment which cleans your system and restarts hbase…