Dev Code that uses that hadoop needs to be customized.  If you have control over your dev tests, you can simply useConfiguration.addDefaultResource(“/path/to/core-site.xml”);This will load the conf file after loading other default resources.In gen…

The people at JetBrains have given the GlusterFS project an “unlimited # of users” license for their cross-platform PyCharm Python IDE: http://www.jetbrains.com/pycharm/download/ Everyone has their own preferences with programming IDE’s, and I’ve used several over the years, generally based on Eclipse.  However, I happen to like PyCharm at the moment, so applied for JetBrain’s “free …Read more

One of the best things about OpenStack development is the rich testing and review frameworks around it.  The most important of which are tempest and devstack.  In this post I will discuss how I recently squashed a particularly painful bug (read: I make dumb mistakes) in my latest patch to Glance. The Bug My patch […]

I went into this expecting problems. Fedora 19 ships with GlusterFS 3.4.0 beta and I’m using GlusterFS 3.3 in production. I expected that I would have to downgrade my Fedora packages so I could use my volumes. I expected problems.
What I didn’t expect …

{% img right http://www.hastexo.com/system/files/imagecache/sidebar/20120221105324808-f2df3ea3e3aeab8_250_0.png %} As I promised on Twitter, this is how I automate a GlusterFS deployment. I’m making a few assumptions here:

• I am using CentOS 6, so should work on RHEL 6 and Scientific Linux 6 too. Others may work, but YMMV.
  • As I use XFS, RHEL users will need the Scalable Storage option. Ext4 will work, but XFS is recommended.
• That you have a way of automating your base OS installation. My personal preference is to use Razor.
• You have a system with at least a complete spare disk dedicated to a GlusterFS brick. That is the best way to run GlusterFS anyway.
• You have 2 nodes and want to replicate the data
• You have a simple setup with only a single network, because I am being lazy. As a proof-of concept this is fine. Modifying this for second network is quite easy, just change the IP address in you use.

{% img https://docs.google.com/drawings/d/1XA7GH3a4BL1uszFXrSsZjysi59Iinh-0RmhqdDbt7QQ/pub?w=673&h=315 ‘simple gluster architecture’ %}

The diagram above shows the basic layout of what to start from in terms of hardware. In terms of software, you just need a basic CentOS 6 install and to have Puppet working.

I use a pair of Puppet modules (both in the Forge): thias/glusterfs and puppetlabs/lvm. The GlusterFS module CAN do the LVM config, but that strikes me as not the best idea. The UNIX philosophy of “do one job well” holds up for Puppet modules as well. You will also need my yumrepos module.

Clone those 3 modules into your modules directory:


cd /etc/puppet/
git clone git://github.com/chriscowley/puppet-yumrepos.git modules/yumrepos
puppet module install puppetlabs/lvm --version 0.1.2
puppet module install thias/glusterfs --version 0.0.3


I have specified the versions as that is what was the latest at the time of writing. You should be able to take the latest as well, but comment with any differences if any. That gives the core of what you need so you can now move on to you nodes.pp.

“`
class basenode {
class { ‘yumrepos’: }
class { ‘yumrepos::epel’: }
}

class glusternode {
class { ‘basenode’: }
class { ‘yumrepos::gluster’: }

volume_group { “vg0”:

ensure => present,
physical_volumes => "/dev/sdb",
require => Physical_volume["/dev/sdb"]

}
physical_volume { “/dev/sdb”:

ensure => present

}
logical_volume { “gv0”:

ensure => present,
require => Volume_group['vg0'],
volume_group => "vg0",
size => "7G",

}
file { [ ‘/export’, ‘/export/gv0’]:

seltype => 'usr_t',
ensure => directory,

}
package { ‘xfsprogs’: ensure => installed
}
filesystem { “/dev/vg0/gv0”:

ensure => present,
fs_type => "xfs",
options => "-i size=512",
require => [Package['xfsprogs'], Logical_volume['gv0'] ],

}

mount { ‘/export/gv0’:

device => '/dev/vg0/gv0',
fstype => 'xfs',
options => 'defaults',
ensure => mounted,
require => [ Filesystem['/dev/vg0/gv0'], File['/export/gv0'] ],

}
class { ‘glusterfs::server’:

peers => $::hostname ? {
  'gluster1' => '192.168.1.38', # Note these are the IPs of the other nodes
  'gluster2' => '192.168.1.84',
},

}
glusterfs::volume { ‘gv0’:

create_options => 'replica 2 192.168.1.38:/export/gv0 192.168.1.84:/export/gv0',
require => Mount['/export/gv0'],

}
}

node ‘gluster1’ {
include glusternode
file { ‘/var/www’: ensure => directory }
glusterfs::mount { ‘/var/www’:

device => $::hostname ? {
  'gluster1' => '192.168.1.84:/gv0',
}

}
}

node ‘gluster2’ {
include glusternode
file { ‘/var/www’: ensure => directory }
glusterfs::mount { ‘/var/www’:

device => $::hostname ? {
  'gluster2' => '192.168.1.38:/gv0',
}

}
}
“`

What does all that do? Starting from the top:

• The basenode class does all your basic configuration across all your hosts. Mine actually does a lot more, but these are the relevant parts.
• The glusternode class is shared between all your GlusterFS nodes. This is where all your Server configuration is.
• Configures LVM
  • Defines the Volume Group “vg0” with the Physical Volume /dev/sdb
  • Creates a Logical Volume “gv0” for GlusterFS use and make it 7GB
• Configures the file system
  • Creates the directory /export/gv0
  • Formats the LV created previously with XFS (installs the package if necessary)
  • Mounts the LV at /export/gv0

This is now all ready for the GlusterFS module to do its stuff. All this happens in those last two sections.

• The class glusterfs::Server sets up the peering between the two hosts. This will actually generate a errors, but do not worry. This because gluster1 successfully peers with gluster2. As a result gluster2 fails to peer with gluster1 as they are already peered.
• Now glusterfs::volume creates a replicated volume, having first ensured that the LV is mounted correctly.
• All this is then included in the node declarations for gluster1 and gluster2.

All that creates the server very nicely. It will need a few passes to get everything in place, while giving a few red herring errors. It should would however, all the errors are there in the README for the GlusterFS module in PuppetForge, so do not panic.

A multi-petabyte scale-out storage system is pretty useless if the data cannot be read by anything. So lets use those nodes and mount the volume. This could also be a separate node (but once again I am being lazy) the process will be exactly the same.

• Create a mount point for it ( `file {‘/var/www’: ensure => directory }
• Define your glusterfs::mount using any of the hosts in the cluster.

Voila, that should all pull together and give you a fully automated GlusterFS set up. The sort of scale that GlusterFS can reach makes this sort of automation absolutely essential in my opinion. This should be relatively easy to convert to Chef or Ansible, whatever takes your fancy. I have just used Puppet because of my familiarity with it.

This is only one way of doing this, and I make no claims to being the most adept Puppet user in the world. All I hope to achieve is that someone finds this useful. Courteous comments welcome.

I’ve been a little slow in making release announcements, so here’s some news: I’ve just released the third stage of my puppet-ipa module. At the moment it now supports installation, managing of hosts, and managing of services. It integrates with … Continue reading →

UPDATE: We’re extending testing until 00:00 UTC on Tuesday, June 25. We want to give everyone a chance to get their RDMA clusters set up. It’s that time again – we want to test the GlusterFS 3.4 beta before we … Continue reading →

UPDATE: We’re extending testing until 00:00 UTC on Tuesday, June 25. We want to give everyone a chance to get their RDMA clusters set up.

It’s that time again – we want to test the GlusterFS 3.4 beta before we unleash it on the world. Like our last test fest, we want you to put the latest GlusterFS beta through real-world usage scenarios that will show you how it compares to previous releases.

This time around, we want to focus this round of testing on Infiniband and RDMA hardware.

For a description of how to do this, see the GlusterFest page on the community wiki. Run the tests, report the results, and report any bugs you found as a result.

As an added bonus, use the gluster-users mailing list as another outlet for your testing. After reporting the results on the GlusterFest page, report them on the list, too, and other users can confirm – or counter – your results.

Find a new bug that is confirmed by the Gluster QE team, and I’ll send you a free t-shirt (see image below).

Testing is underway now, and wraps up at 00:00 UTC on Saturday, June 22 (aka 5pm PT/8pm ET on Friday, June 21).

Last month, over 80 users and developers gathered at Intel’s Shanghai China Campus for a two-day workshop centered on oVirt, the Open Virtualization management platform. Jackson He, General Manager of Intel Asia and Pacific R&D Ltd. and Intel Software and … Continue reading →

Last month, over 80 users and developers gathered at Intel’s Shanghai China Campus for a two-day workshop centered on oVirt, the Open Virtualization management platform.

Jackson He, General Manager of Intel Asia and Pacific R&D Ltd. and Intel Software and Services Group PRC, provided the opening keynote, in which he spoke to a mostly local audience about Intel’s growth in China and continued commitment to open source software including such projects as oVirt, OpenStack, KVM and Hadoop. Intel’s continued commitment to Open Source virtualization was further demonstrated throughout the Workshop with great presentations by Gang Wei and Dongxiao Xu.

With three tracks spread across two days, this was the first workshop that also featured a day long Gluster Operations Track. This track, lead by John Mark Walker, Community Lead for Gluster, allowed for not only introductions and examples of leveraging GlusterFS storage solutions with oVirt, but also more advanced discussions, including a talk on developing with libgfapi, the GlusterFS translator framework, presented by Vijay Bellur, a Senior Principal Software Engineer at Red Hat.

In conjunction with the Gluster track on the first day of the workshop was the primary oVirt Operations track. With Red Hat presentations ranging from an introduction to oVirt to getting into the weeds of Troubleshooting, oVirt attendees were exposed to all levels of operational use cases and deployment tips. Presentations from IBM engineers Shu Ming and Mark Wu provided solid operational discussions covering oVirt testing in a nested virtualization configuration and outlining IBM’s commitment to and planned development objectives for oVirt.

The second day was all about oVirt developers. Of particular interest to attendees was a presentation by Zhengsheng Zhou of IBM discussing work done to support oVirt on Ubuntu. A key highlight of this event is the continued growth and interest around open virtualization solutions, with oVirt serving a foundational role. The interest in making oVirt available to other platforms is greatly encouraging, and we’re excited to see the community grow to include new platforms.

Also on day two, Doron Fediuck of Red Hat presented on oVirt SLAs, enforced by MoM, the Memory Overcommitment Manager. This presentation also provided a roadmap moving forward on this and other key features. Great discussions on this and most of the presentations allowed for developers to get engaged and focus in on where to help moving forward.

Presentations from the event are now available on the oVirt website.

With over 80 attendees representing Intel, IBM, Red Hat as well as the greater oVirt and Gluster communities, we’re pleased that this workshop was a success.

Scouring through large amounts of source code using just “grep” is soooo 1990.  With git, pipes, and scripting languages, you can combine the convenience of grep with simpler and more flexible parsing tools.   Here’s a couple of ways to …

In response to some discussion in the gluster community, I am releasing my puppet-lsi module. It’s quite simple, but it is very useful for rebuilding machines. It could do a lot more, but I wanted to depend on the proprietary … Continue reading →

Today at Red Hat Summit, Jon Masters, Red Hat’s chief ARM architect, demonstrated GlusterFS replicated on two ARM 64 servers, streaming a video. This marks the first successful demo of a distributed filesystem running on ARM 64. Video and podcast to come soon.    

Somebody today asked if GlusterFS could be made as fast as a local filesystem. My answer just came out without a ton of thought behind it, but I found it rather profound.
Comparing (any) clustered filesystem to a local filesystem is like comparing appl…

Setting up HBase can be tricky because of the intermediate states of processes which may be running.  Here are some important configs I found + an idempotent install script, reproducible hbase deployment which cleans your system and restarts hbase…

During my talks, I often try to make the point that horizontally scalable systems are often necessary not only to achieve high aggregate performance but also to overcome performance variation. This is true in general, but especially in a public cloud. In that environment, performance variation both between nodes and over time is much greater […]

Automated cluster creation in VMs by scriptifying your KVM setups.Automatically provisioning VM clusters comes up alot:When you want to simulate / practice installing a clustered app or utility. When you want to confirm that workload distribution is o…

Heading to LinuxCon Japan 2013? If you’ll be attending the conference or will be in Tokyo on May 31st, we’d like to welcome you at the Gluster Community Workshop. We’ll have a full day of talks on all things Gluster, including: Creating a Shared Storage Service with GlusterFS by Keisuke Takahashi Converged Infrastructure: Leveraging oVirt …Read more

Apparently, at RICON East today, Seagate’s James Hughes said something like this. Any distributed filesystem like GlusterFS or Ceph that tries to preserve the POSIX API will go the way of the dodo bird. I don’t actually know the exact quote. The above is from a tweet by Basho’s Seth Thomas, and is admittedly a […]

I have been a little slow on the uptake on this one. I would like to say it is because I was carefully digesting the information, but that is not true; the reality is that I have just had 2 5 day weekends in 2 weeks :-).

The big announcement at thi…