Gluster blog stories provide high-level spotlights on our users all over the world
Since we did not have any weekly Gluster news go out in September, this post tries to capture and summarize action from the entire month of September 2015. == General News == GlusterFS won yet another Bossie in the open source platforms, infrastructure, management, and orchestration software category. Long time users of the project might …Read more
We organized Docker Global Hack Day at Red Hat Office on 19th Sep’15. Though there were lots RSVPs, the turn up for the event was less than expected. We started the day by showing the recording of kick-off event. The … Continue reading →
Facebook hosted a great crowd on Monday, September 14, in Silicon Valley with about 30 attendees and a full night of presentations about Gluster. We started by introducing our new Gluster Community Lead, Amye Scavarda. You’ll see her a lot more in the coming months promoting Gluster and the Gluster community. The talks started with …Read more
Tiering feature is introduced in Gluster 3.7 release. Geo-replication may not perform well with Tiering feature yet. Races can happen since Rebalance moves files from one brick to another brick(hot to cold and cold to hot), but the Changelog/Journal remails in old brick itself. We know there will be problems since each Geo-replication worker(per brick) processes Changelogs belonging to respective brick and sync the data independently. Sync happens as two step operation, Create entry in Slave with the GFID recorded in Changelog, then use Rsync to sync data(using GFID access)
To uncover the bugs we need to setup workload and run multiple times since issues may not happen always. But it is tedious to run multiple times with actual data. How about simulating/mocking it?
Let us consider simple case of Rebalance, A file “f1” is created in Brick1 and after some time it becomes hot and Rebalance moved it to Brick2.
In Changelog we don’t capture the Rebalance Traffic, so in respective brick changelogs will contain,
# Brick1 Changelog CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # Brick2 Changelog DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef
If Brick1 worker processes fast, then Entry is created in Slave and Data Operation succeeds. Since Both the workers can independently, sequence of execution may be like
# Possible Sequence 1 [Brick1] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick1] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef [Brick2] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # Possible Sequence 2 [Brick2] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef [Brick1] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick1] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # Possible Sequence 3 [Brick1] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick2] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef [Brick1] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef
We don’t have any problems with first and last sequence, But in second sequence Rsync will try to sync data before Entry Creation and Fails.
To solve this issue, we thought if we record CREATE from Rebalance traffic then it will solve this problem. So now brick Changelogs looks like,
# Brick1 Changelog CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # Brick2 Changelog CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef
and possible sequences,
# Possible Sequence 1 [Brick1] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick1] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef [Brick2] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick2] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # Possible Sequence 2 [Brick2] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick1] CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 [Brick1] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef [Brick2] DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # and many more...
We do not have that problem, second CREATE will fail with EEXIST, we ignore it since it is safe error. But will this approach solves all the problems with Rebalance? When more FOPs added, it is very difficult to visualize or guess the problem.
To mock the concurrent workload, Collect sequence from each bricks Changelog and mix both the sequences. We should make sure that order in each brick remains same after the mix.
b1 = ["A", "B", "C", "D", "E"] b2 = ["F", "G"]
While mixing b2 in b1, for first element in b2 we can randomly choose a position in b1. Let us say random position we got is 2(Index is 2), and insert “F” in index 2 of b1
# before ["A", "B", "C", "D", "E"] # after ["A", "B", "F", "C", "D", "E"]
Now, to insert “G”, we should randomly choose anywhere after “F”. Once we get the sequence, mock the FOPs and compare with expected values.
I added a gist for testing following workload, it generates multiple sequences for testing.
# f1 created in Brick1, Rebalanced to Brick2 and then Unlinked # Brick1 Changelog CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef # Brick2 Changelog CREATE 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1 DATA 0945daec-6f8c-438e-9bbf-b2ebf07543ef UNLINK 0945daec-6f8c-438e-9bbf-b2ebf07543ef f1
Found two bugs.
I just started simulating with Tiering + Geo-replication workload, I may encounter more problems with Renames(Simple, multiple and cyclic). Will update the results soon.
I am sharing the script since it can be easily modified to work with different workloads and to test other projects/components.
Let me know if this is useful. Comments and Suggestions Welcome.
The Open Source and Standards team in Red Hat is very pleased to announce the addition of a new team member: Amye Scavarda, who will be taking the role of GlusterFS Community Lead.
Amye’s journey to the GlusterFS Project could arguably be said to hav…
The Oh-My-Vagrant project became public about one year ago and at the time it was more of a fancy template than a robust project, but 188 commits (and counting) later, it has gotten surprisingly useful and mature. james@computer:~/code/oh-my-vagrant$ git rev-list … Continue reading →
The Gluster Community currently provides GlusterFS packages for the following distributions: 3.5 3.6 3.7 Fedora 21 ¹ × × Fedora 22 × ¹ × Fedora 23 × × ¹ Fedora 24 × × ¹ RHEL/CentOS 5 × × RHEL/CentOS 6 × × × RHEL/CentOS 7 × × × Ubuntu 12.04 LTS (precise) × × Ubuntu …Read more
OpenMediaVault is a Debian based special purpose Linux Distribution to build a Network Attached Storage (NAS) System. It provides an easy to use web-based interface, Multilanguage support, Volume Management, Monitoring and a plugin system to extend it …
Git submodules are actually a very beautiful thing. You might prefer the word powerful or elegant, but that’s not the point. The downside is that they are sometimes misused, so as always, use with care. I’ve used them in projects … Continue reading →
Once again, time for the annual trek to Portland, Oregon for OSCON — perhaps for the last time! Next year, OSCON is going to be in Austin, TX — which seems like a bit of a mistake to me. Portland and OSCON go together like milk and cookies. If you’re going to be at OSCON, […]
Red Hat is looking for a Gluster Community Manager! Are you prepared to lead an army of ants into the cloud?
The Job Description
The role of a community manager is simple enough to define: Help your community to be wildly successful, grow new use…
On May 12 and 13th, the Gluster contributor community came together to discuss the state of Gluster and the plans for the future in sunny Barcelona, Spain. We had 32 active contributors present and 21 presentations. We discussed a wide range of materia…
GlusterFS 3.7 beta1 RPMs for RHEL, CentOS, (except EL5) and Fedora are available at download.gluster.org. 🙂 http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.7.0beta1/ Please test it out (not on your production system!), and let us know of any problems you hit via either BugZilla or our mailing lists! https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS http://www.gluster.org/community/
Over the past few days we’ve upgraded the version of Gerrit running on review.gluster.org, and moved to using GitHub as the authentication source for user accounts. This switch to GitHub as the authentication source has been fairly painful though. 🙁 There is a manual step (merging of accounts) needed the first time an existing user …Read more
The Gluster community is please to announce the release of GlusterFS-3.4.7. The GlusterFS 3.4.7 release is focused on bug fixes: 33608f5 cluster/dht: Changed log level to DEBUG 076143f protocol: Log ENODATA & ENOATTR at DEBUG in removexattr_cbk a0aa6fb build: argp-standalone, conditional build and build with gcc-5 35fdb73 api: versioned symbols in libgfapi.so for compatibility 8bc612d …Read more
The 4th beta for GlusterFS 3.4.7 is now available for testing. A handful of bugs have been fixed since the 3.4.6 release, check the references below for details. Bug reporters are encouraged to verify the fixes, and we invite others to test this beta to check for regressions. The ETA for 3.4.7 GA is tentatively …Read more
The 2nd beta for GlusterFS 3.4.7 is now available for testing. A handful of bugs have been fixed since the 3.4.6 release, check the references below for details. Bug reporters are encouraged to verify the fixes, and we invite others to test this beta to check for regressions. The ETA for 3.4.7 GA is not …Read more
As you walk down the crowded, narrow Rue des Bouchers on any given Brussels evening, you may find yourself assailed by restaurant barkers intent on getting you to swing into their establishment and dine on their fare. It’s not exactly a welcoming gesture, particularly when some of the barkers embrace the situation and just tell you flat out they want your money.
This past weekend, five kilometers to the south at the Université libre de Bruxelles, other crowded, narrow pathways were full of folks promoting a different sort of fare: free and open source software. If you don’t like crowds, FOSDEM would definitely a place to not be –except for the fact that so much goes on there. Far from unwelcoming, the sense of excitement and camaraderie at this annual event makes it a much more positive environment than similar events within FLOSS.
To describe FOSDEM to the newcomer is at once easy and frustrating –”big” certainly covers it, but doesn’t communicate the scope of an event that spans this small university in the heart of Brussels. The sheer variety of sessions alone are enough to feed the appetite of the most voracious FLOSS advocate.
oVirt certainly had its share of sessions in the conference program. From Gerrit to Gluster to smart VM scheduling, members of the oVirt community were out in force, educating attendees on the latest work being done in our project, as well as the tools and techniques we use to improve our process flow.
oVirt was also fortunate enough to have a booth in the event’s exhibition hall, smack-dab in-between the OpenStack and Puppet stands. Working alongside the OpenStack community folks was fun, given that many people tend to conflate the use cases for our two projects. But the proximity of our booths enabled us to explain to the many visitors who attended when and why would would want to use oVirt, and when an OpenStack solution like RDO would be a better fit.
Yes, free software projects do find themselves competing for mindshare and resources at times, but unlike the barkers of Rue des Bouchers, we’re not going to do it at the expense of the user’s wallet or valuable time. If a particular tool or platform works better for someone, we’d rather they use that happily instead of being unhappy with our offerings.
That doesn’t mean we’re not going to try to improve our own project, though. Self-awareness of problems and the self-discipline to improve them – these are the keys to open source and free software projects’ success. Not slick marketing and empty promises.
That’s never fare you should see at FOSDEM.
Follow the oVirt community on Twitter at @oVirt.
There is plenty of action planned out for GlusterFS in the period between Jan 30th to Feb 8th 2015. Our community enthusiasts will be attending and presenting at various conferences and meetups across the globe. Here is a breakdown of the planned action: GlusterFS QuickStart Tutorial by Lalatendu Mohanty at the CentOS Dojo in Brussels, …Read more