Planet Gluster

Aggregated news from external sources

Catching up with Infrastructure Debt

2017-11-05

If you run an infrastructure, there’s a good chance you have some debt tucked
in your system somewhere. There’s also a good chance that you’re not getting
enough time to fix those debts. There will most likely be a good reason why
something is done in the way it is. This is just how things are in general.
After I joined Gluster, I’ve worked with my fellow sysadmin to tackle our large
infrastructure technical debt over the course of time. It goes like this:

  • We run a pretty old version of Gerrit on CentOS 5.
  • We run a pretty old version of Jenkins on CentOS 6.
  • We run CentOS 6 for all our regressions machines.
  • We run CentOS 6 for all our build machines.
  • We run NetBSD on Rackspace in a setup that is not easy to automate nor is it
    currently part of our automation.
  • We have a bunch of physical machines in a DC, but we haven’t had time to move
    our VMs over and use Rackspace as burstable capacity.

That is in no way and exhaustive list. But we’ve managed to tackle 2.5 items
from the list. Here’s what we did in order:

  • Upgraded Gerrit to the then latest version.
  • Setup Gerrit staging to test newer versions regularly for scheduling
    migration.
  • Created new CentOS 7 VMs on our hardware and moved the builds in there.
  • Moved Gerrit over to a new CentOS 7 host.
  • Wrote ansible scripts to manage most of Gerrit, but deployed currently only
    to staging.
  • Upgraded Jenkins to the latest LTS.
  • Moved Jenkins to a CentOS 7 host (Done last week, more details coming
    up!)

If I look at it, it almost looks like I’ve failed. But again, like dealing with
most infrastructure debt, you touch one thing and you realize it’s broken in
someway and someone depended on that breakage. What I’ve done is I’ve had to
pick and prioritize what things I would spend my time on. At the end of the
day, I have to justify my time in terms of moving the project forward. Fixing
the infrastructure debt for Gerrit was a great example. I could actually focus
on it with everyone’s support. Fixing Jenkins was a priority since we wanted to
use some of the newer features, again I had backing to do that. Moving things
to our hardware is where things get tricky. There’s some financial goals we can
hit if we make the move, but outside of that, we have no reason to move. But
long-term, we want to me mostly in our hardware, since we spent money on it.
This is, understandably going slow. There’s a subtle capacity difference and
the noisy neighbor problem affects us quite strongly when we try to do anything
in this regard.

Source: nigelb (Catching up with Infrastructure Debt)

BLOG

  • 26 Apr 2019
    Gluster Monthly Newsletter, April 2...

    Upcoming Community Happy Hour at Red Hat Summit! Tue, May 7, 2019, 6:30 PM – 7:30 PM EDT https://cephandglusterhappyhour_rhsummit.eventbrite.com has all the details. Gluster 7 Roadmap Discussion kicked off for our 7 roadmap on the mailing lists, see [Gluster-users] GlusterFS v7.0 (and v8.0) roadmap discussion https://lists.gluster.org/pipermail/gluster-users/2019-March/036139.html for more details. Community...

    Read more
  • 24 Apr 2019
    Community Survey Feedback, 2019

    In this year’s survey, we asked quite a few questions about how people are using Gluster, how much storage they’re managing, their primary use for Gluster, and what they’d like to see added. Here’s some of the highlights from this year!

    Read more
  • 24 Apr 2019
    How to Deploy the OpenVPN Encryptio...

    This is part of a new series on using Gluster! OpenVPN is open source software that serves as the basis for a Virtual Private Network capable of supporting a point-to-point or site-to-site connection. Along with the fact that it’s free to use, it also has the benefit of being one...

    Read more