Gluster blog stories provide high-level spotlights on our users all over the world
|What happens when you press “go” ?
We think in abstractions… we code in mega-abstractions… but…
I recently read about Amdahl’s law in this awesome article about caching for high performance web sites: http://architects.dzone.com/articles/role-caching-large-scale. The emphasis was on detection of the true bottleneck in a system. The Linux operating system’s transparent and universally accepted organization lends itself quite well to recursive bottleneck analysis. Nevertheless, its never easy, when given a slow or stalled program, to tell exactly whats wrong.
Why is this hard?
There is no right answer to this question – but part of the problem is that as developers we deal in abstractions which do not map directly to the work done by a system. This is (kind of) a good thing – it frees us to focus on business logic, modularity, and horizontal scalability. Language makers have done an amazing job at separating programmers from machines, and we should applaud them wholeheartedly. But when software runs into scalability or performance problems, abstraction can really come back to bite you. Disks might be slow, resources could be scarce, networking performance may be dismal, or simple libraries may be corrupted or missing. In any case – you’ve got to be able to find out why – and your core programming language chops won’t always help.
But what will help… is an understanding of your operating system. So the following is a quick tour of the Linux innards I always find useful, followed by several links which shed some much needed light on the internals of how programming languages ultimately connect to the kernel.
1) First, the directory structure in linux. Very basic:
/bin: These are binaries that anybody can run.
/sbin: These are binaries that only root runs (ifup, ifconfig, …)
/home: This is where all your data goes.
/tmp: This is where all your temporary files go. Its cleared automatically by the OS, periodically.
/usr: This is where user create/built programs typically go. For example, if you compile your own version of a particular C library because it didn’t work out of the box on your machine, you would put it in /usr/local.
/opt: Applications and programs go here, in particular, the ones that you get from external sources, in one peice. For example, an executable version of google chrome would go in /opt.
/var: System related files go here that are created by programs. Log files (/var/log), mail related stuff, cached (/log/cache/) data from programs that run.
/etc: Configuration files for stuff like networking, disks, etc… Linux runs several services: When you change something, you need to restart the corresponding service… Otherwise you sill see no results.
/dev: The files in this directory represent devices (audio, keyboards, disk drives, etc..)
2) Some quick notes about networking:
Do disks in the cloud work the same?
Yes: They are still mounted as devices, and the device abstraction is sufficiently robust to support extremely diverse disk implementations. Its instructive to consider the way, in the cloud, the “/dev” directory maps to disks. As an experiment, we can create an AMI instance with no external storage, and run the following command:
Subsequently, we can attach an EBS (Elastic Block Storage – a fancy new-fangled cloud storage solution) drive to that same instance using the AWS console. Shortly thereafter, when running the same command (ls /dev), we will see a new entry: This entry represents the raw disk device. By mounting it as a filesystem, we can then write to that detached storage device.
4) How your code works in the Linux environment:
5) Finally – a list of articles that will help you to understand your code, in order of increasing abstraction.
2020 has not been a year we would have been able to predict. With a worldwide pandemic and lives thrown out of gear, as we head into 2021, we are thankful that our community and project continued to receive new developers, users and make small gains. For that and a...
It has been a while since we provided an update to the Gluster community. Across the world various nations, states and localities have put together sets of guidelines around shelter-in-place and quarantine. We request our community members to stay safe, to care for their loved ones, to continue to be...
The initial rounds of conversation around the planning of content for release 8 has helped the project identify one key thing – the need to stagger out features and enhancements over multiple releases. Thus, while release 8 is unlikely to be feature heavy as previous releases, it will be the...