GlusterFS Troubleshooting Guide

From GlusterDocumentation

Jump to: navigation, search

Contents

[edit]

Debugging GlusterFS

[edit]

DEBUG Mode

DEBUG mode is the most verbose level of of logging. It is useful to start GlusterFS binaries in foreground (NON-DAEMON ) mode and redirect all verbose log messages to standard output. Follow the example commands below.

GlusterFS Server Example: 

# glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG

GlusterFS Client Example: 

# glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG --server=192.168.1.1 /mnt
[edit]

Debugging Using strace Tool

strace utility traces glusterfs binaries at the system call level.

GlusterFS Server Example: 

# strace glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG

GlusterFS Client Example: 

# strace glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG --server=192.168.1.1 /mnt

GlusterFS Client Example (look for only open system calls): 

# strace glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG --server=192.168.1.1 /mnt 2>&1 | grep open

OR

If the GlusterFS process already running you can attach to the server and client using their process id's by looking into the process table eg; 'ps ax' output.

GlusterFS server example: 

# strace -p $GLUSTERFSD_PID

GlusterFS client example: 

# strace -p $GLUSTERFS_PID
[edit]

Debugging Using ltrace Tool

ltrace utility traces glusterfs binaries at library call level.

GlusterFS Server Example: 

# ltrace glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG

GlusterFS Client Example: 

# ltrace glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG --server=192.168.1.1 /mnt
[edit]

Using debug/trace Translator

Trace translator produces extensive trace information for debugging purpose. The debug information is written to the GlusterFS log file, which by default is found in /var/log/gluster/glusterfs.log. Trace volume can be inserted or layered on top of any volume which needs to be debugged. All the calls with its arguments/values will be logged. 

### Export volume "brick" with the contents of "/home/export" directory.
volume brick
  type storage/posix                   # POSIX FS translator
  option directory /home/export        # Export this directory
end-volume

### Trace storage/posix translator.
volume trace
  type debug/trace
  subvolumes brick
  option debug on
end-volume
[edit]

GNU Debugger

GNU Debugger (gdb) debugger helps you pin point the problem at the source level. It will be the last resort in your troubleshooting process. To let gdb properly debug your binary, you will have to compile with -g (debug info) and -O0 (disable optimization).

Client Compilation: 

CFLAGS="-O0 -g" ./configure --prefix=/usr --sysconfdir=/etc --disable-server

Server Compilation: 

CFLAGS="-O0 -g" ./configure --prefix=/usr --sysconfdir=/etc --disable-fuse-client
[edit]

Debugging GlusterFS Server

GlusterFS server produces core file under your top level root directory (/core) when there is crash. This file stores the state of the server at the time of failure. Using 'gdb' tool you can go back in time and analyze what went wrong.

Backtrace Example: 

# gdb glusterfsd /core
(gdb) backtrace

Running GlusterFS Server from inside gdb: 

# gdb glusterfsd
(gdb) r  --no-daemon --log-file=/dev/stdout --log-level=DEBUG
[edit]

Debugging GlusterFS Client

GlusterFS client produces core file under current working directory.

Backtrace Example: 

# gdb glusterfs core
(gdb) backtrace

Running GlusterFS Client from inside gdb: 

# gdb glusterfs
(gdb) r  --no-daemon --log-file=/dev/stdout --log-level=DEBUG --server=192.168.1.1 /mnt
[edit]

GlusterFS Error Messages

[edit]

GlusterFS Server Errors

[edit]

[ERROR] glusterfsd: FATAL: could not open specfile: '/etc/glusterfs/glusterfs-server.vol'

glusterfsd server requires volume specification file under /etc/glusterfs/glusterfs-server.vol. Default installation will only provide /etc/glusterfs/glusterfs-server.vol.sample sample file. You need to copy it to /etc/glusterfs/glusterfs-server.vol actual file.

[edit]

gf_log_init: failed to open logfile "/usr/var/log/glusterfs/glusterfsd.log" (Permission denied)

Check if you are running as root user. Use 'whoami' utility.

[edit]

GlusterFS Client Errors

[edit]

fusermount: failed to access mountpoint /mnt: Transport endpoint is not connected

umount /mnt and mount again.

[edit]

Error "Transport endpoint is not connected"

GlusterFS mount failed. Start glusterfs client in DEBUG mode and look out for descriptive error messages.

[edit]

connect to server failed

[edit]

SERVER-ADDRESS: Connection refused

GluserFS Server is not running or dead. Also check your network connections or firewall settings. To check if the server is reachable, try the following command: 

telnet IP-ADDRESS 6996

If server is accessible, your 'telnet' command should connect and block. If not you will see an error message like telnet: Unable to connect to remote host: Connection refused. '6996' is the default GlusterFS port. If you have customized it, then use the corresponding port instead.

[edit]

gf_log_init: failed to open logfile "/usr/var/log/glusterfs/glusterfs.log" (Permission denied)

[edit]

glusterfs: failed to open logfile "/usr/var/log/glusterfs/glusterfs.log"

Check if you are running as root user. Use 'whoami' utility.

[edit]

AppArmour and GlusterFS

Under OpenSuSE GNU/Linux, AppArmour security feature did not allow GlusterFS to create temporary files or network socket connection even as root user. You will see the error messages like Unable to open log file: Operation not permitted or Connection refused error. Disabling apparmour using YaST tool or properly configuring apparmour to recognize glusterfsd or glusterfs / fusermount should solve the problem.

Retrieved from "http://www.gluster.org/docs/index.php/GlusterFS_Troubleshooting_Guide"
Personal tools