<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
gdb doesn't accept the coredump file ...<br>
<br>
<blockquote type="cite">root@nfs05 /var/crash # gdb
/usr/sbin/glusterfsd /var/crash/_usr_sbin_glusterfsd.0.crash <br>
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1<br>
Copyright (C) 2014 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later
<a class="moz-txt-link-rfc2396E" href="http://gnu.org/licenses/gpl.html"><http://gnu.org/licenses/gpl.html></a><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-linux-gnu".<br>
Type "show configuration" for configuration details.<br>
For bug reporting instructions, please see:<br>
<a class="moz-txt-link-rfc2396E" href="http://www.gnu.org/software/gdb/bugs/"><http://www.gnu.org/software/gdb/bugs/></a>.<br>
Find the GDB manual and other documentation resources online at:<br>
<a class="moz-txt-link-rfc2396E" href="http://www.gnu.org/software/gdb/documentation/"><http://www.gnu.org/software/gdb/documentation/></a>.<br>
For help, type "help".<br>
Type "apropos word" to search for commands related to "word"...<br>
Reading symbols from /usr/sbin/glusterfsd...(no debugging symbols
found)...done.<br>
"/var/crash/_usr_sbin_glusterfsd.0.crash" is not a core dump: File
format not recognized<br>
(gdb) bt<br>
No stack.<br>
</blockquote>
<br>
Yann<br>
<br>
<br>
<div class="moz-cite-prefix">Le 29/06/2016 08:39, Anoop C S a
écrit :<br>
</div>
<blockquote cite="mid:1467182368.26839.29.camel@redhat.com"
type="cite">
<pre wrap="">On Tue, 2016-06-28 at 10:49 +0200, Yann LEMARIE wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I found the coredump file, but it's a 15Mo file (zipped), I can't
post it on this mailling list.
</pre>
</blockquote>
<pre wrap="">
Great. In order to exactly pin point the crash location, can you please
attach gdb to extracted coredump file and share us the complete back
trace by executing `bt` command in gdb shell? Apart from gdb you may be
instructed to install some debug-info packages for extracting a useful
back trace while attaching gdb as follows:
# gdb /usr/sbin/glusterfsd <path-to-coredump-file>
If prompted install required packages and reattach the coredump file.
When you are inside (gdb) prompt type 'bt' and paste the back trace.
</pre>
<blockquote type="cite">
<pre wrap="">Here is some parts of the repport :
</pre>
<blockquote type="cite">
<pre wrap="">ProblemType: Crash
Architecture: amd64
Date: Sun Jun 26 11:27:44 2016
DistroRelease: Ubuntu 14.04
ExecutablePath: /usr/sbin/glusterfsd
ExecutableTimestamp: 1460982898
ProcCmdline: /usr/sbin/glusterfsd -s nfs05 --volfile-id
cdn.nfs05.srv-cdn -p /var/lib/glusterd/vols/cdn/run/nfs05-srv-
cdn.pid -S /var/run/gluster/d52ac3e6c0a3fa316a9e8360976f3af5.socket
--brick-name /srv/cdn -l /var/log/glusterfs/bricks/srv-cdn.log --
xlator-option *-posix.glusterd-uuid=6af63b78-a3da-459d-a909-
c010e6c9072c --brick-port 49155 --xlator-option cdn-server.listen-
port=49155
ProcCwd: /
ProcEnviron:
PATH=(custom, no user)
TERM=linux
ProcMaps:
7f25f18d9000-7f25f18da000 ---p 00000000 00:00 0
7f25f18da000-7f25f19da000 rw-p 00000000 00:00
0 [stack:849]
7f25f19da000-7f25f19db000 ---p 00000000 00:00 0
</pre>
</blockquote>
<pre wrap=""> ...
</pre>
<blockquote type="cite">
<pre wrap="">ProcStatus:
Name: glusterfsd
State: D (disk sleep)
Tgid: 7879
Ngid: 0
Pid: 7879
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 64
Groups: 0
VmPeak: 878404 kB
VmSize: 878404 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 96104 kB
VmRSS: 90652 kB
VmData: 792012 kB
VmStk: 276 kB
VmExe: 84 kB
VmLib: 7716 kB
VmPTE: 700 kB
VmSwap: 20688 kB
Threads: 22
SigQ: 0/30034
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000004a01
SigIgn: 0000000000001000
SigCgt: 00000001800000fa
CapInh: 0000000000000000
CapPrm: 0000001fffffffff
CapEff: 0000001fffffffff
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: 7fff
Cpus_allowed_list: 0-14
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 3
nonvoluntary_ctxt_switches: 1
Signal: 11
Uname: Linux 3.13.0-44-generic x86_64
UserGroups:
CoreDump: base64
</pre>
</blockquote>
<pre wrap=""> ...
Yann
Le 28/06/2016 09:31, Anoop C S a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">On Mon, 2016-06-27 at 15:05 +0200, Yann LEMARIE wrote:
</pre>
<blockquote type="cite">
<pre wrap=""> @Anoop,
Where can I find the coredump file ?
</pre>
</blockquote>
<pre wrap="">You will get hints about the crash from entries inside
/var/log/messages(for example pid of the process, location of
coredump
etc).
</pre>
<blockquote type="cite">
<pre wrap="">The crash occurs 2 times last 7 days, each time a sunday morning
with
no reason, no increase of traffic or something like this, the
volume
was mounted since 15 days.
The bricks are used as a CDN like, distributting small images and
css
files with a nginx https service (with a load balancer and 2
EC2), on
a sunday morning there is not a lot of activity ...
</pre>
</blockquote>
<pre wrap="">From the very minimal back trace that we have from brick logs I
would
assume that a truncate operation was being handled by trash
translator
and it crashed.
</pre>
<blockquote type="cite">
<pre wrap="">Volume infos:
</pre>
<blockquote type="cite">
<pre wrap="">root@nfs05 /var/log/glusterfs # gluster volume info cdn
Volume Name: cdn
Type: Replicate
Volume ID: c53b9bae-5e12-4f13-8217-53d8c96c302c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs05:/srv/cdn
Brick2: nfs06:/srv/cdn
Options Reconfigured:
performance.readdir-ahead: on
features.trash: on
features.trash-max-filesize: 20MB
</pre>
</blockquote>
<pre wrap="">
I don't know if there is a link with this crash problem, but I
have
another problem with my 2 servers that make GluserFS's clients
disconnected (from another volume) :
</pre>
<blockquote type="cite">
<pre wrap="">Jun 24 02:28:04 nfs05 kernel: [2039468.818617] xen_netfront:
xennet: skb rides the rocket: 19 slots
Jun 24 02:28:11 nfs05 kernel: [2039475.744086] net_ratelimit:
66
callbacks suppressed
</pre>
</blockquote>
<pre wrap=""> It seem to be a network interface problem :
<a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811</a>
Yann
Le 27/06/2016 12:59, Anoop C S a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">On Mon, 2016-06-27 at 09:47 +0200, Yann LEMARIE wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I'm using GlusterFS since many years and never see this
problem,
but
this is the second time in one week ...
I have 3 volumes with 2 bricks and 1 volume crash with no
reason,
</pre>
</blockquote>
<pre wrap="">Did you observe the crash while mounting the volume? Or can you
be
more
specific on what were you doing just before you saw the crash?
Can
you
please share the output of `gluster volume info <VOLNAME>`?
</pre>
<blockquote type="cite">
<pre wrap=""> I just have to stop/start the volume to make it up again.
The only logs I can find are in syslog :
</pre>
<blockquote type="cite">
<pre wrap="">Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames:
Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10)
Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset:
git://git.gluster.com/glusterfs.git
Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11
Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash:
Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44
Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details:
Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string:
glusterfs
3.7.11
Jun 26 11:27:44 nfs05 srv-cdn[7879]: ---------
Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames:
Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10)
Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset:
git://git.gluster.com/glusterfs.git
Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11
Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash:
Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44
Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details:
Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string:
glusterfs
3.7.11
Jun 26 11:27:44 nfs06 srv-cdn[1787]: ---------
</pre>
</blockquote>
<pre wrap="">
Thanks for your help
Regards
--
Yann Lemarié
iRaiser - Support Technique
<a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a>
_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">
--
Yann Lemarié
iRaiser - Support Technique
<a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a>
_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">
--
Yann Lemarié
iRaiser - Support Technique
<a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a>
_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<div dir="ltr">
<div
style="margin-bottom:0px;margin-left:0px;padding-bottom:5px">
<div style="overflow:hidden">
<div dir="ltr">
<div dir="ltr" style="font-size:13px">Yann Lemarié</div>
<div dir="ltr" style="font-size:13px"><span
style="font-size:x-small">iRaiser - Support Technique</span></div>
<div dir="ltr" style="background-color:white
paddind-left:5px;margin-top:5px;width:145px;margin-bottom:5px;">
<a href="http://www.iraiser.eu" target="_blank"> <img
title="" alt="iRaiser Logotype"
src="cid:part1.09020601.04010302@iraiser.eu"
height="22" width="102"> </a> </div>
<div dir="ltr" style="font-size:13px"><span
style="font-size:x-small"><a
href="mailto:ylemarie@iraiser.eu" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a></a></span></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>