<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Le 10/06/2016 13:08, Mohammed Rafi K C
a écrit :<br>
</div>
<blockquote cite="mid:575A9FC6.60801@redhat.com" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<br>
<br>
<div class="moz-cite-prefix">On 06/10/2016 02:41 PM, Yannick
Perret wrote:<br>
</div>
<blockquote cite="mid:575A844C.1020109@liris.cnrs.fr" type="cite">
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
<div class="moz-cite-prefix">I get no feedback on that but I
think I found the problem:<br>
the glusterfs client grows on memory until no memory available
and them it crashes.<br>
</div>
</blockquote>
<br>
If we you can take a statedump (kill -SIGUSR1 $client_pid) and
send it across, I can take a look to see where it consumes so many
memory. Since you said it is not reproducible with latest Debian
system and If it is not important, that is fine for me.<br>
<br>
</blockquote>
Thanks for your feedback.<br>
As it is clearly for me a transitional installation (moving local
disks from our last old machine to network storage before big
migration/upgrade) and as it works fine (and faster btw) on
up-to-date machines I don't need to investigate more than that:
using NFS gluster mount is fine − even if using directly gluster
would have been better − and will do the job until this old machine
will die.<br>
<blockquote cite="mid:575A9FC6.60801@redhat.com" type="cite"> <br>
<blockquote cite="mid:575A844C.1020109@liris.cnrs.fr" type="cite">
<div class="moz-cite-prefix"> <br>
I performed the same operations on an other machine without
being able to reproduce the problem.<br>
The machine with the problem is an old machine (debian, 3.2.50
kernel, 32bit), whereas the other machine is an up-to-date
debian 64bit.<br>
<br>
To give some stats the glusterfs on the client starts with
less than 810220 of resident size and finished with 3055336
(3Go!) when it crashes again. The volume was mounted only on
this machine, used by only one process (a 'cp -Rp').<br>
<br>
Running the same from a recent machine gives far more stable
memory usage (43364 of resident size and few and small
increasing).<br>
Of course I'm using the same glusterfs version (compiled from
sources on both machines).<br>
<br>
As I can't upgrade this old machine due to version
compatibility with old softs − at least until we replace these
old softs − I will so use a NFS mountpoint from the gluster
servers.<br>
<br>
Whatever I still get on the recent machine very verbose logs
for each directory creation:<br>
[2016-06-10 08:35:12.965438] I
[dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
0-HOME-LIRIS-dht: chunk size = 0xffffffff / 2064114 = 0x820<br>
[2016-06-10 08:35:12.965473] I
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
0-HOME-LIRIS-dht: assigning range size 0xffe76e40 to
HOME-LIRIS-replicate-0<br>
[2016-06-10 08:35:12.966987] I [MSGID: 109036]
[dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
0-HOME-LIRIS-dht: Setting layout of /log_apache_error with
[Subvol_name: HOME-LIRIS-replicate-0, Err: -1 , Start: 0 ,
Stop: 4294967295 ], <br>
</div>
</blockquote>
<br>
This is an INFO level message which says about the layout of a
directory. Gluster-fuse client will print this INFO when it sets
the layout on a directory. This error messages can be safely
ignore.<br>
<br>
<br>
<blockquote cite="mid:575A844C.1020109@liris.cnrs.fr" type="cite">
<div class="moz-cite-prefix"> <br>
I switched clients to WARNING log level (gluster volume set
HOME-LIRIS diagnostics.client-sys-log-level WARNING) which is
fine for me.<br>
But maybe WARNING should be the default log level, at least
for clients, no? In production getting 3 lines per created
directory is useless, and anyone who wants to analyze a
problem will switch to INFO or DEBUG.<br>
</div>
</blockquote>
<br>
I see many uses get panic about this error message. I agree, we
have to do something with this log entry's.<br>
</blockquote>
Yes first I wonder if it was "problems" (i.e. Err: -1). In
particular because you often read logs when you have problems :).
Then I saw it was "INFO" data.<br>
But even with that in mind writing +500 bytes in logs for each
directory creation is a bit too much in production context (log
size, sorting useful data from all these entries…).<br>
Many tools are by default in "warning/error" mode by default, which
may be also the case for gluster, at least for the clients side.<br>
<br>
Thanks.<br>
<br>
Regards,<br>
--<br>
Y.<br>
<br>
<blockquote cite="mid:575A9FC6.60801@redhat.com" type="cite"> <br>
<blockquote cite="mid:575A844C.1020109@liris.cnrs.fr" type="cite">
<div class="moz-cite-prefix"> <br>
Regards,<br>
--<br>
Y.<br>
<br>
<br>
<br>
Le 08/06/2016 17:35, Yannick Perret a écrit :<br>
</div>
<blockquote cite="mid:57583B3A.4010104@liris.cnrs.fr"
type="cite">Hello, <br>
<br>
I have a replica 2 volume managed on 2 identical server, using
3.6.7 version of gluster. Here is the volume info: <br>
Volume Name: HOME-LIRIS <br>
Type: Replicate <br>
Volume ID: 47b4b856-371b-4b8c-8baa-2b7c32d7bb23 <br>
Status: Started <br>
Number of Bricks: 1 x 2 = 2 <br>
Transport-type: tcp <br>
Bricks: <br>
Brick1: sto1.mydomain:/glusterfs/home-liris/data <br>
Brick2: sto2.mydomain:/glusterfs/home-liris/data <br>
<br>
It is mounted on a (single) client with mount -t glusterfs
sto1.mydomain:/HOME-LIRIS /futur-home/ <br>
<br>
I started to copy a directory (~550Go, ~660 directories with
many files) into it. Copy was done using 'cp -Rp'. <br>
<br>
It seems to work fine but I get *many* log entries in the
corresponding mountpoint logs: <br>
[2016-06-07 14:01:27.587300] I
[dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
0-HOME-LIRIS-dht: chunk size = 0xffffffff / 2064114 = 0x820 <br>
[2016-06-07 14:01:27.587338] I
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
0-HOME-LIRIS-dht: assigning range size 0xffe76e40 to
HOME-LIRIS-replicate-0 <br>
[2016-06-07 14:01:27.588436] I [MSGID: 109036]
[dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
0-HOME-LIRIS-dht: Setting layout of /olfamine with
[Subvol_name: HOME-LIRIS-replicate-0, Err: -1 , Start: 0 ,
Stop: 4294967295 ], <br>
<br>
This is repeated for many files (124088 exactly). Is it
normal? If yes I use default settings on the client so I find
it a little bit verbose. If no can someone tell me what is the
problem here? <br>
<br>
Moreover at the end of the log file I have: <br>
[2016-06-08 04:42:58.210617] A [MSGID: 0]
[mem-pool.c:110:__gf_calloc] : no memory available for size
(14651) [call stack follows] <br>
[2016-06-08 04:42:58.219060] A [MSGID: 0]
[mem-pool.c:134:__gf_malloc] : no memory available for size
(21026) [call stack follows] <br>
pending frames: <br>
frame : type(1) op(CREATE) <br>
frame : type(1) op(CREATE) <br>
frame : type(1) op(LOOKUP) <br>
frame : type(0) op(0) <br>
patchset: git://git.gluster.com/glusterfs.git <br>
signal received: 11 <br>
time of crash: <br>
2016-06-08 04:42:58 <br>
configuration details: <br>
argp 1 <br>
backtrace 1 <br>
dlfcn 1 <br>
libpthread 1 <br>
llistxattr 1 <br>
setfsid 1 <br>
spinlock 1 <br>
epoll.h 1 <br>
xattr.h 1 <br>
st_atim.tv_nsec 1 <br>
package-string: glusterfs 3.6.7 <br>
<br>
Which clearly don't seems right. <br>
The data were not all copied (logs of copy got a logical list
of "final transport node not connected" (or similar, it was
translated in my language)). <br>
<br>
I re-mounted the volume and created a directory with 'mkdir
TOTO' and get a similar: <br>
[2016-06-08 15:32:23.692936] I
[dht-selfheal.c:1065:dht_selfheal_layout_new_directory]
0-HOME-LIRIS-dht: chunk size = 0xffffffff / 2064114 = 0x820 <br>
[2016-06-08 15:32:23.692982] I
[dht-selfheal.c:1103:dht_selfheal_layout_new_directory]
0-HOME-LIRIS-dht: assigning range size 0xffe76e40 to
HOME-LIRIS-replicate-0 <br>
[2016-06-08 15:32:23.694144] I [MSGID: 109036]
[dht-common.c:6296:dht_log_new_layout_for_dir_selfheal]
0-HOME-LIRIS-dht: Setting layout of /TOTO with [Subvol_name:
HOME-LIRIS-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295
], <br>
but I don't get such message with files. <br>
<br>
If it can help volumes are ~2To and content is far from that,
and both bricks are ext4 (both same size). <br>
<br>
<br>
Any help would be appreciated. <br>
<br>
Regards, <br>
-- <br>
Y. <br>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>