<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">I noticed one more detail, since I found some entries which are not healed on another gluster filesystem. I resolved the gfids and noticed that *only* the log files generated by the SLURM workload manager (slurm-*.out) are affected. Are there any known problems with SLURM + glusterfs?</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-10-01 12:44 GMT+02:00 Micha Ober <span dir="ltr"><<a href="mailto:micha2k@gmail.com" target="_blank">micha2k@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I noticed that I have two files which are not healed:</div><div><br></div><div><div>root@giant5:~# gluster volume heal gv0 info</div><div>Gathering Heal info on volume gv0 has been successful</div><div><br></div><div>Brick giant1:/gluster/sdc/gv0</div><div>Number of entries: 1</div><div>/holicki/lqcd/slurm-7251.out</div><div><br></div><div>Brick giant2:/gluster/sdc/gv0</div><div>Number of entries: 1</div><div>/holicki/lqcd/slurm-7251.out</div><div><br></div><div>Brick giant3:/gluster/sdc/gv0</div><div>Number of entries: 1</div><div>/holicki/lqcd/slurm-7249.out</div><div><br></div><div>Brick giant4:/gluster/sdc/gv0</div><div>Number of entries: 1</div><div>/holicki/lqcd/slurm-7249.out</div><div><br></div><div>Brick giant5:/gluster/sdc/gv0</div><div>Number of entries: 1</div><div><gfid:e9793d5e-7174-49b0-9fa9-<wbr>90f8c35948e7></div><div><br></div><div>Brick giant6:/gluster/sdc/gv0</div><div>Number of entries: 1</div><div><gfid:e9793d5e-7174-49b0-9fa9-<wbr>90f8c35948e7></div><div><br></div><div>Brick giant1:/gluster/sdd/gv0</div><div>Number of entries: 1</div><div>/jwilhelm/CUDA/Ns16_Nt32_m0.<wbr>0100_beta1.7000_lambda0.0050/<wbr>mu1.000/slurm-5660.out</div><div><br></div><div>Brick giant2:/gluster/sdd/gv0</div><div>Number of entries: 1</div><div>/jwilhelm/CUDA/Ns16_Nt32_m0.<wbr>0100_beta1.7000_lambda0.0050/<wbr>mu1.000/slurm-5660.out</div><div><br></div><div>Brick giant3:/gluster/sdd/gv0</div><div>Number of entries: 0</div><div><br></div><div>Brick giant4:/gluster/sdd/gv0</div><div>Number of entries: 0</div><div><br></div><div>Brick giant5:/gluster/sdd/gv0</div><div>Number of entries: 0</div><div><br></div><div>Brick giant6:/gluster/sdd/gv0</div><div>Number of entries: 0</div></div><div><br></div><div><br></div><div>(Disregard the file "slurm-7251.out", this is/was IO in progress.)</div><div><br></div><div>The logs are filled with entries like this:</div><div><br></div><div><div>[2016-09-30 12:45:26.611375] I [afr-self-heal-data.c:655:afr_<wbr>sh_data_fix] 0-gv0-replicate-3: no active sinks for performing self-heal on file /jwilhelm/CUDA/Ns16_Nt32_m0.<wbr>0100_beta1.7000_lambda0.0050/<wbr>mu1.000/slurm-5660.out</div><div>[2016-09-30 12:45:36.874802] I [afr-self-heal-data.c:655:afr_<wbr>sh_data_fix] 0-gv0-replicate-3: no active sinks for performing self-heal on file /jwilhelm/CUDA/Ns16_Nt32_m0.<wbr>0100_beta1.7000_lambda0.0050/<wbr>mu1.000/slurm-5660.out</div><div>[2016-09-30 12:45:53.701884] I [afr-self-heal-data.c:655:afr_<wbr>sh_data_fix] 0-gv0-replicate-3: no active sinks for performing self-heal on file /jwilhelm/CUDA/Ns16_Nt32_m0.<wbr>0100_beta1.7000_lambda0.0050/<wbr>mu1.000/slurm-5660.out</div></div><div><br></div><div>I checked with md5sum that both files are identical.</div><div>Then, I used setfattr as proposed in an older thread in this mailing list:</div><div><br></div><div>setfattr -n trusted.afr.gv0-client-7 -v 0x000000000000000000000000 /gluster/sdd/gv0/jwilhelm/<wbr>CUDA/Ns16_Nt32_m0.0100_beta1.<wbr>7000_lambda0.0050/mu1.000/<wbr>slurm-5660.out<br></div><div><br></div><div>I did this on both nodes for both clients, so it now looks like this (on both nodes/bricks):</div><div><br></div><div><div>getfattr -d -m . -e hex /gluster/sdd/gv0/jwilhelm/<wbr>CUDA/Ns16_Nt32_m0.0100_beta1.<wbr>7000_lambda0.0050/mu1.000/<wbr>slurm-5660.out</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster/sdd/gv0/jwilhelm/CUDA/<wbr>Ns16_Nt32_m0.0100_beta1.7000_<wbr>lambda0.0050/mu1.000/slurm-<wbr>5660.out</div><div>trusted.afr.gv0-client-6=<wbr>0x000000000000000000000000</div><div>trusted.afr.gv0-client-7=<wbr>0x000000000000000000000000</div><div>trusted.gfid=<wbr>0xcb7978fa42e74a0b97928a871263<wbr>38ac</div><div><br></div></div><div>I triggered heal, but the files do not disappear from heal info. But also, they are not listed in split-brain oder heal-failed.</div><div><br></div><div>I used gfid-resolver.sh for the other file:</div><div><div>e9793d5e-7174-49b0-9fa9-<wbr>90f8c35948e7 == File: /gluster/sdc/gv0/jwilhelm/<wbr>CUDA/Ns16_Nt32_m0.0100_beta1.<wbr>7000_lambda0.0050/mu0.800/<wbr>slurm-5663.out</div></div><div><br></div><div>This file is also marked as dirty:</div><div><br></div><div><div>root@giant5:/var/log/<wbr>glusterfs# getfattr -d -m . -e hex /gluster/sdc/gv0/jwilhelm/<wbr>CUDA/Ns16_Nt32_m0.0100_beta1.<wbr>7000_lambda0.0050/mu0.800/<wbr>slurm-5663.out</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster/sdc/gv0/jwilhelm/CUDA/<wbr>Ns16_Nt32_m0.0100_beta1.7000_<wbr>lambda0.0050/mu0.800/slurm-<wbr>5663.out</div><div>trusted.afr.gv0-client-4=<wbr>0x000000010000000000000000</div><div>trusted.afr.gv0-client-5=<wbr>0x000000010000000000000000</div><div>trusted.gfid=<wbr>0xe9793d5e717449b09fa990f8c359<wbr>48e7</div></div><div><br></div><div><br></div><div>How can I fix this, i.e. get the files healed? I'm using gluster 3.4.2 on Ubuntu 14.04.3.</div><div><br></div><div>I also thought about scheduling a downtime and upgrading gluster, but I don't know if I can do this as long as there are files to be healed.</div><div><br></div><div>Thanks for any advice.</div></div>
</blockquote></div><br></div>