<div dir="ltr">Hello<div><br></div><div>i have some strange problem that happens on one of the servers with glusterfs.</div><div><br></div><div>Configuration:</div><div><br></div><div><div>-- gluster volume info --</div><div><br></div><div>Volume Name: gisdata</div><div>Type: Distribute</div><div>Volume ID: a49bcb69-fcc5-4f39-b556-e23e29da38e7</div><div>Status: Started</div><div>Number of Bricks: 7</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 10.1.0.4:/export/sdc1/brick</div><div>Brick2: 10.1.0.5:/export/sdc1/brick</div><div>Brick3: 10.1.0.6:/export/sdc1/brick</div><div>Brick4: 10.1.0.7:/export/sdc1/brick</div><div>Brick5: 10.1.0.9:/export/sdc1/brick</div><div>Brick6: 10.1.0.10:/export/sdc1/brick</div><div>Brick7: 10.1.0.12:/export/sdc1/brick</div><div>Options Reconfigured:</div><div>server.statedump-path: /tmp</div><div>performance.cache-size: 512MB</div><div>nfs.drc: off</div></div><div><br></div><div>System mount to master server as </div><div><br></div><div>10.1.0.4:/gisdata on /usr/local/mountpoint type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)<br></div><div><br></div><div>and its share by samba to the client who manage files on distribute filesystem.</div><div><br></div><div>Client tryes to delete directory (Data/Projects/DELIVERY/D39) but it fails with error: Can't delete, directory is not empty.</div><div>When i inspect that directory on the Bricks i found it totaly broken</div><div><br></div><div><div> ls -lah /export/sdc1/brick/Data/Projects/DELIVERY/D39/</div><div>ls: cannot access /export/sdc1/brick/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO: No such file or directory</div><div>total 16K</div><div>drwxr-xr-x 3 root 500 8.0K Dec 1 10:52 .</div><div>drwxr-xr-x 40 root 500 4.0K Nov 30 22:43 ..</div><div>?????????? ? ? ? ? ? 494-11-23-1_WD_PO</div></div><div><br></div><div>When i recreate that directory it looks fine but has doubled:</div><div><br></div><div><div>ls -lai /export/sdc1/brick/Data/Projects/DELIVERY/D39/</div><div><div>total 16</div><div> 30075838 drwxr-xr-x 4 root 500 8192 Dec 1 10:56 .</div><div> 9185992 drwxr-xr-x 40 root 500 4096 Nov 30 22:43 ..</div><div>2147631741 drwxr-xr-x 2 root root 6 Dec 1 10:56 494-11-23-1_WD_PO</div><div>2147631741 drwxr-xr-x 2 root root 6 Dec 1 10:56 494-11-23-1_WD_PO</div></div><div><br></div><div>Last logs:</div><div><br></div></div><div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.352895] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.355764] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.358448] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.381921] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.843840] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.846111] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.848384] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.857213] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.678051] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.680209] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.682785] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.697692] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.880597] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.885190] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.889059] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.905035] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.985591] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.990133] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.995346] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:49.012616] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.278130] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.283321] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.286403] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.305513] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 09:52:26.319979] E [dht-rebalance.c:1496:gf_defrag_fix_layout] 0-gisdata-dht: /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO lookup failed</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.172294] I [dht-layout.c:640:dht_layout_normalize] 0-gisdata-dht: found anomalies in /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO. holes=1 overlaps=0</div><div>/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.173616] I [dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of /Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO</div></div><div><br></div><div><br></div><div><br></div><div><div>It happens on server 10.1.0.9. Any ideas how to fix that?</div><div><br></div>-- <br><div class="gmail_signature">_________________________<br>- tetsucceed<br><br></div>
</div></div>