<div dir="ltr">Hi Kotresh,<div><br></div><div>i have used below command to verify any open fd for file. </div><div><br></div><div>"ls -l /proc/*/fd | grep filename".<br></div><div><br></div><div>as soon as write completes there no open fd's, if there is any alternate option. please let me know will also try that.</div><div><br></div><div><br></div><div><br></div><div><br></div><div>Also, below is my scrub status in my test setup. number of skipped files slow reducing day by day. I think files are skipped due to bitrot signature process is not completed yet.</div><div><br></div><div>where can i see scrub skipped files?<br></div><div><br></div><div><br></div><div><div>Volume name : glsvol1</div><div><br></div><div>State of scrub: Active (Idle)</div><div><br></div><div>Scrub impact: normal</div><div><br></div><div>Scrub frequency: daily</div><div><br></div><div>Bitrot error log location: /var/log/glusterfs/bitd.log</div><div><br></div><div>Scrubber error log location: /var/log/glusterfs/scrub.log</div><div><br></div><div><br></div><div>=========================================================</div><div><br></div><div>Node: localhost</div><div><br></div><div>Number of Scrubbed files: 1644</div><div><br></div><div>Number of Skipped files: 1001</div><div><br></div><div>Last completed scrub time: 2016-09-20 11:59:58</div><div><br></div><div>Duration of last scrub (D:M:H:M:S): 0:0:39:26</div><div><br></div><div>Error count: 0</div><div><br></div><div><br></div><div>=========================================================</div><div><br></div><div>Node: 10.1.2.3</div><div><br></div><div>Number of Scrubbed files: 1644</div><div><br></div><div>Number of Skipped files: 1001</div><div><br></div><div>Last completed scrub time: 2016-09-20 10:50:00</div><div><br></div><div>Duration of last scrub (D:M:H:M:S): 0:0:38:17</div><div><br></div><div>Error count: 0</div><div><br></div><div><br></div><div>=========================================================</div><div><br></div><div>Node: 10.1.2.4</div><div><br></div><div>Number of Scrubbed files: 981</div><div><br></div><div>Number of Skipped files: 1664</div><div><br></div><div>Last completed scrub time: 2016-09-20 12:38:01</div><div><br></div><div>Duration of last scrub (D:M:H:M:S): 0:0:35:19</div><div><br></div><div>Error count: 0</div><div><br></div><div><br></div><div>=========================================================</div><div><br></div><div>Node: 10.1.2.1</div><div><br></div><div>Number of Scrubbed files: 1263</div><div><br></div><div>Number of Skipped files: 1382</div><div><br></div><div>Last completed scrub time: 2016-09-20 11:57:21</div><div><br></div><div>Duration of last scrub (D:M:H:M:S): 0:0:37:17</div><div><br></div><div>Error count: 0</div><div><br></div><div><br></div><div>=========================================================</div><div><br></div><div>Node: 10.1.2.2</div><div><br></div><div>Number of Scrubbed files: 1644</div><div><br></div><div>Number of Skipped files: 1001</div><div><br></div><div>Last completed scrub time: 2016-09-20 11:59:25</div><div><br></div><div>Duration of last scrub (D:M:H:M:S): 0:0:39:18</div><div><br></div><div>Error count: 0</div><div><br></div><div>=========================================================</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Thanks </div><div>Amudhan</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 21, 2016 at 11:45 AM, Kotresh Hiremath Ravishankar <span dir="ltr"><<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Amudhan,<br>
<br>
I don't think it's the limitation with read data from the brick.<br>
To limit the usage of CPU, throttling is done using token bucket<br>
algorithm. The log message showed is related to it. But even then<br>
I think it should not take 12 minutes for check-sum calculation unless<br>
there is an fd open (might be internal). Could you please cross verify<br>
if there are any fd opened on that file by looking into /proc? I will<br>
also test it out in the mean time and get back to you.<br>
<span class="im HOEnZb"><br>
Thanks and Regards,<br>
Kotresh H R<br>
<br>
----- Original Message -----<br>
> From: "Amudhan P" <<a href="mailto:amudhan83@gmail.com">amudhan83@gmail.com</a>><br>
</span><div class="HOEnZb"><div class="h5">> To: "Kotresh Hiremath Ravishankar" <<a href="mailto:khiremat@redhat.com">khiremat@redhat.com</a>><br>
> Cc: "Gluster Users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Tuesday, September 20, 2016 3:19:28 PM<br>
> Subject: Re: [Gluster-users] 3.8.3 Bitrot signature process<br>
><br>
> Hi Kotresh,<br>
><br>
> Please correct me if i am wrong, Once a file write completes and as soon as<br>
> closes fds, bitrot waits for 120 seconds and starts hashing and update<br>
> signature for the file in brick.<br>
><br>
> But, what i am feeling that bitrot takes too much of time to complete<br>
> hashing.<br>
><br>
> below is test result i would like to share.<br>
><br>
><br>
> writing data in below path using dd :<br>
><br>
> /mnt/gluster/data/G (mount point)<br>
> -rw-r--r-- 1 root root 10M Sep 20 12:19 test53-bs10M-c1.nul<br>
> -rw-r--r-- 1 root root 100M Sep 20 12:19 test54-bs10M-c10.nul<br>
><br>
> No any other write or read process is going on.<br>
><br>
><br>
> Checking file data in one of the brick.<br>
><br>
> -rw-r--r-- 2 root root 2.5M Sep 20 12:23 test53-bs10M-c1.nul<br>
> -rw-r--r-- 2 root root 25M Sep 20 12:23 test54-bs10M-c10.nul<br>
><br>
> file's stat and getfattr info from brick, after write process completed.<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ stat test53-bs10M-c1.nul<br>
> File: ‘test53-bs10M-c1.nul’<br>
> Size: 2621440 Blocks: 5120 IO Block: 4096 regular file<br>
> Device: 821h/2081d Inode: 536874168 Links: 2<br>
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<br>
> Access: 2016-09-20 12:23:28.798886647 +0530<br>
> Modify: 2016-09-20 12:23:28.994886646 +0530<br>
> Change: 2016-09-20 12:23:28.998886646 +0530<br>
> Birth: -<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ stat test54-bs10M-c10.nul<br>
> File: ‘test54-bs10M-c10.nul’<br>
> Size: 26214400 Blocks: 51200 IO Block: 4096 regular file<br>
> Device: 821h/2081d Inode: 536874169 Links: 2<br>
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<br>
> Access: 2016-09-20 12:23:42.902886624 +0530<br>
> Modify: 2016-09-20 12:23:44.378886622 +0530<br>
> Change: 2016-09-20 12:23:44.378886622 +0530<br>
> Birth: -<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ sudo getfattr -m. -e hex -d<br>
> test53-bs10M-c1.nul<br>
> # file: test53-bs10M-c1.nul<br>
> trusted.bit-rot.version=<wbr>0x020000000000000057daa7b50002<wbr>e5b4<br>
> trusted.ec.config=<wbr>0x0000080501000200<br>
> trusted.ec.size=<wbr>0x0000000000a00000<br>
> trusted.ec.version=<wbr>0x0000000000000050000000000000<wbr>0050<br>
> trusted.gfid=<wbr>0xe2416bd1aae4403c88f44286273b<wbr>be99<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ sudo getfattr -m. -e hex -d<br>
> test54-bs10M-c10.nul<br>
> # file: test54-bs10M-c10.nul<br>
> trusted.bit-rot.version=<wbr>0x020000000000000057daa7b50002<wbr>e5b4<br>
> trusted.ec.config=<wbr>0x0000080501000200<br>
> trusted.ec.size=<wbr>0x0000000006400000<br>
> trusted.ec.version=<wbr>0x0000000000000320000000000000<wbr>0320<br>
> trusted.gfid=<wbr>0x54e018dd8c5a4bd79e0317729d8a<wbr>57c5<br>
><br>
><br>
><br>
> file's stat and getfattr info from brick, after bitrot signature updated.<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ stat test53-bs10M-c1.nul<br>
> File: ‘test53-bs10M-c1.nul’<br>
> Size: 2621440 Blocks: 5120 IO Block: 4096 regular file<br>
> Device: 821h/2081d Inode: 536874168 Links: 2<br>
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<br>
> Access: 2016-09-20 12:25:31.494886450 +0530<br>
> Modify: 2016-09-20 12:23:28.994886646 +0530<br>
> Change: 2016-09-20 12:27:00.994886307 +0530<br>
> Birth: -<br>
><br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ sudo getfattr -m. -e hex -d<br>
> test53-bs10M-c1.nul<br>
> # file: test53-bs10M-c1.nul<br>
> trusted.bit-rot.signature=<wbr>0x0102000000000000006de7493c5c<wbr>90f643357c268fbaaf461c1567e033<wbr>4e4948023ce17268403aa37a<br>
> trusted.bit-rot.version=<wbr>0x020000000000000057daa7b50002<wbr>e5b4<br>
> trusted.ec.config=<wbr>0x0000080501000200<br>
> trusted.ec.size=<wbr>0x0000000000a00000<br>
> trusted.ec.version=<wbr>0x0000000000000050000000000000<wbr>0050<br>
> trusted.gfid=<wbr>0xe2416bd1aae4403c88f44286273b<wbr>be99<br>
><br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ stat test54-bs10M-c10.nul<br>
> File: ‘test54-bs10M-c10.nul’<br>
> Size: 26214400 Blocks: 51200 IO Block: 4096 regular file<br>
> Device: 821h/2081d Inode: 536874169 Links: 2<br>
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<br>
> Access: 2016-09-20 12:25:47.510886425 +0530<br>
> Modify: 2016-09-20 12:23:44.378886622 +0530<br>
> Change: 2016-09-20 12:38:05.954885243 +0530<br>
> Birth: -<br>
><br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ sudo getfattr -m. -e hex -d<br>
> test54-bs10M-c10.nul<br>
> # file: test54-bs10M-c10.nul<br>
> trusted.bit-rot.signature=<wbr>0x010200000000000000394c345f0b<wbr>0c63ee652627a62eed069244d35c4d<wbr>5134e4f07d4eabb51afda47e<br>
> trusted.bit-rot.version=<wbr>0x020000000000000057daa7b50002<wbr>e5b4<br>
> trusted.ec.config=<wbr>0x0000080501000200<br>
> trusted.ec.size=<wbr>0x0000000006400000<br>
> trusted.ec.version=<wbr>0x0000000000000320000000000000<wbr>0320<br>
> trusted.gfid=<wbr>0x54e018dd8c5a4bd79e0317729d8a<wbr>57c5<br>
><br>
><br>
> (Actual time taken for reading file from brick for md5sum)<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ time md5sum test53-bs10M-c1.nul<br>
> 8354dcaa18a1ecb52d0895bf00888c<wbr>44 test53-bs10M-c1.nul<br>
><br>
> real 0m0.045s<br>
> user 0m0.007s<br>
> sys 0m0.003s<br>
><br>
> gfstst-node5:/media/disk2/<wbr>brick2/data/G$ time md5sum test54-bs10M-c10.nul<br>
> bed3c0a4a1407f584989b4009e9ce3<wbr>3f test54-bs10M-c10.nul<br>
><br>
> real 0m0.166s<br>
> user 0m0.062s<br>
> sys 0m0.011s<br>
><br>
> As you can see that 'test54-bs10M-c10.nul' file took around 12 minutes to<br>
> update bitort signature (pls refer stat output for the file).<br>
><br>
> what would be the cause for such a slow read?. Any limitation in read data<br>
> from brick?<br>
><br>
> Also, i am seeing this line bitd.log, what does this mean?<br>
> [bit-rot.c:1784:br_rate_limit_<wbr>signer] 0-glsvol1-bit-rot-0: [Rate Limit<br>
> Info] "tokens/sec (rate): 131072, maxlimit: 524288<br>
><br>
><br>
> Thanks<br>
> Amudhan P<br>
><br>
><br>
><br>
> On Mon, Sep 19, 2016 at 1:00 PM, Kotresh Hiremath Ravishankar <<br>
> <a href="mailto:khiremat@redhat.com">khiremat@redhat.com</a>> wrote:<br>
><br>
> > Hi Amudhan,<br>
> ><br>
> > Thanks for testing out the bitrot feature and sorry for the delayed<br>
> > response.<br>
> > Please find the answers inline.<br>
> ><br>
> > Thanks and Regards,<br>
> > Kotresh H R<br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "Amudhan P" <<a href="mailto:amudhan83@gmail.com">amudhan83@gmail.com</a>><br>
> > > To: "Gluster Users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> > > Sent: Friday, September 16, 2016 4:14:10 PM<br>
> > > Subject: Re: [Gluster-users] 3.8.3 Bitrot signature process<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > Can anyone reply to this mail.<br>
> > ><br>
> > > On Tue, Sep 13, 2016 at 12:49 PM, Amudhan P < <a href="mailto:amudhan83@gmail.com">amudhan83@gmail.com</a> ><br>
> > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > I am testing bitrot feature in Gluster 3.8.3 with disperse EC volume 4+1.<br>
> > ><br>
> > > When i write single small file (< 10MB) after 2 seconds i can see bitrot<br>
> > > signature in bricks for the file, but when i write multiple files with<br>
> > > different size ( > 10MB) it takes long time (> 24hrs) to see bitrot<br>
> > > signature in all the files.<br>
> ><br>
> > The default timeout for signing to happen is 120 seconds. So the<br>
> > signing will happen<br>
> > 120 secs after the last fd gets closed on that file. So if the file is<br>
> > being written<br>
> > continuously, it will not be signed until 120 secs after it's last fd is<br>
> > closed.<br>
> > ><br>
> > > My questions are.<br>
> > > 1. I have enabled scrub schedule as hourly and throttle as normal, does<br>
> > this<br>
> > > make any impact in delaying bitrot signature?<br>
> > No.<br>
> > > 2. other than "bitd.log" where else i can watch current status of bitrot,<br>
> > > like number of files added for signature and file status?<br>
> > Signature will happen after 120 sec of last fd closure, as said above.<br>
> > There is not status command which tracks the signature of the files.<br>
> > But there is bitrot status command which tracks the number of files<br>
> > scrubbed.<br>
> ><br>
> > #gluster vol bitrot <volname> scrub status<br>
> ><br>
> ><br>
> > > 3. where i can confirm that all the files in the brick are bitrot signed?<br>
> ><br>
> > As said, signing information of all the files is not tracked.<br>
> ><br>
> > > 4. is there any file read size limit in bitrot?<br>
> ><br>
> > I didn't get. Could you please elaborate this ?<br>
> ><br>
> > > 5. options for tuning bitrot for faster signing of files?<br>
> ><br>
> > Bitrot feature is mainly to detect silent corruption (bitflips) of<br>
> > files due to long<br>
> > term storage. Hence the default is 120 sec of last fd closure, the<br>
> > signing happens.<br>
> > But there is a tune able which can change the default 120 sec but<br>
> > that's only for<br>
> > testing purposes and we don't recommend it.<br>
> ><br>
> > gluster vol get master features.expiry-time<br>
> ><br>
> > For testing purposes, you can change this default and test.<br>
> > ><br>
> > > Thanks<br>
> > > Amudhan<br>
> > ><br>
> > ><br>
> > > ______________________________<wbr>_________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>