<div dir="ltr"><span style="font-size:13px">Reviving the old thread..</span><div style="font-size:13px"><br></div><div style="font-size:13px">I was thinking about a possible NDMP based migration tool built into GlusterFS.</div><div style="font-size:13px">Migration from any other file system storage (source) to gluster storage (target) can be challenging because of lack of a tool built into GlusterFS. Host based migration is a common approach but it can be painful for large datasets. <br></div><div style="font-size:13px"><br></div><div style="font-size:13px">The proposed m<span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap">igration tool will transfer data over the Network Data Management Protocol (NDMP). This tool will make use of the snapshots feature on the source system to ensure that a point-in- time copy of the file system is transferred to the target. The proposed tool can also be utilized to migrate from gluster on old hardware to gluster on new hardware.</span></div><div style="font-size:13px"><span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap"><br></span></div><div style="font-size:13px"><span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap">Following is the proposed workflow:</span></div><div style="font-size:13px"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><br></div><div style="font-size:13px"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;white-space:pre-wrap;background-color:transparent;line-height:1.38;color:rgb(0,0,0);font-family:Arial"> <b>1) </b></span><span style="font-size:13.333333333333332px;white-space:pre-wrap;background-color:transparent;line-height:1.38;color:rgb(0,0,0);font-family:Arial"><b>First copy</b></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Migrate recursively from source directory to the gluster cluster. This is expected to be a time consuming step.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><b> 2) Incremental copy</b></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Migrating any changes that occur after the first copy. </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap"> 3) Cutover</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Restrict clients from accessing data on the source , migrate remaining changes , and redirect clients to access their data on the gluster cluster. </span></p></div><div style="font-size:13px"><span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap"><br></span></div><div style="font-size:13px"><span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap">Note that there may be issues with migrating existing file permissions, quota enforcements, etc on the source.</span></div><div style="font-size:13px"><span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap"><br></span></div><div style="font-size:13px"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Missing pieces in gluster</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">-NDMP support</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">-May have to revive “ndmp-server” project?</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Please share your thoughts.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.333333333333332px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span></p></div><div style="font-size:13px"><span style="color:rgb(0,0,0);font-family:Arial;line-height:18px;white-space:pre-wrap">Regards</span></div><div class="" style="margin:2px 0px 0px;font-size:13px"><div id=":1uy" class="" tabindex="0"><img class="" src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif" style=""></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 1, 2014 at 6:43 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Dec 01, 2014 at 05:40:53PM +0530, Aravinda wrote:<br>
> Hi,<br>
><br>
> We are trying to implement backup support for GlusterFS. Many Network backup<br>
> utilities like Bacula(open source), Amanda(open source), Symantec NetBackup<br>
> support NDMP(<a href="http://www.ndmp.org/" rel="noreferrer" target="_blank">http://www.ndmp.org/</a>). Comparison is available here<br>
> <a href="http://wiki.bacula.org/doku.php?id=comparisons" rel="noreferrer" target="_blank">http://wiki.bacula.org/doku.<wbr>php?id=comparisons</a><br>
><br>
> Plan is to create glusterfs-ndmp-server, which utilizes glusterfs changelogs<br>
> to detect changes for incremental backup. Design is not yet finalized,<br>
> Comments & Suggestions Welcome.<br>
><br>
> Looks like a project(<a href="https://forge.gluster.org/ndmp-server" rel="noreferrer" target="_blank">https://forge.gluster.<wbr>org/ndmp-server</a>) in forge.gluster<br>
> is discontinued.<br>
><br>
> PS: NDMP support is not available in Open Source editions of Bacula and<br>
> Amanda, but available in Enterprise Editions.<br>
<br>
</span>Bareos (<a href="http://www.bareos.org" rel="noreferrer" target="_blank">www.bareos.org</a>) a full Open Source fork of Bacula seems to<br>
support NDMP:<br>
- <a href="https://github.com/bareos/bareos/blob/master/README.NDMP" rel="noreferrer" target="_blank">https://github.com/bareos/<wbr>bareos/blob/master/README.NDMP</a><br>
- <a href="http://doc.bareos.org/master/html/bareos-manual-main-reference.html#x1-12800010.3" rel="noreferrer" target="_blank">http://doc.bareos.org/master/<wbr>html/bareos-manual-main-<wbr>reference.html#x1-12800010.3</a><br>
<br>
I think Bareos has already support to backup to a Gluster Volume.<br>
<br>
>From my understanding you are proposing a feature that integrates a<br>
(async/changelog) NDMP-client in the Gluster xlator stack that writes to<br>
a NDMP-server?<br>
<br>
HTH,<br>
Niels<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> References:<br>
> -----------<br>
> 1. NDMP support in NetBackup<br>
> <a href="http://www.symantec.com/business/support/index?page=content&id=DOC6456" rel="noreferrer" target="_blank">http://www.symantec.com/<wbr>business/support/index?page=<wbr>content&id=DOC6456</a><br>
> 2. NDMP Presentation<br>
> <a href="http://www.ndmp.org/download/sdk_v4/ndmp-overview-r2.ppt" rel="noreferrer" target="_blank">http://www.ndmp.org/download/<wbr>sdk_v4/ndmp-overview-r2.ppt</a><br>
> 3. NDMP website <a href="http://www.ndmp.org/" rel="noreferrer" target="_blank">http://www.ndmp.org/</a><br>
> 4. Bacula and Amanda website <a href="http://bacula.org/" rel="noreferrer" target="_blank">http://bacula.org/</a> and <a href="http://www.amanda.org/" rel="noreferrer" target="_blank">http://www.amanda.org/</a><br>
><br>
> --<br>
> regards<br>
> Aravinda<br>
> ______________________________<wbr>_________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://supercolony.gluster.<wbr>org/mailman/listinfo/gluster-<wbr>devel</a><br>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://supercolony.gluster.<wbr>org/mailman/listinfo/gluster-<wbr>devel</a><br>
<br></blockquote></div><br></div>