<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"M S Vishwanath Bhat" <msvbhat@gmail.com><br><b>To: </b>"Niels de Vos" <ndevos@redhat.com><br><b>Cc: </b>"Raghavendra Talur" <rtalur@redhat.com>, "Jonathan Holloway" <jholloway@redhat.com>, "Gluster Devel" <gluster-devel@gluster.org>, "Kaushal M" <kshlmster@gmail.com><br><b>Sent: </b>Wednesday, March 30, 2016 7:05:52 AM<br><b>Subject: </b>Re: [Gluster-devel] Location of distaf tests<br><div><br></div><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On 24 March 2016 at 15:00, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">On Thu, Mar 24, 2016 at 12:05:29PM +0530, Raghavendra Talur wrote:<br>
> On Wed, Mar 23, 2016 at 9:56 PM, M S Vishwanath Bhat <<a href="mailto:msvbhat@gmail.com" target="_blank">msvbhat@gmail.com</a>><br>
> wrote:<br>
><br>
> ><br>
> ><br>
> > On 18 March 2016 at 02:47, Jonathan Holloway <<a href="mailto:jholloway@redhat.com" target="_blank">jholloway@redhat.com</a>> wrote:<br>
> ><br>
> >><br>
> >><br>
</span>> >> ------------------------------<br>
> >><br>
> >> *From: *"M S Vishwanath Bhat" <<a href="mailto:msvbhat@gmail.com" target="_blank">msvbhat@gmail.com</a>><br>
> >> *To: *"Niels de Vos" <<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>><br>
> >> *Cc: *"Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> >> *Sent: *Thursday, March 17, 2016 8:18:23 AM<br>
> >> *Subject: *Re: [Gluster-devel] Location of distaf tests<br><div><div class="h5">> >><br>
> >><br>
> >><br>
> >><br>
> >> On 17 March 2016 at 10:50, Niels de Vos <<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>> wrote:<br>
> >><br>
> >>> On Wed, Mar 09, 2016 at 08:26:44PM +0530, M S Vishwanath Bhat wrote:<br>
> >>> > On 9 March 2016 at 19:39, Kaushal M <<a href="mailto:kshlmster@gmail.com" target="_blank">kshlmster@gmail.com</a>> wrote:<br>
> >>> ><br>
> >>> > > On Wed, Mar 9, 2016 at 7:02 PM, M S Vishwanath Bhat <<br>
> >>> <a href="mailto:msvbhat@gmail.com" target="_blank">msvbhat@gmail.com</a>><br>
> >>> > > wrote:<br>
> >>> > > > Hi,<br>
> >>> > > ><br>
> >>> > > > When we were discussing about the readiness of distaf for upstream<br>
> >>> test<br>
> >>> > > > automation, this question came up, That we should have a process or<br>
> >>> > > workflow<br>
> >>> > > > for proposing, reviewing and including the tests somewhere.<br>
> >>> > > ><br>
> >>> > > > Right now the tests are part of distaf repository<br>
> >>> > > > (<a href="http://github.com/gluster/distaf" rel="noreferrer" target="_blank">github.com/gluster/distaf</a>) itself. And contributing to distaf is<br>
> >>> by<br>
> >>> > > sending<br>
> >>> > > > a PR. But we want this to be included in gerrit so that review and<br>
> >>> > > > contributing process becomes much easier. But the question still<br>
> >>> > > remains...<br>
> >>> > > > where? Right now I can think of below options.<br>
> >>> > > ><br>
> >>> > > > * Use the same distaf repo in github for tests as well.<br>
> >>> > > > * Create a separate repo distaf_gluster_tests (or something<br>
> >>> similar) and<br>
> >>> > > > have all the tests there.<br>
> >>> > > > * Or have a tests/distaf/ directory inside glusterfs repository.<br>
> >>> And this<br>
> >>> > > > tests can be bundled in a rpm and distributed. This directory will<br>
> >>> have<br>
> >>> > > both<br>
> >>> > > > the test cases and related library functions.<br>
> >>> > ><br>
> >>> > > I prefer this approach. It makes it easier for developers to submit<br>
> >>> > > tests along with their changes, as is the case with our regression<br>
> >>> > > tests now.<br>
> >>> > ><br>
> >>> > > By library functions, I'm assuming you mean helper libraries related<br>
> >>> > > to gluster, which will be used in the tests which will be written.<br>
> >>> > ><br>
> >>> ><br>
> >>> > Yes, I mean helper functions which are related to gluster. The<br>
> >>> framework<br>
> >>> > itself will be made a python package. At least that's the plan.<br>
> >>><br>
> >>> +1 for the tests in tests/distaf/ . I think I would like the libs to be<br>
> >>> part of the main distaf project. That would make it easier to run distaf<br>
> >>> for other projects that integrate with Gluster (like Samba,<br>
> >>> NFS-Ganesha and the like). The upstream projects can that use distaf for<br>
> >>> their integration with Gluster if they choose so.<br>
> >>><br>
> >><br>
> >> My concern with having libs in main distaf project is that, libs and test<br>
> >> cases are very much tied together. So when someone writes a testcase, they<br>
> >> write and/or modify the related gluster libs along with it. And the one<br>
> >> needs to go back and forth between the testcase and gluster libs while<br>
> >> writing and debugging the case. If they are in separate repo/package, it<br>
> >> makes it bit difficult (not impossible) for test case writer.<br>
> >><br>
> >> If everyone is of the same opinion that gluster libs should be part of<br>
> >> main distaf repo, I can still do it when I package it next week.<br>
> >><br>
> >><br>
> >> I'm now thinking we should go ahead and keep it three distinct projects:<br>
> >> - Gluster: testcase--and supporting functions that are not a good fit for<br>
> >> the libs<br>
> >> - distaf-libs-gluster: core libarary and utility functions<br>
> >> - distaf: framework only<br>
> >><br>
> ><br>
> > But that (having more than one repo) kind of defeats the purpose of<br>
> > developers having to concern themselves with only single repo<br>
> > (glusterfs.git). Since I couldn't discuss this in weekly gluster meeting in<br>
> > #gluster-meeting, let me propose my solution here.<br>
> ><br>
><br>
> This proposal from Jonathan of 3 repos is what I like the best for long<br>
> term.<br><br></div></div>Same here, although having distaf-libs-gluster in the man distaf<br>
framework might be good too. Many frameworks also provide plugins of<br>
some kind, I see the libs as something like a plugin.<br><br>
In the end, I do not really care very much. Just check with whoever is<br>
writing test-cases that need library extensions. Hopefully the library<br>
changes are rare (on future) and new test-cases do not need any<br>
framework/libs modifications. Once the libs are (more) stable, maybe<br>
move them to the main distaf repo.<br></blockquote><div><br></div><div>Okay, We had an internal meeting to finalise on this. Hope it is Okay.</div><div><br></div><div>So as of now, we'll have both distaf test cases and libraries together. This will ease the process of sending test cases for all people involved.</div><div><br></div><div>I have sent a initial patch for that <a href="http://review.gluster.org/#/c/13853/" target="_blank">http://review.gluster.org/#/c/13853/</a>. Please take some time to review it. Since I will be on leave for next two weeks until 15th April, someone please address any comments that is given to the patch.</div></div></div></div></blockquote><div><br>I'm getting a 500 Server Error when trying to access upstream gerrit under my id at the moment, so I'll comment on the patch at next opportunity.<br></div><div><br></div><div>Anyone know of any snags we might encounter with namespace_packages?<br></div><div>Since we're going ahead and housing distaf-libs in the glusterfs repo now with the idea of moving to a separate distaflibs repo later,<br>we could leverage namespace_packages to have all libraries fall under a single installed namespace while development stays completely separate.<br>The distaf_libs tree under the glusterfs repo could be something like the following:<br><br>tests/distaf/distaflibs-gluster/ <--- root dir for python config files, docs, etc.<br>tests/distaf/distaflibs-gluster/setup.py<br>tests/distaf/distaflibs-gluster/distaflibs/__init__.py <--- namespace_package placeholder<br>tests/distaf/distaflibs-gluster/distaflibs/gluster <--- contains the actual package and module files<br>tests/distaf/distaflibs-gluster/distaflibs/gluster/__init__.py<br>tests/distaf/distaflibs-gluster/distaflibs/gluster/brick_ops.py<br>...<br>tests/distaf/distaflibs_gluster/distaflibs/gluster/volume_ops.py<br><br>This would install to /usr/lib/python2.7/site-packages/distaflibs/gluster<br>and import via "from distaflibs.gluster import volume_ops", etc.<br><br>If I write a separate library, for say lvm, under my own github, etc., it would be setup similar to...<br><wherever>/distaflibs-lvm/<br><wherever>/distaflibs-lvm/setup.py<br><wherever>/distaflibs-lvm/distaflibs/__init__.py<br><wherever>/distaflibs-lvm/distaflibs/lvm<br><wherever>/distaflibs-lvm/distaflibs/lvm/__init__.py<br><wherever>/distaflibs-lvm/distaflibs/lvm/pv.py<br><br>Like distaflibs-gluster, it would also be installed under /usr/lib/python2.7/distaflibs<br>and import via "from distaflibs.lvm import pv", etc.<br><br>The directory structure seems a little redundant, but allows for easy separation of package configuration while installing into the same package tree.<br>When we are ready to move out of glusterfs repo, it's just a matter of moving the distaflibs-gluster directory somewhere else and all imports will remain the same.</div><div>If down the road we decide to host other common/core distaflibs packages, the directory structure makes it simple to copy them in with no changes.<br><br>Sphinx uses something similar for sphinxcontrib and zope uses namespace_packages pretty heavily.<br><br>Cheers,<br>Jonathan<br><br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>So after 3.8, when we have some tests already in repo, we can analyse the robustness of the libs and move out the related libs to different repo. But we can take that call later and for now, at least for ease of use, lets have both together.</div><div><br></div><div>Best Regards,</div><div>Vishwanath</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><br>
> > Library functions and test cases both go into a subdirectory inside<br>
> > glusterfs.git. This helps for devs and people writing test case for<br>
> > gluster. And we find a way to sync the libraries to distaf.git, so that<br>
> > each time a package is made, it is available for integrating projects to<br>
> > make use of. I know this is not very elegant way of dealing with the<br>
> > problem, but I couldn't think of anything which would fit all requirements.<br>
> ><br>
><br>
> If you think distaf-libs-gluster can have more improvements and it makes<br>
> sense to have them with tests for now; we can have 2 repos temporarily<br>
> 1. Gluster git has tests + distaf-libs-gluster ( We can even avoid<br>
> packaging these two for now. Just let them be part of Gluster repo. )<br>
> 2. distaf repo has distaf framework only<br>
<br>
</span>I really would like to see the tests (and distaf-libs) packaged too.<br>
This makes it much easier for writing and running test cases. All the<br>
tests that we do in the CentOS CI should only install RPMs, that makes<br>
it very simple to run the same tests on different environments.<br><br>
At the moment we package the tests/ directory for non-official builds<br>
already. We do not want to provide the glusterfs-regression-tests RPM in<br>
repositories for users where they can install the package and easily<br>
break their existing deployment. But installing the package on a<br>
test-machine with the matching binary RPMs makes testing pretty smooth.<br><span class=""><span style="color: #888888;" data-mce-style="color: #888888;" color="#888888"><br>
Niels<br>
</span></span><div class=""><div class="h5"><br>
><br>
><br>
> ><br>
> > Jonathan, Niels, Kaushal, Raghavendra Talur - Your thoughts?<br>
> ><br>
> > Best Regards,<br>
> > Vishwanath<br>
> ><br>
> ><br>
> >> Here's some of my reasoning...<br>
> >> - We're already talking about not putting the libs in the Gluster repo as<br>
> >> an option, so from a convenience perspective it's not as relevant which<br>
> >> repo is being used (distaf or distaf-gluster-libs).<br>
> >> - And if we're already talking about the possibility of a refactor to<br>
> >> split the libs out down the road, it's not much different to talk refactor<br>
> >> to roll the libs into the distaf project should they become a problem to<br>
> >> maintain.<br>
> >> - With that said, if looking to have distaf adopted by a wider<br>
> >> non-Gluster audience, we'll want to split the libs out down the road<br>
> >> anyway. It would also lay the foundation for others<br>
> >> (distaf-libs-<your-project-here>).<br>
> >> - The Maintainer/Reviewer/Contributor demographic isn't necessarily the<br>
> >> same across all three and the Maintainer/Reviewer will most likely play a<br>
> >> larger role initially as we build out the libs and figure out what goes<br>
> >> where, what works, etc.<br>
> >> I'm just thinking it would be easier to manage that separately in<br>
> >> the beginning.<br>
> >> - If we are including the libs in the distaf project, how will the libs<br>
> >> be consumed? Roll them in the distaf package? Separate package? Git clone<br>
> >> and setup.py?<br>
> >> I like the concept of separate distaf and distaf-libs-gluster<br>
> >> packages that can be versioned and maintained separately. It keeps them<br>
> >> flexible and clean/clear of each other, and any framework changes that are<br>
> >> required by a library change can be managed through the packaging.<br>
> >> We can still do that if the libs are in the same repo, but in my<br>
> >> opinion it's not as clean and the value add in having them combined is<br>
> >> minimal.<br>
> >> - It shouldn't take that long to create the new project and may save us<br>
> >> some inevitable headache down the road.<br>
> >><br>
> >> Either way, I agree with Niels that the sooner the better--so if a<br>
> >> separate repo for distaf-libs-gluster would create an unacceptable delay,<br>
> >> rolling libs into the distaf project works for me.<br>
> >><br>
> >> Cheers,<br>
> >> Jonathan<br>
> >><br>
> >> Best Regards,<br>
> >> Vishwanath<br>
> >><br>
> >>><br>
> >>> > > I'm also in favor of including them here as well. This will help keep<br>
> >>> > > DiSTAF free of an Gluster specific cruft and allow it to be<br>
> >>> (possibly)<br>
> >>> > > reusable by others.<br>
> >>> > ><br>
> >>> ><br>
> >>> > The recent changes makes it specific to gluster, but very easy to make<br>
> >>> it<br>
> >>> > generic.<br>
> >>><br>
> >>> Starting Gluster specific is fine, later on when things are in use and<br>
> >>> working the way we need it should be trivial to make it more generic.<br>
> >>> Just keep a generic way for adding new functionalities for now, and file<br>
> >>> issues in GitHub for the parts that need to get refactored.<br>
> >>><br>
> >>> The sooner we can start using distaf and gain (more) real world<br>
> >>> experience the better.<br>
> >>><br>
> >>> Thanks,<br>
> >>> Niels<br>
> >>><br>
> >>><br>
> >>> > Best Regards,<br>
> >>> > Vishwanath<br>
> >>> ><br>
> >>> ><br>
> >>> > ><br>
> >>> > > ><br>
> >>> > > > Please let us know what your preferred option is. If you have any<br>
> >>> other<br>
> >>> > > > ideas, please let us know as well.<br>
> >>> > > ><br>
> >>> > > > Best Regards,<br>
> >>> > > > Vishwanath<br>
> >>> > > ><br>
> >>> > > ><br>
> >>> > > > _______________________________________________<br>
> >>> > > > Gluster-devel mailing list<br>
> >>> > > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> >>> > > > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> >>> > ><br>
> >>><br>
> >>> > _______________________________________________<br>
> >>> > Gluster-devel mailing list<br>
> >>> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> >>> > <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> >>><br>
> >>><br>
> >><br>
> >> _______________________________________________<br>
> >> Gluster-devel mailing list<br>
> >> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> >> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
> >><br>
> >><br>
> >><br>
> ><br></div></div></blockquote></div><br></div></div></blockquote><div><br></div></div></body></html>