<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 13, 2016 at 3:49 PM, Niels de Vos <span dir="ltr">&lt;<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Wed, Jan 13, 2016 at 10:35:42AM +0100, Xavier Hernandez wrote:<br>
&gt; The same has happened to <a href="http://slave34.cloud.gluster.org" rel="noreferrer" target="_blank">slave34.cloud.gluster.org</a>. I&#39;ve disabled it to<br>
&gt; allow regressions to be run on other slaves.<br>
&gt;<br>
&gt; There are two files owned by root inside<br>
&gt; /home/jenkins/root/workspace/rackspace-regression-2GB-triggered:<br>
&gt;<br>
&gt; -rwxr-xr-x  1 root    root     10124 Jan  7 17:54 file_lock<br>
&gt; drwxr-xr-x  3 root    root      4096 Jan  7 18:31 <a href="http://slave34.cloud.gluster.org" rel="noreferrer" target="_blank">slave34.cloud.gluster.org</a>:<br>
<br>
</span>Thanks!<br>
<br>
I&#39;ve looked into this a little more now, and might have identified the<br>
problem.<br>
<br>
This one failed with an unrelated error:<br>
<br>
  <a href="https://build.gluster.org/job/rackspace-regression-2GB-triggered/17413/console" rel="noreferrer" target="_blank">https://build.gluster.org/job/rackspace-regression-2GB-triggered/17413/console</a><br>
<br>
  ...<br>
  Building remotely on <a href="http://slave34.cloud.gluster.org" rel="noreferrer" target="_blank">slave34.cloud.gluster.org</a> (rackspace_regression_2gb) in workspace /home/jenkins/root/workspace/rackspace-regression-2GB-triggered<br>
   &gt; git rev-parse --is-inside-work-tree # timeout=10<br>
  Fetching changes from the remote Git repository<br>
   &gt; git config remote.origin.url git://<a href="http://review.gluster.org/glusterfs.git" rel="noreferrer" target="_blank">review.gluster.org/glusterfs.git</a> # timeout=10<br>
  Fetching upstream changes from git://<a href="http://review.gluster.org/glusterfs.git" rel="noreferrer" target="_blank">review.gluster.org/glusterfs.git</a><br>
  ...<br>
<br>
The next run on slave34 failed because of the weird directory:<br>
<br>
  <a href="https://build.gluster.org/job/rackspace-regression-2GB-triggered/17440/console" rel="noreferrer" target="_blank">https://build.gluster.org/job/rackspace-regression-2GB-triggered/17440/console</a><br>
<br>
  ...<br>
  Building remotely on <a href="http://slave34.cloud.gluster.org" rel="noreferrer" target="_blank">slave34.cloud.gluster.org</a> (rackspace_regression_2gb) in workspace /home/jenkins/root/workspace/rackspace-regression-2GB-triggered<br>
  Wiping out workspace first.<br>
  java.io.IOException: remote file operation failed: /home/jenkins/root/workspace/rackspace-regression-2GB-triggered at hudson.remoting.Channel@62ecdacb:<a href="http://slave34.cloud.gluster.org" rel="noreferrer" target="_blank">slave34.cloud.gluster.org</a>:<br>
  ...<br>
<br>
Note the &quot;Wiping out workspace first.&quot; line. This comes from an option<br>
in the regression job. This seems to be a recently added &quot;Additional<br>
Behaviour&quot; in the Jenkins job. Did anyone add this on purpose, or was<br>
that automatically done with a Jenkins update or something?<br></blockquote><div><br></div><div>The three additional behaviour added in regression configuration were added by me and I simply copied</div><div>whatever was there in smoke configuration page. We can try removing this configuration line but the tests weren&#39;t getting started without this(we got it running after a restart so this might be false requirement).</div><div><br></div><div>Technically, it is not a harmful configuration and wiki pages recommend it. It says such errors occur only if files were created and left open/locked by some tests or were created with different permissions. We still need to identify what tests are responsible for this.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="HOEnZb"><font color="#888888"><br>
Niels<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
&gt;<br>
&gt; Xavi<br>
&gt;<br>
&gt; On 12/01/16 12:06, Niels de Vos wrote:<br>
&gt; &gt;Hi,<br>
&gt; &gt;<br>
&gt; &gt;I&#39;ve disabled <a href="http://slave32.cloud.gluster.org" rel="noreferrer" target="_blank">slave32.cloud.gluster.org</a> because it failed multiple<br>
&gt; &gt;regression tests with a weird error. After disabling slave32 and<br>
&gt; &gt;retriggering the failed run, the same job executed fine on a different<br>
&gt; &gt;slave.<br>
&gt; &gt;<br>
&gt; &gt;The affected directory is owned by root, so the jenkins user is not<br>
&gt; &gt;allowed to wipe it. Does anyone know how this could happen? The dirname<br>
&gt; &gt;is rather awkward too...<br>
&gt; &gt;<br>
&gt; &gt;   /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/slave32.cloud.gluster.org:/d<br>
&gt; &gt;<br>
&gt; &gt;I think we can just remove that dir and the slave can be enabled again.<br>
&gt; &gt;Leaving the status as is for further investigation.<br>
&gt; &gt;<br>
&gt; &gt;Thanks,<br>
&gt; &gt;Niels<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;Full error:<br>
&gt; &gt;<br>
&gt; &gt;     Wiping out workspace first.<br>
&gt; &gt;     java.io.IOException: remote file operation failed: /home/jenkins/root/workspace/rackspace-regression-2GB-triggered at hudson.remoting.Channel@7bc1e07d:<a href="http://slave32.cloud.gluster.org" rel="noreferrer" target="_blank">slave32.cloud.gluster.org</a>: java.nio.file.AccessDeniedException: /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/slave32.cloud.gluster.org:/d<br>
&gt; &gt;             at hudson.FilePath.act(FilePath.java:986)<br>
&gt; &gt;             at hudson.FilePath.act(FilePath.java:968)<br>
&gt; &gt;             at hudson.FilePath.deleteContents(FilePath.java:1183)<br>
&gt; &gt;             at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:28)<br>
&gt; &gt;             at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1040)<br>
&gt; &gt;             at hudson.scm.SCM.checkout(SCM.java:485)<br>
&gt; &gt;             at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)<br>
&gt; &gt;             at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)<br>
&gt; &gt;             at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)<br>
&gt; &gt;             at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)<br>
&gt; &gt;             at hudson.model.Run.execute(Run.java:1738)<br>
&gt; &gt;             at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)<br>
&gt; &gt;             at hudson.model.ResourceController.execute(ResourceController.java:98)<br>
&gt; &gt;             at hudson.model.Executor.run(Executor.java:410)<br>
&gt; &gt;     Caused by: java.nio.file.AccessDeniedException: /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/slave32.cloud.gluster.org:/d<br>
&gt; &gt;             at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)<br>
&gt; &gt;             at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)<br>
&gt; &gt;             at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)<br>
&gt; &gt;             at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)<br>
&gt; &gt;             at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)<br>
&gt; &gt;             at java.nio.file.Files.delete(Files.java:1079)<br>
&gt; &gt;             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br>
&gt; &gt;             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)<br>
&gt; &gt;             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
&gt; &gt;             at java.lang.reflect.Method.invoke(Method.java:606)<br>
&gt; &gt;             at hudson.Util.deleteFile(Util.java:255)<br>
&gt; &gt;             at hudson.FilePath.deleteRecursive(FilePath.java:1203)<br>
&gt; &gt;             at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)<br>
&gt; &gt;             at hudson.FilePath.deleteRecursive(FilePath.java:1194)<br>
&gt; &gt;             at hudson.FilePath.deleteContentsRecursive(FilePath.java:1212)<br>
&gt; &gt;             at hudson.FilePath.access$1100(FilePath.java:190)<br>
&gt; &gt;             at hudson.FilePath$15.invoke(FilePath.java:1186)<br>
&gt; &gt;             at hudson.FilePath$15.invoke(FilePath.java:1183)<br>
&gt; &gt;             at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719)<br>
&gt; &gt;             at hudson.remoting.UserRequest.perform(UserRequest.java:120)<br>
&gt; &gt;             at hudson.remoting.UserRequest.perform(UserRequest.java:48)<br>
&gt; &gt;             at hudson.remoting.Request$2.run(Request.java:326)<br>
&gt; &gt;             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)<br>
&gt; &gt;             at java.util.concurrent.FutureTask.run(FutureTask.java:262)<br>
&gt; &gt;             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
&gt; &gt;             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
&gt; &gt;             at java.lang.Thread.run(Thread.java:745)<br>
&gt; &gt;             at ......remote call to <a href="http://slave32.cloud.gluster.org" rel="noreferrer" target="_blank">slave32.cloud.gluster.org</a>(Native Method)<br>
&gt; &gt;             at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)<br>
&gt; &gt;             at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)<br>
&gt; &gt;             at hudson.remoting.Channel.call(Channel.java:781)<br>
&gt; &gt;             at hudson.FilePath.act(FilePath.java:979)<br>
&gt; &gt;             ... 13 more<br>
&gt; &gt;     Finished: FAILURE<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;_______________________________________________<br>
&gt; &gt;Gluster-infra mailing list<br>
&gt; &gt;<a href="mailto:Gluster-infra@gluster.org">Gluster-infra@gluster.org</a><br>
&gt; &gt;<a href="http://www.gluster.org/mailman/listinfo/gluster-infra" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-infra</a><br>
&gt; &gt;<br>
</div></div></blockquote></div><br></div></div>