<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Mon, Jul 27, 2015 at 5:03 PM Ryan Clough <<a href="mailto:ryan.clough@dsic.com">ryan.clough@dsic.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><div class="gmail_default" style="font-family:monospace,monospace">Hello,<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">I have cross-posted this question in the bareos-users mailing list.<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Wondering
if anyone has tried this because I am unable to backup data that is
mounted via Gluster Fuse or Gluster NFS. Basically, I have the Gluster
volume mounted on the Bareos Director which also has the tape changer
attached.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">Here is some information about versions:<br></div><div class="gmail_default" style="font-family:monospace,monospace">Bareos version 14.2.2<br></div><div class="gmail_default" style="font-family:monospace,monospace">Gluster version 3.7.2<br></div><div class="gmail_default" style="font-family:monospace,monospace">Scientific Linux version 6.6<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">Our Gluster volume consists of two nodes in distribute only. Here is the configuration of our volume:<br>[root@hgluster02 ~]# gluster volume info<br> <br>Volume Name: export_volume<br>Type: Distribute<br>Volume ID: c74cc970-31e2-4924-a244-4c70d958dadb<br>Status: Started<br>Number of Bricks: 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: hgluster01:/gluster_data<br>Brick2: hgluster02:/gluster_data<br>Options Reconfigured:<br>performance.io-thread-count: 24<br>server.event-threads: 20<br>client.event-threads: 4<br>performance.readdir-ahead: on<br>features.inode-quota: on<br>features.quota: on<br>nfs.disable: off<br>auth.allow: 192.168.10.*,10.0.10.*,10.8.0.*,10.2.0.*,10.0.60.*<br>server.allow-insecure: on<br>server.root-squash: on<br>performance.read-ahead: on<br>features.quota-deem-statfs: on<br>diagnostics.brick-log-level: WARNING<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">When
I try to backup a directory from Gluster Fuse or Gluster NFS mount and I
monitor the network communication I only see data being pulled from the
hgluster01 brick. When the job finishes Bareos thinks that it completed
without error but included in the messages for the job are lots and
lots of permission denied errors like this:<br>15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open "/export/rclough/psdv-2014-archives-2/scan_111.tar.bak": ERR=Permission denied.<br>15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open "/export/rclough/psdv-2014-archives-2/run_219.tar.bak": ERR=Permission denied.<br>15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open "/export/rclough/psdv-2014-archives-2/scan_112.tar.bak": ERR=Permission denied.<br>15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open "/export/rclough/psdv-2014-archives-2/run_220.tar.bak": ERR=Permission denied.<br>15-Jul 02:03 ripper.red.dsic.com-fd JobId 613: Cannot open "/export/rclough/psdv-2014-archives-2/scan_114.tar.bak": ERR=Permission denied.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">At
first I thought this might be a root-squash problem but, if I try to
read/copy a file using the root user from the Bareos server that is
trying to do the backup, I can read files just fine.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">When
the job finishes is reports that it finished "OK -- with warnings" but,
again the log for the job is filled with "ERR=Permission denied"
messages. In my opinion, this job did not finish OK and should be
Failed. Some of the files from the HGluster02 brick are backed up but
all of the ones with permission errors do not. When I restore the job,
all of the files with permission errors are empty.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">Has
anyone successfully used Bareos to backup data from Gluster mounts?
This is an important use case for us because this is the largest single
volume that we have to prepare large amounts of data to be archived.<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div>Thank you for your time,</div><div><div><div dir="ltr"><div><div dir="ltr"><br></div></div></div></div></div></div></blockquote><div>How did I not see this earlier? I'm seeing a very similar problem. I just posted this to the bareos-user list:</div><div><br></div><div><div>Help! I've run out of know-how while trying to fix this myself...</div><div><br></div><div>Environment: CentOS 7, x86_64</div><div>Bareos version: 14.2.2-46.1.el7 (via <a href="http://download.bareos.org/bareos/release/14.2/CentOS_7/">http://download.bareos.org/bareos/release/14.2/CentOS_7/</a> repo)</div><div>Gluster version: 3.7.3-1.el7 (via <a href="http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/epel-$releasever/$basearch/">http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/epel-$releasever/$basearch/</a> repo)</div><div><br></div><div>Symptom: Bareos attempts to mount a volume, and spits back a Permission Denied error, as though it didn't have permission to access the relevant file.</div><div><br></div><div>I've been seeing this at least since Gluster version 3.7.2, which I updated to owing to a need to expand my backend storage (and 3.7.1, which worked fine) had a bug that broke bricks while rebalancing.</div><div><br></div><div>I've verified that the bareos storage daemon is running as the bareos user, and I've also, by way of FUSE mount into the gluster volume, verified ownership of the volume:</div><div><br></div><div># ls -l Email-Incremental-0155 </div><div>-rw-r-----. 1 bareos bareos 1073728379 Jun 10 21:04 Email-Incremental-0155</div><div><br></div><div>And uid/gid, for reference:</div><div><br></div><div># ls -ln Email-Incremental-0155 </div><div>-rw-r-----. 1 997 995 1073728379 Jun 10 21:04 Email-Incremental-0155</div><div><br></div><div>And in the gluster volume, the storage owner-{uid,gid}:</div><div># gluster volume info bareos</div><div><br></div><div>Volume Name: bareos</div><div>Type: Distribute</div><div>Volume ID: f4cb7aac-3631-41cc-9afa-f182a514d116</div><div>Status: Started</div><div>Number of Bricks: 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: backup-stor-1[censored]:/var/gluster/bareos/brick-bareos</div><div>Brick2: backup-stor-2[censored]:/var/gluster/bareos/brick-bareos</div><div>Options Reconfigured:</div><div>server.allow-insecure: on</div><div>performance.readdir-ahead: off</div><div>nfs.disable: on</div><div>performance.cache-size: 128MB</div><div>performance.write-behind-window-size: 256MB</div><div>performance.cache-refresh-timeout: 10</div><div>performance.io-thread-count: 16</div><div>performance.cache-max-file-size: 4TB</div><div>performance.flush-behind: on</div><div>performance.client-io-threads: on</div><div>storage.owner-uid: 997</div><div>storage.owner-gid: 995</div><div>features.bitrot: off</div><div>features.scrub: Inactive</div><div>features.scrub-freq: daily</div><div>features.scrub-throttle: lazy</div><div><br></div><div>In this run, the storage daemon and the file daemon happen to be on the same node. Here's trace output at level 200, obtained running "tail -f *.trace" in bareos-sd's cwd:</div><div><br></div><div>==> backup-director-sd.trace <==</div><div>backup-director-sd: fd_cmds.c:219-0 <filed: append open session</div><div>backup-director-sd: fd_cmds.c:303-0 Append open session: append open session</div><div>backup-director-sd: fd_cmds.c:314-0 >filed: 3000 OK open ticket = 1</div><div>backup-director-sd: fd_cmds.c:219-0 <filed: append data 1</div><div>backup-director-sd: fd_cmds.c:265-0 Append data: append data 1</div><div>backup-director-sd: fd_cmds.c:267-0 <filed: append data 1</div><div>backup-director-sd: append.c:69-0 Start append data. res=1</div><div>backup-director-sd: acquire.c:369-0 acquire_append device is disk</div><div>backup-director-sd: acquire.c:404-0 jid=924 Do mount_next_write_vol</div><div>backup-director-sd: mount.c:71-0 Enter mount_next_volume(release=0) dev="GlusterStorage4" (gluster://backup-stor-1[censored]/bareos/bareos)</div><div>backup-director-sd: mount.c:84-0 mount_next_vol retry=0</div><div>backup-director-sd: mount.c:604-0 No swap_dev set</div><div>backup-director-sd: askdir.c:246-0 >dird CatReq Job=server2-email.2015-07-29_16.32.34_09 GetVolInfo VolName=Email-Incremental-0155 write=1</div><div>backup-director-sd: askdir.c:175-0 <dird 1000 OK VolName=Email-Incremental-0155 VolJobs=0 VolFiles=0 VolBlocks=0 VolBytes=1 VolMounts=3 VolErrors=0 VolWrites=16646 MaxVolBytes=1073741824 VolCapacityBytes=0 VolStatus=Recycle Slot=0 MaxVolJobs=0 MaxVolFiles=0 InChanger=0 VolReadTime=0 VolWriteTime=8455280 EndFile=0 EndBlock=1073728378 LabelType=0 MediaId=156 EncryptionKey= MinBlocksize=0 MaxBlocksize=0</div><div>backup-director-sd: askdir.c:211-0 do_get_volume_info return true slot=0 Volume=Email-Incremental-0155, VolminBlocksize=0 VolMaxBlocksize=0</div><div>backup-director-sd: askdir.c:213-0 setting dcr->VolMinBlocksize(0) to vol.VolMinBlocksize(0)</div><div>backup-director-sd: askdir.c:215-0 setting dcr->VolMaxBlocksize(0) to vol.VolMaxBlocksize(0)</div><div>backup-director-sd: mount.c:122-0 After find_next_append. Vol=Email-Incremental-0155 Slot=0</div><div>backup-director-sd: autochanger.c:99-0 Device "GlusterStorage4" (gluster://backup-stor-1[censored]/bareos/bareos) is not an autochanger</div><div>backup-director-sd: mount.c:144-0 autoload_dev returns 0</div><div>backup-director-sd: mount.c:175-0 want vol=Email-Incremental-0155 devvol= dev="GlusterStorage4" (gluster://backup-stor-1[censored]/bareos/bareos)</div><div>backup-director-sd: dev.c:536-0 open dev: type=5 dev_name="GlusterStorage4" (gluster://backup-stor-1[censored]/bareos/bareos) vol=Email-Incremental-0155 mode=OPEN_READ_WRITE</div><div>backup-director-sd: dev.c:540-0 call open_device mode=OPEN_READ_WRITE</div><div>backup-director-sd: dev.c:941-0 Enter mount</div><div>backup-director-sd: dev.c:610-0 open disk: mode=OPEN_READ_WRITE open(gluster://backup-stor-1[censored]/bareos/bareos/Email-Incremental-0155, 0x2, 0640)</div><div><br></div><div>==> backup-director-fd.trace <==</div><div><br></div><div>==> backup-director-sd.trace <==</div><div>backup-director-sd: dev.c:617-0 open failed: dev.c:616 Could not open: gluster://backup-stor-1[censored]/bareos/bareos/Email-Incremental-0155, ERR=Permission denied</div></div></div></div>