How can I troubleshoot External Backup issues?

Question:

While configuring the External backups, I am getting an error message on the copy of test/backup files to the shared resource.

How can I troubleshoot External backup issues?  

Answer:

As a first step, we can manually mount the SMB share from the command line. This will tell us if we can mount the remote share, access it and write a test file to it.
 

Connect to the affected Appliance's CLI and follow the steps below:

1. Create a test directory where we will mount the SMB share on:

   CODE

   sudo mkdir /mnt/testdir

2. Manually mount the share using mount command:

   CODE

   sudo mount -t cifs //<server>/<path> /mnt/testdir -o username=<username>

    Where: 

   • <server> is the share SMB sever you configure in the Web Console

   • <path> is the location of the shared path

   • <username>: The user you will use to access the share  

3. The command execution will prompt for the user's password (<username>).

   • Please type it and press Enter.

If succeeded, you won't get any output. In this case, proceed with the following commands sequence:

Check if the remote share was mounted on /mnt/testdir

sudo mount

Check if you have permission to write a file on this remote share and path

sudo touch /mnt/testdir/test.test

Check if the file was created and its permissions. Should this step fail you are likely looking at a permissions issue on the server side hosting the SMB share.

sudo ls -la /mnt/testdir

Unmount the remote share

sudo umount /mnt/testdir

Verify if it's unmounted

sudo mount

Verify if the test.test file was perhaps written to the appliance's local filesystem. This can happen if the mount failed earlier.

sudo ls -la /mnt/testdir

Check for mount errors in the system logs

sudo tail -200 /var/log/messages

If an output with an error is given, please proceed with the "Issues and Solutions" section.  

Issues and solutions

There are some common issues among our customers, which prevent external backups to work. 

Below you will find the most common errors and their possible solutions:

Use of hidden path

A common error message you can get in the Web Console is the following:

"The share name field contains an invalid share name"

Make sure you are not using any invalid characters. We do not support special characters through the Web Console Graphical User Interface (GUI).

If you are trying to map a shared resource with a hidden path (that is, adding "$" at the end), you need to set this manually via CLI:

1. Save the SMB share path without $ via the Web Console GUI.  

2. Login to the Appliance via CLI and edit /var/nexthink/common/conf/backup-config.xml  

3. Then, in the <Share>YOUR_PATH</Share> section add $ to the share (YOUR_PATH$).  So, it will look like this:

   CODE

   ...  <BackupDirectory enabled="true">  ...
   ...  <Share>YOUR_PATH$</Share>  ...
   ...  </BackupDirectory> </Configuration> 

   NOTE: sometimes, you will need to do this in the <Path> section. This is for the cases where a long shared path is used. Example: //server/share/path.  

4. Save the file,

Please take note that due to the addition on a "$" character the "Copy test file to SMB share" will report a failure. However, the backup to the share should work as expected.

Permission, configuration or unsupported version issues

In this section you will find solutions proposed to the error messages listed below:

"mount error(13): Permission denied"

"mount error(95): Operation not supported"

"mount error(112): Host is down"

"mount error(13) Device or Resource Busy"

SMB version needs to be specified

In this case, most-likely the smbv1 or higher negotiation support on windows server side was removed. For this the recommendation is to add an <ExtraParameters> configuration in the primary Appliance (Portal):

Edit /var/nexthink/common/conf/backup-config.xml and add this configuration:

<?xml version="1.0"?> 
  <Configuration origin="centralized" version="1547027001">  
  <BackupDirectory enabled="true">  
  <Server>xxxxxx</Server>  
  <Share>xxxxxx</Share>  
  <Path>xxxxxx</Path>  
  <Username>xxxxxx</Username>  
  <Password>xxxxxx</Password>  
  <Domain>xxxxxx</Domain>  
  <ExtraParameters>vers=2.1</ExtraParameters>  
  <Notification>false</Notification>  
  </BackupDirectory> 
</Configuration>

If present, and other extra parameter are already there, please add the "vers" parameter right after a comma (omitting space). Example:

... <ExtraParameters>vers=2.1</ExtraParameters> ...

This will force the mount command to add SMB version 2.1 in order to be able to use it.

NOTE: Please make sure to specify the correct vers option. If you get "mount error(22): Invalid argument" error, you need to make sure you added the specific minor SMB version, even if it's a zero. Example for SMB 3.0: vers=3.0 

After applying changes, please restart the webconsole service on the appliance.

This can be done by issuing the following from the commandline

sudo systemctl restart nxconsole

Mount point is already in use

The typical error for this is “Device or Resource Busy” .

Please check the output of the command below to see what is already mounted on the mount point specified.

sudo mount

Look for an entry like this:

//SERVER/SHARE/BACKUP on /var/nexthink/console/helpers/mnt type cifs (rw ,relatime,vers=default,cache=strict,username=USER,domain=YOURDOMAIN,uid=0 ,noforceuid,gid=0,noforcegid,addr=X.X.X.X,file_mode=0755,dir_mode=0755,sof t,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo =1)

And try to unmount it:

sudo umount /var/nexthink/console/helpers/mnt

This error could also appear if for some reason the script that is responsible for taking the backup got stuck or interrupted and never unmounted the remote share. It might be necessary to terminate the script or simply reboot the appliance to clear.

Incorrect shared path permissions

Sometimes, the issue is that the permissions were not correctly set in the SMB server for the specified path.

You may need to double-check with your IT/Network Team to make sure the user you are configuring for External backups has read and write permissions to the specified destination path.

The SMB share path is composed in Windows format, that is \\server-name\shared-folder\path.

Please note that having read/write permissions up to  "\shared-folder" is not enough. You may need to double-check you have NTFS level permissions for "\path" too.

Invalid permission in mount point

The error message you can expect is the one below:

"mount error(5): Input/output error"

This error might come up when you try to mount the share in an unreadable/unwritable mount point.

To solve this, make sure the external backup mount point have the right permissions:

su - ll /var/nexthink/console/helpers/mnt

You should get the following output:

[root@nxengine 2019-01-11 11:36:12 /]$ ll /var/nexthink/console/helpers/ total 56 ... drwx--x--x 2 root root 4096 Jan 4 10:28 mnt ..

If you are mounting manually for tests, please check the directory you created is readable and writable by the user executing the mount command. You can also use sudo to execute such command with elevated privileges and make sure it will be read and written.  

Invalid configuration parameters

The error message you can expect for this kind of issue is the following:

"mount error(22): Invalid argument"

This issue is mostly related to an invalid option value added in extra parameters. Examples can be an invalid number of SMB version specified for "vers" option (e.g. vers=3 instead of vers=3.0)

Correct the issues, save changes and then try the file copy test from the Web Console again.

For any other errors, or if the solutions above did not solve your issues, please contact Nexthink Product Support team, and provide:

• Support script output file of the affected Appliance

• Details on the troubleshooting done so far.

• Any outputs/screenshots of the results obtained when running commands or trying different settings