ShareFile PowerShell and Windows Task Scheduler for local backup

  • 1
  • Problem
  • Updated 2 months ago
We are trying to accomplish backing up our entire ShareFile structure locally to a NAS. I have this currently setup and running. Only problem is, the task seems to be sporadic on how long it runs. Physically looking over the folder structure and files, I'm seeing that it's not finishing the sync before "Task Completed" as the Task Scheduler History shows. Sometimes it'll run just for a few minutes, other times it'll run for over 24 hours.

From what I can tell, everything is configured correctly, and I've also tested small syncs with success. The task is being ran with a domain account and proper permissions to the NAS for that account. The ShareFile account used is a Super User. Just for further information, this structure is over 2TB.

The Script

Add-PSSnapIn ShareFile
$sfLogin = Get-SfClient -Name "C:\ShareFileBackup\sflogin.sfps"
New-PSDrive -Name SFBackup -PSProvider ShareFile -Root "/" -Client $sfLogin
Sync-SfItem -ShareFilePath "/" -Synchronize -Download -LocalPath "\\wattnas\ShareFile" -Recursive -Strict

The Scheduled Task







Regards,
Barrett

Photo of Barrett Cowan

Barrett Cowan

  • 2 Posts
  • 0 Reply Likes

Posted 1 year ago

  • 1
Photo of Barrett Cowan

Barrett Cowan

  • 2 Posts
  • 0 Reply Likes
Still seeing the same behavior here. Anyone have any input? I have a feeling this may be an issue with the size of the entire structure and the -Strict option possibly. Is there anyway to log, I'm not seeing any options for it?
Photo of Peter Oven

Peter Oven

  • 21 Posts
  • 7 Reply Likes
With 2 TB I am not surprised that it's not completing the sync. I have 500 GB or so and I had to break the sync into sections, so that a failure of one part does not cause the whole thing to fail. I just run this every week using a scheduled task.


# Sharefile Backup Script
# Peter Oven
# 20180404 V1.0
# 20190717 V2.0
# 20190724 V2.1
#
# developed using Powershell SDK v1.92
# Based on code found at the following web pages
# https://github.com/citrix/ShareFile-PowerShell/releases
# https://support.citrix.com/article/CTX207460
# https://www.citrix.com/blogs/2014/05/16/getting-started-with-the-powershell-sdk/
# https://www.citrix.com/blogs/2014/11/13/transitioning-from-sfcli-to-sharefile-powershell/
# https://www.vexasoft.com/blogs/powershell/7255220-powershell-tutorial-try-catch-finally-and-error-handling-in-powershell

# Directories to Back Up
$sharedFoldersLocations = "External","WES Energy and Environment"
$wesLocations = "Individual Storage","WES Business","USDA","Reference Materials","Projects","AutoCAD Resources","BoilerMACT Compliance","Miscellaneous","Contracts","Proposals","Project Cost Estimating","USDA Technical Support"

# for manual adjustments to limit the backup extent
#$sharedFoldersLocations = $null
#$wesLocations = "BoilerMACT Compliance"

# logging system https://stackoverflow.com/questions/7834656/create-log-file-in-powershell
$logfile = "C:\SF\logs\sf-Backup.log"
Function Write-Log {
[CmdletBinding()]
Param(
[Parameter(Mandatory=$False)]
[ValidateSet("INFO","WARN","ERROR","FATAL","DEBUG")]
[String]
$Level = "INFO",
[Parameter(Mandatory=$True)]
[string]
$Message,
[Parameter(Mandatory=$False)]
[string]
$logfile
)
$Stamp = (Get-Date).toString("yyyy/MM/dd HH:mm:ss")
$Line = "$Stamp $Level $Message"
If($logfile) {
Add-Content $logfile -Value $Line
}
Else {
Write-Output $Line
}
}

Function doSync {
[CmdletBinding()]
Param(
[Parameter(Mandatory=$True)]
[string]
$loc,
[Parameter(Mandatory=$True)]
[string]
$pathPrepend,
[Parameter(Mandatory=$True)]
[string]
$logfile
)
Add-PSSnapin ShareFile
# load login token
$sfc = Get-SfClient –Name "c:\SF\sflogin-poven.sfps"
# set up temporary sharefile drive
New-PSDrive -Name sfdrive -PSProvider ShareFile -Root "/" -Client $sfc
Write-Log "INFO" ("Starting "+$loc) $logfile
Try {
# Strict will delete locally any files which have been deleted on the sharefile account
#Sync-SfItem -ShareFilePath ("/Shared Folders"+$pathPrepend+$loc+"/") -Synchronize -Download -LocalPath ("c:\SF\Shared Folders"+$pathPrepend) -Recursive -Strict -ErrorAction Stop
Sync-SfItem -ShareFilePath ("/Shared Folders"+$pathPrepend+$loc+"/") -Synchronize -Download -LocalPath ("c:\SF\Shared Folders"+$pathPrepend) -Recursive -ErrorAction Stop
Write-Log "INFO" ("Backup of "+$loc+" complete") $logfile
}
Catch {
Write-Log "ERROR" ("Backup of "+$loc+" failed") $logfile
Write-Log "ERROR" $_.Exception.Message $logfile
}
Finally {
Remove-PSDrive sfdrive
Remove-PSSnapin ShareFile
}
}

foreach($location in $sharedFoldersLocations)
{
doSync $location "/" $logfile
}

foreach($location in $wesLocations)
{
doSync $location "/WES/" $logfile
}



Photo of ShareFile Admin VA

ShareFile Admin VA

  • 50 Posts
  • 10 Reply Likes
Hi Peter, thanks so much for sharing this.  I have limited (hardly any) PowerShell experience but am learning a lot thanks to all the helpful into being shared.

Can you elaborate/explain on this section of the script? I'm not how to adjust it to match my environment and what "weslocations" refers to...
# Directories to Back Up
$sharedFoldersLocations = "External","WES Energy and Environment"
$wesLocations = "Individual Storage","WES Business","USDA","Reference Materials","Projects","AutoCAD Resources","BoilerMACT Compliance","Miscellaneous","Contracts","Proposals","Project Cost Estimating","USDA Technical Support"

# for manual adjustments to limit the backup extent
#$sharedFoldersLocations = $null
#$wesLocations = "BoilerMACT Compliance"


Photo of Peter Oven

Peter Oven

  • 21 Posts
  • 7 Reply Likes
The $sharedFoldersLocations and $wesLocations are arrays which contain key directories in my shared folders, this allows me to do the backup piecemeal instead of making it do the entire Shared Folders at once. The reason for this is that it eliminates a single point of failure. From time to time there will be an error in talking to Sharefile servers for some reason, and if this happens, the worst is that for one week, just one segment of my sharefile drive will not be up to date in the backup, but all the other parts will be up to date. The logging feature in sharefile powershell API does not seem to work so I have no way to trace the error to correct it other than to retry the entire backup of the requested path. The foreach loops at the end are what step through these arrays to actually do the backing up.

You need to figure out, based on how many GB you have stored, whether you need to break yours up or if you want to do it all at once. My directory S:\Shared Folders\WES\Projects is 211 GB. That's the biggest directory I backup at once.

Here is a screenshot of my directory structure which may clarify things. I have three directories in Shared Folders, but WES is so big that I went into that another level and individually backed up all of those directories.



Finally, I wanted to comment that the -Strict argument does not work very well, it caused lots of failures in the sync so I stopped using it, that's why it's commented out in my script. Basically I am accumulating any deleted files or directories in my backup, I am not attempting to remove them right now. I had written a script to do this but it used the Citrix Files program via the S: drive, and it didn't work reliably.
Photo of ShareFile Admin VA

ShareFile Admin VA

  • 50 Posts
  • 10 Reply Likes
Peter, can't thank you enough for all the info.  This is excellent.  I have it working conceptually.  One last question (for now) - why are the following commands part of the loop? Couldn't they be run once and the other pieces run between them? Just wondering if that's by design or incidental...

Add-PSSnapin ShareFile
$sfc = Get-SfClient –Name 
New-PSDrive -Name sfdrive -PSProvider ShareFile -Root "/" -Client $sfc
Remove-PSDrive sfdrive
Remove-PSSnapin ShareFile
Photo of Peter Oven

Peter Oven

  • 21 Posts
  • 7 Reply Likes
You have a good point that these should not be needed except once at the beginning and once at the end. I intentionally put them inside the function that runs over and over, in order to fully "clean up" and "start fresh" with each directory that I am backing up. My intention was that this would add reliability to the script since if an unknown Citrix error happens during one backup, that could potentially corrupt the structures or whatever in the "sfdrive", that directory backup might fail, but then my script will close out that dfdrive and remove the snapin, and then start fresh on the next directory in the sequence with reloading the snapin and creating a new (identical) sfdrive.

If this was a script that was being called in realtime by a  user's actions, then maybe it would be worth optimizing, but the extra adding and cleanup of the snapin and sfdrive in this case is negligible and the entire process runs in the background on a computer dedicated only to this task.

I have basically no error handling from the Citrix command other than fatal errors. If Citrix developed better error handling in their snapin, or a logging feature that worked, maybe I would do it differently.
Photo of ShareFile Admin VA

ShareFile Admin VA

  • 50 Posts
  • 10 Reply Likes
Thanks again.  We figured as much but wanted to confirm.