Best way to sync file server?

  • 1
  • Question
  • Updated 6 months ago
hi everyone,
I am looking for the best way to sync a file server folder to sharefile. This folder is pretty big: 126GB, 112.000 files, 29.600 folders.
I started using the sharefile migration tool - took about 7 hours to upload all files - everything great so far.
Because a lot of files change on daily basis I need to keep the files in sync between the file server and sharefile.
I guess the perfect solution would have been the sharefile enterprise sync manager - but this is not supported anymore as far as I can see.
Next solution would be the powershell sdk - I created a script and am using the following command:

Sync-SfItem -LocalPath $LocalPath -ShareFilePath "sfDrive:" -Recursive -Strict -Synchronize -Upload

I hope the parameters are correct (I want to upload only the changed files)
Script is working - but it seems to take way too long. Last run took about 2 days! and ended with the following error:

Sync-SfItem : A system error occurred.

I have no idea what this should tell me - I know it uploaded several files, but definitly not all new files. Any way to troubleshoot this?

Is the sync-sfitem not the right tool for syncing such a big folder? Or do I need to create several sync-sfitem commands for the different subfolders, so that each command doesn't have to check that much data?

Or should I use another way? Network share to sharefile and robocopy? FTP Upload?

Hopy you guys can help me!
Photo of Arne Lammert

Arne Lammert

  • 8 Posts
  • 0 Reply Likes

Posted 7 months ago

  • 1
Photo of Luke Skrzypek

Luke Skrzypek, Employee

  • 57 Posts
  • 7 Reply Likes
Hi Arne,

Your Powershell command appears correct. Making multiple calls to Sync-SfItem for different subfolders might be ideal, as that is a large number of files the cmdlet has to check for differences.

Alternatively, you could try Drive Mapper to map your ShareFile folder as a mapped drive.
https://support.citrix.com/article/CTX207791
I'd suggest not using FTP as that will likely take longer.

I'm sorry about the unhelpful error message, if you're willing to share your ShareFile subdomain and a general time when the error happened, I can try to see if there are any errors logged on the backend.
You can do that either here, or by uploading a text file here https://skrzypek.sharefile.com/r-rad7ce689b1d45338
Photo of Luke Skrzypek

Luke Skrzypek, Employee

  • 57 Posts
  • 7 Reply Likes
For drive mapper, I think the closest thing would be to set up SAML
Windows Integrated Authentication to avoid login prompts. https://citrix.sharefile.com/share/view/sc1aee5c38ff45ab9

For Powershell, can you post more of your script? Specifically, how you're authenticating and creating the sfdrive?
Photo of Arne Lammert

Arne Lammert

  • 8 Posts
  • 0 Reply Likes
Drive mapper:
As far as I can tell the drive mapper doesn't run as a service, so it needs to have someone logged in in Windows to work, doesn't it?
Wouldn't SAML need an active windows user either to work?

Powershell:
here are the important parts of the script:
ran this one time to create spfs:
$sfClient = New-SfClient -Name ((Join-Path $env:USERPROFILE "Documents") + "\YourSubdomain.sfps") -Account <*hidden*>
- script commands to run everytime:
$sfClient = Get-SfClient -Name ((Join-Path $env:USERPROFILE "Documents") + "\YourSubdomain.sfps")
New-PSDrive -Name sfDrive -PSProvider ShareFile -Client $sfClient -Root $ShareFileFolder
Sync-SfItem -LocalPath $LocalPath -ShareFilePath "sfDrive:" -Recursive -Strict -Synchronize -Upload
(Edited)
Photo of Luke Skrzypek

Luke Skrzypek, Employee

  • 57 Posts
  • 7 Reply Likes
Drive mapper:
You're right that it doesn't run as a service. However, your backup script/task could programmaticly launch the DriveMapper executable to mount the drive. I was thinking that if your tasks runs as a SAML user it could skip the authentication prompt, although I haven't tried it personally.

Powershell:
Could you share what user you're authenticating as? Same link as before will work if you want to keep it private.
 
Photo of Arne Lammert

Arne Lammert

  • 8 Posts
  • 0 Reply Likes
Drive Mapper:
Ah ok, I will try that, thanks!

Powershell:
I uploaded the user, hope thats what you meant
Photo of Luke Skrzypek

Luke Skrzypek, Employee

  • 57 Posts
  • 7 Reply Likes
Hi Arne,

I checked and we had some networking issues yesterday that coincide with the times when you saw the errors. They have since been resolved.
Photo of Zack Eleveld

Zack Eleveld

  • 21 Posts
  • 4 Reply Likes
I am assuming that you are already using powershell to filter out files based on their change dates.  Is this a one way sync or a upload/download scenario.  if you are not testing locally before syncing, the longest part of your Sync-sfitem command is probably testing to see if the files are the same or different.
Photo of Arne Lammert

Arne Lammert

  • 8 Posts
  • 0 Reply Likes
Hi Zack,
its a one way sync from fileserver to sharefile. I tested it now several times - with sync sfitem it takes 2 days to sync all folders. (I already split this process into several smaller syncs)
What exactly do you mean with filtering files out via powershell based on the change dates?
Photo of Zack Eleveld

Zack Eleveld

  • 21 Posts
  • 4 Reply Likes
The file system knows which files have been updated.  Powershell is a very powerful manipulator of filesystem objects.  So, you can ask powershell to give you a list of all files that have been updated within X amount of time.  Then, only attempt to sync items that have been updated.

For us, it looks something like this:
$backTime = new-timespan -Days $backDays -Hours $backHours -Minutes 0
$searchTime = (get-date) - $backTime  # time used in looking back at the files updated within $backtime.  Lock it in at the start
        #get list of local files, oldest first
$locFiles = get-childitem $searchDir -recurse -exclude *.url, *.lnk, *.sfprime, *.msg, *.eml | Where {$_.LastWriteTime -gt $searchTime}| Where-object {$_.mode -notmatch "d"} | Sort lastwritetime
write-host $locFiles.count.toString() "updated within" $backTime.totalHours.toString("0.00") "hours."

You have to also look for new directories if new directories can be created too.  To do that, we simply get two filesystem objects, one on the SF side and one from the Windows side and do a compare:  (this example is highly dependent on our file structure.  Though, it's possible to make it generic, this is just to give you an idea.  It's also possible to just use the above and determine if the directory the new/updated file is in doesn't exist in SF, sync the directory instead of the file.)
	# collect array of directories in the lcoal file that are case file level dirs
$locCL = get-ChildItem ($baseDir+$myDirSrch+"*")  | Where-object {$_.mode -match "d"}  | Select-Object name, Parent
# Collect list of SF directories (this is within a loop) $dirStr = $baseDir + $curSub
$sfSeek = $dirStr.replace($baseDir,$sfBaseDir).replace("\","/")
$TEMPsfCL = get-childitem sfdrive:$sfSeek | select filename
if ($sfCL.count -eq 0 ) {$sfCL = $TEMPsfCL}
#echo $tempsfcl
elseif ([Array]$TEMPsfCL.count -gt 0 -or [bool]$TEMPsfCL) {[Array]$sfCL += $TEMPsfCL}
# Compare the two arrays of files to produce a sync list
$locNotSF = compare-object -referenceobject $locCL  -differenceobject $sfCL -PassThru -Property FileName | where { $_.sideIndicator -eq "<="}
foreach ($fName in $locNotSF) {
$lFile = $fName.Name
$lBpath = $myDirSrch.Replace("*",$fName.Parent.Name)
# write-host LF $lFile
$lSrc = $baseDir + $lBpath + $lFile
		$SFtoLoc = $sfBaseDir + $lBpath.Replace("\","/")
		# double check that case file isn't there
		$SFdblChk = $lSrc.Replace($baseDir,$sfBaseDir).replace("\","/") -replace "\d{3}-\d{2}-"
		$sfFil = Get-ChildItem sfdrive:$SFdblChk -ErrorAction Ignore
		if (-not [bool]$sfFil) {
			write-host "Syncing: " $lFile
			# sync the folder
			Sync-SfItem -LocalPath $lSrc -Upload -Synchronize -ShareFilePath $SFtoLoc -Recursive
			# increment Counter
			$NSc ++
		} # only sync on double check not there
	} # end foreach item to sync
It's not that hard if you have any Powershell experience and it works better than trying to sync directories or use any of the SF tools, especially if you have a relatively small amount of files that are changed every day.
Photo of Arne Lammert

Arne Lammert

  • 8 Posts
  • 0 Reply Likes
Wow that looks really nice! I will dig into that on monday and report back. Thanks a lot! This should really help