FreeFileSync FFS very slow. Other ways to copy 1.2TB of files to external HDD? [SOLVED]

Help Desk General Help
1

I’m using Ubuntu 22.04 on a laptop. Earlier today I started a FFS mirror job of my 1.2TB user files to an external USB-C connected spinning hard drive. FFS says the job will take 30 days…!!! Also, while FFS was running, I was unable to start any other apps on my 40GB-RAM Ryzen 7 laptop. So naturally I stopped the FFS job. My main drive is a laptop internal 2TB nvme.

Is there a faster way to copy files to an external USB-C connected spinning hard drive? Should I use an USB-C connected external nvme drive instead? (I’ll buy one if needed to get decent copy times.)

I’m thinking if I find a a faster way to copy files to the external drive, I’ll use FFS to sync future jobs (which will be small and go very fast).

1 Like
2

Glad I ran into this, I have almost 40 external drives I use for Cold storage, Backups and migrations. There’s a couple questions I have :

  1. What File system is on the External USB-C drive? XFS, ext4, Bcache or other?
  2. How many other devices were connected to the USB ports, Although AMD uses PCIe and good bandwidth for their devices, It’s good to know in case there is some sort of bottle neck.
  3. While the device is a USB-C HDD, do you happen to know the buffer size for the HDD? Good devices will have more than 64MB cache.
  4. I use rsync to do my transfers, Is there a way for you to see how much data is being transferred to the USB-C device? How much bandwidth in total?

Most devices (HDDs) start out well for the first 2-4Gbp/s, then absolutely TANK, going down to 10Mb/s for the duration. If you are considering a NVMe external device that can be potentially better, just know the limitations of USB-C, the NVMe and other components involved. i can post recommendations if needed.

1 Like
3

[[ Hopefully the changes I made will lead to a fast and successful backup. I’ll report back w results. Thanks! ]]

1 Like
4

I’ve never used Grsync, but rsync does have all the tools necessary for you to see the transfer speeds and relevant information. rsync can do so much more too ! file checksum, etc.

rsync -ahP the P flag is for Progress here is a screenshot of what a typical transfer looks like for me :

2023-12-31_07-41-50
2023-12-31_07-41-501185×327 21.7 KB

if things get too bogged down you can limit the bandwidth of the transfer potentially smoothing the transfer of files, or if you are network contrained with --bwlimit=XM
where X is the amount in Megabytes.

1 Like
5

Cool. Rsync looks good but my old brain is unable to wrap itself around a CLI app and all the various commands. Grsync seems to be working good.

I found “Mission Center”, a nice GUI app that displays the disk data transfer speeds. Perfect for my needs.

6

I’ll download Grsync for a minute and see if it has any tools useful for you.

So this turned into a minor rabbit hole for me, because over 3yrs ago, I packaged grsync into a flatpak but did not publish it on Flathub. The process was different back then I recall, and I didn’t want to be responsible for release.

Grsync is still not on flathub, and does not appear to be going any time soon either.
Also, the app still looks dated and intuitive IMO.

Here are some screenshots :

2023-12-31_09-34-11
2023-12-31_09-34-111198×535 43.4 KB

2023-12-31_09-36-22
2023-12-31_09-36-221297×581 78.9 KB

So hovering over the names, check box actually gives you the rsync flag it corresponds too ! :laughing:

Either way, Check “Show transfer progress” to get a idea of what transfer speeds you are getting.

1 Like
7

I got Grsync using apt.

I looked into Grsync some more and I didn’t see a way to sync more than one directory at a time. I googled this question and found how to do it in Rsync, but not in Grsync. So I switched back to FreeFileSync. I now have my external HDD drive directly connected to my laptop rather than going thru my USB hub. Switching to the direct connection causes FFS to speed up a lot. The backup to my external HDD will now take only 15hrs rather than 30 days. Which is OK cuz I only do a full backup once and then future syncs only take a few seconds.

Also I really like the FFS GUI, it’s great at showing me whats going on and exactly what I’ll be syncing. Much more intuitive than Grsync (at least to my eyes).

The MissionCenter app is great at displaying real time data transfer speeds.

8

Whoa ! I see a lot of Sync and Backup being thrown around there. Those two concepts are often misunderstood and interchanged, and they are definitely NOT the same. I’m not sure I’ll go into what that all means here, as you seem to be getting the reulst you want. But if you need any help, just reply !

9

My overall data integrity tasks are the following:

(note: my system drive has both /root and /home, i.e. everything i.e. all my data)

  • Every other day I do a full Clonezilla clone of my system drive to 2 different external nvme drives. So I always have 2 recent clones of my whole system drive that I can use to easily restore my whole system (OS, /root and /home (everything)).

  • Every day I use FreeFileSync to mirror sync all my user files to one of 3 external HDDs. So I always have 3 recent external backup copies of all my user (personal) files (the most important files!).

  • The nvme drives and HDDs are physically disconnected when a clone or sync is complete. They’re not continually connected to my laptop.

If you have suggestions to improve this backup strategy, feel free to let me know.

FYI: For various reasons it’s currently not possible for me to do online backups to Backblaze or S3 at this time.