this post was submitted on 12 Jan 2025
20 points (88.5% liked)

Linux

48954 readers
558 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
 

For context:

I'm copying the same files to the same USB drive for comparison from Windows and from my Fedora 41 Workstation.

Around 10k photos.

Windows PC: Dual Core AMD Athlon from 2009, 4GB RAM, old HDD, takes around 40min to copy the files to USB

Linux PC: 5800X3D, 64GB RAM, NVMe SSD, takes around 3h to copy the same files to the same USB stick

I've tried chagning from NTFS to exFAT but the same result. What can I do to improve this? It's really annoying.

top 19 comments
sorted by: hot top controversial new old
[–] AnAmericanPotato@programming.dev 1 points 29 minutes ago* (last edited 24 minutes ago)

Silly question perhaps, but are you sure you're using the correct port on your Linux system? If I plug my external HD into a USB2 port, I'm stuck at 30-40MB/sec, while on a USB3 port I get ~150-180MB/sec. That's proportionally similar to the difference you described so I wonder if that's the culprit.

You can verify this in a few different ways. From Terminal, if you run lsusb you'll see a list of all your USB hubs and devices.

It should look something like this:

Bus 002 Device 001: ID xxxx:yyyy Linux Foundation 3.0 root hub
Bus 002 Device 002: ID xxxx:yyyy <HDD device name>
Bus 003 Device 001: ID xxxx:yyyy Linux Foundation 2.0 root hub
Bus 004 Device 001: ID xxxx:yyyy Linux Foundation 3.0 root hub

So you can see three hubs, one of which is 2.0 and the other two are 3.0. The HDD is on bus 002, which we can see is a USB 3.0 hub by looking at the description of Bus 002 Device 001. That's good.

If you see it on a 2.0 bus, or on a bus with many other devices on it, that's bad and you should re-organize your USB devices so your low-speed peripherals (mouse, keyboard, etc.) are on a USB2 bus and only high-speed devices are on the USB3 bus.

You can also consult your motherboard's manual, or just look at the colors of your USB ports. By convention, gray ports are USB 1.0, blue ports are 2.0, and green ports are 3.x.

If you're running KDE, you can also view these details in the GUI with kinfocenter. Not sure what the Gnome equivalent is.

[–] eager_eagle@lemmy.world 4 points 2 hours ago

rsync -aP /

I find it faster and more reliable than most GUI explorers

[–] lnxtx@feddit.nl 15 points 4 hours ago (2 children)

I/O scheduler issue?

Connect that USB stick, and run command:

grep '' /sys/block/*/queue/scheduler

show output.

[–] WereCat@lemmy.world 2 points 4 hours ago (1 children)
[–] lnxtx@feddit.nl 1 points 3 hours ago

I see nothing wrong here. So issue must be somewhere else.

[–] WereCat@lemmy.world 2 points 4 hours ago (1 children)

Can I run it while transferring?

[–] lnxtx@feddit.nl 1 points 4 hours ago
[–] neidu3@sh.itjust.works 12 points 5 hours ago* (last edited 5 hours ago) (2 children)

I find that it's around the same, except linux waits on updating the UI until all write buffers are flushed, whereas Windows does not.

[–] Buffalox@lemmy.world 4 points 4 hours ago* (last edited 4 hours ago) (1 children)

except linux waits on updating the UI until all write buffers are flushed, whereas Windows does not.

I wish that were true here. But when I copy to USB the file manager ( XFCE/Thunar ) shows the copy is finished and closes the copy notifications way way before it's even half done, when I copy movies to a stick.
I use fast USB 3 stick on USB 3 port, and I don't get anywhere near the write speed the stick manufacturer claims. So I always open a terminal and run sync, to see when it's actually finished.

I hate to the extreme when systems don't account for write cache before claiming a copy is finished, it's such an ancient problem we've had since the 90's, and I find it embarrassing that such problems still exist on modern systems.

[–] WereCat@lemmy.world 3 points 4 hours ago (1 children)

I've ran sync and it exited already 4 times and the copy is still going

[–] Buffalox@lemmy.world 2 points 3 hours ago

Yes that's annoying too, I have no clue why it does that, but when the sync says "clear", I always wait a couple seconds, and run sync again a couple of times, to see if it's actually finished. And only THEN unmount the stick.
Copy to USB does not seem very solid on Linux IMO. So I also ALWAYS buy sticks with activity LED.
But even that can fool you, sometimes when I think a smaller copy is finished, because the LED stops blinking, it suddenly starts up again, after having paused for about 1½ second?!?!

[–] WereCat@lemmy.world 4 points 5 hours ago (1 children)

That's nice but I managed to copy 300GB worth of data from the Windows PC to my Linux PC in around 3h to make a backup while I reinstall system and now I've been stuck for half a day copying the data back to the old Windows PC and I've not even finished 100GB yet... I've noticed this issue long ago but I ignored it as I never really had to copy this much data. Now it's just infuriating.

[–] schizo@forum.uncomfortable.business 7 points 4 hours ago (1 children)

One thing I ran into, though it was a while ago, was that disk caching being on would trash performance for writes on removable media for me.

The issue ended up being that the kernel would keep flushing the cache to disk, and while it was doing that none of your transfers are happening. So, it'd end up doubling or more the copy time because the write cache wasn't actually helping removable drives.

It might be worth remounting without any caching, if it's on, and seeing if that fixes the mess.

But, as I said, this has been a few years, so that may no longer be actively the case.

[–] WereCat@lemmy.world 4 points 4 hours ago

This actually sounds like it could be the case, I'll explore tomorrow as I'm already in bed. Thanks for suggestion.

[–] eugenia@lemmy.ml 3 points 4 hours ago* (last edited 4 hours ago)

That's just the state of things. I have experienced this as well, trying to copy a 160 GB usb stick to another one (my old itunes library). Windows manages fine, but neither Linux nor MacOS do it properly. They crawl, and in macos' case, it gets much slower as time goes by, and I had to stop the transfer. Overall, it's how these things are implemented. It's ok for a few gigabytes, but not a good case for many small files (e.g. 3-5 mb each) with many subfolders, and many GBs overall. Seems to me that some cache is overfilling, while windows is more diligent to clear up that cache in time, before things get into a crawl. Just a weak implementation for both Linux and MacOS IMHO, and while I'm a full time Linux user, I'm not afraid to say it how I experienced it under a debian and ubuntu.

[–] nomecks@lemmy.wtf 3 points 4 hours ago* (last edited 2 hours ago) (2 children)

How are you copying them? It could potentially be faster to use rsync. You can try mounting with noatime as a flag as well.

[–] Buffalox@lemmy.world 5 points 4 hours ago* (last edited 4 hours ago) (1 children)

noatime

Oh boy, yes I remember I used to disable that shit ages ago in fstab, that's quite annoying that's still necessary!!!
How do you do that for USB sticks?

[–] nomecks@lemmy.wtf 1 points 2 hours ago* (last edited 2 hours ago)

I'm pretty sure you can just mount it with mount and specify noatime

[–] WereCat@lemmy.world 2 points 4 hours ago

Ctrl c, ctrl v