Wow, the animated gif of the Defrag running sent me way back.
I used to run that like it was religion, watching the blocks all move together and change color, and knowing the machine would run just a bit faster after it was all done was a bit of magic.
It gave me flashbacks to high school computer class, in which we were taught to use a tool misnamed COMPRESS -- actually a defragger -- to maintain our hard disks. It displayed a similar animation in text mode as it worked. This was on OG IBM 5150s -- ancient even at the time.
It's funny how, in these late-stage Moore's Law days, a 10-year-old PC is old certainly, but not as hopelessly outmoded now as the 5150 was then.
Just wanted to say thanks for doing this research on flash speeds and posting your results for the last few years, they've helped me pick both reasonable sd-cards and mid-cost USB thumb drives that work for my budget and RPi 4.
Samsung Evo+ cards with the same speed rating (say U1) and model number can vary in performance quite a bit. I have tried three of them on my rpi4, and found one which was considerably faster than the others. To be sure I verified it with crystaldiskmark and indeed, it was up to 2x faster in writes and reads.
Samsung cards tend to outperform other brands in that price range.
tl;dr - the Pi 4 now has USB boot enabled by default, and you can get way better performance from a fast USB 3.0 drive than the built-in microSD.
I like the Arcanite for large files and basic storage needs, or an NVMe + USB 3.0 adapter for the best performance for a reasonable price.
UASP can speed up operations by up to 50%, and many SSDs and high end flash drives do support it. TRIM support is less prevalent, and you often need to manually enable it (there's a guide linked in the post).
Weirdest finding: the Arcanite flash drive firmware indicates it supports trim, but when I tried running fstrim something blew up in the drive, and I could no longer read/mount/initialize it from Mac/Linux/Windows (I had to buy a replacement to finish the testing).
Thanks for doing all that testing! Nice to see that answer pretty much boils down to "Use something decent-ish, mostly it doesn't matter which choice you make"... (I would have guessed nVME would have been worth spending extra on, turns out not by enough to care about, at least not in a Pi4.)
Yeah the Pi 4's USB 3.0 bus isn't able to take advantage of the speeds faster drives can put out; I'm hoping the next compute module can support NVMe or SATA more directly (something Eben Upton hinted at in a podcast earlier this summer). A Pi 4 B+ or Pi 5 could even fit a slot underneath?
I'd expect the compute flavour to expose the PCIe bus, so SATA or NVMe would be easily added as no USB3 chip hogging it. Indeed I'd say that is not only a given avenue, but the angle Mr Upton was hinting towards given the I/O side of things currently from the SOC.
Like many - pre Pi4, was wishing for a SATA port, but overall, if they did do some upgrade to the Pi4 in another B+ or 5 flavour - then I'd be more interested in a faster USB flavour, like 3.2 or USB-C. Reason being is that it would be more flexible and those who want SATA or NVMe would be able to tap into that. Certainly more flexible in usage and that does seem to be key.
In hindsight, how would you have used the documentation of the Raspberry Pi architecture to determine whether UASP is supported?
Also, I'm really curious if your only problem with that Arcanite was only that it was a defective unit. If anyone reading has any other ideas for diagnosing that problem then I'd love to know them.
Great guide, Jeff. Really enjoyable (and informative!).
> In hindsight, how would you have used the documentation of the Raspberry Pi architecture to determine whether UASP is supported?
Doesn't really depend on the Pi hardware at all, it depends on the kernel driver (kernel must be new enough) and the USB storage device (does it implement UASP in addition to legacy bulk-only-transport?).
Not likely to be smoke. However TRIM failures were very common with SSDs early on and even today some distros disable or partially disable it by default (see for example this Debian warning: https://wiki.debian.org/SSDOptimization#WARNING). Upstream Linux does not enable TRIM for MD-raid devices unless you use a special kernel boot parameter. "Continuous TRIM" (which means setting the discard parameter on the filesystem so that files are automatically trimmed on deletion) is also a good way to reveal bugs and problems, some of which will merely kill performance when you delete files, and others will nuke your data.
No smoke, just stopped working entirely, and can't mount it or partition it at all. Closest I can get is with Disk Utility on the Mac, if I partition it, it acts like it's working, then it gives an error on the last block of the device, something like that.
Does anyone have any experience how reliable is such modern USB attached storage when run 24/7? Some years back I made a box (x86, not RPi) with a bunch of USB connected drives (from Western Digital, if I remember correctly). After a few days of uptime the drives would simply drop from the bus. I was guessing there was a bug in the drive’s USB controller.
These devices are usually meant to be used as a carrier that’s only connected to a computer to load/off load data. Long-term operation probably isn’t a priority when developing them.
Not related to USB, but flash storage quality in general varies a lot.
We have a few hundred Raspberry Pi in operation at customer sites, running 24/7. Back when we started our business we used generic Micro SD (can't remember the brand). They would fail after 2-6 months. This was a big hassle. We have customers all over Germany and would need to ship them a complete RPI as a replacement, as the customers are usually not tech-savvy enough to just replace the SD.
We now use SanDisk's "Industrial" Micro SD and haven't had this problem since.
Also, reducing writes to the SD is certainly a good idea. Ideally - if you can - mount the system as read only.
Interesting you mention "Industrial" cards as I have been looking at cards in this range after reading quite a few issues with RPIs over the past few months. I have a RPI3 with a SanDisk Extreme, running for 18 months 24/7 with no issues yet. Plenty of re-imaging and high temps. RPI 4 has a SanDisk Ultra, only been running for 3 months 24/7. Now whilst we all should have backups anyway, I really don't want the worry or hassle of restoring backups couple of times a year, I would rather have a card that will last years rather than months.
Slightly off topic but I have also been looking at using net boot instead. Not only would it be just as fast as the Micro SD cards but much easier to perform backups locally and offsite.
Most of the usb sticks seem to perform about the same as microSD cards, and I wouldn't trust them (especially the tiny ones that get super hot like the Ultra Fit) for booting anything important. USB SATA/NVMe SSDs are just as reliable as internal drives, in my experience. I've had a few of them running 24x7 on my NAS for years (with decent amounts of activity).
And the Corsair GTX, while massive, is as fast as an SSD and supports UASP and TRIM. It's just big, and relatively expensive, for a 'thumb drive'.
I ran 5 USB drivers (4x USB3 and 1x USB2) of a powered USB3 hub attached to a RPi4 a few weeks back as was clearing up some drives and data juggling between them. Had that running for few days with no issues at all and was thrashed a few times with moves/copies across all the drives.
So indications are good, and the drives in this case was a WD SSD, 3x Toshiba bus powered drives and an ancient 320gb usb externally powered WD affair. So a but of a mix. Though I did have it plugged into a UPS, which like all things tech - kinda helps smooth out the random chaos of electricity and tech.
Most storage corruption problems related Rasberry PIs are just PSU problems. Flash drives and SD cards draw their power directly from the Rasberry PI and since those only use a crappy micro USB port or a buggy USB C port you're going to run into issues, especially if you use a random USB charger instead of the official power supplies. External storage like an SSD that uses its own power supply (via the USB hub) will be far more reliable.
I have used a usb stick (SanDisk Ultra Fit) for the Freenas O.S.. That drive died in 6 months (24/7). In general Freenas suggests to use 2 mirrored, quality, name-brand USB drives as they can `wear out or fail unexpectedly`[1]. They not suggest USB drives for data storage and I never tried.
I use a raspberry pi 4 (before that 3b) as NAS. A usb harddrive is used for data storage. The workload is low, mostly photos and music streaming, but I've not seen USB problems so far.
Been running 2 Samsung 500GB ssd (1 USB3 direct, 1 via USB3 hub) on rpi4 24/7 since boot from USB first landed (May?). Mostly video recording/backup workload. I’ve had no issues and sitting at 36d uptime right now.
I wouldn't trust any of those USB sticks for anything other than very light, mostly read-only use. However the SSD and NVME drives are likely to be a lot more reliable - since they are just desktop NVME/SSDs in a shell that converts the interface to USB 3 they are as reliable as an NVME/SSD in a desktop or laptop.
Be aware (as I found out a couple of days ago) the RPi 4 USB port cannot power an external USB hard disk unless you use a separate powered hub. Should be fine to use a USB SSD / USB NVME though.
I've been using an external portable (2.5") seagate drive[0] for a while with my pi 4 to hold music and videos. It's plugged straight into the USB 3 port with no problems so far.
You will have problems trying to power a desktop 3.5" drive though.
I’ve been running a NUC as a home server, with a cheap dual bay USB 3.1 to SATA enclosure, for about two years now. I haven’t had any reliability issues with it.
For normal laptops it won’t be so much difference in performance, as LUKS by default uses AES-XT as the encryption type, and most modern CPUs have AES accelerators in them.
For rPi though? Not sure they have an AES accelerator, and if they don’t then it’s going to be a huge cost to performance.
Not only because USB is using a CPU core to operate (because USB uses CPU not dedicated hardware like FireWire) but also because it has to transparently translate and encrypt every access.
The ARM chips in the all of the Raspberry pis do not have AES accelerators. Encrypting your pis drive is going to be a serious hit on performance like you said. I don't recommend it.
If you really really really want to encrypt your Rpi then you need to use a disk cipher like Adiantum. It is based off of ChaCha12 and was made specifically for low end CPUs that lack AES accelerators.
I know it doesn't have AES encryption acceleration (at least in the CPU).
That's why it's an interesting thing to measure, how much the competition between boot tasks and AES encryption affect the boot. It may not be much because I suspect boot is IO bound, but I don't know.
These reports always comment on raw performance without checking if the drive actually meets the storage contract with the host.
That's like having a 'hall of fame' for the fastest marathon runners, but not having anyone keep an eye on if they took a taxi half the route...
These benchmarks should always have a section where a drive has its power pulled repeatedly while under heavy load. If, ever, a single sector becomes overwritten, rolled back, or unreadable, except those being written between a pair of write barriers, then the drive has failed the test, and all other performance results thrown out. That drive took the taxi.
I've a rpi4 that I'm setting up as a media player. Files come from a NAS. In the past (rpi3) I've just used an SD card in the Pi. I'm curious if I'd get better performance using USB. Also curious if I did use USB if I could setup automatic caching to the drive.
I'm using an RPi 4 as a NAS/home server and the difference between SD speeds and USB speeds was pretty noticeable (subjectively); even simple linux commands (ls, cp -r, mv) sped up after switching to a SanDisk Ultra Flair. Except for one USB drive...
Recently I had the misfortune of having a debian update screw up my docker daemon and I ran over to Best Buy to grab any-old-USB drive to use as a new OS drive. As it happens the SD copy over went fine, but as I performed basic "setting the server up" operations like copying over the old data (50 GB of various sizes) and firing up docker-compose scripts, performance deteriorated rapidly to the point where a docker install of a lighttpd container took at least 30 minutes and the system became effectively unusable.
The poor-performing drive was the PNY Elite-X Fit, USB 3.0, 128GB, purchased for ~$22
I've switched back to a SanDisk SD card for the moment which is perfectly adequate for the next few months but I hope to get back to running off a USB soon when I'm not so busy.
For the Pi 3 B+ and previous generations, the difference will be much less pronounced, but still can be measurable.
For the Pi 4, a faster USB drive (e.g. SSD or NVMe in adapter) will be a massive improvement for many things (though some operations like launching a browser won't be vastly improved).
I used to run that like it was religion, watching the blocks all move together and change color, and knowing the machine would run just a bit faster after it was all done was a bit of magic.
It's funny how, in these late-stage Moore's Law days, a 10-year-old PC is old certainly, but not as hopelessly outmoded now as the 5150 was then.
http://hultbergs.org/defrag/
Samsung cards tend to outperform other brands in that price range.
I like the Arcanite for large files and basic storage needs, or an NVMe + USB 3.0 adapter for the best performance for a reasonable price.
UASP can speed up operations by up to 50%, and many SSDs and high end flash drives do support it. TRIM support is less prevalent, and you often need to manually enable it (there's a guide linked in the post).
Weirdest finding: the Arcanite flash drive firmware indicates it supports trim, but when I tried running fstrim something blew up in the drive, and I could no longer read/mount/initialize it from Mac/Linux/Windows (I had to buy a replacement to finish the testing).
Like many - pre Pi4, was wishing for a SATA port, but overall, if they did do some upgrade to the Pi4 in another B+ or 5 flavour - then I'd be more interested in a faster USB flavour, like 3.2 or USB-C. Reason being is that it would be more flexible and those who want SATA or NVMe would be able to tap into that. Certainly more flexible in usage and that does seem to be key.
Also, I'm really curious if your only problem with that Arcanite was only that it was a defective unit. If anyone reading has any other ideas for diagnosing that problem then I'd love to know them.
Great guide, Jeff. Really enjoyable (and informative!).
Doesn't really depend on the Pi hardware at all, it depends on the kernel driver (kernel must be new enough) and the USB storage device (does it implement UASP in addition to legacy bulk-only-transport?).
https://www.devever.net/~hl/usbuas has a great explanation of the nitty-gritty details.
These devices are usually meant to be used as a carrier that’s only connected to a computer to load/off load data. Long-term operation probably isn’t a priority when developing them.
We have a few hundred Raspberry Pi in operation at customer sites, running 24/7. Back when we started our business we used generic Micro SD (can't remember the brand). They would fail after 2-6 months. This was a big hassle. We have customers all over Germany and would need to ship them a complete RPI as a replacement, as the customers are usually not tech-savvy enough to just replace the SD.
We now use SanDisk's "Industrial" Micro SD and haven't had this problem since.
Also, reducing writes to the SD is certainly a good idea. Ideally - if you can - mount the system as read only.
Slightly off topic but I have also been looking at using net boot instead. Not only would it be just as fast as the Micro SD cards but much easier to perform backups locally and offsite.
And the Corsair GTX, while massive, is as fast as an SSD and supports UASP and TRIM. It's just big, and relatively expensive, for a 'thumb drive'.
I have run 24/7 many SSD's attached via USB, with no reliability problems.
Usually buying SATA or NVME SSD's and separate USB adapters for them is cheaper than buying USB SSD's, which actually contain an internal USB adapter.
So indications are good, and the drives in this case was a WD SSD, 3x Toshiba bus powered drives and an ancient 320gb usb externally powered WD affair. So a but of a mix. Though I did have it plugged into a UPS, which like all things tech - kinda helps smooth out the random chaos of electricity and tech.
I have used a usb stick (SanDisk Ultra Fit) for the Freenas O.S.. That drive died in 6 months (24/7). In general Freenas suggests to use 2 mirrored, quality, name-brand USB drives as they can `wear out or fail unexpectedly`[1]. They not suggest USB drives for data storage and I never tried.
[1] https://www.ixsystems.com/documentation/freenas/11.3-U4/intr...
Be aware (as I found out a couple of days ago) the RPi 4 USB port cannot power an external USB hard disk unless you use a separate powered hub. Should be fine to use a USB SSD / USB NVME though.
You will have problems trying to power a desktop 3.5" drive though.
[0] https://www.amazon.co.uk/Seagate-Expansion-Portable-PlayStat...
For rPi though? Not sure they have an AES accelerator, and if they don’t then it’s going to be a huge cost to performance.
Not only because USB is using a CPU core to operate (because USB uses CPU not dedicated hardware like FireWire) but also because it has to transparently translate and encrypt every access.
It seems like the raspberry actually has zero crypto extensions anyway: https://www.raspberrypi.org/forums/viewtopic.php?t=207888
The ARM chips in the all of the Raspberry pis do not have AES accelerators. Encrypting your pis drive is going to be a serious hit on performance like you said. I don't recommend it.
If you really really really want to encrypt your Rpi then you need to use a disk cipher like Adiantum. It is based off of ChaCha12 and was made specifically for low end CPUs that lack AES accelerators.
That's why it's an interesting thing to measure, how much the competition between boot tasks and AES encryption affect the boot. It may not be much because I suspect boot is IO bound, but I don't know.
That's like having a 'hall of fame' for the fastest marathon runners, but not having anyone keep an eye on if they took a taxi half the route...
These benchmarks should always have a section where a drive has its power pulled repeatedly while under heavy load. If, ever, a single sector becomes overwritten, rolled back, or unreadable, except those being written between a pair of write barriers, then the drive has failed the test, and all other performance results thrown out. That drive took the taxi.
Recently I had the misfortune of having a debian update screw up my docker daemon and I ran over to Best Buy to grab any-old-USB drive to use as a new OS drive. As it happens the SD copy over went fine, but as I performed basic "setting the server up" operations like copying over the old data (50 GB of various sizes) and firing up docker-compose scripts, performance deteriorated rapidly to the point where a docker install of a lighttpd container took at least 30 minutes and the system became effectively unusable.
The poor-performing drive was the PNY Elite-X Fit, USB 3.0, 128GB, purchased for ~$22
I've switched back to a SanDisk SD card for the moment which is perfectly adequate for the next few months but I hope to get back to running off a USB soon when I'm not so busy.
For the Pi 4, a faster USB drive (e.g. SSD or NVMe in adapter) will be a massive improvement for many things (though some operations like launching a browser won't be vastly improved).