|
My meme for the engineering triangle for project documentation has two sides labelled "complete" and "correct".
The third side is un-labelled to represent non-existence.
But it's funny 'cuz it's true.
|
|
|
|
|
And "It can't be too odd" you'd be wrong. I have seen so many developers blow past this area it makes my eyes water. Marc has a good list, but personally I would condense it. The real question is, what are your doc requirements? I had to write MILSPEC docs a long time ago. DD something or other, very specific and completely useless. I went to the USAF Captain handling the project and told him I could give him what he asked for but it would be complete crap and useless. Alternatively, I could give him something useful and we just wrap it in the boilerplate of the doc spec. He was happy.
- if you are documenting code stuff, it must be automated. Else, the doc will become a project of it's own, no one will maintain that part and worse, it will become obsolete and not reflect the code. Frankly, this is mostly useless documentation.
- what you need to build the project - good stuff. Software to install, build procedures, etc.
- my biggest career beef - I rarely come across doc that explains why the code is the way the code is, the abrasions, broken limbs and late nights of things that have bit you, and on and on.
- DO NOT INCLUDE WEB REFERENCES TO INFORMATION. Especially Microsoft. Print the info in PDF form and archive it with the doc. Microsoft routinely prunes useful info for the misguided reason that nobody could possibly care about this any more.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
[tl;dr]: Is RAID5 really causing such a huge performance hit?
I have a system (a Hyper-V VM host) with both eSATA and USB3.0 connectors.
I have a retired set of 8TB drives. I got myself a Mediasonic HFR2-SU3S2 PRORAID 4-drive enclosure, which can use either connector.
I love how trivial this enclosure's RAID setup is. I chose RAID5, so I have a total of 24TB worth of storage. Performance however makes it downright unusable. I could leave my VMs powered down overnight to back them up, but what I'm currently seeing could take days. Backing up a VM while it's running is just not a good idea (I use robocopy) so the VMs have to remain down while backing them up. That's not gonna fly during my workweek.
I made sure that, whether I'm using USB3 or eSATA, the "Better Performance" radio button is selected in Device Manager / Disk drives / [the RAID enclosure] / Properties / Policies.
Write operations hold steady at ~2.6MB/s. Active time is flat at 100%.
Same setup, but using eSATA instead, holds steady at ~5MB/s. Better, but still way below expectations. I'm questioning what my expectations should be.
The OS sees the RAID, not individual drives. On top of that, I use VeraCrypt to encrypt the entire RAID. I understand RAID involves some overhead, especially for Write operations--parity calculations would be done by the enclosure hardware, not my VM host's CPU. OTOH, VeraCrypt also introduces its own overhead, and that would be done by the host's CPU (which holds steady at ~3-4% when copying, so that's hardly the killer).
Before I got the RAID enclosure, I backed up the VMs onto a single external disk over USB3, and there was always plenty of time to do the whole thing overnight. I forget what I got in terms of transfer rate, but I'll be sure to pay attention the next time I do it - surely at least 10x the current performance. That single disk is also encrypted with VeraCrypt, so--unless I'm missing something--the only thing left that can account for the difference in transfer rate is the fact that the target drives are set up in a RAID, as opposed to transferring to a single drive.
My (somewhat rhetorical) question is: Really?
Does my diagnostic make sense? Is the fact that I'm backing up to a RAID the real performance killer? Everything is otherwise the same - both the RAID and my single external drive are connected via USB3, and using VeraCrypt.
Does it make sense at all that RAID5 kills performance to the extent I'm seeing?
What would you expect with a setup like this? I know I'll never get close to USB3's theoretical maximum throughput, but this is insane.
[The RAID isn't indicating any failure, and the last time I've used the drives individually, they were all working fine]
|
|
|
|
|
(Definitely didn't read the whole thing.)
When selecting a RAID level, you have to consider the ratio of reads to writes. Most situations have many more reads than writes and (if I recall correctly) RAID 5 is designed for that. But if your situation doesn't have so many reads, a different RAID level may provide better performance.
|
|
|
|
|
Well, I really only bought the RAID enclosure to maintain an extra backup set. I'm already backing up on single drives, but I had more than enough retired 8TB drives to make a RAID out of them. While I'll sacrifice capacity for performance, there's a limit to what I'll find acceptable. I thought RAID5 was a good compromise. Of course I'd never settle for a purely striped setup.
In theory, the only time I'll ever read back what I have on that RAID is if I my 'live' drives fail and need to restore from the backup. Otherwise the intend is to just re-sync (once a week?) what's different from my live drive back onto it (essentially incremental updates - robocopy's good at that).
At this point I'm thinking maybe I should just do my backups straight from my live drive, onto a single external drive, and then let the RAID bring itself up to date by syncing from the backup instead. Then it doesn't matter if that takes days, since that won't force me to have my VMs powered down when the extra backup is taking place...
|
|
|
|
|
RAID 0+1 might be a better option for RAID mode in your use case.
Basically, striped, but with an extra drive for parity, so a bit of both worlds.
Something is going on here though... Still thinking what it would be.
|
|
|
|
|
First off, are you sure you've got RAID 5 selected? If you do, and you have 4 8TB drives, the total storage should be far less than 24TB. Depending on how the controller sets the mirrored slices, then you might only see 8 TB.
If the enclosure supports non-raid operation, I'd be tempted to try just one drive with no encryption, and see what transfer speed you get. If it's still poor, then that might suggest that the enclosure itself has poor transfer rates. You might also try it with the other drives, just in case one or more of them has some hardware issue that's slowing the whole thing down. If you get good transfer rates with a single drive, I'd move on to basic mirror (RAID-1), and see how that works out. Maybe try some of the other RAID configurations as available to see if there's a sweet spot that provides good transfer rates and protection from drive failure.
"A little song, a little dance, a little seltzer down your pants"
Chuckles the clown
|
|
|
|
|
I think just about every discussion I've ever seen about RAID capacities said essentially that the capacity for a RAID5 setup (assuming drives of identical sizes) is N-1 (where N is the capacity of one drive). So 24TB sounds right to me.
I suppose I will take the time to try out various combinations, but this is all taking a very long time. I had initially started off with four 4TB drives, and was essentially seeing the same thing under the same conditions.
|
|
|
|
|
You are right about the capacity. I don't know what I was thinking. Something like the stripes were duplicated over the array, something like RAID 5+0 (?). 
"A little song, a little dance, a little seltzer down your pants"
Chuckles the clown
|
|
|
|
|
I forgot all about RAID 5+0; I thought for a minute that was the equivalent of RAID-10, but it's apparently not. Neither is RAID6. Now unless I find something with tables and pretty pictures, I'm more confused than ever about which is which.  But that's not the point.
Suffice it to say that given the capacity/redundancy and the drives I have at my disposal (four x 8TB), RAID5 seemed like the best compromise to me. I would hate to have 32TB worth of drives but get to use "only" half the capacity.
|
|
|
|
|
1. Just a SWAG here, but that level of NAS suggests minimal processing power. Since it is essentially a software RAID, the more writing across disks, the more overhead. Many of these boxes run a slimmed verion of TrueNAS or embedded Linux.
I run a TrueNAS NAS box on a 10 year old Precision 5810 with 2 5TB NAS drives, 5400 RPM. Uses the ZFS file system in a mirrored RAID (software not hardware). I backup via ethernet. Here is a small sample, using Robocopy:
Total Copied Skipped Mismatch FAILED Extras
Dirs : 515 514 1 0 0 0
Files : 2483 2483 0 0 0 0
Bytes : 16.795 g 16.795 g 0 0 0 0
Times : 0:04:39 0:04:12 0:00:00 0:00:27
Speed : 71555803 Bytes/sec.
Speed : 4094.455 MegaBytes/min.
Ended : Tuesday, April 16, 2024 12:18:17 PM
Not great but OK (~69MB). Network supports about 125MB.
I agree with the idea of putting a single drive in your NAS for testing.
2. Veeam will back up running VM's. Community version is free for something like 10 VM's. Not full featured, but with a PS script, you can automate.
Oops, wait: I am not affiliated, associated, known by, etc.
>64
It’s weird being the same age as old people. Live every day like it is your last; one day, it will be.
|
|
|
|
|
I'd be happy with 70MB/s. I'm pretty sure this is roughly what I had with my single drive.
Does Veeam produce straight copies of what you backup, or does it produce backup files that can only be read back by their own software?
|
|
|
|
|
proprietary, but you can mount the backup for individual files/folders, you do have to use their recovery software. I am guessing that veracrypted files could complicate things.
We use a combination of Veeam and Robocopy.
Robocopy for changed/new files nightly to a NAS. NAS has limited connectivity by IP, and there are no mappings to it.
VM's backed up with Veeam monthly (some semi monthly) to removable media, then taken off site.
Got hit by ransomeware before it was fashionable, some 12 or so years ago. Had to recover about 3 days of file changes, rest was backed up and disconnected.
Early ransomeware only encrypted a few file types in those days, ruined a Saturday afternoon.
In the SOHO, I run Linux, as well as ESXi to host VM's and the Veeam agent for Linux works fine although the UI is er um, less than stellar (their support is stellar).
Again, that 4 letter word beginning with F: Free.
I like it better than timeshift because you can image drives and/or partitions.
>64
It’s weird being the same age as old people. Live every day like it is your last; one day, it will be.
|
|
|
|
|
Once bitten, twice shy...I got hit hard a long time ago by some proprietary backup software that created files only it could read, and (I guess) one misplaced bit at the worst possible location was enough to make the entire file unusable. Of course that's when I needed to use the backup.
Since that time I've been sticking with robocopy; the entire tree is identical to the source, and no third-party software whatsoever is needed to access any specific file and without mounting anything. Of course that does nothing for bootable disks and the like, but considering I'm primarily interested in backing up VHD/VHDX files (and not physical disks), I can get away with it.
The entire "disk" is encrypted with VeraCrypt which, in a case like this, is preferable over a file container (.VC files). Since this is a hardware RAID, neither Windows nor VeraCrypt are really aware the underlying disk is actually a RAID (maybe they are, but since this is "supposed" to be transparent, I'm guessing they don't do anything special about it.
|
|
|
|
|
To add to the discussion, here are some thoughts and information:
- Poor throughput could be linked to the RAID5 being either degraded or still being built. I found some Youtube videos showing that (for your model) you would get about max 40 MB/s write speed in such a case, opposed to about 80 MB/s with a healthy array.
- As mentioned by others, these entry level enclosures have pretty low hardware specs and most often do software RAID inside the box. This can seriously impact performance.
- USB 3.0 throughput can reach 5 Gb/s, or 500 MB/s after overhead etc. USB 3.1 can do double that. USB 2 can do about 1/10 of that. So you are getting USB2 speeds at best.
- Have you checked transfer speeds on another PC?
- Finally, your NAS initially came out in 2010, which is a century ago in IT terms.
I run a pair of Asustor NAS with 2.5Gb/s ethernet interfaces. I get 250 MB/s read and write speeds with a simple Windows Explorer copy [^]. These NAS do however have a couple of NVMe SSD cache drives in them. A third NAS with 1 Gb/s and no cache drives gives a solid 100 MB/s. I have yet to test VM perfomance, but the NAS can expose iSCSI volumes so I am expecting to see similar performance from inside a VM.
So old that I did my first coding in octal via switches on a DEC PDP 8
|
|
|
|
|
As part of my analysis, I looked at the manufacturer's web page for the NAS box. It claims that the RAID is done in hardware.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
I have a WD PR4100 NAS configured to use RAID5 with 8TB WD Red drives. When writing to the NAS over a 1GB/s network, I get speeds of at least 60-100 Mbps. Unless you are writing large amounts of tiny files, I would expect at least that sort of performance out of any NAS.
USB 3 has a maximum transfer rate of 5 Gbps, and USB 2 has a maximum transfer rate of 480 Mbps. Even using USB 2, the transfer rates are very low. I would try each single drive in a different enclosure, if you have one, and then try one of the single drives in the NAS enclosure. This should give you enough information to determine whether it is one of the drives or the NAS that is the bottleneck.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
Daniel Pfeffer wrote: Unless you are writing large amounts of tiny files, I would expect at least that sort of performance out of any NAS.
So far I've only been trying to backup VM files - so, very few, but rather large files (multiple dozen GBs in size).
Daniel Pfeffer wrote: Even using USB 2, the transfer rates are very low
Agreed, I've been using other single drives using the same USB3 port and connector, and get much better speeds. So I have to rule out this beign stuck in USB 2 mode.
Your idea of testing out each drive separately is a good one. I wonder if I could power down the RAID, take one drive, put it in an enclosure and, without reformating it, run some sort of non-destructive speed test, then put the drive back in the RAID. I'd hate to rebuild it all over again at this point.
|
|
|
|
|
dandy72 wrote: I wonder if I could power down the RAID, take one drive, put it in an enclosure and, without reformating it, run some sort of non-destructive speed test
Given that you want to test WRITE speed I would say no, not unless you can find a non-destructive write test. I don't know of any test program that works like that.
You may also have to take into account the behaviour of the NAS. It may detect that the drive is part of a very degraded RAID-5 set and refuse to mount it. You may have to tell the NAS that this is a JBOD and have it reconfigure the drive before it can be accessed.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
First, RAID 5 with 8 TB drives is asking for complete, 100% data loss. The reason is that if you have one drive fail, when you replace it the entire RAID set has to be restored and during that restore even a single burp from one of the other three drives will double fault your data store, causing the entire store to corrupt. RAID 5 is also very computation expensive in that the data has to be split up across the drives during writes and then recombined on reads.
Second, if you want performance from large drives in a RAID environment, use RAID 1 or RAID 10. These are striped. Personally, I would use RAID 10 in your case, resulting in 16 TB of resilient storage. RAID 1 and 10 are mirrored so there is no computation involved, just two writes. During read either drive in the pair may be read depending on drive availability.
Finally, veracrypt is a good product, but if your system supports BitLocker at the hyper-v level, use it instead. There is a lot less overhead with BitLocker relative to veracrypt. I don't think this is causing your performance problems, though. I suspect one of your drives is on the way out and the RAID controller is having to do a lot of extra reads and writes to keep that drive responding.
|
|
|
|
|
Very interesting thoughts from someone who obviously has a lot more RAID experience than I do. I appreciate you sharing that.
Are failures during rebuilds so common that RAID admins consider them to be a real concern? I don't question the possibility of having two failures occurring close to each other in time...I'm wondering if you're suggesting that a RAID 5 config exacerbates the likelihood of multiple failures. I suppose all drives are working harder while the whole thing is being rebuilt...
In a way, I'm ok(-ish) with the possible loss - the whole RAID is intended to only act as an additional backup set (not my only backup set). So long as it doesn't take a week to rebuild if I do ever encounter a failure.
And since this is only for backups, I'm not terribly concerned about squeezing every last iota of performance - but again, not sacrificing performance to the extent that I'm seeing right now.
As mentioned elsewhere in this thread, I thought RAID5 was a good compromise between redundancy and capacity. I'm not a huge fan of the thought of pure duplication (eg, losing 50% of total capacity). OTOH, I'd be willing to do it if it guaranteed it brought performance back to where it ought to be. But if it only made things marginally faster, I don't think I'd be in a much better place.
|
|
|
|
|
dandy72 wrote: suggesting that a RAID 5 config exacerbates
IMO, it's the drive count. For RAID 5, it's just a sort of high barrier to entry where it's just not worth doing unless you're using a fairly large number of drives (9+) because fewer drives tends inherently mean the recoverability/performance benefits would be better in a different RAID mode.
You're right though. Something with that enclosure or one of those drives or something about the NAS networking is not right.
|
|
|
|
|
Just a thought, have you tried the enclosure on a USB 2 socket?
So old that I did my first coding in octal via switches on a DEC PDP 8
|
|
|
|
|
Years ago, I lived in the land of RAID. In my case, I had the need for high throughput for industrial systems. In circa 2000 all we had were spinners, and the fastest were SCSI. So, RAID made sense for higher performance and redundancy. I forget the manufacturer but they were high end and we paid a lot of $$ for these units. Redundant power supplies, controllers, drives, etc. Then one day, one of the controllers failed, and we lost the entire RAID. Seems the controller boards were sort of redundant. As the manager for the data group, it was an interesting conversation with the tech, support and ultimately a VP. It started with "you have to be elephanting kidding me?" Where upon I replaced their product.
Why do I say this? First, RAID helped but tech has passed on. If you are doing this for a science experiment, fine. For day to day, it's just not worth the hassle. Go buy some 4 tb usb drives and move on.
But let's talk about specifications. It is true that USB 2.0 supports UP TO 480 Mbps. USB 3.0 supports UP TO 5.0 Gbps. Note the "UP TO." The interface may support it, but I have yet to find a device that even approaches this transfer rate. I admit I have not tried a RAM disk. I have a fairly high end laptop with USB 3.2 on it. I can plug in a USB 3.2 SSD and copy my VMs to it. The burst speed is actually quite good, but when the cache fills, the transfer rate drops to 10% of the spec rate (I'm ballparking here, it's been a while since I did the test).
So, if you are moving large files - like me - 150GB for a VM via cut/paste, the cache will fill up and you will see perf degrade. This is with modern hardware. Your old spinners? I'd expect worse.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
Just to confirm, the USB3 connection is USB-C? I have a NVME enclosure which had full USB3 speeds in one cable orientation, but if flipped 180 it would be USB2. I would hope it's not something as simple as that. Can't hurt to try. I know that's the whole point of USB-C, however, that doesn't mean everyone handles things correctly
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
