|
My experience with Doxygen (my own direct use and how I have seen numerous co-workers use it) is that it is fine for the bottom level documentation. The reader will know all the details about the third argument to a given method, but without a clue about when, where and why that method is called at all. The protocol, both any line protocol and the protocol of interaction between methods in one module and the interaction between modules.
In principle, you can add information about module and object interaction, line protocol specifications etc. My experience is that developers never get around to it Round Tuit[^]. Documenting the third parameter in a Doxygen comment is easy to do when you are editing the code; the higher level documentation is postponed until the next tuit delivery. (But sending the order has been postponed ...  ).
As long as the intermediate levels are documented well, from the top level, the main subsystems and their interactions, and then stepwise down to modules and objects, then the reader might as well dive right into the source code. When you understand the use of some component, you have no need for a Doxygen carbon copy of the details that you will find in the source code anyway. Too often, Doxygen 'documentation' is little more than a rephrasing of method header.
Obviously, this very much depends on how developers use Doxygen. I am referring to what I have observed, not what you in theory can do with Doxygen.
Equally important: This is certainly not Doxygen-specific. Give the same developers another documentation system that they can maintain as part of the code editing, and the result will be very much of the same. You see it everywhere, in almost all software and documentation available on internet: Method headers and parameters are documented; how these are intended to be used is often completely omitted. If there are some traces of it, it is often poorly written. (The best way learn the use of some library, server etc. is to search for some other open source code to see how they are using it!)
So, my advice is: Forget about the source code documentation, at the method header and parameter level (with the exception of APIs exported to users outside your organization). Document it top-down, with focus on component interactions. If there are explicit protocols across a line, describe it, but document the protocol, not the API! E.g. MSCs can be very helpful to understand the component interactions. You can even make MSCs for protocols between modules or objects, even if they are not line protocols.
Document data structures thoroughly, but again: Top down. What kind of information is held in each structure, which components have access to it, which component maintains it. I am so old that I still think of ER as a good data modelling tool; today it is about as non-PC as the riverfall model, but you can have an ER-like approach even if you do not use ER notation. Stick to abstract data types - at this level it doesn't matter if a counter is 16, 32 or 64 bits. It does matter that it is a counter, though!
If it suits the software, state diagrams and/or state tables are great documentation tools. Unfortunately, few young developers have a good grasp on state transitions; it is more like a vague, remote concept in the same class as the OSI 7-layer model .
So how far down should the documentation go? My answer is clear (but lots of people will frown): Write your documentation so that it is independent of programming language. If the system is re-implemented in, say, Fortran or Pascal, all of you documentation should still be valid. It should contain the language independent stuff, all the language independent stuff and nothing but the language independent stuff.
When you need code snippets or declarations to make your point, leave it to the source code itself. Your documentation tells what it should do. How does it can be seen from the code.
There is one big pitfall, though: When you start describing your system as a set of well documented(/defined) protocols, MSCs, a bird's eye on the data structures, state transitions etc., chances are that you will utter a few nasty words about how the system should have been written. You'll discover messy transitions, irregular use of interaction protocols, undisciplined access to data structures, ... There will be a lot of new entries in the ToDo-list! Just make sure to get that tuit order in the mail as soon as possible
Religious freedom is the freedom to say that two plus two make five.
|
|
|
|
|
The real problem is that there is not enough money or time in the budget for documenting and the programmer is usually to lazy to do it.
Definition of a burocrate; Delegate, Take Credit, shift blame.
PartsBin an Electronics Part Organizer - Release Version 1.3.1 JaxCoder.com
Latest Article: EventAggregator
|
|
|
|
|
There are people/companies promoting 'Test Driven Development', writing all the tests, from module level and upwards, before any coding is started. I have even talked with people claiming to practice it 
For all my private projects, I practice 'Documentation Driven Development': I write both (end) User Guide and system documentation (implementation language independent, as noted earlier) before I start coding. Admittedly, it happens that the 'ideal' view of the system, described in the documentation, cannot be realized 100% in practice. Yet I would say that having the 'ideal' solution in the documentation is a great help to make sure that adaptations are done in a structured and architecturally sound ways, so that they do not appear as random hacks. Both end user and system documentation is of course updated correspondingly.
The one big software suite I was working on that tried to follow a TDD style (admittedly, tests were implemented in parallel with the code, but by a completely independent team) did experience very similar situations: The original spec, underlaying both tests and code, could not reasonably be implemented exactly as given, so both code and tests had to be modified midway. Specs are not carved in stone.
Religious freedom is the freedom to say that two plus two make five.
|
|
|
|
|
You sir are an anomaly.
I know people that write a spec to go by but usually written by a project manager. But I guess if you are a one person shop that would be you.
I start out with a general plan but, as an example my current project, I started with a very generic idea of what I wanted to accomplish and it just grew and grew. The one saving grace is that I programmed it so that it could expand...and it has grown exponentially.
I document code pretty heavily as I go, but writing user manual or any kind of formal docs is VERY difficult for me.
Definition of a burocrate; Delegate, Take Credit, shift blame.
PartsBin an Electronics Part Organizer - Release Version 1.3.1 JaxCoder.com
Latest Article: EventAggregator
|
|
|
|
|
> I have written a bit of Code to talk to bit of hardware.
Does the hardware in question has a datasheet?
If hard to find, provide link or better the actual PDF you used to write the code.
Some example applications are often more useful than documentation in practice.
At least a minimalistic "hello world" app to get started, maybe a performance testing one.
The more complex the hardware the more example apps might be needed.
What is also useful is adding some (minimal) debugging.
E.g. an I2C based class should contain something like bool isConnected().
A SPI based class should have means to set the clock speed.
This allows 1st order testing for users of your code.
my 2 cents
|
|
|
|
|
The bit of Hardware are the guts of scale which then lashed to a mechanical rig (not of my design) so the scale documentation is of limited use. The comms is over USB which is part of the problem my first attack I started to go down the rabbit hole...
|
|
|
|
|
Its not a silly question to ask 
I would at least include:
- source code documentation (Doxygen is your friend)
- build instructions (which you tested on a clean machine)
- description of the communication protocol between PC and the hardware
- installation instructions (which you tested on a clean machine)
- use instructions (e.g. command line arguments)
- eventually an high-level flowchart of how the system operates
The first two items are helpful when in the future, you or somebody else needs to do some work.
The other are useful for the actual users and typically will be part of manual. Which I strongly
advise you to make, if somebody else isn't doing. I have found that "have you read the manual ?"
substantially decreases support effort
A lot of work ? The first time yes, a few days at least, but then it is just minor edits to the documents.
I use Doxygen for generating the documentation of the source code, of the language own tool if better.
And Sphinx[^]
for writing the manual and instructions.
|
|
|
|
|
My meme for the engineering triangle for project documentation has two sides labelled "complete" and "correct".
The third side is un-labelled to represent non-existence.
But it's funny 'cuz it's true.
|
|
|
|
|
[tl;dr]: Is RAID5 really causing such a huge performance hit?
I have a system (a Hyper-V VM host) with both eSATA and USB3.0 connectors.
I have a retired set of 8TB drives. I got myself a Mediasonic HFR2-SU3S2 PRORAID 4-drive enclosure, which can use either connector.
I love how trivial this enclosure's RAID setup is. I chose RAID5, so I have a total of 24TB worth of storage. Performance however makes it downright unusable. I could leave my VMs powered down overnight to back them up, but what I'm currently seeing could take days. Backing up a VM while it's running is just not a good idea (I use robocopy) so the VMs have to remain down while backing them up. That's not gonna fly during my workweek.
I made sure that, whether I'm using USB3 or eSATA, the "Better Performance" radio button is selected in Device Manager / Disk drives / [the RAID enclosure] / Properties / Policies.
Write operations hold steady at ~2.6MB/s. Active time is flat at 100%.
Same setup, but using eSATA instead, holds steady at ~5MB/s. Better, but still way below expectations. I'm questioning what my expectations should be.
The OS sees the RAID, not individual drives. On top of that, I use VeraCrypt to encrypt the entire RAID. I understand RAID involves some overhead, especially for Write operations--parity calculations would be done by the enclosure hardware, not my VM host's CPU. OTOH, VeraCrypt also introduces its own overhead, and that would be done by the host's CPU (which holds steady at ~3-4% when copying, so that's hardly the killer).
Before I got the RAID enclosure, I backed up the VMs onto a single external disk over USB3, and there was always plenty of time to do the whole thing overnight. I forget what I got in terms of transfer rate, but I'll be sure to pay attention the next time I do it - surely at least 10x the current performance. That single disk is also encrypted with VeraCrypt, so--unless I'm missing something--the only thing left that can account for the difference in transfer rate is the fact that the target drives are set up in a RAID, as opposed to transferring to a single drive.
My (somewhat rhetorical) question is: Really?
Does my diagnostic make sense? Is the fact that I'm backing up to a RAID the real performance killer? Everything is otherwise the same - both the RAID and my single external drive are connected via USB3, and using VeraCrypt.
Does it make sense at all that RAID5 kills performance to the extent I'm seeing?
What would you expect with a setup like this? I know I'll never get close to USB3's theoretical maximum throughput, but this is insane.
[The RAID isn't indicating any failure, and the last time I've used the drives individually, they were all working fine]
|
|
|
|
|
(Definitely didn't read the whole thing.)
When selecting a RAID level, you have to consider the ratio of reads to writes. Most situations have many more reads than writes and (if I recall correctly) RAID 5 is designed for that. But if your situation doesn't have so many reads, a different RAID level may provide better performance.
|
|
|
|
|
Well, I really only bought the RAID enclosure to maintain an extra backup set. I'm already backing up on single drives, but I had more than enough retired 8TB drives to make a RAID out of them. While I'll sacrifice capacity for performance, there's a limit to what I'll find acceptable. I thought RAID5 was a good compromise. Of course I'd never settle for a purely striped setup.
In theory, the only time I'll ever read back what I have on that RAID is if I my 'live' drives fail and need to restore from the backup. Otherwise the intend is to just re-sync (once a week?) what's different from my live drive back onto it (essentially incremental updates - robocopy's good at that).
At this point I'm thinking maybe I should just do my backups straight from my live drive, onto a single external drive, and then let the RAID bring itself up to date by syncing from the backup instead. Then it doesn't matter if that takes days, since that won't force me to have my VMs powered down when the extra backup is taking place...
|
|
|
|
|
RAID 0+1 might be a better option for RAID mode in your use case.
Basically, striped, but with an extra drive for parity, so a bit of both worlds.
Something is going on here though... Still thinking what it would be.
|
|
|
|
|
First off, are you sure you've got RAID 5 selected? If you do, and you have 4 8TB drives, the total storage should be far less than 24TB. Depending on how the controller sets the mirrored slices, then you might only see 8 TB.
If the enclosure supports non-raid operation, I'd be tempted to try just one drive with no encryption, and see what transfer speed you get. If it's still poor, then that might suggest that the enclosure itself has poor transfer rates. You might also try it with the other drives, just in case one or more of them has some hardware issue that's slowing the whole thing down. If you get good transfer rates with a single drive, I'd move on to basic mirror (RAID-1), and see how that works out. Maybe try some of the other RAID configurations as available to see if there's a sweet spot that provides good transfer rates and protection from drive failure.
"A little song, a little dance, a little seltzer down your pants"
Chuckles the clown
|
|
|
|
|
I think just about every discussion I've ever seen about RAID capacities said essentially that the capacity for a RAID5 setup (assuming drives of identical sizes) is N-1 (where N is the capacity of one drive). So 24TB sounds right to me.
I suppose I will take the time to try out various combinations, but this is all taking a very long time. I had initially started off with four 4TB drives, and was essentially seeing the same thing under the same conditions.
|
|
|
|
|
You are right about the capacity. I don't know what I was thinking. Something like the stripes were duplicated over the array, something like RAID 5+0 (?). 
"A little song, a little dance, a little seltzer down your pants"
Chuckles the clown
|
|
|
|
|
I forgot all about RAID 5+0; I thought for a minute that was the equivalent of RAID-10, but it's apparently not. Neither is RAID6. Now unless I find something with tables and pretty pictures, I'm more confused than ever about which is which. But that's not the point.
Suffice it to say that given the capacity/redundancy and the drives I have at my disposal (four x 8TB), RAID5 seemed like the best compromise to me. I would hate to have 32TB worth of drives but get to use "only" half the capacity.
|
|
|
|
|
1. Just a SWAG here, but that level of NAS suggests minimal processing power. Since it is essentially a software RAID, the more writing across disks, the more overhead. Many of these boxes run a slimmed verion of TrueNAS or embedded Linux.
I run a TrueNAS NAS box on a 10 year old Precision 5810 with 2 5TB NAS drives, 5400 RPM. Uses the ZFS file system in a mirrored RAID (software not hardware). I backup via ethernet. Here is a small sample, using Robocopy:
Total Copied Skipped Mismatch FAILED Extras
Dirs : 515 514 1 0 0 0
Files : 2483 2483 0 0 0 0
Bytes : 16.795 g 16.795 g 0 0 0 0
Times : 0:04:39 0:04:12 0:00:00 0:00:27
Speed : 71555803 Bytes/sec.
Speed : 4094.455 MegaBytes/min.
Ended : Tuesday, April 16, 2024 12:18:17 PM
Not great but OK (~69MB). Network supports about 125MB.
I agree with the idea of putting a single drive in your NAS for testing.
2. Veeam will back up running VM's. Community version is free for something like 10 VM's. Not full featured, but with a PS script, you can automate.
Oops, wait: I am not affiliated, associated, known by, etc.
>64
It’s weird being the same age as old people. Live every day like it is your last; one day, it will be.
|
|
|
|
|
I'd be happy with 70MB/s. I'm pretty sure this is roughly what I had with my single drive.
Does Veeam produce straight copies of what you backup, or does it produce backup files that can only be read back by their own software?
|
|
|
|
|
proprietary, but you can mount the backup for individual files/folders, you do have to use their recovery software. I am guessing that veracrypted files could complicate things.
We use a combination of Veeam and Robocopy.
Robocopy for changed/new files nightly to a NAS. NAS has limited connectivity by IP, and there are no mappings to it.
VM's backed up with Veeam monthly (some semi monthly) to removable media, then taken off site.
Got hit by ransomeware before it was fashionable, some 12 or so years ago. Had to recover about 3 days of file changes, rest was backed up and disconnected.
Early ransomeware only encrypted a few file types in those days, ruined a Saturday afternoon.
In the SOHO, I run Linux, as well as ESXi to host VM's and the Veeam agent for Linux works fine although the UI is er um, less than stellar (their support is stellar).
Again, that 4 letter word beginning with F: Free.
I like it better than timeshift because you can image drives and/or partitions.
>64
It’s weird being the same age as old people. Live every day like it is your last; one day, it will be.
|
|
|
|
|
Once bitten, twice shy...I got hit hard a long time ago by some proprietary backup software that created files only it could read, and (I guess) one misplaced bit at the worst possible location was enough to make the entire file unusable. Of course that's when I needed to use the backup.
Since that time I've been sticking with robocopy; the entire tree is identical to the source, and no third-party software whatsoever is needed to access any specific file and without mounting anything. Of course that does nothing for bootable disks and the like, but considering I'm primarily interested in backing up VHD/VHDX files (and not physical disks), I can get away with it.
The entire "disk" is encrypted with VeraCrypt which, in a case like this, is preferable over a file container (.VC files). Since this is a hardware RAID, neither Windows nor VeraCrypt are really aware the underlying disk is actually a RAID (maybe they are, but since this is "supposed" to be transparent, I'm guessing they don't do anything special about it.
|
|
|
|
|
To add to the discussion, here are some thoughts and information:
- Poor throughput could be linked to the RAID5 being either degraded or still being built. I found some Youtube videos showing that (for your model) you would get about max 40 MB/s write speed in such a case, opposed to about 80 MB/s with a healthy array.
- As mentioned by others, these entry level enclosures have pretty low hardware specs and most often do software RAID inside the box. This can seriously impact performance.
- USB 3.0 throughput can reach 5 Gb/s, or 500 MB/s after overhead etc. USB 3.1 can do double that. USB 2 can do about 1/10 of that. So you are getting USB2 speeds at best.
- Have you checked transfer speeds on another PC?
- Finally, your NAS initially came out in 2010, which is a century ago in IT terms.
I run a pair of Asustor NAS with 2.5Gb/s ethernet interfaces. I get 250 MB/s read and write speeds with a simple Windows Explorer copy [^]. These NAS do however have a couple of NVMe SSD cache drives in them. A third NAS with 1 Gb/s and no cache drives gives a solid 100 MB/s. I have yet to test VM perfomance, but the NAS can expose iSCSI volumes so I am expecting to see similar performance from inside a VM.
So old that I did my first coding in octal via switches on a DEC PDP 8
|
|
|
|
|
As part of my analysis, I looked at the manufacturer's web page for the NAS box. It claims that the RAID is done in hardware.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
I have a WD PR4100 NAS configured to use RAID5 with 8TB WD Red drives. When writing to the NAS over a 1GB/s network, I get speeds of at least 60-100 Mbps. Unless you are writing large amounts of tiny files, I would expect at least that sort of performance out of any NAS.
USB 3 has a maximum transfer rate of 5 Gbps, and USB 2 has a maximum transfer rate of 480 Mbps. Even using USB 2, the transfer rates are very low. I would try each single drive in a different enclosure, if you have one, and then try one of the single drives in the NAS enclosure. This should give you enough information to determine whether it is one of the drives or the NAS that is the bottleneck.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
Daniel Pfeffer wrote: Unless you are writing large amounts of tiny files, I would expect at least that sort of performance out of any NAS.
So far I've only been trying to backup VM files - so, very few, but rather large files (multiple dozen GBs in size).
Daniel Pfeffer wrote: Even using USB 2, the transfer rates are very low
Agreed, I've been using other single drives using the same USB3 port and connector, and get much better speeds. So I have to rule out this beign stuck in USB 2 mode.
Your idea of testing out each drive separately is a good one. I wonder if I could power down the RAID, take one drive, put it in an enclosure and, without reformating it, run some sort of non-destructive speed test, then put the drive back in the RAID. I'd hate to rebuild it all over again at this point.
|
|
|
|
|
dandy72 wrote: I wonder if I could power down the RAID, take one drive, put it in an enclosure and, without reformating it, run some sort of non-destructive speed test
Given that you want to test WRITE speed I would say no, not unless you can find a non-destructive write test. I don't know of any test program that works like that.
You may also have to take into account the behaviour of the NAS. It may detect that the drive is part of a very degraded RAID-5 set and refuse to mount it. You may have to tell the NAS that this is a JBOD and have it reconfigure the drive before it can be accessed.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
