|
This is about file systems in general, although with a primary emphasis on NTFS:
If you are expecting to stor a huge number of files - in the order of 100 k or more - on a disk, is there any significant advantage of spreading them over a number of subdirectories (based on some sort of hash)? Or are modern file systems capable of handling a huge number of files in a single level directory?`
If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage?
Is this differnent e.g among differnt FAT variants, and with NTFS?
|
|
|
|
|
I can tell you from experience that Windows does not do well with thousands of files in a single directory.
You will be much better off distributing them over many sub-dirs.
Off Topic: I think it's time for you to choose a user name instead of Member 7989122. 
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Can you provide an explanation of why it would be that way? Or is it at the "gut feeling" level?
|
|
|
|
|
People don’t relate well to numbers and this is a place where camaraderie is important. A name - even an obvious alias will make the interactions more personable.
If you can't laugh at yourself - ask me and I will do it for you.
|
|
|
|
|
Does that mean I am friends with OriginalGiff ?
|
|
|
|
|
I can't explain why that is, but it's quite simple to test. Write a small piece of code that copies an image file into the same directory multiple times. Doesn't have to be 100.000, I think 10-20.000 will suffice.
Then try to open that directory with Explorer.
That'll give you an idea about the problem. 
Anything that is unrelated to elephants is irrelephant
Anonymous
-
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-
Never argue with a fool. Onlookers may not be able to tell the difference.
Mark Twain
|
|
|
|
|
Probably because a windows folder isn't designed to contained 10,000 files, unlike database table which is expected to contain millions of rows. Or spreadsheets.
When we browse into a folder using windows explorer, it tried to read all file names inside that folder. There's no virtualization or partial loading. Reading 10,000 file names and extensions is surely detrimental.
EDIT : it's probably fine as long as you don't browse it using any explorer view.
|
|
|
|
|
At one point Microsoft actually recommended no more than 10,000 files per directory in NTFS. This was years ago, however.
The real reason is that file name scans inside a directory are sequential.
|
|
|
|
|
I've had experience with a large numbers of files in a directory. They load slow, if at all when you try to view them in the directory.
|
|
|
|
|
Richard Andrew x64 wrote: I think it's time for you to choose a user name instead of Member 7989122.
I don't know; see Peanuts for 30-Sep-1963 to 04-Oct-1963.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
Don't rush him. It's only been a little more than 9 years. He probably needs time to think of one...
Anything that is unrelated to elephants is irrelephant
Anonymous
-
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-
Never argue with a fool. Onlookers may not be able to tell the difference.
Mark Twain
modified 14-Jul-20 6:18am.
|
|
|
|
|
What size files? Do they have to be files? What sorts of access? How frequently?
|
|
|
|
|
Why would the size of the files matter? Very few are small enough to fit in the available space of the directory entry.
Yes, they are files, by definition. Mostly, new files are added to the directory. This is the most common access. File access is far more infrequent.
|
|
|
|
|
Why would the size of the files matter? -- Because if the "files" are small enough, sticking them in some other cataloging system might be a better idea. Maybe a database, maybe a custom archiving system. Think of things like version control systems.
File access is far more infrequent. -- Then just do whatever you want, it won't matter.
|
|
|
|
|
In this case I would create a sub directory structure based on the date of file addition.
|
|
|
|
|
File size is critically important. If you're breaking across the block size by just a little bit, the rest of the block is dead space.
Assuming 4k block size and files storing 1K of data. That's 3k of wasted space on disk, per file.
If you zip up the files, they'll store much, much more efficiently.
We have this problem with hundreds of thousands of small text files. We sweep them up and zip them into archive folders on occasion to clean up the folders and reclaim disk space.
|
|
|
|
|
As far as I know, Windows itself doesn't mind it too much if there are lots of files in a folder. Explorer is an other matter. So you can put lots of files in a folder, but you can never look at them.
And FAT32 can only have 65534 files in a folder.
|
|
|
|
|
I hope to persue most users to go for NTFS rather than FAT32.
The most common access will be through an application, which will read the directory programmatically. Windows Explorer access can be considered an exception (although not that exceptional!).
|
|
|
|
|
If users would be copying these files to a USB stick for any reason, you may run into a problem as formatting a stick using FAT32 is a distinct possibility.
|
|
|
|
|
You could always format the USB stick with NTFS.
|
|
|
|
|
You could, but how many users actually read the documentation for your app?
|
|
|
|
|
Windows explorer will be your bottleneck ... while you sit and wait while it "builds" a 100k tree view. Odds are, it will "hang". "Reading" directories is not a big deal; how you "display" them is.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I’m “curious” why you “quoted” those “words” in your “post”.
JohnnyCee
|
|
|
|
|
Too lazy to use italics.
"Builds": iterating and instantiating.
"Hangs": no response or exceeding an acceptable response time.
"reading": file i/o
"display": where one loads a visual element for each file object.
Better?
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I don't think emphasis is required for those words, but you do you.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
