|
I've had experience with a large numbers of files in a directory. They load slow, if at all when you try to view them in the directory.
|
|
|
|
|
Richard Andrew x64 wrote: I think it's time for you to choose a user name instead of Member 7989122.
I don't know; see Peanuts for 30-Sep-1963 to 04-Oct-1963. 
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
Don't rush him. It's only been a little more than 9 years. He probably needs time to think of one... 
Anything that is unrelated to elephants is irrelephant
Anonymous
-
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-
Never argue with a fool. Onlookers may not be able to tell the difference.
Mark Twain
modified 14-Jul-20 6:18am.
|
|
|
|
|
What size files? Do they have to be files? What sorts of access? How frequently?
|
|
|
|
|
Why would the size of the files matter? Very few are small enough to fit in the available space of the directory entry.
Yes, they are files, by definition. Mostly, new files are added to the directory. This is the most common access. File access is far more infrequent.
|
|
|
|
|
Why would the size of the files matter? -- Because if the "files" are small enough, sticking them in some other cataloging system might be a better idea. Maybe a database, maybe a custom archiving system. Think of things like version control systems.
File access is far more infrequent. -- Then just do whatever you want, it won't matter.
|
|
|
|
|
In this case I would create a sub directory structure based on the date of file addition.
|
|
|
|
|
File size is critically important. If you're breaking across the block size by just a little bit, the rest of the block is dead space.
Assuming 4k block size and files storing 1K of data. That's 3k of wasted space on disk, per file.
If you zip up the files, they'll store much, much more efficiently.
We have this problem with hundreds of thousands of small text files. We sweep them up and zip them into archive folders on occasion to clean up the folders and reclaim disk space.
|
|
|
|
|
As far as I know, Windows itself doesn't mind it too much if there are lots of files in a folder. Explorer is an other matter. So you can put lots of files in a folder, but you can never look at them.
And FAT32 can only have 65534 files in a folder.
|
|
|
|
|
I hope to persue most users to go for NTFS rather than FAT32.
The most common access will be through an application, which will read the directory programmatically. Windows Explorer access can be considered an exception (although not that exceptional!).
|
|
|
|
|
If users would be copying these files to a USB stick for any reason, you may run into a problem as formatting a stick using FAT32 is a distinct possibility.
|
|
|
|
|
You could always format the USB stick with NTFS.
|
|
|
|
|
You could, but how many users actually read the documentation for your app?
|
|
|
|
|
Windows explorer will be your bottleneck ... while you sit and wait while it "builds" a 100k tree view. Odds are, it will "hang". "Reading" directories is not a big deal; how you "display" them is.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I’m “curious” why you “quoted” those “words” in your “post”.
JohnnyCee
|
|
|
|
|
Too lazy to use italics.
"Builds": iterating and instantiating.
"Hangs": no response or exceeding an acceptable response time.
"reading": file i/o
"display": where one loads a visual element for each file object.
Better?
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
I don't think emphasis is required for those words, but you do you.
|
|
|
|
|
It's from writing too many User Manuals.
As in: the CD disk "tray" is not a "cup holder."
Glad to know you and your users are more sophisticated and have time to sweat this stuff.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
IIRC, NTFS uses a B-tree variant to store file names in a directory. This guarantees fast access to a single file, but may slow down access if you are trying e.g. to enumerate all files in the directory.
FAT32 has a limit of just under 64K entries. The search is linear. Note that a long filename takes at least two entries - one for the short name and one for the long name.
I don't know how exFAT stores directories.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
File-access; so, mostly reading "files"?
A database would give you the most flexibility and performance.
--edit
You can easily expand Sql Server over multiple servers if need be, with more control over sharding and backups than with a regular filesystem.
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
NTFS uses database techniques for file management.
|
|
|
|
|
Which is not the same as using a database. The Dokan libraries have proven that a DB is very capable as a FS.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Member 7989122 wrote: If there are reasons to distribute the files over a series of subdirectories, what are the reasons (/explanations) why it would be an advantage?
If performance downgrade with number of files in a directory, there is only 1 explanation:
The directory is organized as a flat list of files, unsorted.
This imply that to find a file, you have to scan the list/directory sequentially. In O(n).
If an OS can have the directory sorted in the order you look for (file name), cost of finding a file is in O(log(n))
Patrice
“Everything should be made as simple as possible, but no simpler.” Albert Einstein
|
|
|
|
|
Maximum number of files on disk: 4,294,967,295
As already mentioned, the problems will start when you try to browse the disk in question with pretty much any existing application.
A better option would be to put the files in a database as blobs. At that point, you'll only have one file on the disk for the database itself. It wouldf also be easier to organize and manage than a complex folder hierarchy.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
We have some directories that contain that big number of files, the record I can remember right now is around 450k files in a folder.
They come from long time meassurements that trigger a data file a between 3 and 5 in a minute, each between 1 and 5 Mb.
Accessing the directory is slow, changing the order from name to timestamp is slow, moving the directory to another place is slow, getting the properties of the folder is slow, deleting the folder once is not needed anymore is slow.
Windwos 10 even slower specially the "folder properties" it needs over 15 minutes to count the files and give the size of the folder.
Windows 7 did it in 30 or 40 seconds.
We can't move that to FAT drives, due to number limitations as other said. Need to be NFTS.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
