Click here to Skip to main content
15,396,078 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.

 
GeneralRe: Flushing to paging disk Pin
trønderen16-Jan-21 6:39
Membertrønderen16-Jan-21 6:39 
GeneralRe: Flushing to paging disk Pin
Mircea Neacsu16-Jan-21 5:56
mvaMircea Neacsu16-Jan-21 5:56 
GeneralRe: Flushing to paging disk Pin
trønderen16-Jan-21 6:25
Membertrønderen16-Jan-21 6:25 
GeneralRe: Flushing to paging disk Pin
Mircea Neacsu16-Jan-21 7:33
mvaMircea Neacsu16-Jan-21 7:33 
GeneralRe: Flushing to paging disk Pin
Gary R. Wheeler16-Jan-21 7:02
MemberGary R. Wheeler16-Jan-21 7:02 
GeneralRe: Flushing to paging disk Pin
trønderen17-Jan-21 4:13
Membertrønderen17-Jan-21 4:13 
GeneralRe: Flushing to paging disk Pin
Gary R. Wheeler18-Jan-21 10:12
MemberGary R. Wheeler18-Jan-21 10:12 
GeneralRe: Flushing to paging disk Pin
trønderen19-Jan-21 3:56
Membertrønderen19-Jan-21 3:56 
Gary R. Wheeler wrote:
Do you think it's possible the reason file I/O is very low is because I/O is being satisfied from cache a large amount of the time?
Accessing disk in large chuncks is certainly more efficient than doing it in small chunks, whoever does it (the application, OS, disc driver or disc firmware). As long as you access a contiguous sequence of disk sectors, the time cost and disk load is almost constant, independent of data volume. Before disks with RAM caches were common, before the OS did much buffering, your high performance application might read 64 KiByte at a time and gain a lot of speed. (And for that purpose, keeping your FAT disk defragmented was essential!).

DOS did no buffering; it already occupied 384 KiByte of the 1 MiByte address space available, leaving 640 KiByte to the application (well known fellow allegedly considered that it "ought to be enough for anyone"...).

The law of diminishing returns soon comes to play. Reading beyond the end of the file fragment is a waste (your application or OS won't do it; the disc cache has no awareness of fragment limits). If you have flagged your NTFS file as encrypted or compressed, it is anyhow processed in 64 KiByte chunks. At least some RAID solutions does striping in 64 KiByte chunks. Quite a few files - by number, not by total volume - are less than 64 KiByte in size, or not very much more (in particular in software development environments). The performance benefit of reading up to 64 KiByte chunks may be significant, but for the all over system performance, it drops rapidly off beyond that.

Today, RAM is so cheap that we uncritically buffer, whether beneficial or not. The benefit of OS prefetching (i.e. transferring large chunks) has diminished a lot the last few years, due to a couple of other fairly recent (on a historical scale) developments:

Nowadays, most system disks (and almost all new ones), and an increasing share of data disks, are solid state - still slower than RAM, but the factor is more like one to ten, rather than one to ten thousand. If you turn off all buffering, always reading a single page at a time, flash disk slowdown would not be much noticeable on application performance; speed would be almost like before.

Second: Most new magnetic discs have on-disk RAM buffers, reading an entire track (or a significant portion of it) into their own RAM, whether asked to or not. On the next single-page request from the OS, data goes from one RAM buffer (in the disc) to another RAM buffer (in OS managed memory), at a speed usually limited only by the disk interface.

Certainly: If your application makes 16 single-page (4 KiByte) disc accesses rather than a single 64 KiByte access, management work done by the CPU is higher. If the OS doesn't find the pages in its own buffer, it may have to make up to 16 separate disc accesses. This takes some CPU capacity as well. Yet you never see the CPU load rocket when you access the disk. CPU load is insignificant.

Opening or creating a file may require quite a few disc accesses, for accessing / updating its directory, reading or allocating an MFT entry, updating the allocation bit map (create, write), ... These are file system structures that the OS repeatedly accesses, and can benefit from caching. But they are OS owned data, not user data.


I could see performance improvements from a large cache even for typical consumer apps like web browsers: video, images, large data chunks, and so on.
Tuning web caching may be quite different from tuning disc caching, but they do share some characteristics. For video and large data chunks: How often do you watch that same video again, while it is still in memory? I'd say: Not very often. You download some huge software - say, a new OS image. How often do you repeat that download before the first one is our of the cache? Web caching saves a lot of tiny little transfers, such as logos or icons used on every page presented by a web site. But first: The cache is maintained in the file system, by the browser - not in RAM by the OS.

Second: HTTP allows an expiry time for a chunk of data (such as a logo), but many web sites are lazy at setting this properly, so web browsers commonly make a request anyway, asking if the logo has been updated recently. If it hasn't, there is no need to transfer those two hundred bytes again. Maybe it took a couple thousand bytes to save the transfer of two hundred...

Images are in a middle-between (and they are usually cached by the browser): I have tried timing web newspaper front pages with a couple dozen of photos, before and after a complete cleaning of the browser cache. The first access was measurably slower, when all the logos and icons had to be retransferred. On the second access to the front page, the speed was the same as before the cleanup. So the caching obviously gave some speedup. Not much, but measurable.

Yet, the trend today is exactly the opposite: Lots of sites presenting scores of images in a huge display, deliberately do not fetch all of them, but only those currently visible in the browser window. Do not waste resources on retrieving anything that might not be needed after all!

This may be justifiable, considering the speed of an Internet connection vs. the speed of a flash disk transfer. Also, the probability of a disk file user accessing the entire file may higher than that of a web user wanting to see the complete display of all pictures. Also, the difference between an application (the web browser) managing a cache in disk files, vs. the OS managing a cache in RAM, is quite significant. So several considerations regarding caching cannot be directly transferred from the one area to the other.
GeneralRe: Flushing to paging disk Pin
Kornfeld Eliyahu Peter16-Jan-21 8:44
professionalKornfeld Eliyahu Peter16-Jan-21 8:44 
GeneralRe: Flushing to paging disk Pin
trønderen17-Jan-21 4:49
Membertrønderen17-Jan-21 4:49 
GeneralRe: Flushing to paging disk Pin
Slacker00716-Jan-21 9:38
professionalSlacker00716-Jan-21 9:38 
GeneralRe: Flushing to paging disk Pin
trønderen16-Jan-21 11:35
Membertrønderen16-Jan-21 11:35 
JokeThe secret to great coffee Pin
Cp-Coder16-Jan-21 1:55
MemberCp-Coder16-Jan-21 1:55 
GeneralRe: The secret to great coffee Pin
Mike Hankey16-Jan-21 3:23
professionalMike Hankey16-Jan-21 3:23 
GeneralRe: The secret to great coffee Pin
Gary R. Wheeler16-Jan-21 7:04
MemberGary R. Wheeler16-Jan-21 7:04 
GeneralRe: The secret to great coffee Pin
trønderen16-Jan-21 9:13
Membertrønderen16-Jan-21 9:13 
GeneralRe: The secret to great coffee Pin
W Balboos, GHB17-Jan-21 7:42
mveW Balboos, GHB17-Jan-21 7:42 
GeneralRe: The secret to great coffee Pin
BryanFazekas19-Jan-21 1:13
MemberBryanFazekas19-Jan-21 1:13 
GeneralRe: The secret to great coffee Pin
W Balboos, GHB19-Jan-21 1:16
mveW Balboos, GHB19-Jan-21 1:16 
GeneralRe: The secret to great coffee Pin
BryanFazekas19-Jan-21 1:18
MemberBryanFazekas19-Jan-21 1:18 
JokeRe: The secret to great coffee Pin
W Balboos, GHB19-Jan-21 1:20
mveW Balboos, GHB19-Jan-21 1:20 
GeneralI can't believe I missed this! Pin
Sander Rossel16-Jan-21 0:34
professionalSander Rossel16-Jan-21 0:34 
GeneralRe: I can't believe I missed this! Pin
OriginalGriff16-Jan-21 0:50
mveOriginalGriff16-Jan-21 0:50 
GeneralRe: I can't believe I missed this! Pin
RickZeeland16-Jan-21 1:00
mveRickZeeland16-Jan-21 1:00 
GeneralRe: I can't believe I missed this! Pin
Mike Hankey16-Jan-21 3:23
professionalMike Hankey16-Jan-21 3:23 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.


Straw Poll

Do you expect users to read the instructions when using your apps?
  Results   127 votes