|
System commit limit sounds like it might be the amount of dirty memory it can write to the swap, or maybe that it can use as swap? Not sure, but I'm more inclined to think it's the former rather than the latter, if any.
Edit: I'm surprised it wouldn't be expressed in pages rather than GB though.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Step away from the misbehaving application for a moment. Fire up a VM running the Windows OS you want, and write a bad, evil application. It does not have to be complex. Using a VM allows you to crash/hang it instead of your main box. Try malloc (comes off the heap) and static allocation (process space?). I'm rusty. Just play around.
That said, trust nothing Microsoft "documents." I fully confess that I am jaded. Trust but verify.
I have had my left leg in the embedded world and the right leg on the desktop. My two common errors has been buffer overflows into malloc'd areas and stack overflows. Either way, it's an application issue.
Keep learning.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
I need to amend my earlier statement about 32-bit processes under Windows.
While it's not important, historically the process address space was divvied such that the lower 2GB was "user space" and the upper 2GB was "kernel space" (I may have that backward, but either way, it's half and half)
Some apps could be "3GB aware", sometimes run with a command line switch like /3GB to enable it. In that case, the kernel was only mapped to 1GB of the address space. I'm not sure why all apps weren't this way, other than compatibility. An example of a 32 process that could be 3GB aware with a command line switch is the old 32-bit versions of Image Line's FL Studio DAW software.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
For Linux and friends, start with man getrlimit and apropos oom . The whole OOM-killer environment is intriguing, to say the least. I believe it came from the *nix legacy of lots of users timesharing a machine with limited resources, particularly memory. A competitive, rather than cooperative, workload.
Software rusts. Simon Stephenson, ca 1994. So does this signature. me, 2012
|
|
|
|
|
raddevus wrote: So when a developer notices that the app he wrote running on
Presumably this is your real question and seems like it was already answered.
However one somewhat related gotcha with C# (if relevant) is the Large Object Heap. Which means an app can "run out" of memory long before physical/virtual memory is used.
|
|
|
|
|
jschell wrote: C# (if relevant) is the Large Object Heap. Which means an app can "run out" of memory long before physical/virtual memory is used.
Both of the Apps (Services) in question are indeed written in C# and the point you make is highly relevant and I really appreciate you mentioning it. Very helpful.
My question really hasn't been answered.
I am still very curious if a 64 bit app can eat all of the memory on a large server (64GB RAM or something larger).
I'm guessing that it cannot since
1. I believe that any app cannot allocate RAM beyond its address space.
2. This would be a huge security hole since any malicious app could just eat all RAM
Someone mentioned a "leak" but even a leak is bound to the processes address space & once the process has eaten / leaked all that RAM then it would throw the "out of memory" exception.
I'm just not sure if the OS allows address spaces to be much larger than 4GB and if it does what the "default" value would be for each address space on 64 bit Windows OS (Server).
Very difficult thing to find.
I would write an app that eats all the RAM -- and have done that, but the box I have is limited to 8GB or even 16GB and you need more RAM and running the Win Server OS to really determine this.
|
|
|
|
|
raddevus wrote: I am still very curious if a 64 bit app can eat all of the memory on a large server (64GB RAM or something larger).
I'm guessing that it cannot since
1. I believe that any app cannot allocate RAM beyond its address space. If you by 'address space' refer to the entire 2**64 bytes space that a 64 bit process can cover by its addresses, 16 exbi bytes (more than 16 million gigabytes), your assumption is right: A process cannot allocate that much space. And we will never see a computer with 16 exbi bytes of RAM. Never ever.
In no general machine (excluding e.g. embedded processors) of the last 30-40 years has the address indicated by the program code been used directly as the physical RAM address. The virtual address in the program is translated to a different physical address in RAM through a set of hardware translation tables, managed by the OS, called the Memory Management System (MMS). Each process has its own set of MMS tables. The OS sets up the MMS tables for a tiny slice of the virtual address space. If the program presents a virtual address within this slice, the range covered by the MMS tables for that process, it is translated to a physical RAM address. If the virtual address is outside the range covered by the MMS tables, an interrupt is generated, and the OS will terminate the process. (Well, it might offer a mechanism for reporting the interrupt e.g. to a debugger that can inspect the process state before it is cleaned out.)
If you by 'its address space' refer to just that slice of the total 64 bit virtual address space for which the OS has set up translation tables, then you are essentially right. The size of this slice can be a few hundred kiB, a few GiB, or many GiB - but the OS will not give you more than it is capable of handling.
When an app allocates RAM, the allocated space is, at the outset, within the address space translated by its MMS tables. If the malloc/new/... maps down to an OS request, the OS may say: 'There isn't enough unused space in the already mapped virtual address space, so I have to add another entry to the mapping tables, expanding the address space available to that process'. Before the OS does that, it will check that the process does not already control an excessive amount of address space. The limit is set by the OS to any value that it can handle.
In many systems, malloc/new/... starts out as a call to a runtime library in the process address space. As long as there is allocated, but unused, space available, the OS (and MMS) knows nothing about the allocation - it is nothing but making use of resources already allocated to the process. The library function has been given a range of valid addresses (the 'heap') to manage; allocation is left to that manager function. The OS may provide a function for the heap manager, so that if the heap overfills, the heap manager can ask the OS for an extension of the process user space, just like when the OS takes care of the entire allocation task: The OS will check that one process doesn't run away with a too big share of the resources.
2. This would be a huge security hole since any malicious app could just eat all RAM The MMS translation tables are managed by the OS; a user process doesn't have the privilege. So any extension of the address space, i.e. all new translation entries added to the tables, is done by the OS. If it accepts to give any process as much address space as is asked for, with no limits, no questions asked, no check against what the process already has, then You Asked For It, You Got It. Any decent OS will make checks, and put a cap on the virtual memory allocated to each process.
So the app can eat up all the (virtual) memory explicitly given to it by the OS that says 'Here, these addresses are yours, for now'. The OS takes the responsibility for mapping all those addresses to valid RAM. But no more. Any other address, not endorsed by the OS, causes a fatal interrupt.
An important point: When the app presents a virtual address to the MMS tables, the table entry may (and usually will) contain the physical RAM address. The table entry has a number of additional flags: One of them may indicate that the address is not a RAM address, but an address in a 'swap file' or 'page file'. This causes a (non-fatal) 'page fault' interrupt: An OS routine is triggered to first make sure that there is a free, unused RAM page. If not, it will select one of the active ones to be written to the page file to free up a memory page, its table entry is changed from the its RAM address to the file address it is written to, and the appropriate flag is set to show that this page is now on disk, not in RAM. Then the requested page is read from disk into the free RAM page, the disk address is changed to the RAM address in the MMS table entry, and the flag is reset, the one that said that this page resides on disk.
This entire operation takes so much time that the OS will let another process use the CPU while it waits for the disk access to complete. The requesting process is suspended. When the disk page has been read into RAM, the suspension is lifted. The process will retry the memory access, which will now succeed without generating an interrupt.
In the days when RAM was expensive, the total sum of virtual address space allocated to processes could be many times the RAM size. The OS might allocate virtual addresses up to the amount of disk space it could spare for the page file. In those days, you could bring a system to a near-halt by initiating a large number of processes requesting so much memory that almost all of it would be in a page file, and referencing data 'all over'. So, upon the first page fault, the OS lets a second process use the CPU, almost immediately causing another page fault, that is queued up for processing. A third process gets the CPU, causing yet another page fault, and so it goes on, the list of suspended processes growing, the disk being busy more or less 100% of the time, and any operation requesting disk data (that includes a lot of OS functions!) stalled in wait for disk capacity. This was referred to as 'thrashing', when the machine spends almost all its resources on managing waiting queues, with minimal resources available to complete tasks to clean up the traffic jam.
Today, RAM is so cheap that few active developers have experienced real thrashing, so bad that you had to restart the machine. Lots of people don't even know the meaning of the term 'thrashing'. Maybe OSes still come with thrashing prevention mechanisms: If a dangerous build up of queues is detected, maybe half of the processes are suspended, to let the other half complete its job first. The suspended ones will experience a grave delay, but that is better than none of them being able to complete. Maybe today's OSes have removed such processing, thinking them no longer needed.
Someone mentioned a "leak" but even a leak is bound to the processes address space & once the process has eaten / leaked all that RAM then it would throw the "out of memory" exception. This occurs within the valid address space of a single process. If the OS does not provide a function for extending the address space, no other process is affected by the leak. If the OS does honor extension requests without question, it may of course run out of paging file space. (Or, if you have disabled paging, which is possible at least under Windows, the RAM size may be the limit.) Any decent OS should allocate an unreasonable large address space, whether in the page file or in RAM.
I'm just not sure if the OS allows address spaces to be much larger than 4GB In a 64 bit environment, 4GiB is no 'natural' limit. It is set by the OS to any value it finds reasonable. I would say that if you expect to have a real need beyond 4GiB, then it isn't too much to ask that you indicate your needs to the OS at startup to have the limit raised for you process. The OS may have a default limit of 4GiB. That doesn't imply any real allocation of that size, it just puts a cap on the total memory that the process may request while running, whether by heap allocation, adding DLLs to its address space or otherwise.
I would write an app that eats all the RAM -- and have done that, but the box I have is limited to 8GB or even 16GB and you need more RAM and running the Win Server OS to really determine this. Unless you turn paging off, when the OS runs out of free RAM, it will start throwing pages out to the paging disk before you manage to crash the system by allocating 'all' RAM to your process. In early, primitive OSes you might succeed in crashing the machine that way, but not with any modern, general OS. ("Modern" goes as far back as the 1970s.)
To study how memory is used, I'd recommend the SysInternals RAMMap program (RAMMap[^]). Even the standard Resource Monitor ('Memory' tab) give you a lot of info, but RAMMap gives you more, and it has function (in the 'Empty' tab) to do a cleanup, throwing out of RAM everything that doesn't need to be in RAM; it can be fetched later if needed.
Religious freedom is the freedom to say that two plus two make five.
|
|
|
|
|
Wordle 1,189 4/6
🟩⬜⬜🟨⬜
🟩⬜🟩⬜🟩
🟩⬜🟩🟩🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 1,189 4/6*
⬜⬜⬜⬜🟩
⬜⬜⬜🟨🟩
🟩⬜⬜⬜🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 1,189 3/6
⬜🟨⬜🟨🟩
🟩⬜🟩⬜🟩
🟩🟩🟩🟩🟩
"A little time, a little trouble, your better day"
Badfinger
|
|
|
|
|
Wordle 1,189 5/6*
⬜⬜🟩🟨🟩
🟩⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩🟩🟩🟩🟩
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
⬜⬜⬜⬜🟩
⬜⬜🟩⬜🟨
⬜⬜🟩🟨🟩
🟩⬜🟩⬜🟩
🟩🟩🟩🟩🟩
In a closed society where everybody's guilty, the only crime is getting caught. In a world of thieves, the only final sin is stupidity. - Hunter S Thompson - RIP
|
|
|
|
|
Wordle 1,189 3/6*
⬜⬜🟨⬜⬜
🟨🟨⬜⬜🟨
🟩🟩🟩🟩🟩
Happiness will never come to those who fail to appreciate what they already have. -Anon
And those who were seen dancing were thought to be insane by those who could not hear the music. -Frederick Nietzsche
|
|
|
|
|
Wordle 1,189 4/6
⬛⬛🟩⬛🟩
🟩⬛🟩⬛🟩
🟩⬛🟩🟩🟩
🟩🟩🟩🟩🟩
Ok, I have had my coffee, so you can all come out now!
|
|
|
|
|
Wordle 1,189 4/6*
⬛⬛🟩⬛🟩
🟩⬛🟩⬛🟩
🟩⬛🟩🟩🟩
🟩🟩🟩🟩🟩
|
|
|
|
|
Wordle 1,189 3/6
⬜⬜🟩⬜🟩
🟩⬜🟩⬜🟩
🟩🟩🟩🟩🟩
Within you lies the power for good - Use it!
|
|
|
|
|
I'm the kind of stuck that doesn't resolve itself to a programming question.
As an adult I used to work with someone I taught coding to back when we were teens, and we'd buddy code, and accomplish things neither one of us could do on our own. We worked really well together.
Eventually he went off to NYC to make obscene amounts of money.
But that's the kind of help I need. This SVG stuff is absolutely kicking my ass.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
honey the codewitch wrote: off to NYC to make obscene amounts of money
Sorry, can't help you there.
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
|
Quote: [...] obscene amounts of money.
honey the codewitch wrote: [...] that's the kind of help I need. Me too.
On the serious side, I strongly doubt the magician would happen again.
"In testa che avete, Signor di Ceprano?"
-- Rigoletto
|
|
|
|
|
Get a rubber duck*?
* Made a typo there once, made for an interesting conversation
|
|
|
|
|
Well it's one of those problems where something has gone dramatically wrong and I don't even know where in the codebase to start looking. It's gotten to the point where I've decided to support TinyVG instead, eventually in addition to SVG
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Sander Rossel wrote: Made a typo there once, made for an interesting conversation
A ... robber duck ?
|
|
|
|
|
Yeah, let's dick to that
|
|
|
|
|
Take a break from it for a while, if you can. Don't know how many times I've been stuck and taken a break and when I came back to it...bam.
|
|
|
|
|