|
I really didn't see a proper forum for this question - logging begs toward philosophy and religion.The general forum has provided excellent feedback on bat files and glasses. So it's time to get serious. If there is a better place for this, don't post answers just point me and I'll relocate.
Logging. Application level. I've been in this business for a long time, and there does not seem to be a method to the madness. Most logging is reactive. Jeese, it stopped working guess I better monitor memory, etc. So logging is added after the fact. Well that helped - not. I'd like to hear from others what they just know they need to log. FWIW, I don't care about the application environment - web, server, desktop or embedded.
There is an article brewing... my opinion is that every application needs logging.
thanks all. 60 degrees in Atlanta today, nice breeze and sunny skies. Sorry Texas.
Charlie Gilley
<italic>Stuck in a dysfunctional matrix from which I must escape...
"Where liberty dwells, there is my country." B. Franklin, 1783
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
|
|
|
|
|
A philosophy question then.
We have four main types of logging:
- Traffic logging. The boring standard old stuff.
- Error logging. Reactive by definition
- Process logging. How many times did X happen per time interval
- Health logging. Preemptive.
Our Health logging is the most interesting because it's broken into a few main areas (disc, database, file and cache server) and is all defined using a set of rules in a JSON file that can easily be updated.
We build rules to run a test against systems that could be in error to spot conditions that may lead to an error, or may indicate that something else isn't working which may in turn cause an error. For instance, is memory getting low? Are we seeing too many network outages? Our background process queue is starting to fill up, we're not seeing as many votes as we'd expect, our cache server seems to be a bit slower than we'd expect. Each of the rules is run every 5 minutes and then we build a page that has a big green tick or a big red X. If the X appears our monitoring guys let us know and we can dig in and see what's happening before things get out of control.
Our logging / monitoring is always (almost by definition) going to be reactive, but over time we've built up a great set of rules that enable us to head issues off at the pass.
cheers
Chris Maunder
|
|
|
|
|
Chris Maunder wrote: Our logging / monitoring is always (almost by definition) going to be reactive, but over time we've built up a great set of rules that enable us to head issues off at the pass.
That seems to me to be the hallmark of a mature application that logs. After a certain point it goes from being totally reactive to somewhat preemptive.
Real programmers use butterflies
|
|
|
|
|
(5.) Security / bozo logging:
As in, "Who auto-generated those 500 purchase orders?" (That now have to be reversed).
True story; the new Accounting Manager seeing "what does this do?". The "action log" told the story.
(Interesting: CP changed my unbracketed 5 to a 1).
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
Gerry Schmitz wrote: Interesting: CP changed my unbracketed 5 to a 1
Ye - you have Markdown selected, so it gets a little ahead of itself sometimes)
But I do like your point: Auditing. Super important. Always nice to be able to point the finger.
cheers
Chris Maunder
|
|
|
|
|
I used worked to in automation systems that dealt with millions of dollars worth of product at a time - cassettes of wafers full of chips. We learned really, really fast that a production log was essential. We were often asked to explain why entire cassettes were scrapped and the answers boiled down to very poor training by the customer. Their operators somehow got into their head that if the slightest thing went wrong then click abort which effectively trashes all product in process - sometimes three or four cassettes at once. It had multi-level "are you really, really sure?" prompts but they did it anyway.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
|
|
|
|
|
Have a meta story on that subject.
When Windows Server 2000 was new I found the setting for "Audit logging" and thought: "Cool, have to test that".
Not much later, the server is basically unresponsive.
Have never touched that setting since. 
|
|
|
|
|
I don't like logging which is why I don't instrument for it until i have to, or if i'm working on a system that I can't debug for whatever reason.
I don't like it because
A) it's yet another thing to manage. Who cleans up your log files, just for example?
B) it's more work and in most applications on a modern machine I've written them such that I can identify the problem with a little bit of tinkering in a debugger. So it just doesn't pay for itself the way i code, or at least it doesn't pay for itself to do so preemptively
C) This is mainly a complaint i have with my C++ apps for reasons, including reasons of not having much of a unified way to do reliable logging that's consistent from app to app, but there's just so much buy in for logging and then usually it means injecting those dependencies into every single part of your code, which means all your libraries you produce and such have a fixed dependency on a logging subsystem, which now you have to manage as well. And across all the libs you've built that use it. Unless of course, you're just basically doing dirty old printfs...
I should add the caveat that I do not do enterprise application development anymore. If there were ever a good case for logging, it's on large incredibly complicated applications running on disparate, possibly remote systems.
Real programmers use butterflies
|
|
|
|
|
honey the codewitch wrote: enterprise application development anymore. If there were ever a good case for logging
ding ding ding ding!
you win.
#SupportHeForShe
Government can give you nothing but what it takes from somebody else. A government big enough to give you everything you want is big enough to take everything you've got, including your freedom.-Ezra Taft Benson
You must accept 1 of 2 basic premises: Either we are alone in the universe or we are not alone. Either way, the implications are staggering!-Wernher von Braun
|
|
|
|
|
honey the codewitch wrote: If there were ever a good case for logging, it's on large incredibly complicated applications running on disparate, possibly remote systems
Exactly our use case, and the reason why we built our own system. It's based on TCP/IP sockets and each software component contains a small server. Our client application can connect to as many of these servers as you like. The servers send messages (printf() on steroids) and the client can send commands back to the servers. The client can save traces to a file, and can be set up for recording continuously over a long term. It's proven invaluable given that our product runs on multiple computers and each component is heavily multithreaded. The client application is our oldest piece of software still in continual development. It got its start in 2000, and gets a new release every now and then as we discover a need for a new feature (or I get bored and want something fun to work on for a change  ).
Software Zen: delete this;
|
|
|
|
|
Right on! and it's encouraging that I'm not actually getting ripped apart in the comments over my remarks.
This place is really heavy on business development, i think because that's what most development is, and logging is big business. 
I'm in the minority it seems, in that I don't do bizdev, at least not anymore. I write commercial software, but it's not b2b or ecom or anything like that.
Logging is useful for me in certain situations, but in a lot of them it just isn't.
Real programmers use butterflies
|
|
|
|
|
In my case, I have embedded systems all over the world. There is no way to attach a debugger.
In addition to having thousands of systems all over, toss in multiple versions of code and a varied mix of connected hardware, all running in different production environments, and we have a real opportunity for understanding what data is necessary to collect to allow us to figure out what is wrong.
I do like the idea of different flavors of logging. We do this now on one side the system, but on mine, I'm just dumping into a text file what I think is useful. I'm going to think about dividing it up.
As for managing the log files, we set a max side and roll across 3 of them.
Charlie Gilley
<italic>Stuck in a dysfunctional matrix from which I must escape...
"Where liberty dwells, there is my country." B. Franklin, 1783
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
|
|
|
|
|
|
Thanks, I'll grab them now.
Charlie Gilley
<italic>Stuck in a dysfunctional matrix from which I must escape...
"Where liberty dwells, there is my country." B. Franklin, 1783
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
|
|
|
|
|
Last company, we logged everything; enter any method, its name and parameters logged.
I prefer mini-dumps.
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
charlieg wrote: I'd like to hear from others what they just know they need to log.
I've always found a stack trace and minidump to be very useful. Combining it with a log only helps to find what the user was doing when the exception occured which might not be of interest.
Best Wishes,
-David Delaune
|
|
|
|
|
Randor wrote: Combining it with a log only helps to find what the user was doing when the exception occured which might not be of interest.
This. I'm so glad you wrote this because it so succinctly describes what logging is for - tracking the user's movements through the app, rather than what the app is doing.
Now in my code on my jtagless embeddeds i have to log what the app is doing too, because i don't have a debugger, but generally - i'd say in most cases, you're right to bring this up.
If you're logging things other than the user's movement through the app you just might be using the wrong tool for the job.
Real programmers use butterflies
|
|
|
|
|
Yeah,
Situational awareness via telemetry is also important though. It helps developers learn what features are being used in your application (or operating system). By tracking the user movements through your application (or operating system) you can easily identify which features are not being used.
In the old days developers would add/design features and have no clue that absolutely nobody used it.
|
|
|
|
|
I'm going to remain hopelessly old fashioned in that regard and it has to do with something I wrote to someone earlier.
I won't take part in writing software that surveils users without their continuous and ongoing consent.
Now, gathering telemetry data doesn't necessarily *always* violate that. Figuring out if a feature is being used is kind of benign, but it all depends i guess on the level of gathering of information (are you trying to pick out their relative's birthdays from their contacts?_ and also what are you trying to do with that information? - telemetry is a muddy area for me and one where I'm wary.
At the end of the day, would I be okay with a neighbor looking at me the way the software is? I guess maybe that's a rule of thumb for me, but I just came up with it and i haven't really put it to the test.
But I will not write software that abuses or harms the people that use it, or makes it easier for other people to harm others, where I can help it. For me that covers a lot of ground, and surveilling without their continuous and ongoing consent comes under abusing others - for me.
I'm making moral arguments here, but I am not a believer in an objective morality - only that there's things I can live with and things I can't.
I haven't said any of this in judgment of others. I personally care elephant-all where other devs draw the line, but for me I have a vicious and fickle conscience and I cross it at my peril.
Real programmers use butterflies
|
|
|
|
|
Randor wrote: In the old days developers would add/design features and have no clue that absolutely nobody used it.
True story from the early 1970s: When I first visited the team that wrote the assembler for the machines we built, I asked about some of the more arcane pseudo-ops. Things like controlling listing width.
Turns out they were "sacrificial". There were features the devs wanted but management didn't grok, and the devs knew there would be pressure to trim the feature list. So they threw in a bunch of useless ones, so they could be noble later and trim the list at management's request (keeping their goodies, of course). The trimming wasn't perfect...
Cheers,
Peter
Software rusts. Simon Stephenson, ca 1994. So does this signature. me, 2012
|
|
|
|
|
I take a little bit different approach to logging. I work on automation systems and I have multiple levels of logging going on simultaneously. I'll mention just a couple. One is the trace buffer. It's always going and is essentially major events of note. The next one is what I call debug messages. These work like traces but have a mask associated with them that can be changed at run-time. The masks are bit-oriented and each bit is associated with a given functional aspect of the system. There are about two dozen of them defined now and there can be up to 63 because the zero bit is always set on. I wrote a special facility for this that uses shared memory to buffer up a whole bunch of messages and access is controlled by a mutex so multiple processes can use the same buffer. As it turns out, this is really, really fast which was my primary goal. I have measured the overhead for each message to be less than one microsecond on my old laptop. The same test was performed using printf calls to a console and the overhead was measured to be greater than 12uS. Writing directly to a text file is unacceptable for a variety of reasons. I tried that a few decades ago and abandoned it entirely. My systems involve multiple processes, each with multiple threads, and I worked for a long time to come up with a mechanism that solves all the problems and I'm pretty happy with this one. There's a lot more to it but I'll stop there.
"They have a consciousness, they have a life, they have a soul! Damn you! Let the rabbits wear glasses! Save our brothers! Can I get an amen?"
|
|
|
|
|
This is similar to what I do. I avoid mutex overhead by using a std::atomic_uint32_t as an index into a circular trace buffer, with each of its entries simply being a pointer to whatever holds the captured information.
|
|
|
|
|
Perhaps "time travel/reverse" debugging would be useful
|
|
|
|
|
That's very useful for debugging but out of the question on a live system. I can't count the number of times I've stepped over a function call, assuming the problem had to be further down, only to discover that I should have stepped into that function, and now the entire scenario has to be rerun from the beginning. 
|
|
|
|
|
Perhaps I do not understand time travel debuggers as I have yet to utilize one. Please permit me to inquire further re your response. Would not a time travel debugger permit stepping backward and entering the function you refer to after having discovering there is where the problem lies without having to restart? Thank You Kindly - Cheerio
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
