|
Look at the points where you expose data from the backend (typically a REST service or similar). If an exception is raised, you generally speaking just return "500 Internal Server error" without any further details - and (very important) log those details on the server.
In case you use exception to trigger responses like like "404 NOT FOUND" you need to make sure they do not show stack traces or internal details as well.
This is typically done in a central place in the response handler of the API so it will automatically kick in for any exception raised - you need to write this code and test it so you trust it no matter which exception is raised from the lower levels of the code. Then you do not need to hunt your entire code base for possible exceptions - which is an impossible task anyway as some system exceptions can be raised from anywhere.
Java tried to get the code to declare all exceptions so you could track it down (checked exceptions) but it was not a great success.
UPDATE: And make sure you only catch exceptions in 2 cases:
1) When you can FIX the issue - i.e. no error will be returned to the client, and hence no information will be leaked. Example: You want to read some data from a disk cache. If the read throws an exception catch it and recalculate the data.
2) When you can throw an exception with more details. Example: You try to read the data but the exception just says "Access denied". You know what you where trying to read and which user you where running as, so catch it and throw a new exception adding these datails - keeping the original exception as inner exception.
|
|
|
|
|
Isn't there an option to hide that?
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Probably, but hard to say without any details. I am surprised it is not the default behavior of whatever framework he is using, so maybe the default was just changed to make it easier to debug. In ASP.NET you do for example get the "to read the error details, change this setting" message - and someone not understanding the impact might just have followed these instructions. In that case the only thing he needs to do is revert that setting. But I am not going to google the solution to every imaginable framework out there, just because the poster forgot to mention what they are using. 
|
|
|
|
|
lmoelleb wrote: I am surprised it is not the default behavior of whatever framework he is using, so maybe the default was just changed to make it easier to debug.
It is hidden by default because of hackers.
Details should be hidden if it is not on localhost.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Of course. That is kind of what I have been saying the entire time
|
|
|
|
|
I'm currently on coffee two.
Not entirely alive yet.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
If you go and do a "replace all"..
..then have your exceptions inherit from a custom class inheriting from Exception. For logging, use FirstChanceException as suggested by Gerry; it will show a lot more than the exceptions you throw yourself.
Bastard Programmer from Hell
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Simon_Whale wrote: list all the exceptions that either the code catches or raises so that I can do a review
That process would need to find that those in all of the libraries that you use also.
Simon_Whale wrote: reason for the review is the result of a pen test,
Probably because you have an API and you are returning the exception stack trace as part of the API error handling. The solution to that is the following
1. Catch all exceptions so "catch(Exception e)" in your API layer.
2. Log the exception. You should of course already be doing this.
3. Return an error result from the API. Do not return the exception. Probably 500. But over time you can differentiate this in the error handling code.
I would suggest that you also create an id and log that as well and return that as part of the error message. That way you can track an error reported from a caller back to the specific log entry. You can use a GUID but it is going to be more user friendly, especially if a UI is involved, if you create a simple id. It can rotate over time because the context of the error (when reported) is enough to localize in the logs.
|
|
|
|
|
C# >= 4
Curious to know if you've had to deal with the issues described here.
1) assume you have a generic method of Type T
2) in that method a Func<IEnumerable<T>, IEnumerable<T>> is called on an IEnumerable of T
3) the result of calling the func is in a variable 'temp ... when you examine 'temp in the Command Window at run-time after hitting a break-point:
> ? temp
{System.Linq.Enumerable.WhereListIterator<int>}
Current: 0
Results View: Expanding the Results View will enumerate the IEnumerable
> ? temp == null
false
> ? temp == default
false
> ? temp is null
false
> ? temp.Any()
false
> ? temp.Count()
04) in the case where the Func call results in no content (no items)
4a) you want to evaluate whether 'temp has any "items" ... i.e., has a Count of #0 (as shown above)
4b) but, you do not want to trigger iteration/enumeration of 'temp by calling Count()
I have read that .Any() will not cause iteration in this very interesting SO thread: [^].
And, on the same thread: [^]: is mention of a new C#6 Linq function 'TryGetNonEnumeratedCount
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
There is no way to do it in general, not even in theory.
TryGetNonEnumeratedCount is neat, but only actually outputs the count when it is called on a collection that knows its count. For general enumerables such as WhereListIterator, it returns false. So OK, it doesn't evaluate the enumerable, but it also doesn't help.
Any on a WhereListIterator has to evaluate until it finds an element or runs out of input. There's nothing else it could do. BTW Any includes TryGetNonEnumeratedCount-like tricks, examining the type of the object it is called on, and if it is an ICollection then it just checks whether its count is zero (there are additional details but they don't help here either), but those tricks cannot apply to a filtered enumerable.
If there was any method that could in general predict whether a filtered enumerable is going to have anything in it (without iterating through it), that would have strange implications. It would act as a powerful oracle (of the theoretical computer science variety) that can extract information from a black box function without ever actually evaluating the function. Spooky.
|
|
|
|
|
thanks, Harold, as I'd expect from you, a very thoughtful reply.
I've found very little documentation on 'TryGetNonEnumeratedCount so far. My impression in the SO thread cited is that this new Linq method is somehow more efficient.
Given .Any short-circuits when it encounters a first instance: I find myself wondering if under-the-hood .Any is in any way more "efficient" in a function that returns an IEnumerable ... compared to invoking Count().
In this context: in a loop where a series of functions are called in sequence, and the goal is to exit the loop when any function returns an empty IEnumerable ... .Any can be used while testing for null or == default are useless.
cheers, Bill
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
TryGetNonEnumeratedCount is always efficient, but that's because in the case that the enumerable doesn't already know its count, instead of falling back to counting elements one-by-one (as Count does), it just gives up. It is Count but without the default/slow path, only the special cases. I wouldn't really call it "more efficient", it's more like a niche thing that you can use when you would like to have a count but you don't really need it (such as to create a collection with an initial size, which isn't worth counting one-by-one for, but if getting the count is basically free then you may as well use it).
Any can certainly be more efficient than Count, thanks to the early-out. In some cases Count is actually more efficient, but let's put that aside, that's not really relevant to the case you're dealing with.
So far I have no specific reason to think that there is any better solution than Any. Maybe if there is some special property of the Mystery Function that's being called, or of the overall algorithm, there may be some other technique.
|
|
|
|
|
Thanks, again !
A few gaps in (my understanding of) the puzzle:
1) "TryGetNonEnumeratedCount ... in the case that the enumerable doesn't already know its count, instead of falling back to counting elements one-by-one (as Count does), it just gives up"
If an IEnumerable "knew" its count ... is that a contradiction in terms ? Clearly, some collections "know" their count, because it is a Property not a method.. But, a generic IEnumerable created at run-time ?
2) "In some cases Count is actually more efficient"
fascinating; I try to imagine a case of a generic IEnumerable where Count() would be more efficient ... but, draw a blank
3) if you had a function that returned a generic IEnumerable, and in that Function you used Count(): would there be any structural differences between that returned result and a returned result IEnumerable you had never used Count on.
4) comparing a a generic IEnumerable you used Count() on with one you used Any() on: would there be any structural differences ?
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: If an IEnumerable "knew" its count ... is that a contradiction in terms ?
Not really. For example, a List<T> is also an IEnumerable<T>; the List<T> "knows" its count.
The TryGetNonEnumeratedCount method simply attempts to cast the IEnumerable<T> to a few "pre-counted" interfaces at run-time, and returns the Count property from the first one that succeeds. You can see the source code on GitHub:
runtime/Count.cs at 57bfe474518ab5b7cfe6bf7424a79ce3af9d6657 · dotnet/runtime · GitHub[^]
It tries ICollection<T>, IIListProvider<T>, and the non-generic ICollection. That should cover most cases, although it notably doesn't try the IReadOnlyCollection<T> introduced in .NET 4.5 - I can't find an official comment, but this SO thread[^] discusses that.
You can see the original proposal for this method on the runtime repository:
Non-enumerating Count Linq Method · Issue #27183 · dotnet/runtime · GitHub[^]
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Thanks,Richard Deeming wrote: a List<T> is also an IEnumerable<T>; the List<T> "knows" its count.
i find this a specious comparison, in the context of this thread; i would distinguish between an IEnumerable created at run-time that has been "topped up" in code, and a List that has been initialized, and had values added.
With 'List 'Count is a Property, with 'IEnumerable 'Count() is a method call that iterates over the 'IEnumerable using 'MoveNext ... if the Enumerable is not other certain Types, like an IList.Richard Deeming wrote: The TryGetNonEnumeratedCount method simply attempts to cast the IEnumerable<T> to a few "pre-counted" interfaces
The source for IEnumerable.Coubt() uses the same short-circuiting behavior.
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: I find this a specious comparison, in the context of this thread
Not sure I follow you here. You're writing code which can operate on an instance of any class that implements IEnumerable<T>. List<T> is such a class.
Your code wants to get the number of items in the source sequence, but only if that operation doesn't involve iterating over the entire collection. If the method was passed a List<T>, you can do it. If it was passed the result of an iterator method (for example), it can't.
It's all about examining the run-time type of the variable to access properties that aren't available on the compile-time type. From a purist perspective, that's not a good approach. But from a practical perspective, there are times when it's useful.
BillWoodruff wrote: The source for IEnumerable.Coubt() uses the same short-circuiting behavior.
The difference is, if the value passed in doesn't match one of the specific interfaces, Enumerable.Count(IEnumerable<T>) will proceed to enumerate the entire sequence, whereas TryGetNonEnumeratedCount will give up.
For example, given:
public static IEnumerable<int> Answers()
{
while (true)
{
yield return 42;
}
} using:
if (Answers().TryGetNonEnumeratedCount(out int count))
{
Console.WriteLine(count);
}
else
{
Console.WriteLine("Nope");
} will output "Nope", whereas:
int count = Answers().Count();
Console.WriteLine(count); will never return.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Hi, Richard, thanks !
Perhaps i have not emphasized enough that this thread is about a not-iterated over IEnumerable.Richard Deeming wrote: Not sure I follow you here. You're writing code which can operate on an instance of any class that implements IEnumerable<T>. List<T> is such a class.
Yes, the input to the first Func<IEnumerable<T>, IEnumerable<T>> in the series is an instance of a collection: the subsequent Func<IEnumerable<T>, IEnumerable<T>> calls take as input the previously created IEnumerable<T> results.
Because getting the count is not necessary here, 'Any() appears to be the only possibly efficient way to see if the IEnumerable is not "empty." It will have to do
i may be too "picky" here (based on how i see students respond to this topic) when i express my semantic discomfort with the equating of an instance of List<T> with IEnumerable<T>/
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
Ah, but this sub-thread was a digression down the avenue of your question:
BillWoodruff wrote: If an IEnumerable "knew" its count ... is that a contradiction in terms?
BillWoodruff wrote: equating of an instance of List<T> with IEnumerable<T>
Logically, every List<T> is an IEnumerable<T>, but not all IEnumerable<T> are List<T>.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
Hi, Yes, i should welcome digressions coming up in soil i've freshly plowed
Richard Deeming wrote: Logically, every List<T> is an IEnumerable<T>, but not all IEnumerable<T> are List<T>
Semantically, i respectfully disagree with that statement: imho, all generic Lists implement enumerability; but, an IEnumerable is a very special type of object that a generic list is not equatable to.
The list is a baby, alive and kicking; the IEnumerable is a fetus waiting to be born 
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: an IEnumerable is a very special type of object that a generic list is not equatable to
Since you can't create an instance of an interface, there isn't really an "IEnumerable object". There are only objects which implement IEnumerable.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
We may not end up eye=>to<=eye on this one ... of course, on a practical level, we both use IEnumerable with skill (for most of us it takes some time to achieve a sense of mastery).
Let me ask you to imagine you are teaching a bright young person how to use Linq: they ask you, one day: "what is an IEnumerable before it gets iterated ?" ... what metaphor, or analogy, would you use to communicate your understanding ?
I run a Linq operator on a collection that returns an IEnumerable<T>: I get a result which is a ""System.Linq.Enumerable+WhereListIterator`1[[System.Int32, System.Private.CoreLib" ... I call that an instance of a special object, a kind of "alternate reality" with some unique methods ('MoveNext, etc. and facilities (deferred execution.
List<int> ints0 = new List<int>() { 1, 2, 3, 4, 5, 6, 7, 8 };
var ienm = ints0.Where(itm => itm > 2);I examine the Type:
> ? ienm.GetType()
{Name = "WhereListIterator`1" FullName = "System.Linq.Enumerable+WhereListIterator`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]"}
Assembly: {System.Linq, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a}
AssemblyQualifiedName: "System.Linq.Enumerable+WhereListIterator`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Linq, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
Attributes: NestedPrivate | Sealed | BeforeFieldInit
BaseType: {Name = "Iterator`1" FullName = "System.Linq.Enumerable+Iterator`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]"}
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
modified 20-Jan-22 10:33am.
|
|
|
|
|
BillWoodruff wrote: Clearly, some collections "know" their count, because it is a Property not a method.. But, a generic IEnumerable created at run-time ?
Right, so for most things, TryGetNonEnumeratedCount just returns false. It doesn't give the count in general, it gives the count in special cases such as when the IEnumerable<T> is actually a List<T> or something similar that has a known count, that's the point. BTW you can check the source.
BillWoodruff wrote: fascinating; I try to imagine a case of a generic IEnumerable where Count() would be more efficient ... but, draw a blank
There's little chance of that, it's about the special cases again, you can see someone elses benchmarks here. They are cases in which Any and Count are both pretty fast (no significant iteration happens), but not equally fast and Count won. Those results are probably not valid anymore for .NET 6, in which Any has similar shortcuts as Count and TryGetNonEnumeratedCount have, but it made sense back in the days of .NET 4.8 and such, back then Count already had the shortcuts (TryGetNonEnumeratedCount didn't exist yet) but Any didn't have shortcuts so it wasted time creating an enumerator and calling MoveNext on it even for collections with a known count.
About 3 & 4, it depends. Depending on where those IEnumerables come from, that might perform a database query twice or call a function with a side-effect extra times (the IEnumerable could have different contents the next time, including changing its length). The safe thing to do is ToList-ing the IEnumerable at this point.. but that's the opposite of what you were trying to do. The predicate passed into a (for example) Where should be a pure function (then multiple evaluations are safe, just cost time) but C# has no mechanism to enforce that.
Sticking with Where for now, aside from literally changing the result, there is also potentially a lot of time wasted (depending on where the source IEnumerable comes from, how expensive the Where predicate is, and how often the predicate is true) by first finding the first element that passes the predicate (for Any), and then doing it again when the IEnumerable is used "for real". It's avoidable but a little messy. Duplicate work is easier to avoid in the case that Any returns false.
TBH I'm leaning more and more in favour of avoiding this whole mess, but IDK what kind of algorithm you're dealing with..
|
|
|
|
|
Thanks ! Really appreciate the details on the behavior of count/any in .NET history.
The timing study you cited is very interesting, but, my concern here is with IEnumerables that have not yet been iterated over.
i have seen an example of a generic class that inherits from IEnumerable<t>, overrides 'MoveNext, and sets a boolean property initialized to 'false to 'true, the first time 'MoveNext is called. i regret to say i can't find that code now ... which is driving me nuts.
If i understood what i have read recently, 'Any() will call 'MoveNext at least once, but, on return the Enumerable's 'current property will be #0. Interesting the source for 'Any shows it tries to convert the input to an IList: if that succeeds, it returns the IList .Count property.
i should make the context more specific here:
1) at runtime you have a collection, say: a List<int>
2. function chaining: you run that List through a series of Func methods each of which returns an IEnumerable<int>. assume you have good reasons for writing separate Funcs
2a) one case is cumulative: you collect the outcome of each Func, and concatenate that with some external IEnumerable variable you've defined/initialized. in this case a no-values result may not make a difference since such a rsult dioes not throw an error if concatenated.
2b) another case: you are filtering the source data, selecting the next items from the results of calling the previous Func. A series of "sieves" if you will.
3) consider the case where you want to do something when any of the series of Func calls returns an IEnumerable with no elements ... assume you have a method/Action defined that can be invoked by any of the series of Funcs in this "no result" case.
4) now consider the final IEnumerable result of the series of Funcs, assuming they were all invoked, and returned some values.
a) the benefits of using 'Any after each Func call seem obvious ... but ...
b) in the context of iterating the final IEnumerable result:
1) would converting the result of each Func call to a List or Array, have any benefits ?
or, any downside ?
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
BillWoodruff wrote: My impression in the SO thread cited is that this new Linq method is somehow more efficient.
Based on my experience I would not trust linq to be efficient at all. Made worse by people not even understanding what it might do.
The most major case is when linq is used with database queries. Via profiling I have seen numerous cases where the linq code that would seem to translate into a single SQL query that would return a very limited set instead pulls major parts of the table or even the entire table into memory and then processes via C# to get the results.
The second case is where a programmer is handed a dictionary and then uses linq to search for a key in the dictionary. But the form that they use means that linq will iterate all of the keys and look for a match rather than using the key directly. Not saying there isn't a different way to use it but rather I have seen it programmed this way so often that I have started to look for it.
|
|
|
|
|
Thanks ! I appreciate hearing you observations.
Re Dictionary: since Keys are implemented as Hashes, does this affect search by Linq ?
The canonical way to search for a (possibly not present) Key in a Dictionary is via 'TryGetValue: tale a look at the internal implementation of 'FindVal which 'TryGetValue uses.
Dictionary iteration: [^]
«The mind is not a vessel to be filled but a fire to be kindled» Plutarch
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
