Click here to Skip to main content
15,887,135 members
Home / Discussions / C#
   

C#

 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8612-Jan-16 16:29
JD8612-Jan-16 16:29 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
Luc Pattyn12-Jan-16 17:16
sitebuilderLuc Pattyn12-Jan-16 17:16 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8612-Jan-16 17:30
JD8612-Jan-16 17:30 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
Luc Pattyn12-Jan-16 17:46
sitebuilderLuc Pattyn12-Jan-16 17:46 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8614-Jan-16 4:07
JD8614-Jan-16 4:07 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8614-Jan-16 4:26
JD8614-Jan-16 4:26 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8614-Jan-16 14:05
JD8614-Jan-16 14:05 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
Luc Pattyn14-Jan-16 18:26
sitebuilderLuc Pattyn14-Jan-16 18:26 
Hi,

1. once again I must ask you to be more specific; "slowly climbing" doesn't offer much info.

2. changing the code and changing run-time parameters too much makes it extremely hard if not impossible to understand what sporadic measurement results actually mean.

3. I won't be in tomorrow.

4. Today I have been, and still am, researching the behavior of the Large Object Heap, and probably will write a CP article about it; if so, that will take a few weeks though.
In summary, while there may be a way to make it perform as required (with a lot of conditions), I am still inclined to avoid large objects as much as possible. And hence:

5. I developed a little class that would keep your huge message lists in the regular heap, hence avoiding the LOH fragmentation you are currently bound to get. It does not offer an initial capacity, I don't think that would be very useful. It has been tested with foreach and with LINQ. Here it is:

C#
using System;
using System.Collections;
using System.Collections.Generic;
using System.Text;

namespace LargeObjectsHeapTester {
	//
	// This class collects a sequence of items of type T (value or reference type)
	// and keeps them in the order they get added.
	// A list-of-lists technique is used to keep everything in the regular heap,
	// i.e. this class should not challenge the large object heap unless
	// the number of items collected gets really big (above 100 million).
	// To control what is inside the collection:
	//	- use Add to add an item at the end;
	//	- use Clear to clear the collection.
	// To obtain information about its content:
	//	- use the Count property;
	//	- use the IEnumerable<T> interface (or the old-fashioned IEnumerable if you must)
	//	  (both foreach and LINQ rely on the IEnumerable<T> interface when present)
	//
	// Note: value types such as structs should be small (max 8B) otherwise
	// the level1 lists may not fit in the regular heap.
	public class ListOfLists<T> :IEnumerable, IEnumerable<T> {
		public const int MAX_ITEMS_IN_LEVEL1=10000;
		//public static ILog Logger;
		private List<List<T>> level0;
		private List<T> currentLevel1;

		// constructor
		public ListOfLists() {
			Clear();
		}

		// logging utility
		private void log(string msg) {
			//if(Logger!=null) Logger.log(msg);
		}

		// empty the collection
		public void Clear() {
			level0=new List<List<T>>();
			currentLevel1=new List<T>();
			level0.Add(currentLevel1);
		}

		// add an item at the end of the collection
		public void Add(T item) {
			if(currentLevel1.Count>=MAX_ITEMS_IN_LEVEL1) {
				currentLevel1=new List<T>();
				level0.Add(currentLevel1);
			}
			currentLevel1.Add(item);
		}

		// get the number of items in the collection
		public int Count {
			get {
				int totalCount=0;
				foreach(List<T> level1 in level0) totalCount+=level1.Count;
				return totalCount;
			}
		}

		// log the internals
		public void DumpCounts() {
			int totalCount=0;
			log("level0.Count="+level0.Count);
			int i=0;
			foreach(List<T> level1 in level0) {
				int count1=level1.Count;
				log("level0["+i+"].Count="+count1);
				totalCount+=count1;
				i++;
			}
			log("total count="+totalCount);
		}

		// get an old-style enumerator (typeless)
		IEnumerator IEnumerable.GetEnumerator() {
			return this.GetEnumerator();
		}

		// get an enumerator (this one gets used by foreach and LINQ)
		public IEnumerator<T> GetEnumerator() {
			foreach(List<T> level1 in level0) {
				foreach(T item in level1) {
					yield return item;
				}
			}
		}
	}
}


6. Your LINQ statements such as
var allMailboxes = db.Users.Where(x ... ... ... ).ToList();                                              

clearly also create lists; I don't know how large they are, if more than 10000 elements, they too will contribute to the LOH fragmentation, and have escaped every remedy I have suggested so far.

Questions:
a) how many users satisfy x.MailboxPlan > 0
b) what would be a good upper limit for the Count of this allMailboxes?
c) how many users would there typically be in the below a.Users (sent or received)
List<MessageTrackingLog> totalSentLogs = sentLogs.Where(a => a.Users.Contains(x.EmailAddress, StringComparer.OrdinalIgnoreCase)).ToList();


That's it for now.

Smile | :)
Luc Pattyn [My Articles] Nil Volentibus Arduum


modified 15-Jan-16 8:18am.

GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8617-Jan-16 16:09
JD8617-Jan-16 16:09 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
Luc Pattyn17-Jan-16 17:10
sitebuilderLuc Pattyn17-Jan-16 17:10 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8617-Jan-16 17:21
JD8617-Jan-16 17:21 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
Luc Pattyn17-Jan-16 18:32
sitebuilderLuc Pattyn17-Jan-16 18:32 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8619-Jan-16 3:26
JD8619-Jan-16 3:26 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
JD8621-Jan-16 4:04
JD8621-Jan-16 4:04 
GeneralRe: Memory leak trouble with Linq to Sql and multiple threads Pin
Luc Pattyn21-Jan-16 15:06
sitebuilderLuc Pattyn21-Jan-16 15:06 
Questionbetter way to Linqify this ? Pin
BillWoodruff9-Jan-16 2:01
professionalBillWoodruff9-Jan-16 2:01 
AnswerRe: better way to Linqify this ? Pin
Sascha Lefèvre9-Jan-16 2:25
professionalSascha Lefèvre9-Jan-16 2:25 
GeneralRe: better way to Linqify this ? Pin
BillWoodruff11-Jan-16 8:06
professionalBillWoodruff11-Jan-16 8:06 
GeneralRe: better way to Linqify this ? Pin
Sascha Lefèvre11-Jan-16 8:37
professionalSascha Lefèvre11-Jan-16 8:37 
GeneralRe: better way to Linqify this ? Pin
Richard Deeming11-Jan-16 11:00
mveRichard Deeming11-Jan-16 11:00 
GeneralRe: better way to Linqify this ? Pin
Sascha Lefèvre11-Jan-16 23:00
professionalSascha Lefèvre11-Jan-16 23:00 
GeneralRe: better way to Linqify this ? Pin
BillWoodruff12-Jan-16 0:19
professionalBillWoodruff12-Jan-16 0:19 
AnswerRe: better way to Linqify this ? Pin
PIEBALDconsult9-Jan-16 6:09
mvePIEBALDconsult9-Jan-16 6:09 
GeneralRe: better way to Linqify this ? Pin
OriginalGriff9-Jan-16 6:17
mveOriginalGriff9-Jan-16 6:17 
GeneralRe: better way to Linqify this ? Pin
Jörgen Andersson9-Jan-16 12:53
professionalJörgen Andersson9-Jan-16 12:53 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.