Click here to Skip to main content
15,915,786 members

Welcome to the Lounge

   

For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?

The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.

 
GeneralRe: I feel old :( Pin
Ron Anders4-Nov-13 16:39
Ron Anders4-Nov-13 16:39 
GeneralRe: I feel old :( Pin
Marc Clifton5-Nov-13 5:48
mvaMarc Clifton5-Nov-13 5:48 
GeneralFacinating Aida PinPopular
DaveAuld4-Nov-13 6:08
professionalDaveAuld4-Nov-13 6:08 
GeneralRe: Facinating Aida Pin
OriginalGriff4-Nov-13 6:24
mveOriginalGriff4-Nov-13 6:24 
JokeBar humor PinPopular
Ravi Bhavnani4-Nov-13 5:46
professionalRavi Bhavnani4-Nov-13 5:46 
GeneralRe: Bar humor Pin
Mike Hankey4-Nov-13 6:04
mveMike Hankey4-Nov-13 6:04 
GeneralRe: Bar humor Pin
Ron Anders4-Nov-13 14:31
Ron Anders4-Nov-13 14:31 
GeneralThe rad trap (warning code but not a question) Pin
Ennis Ray Lynch, Jr.4-Nov-13 5:41
Ennis Ray Lynch, Jr.4-Nov-13 5:41 
Preface ... I knew better but I made the mistake anyway and I just felt like sharing, you probably don't want to read : )

My current mission is to author applications that have never been written before, ever, rapidly, for the power industry on "Massive Amounts" of data for users that want it yesterday but will change their minds constantly. So pretend you want to know how long a users power was out (and I know ahead of time the rules will change on me)

public class Foo{
    public DateTime PowerOutTime;
    public DateTime PowerOnTime;
    public TimeSpan FigureOutJustHowGDLongThePowerWasOut;
    public string CustomerId;
}

Populate the list with a few hundred million records, do some work. Output, slow, of course, data-loading takes a long time, up to 60 to 100 seconds for a day. Ignore the fact that I can't do this work in the DB (there are reasons)

First naive mistake, Let's put it in a generic List

List<Foo> items = ...


Sure the data calculation part is fast but for technical reasons this is very slow, (Allocation of contigous memory streams, copying, adding etc). Even better, when you are really sloppy and work with 7 days of customer data System.OutOfMemoryException : ). So what to do? Still blinded by the R.A.D. issue I continued ...

LinkedList<Foo> items = ...


Golden, no longer an out of memory issue for n<some value="" ...="" workable.="" still="" slow.="" heck="" because="" of="" network="" security,="" locality,="" and="" data-source="" issues="" it="" could="" take="" a="" day="" or="" two="" to="" get="" the="" data="" Frown | :( "" ok,="" so,="" why="" not="" copy="" locally="" first="" with="" compressed="" file="" stream="" gravy,="" hundred="" mb="" zipped="" under="" 100mb="" but="" wait,="" is="" this="" so="" slow="" ...

did="" you="" know="" that="" zipping="" operation="" in="" .net?="" (well="" large="" files)="" hdd="" cheap,="" toss="" zip.="" now="" it's="" loading="" more="" reasonably="" reliably.="" oh=""

the="" source="" local="" system
the="" zipped
the="" format="" binary="" small="" (minimally="" so)="" populating="" objects="" processing="" slow

next,="" up,="" well,="" lets="" just="" process="" order="" oops,="" can't.="" would="" have="" be="" sorted="" by="" customer="" date="" very="" (crashed="" oracle="" postgress="" too="" :="" (="" db's="" should="" able="" sort="" another="" issue)="" still,="" ..="" (in="" joker="" serious="" voice)

then="" hit="" me:
<pre="">
"Luke, pretend this is C++ and you are using a struct and a pointer offset"
"Ben, I can't C# doesn't allow safe casting"
"Use the force"


So I mapped the file to memory, created a struct and a static method to do the math for the offsets for the struct in a thread-safe manner and BAM my test set of 100 megs went from 30 seconds to load and process to under 1 second using only one-thread.

Honestly *rolls eyes*.

The worst part is that I can't use the code. Things change to much. At least I know what is causing the issue and worse, technically, I learned in college what was causing the issue at still I ignored it in favor of RAD. It was my crappy code the entire time! (Well not including the fact that the raw sources are still too slow but let's not count things out of my control)


Just remember kids, sometimes you need a better perspective of the problem.

(Oh and as a bonus we are now one step closer to bringing the power back on ... before it goes out but that is confidential : ) )
Need custom software developed? I do custom programming based primarily on MS tools with an emphasis on C# development and consulting.

"And they, since they Were not the one dead, turned to their affairs" -- Robert Frost

"All users always want Excel" --Ennis Lynch

GeneralRe: The rad trap (warning code but not a question) Pin
Marc Clifton4-Nov-13 6:02
mvaMarc Clifton4-Nov-13 6:02 
GeneralRe zipping Pin
Ennis Ray Lynch, Jr.4-Nov-13 6:10
Ennis Ray Lynch, Jr.4-Nov-13 6:10 
GeneralRe: The rad trap (warning code but not a question) Pin
Roger Wright4-Nov-13 8:10
professionalRoger Wright4-Nov-13 8:10 
GeneralRe: The rad trap (warning code but not a question) Pin
Ennis Ray Lynch, Jr.4-Nov-13 8:26
Ennis Ray Lynch, Jr.4-Nov-13 8:26 
GeneralRe: The rad trap (warning code but not a question) Pin
Roger Wright4-Nov-13 9:47
professionalRoger Wright4-Nov-13 9:47 
GeneralRe: The rad trap (warning code but not a question) Pin
Ennis Ray Lynch, Jr.4-Nov-13 10:08
Ennis Ray Lynch, Jr.4-Nov-13 10:08 
GeneralRe: The rad trap (warning code but not a question) Pin
TheGreatAndPowerfulOz4-Nov-13 10:19
TheGreatAndPowerfulOz4-Nov-13 10:19 
GeneralRe: The rad trap (warning code but not a question) Pin
Roger Wright4-Nov-13 10:25
professionalRoger Wright4-Nov-13 10:25 
GeneralRe: The rad trap (warning code but not a question) Pin
Shaun Wilde4-Nov-13 13:39
Shaun Wilde4-Nov-13 13:39 
QuestionData Entry Systems Pin
Marcus J. Smith4-Nov-13 5:29
professionalMarcus J. Smith4-Nov-13 5:29 
AnswerRe: Data Entry Systems Pin
mikepwilson4-Nov-13 5:41
mikepwilson4-Nov-13 5:41 
JokeMicrosoft Makes one Pin
Ennis Ray Lynch, Jr.4-Nov-13 5:42
Ennis Ray Lynch, Jr.4-Nov-13 5:42 
AnswerRe: Data Entry Systems Pin
Marc Clifton4-Nov-13 5:52
mvaMarc Clifton4-Nov-13 5:52 
GeneralRe: Data Entry Systems Pin
Marcus J. Smith4-Nov-13 6:27
professionalMarcus J. Smith4-Nov-13 6:27 
GeneralRe: Data Entry Systems Pin
Marc Clifton4-Nov-13 7:21
mvaMarc Clifton4-Nov-13 7:21 
GeneralRe: Data Entry Systems Pin
Marcus J. Smith4-Nov-13 7:30
professionalMarcus J. Smith4-Nov-13 7:30 
GeneralRe: Data Entry Systems Pin
Marc Clifton4-Nov-13 8:22
mvaMarc Clifton4-Nov-13 8:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.