|
Predictable but funny. 
|
|
|
|
|
Meh.
You've done better brother. 
|
|
|
|
|
Preface ... I knew better but I made the mistake anyway and I just felt like sharing, you probably don't want to read : )
My current mission is to author applications that have never been written before, ever, rapidly, for the power industry on "Massive Amounts" of data for users that want it yesterday but will change their minds constantly. So pretend you want to know how long a users power was out (and I know ahead of time the rules will change on me)
public class Foo{
public DateTime PowerOutTime;
public DateTime PowerOnTime;
public TimeSpan FigureOutJustHowGDLongThePowerWasOut;
public string CustomerId;
}
Populate the list with a few hundred million records, do some work. Output, slow, of course, data-loading takes a long time, up to 60 to 100 seconds for a day. Ignore the fact that I can't do this work in the DB (there are reasons)
First naive mistake, Let's put it in a generic List
List<Foo> items = ...
Sure the data calculation part is fast but for technical reasons this is very slow, (Allocation of contigous memory streams, copying, adding etc). Even better, when you are really sloppy and work with 7 days of customer data System.OutOfMemoryException : ). So what to do? Still blinded by the R.A.D. issue I continued ...
LinkedList<Foo> items = ...
Golden, no longer an out of memory issue for n<some value="" ...="" workable.="" still="" slow.="" heck="" because="" of="" network="" security,="" locality,="" and="" data-source="" issues="" it="" could="" take="" a="" day="" or="" two="" to="" get="" the="" data=""  "" ok,="" so,="" why="" not="" copy="" locally="" first="" with="" compressed="" file="" stream="" gravy,="" hundred="" mb="" zipped="" under="" 100mb="" but="" wait,="" is="" this="" so="" slow="" ...
did="" you="" know="" that="" zipping="" operation="" in="" .net?="" (well="" large="" files)="" hdd="" cheap,="" toss="" zip.="" now="" it's="" loading="" more="" reasonably="" reliably.="" oh=""
the="" source="" local="" system
the="" zipped
the="" format="" binary="" small="" (minimally="" so)="" populating="" objects="" processing="" slow
next,="" up,="" well,="" lets="" just="" process="" order="" oops,="" can't.="" would="" have="" be="" sorted="" by="" customer="" date="" very="" (crashed="" oracle="" postgress="" too="" :="" (="" db's="" should="" able="" sort="" another="" issue)="" still,="" ..="" (in="" joker="" serious="" voice)
then="" hit="" me:
<pre="">
"Luke, pretend this is C++ and you are using a struct and a pointer offset"
"Ben, I can't C# doesn't allow safe casting"
"Use the force"
So I mapped the file to memory, created a struct and a static method to do the math for the offsets for the struct in a thread-safe manner and BAM my test set of 100 megs went from 30 seconds to load and process to under 1 second using only one-thread.
Honestly *rolls eyes*.
The worst part is that I can't use the code. Things change to much. At least I know what is causing the issue and worse, technically, I learned in college what was causing the issue at still I ignored it in favor of RAD. It was my crappy code the entire time! (Well not including the fact that the raw sources are still too slow but let's not count things out of my control)
Just remember kids, sometimes you need a better perspective of the problem.
(Oh and as a bonus we are now one step closer to bringing the power back on ... before it goes out but that is confidential : ) )
|
|
|
|
|
Piece-wise memory allocation is painful. For a particular project analyzing end-of-life failure modes on satellite switch rings, I have a function that "simply" spins up a list (several million members) of "random" (but balanced) failures to test. It takes minutes to generate the list, most of this is memory allocations going on behind the scenes of the list management. I never bothered to "un-safe" the algorithm. 
And yes, zip is painfully slow. And when I looked at using a serialized zip stream, I discovered it's not a true stream - you can't just shove data in and expect it to stream it out as it works through the compression - the whole damn thing has to be compressed in memory first before it starts spitting out the compressed data to the stream. I discovered that years ago, I think I even wrote an article about it.
Marc
|
|
|
|
|
For me the most painful lesson with the GZip was trying to zip multiple items, separately, to one stream. That was a mistake! Seems a lot of people have it too.
|
|
|
|
|
Keep working on it, Ennis! This industry is pathetic.
I've tried for months to develop a report based on hourly meter readings from our supplier, delivered daily to my mailbox, just to generate a meaningful value for system losses. It turns out that it's impossible. Even the people who use this telemetry data to generate our monthly power bills tell me that it's useless to look at these values, because they can't trust them, and each meter reports a different number of digits' resolution. They actually still rely on monthly manual meter reads to generate our bill! This from fully automated generation and transmission facilities owned and operated by US Government agencies!!!
Will Rogers never met me.
|
|
|
|
|
How about, Just assume .9 power factor for all residential meters and give me the KVA for yesterday but only have KW/h at the hour and Voltage at the hour? And, your not allowed to have an index because it takes up too much space
|
|
|
|
|
|
Well, to be honest, everyone involved including myself understands the difference (which is surprising because I am bad at math). Unfortunately, when you have millions of meters all getting their readings at the hourly level we have to make guesses. I was more surprised that using a constant for power factor was OK, I could understand using a "best guess" based on available data KVA.
Don't pity me too much we can actually bill from our data :p
|
|
|
|
|
Ennis Ray Lynch, Jr. wrote: we can actually bill from our data
 
If your actions inspire others to dream more, learn more, do more and become more, you are a leader.-John Q. Adams
You must accept one of two basic premises: Either we are alone in the universe, or we are not alone in the universe. And either way, the implications are staggering.-Wernher von Braun
Only two things are infinite, the universe and human stupidity, and I'm not sure about the former.-Albert Einstein
|
|
|
|
|
Yeah, power factors don't change much, except in areas dominated by motors and other highly inductive loads that are switched on and off at various times. I have one substation that has a power factor for the entire service area of 1.000 - consistently! It's almost entirely residential, with a few well pumps, but there's also a 1.5 MG/D sewer plant on the line. How we pull that off is a mystery - just a lucky circumstance that the predominantly capacitive power lines exactly balance the inductive loads. I couldn't design it that way! 
For residential, though, 0.9 is awfully low. Even in summer, when we have summer air conditioning loads that double the usual lighting loads of winter, we run a PF of about 0.95. The rest of the year it's 0.97 to 0.99.
Will Rogers never met me.
|
|
|
|
|
RAD = Rapid Application Development
This is not the same as the Development of Rapid Applications!
I'll be more enthusiastic about encouraging thinking outside the box when there's evidence of any thinking going on inside it. - pTerrysawilde @ GitHub
|
|
|
|
|
I work for a company where I have rolled our own DES and it has been a successful system for many years. Now we are evaluating whether to build new, retrofit, or purchase something. I am trying to review software but am only finding stuff intended to provide mechanisms to enter fairly simple data, small amounts of data, or surveys. Do you all know of companies that offer software that is truly customizable and intended to allow for hundreds/thousands of complex data edits? I just need help finding the companies that offer those types of systems instead of these data entry creation systems that are intended to allow for name and phone number type entry.
CleaKO
"Now, a man would have opened both gates, driven through and not bothered to close either gate." - Marc Clifton (The Lounge)
|
|
|
|
|
Sounds like you just need to bang together a gui app in Delphi, or visual whatever.
All of those tools make that exact type of operation pretty easy. Drag a bunch of controls, get the tab order right, hook them to a data source and add whatever validation on the 'lose focus' operation.
|
|
|
|
|
It's called C#, it's fairly configurable. If you use the ASP.NET module it can be web based as well. A common configuration option is the SQL Server persistence module
|
|
|
|
|
I haven't seen anything that can handle the real-life complexities gracefully, which is why I've rolled my own in the past as well, and why I've seen others roll their own as well in other platforms.
Marc
|
|
|
|
|
That's what I figured. I already wrote my own and we were just evaluating whether there are systems that are already out there that can be configured to do what we want. I didn't have high hopes.
To answer the other 2 questions, I already wrote my own so don't have a problem with that.
CleaKO
"Now, a man would have opened both gates, driven through and not bothered to close either gate." - Marc Clifton (The Lounge)
|
|
|
|
|
CleaKO wrote: I already wrote my own and we were just evaluating whether there are systems that are already out
Well, why don't you quickly whip together a website and show it to management "Oooh, I recommend THIS product", and sell it to them all over again? 
Marc
|
|
|
|
|
There's an idea.
I don't post much on here anymore so is it strange that you provided the quote in my signature?
CleaKO
"Now, a man would have opened both gates, driven through and not bothered to close either gate." - Marc Clifton (The Lounge)
|
|
|
|
|
CleaKO wrote: I don't post much on here anymore so is it strange that you provided the quote in my signature?
Haha. Sadly, I didn't even notice (the font is so tiny). Then again, I'm on here every day, and I'm also seeing if anyone notices my sig and my latest article link. and votes on the article!!!
Marc
|
|
|
|
|
Adopt our data service  , which allows batch input as a UI product and also allows custom built input front end system as a service. See here[^]. And here[^] is a more real life example built for StackExchange data dump (it's read only, however).
|
|
|
|
|
I am looking for a working in C# that can convert english text that exists in a form to Arabic text before the text is passed to printer. I will be really very very thank full if someone could help me with this.
|
|
|
|
|
Have you looked into Yamli?
|
|
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
