|
I started with multi-gb size XML files that would cause out-of-memory exceptions.
After I resolved that issue (by using XmlTextReader/XElement instead of XDocument), it was taking over 20 minutes to process the sample file set (434,000 rows). This is where I was on Friday.
I added some TPL love and got it down to about 12 minutes.
I reworked the target tables. At first, I was creating a single monolithic flat table, and a lot of data was being repeated for every record. I created two tables (one that holds the repeating data, and one that holds the unique data), and got it down to just a little over 6 minutes.
This code will replace a Perl script that:
0) Creates a bunch of smaller XML files, probably so it doesn't run out of memory
1) Took over an hour to process the same set of sample files.
2) Did not return all of the data (because the stakeholder doesn't really need it)
Added benefits:
0) My code can process ZIP (extracts NESSUS files from them for processing) and bare naked NESSUS files
1) Processes multiple files with a single execution
2) Puts ALL of the data into the database
Caveat: Because of the limited hardware the app will actually be running on, processing time will take longer (because of the TPL's default behavior), but not significantly so, and CERTAINLY not take an hour.
It's been a good weekend. 
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
And now ... the overtime claim ... 
Sent from my Amstrad PC 1640
Never throw anything away, Griff
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Nope, just did it for the fun of it.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
Are you sure you didn't do it to implicitly say someone... I told you so?  
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Nope - there were several challenging aspects to the project, and I enjoy a decent programming challenge.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
When you're stuck maintaining an existing project, optimizing the living crap out of somebody else's inefficient design can be rather rewarding, even when it's only to make yourself feel smug. Not suggesting this is the case here. Those time figures speak for themselves.
|
|
|
|
|
And now the streaming SignalR version please 
|
|
|
|
|
Just don't get too pissed off if it's sidelined for political or egoistic reasons.
That happens all the time, and (rightly) blowing your top only gives them more ammunition to snipe at you with.
I wanna be a eunuchs developer! Pass me a bread knife!
|
|
|
|
|
Well, I was asked to do it if I had time, so...
I don't really care if it's used or not. Pretty much everything I've ever written for the government is no longer used for the very reasons you listed. I've kinda GOTTEN used to it. I see it as a learning experience, especially due to the large XML file stuff I had to work around.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
That's a good perspective.
You've done work to be proud of.
If the @rseholes are too busy blowing up their reps to be proud of it, it's because they're just @rseholes.
I wanna be a eunuchs developer! Pass me a bread knife!
|
|
|
|
|
I think you have the basis of an article here John
We can’t stop here, this is bat country - Hunter S Thompson RIP
|
|
|
|
|
i might...
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
Please do! I was looking through the whole thread thinking how this would make a great article and hoping you would be inspired
<sig notetoself="think of a better signature">
<first>Jim</first> <last>Meadors</last>
</sig>
|
|
|
|
|
John Simmons / outlaw programmer wrote: everything I've ever written for the government is no longer used
That's the reason I'm glad I got out of defense contracting. Of the last two projects I did for the Air Force in he late 1980's, one took over ten man-years to develop and was used for two weeks. The second was a bit smaller, taking about 3 man-years, but was never used at all.
The only thing worse than when they call your baby ugly is when they are indifferent to it.
Software Zen: delete this;
|
|
|
|
|
I simply don't care anymore - I was employed, and I did what they wanted. If they want to throw it all in the crapper, that's their perogative.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
John Simmons / outlaw programmer wrote: I simply don't care anymore - I was employed, and I did what they wanted. If they want to throw it all in the crapper, that's their perogative.
The voice of the experienced software developer.
Once said (by me) to a younger developer who was commiserating that none of the stuff he's working on will ever be remembered by anyone: "Sorry to disappoint you Michelangelo, but we're not building cathedrals."
|
|
|
|
|
Is the Tartanula's worst enemy the Plaidypus?
Sent from my Amstrad PC 1640
Never throw anything away, Griff
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
Sometimes you give me the creeps.
"If we don't change direction, we'll end up where we're going"
|
|
|
|
|
Not sure... let me check...
|
|
|
|
|
That pun's the worsted!
Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.
|
|
|
|
|
I phoned the local zoo to find out. They said “Argyle look into it and get back to you”
Socialism is the Axe Body Spray of political ideologies: It never does what it claims to do, but people too young to know better keep buying it anyway. (Glenn Reynolds)
|
|
|
|
|
This thread should have been kilt before posting.
| Ravings en masse^ |
|---|
| "The difference between genius and stupidity is that genius has its limits." - Albert Einstein | | "If you are searching for perfection in others, then you seek disappointment. If you are seek perfection in yourself, then you will find failure." - Balboos HaGadol Mar 2010 |
|
|
|
|
|
It was one of the first low power CMOS processors, at a time when nobody thought that would ever be important. It also was a RISC processor, long before that term was even invented. It was called an 'odd beast' and 'weird' for that, until more than a decade later the first ARM processors were made. It was also the first microprocessor in space. That gave it an exceptionally long lifetime, until it went out of production. It was said to have been in the Voyager and Viking probes, but that does not seem to be right. It was definitely in the Space Shuttles and plunged into Jupiter's atmosphere with Galileo.
It just does not deserve this[^]! (Edit: wrong picture, now corrected!)
* Not really. This actually demonstrates, how 'microcontrollerish' the little processor was, despite being able to drive its bus like full microprocessor. Even more so the later versions (CDP1805 or CDP1806) with onboard ROM and RAM. Still, with only an external ROM and a static RAM, the CDP1802 on that board could act as a minimal computer wich can be used via an external terminal (emulation nowadays) and RS232. Not so bad for 1976.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
modified 18-Feb-19 9:00am.
|
|
|
|
|
well if that's what they sent into space no wonder the aliens never called...
Alien auditors report circa 1970's
humans have developed computers and what do they use them for? smiley faces! lets check back in a few years see if any better signs of intelligence manifest.
[approx 3 Jovian years later]
Alien auditors report update circa 2019
what have the humans achieved since our last report:
pictures of lunch, neighbor's cat, fake news, ...
seems humans are now 100% invested in something called "social media"
... what a bunch of self loving arrogant pricks, 50 years of computers and all they do is look at themselves? ...
introduce humans to the galactic federation and technologies No ![Elephant | [mastadon]](https://codeproject.freetls.fastly.net/script/Forums/Images/elephant.gif) ing way
Message Signature
(Click to edit ->)
|
|
|
|
|
Let's hope that's not what aliens see when they get one into their tentacles. The designer did not intend to make a super processor for mainframes. Instead, he wanted it to be inexpensive and need as few external parts as possible. He also wanted to bring computers in everybody's home by making them as inexpensive and easy to build as he could. Right after the CDP1802 was released, he designed the Elf and published it in Popular Electronics. These were the days of the Altair. When you wanted to have a computer, you had to build it yourself. The little Elf was ideal with a price under 100$ and really far easier to put together than an Altair. Later they even added simple graphics capability without raising the price over 100$.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
