|
I wrote a command line app that imports a NESSUS security scan XML data file - the largest I've seen to date is about 8gb. We import the data into a SQL server database. It's not multi-threaded at all that I recall. I do remember that the file was too big for XDoument to work.
I feel your pain.
".45 ACP - because shooting twice is just silly" - JSOP, 2010
-----
You can never have too much ammo - unless you're swimming, or on fire. - JSOP, 2010
-----
When you pry the gun from my cold dead hands, be careful - the barrel will be very hot. - JSOP, 2013
|
|
|
|
|
If the parsing can be partitioned into n subproblems, where n is the number of cores, then I would consider creating n daemons and locking each one into its own core. If any of them block, offloading the blocking operations to thread pools might help.
Partitioning the problem will help to reduce semaphore contention and cache collisions.
But I haven't had to populate a large database this way, so I could be full of shite. 
|
|
|
|
|
This is exactly what I didn't want to have to learn. 
At least all proper databases already handle parallel execution properly.
|
|
|
|
|
yup - learn't the hard way, 1st identify where the program uses it's resources
|
|
|
|
|
More proof that some people have real problems.
So stop complaining people, you could be Jörgen today.
|
|
|
|
|
Ron Anders wrote: So stop complaining people, you could be Jörgen today.
...and have no toilet paper.
Jeremy Falcon
|
|
|
|
|
Isn't it enough if I'm being me?
|
|
|
|
|
I was vaguely reminded of an episode of Home Improvement, where one of the kids got himself in some trouble, so one of the brothers says "I wouldn't wanna be you right now", and the other responds with "I wouldn't wanna be you, ever".
|
|
|
|
|
I was going to say - size of the work done in each task is key...
But the underlying technology can also have an effect, by reducing the cost of task creation. If you're using a work queue on top of a thread pool, you're not creating a thread for each task, you're pushing/popping tasks on and off a queue.
I created a little tool to detect duplicate files using that sort of parallelism. It contains two main areas of parallelism:
- The file search library that I use adds a new task for each directory it sees. Each task processes just the files that are immediate children of the directory the task was created for.
- The detection of duplicates is split so that each task hashes a group of files that have the same size. This is performed using a data parallelism library, which makes parallelising things very easy.
The amount of speedup I get isn't anywhere near the number of processor cores in use (I get a factor of just over two speedup on an eight core machine), but I think that the amount of IO being done serialises the processing to a certain degree. Benchmarking ripgrep, another tool that uses similar parallelism, shows that running with 8 threads (on 8 logical/4 physical cores) is just over 3x faster than using 1.
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
|
|
|
|
|
why are u even parsing xml files and that too 80gb !!! and then saving it to the database !!! .. u could try to use the sql server bulk import tools to do this and avoid programming such stuff all together...
Caveat Emptor.
"Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
|
|
|
|
|
Because I want to have the data extracted into normalized tables.
|
|
|
|
|
|
I've missed out on that possibility completely.
A bit late now, but I'll take a look at it anyway. 
|
|
|
|
|
I think I see the reason why I missed out on that possibility, it does not seem to exist on SQL Server 2012.
|
|
|
|
|
if its the dev env u have you can run the sql server setup and select the components needed to get SSIS services and vs based client tools.is on the iso or dvd etc.. .. also there is OPENROWSET Simple way to Import XML Data into SQL Server with T-SQL .....
Caveat Emptor.
"Progress doesn't come from early risers – progress is made by lazy men looking for easier ways to do things." Lazarus Long
|
|
|
|
|
I think I see the reason why I missed out on that possibility, XMLSource does not seem to exist on SQL Server 2012.
|
|
|
|
|
Welcome to the cool club though. Ladies can't resist an async coder. #science
Jeremy Falcon
|
|
|
|
|
That's seriously the best answer today. 
|
|
|
|
|
Jeremy Falcon
|
|
|
|
|
Jeremy Falcon wrote: Ladies can't resist an async coder.
What for...certainly not her and her sister...
|
|
|
|
|
I can't help, but reading "parse" in the body of the message...
This is clearly a case for... HONEY THE @CODE-WITCH tatatataaaaaaa 
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
are you sure the bottleneck isnt disk i/o?
Real programmers use butterflies
|
|
|
|
|
Yes, just to make sure.
I've made test runs just reading an ID from every record which goes twice as fast, and that's on a slow HDD here at home.
And when I move this to a server the disks will be considerably faster.
|
|
|
|
|
40GB of your 80GB XML file are tags. So much for the overhead.
The suggestion is to worldwide drop all markup languages (XML, JSON and similar shiite)
|
|
|
|
|
But compact binary formats are almost impossible to patch up using vi. Linux guys will feel completely lost!
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
