|
Hi All;
I have a large dataset stored in a CSV file (about 40,000 Rows and 10,000 Columns). I need to load it into a C# Windows application. So, any idea to do this. I tried different code, but some are able to loao 40,000 R * 255 C, and other codes are able to load 5,000 R and 10,000 C.
Thanks
losan1985
|
|
|
|
|
It's probably going to take rolling your own custom class to hold it all. I don't know of anything "off-the-shelf" that will hold 10,000 columns. Frankly, I've never even HEARD of such a wide CSV file ever being used.
It shouldn't be very hard at all to create a List<list<int>> or whatever your item data type is. Basically, a List of List of Integers.
|
|
|
|
|
You probably can't load it all at once (at least in a 32-bit OS). 40,000 x 10,000 = 400,000,000 bytes if each cell is 1 byte. If you assume an average of 16 bytes (since you didn't say) per cell, thats 6,400,000,000 bytes = 5GB of data. You only have 2GB of address space for your application. You can do it on a 64-bit OS though.
With that being said, I doubt you really need 40,000 x 10,000 cells loaded in memory at once. What is a person going to do with all that data?
You might want to consider loading only the portion you need.
|
|
|
|
|
What is your actual requirement? You are loading this data for a reason. What is that reason? For instance, are you performing some calculation on certain columns? By breaking down your requirements, we can work out a practical solution.
|
|
|
|
|
yes, this may solve your problem....
|
|
|
|
|
As POH has said your design has to be wrong for this to be a valid requirement. Go back and look at how the CSV was created, why does it require 10k columns (what a ridiculous number). Can your source break it up into more swallowable chunks. Do you need all 10k columns.
Can you load and process 1 row at a time, presumably you want to dump this into some more reasonable format.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
You'll need to build in a sort of paging mechanism that only loads that part that is shown on the screen.
|
|
|
|
|
As Sledgehammer01 says, that's an unreasonably large amount of data for most purposes. It's 400 million cells and so you're talking about GB of memory, depending on exactly what's in there. What do you want to do with this dataset? You almost certainly want a load-on-demand adapter of some kind, so you can run through the data without actually having it all in memory at once.
This library is rather good; I used it in a real application (though not dealing with massive datasets) without problem.
|
|
|
|
|
Hi losan,
I found your post very interesting because I've never encountered a data set that large. Are you trying to analyze that data? If so, I may be able to help.
I have a product (www.patternscope.com) that finds patterns in extremely large data sets. I think your data set would be good for stress-testing the application, and it fits perfectly with two planned developments:
1. Reading CSV data (currently it only reads databases through ODBC, or flat files), and
2. Making a C#-callable API that you could use in your C# application to handle that much data (e.g. queries, retrieval, and analysis).
My product extracts the patterns that comprise the raw data. These patterns are a fraction of the size of the original raw data, so they fit entirely into memory, even when the raw data is larger than the memory available.
The patterns have the same information content as the raw data, so can be processed (e.g. queries or analysis) many times faster.
If you could give me a copy of your data set, I could give you a free copy of PatternScope (after I adapt it for reading CSV data) which you could use to analyze the data, followed by a DLL you could call from C# for processing the data in your program.
What does this data represent?
|
|
|
|
|
Hi;
Here is a link for the Dataset
"www.dropbox.com/s/een9zlqce4vqqrl/ProjectData3.csv"
What I need to do is to apply the collaborative filtering algorithms in the dataset. The data set is about Tweets, who is going to retweet from another person.
Thanks
losan1985
|
|
|
|
|
Thanks. When I've adapted PatternScope for comma-separated values, I'll send you a copy. Collaborative filtering looks interesting.
|
|
|
|
|
My requirement is Capturing Video from web cam and saving it in .avi format. I am currently using DirectShowNet Library, But the captured video size is too big. For 10 sec video it takes 70 MB storage size. I have tried other web cam libraries but nothing works.
I have tried AForge, EMGU, DirectX and some other libraries, but the problem still exists (video size is too big).
If you have ever worked on this type of problem, please help me or suggest me for some other libraries.
___ ___ ___
|__ |_| |\ | | |_| \ /
__| | | | \| |__| | | /
|
|
|
|
|
Raw Video Data has it's cost (that's the big size on the disk it eats  ) You'll either need to a) reduce the resolution you're recording a video or b) encode the video with a codec to reduce the size.
I haven't done something like that programatically myself yet. However already used some converters which are available for free on the internet to convert the videos. (As far as I remember you should be even able to use the windows live movie maker )
|
|
|
|
|
Yes, I think you are right. I need to encode/compress the RAW Video data into proper format.
Can you please suggest any library for doing the same. Thanks
___ ___ ___
|__ |_| |\ | | |_| \ /
__| | | | \| |__| | | /
|
|
|
|
|
As I've said. I haven't done it myself with a library myself.
Maybe have a look at this question here:
video compression in c#[^]
|
|
|
|
|
OK,
I found a converter named "FFMPEG" and working on it. BTW Thanks.
___ ___ ___
|__ |_| |\ | | |_| \ /
__| | | | \| |__| | | /
|
|
|
|
|
When I call with ATDxxxx; on Port 16: "Mobile Connect - PC UI Interface"
It connect but audio stream in Port 18: "Mobile Connect - Application Interface"
How to redirect audio stream in/out the serial port 18?
Please help!
 
|
|
|
|
|
You should try asking in the Hardware and Devices forum.
"It's true that hard work never killed anyone. But I figure, why take the chance." - Ronald Reagan
That's what machines are for.
Got a problem?
Sleep on it.
|
|
|
|
|
Hi guys ...
I was looking for a way that allow me to send a message to cmd directly ... , I've searched about an answer, but I didn't find what I want.
I want my windows Form to send messages to CMD without open a new one
I mean it's already opened on my PC but I want a way to send messages only .. So can I do that ?
I think I'll need DllImport or something like that to get it work
Thanks for your help ..
|
|
|
|
|
A "low-tech" approach is sending key-strokes to the cmd window, i.e. send the keystrokes for the characters you would type otherwise.
|
|
|
|
|
What are you actually trying to achieve in the command window? are you after running a batch file or something?
If so have a look at the ProcessStartInfo class[^]
Every day, thousands of innocent plants are killed by vegetarians.
Help end the violence EAT BACON
|
|
|
|
|
Hi,
I do not think it is possible in command prompt, it sound like you need a terminal window. command prompt doesn't have a gateway like terminal. In terminal window you can send messages using a gateway. with command prompt it is not possible unless you use the powershell or install powershell extension and inject a powershell command to copy/print your message to command prompt using special powershell commands.
I hope this helps.
Think! Don't write a line of code unless you absolutely need to.
|
|
|
|
|
It might help if you told us why you need to do it that way.
You would need to get the Process ID of the command prompt window first.
But, AFAIK it can't be done without intercepting stdin, stdout. I don't believe sendkeys would work.
"It's true that hard work never killed anyone. But I figure, why take the chance." - Ronald Reagan
That's what machines are for.
Got a problem?
Sleep on it.
|
|
|
|
|
There are numerous things you need to think about here. First of all, what happens if you have multiple cmd windows open? Perhaps some of them are running specific long running tasks that shouldn't be interrupted. It could be that some of them have shells displayed that are waiting for user input, so sending information to them could cause problems.
The question here would have to be, what is your business requirement for not opening a new cmd window? What you might want to consider is the approach taken in the top answer here[^].
|
|
|
|
|
Hacking purposes xD
I've gotta to write a lot of lines ...
and since I am a not bad C sharper, then I don't want to bother my self writing them ...
So I want To write a program that send my commands.
I already done this with another program ...
foreach (Process Proc in Process.GetProcessesByName("NameOfProcess"))
{
}
This about what gonna happen if mote than one Program are running at the same time, It's gonna send to both of them with this code ..
What I understood till now ..
There is no way to send messages to CMD ??
Thank you guys for your Replys ..
they made the things more clear for me
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
