Re: Import data - C# Discussion Boards

19-Nov-04 7:54

In addition to what Daniel said, you should also take a look at a few examples of similar classes by viewing some of your favorite (and similar) classes in ildasm.exe that installs with the .NET Framework SDK or a decompiler like the .NET Reflector[^].

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

Import data

bouli19-Nov-04 6:57

19-Nov-04 6:57

Hello gurus,

I would like to know if there is a data importer in .NET like in the excel's or access' "Import Data" option? Or must I reinvent the wheel?
As most software have this "Import Data" I guess it's already built in .NET but where is it?
Which class does that? I feel this functionality of data import exist in the framework.

Thanks for the answers.

Best regards.

There is no spoon.

Heath Stewart19-Nov-04 8:03

19-Nov-04 8:03

The .NET Framework base class library (BCL) is just that - a class library. It does not - and should not - contain such functions as data import, but should - and does - facilitate it. That's what a base class library is for. It's not a piece of software like Excel - which is a functional product.

Besides, into what format are you importing data? There's no standard format in .NET for data like an XLS for Excel or something. The only thing that comes close is to import data from either a database or an XML file (the serialized representation) into a DataSet. Importing data into your own classes is specific to your implementation.

The BCL facilitates that through I/O classes, data readers and adapters, the DataGrid, and just about anything else you can use to import and manipulate data. How you do that depends on your requirements, but there are classes to help that like the FileStream, the TextReader, a DataReader derivative, the DataAdapter derivatives, and many, many more.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

bouli19-Nov-04 8:36

19-Nov-04 8:36

Hi Heath,

Thanks for your answer.

In a first time, I need to import data from a text file with a delimiter.
I guess that things like ODBC does automatically that. or maybe there is something already done in .NET.

In my 1st esquisse of the importer, I use the DataSet class to store the read data from the text file, and I use the StreamReader to read the contents of the text file.

Storing in a DataSet object the contents of the analyzed text file goes well.
BUT...

The day the data from the text file takes a huge amount of disk space (more than the physical memory itself), the entire data can't fit into the memory. So, we need to read little by little the data. There are several text files that contains exported data from an Database server that are huge in size. Then, using DataSet won't feet in memory.

I guess that MS has done something that matches with this problem, because most software does data importation. In an old time, ODBC was the answer to data importation/exportation.

So, I guess also that .NET provide something like ODBC and text drivers to read data stored in text is a specified formating (separated by a delimiter like coma, tab...)

Best regards.

There is no spoon.

Nick Parker19-Nov-04 9:32

Nick Parker

19-Nov-04 9:32

bouli wrote:
In a first time, I need to import data from a text file with a delimiter.

Why not just read through the file and as you read each line into a string, use the Split method of the string using your delimiter to break it into pieces and the process the results.

- Nick Parker
My Blog | My Articles

bouli19-Nov-04 9:47

19-Nov-04 9:47

That's what I do.
but at the end, the whole data remains in memory.

There is no spoon.

Dave Kreskowiak19-Nov-04 10:00

Dave Kreskowiak

19-Nov-04 10:00

That's because your not finishing the processing of the data. You don't add all those records to the database (or whatever your using as a database) all at once in a single dataset. You read a record out of your file, parse it up, process it, then send it to it's destination. You DON'T store it in a dataset, or any other memory object for that matter. Let the database do it's job. Don't try and do the job for it. You already found out why...

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

bouli19-Nov-04 10:07

19-Nov-04 10:07

so, how can I finish processing the data?
I would like to store these parsed rows. I don't see any methods to write datasets (or datarows) in a file...

There is no spoon.

Heath Stewart19-Nov-04 10:10

19-Nov-04 10:10

bouli wrote:
I would like to store these parsed rows.

So read in X number of rows then record them in a database, then read X more and continue till you're done processing the file. That's how bulk import applications work. The DataSet is not always the best solution to use, either, but if you want to save a DataSet to a file use DataSet.WriteXml and to read it use DataSet.ReadXml. This will operate on XML files, though, no delimited files.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

bouli19-Nov-04 10:14

19-Nov-04 10:14

If you have another suggestion, it's welcome.
the goal is to save the processed lines from the text file, in my own format.
The rows I read are correct, the contents of the rows in memory are correct, but I don't want all in memory. once I processed the row, I store immediatly the row in order to keep only one row in memory.

There is no spoon.

Heath Stewart19-Nov-04 10:16

19-Nov-04 10:16

We just told you how to do it. Read your file line-by-line, process it however you require, then save it. This is a simple, entry-level problem with file I/O. As we've all said - don't read the whole thing into memory, only deal with one or a few rows at a time and persist the processed data however you require. It's your requirements (which you failed to tell us about anyway: "process" could be anything) so you need to figure out how to process the data. The System.IO and System.Data namespace classes are there to facilitate retreiving and storing data.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

bouli19-Nov-04 10:29

19-Nov-04 10:29

ok.
So instead of using the ReadToEnd() method of StreamReader, I may do a loop and use the ReadLine() method, like in the MSDN sample.

There is no spoon.

Nick Parker19-Nov-04 19:41

Nick Parker

19-Nov-04 19:41

Yes, you could do that.

- Nick Parker
My Blog | My Articles

bouli19-Nov-04 21:52

19-Nov-04 21:52

I have a more simple idea.

I will simply save my read data from text file in an ms-access database (mdb).
Then ms-access will do the queries.

What is the namespace that creates and manipulates ms-access database format (mdb)? Is it ADO?

I think it's the simplest.

There is no spoon.

Nick Parker20-Nov-04 5:07

Nick Parker

20-Nov-04 5:07

I would recommend that you check out the documentation on MSDN, it will be very helpful. In particular, check into the OleDbConnection Class[^]. HTH.

- Nick Parker
My Blog | My Articles

bouli20-Nov-04 5:08

20-Nov-04 5:08

Ok. I'm going to check this class.
Thanks Smile | :)

There is no spoon.

Heath Stewart19-Nov-04 10:13

19-Nov-04 10:13

The .NET Framework is not an application - it's a framework (hence the name) on which applications are built. The JRE (Java runtime environment) doesn't implement bulk importers either, but gives you the classes (i.e., the framework) with which to create bulk importers.

Yes, ODBC and OLE DB both have a driver to treat delimited text files as data sources and you can use that using either the System.Data.Odbc or System.Data.OleDb classes, buth that would be very inefficient because you're P/Invoking (indirectly) native calls and marshaling data. Reading it faster like the other reply (from Nick) mentions is much faster.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

enum Questions

niceguyeddie19-Nov-04 6:31

niceguyeddie

19-Nov-04 6:31

I'm new at C#, okay with C++

Lets say hypothetically that i'm making a video game and I want to make a button that fires a weapon.. I've made an enum for my "special keys"... so instead of using the Keys enum which doesn't give the keys any more of a description more than what letter it is. So I wanted to make a new Keys enum called myKeys and do something like this

<br />
enum myKeys <br />
{<br />
     SpecialKey1 = Keys.N,<br />
     FireWeapon  = Keys.P,<br />
     //etc....<br />
}<br />
<br />
protected override void OnKeyUp(KeyEventArgs e)<br />
{<br />
     if (e.KeyCode == myKeys.SpecialKey1)  //Error<br />
     {<br />
            //Do something;<br />
     }<br />
     if (e.KeyCode == Keys.N)  //NO error<br />
     {<br />
            //Do something;<br />
     }<br />
}<br />

What am I doing wrong here?

This is the error...

<br />
 Operator '==' cannot be applied to operands of type 'System.Windows.Forms.Keys' and 'myApp.myKeys'<br />

Seems like neither of the above should work by looking at this error? What going on here? Suggestions for other ways of accomplishing the same task

Thanks for responses guys

Daniel Turini19-Nov-04 6:55

Daniel Turini

19-Nov-04 6:55

niceguyeddie wrote:
Operator '==' cannot be applied to operands of type 'System.Windows.Forms.Keys' and 'myApp.myKeys'
Each enum is a different type, you'll need to explicitly cast both to int to compare them.

Yes, even I am blogging now!

Dennis C. Dietrich19-Nov-04 6:59

Dennis C. Dietrich

19-Nov-04 6:59

niceguyeddie wrote:
What am I doing wrong here?

C# is type-safe and the type of KeyEventArgs.KeyCode[^] is Keys[^]. Therefore you have to explicitly cast the type:

if (e.KeyCode == (Keys)myKeys.SpecialKey1)
{
}

Lets say hypothetically that i'm making a video game and I want to make a button that fires a weapon.

I would definitely not use an enumeration to store the keys because a modern video game should provide a setup for reassigning the keys.

Best regards
Dennis

niceguyeddie19-Nov-04 7:25

niceguyeddie

19-Nov-04 7:25

thanks guys, that completly makes sense.. those are the things that make you pull hair out for an hour because the solution is so easy yet just can't seem to see it.

this is actually not for a video game, the keys won't have to be reassigned. I can recompile if I need to change something.

Heath Stewart19-Nov-04 8:12

19-Nov-04 8:12

On a different note (since your question was accurately answered) you should consider not hard-coding certain keys. Many people agree that a good game allows for key mapping, and - in a basic capacity - it's not that hard to add.

Lets say you host your game in a Form (this is typically the came even with Managed DirectX). There is a method you can override (actually several) named ProcessDialogKey. You would also keep a key map of what keys map to which actions. One way would be to use the hash table and an enumeration (like you already have) of Keys to Actions (assuming Actions is your action enumeration), which you could persist in a file, database, etc.

Take the following example (doesn't include persistence):

protected override bool ProcessDialogKey(Keys keyData)
{
  if (keyMap.ContainsKey(keyData))
  {
    Action a = (Action)keyMap[keyData];
    switch (a)
    {
      case Action.Shoot:
        Shoot();
        break;
      case Action.Duck;
        Duck();
        break;
      // ...
    }
    return true;
  }
  return false;
}

Depending on how your game's Form is set up, you may also need to set the Form's KeyPreview property to true to make sure that the form processes the keys before its child controls do.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

Daniel Turini20-Nov-04 5:47

Daniel Turini

20-Nov-04 5:47

Heath Stewart wrote:
switch (a)
{
case Action.Shoot:
Shoot();
break;
case Action.Duck;
Duck();
break;
// ...
}
return true;
}
Actually, you could get rid of this switch by storing interface pointers in the hashtable and simply call an "executeAction" method, or something like that.

Yes, even I am blogging now!