C / C++ / MFC

19-Jul-04 7:01

Hi,

I'm developing a database application using CDatabase & CRecordSet classes. My SQL Server database
is sitting on other machine and it's effecting application performance when I tried to insert large number of records
and retrieving the records for generating some reports(i.e. to fetch 1000000 records at a time for creating a report). Application is getting too slow.

Need some suggestions in this regard.

Thanks,

G. Steudtel19-Jul-04 7:10

G. Steudtel

19-Jul-04 7:10

Hi,
probably instead of using mysql_store_result you should using mysql_use_result. It will give you a boost, but also a lot of drawbacks. See the C-API documentation for that.

GSte

Ivan Cachicatari19-Jul-04 7:12

Ivan Cachicatari

19-Jul-04 7:12

Hi,

I recommend to segment this query or install more physics memory.

navisoft
www.latindevelopers.com

Alexander Wiseman19-Jul-04 7:25

19-Jul-04 7:25

How are you using CRecordset? Are you predefining the query and the columns it returns with CRecordset, or are you simply creating a CRecordset from a query at run-time? If you are doing the former (predefining the query and the columns), there are some operations in which CRecordset performs very inefficient methods on the data, and it could be that the slowness you are seeing is because CRecordset is clogging the system memory.

Check which way you are using CRecordset and also pull up Task Manager while your long query is running to see how much system memory it is sucking up.

Sincerely,
Alexander Wiseman

Est melior esse quam videri
It is better to be than to seem

Sridhar Sanikommu19-Jul-04 7:35

19-Jul-04 7:35

Hi,

I'm using the former one. I have to read a text file which has more than million records and insert each record at a time. Also, for genearting some reports, i've to fetch huge number of records at a time from the database and process all the records.

Thanx,

Alexander Wiseman19-Jul-04 10:14

19-Jul-04 10:14

So, just to clarify exactly what you are trying to do: you are opening and reading a text file from which data you want to insert records into a database. Because you want to insert so many records, I would suggest using a SQL command to insert the data (INSERT INTO...) instead of CRecordset::AddNew() and CRecordsSet::Update(). The CRecordset functions for adding records tend to be slower than simply executing a SQL statement against the database. You can execute a SQL statement by using the function in CDatabase (I forget the exact name, but it is something like "Execute").

For fetching large numbers of records, I would also suggest that you move away from the default RFX (Recordset Field Exchange - where you predefine the columns returned from the query) implemented with CRecordset. Again, this tends to be slower than simple qeurying the database with SQL statement and not using RFX. For some more information on this topic, take a look at the ODBC section on CodeProject: http://www.codeproject.com/database/#ODBC[^].

If you need more specific information, or some example code, please let me know.

Sincerely,
Alexander Wiseman

Est melior esse quam videri
It is better to be than to seem

Sridhar Sanikommu19-Jul-04 11:28

19-Jul-04 11:28

Hi Alexander,

Thanks for your input. Right now, I'm not using RFX for fetching the records nor
AddNew() and Update() functions of CRecordset classes. As you said, i'm reading each record from the text file and create INSERT INTO.. query and execute it using CDatabase
ExecuteSQL() function. However, for every call to this function, database has to parse , prepare an execution plan and compile it. Instead i'm thinking of using stored procedures. But my application has to support multipe databases like MSAccess, MySQl, Oracle & MS SQL Server. I don't think MSAccess supports Stored procedures and I'm not sure of MySQL either.

Do you have any generic database module? If so, can you send it to me?

I would appreciate your help.

Thanx,

Alexander Wiseman19-Jul-04 15:52

19-Jul-04 15:52

Hi Sridhar,

Unfortunately, there won't be something which you can code which will take the place of a stored procedure for those DBs that don't support them. MySQL should have something you can use for optimization, although I thought it supported stored procedures. Maybe your best bet is to implement the insertion code inside a worker thread, so the main application UI doesn't seem like it isn't responding when you are doing the insert. Long operations, when necessary, are OK as long as you give the user some indication of what is actually happening and maybe a "Stop" button.

Although I can't think of anything else right now, I will keep my eyes open for something that can help you.

Sincerely,
Alexander Wiseman

Est melior esse quam videri
It is better to be than to seem

Anonymous19-Jul-04 16:36

Anonymous

19-Jul-04 16:36

Hi Alexander,

I thought of implementing intensive database operations in the worker thread.
Think this strategy will work for inserting records. But, what about fetching if you implement it in the worker thread?

I mean, if my query is taking X number of sec or minutes to fetch Million plus records(Select * from table where condition). HOw do I cancel the worker thread if it's still in the process of retrieving?

Some insight would be greatly appreciated.

Thanks,

Sridhar Sanikommu19-Jul-04 16:38

19-Jul-04 16:38

Finding files recursively

Steen Krogsgaard19-Jul-04 23:01

Steen Krogsgaard

19-Jul-04 23:01

Hi,

MSAccess (or rather Jet, the dbengine), supports stored procedures, at least it supports CREATE PROC and EXECUTE statements. See http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnacc2k/html/acintsql.asp[^]
Interestingly, this article on MSDN http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/vcconstoredprocedures.asp[^] says that the OLEDB provider for Jet does not support stored procs, but it nevertheless works for me.

Cheers
Steen.

"To claim that computer games influence children is ridiculous. If Pacman had influenced children born in the 80'ies we would see a lot of youngsters running around in dark rooms eating pills while listening to monotonous music"

Ivan Cachicatari19-Jul-04 6:55

Ivan Cachicatari

19-Jul-04 6:55

Hello everyone,

I am searching the way to find for in archives and folders recursively.

You can say me the optimal way to do it?

Thanks!

navisoft
www.latindevelopers.com

Re: Finding files recursively

Alexander Wiseman19-Jul-04 7:21