|
I am designing a distributed application which is implemented using remoting. Basically the server side remote object is used to retrieve information from the database.
Currently in this object i am using a single dataset to store all data, which runs fine as we have not yet tested against large volumes of data.
Now the thing is that i would like the application to be scalable for which i might have to use things like object pooling, now if i pool my objects then each object will have a differnet instance of the dataset and there will be no synchronization between them. But i have a feeling that directly manipulating the database will degrade my performance.
Anyone have any stats, ideas or suggestions or any experience with a similar problem?
May the Source be with you
Sonork ID 100.9997 sijinjoseph
|
|
|
|
|
Avoid using large datasets in remoting applications. Even if you specify binary serialization, the contained objects (datatables, datarelations) still get serialized to HTML diffgrams (very verbose) and consume immense ammounts of memory and bandwith if many rows are involved. My experience is that this also leaks memory (although this may just have been due to costruction in the "large object heap" which seems only to be garbage collected if the moon is full). This MSDN magazine article <a href="http://msdn.microsoft.com/msdnmag/issues/02/12/CuttingEdge/default.aspx">http://msdn.microsoft.com/msdnmag/issues/02/12/CuttingEdge/default.aspx</a>[<a target=_blank title='New Window' href="http://msdn.microsoft.com/msdnmag/issues/02/12/CuttingEdge/default.aspx">^</a>] goes into much more detail, and suggests some workarounds.
We ended up loading a custom class instance (basically an arraylist of arraylits), serializing that, deserializing and reconstructing a dataset on the client side. Cut memory usage by factor of 4 and increased performance by factor of 3. No more memory leaks.
When all else fails, read the directions...
OldRob
|
|
|
|
|
Thanks a lot mate, but actually i don't actually pass the dataset over the wire at any time but still a very informative link.
Could you tell me what your experience was in using a dataset i mean for large databases with the enitre database in memory do you think it will be a good solution?
May the Source be with you
Sonork ID 100.9997 sijinjoseph
|
|
|
|
|
Depends on what you mean by large ... If you plan to keep several million rows of two or three hundred columns in memory as a read only db, I would think this would strain even a large box. Not too sure the class is thread-safe for multiple readers(enumerators...), pretty certain it is unsafe for multiple writers. We were working with a 30GB database,so no thought of keeing it in ram. Datasets/DataAdapters are not real performant for heavy insert/update use, sqlcommands with stored procs work best for this, and sqlDataReaders are much more performant for populating read-only caches...
The DataSet is a rather complex structure, carring a lot of overhead (duplicate copies of all the data, state information for every row, complete schema information... really is an in-memory db of sorts)
Also, the remoting issue is just as severe when passing data between processes in the same box (appdomain to appdomain). We were fairly convinced that this resulted in real memory leaks with datasets of 10000 rows x 300 columns or so (worse if multiple datatables adding up to this..)
OldRob
|
|
|
|
|
Hello,
I have a database design problem,
I'm using MS Access databases btw.
I have several tables, which need to be syncronised between databases.
There are 3 databases and records can be added to any of the 3 and need to
be updated on the others.
Table T1 has a primary key which is referenced by T2.
Now to make an update to a remote database I came up with the following.
Create a table called Database which has an unique ID.
Create a table called Mapping which has the following
Id (AutoNumber)
TableId - unique number 1- T1 , 2-T2
DatabaseId - Id from Database Table
RemoteId - Remote Id of this entry
LocalId - local Id
To syncronize, when a remote client connects to the local database, it goes through table T1, and looks in the Mapping table for the localId, with its DatabaseId. If it is not found, it is a new entry and creates a new T1 entry on its (remote) database. It also creates a Mapping entry on the local database and remote database.
Does this sound like a good plan?
Or are there existing methods which work better?
Cheers
John
|
|
|
|
|
I have done a db program.Using ado to access oracle806 db,I used Oracle OleDB provider to improve performance.
It's fine on my pcs(xp/98,oracle 817 client) .However ,
After release it(using vc++6.0 install shield),it does not work on some pcs.It seem program do not connect to oracle.
On others,it work well.Try connecting the oracle with SQL*PLUS,i find the net8 is ok.What's wrong with my program? I suppose the program is Oracle OleDB provider.
Help me ,please.
|
|
|
|
|
Greetings,
I am getting an exception when I use Append and adDBTimeStamp when creating a table with this parameter.
I am using visual c++ .net.
Here is my code below.
tblNew->Columns->Append(_bstr_t("MyDate"), adDBTimeStamp,0);
I get an exception when I try to create this, can anyone give me some advice?
Thanks in advance!
|
|
|
|
|
What exeption you get?And do use sql or access?
Mazy
"And the carpet needs a haircut, and the spotlight looks like a prison break
And the telephone's out of cigarettes, and the balcony is on the make
And the piano has been drinking, the piano has been drinking...not me...not me-Tom Waits
|
|
|
|
|
Hello,
I am using sql server.
It doesn't give the exception name, and it skips the exception handler below.
catch(_com_error &e)
{
}
I did get it to work using "CREATE TABLE", and using a "datetime" parameter,
so I do have a workaround. But it seems that the ADOX call didn't work.
I can't seem to get my query working now. Here is a query I had previously used
using ADO to query an Access database, but does not seem to work with SQL server.
The query:
----------
USE MyTestDB
SELECT *
FROM MyTestTable
WHERE StartTime>= #10/11/2002# and StartTime <= #11/11/2002#
It complains about the "#" symbol and says it is a syntax error.
Any ideas as to what symbol I should use in it's place, or a different way
to do my query.
Thanks for answering my post!
|
|
|
|
|
Use ' instead of '#'.Access ans sqlserver have some differences in some syntax.And also I dont think you need USE MyTestDB.You have to login to that database with your connection string.For your other problem,I suggest to see the samples of MSDN about 'Append' and other functions that you use
Mazy
"And the carpet needs a haircut, and the spotlight looks like a prison break
And the telephone's out of cigarettes, and the balcony is on the make
And the piano has been drinking, the piano has been drinking...not me...not me-Tom Waits
|
|
|
|
|
Thanks for the tip about the "'" symbol, and the msdn "Append"!
|
|
|
|
|
I am trying to write a tool to help me explore my MSDE databases, and I can connect fine, but I cannot figure out how to enumerate the available databases or tables so that I can show them. Any help appreciated.
Christian
No offense, but I don't really want to encourage the creation of another VB developer. - Larry Antram 22 Oct 2002
Hey, at least Logo had, at it's inception, a mechanical turtle. VB has always lacked even that... - Shog9 04-09-2002
During last 10 years, with invention of VB and similar programming environments, every ill-educated moron became able to develop software. - Alex E. - 12-Sept-2002
|
|
|
|
|
Hi, Christian:
You can use ADOX for more serious work, the sp_help stored procedure or use the not-so-documented system tables for this:
select * from master.dbo.SysDatabases
User tables:
select * from SysObjects where type = 'U'
System tables:
select * from SysObjects where type = 'S'
[edit]
You can also try this in SQL Server 2000:
select * from INFORMATION_SCHEMA.TABLES
[/edit]
lazy isn't my middle name.. its my first.. people just keep calling me Mel cause that's what they put on my drivers license. - Mel Feik
|
|
|
|
|
Daniel Turini wrote:
select * from INFORMATION_SCHEMA.TABLES
This also works in SQL Server 7
Dave.
|
|
|
|
|
Davey wrote:
This also works in SQL Server 7
I should have say "I've only tested it on SQL Server 2000"
lazy isn't my middle name.. its my first.. people just keep calling me Mel cause that's what they put on my drivers license. - Mel Feik
|
|
|
|
|
Thanks, but while this has got me started, the fact is that I can't find a way to list all the tables of a specific database.
Daniel Turini wrote:
select * from INFORMATION_SCHEMA.TABLES
is listing everything in master, but nothing from Northwind, the other DB I have created, or any others.
I don't want to use ADOX if I can avoid it. I'm hoping I can do something like select * from TABLES where DBName = 'Northwind'.
Christian
No offense, but I don't really want to encourage the creation of another VB developer. - Larry Antram 22 Oct 2002
Hey, at least Logo had, at it's inception, a mechanical turtle. VB has always lacked even that... - Shog9 04-09-2002
During last 10 years, with invention of VB and similar programming environments, every ill-educated moron became able to develop software. - Alex E. - 12-Sept-2002
|
|
|
|
|
As usual, as soon as I ask, I solve it
Christian
No offense, but I don't really want to encourage the creation of another VB developer. - Larry Antram 22 Oct 2002
Hey, at least Logo had, at it's inception, a mechanical turtle. VB has always lacked even that... - Shog9 04-09-2002
During last 10 years, with invention of VB and similar programming environments, every ill-educated moron became able to develop software. - Alex E. - 12-Sept-2002
|
|
|
|
|
Hello,
I would really like to use .NET for an application that has a website interface and a windows downloadable C++ MFC application. Both need a database backend.
For the web application, I have chosen, ASP and Access ( migrate to SQL 7 later)
For the windows C++ MFC app, the choice is not so clear.
I need the smallest possible size for the windows download and it needs to be backwards compatible to windows 98.
Even using access would require the MDAC redist which is 5MB...too large.
I thought about writing my own small database interface,
however it feels like reinventing the wheel.
.NET would do the job nicely, but a 20MEG download???
Fine for people with cable but not for 56k's
Any other suggestions?
Cheers
John
|
|
|
|
|
I would structure this as follows:
For the web application: ASP.NET. Easy to develop, fast, scalable and, because the server is only 1 machine, a one-time 20Mb download will not be a hassle.
For the client: MFC C++ application. Obvious solution.
It's not clear why you need databases on the client: accessing the server database or a local "cached" database?
A) Accessing the server database: develop a web service. It's fast and as easy as running a wizard on VS.NET. The client side is easier with C# or MC++, but even with MFC is a matter of using a couple of CInternet* MFC classes and a bit of MSXML for parsing the resulting document. It's fast to develop and flexible.
B) Local database: if the database is small, you can replace it for a small XML file. This way you do not need to reinvent the wheel, and do not become "glued" to a proprietary file format.
Remember, a XML file is not a high-performance solution for massive database manipulation, specially if you need lots of "GROUP BY", but for simple searchs and updates, MSXML is quite fast and can be a compatible solution with Windows 98.
lazy isn't my middle name.. its my first.. people just keep calling me Mel cause that's what they put on my drivers license. - Mel Feik
|
|
|
|
|
For the Client:
It will be offline mostly which is why the need for a local cached version. It requires alot of add, del, insert,select, update operations to be done on the local database.
A) accessing server database
I had considered using MSXML(XMLHTTP ) for the accessing the server database, but hadnt considered using MSXML for the local database.
Not sure if MSXML is appropriate for database solution.
Is a small DAO redistributable still availble from MS?
In that case I could use DAO?
Cheers
John
|
|
|
|
|
johnb wrote:
Is a small DAO redistributable still availble from MS?
In that case I could use DAO?
Argh!
I would keep a safe distance from DAO, if I were you.
And I have some nightmare stories I lived with MDAC redistributables, too 
lazy isn't my middle name.. its my first.. people just keep calling me Mel cause that's what they put on my drivers license. - Mel Feik
|
|
|
|
|
Maybe ADO is the answer, XP, 2000 already have a version of MDAC installed,
I guess, win98 users will have to install MDAC2.7 approx 5MB download.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
