|
you can do this using SSIS package. Please check this
♫ 99 little bugs in the code,
99 bugs in the code
We fix a bug, compile it again
101 little bugs in the code ♫
|
|
|
|
|
Thanks for your Reply.
i apologize for reposting question in Asp.net and c#.
Would SSIS be capable for importing data from Large Xml Document(more than 10GB)?
did you try it?
|
|
|
|
|
I suggest you proceed cautiously.
That is a big file and the vast majority of solutions deal with smaller files.
If it was me I think I would do the following
1. Create an app that does nothing but divide it into smaller files. Might not even do validation.
2. Create a second app that consumes the smaller files, validates, logs the file processed and then post to the database.
Advantages in the above.
1. If an error occurs you are going to need to manually review it. And reviewing a file that is 10/100 meg is easier than 10 gig.
2. If an error occurs you can manually fix and continue with the file that errored, rather than restarting everything.
3. You don't need to deal with the issue that the file is just too big for some part of the process.
Regardless however you must have the following features.
1. Log as you complete each block (insert, whatever) of work so you not only know where something failed but also can track that it succeeded.
2. Insure that transaction blocks are small. 1000 is probably a good number.
3. Design for the possibility that something will fail so you have a way to restart it at the failure point.
|
|
|
|
|
I have 2GB RAM in my machine. i tried SSIS package for 1.5GB file it's giving error.
Error: 0xC02092AF at Data Flow Task, XML Source [1]: The component "XML Source" (1) was unable to process the XML data. Insufficient memory to continue the execution of the program.
If SSIS is not right choice than i have to follow the process as jschell suggest. can anyone give me some code sample for splitinf file into smaller one. i can split 10GB file into 10 1GB smaller files not more.
If i process 10 1GB files than how can i maintain transaction in all files? i mean to say if 1 file fails all others files should be rollbacked.
|
|
|
|
|
ntiin wrote: If i process 10 1GB files than how can i maintain transaction in all files? i
mean to say if 1 file fails all others files should be rollbacked.
I doubt that is reasonable. It might not even be possible.
|
|
|
|
|
Your reply to this topic has not just helped him... 
I never thought of divide and conquering files for those advantages, thanks!
|
|
|
|
|
Yes this is not much helpful for me .
i am tring SSIS Package to do so.
Can anyone help me on topic "SSIS Package for Importing XML files"?
|
|
|
|
|
This is the Sql-Query
SELECT "tempBillDetails"."SampleNumber", "tempBillDetails"."Description", "tempBillDetails"."Amount", "tempBilling"."BillNo", "tempBilling"."BillDate", "tempBilling"."PartyName", "NewSampleEntrys"."NameOfSample", "NewSampleEntrys"."BatchNo", "tempBilling"."AgmarkCharges", "tempBilling"."DisplayName", "tempBilling"."OtherCharges", "tempBilling"."Discount", "tempBilling"."OtherChargesFor", "Charges"."FTest", "Charges"."Flag"
FROM ("Quali"."dbo"."NewSampleEntrys" "NewSampleEntrys" INNER JOIN (("Quali"."dbo"."tempBillDetails" "tempBillDetails" left OUTER JOIN "Quali"."dbo"."tempBilling" "tempBilling" ON "tempBillDetails"."BillNo"="tempBilling"."BillNo") INNER JOIN "Quali"."dbo"."SampleRegistration" "SampleRegistration" ON "tempBillDetails"."SampleNumber"="SampleRegistration"."SampleNumber") ON "NewSampleEntrys"."QLID"="SampleRegistration"."QlCode") INNER JOIN "Quali"."dbo"."Charges" "Charges" ON "SampleRegistration"."QlCode"="Charges"."QlCode"
ORDER BY "tempBillDetails"."SampleNumber"
it work fine if all tables having data.
my problem is that it only "tempBilling" contain record and no matching record in other table it show nothing i want that tempBilling data sholud be display.
how can i solve this problem plz help.
modified on Monday, April 18, 2011 2:57 AM
|
|
|
|
|
|
Luc Pattyn wrote: Pick the correct join: Visual Representation of SQL Joins[^]
Gr8..entry for SQL Join, It's really helpful to understand SQL join.
Thanks
Md. Marufuzzaman
I will not say I have failed 1000 times; I will say that I have discovered 1000 ways that can cause failure – Thomas Edison.
|
|
|
|
|
Hi,
I have these related tables from access and it's killing me why can't they work like a master/detail in two datagridview controls when dragging them in the designer from the dataset tree to into the form.
The master works just fine, but the detail table is not loading the relative records.
Please help.
|
|
|
|
|
|
Thanks, it works fine now.. I just had to reinstall VS!!
|
|
|
|
|
Hi,
I want to make a database for an mlm project (binary tree). I'm confused
to choose data storing model. I've got two option "adjacency list model"
and "Modified Preorder Tree Traversal". Which one is better for this kind of project ?
My project may contain a large amount of records. Please suggest better model for my project.
Thanks & Regards
|
|
|
|
|
I suspect you're over-thinking the problem.
|
|
|
|
|
|
I have develop 3 binary tree related package. ( MLM and Microfinance). Just try to put Parent Id and thats enough to proceeds.. And if u have left right combination is there then put Position with varchar. which ontain Only R or L ( for left or Right.
|
|
|
|
|
Hi!
I have a small C# program that runs in multiple instances over a LAN and uses a database running on a seperate server. Now I need to make changes to the database layout by adding a few tables and extending an existing table with a few more columns.
How do I do this the right way? Can anybody point me in the direction of a tutroial or such information to get me started?
If it helps : The database is "Microsoft SQL Database 2008" and C# is the 2008 flavour.
Sure, I can make a new database and then export/import the from the old database, but I would much rather use a update through an installation-package so that the person updating don't have to be a certified tech. This change need's to be done on several locations and some of the users are just that - users 
Thanks in advance!
|
|
|
|
|
I'd write a script and ask them to run it.
|
|
|
|
|
Will the applications be running when you do the update?
When you say "several locations" you mean that there are the database itself exists in several locations?
Probably the absolute easiest way from the users stand point would be the following
1. Create a C# program that does nothing by applies the changes. Do NOT put this in the main app.
2. Create an installer which runs that app. It might also update the real app, see below.
Also as a suggestion the following might be useful.
1. Create a table which the sole purpose is to keep track of the database schema version. Thus for this release you would insert into that table something like 'MyApp', '2.0', <current date="">
2. Have your application verify the version on start up and refuse to run (error message) if the version does not match.
If you do this your C# database update program can verify the database version before applying the update. Note as well that you do not need to embed the update SQL in the database update application. It can read it from an included SQL file.
|
|
|
|
|
Thanks! That sounds like what I need 
Small clarification for posterity:
Each LAN installation has one databas and several C#-clients. THis installation is then done on several LANs.
|
|
|
|
|
Let's say I have a store procedure returning Value, Desired_Row_Header, and Desired_Column_Header.
EXEC [dbo].[scsp_My_Query]
Can I pivot the results of the stored procedure directly or do I have to embed the pivot in the stored procedure?
I am currently fetching the results off the database and manually creating pivot tables in code. I'd like to move it on the database, especially if I can do the pivot based on the column position without having to lookup the names of each column.
|
|
|
|
|
Have you tried
CREATE TABLE #tmp(Value varchar(20), Desired_Row_Header varchar(20), Desired_Column_Header varchar(20))
INSERT INTO #tmp
exec scsp_My_Query
Now do your pivot on #tmp
Common sense is admitting there is cause and effect and that you can exert some control over what you understand.
|
|
|
|
|
No, I hadn't tried doing that. It sounds like a good idea and I will try it out.
I have gotten some pivots to work with arbitrary # of columns, but I've had to use dynamic sql.
|
|
|
|
|
T2102 wrote: I have gotten some pivots to work with arbitrary # of columns, but I've had to use dynamic sql.
Yea if the columns before execution time are not known then you will need to resort to dynamic SQL.
Common sense is admitting there is cause and effect and that you can exert some control over what you understand.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
