How do I create unit test for etl job

Question

1.00/5 (2 votes)

See more:

how should i create ETL test cases...
what we do is get all data from test server, process and load into some other table.
i need some sample code to test for such scenarios

What I have tried:

1. how to create unit test project for etl job
2. how to test if data is being loaded or not

Posted 3-Sep-21 13:09pm

BishnuKarki

Updated 5-Sep-21 22:06pm

Add a Solution

Comments

James Walsh Jr 3-Sep-21 20:15pm

Without knowing what you're using for your ETL process, it's impossible to answer this question.
What ETL tool are you using, what platform? What have you tried? Are you getting a specific error?
Alot more detail is required before anyone can help you.

PIEBALDconsult 4-Sep-21 0:05am

No idea, but you kinda don't.
I've used SSIS quite a bit, but otherwise I prefer to write my own ETL tools.
The situation I'm in now is that my boss has asked me to verify that other team members have "loaded the data correctly". I could just weep.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Gerry Schmitz · Answer 1 · 2021-09-05T22:06:00

Solution 1

You create "expected output"; e.g. a table, work book, flat file, csv, whatver. Then you run your ETL and compare it to "expected". They should match for a successfull test; you do a tablec compare; file compare; etc. using any "compare tool", including SQL. That's how audit groups do it. The "expected output" is retained for any future regression / unit testing.

Posted 5-Sep-21 22:06pm

Gerry Schmitz

Comments

PIEBALDconsult 6-Sep-21 12:18pm

But that might not flex the same bits of functionality which is to be tested.
And there could be a problem in the code you use during your "test" which is not present in the code you are meant to be testing -- leading to a false-positive or -negative result.

[no name] 10-Sep-21 12:42pm

It's an "extract", then an (optional) translate, then a load. It's not that complicated. The "audit groups" I worked with would create master files of how they "should look" after a series of transactions agains a "known" starting master. Finance. No idea where you're coming from.

PIEBALDconsult 10-Sep-21 12:47pm

Loading from "master files" as you suggest is not the same as loading from a database as the OP implies -- you would be testing a different code path.