Click here to Skip to main content
15,881,173 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
I am using Azure data factory(ADF) Pipeline-Copy Activity to copy data from cosmos db to datalake.I have created linkedservice for source (Cosmos db ) and destination (Data lake gen 2- Delimited format (.tsv))as I want my output in tsv format or csv..
which is working fine for me for simple objects.
But,I want some suggestion for copying complex objects from cosmos db to flattened tsv format.

eg:-> Source :->
{
"id":"30b00315-bae9343442333",
"TestId":"12345",
"Information":
[
 {"Info":
  [
   {"InfoID":1,"result":"4"},
   {"InfoID":2,"result":"3"},
   {"InfoID":3,"result":"3"},
   {"InfoID":4,"result":"4"},
  ]

 "lastModified":"2021-01-2T08:36:09.4961916",
 "lastModifiedBy":"SYSTEM",
 "flag":1
 }
],
"created":"2021-01-12T08:36:09.49561916",
"createdBy":"SYSTEM",
"isNew":true,
"eTag":"\"6100478c-0000-0700-342343440000\""
}

Destination should be in:->
Id	                   TestId InfoId Result	LastModified LastModifiedBy	CreatedBy flag
30b00315-bae9343442333	12345	1	 4	    2021-01-2   	System	   2021-01-2	1
30b00315-bae9343442333	12345	2	 3	    2021-01-2	    System	   2021-01-2	1
30b00315-bae9343442333	12345	3	 3	    2021-01-2	    System	   2021-01-2	1
30b00315-bae9343442333	12345	4	 4	    2021-01-2	    System	   2021-01-2	1


As you can see above in source,Information is array and inside it,there is another field Info which is another array.I want this complex nested arrays in flattened structure.
How we can achieve it? Please suggest some way.


Thanks in advance.


What I have tried:

1.Copy Activity Mapping schema :- But it will take only first index.
2.I have also tried for Data flow activity But it wont work for nested array objects.Let me know if i am missing something from my end.
Posted
Comments
[no name] 7-Jun-21 14:52pm    
https://stackoverflow.com/questions/66233023/export-azure-cosmos-db-table-to-csv
Prasad Nikumbh 8-Jun-21 2:32am    
Thanks for your reply.But i want the result in flattened format only.
[no name] 8-Jun-21 13:41pm    
I can't "see" the csv contents, but I'll take your word for it.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900