i have two sources two flatfiles same structure
one flatfile contain 100 million recs
another flatfile contain 10 million recs
i have to connect single target(performance oriented answer)
what are the steps we have to do this scenario
Answers were Sorted based on User's Feedback
Answer / babu
Hi Teja,
The two sources are haveing same structure,so beter to
use list file,to load the data into single TARGET.
Procedure:
----------
1.Place the two source paths in a note pad and
save as it '.list' format.
2.create a maping as per your requirement in
that mapping you have to take only one source no need to
take two sources.
3. Create a session and workflow.
4.In session give a SOURCE FILE PATH
is '.list' file and SOURCE FILE TYPEA as INDIRECT.
Let me know if any thing wrong.
Thanks,
dwarak.vbr@gmail.com
| Is This Answer Correct ? | 16 Yes | 0 No |
Answer / srikanth
Hi,
Both teja and Babu Answers are correct..
But here we need to consider Performance wise which one is
better.
here we have only two flat files one is having 100 million
records and second one having 10 million records.
union transformation can supports for flat files also and it
is acting as union all(don't worry about this).
first drag the 100 million records source and then drag the
10 million records source on mapping designer.
connect these two sources to union transformation.
union transformation can merge the Data By COLUMN by COLUMN.
so it is trying to merge only 10 million records with
existing 100 million records
Then we can tune the performance here.
if we have lot of flat files(same structure) then we have to
go for file indirect.
| Is This Answer Correct ? | 1 Yes | 0 No |
Answer / sudheer
As u r using source as flat file so u can go for partitioning preferably Single-Thread or Multi-Thread partitioning.
| Is This Answer Correct ? | 0 Yes | 0 No |
Answer / tejaetl
i answered this question both the flatfiles connet to union
connect to single target
| Is This Answer Correct ? | 0 Yes | 1 No |
What do you mean by worklet?
The question was on time stamp. what is the difference between HH and HH24 when to use when.
Insert else update option in which situation we will use
How to load last n records of file into target table - informatica
can anyone suggest best free Talend data integration training online
write a query to follwoing table amount year quarter 254556 2003 first 546332 2003 second 129034 2003 third 490223 2003 fourth 165768 2004 first 265443 2004 second 510412 2004 third 435690 2004 fourth i want the output year q1_amount q2_amount q3_amount q4_amount 2003 254556 546332 129034 490223 2004 165768 265443 510412 435690
Can you create a flatfile target
Partition, what happens if the specified key range is shorter and longer
I have 4 columns in a table say, name ,address, salary and city. So based on distinct city names, i need to combine the other data and send it to a flat file. Distinct city names can be 4 or 5 or 6. So it has to dynamically generate flat files(4 or 5 or 6) corresponding to the city names.
wht target override?wht advantages it has compare to target update?
Explain in detail scd type 2 through mapping.
if the session continuous run what can we do. means shall we stop the session or abort session. what is the correct reasion. why this situation occurs