Drop duplicate records ...
SOURCE LIKE ..........
ID flag1 flag2
100 N Y
100 N N
100 Y N
101 Y Y
101 N Y
102 Y N
103 N N
104 Y Y
105 N N
106 N Y
102 N Y
105 Y Y
in above file if any id having both the flags as "N" then
that corresponding id records should be dropped,
in above case o/p should be as
ID flag1 flag2
101 Y Y
101 N Y
102 Y N
102 N Y
104 Y Y
106 N Y
Steps to do :
1) Identified the id’s that got duplicated (both the
flag values having vales “N”)
2) Look up with these id’s to existing id’s to drop .
- 2 Answers
- 4663 Views
- I also Faced
- E-Mail Answers
Answers were Sorted based on User's Feedback
Answer / dipal
step-1
Filter the record based on condition
Flag1=N AND Flag2=N ....link1
also defined a reject link
step-2
read link1 as left link and reject link as right link and
do inner join with Look up stage based on id
also define a reject link.
now the reject link will have required output.
| Is This Answer Correct ? | 3 Yes | 0 No |
Answer / vz
Put a constraint in Transformer stage as shown bellow.
flag1=y or flag2=y
means
feald1=y or feald2=y
I think it's help you.
| Is This Answer Correct ? | 3 Yes | 1 No |
types of errors in datastage?
What is data partitioning?
How the ipc stage work?
What is the method of removing duplicates, without the remove duplicate stage?
Field,NVL,INDEX,REPLACE,TRANSLATE,COLESC
In work load management there are three options of Low priority, Medium priority and High Priority Jobs which can be used for resource management. why this feature is developed when there is already jobs prescheduled by scheduler or autosys. what will be the use of workload management then?
How to exclude first and last lines while reading data into a sequential file(having some 1000 records).I guess probably by using unix filter option but not sure which to use
I have 100 records how can I load at a time from the single time
Does datastage support slowly changing dimensions ?
I have a file it contain 2 records like empname,company as Ram, Tcs and Ram, IBM. But i want empname, company1,company2 as Ram, TCS,IBM in the target. How?
Hi I am Vijay In my source i've 10 records in a single column.... but i want to split those records into 5 sequential files each seq file contains 2 records.?.... can any body help me?
15 Answers Scope International,
HOW CAN U DO ERROR HANDLING IN DATA STAGE?
Teradata 
