There are several ways of achieving this. We can do it through expression

How do we eliminate duplicate records in a flat file without using Sorter and Aggregator?

Question Posted / priyank

7 Answers
21791 Views
iNautix, I also Faced
E-Mail Answers

Answer Posted / priyank

There are several ways of achieving this. We can do it
through expression transformation and other is look up on
the target.

Expression transformation:

Create ports,

Var_PREV_KEY=Key
Var_CURR_KEY=Var_PREV_KEY
Var_CHK_DUPLICATE --> IIF(Var_CURR_KEY=Key,'DUP','NODUP')
OUT_DUPLICATE --> Var_CHK_DUPLICATE

Note: I have taken a scenario where the target table
contains only 1 Key. In case of multiple keys, will have to
create a few more Variable ports for both CURR and PREV and
in the Var_CHK_DUPLICATE port, we need to add those checks
with an 'AND' operator.E.g. For 2 keys,

Var_PREV_KEY1=Key1
Var_CURR_KEY1=Var_PREV_KEY1
Var_PREV_KEY2=Key2
Var_CURR_KEY2=Var_PREV_KEY2
Var_CHK_DUPLICATE --> IIF(Var_CURR_KEY1=Key1 AND
Var_CURR_KEY2=Key2,'DUP','NODUP')
OUT_DUPLICATE --> Var_CHK_DUPLICATE

If the Informatica version is Unix installation, then in
the pre session command you can give an unix command to
remove the duplicates from the file like

sort <file_name> | uniq > <file_name>.new

Hope it helps.

Is This Answer Correct ?

4 Yes

12 No

Post New Answer View All Answers

Please Help Members By Posting Answers For Below Questions

what is size of u r database?

2319

What is the sql query overwrite in source qualifier in informatica

1099

Explain where to store informatica rejected data? How to extract the informatica rejected data?

1020

what are factless facts? And in which scenario will you use such kinds of fact tables.

1817

How to use procedural logic inside infromatica?

1255

Mention few power centre client applications with their basic purpose?

961

What are the various test procedures used to check whether the data is loaded in the backend, performance of the mapping, and quality of the data loaded in informatica?

1089

hi real timers . iam waiting for ur reply regarding ETL TESTING

2282

What are connected or unconnected transformations?

1129

What are the steps involved in the migration from older version to newer version of Informatica Server?

2119

can we override a native sql query within informatica? How do we do it?

1135

Explain lookup transformation source types in informatica

1108

wf dont have integration severances how you can run?

1947

What is a dimensional model?

993

What are different types of transformations available in informatica?

1005