if the source is database means we can delete the duplicate
records by enabling the option select distinct in source
qualifier t/r properties or by writing the following query
in source qualifier filter
delete from emp where rowid not in (select min(rowid) from
emp group by empno);
if the source is flat file means by enabling the option
distinct in sorter t/r we can delete the duplicate records
Source and the target will the same table from which the
duplicate needs to be deleted from. Then use a lookup trans
on to this same table and then use an update trans with
DD_DELETE if the row is duplicate else do nothing.
There are 2 ways to do this and both of them are efficient.
Method 1: Sorter -Filter.
Send all the data to a sorter and , sort by all feilds that
u want to remove duplicacy from . note that in the
preoperties tab, select Unique .
This will select and send forward only Unique Data .
Method 2; Use an Aggregator
Use AGG Transformation and group by the keys /feilds that u
want to remove duplicacy from.