1. Categories >> Software >> Data Warehouse >> Data Stage
  2. Suggest New Category

How can remove duplicates in a file using UNIX?

Question Posted / vishwababu

Answers were Sorted based on User's Feedback



How can remove duplicates in a file using UNIX?..

Answer / j madhava rao

$sort -u filename will sorts the data and removes the duplicates lines from the output

Is This Answer Correct ?    17 Yes 1 No

How can remove duplicates in a file using UNIX?..

Answer / vishwababu

You can use UNIQ to remove duplicates frm a file but it can
consider only same consecutive records.
Ex: if u hve records as
1 vishwa
1 Shruti
2 Ravi
3 Naveen
1 vishwa
and if u use UNIQ cmd, then you wil get vishwa, Ravi, Naveen
and vishwa.
So perform sort before using UNIQ so tat to avoid duplicates

Is This Answer Correct ?    7 Yes 7 No

How can remove duplicates in a file using UNIX?..

Answer / prasad

Uniq -u File_name

It will remove duplicates records.

Is This Answer Correct ?    0 Yes 2 No

Post New Answer

More Data Stage Interview Questions

Create a job to load all unique products in one table and the duplicate rows in to another table. The first table should contain the following output A D The second target should contain the following output B B B C C Q2. Create a job to load each product once into one table and the remaining products which are duplicated into another table. The first table should contain the following output A B C D The second table should contain the following output B B C

3 Answers   Deloitte, IBM,

What are the benefits of datastage?

0 Answers  

Differentiate between datastage and informatica?

0 Answers  

how to handle null values in sequential file?

1 Answers   HCL,

which unix commands mostly used in datastage

3 Answers   HSBC,




Out of 4 mill records only 3 mill records are loaded to target and then job aborted. How to load only those 1 mill(not loaded records) for next run. This job is not sequential job, it is stand alone parallel job.What are the possibilities available in datastage8.1?

7 Answers   IBM,

how many stages are there in Datastage... Please clarify mee??

3 Answers  

Hi Can any one help regarding below INPUT NAME LOC Ram hyd Ram ban Raj chn Raj Pun Sam del OUPUT NAME LOC Ram Hyd ban Raj chn pun sam del

3 Answers   HCL,

What are the various kinds of the hash file?

0 Answers  

How to Remove Duplicate using SQL?

0 Answers   CTS,

how to configure databases through datastage

2 Answers  

Explain how a source file is populated?

0 Answers  

For more Data Stage Interview Questions Click Here
Categories