如何删除重复的行（sed或awk）

najia01

现有文件如下：
---------------------------------------------------
my friends, chenhong
my friends, chenhong
my friends, chenhong
my teacher, liyong
my teacher, liyong
my teacher, liyong
my father, wuzhongyi
my father, wuzhongyi
my father, wuzhongyi
my sister, wushiying
my sister, wushiying
my sister, wushiying
---------------------------------------------------
现在欲把文件变成如下：
---------------------------------------------------
my friends, chenhong
my teacher, liyong
my father, wuzhongyi
my sister, wushiying
---------------------------------------------------
该如何用sed或者awk，如何解决？

doubleelec

一定要 sed 或 awk 吗？用 uniq 行不?

najia01

也行，如何使用？
我个人觉得，sed awk grep组合应能解决所有文本处理的问题。
呵呵～～

doubleelec

uniq:

1. 
   
2. uniq file
   

复制代码

awk:

1. 
   
2. awk '{if ($0!=line) print;line=$0}' file
   

复制代码

najia01

可否解释一下这行的意思：
-------------------------------------------------------
awk '{if ($0!=line) print;line=$0}' file
-------------------------------------------------------
谢谢！

najia01

我猜大概是这样：
line应为一个变量。

if ($0!=line) print;line=$0;
可以这样理解：
因为awk也是一次读入一行，line第一次为空，所以自然就不等于$0（$0为
"my friend,chenhong"），所以就打印了；接着把line的值赋为$0;然后awk又读
入一行，由于此时$0的值与line相同（均为"my friend,chenhong"），所以就不
打印了。当读入"my teacher, liyong"时，$0与line(值为
"my friend,chenhong")又不同了，所以打印出来。
其余的以此类推。

不知道这样理解，对不对？

doubleelec

没错

如果用 sed 就比较难了，谁来一个？

galilette

sed -f rmdup.sed yourfile
here is the rmdup.sed sed script:

1. 
   
2. #n rmdup.sed - ReMove DUPlicate consecutive lines
   
3. 
   
4. # read next line into pattern space (if not the last line)
   
5. $!N
   
6. 
   
7. # check if pattern space consists of two identical lines
   
8. s/^\(.*\)\n\1$/&/
   
9. # if yes, goto label RmLn, which will remove the first line in pattern space
   
10. t RmLn
    
11. # if not, print the first line (and remove it)
    
12. P
    
13. 
    
14. # garbage handling which simply deletes the first line in the pattern space
    
15. : RmLn
    
16. D
    

复制代码

najia01

如果文件a如下：
---------------------------------------------------
my friends, chenhong
my teacher, liyong
my teacher, liyong
my father, wuzhongyi
my sister, wushiying
my sister, wushiying
my friends, chenhong
my teacher, liyong
my father, wuzhongyi
my sister, wushiying
my friends, chenhong
my father, wuzhongyi
---------------------------------------------------
现在欲把文件变成如下：
---------------------------------------------------
my friends, chenhong
my teacher, liyong
my father, wuzhongyi
my sister, wushiying
---------------------------------------------------
又该如何？

galilette

use `sort' first. there is no EFFICIENT way of sorting in sed/awk