-
Notifications
You must be signed in to change notification settings - Fork 0
/
103-4-use-streams-pipes-and-redirects
182 lines (152 loc) · 7.1 KB
/
103-4-use-streams-pipes-and-redirects
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
###########################################
103.4. Use streams, pipes and redirects
###########################################
Candidates should be able to redirect streams and connect them in order to efficiently process textual data. Tasks include redirecting standard input, standard output and standard error, piping the output of one command to the input of another command, using the output of one command as arguments to another command and sending output to both stdout and a file.
Objectives
& Redirecting standard input, standard output and standard error.
& Pipe the output of one command to the input of another command.
& Use the output of one command as arguments to another command.
& Send output to both stdout and a file.
& tee
& xargs
We've already talked about basics of piping and redirects in previous sections. Here you will get a deeper understanding of these concepts.
###########################################
Redirecting standard IO
###########################################
On a linux system most shells use streams for input and output (a list of characters). . These streams can be from (and toward) various things including keyboard, block devices (hards, usb stick, ..), window of a program, fiels, ...
1-stdout is the standard output stream, which displays output from commands (file descriptor 1)
2-stderr is the standard error stream, which displays error output from commands (file descriptor 2)
3-stdin is the standard input stream, which provides input to commands (file descriptor 0)
What are these file descriptions? There are used to control the output. If you need to control where your output goes, you can add n> or n>>.
* n> redirects file description n to a file or device. If the file already exists it is overwritten and if it does not exists, it will be created.
` * n>> redirects file description n to a file or device. If the file already exists the stream will be appended to the end of it and if it does not exists, it will
be created.
if the n is not given, the default is standard output
The user who runs the command should have write access to the file.
$ ls
fiona habib mahmoodrm minoo mojtaba sina
$ ls j*
ls: cannot access j*: No such file or directory
$ ls m*
mahmoodrm minoo mojtaba
$ ls j* m*
ls: cannot access j*: No such file or directory
mahmoodrm minoo mojtaba
$ ls j* m* > output 2> errors
$ cat output
mahmoodrm
minoo
mojtaba
$ cat errors
ls: cannot access j*: No such file or directory
$
#########################################################
Redirecting both stdout and stderr to one location
#########################################################
Sometimes (say during automated tasks) we prefer to send both standard output and standard error to same place, Use &> and &>> to say both stderr and stdout.
It is also possible to use &1 and &2 and &0 to refer to current place of stdout, stderr & stdin. In this case ls > file1 2>&1 means redirect output to file1 and output stderr to same place as stdout (file1)
Be careful! ls 2>&1 > file1 means print stderr to current location of stdout (screen) and then change the stdout to file1
###########################################
sending to null
###########################################
In linux, /dev/null is like a trash-can. You can send anything there and it disappears. So it is normal to say:
$ ls j* m* > file1
ls: cannot access j*: No such file or directory
$ ls j* m* > file1 2>/dev/null
$ cat file1
mahmoodrm
minoo
mojtaba
$
###########################################
redirecting input
###########################################
The < operand redirects the input.
$ cat uses
you fedora
jadi ubuntu
rubic windows
neda mac
narsin arch
$ tr ' ' ',' < uses
you,fedora
jadi,ubuntu
rubic,windows
neda,mac
narsin,arch
###########################################
here-documents
###########################################
Many shells, have here-documents (also called here-docs) as a way of input. You use << and a WORD and then whatever you input is considered stdin till you give only the WORD in one line.
$ tr ' ' '.' << END_OF_DATA
> this is a line
> and then this
>
> we'll still type
> and,
> done!
> END_OF_DATA
this.is.a.line
and.then.this
we'll.still.type
and,
done!
Here-Documnts are very useful if you are writing scripts and automated tasks.
###########################################
Pipes
###########################################
Piping is sending one commands output to another commands input (Piping the stdout to stdin). You use | for this task.
As previously seen, many commands use a hyphen - in place of a filename as an argument to indicate when the input should come from stdin rather than a file.
$ cat what_i_have.txt
laptop
socks
tshirt
ball
socks
glasses
$ cut -f1 -d' ' what_i_have.txt | sort | uniq -c | sort -nr
2 socks
1 tshirt
1 laptop
1 glasses
1 ball
If you need to start your pipeline with the contents of a file, start with cat filename | ... or use a < stdin redirect.
###########################################
xargs
###########################################
This command reads input from stdin and uses them as arguments.
$ ls | xargs echo these are files:
these are files: errors f file1 fiona habib mahmoodrm minoo mojtaba output output.txt sina uses what_i_have.txt
if you do not give any command to the xargs , the echo will be the default command (it will show the stdin).
Have in mind that xargs breaks input based on blanks and use any part as an argument. You can limit the number of arguments with --max-args (same as -n) switch and escape blanks or quote them to prevent them from breaking.
One important switch is -I. This is useful if you need to pass stdin arguments in the middle (or even start) of your commands. use the form xargs -I SOMETHING echo here is SOMETHING end:
$ cat what_i_have.txt
laptop
socks
tshirt
ball
socks
glasses
$ cat what_i_have.txt | xargs -I DATA echo I have DATA and I love it.
I have laptop and I love it.
I have socks and I love it.
I have tshirt and I love it.
I have ball and I love it.
I have socks and I love it.
I have glasses and I love it.
If you use -L, the input will break by line and not by blanks.
###########################################
tee
###########################################
What if you need to see the output on screen and also save it to a file? Ugly way is redirecting to the file and using tail -f file in another window. Nice way is using tee and giving it one or more filenames for standard output (if you need to save stderr, first redirect it to stdout). This will write the output to those files and also writes them to the screen:
$ ls -1 | tee allfiles myfiles
allfiles
f
fiona
habib
mahmoodrm
minoo
mojtaba
myfiles
sina
if you want to prevent overwriting files, use the -a switch to append to files if exists.