ch04-sed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Chapter 04: `sed` Tutorial\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sed` is the stream editor which can conduct text transformation on the input stream (text file or input pipeline).\n",
    "\n",
    "Here is the basic command synopsis for `sed`:\n",
    "```bash\n",
    "sed SCRIPT INPUTFILE\n",
    "```\n",
    "\n",
    "And of course `sed` has its own full synopsis:\n",
    "```bash\n",
    "sed [OPTIONS] SCRIPT INPUTFILE\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synopsis\n",
    "```bash\n",
    "sed -n -e '{COMMAND}' FILENAME  ## 没有地址的情况，这时候不用花括号也可\n",
    "sed -n -e 'ADDR1{COMMAND}' FILENAME  ## 单个地址的情况\n",
    "sed -n -e 'ADDR1,ADDR2{COMMAND}' FILENAME ## 一对地址的情况，表示一个范围\n",
    "```\n",
    "\n",
    "<font color='red'>NOTE</font>: Although the option `-e` is not compulsory, we still recommend you to use this option especially when you have multiple commands (scripts) to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zero\r\n",
      "one\r\n",
      "two\r\n",
      "three\r\n",
      "four\r\n",
      "five\r\n"
     ]
    }
   ],
   "source": [
    "sed -n -e 'p' re/numbers.txt | head -6\n",
    "#相当于sed -e '' re/numbers.txt，因为没有用-n选项，默认输出模式空间的内容"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Addresses\n",
    "\n",
    "1. When no addresses are given, command will be executed on all the input lines;\n",
    "2. When only one address is given (`3p`), command will be executed on the matching line; the address can also be regular expression\n",
    "3. When two addresses are given (`3,5p`), commmand will be executed on the range between the two lines, the two addresses can be either __numeric__ or __regular expression__.\n",
    "\n",
    "\n",
    "\n",
    "#### <font color='blue'>Exercise</font>\n",
    "1. Have a look at the manual of `sed`, tell what the following command will do:\n",
    "```bash\n",
    "sed -n -e 'p' re/numbers.txt     # no address\n",
    "sed -n -e '3p' re/numbers.txt    # single address\n",
    "sed -n -e '/^t/p' re/numbers.txt # single address, but defined by a regex\n",
    "sed -n -e '3,5p' re/numbers.txt  # two addresses, starting from 3, ending at 5\n",
    "sed -n -e '/^o/, /^t/p' re/numbers.txt # two regexes as starting and ending address\n",
    "sed -n -e '0~5p' re/numbers.txt  # starting from head, every 5\n",
    "sed -n -e '1,+5p' re/numbers.txt # starting from 1, continuous 5 lines\n",
    "sed -n -e '1,~5p' re/numbers.txt # starting from 1, ending at the multiplier of 5\n",
    "sed -n -e '1,1p' re/numbers.txt  # starting from 1, ending at 1\n",
    "```\n",
    "2. Which of the following commands are valid? Why?\n",
    "```bash\n",
    "sed -n -e '0,1p' re/numbers.txt      # the first address is 0, second is also a number\n",
    "sed -n -e '0,/^z/p' re/numbers.txt   # the first address is 0, second is regex\n",
    "sed -n -e '/^o/,/^o/p' re/numbers.txt # the two address are two same regexes\n",
    "sed -n -e '/^t/,/^t/p' re/numbers.txt\n",
    "```\n",
    "3. Tell what will be printed for the following two commands, and tell the reasons.\n",
    "```bash\n",
    "sed -n -e '/^t/{p;q}' re/numbers.txt\n",
    "sed -n -e '/^t/p;q' re/numbers.txt\n",
    "```\n",
    "4. Write down the output\n",
    "```bash\n",
    "sed -n -e '8,~5!p' re/numbers.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Options (`sed`的常用选项的用法)\n",
    "\n",
    "`sed` can have different options:\n",
    "\n",
    "| Option | Description |\n",
    "| --- | --- |\n",
    "| -n, --quiet, --silent | Suppress automatic printing of pattern space. 不在每个工作周期结束后默认自动输出模式空间中的内容 |\n",
    "| -e SCRIPT, --expression=SCRIPT | Add the script to the commands to be executed. 一般用于多个不同的命令作用于不同的地址（条件） |\n",
    "| -f SCRIPT-FILE, --file=SCRIPT-FILE | Use the scripts in SCRIPT-FILE. 将命令置入一个文件中 |\n",
    "| -i[SUFFIX], --in-place[=SUFFIX] | Edit files in place (backup if SUFFIX is supplied). 直接修改文件，而不是修改缓存中的内容，但可以将源文件进行备份 |\n",
    "|  -l N, --line-length=N | Specify the desired line-wrap length for command `l`. |\n",
    "| -r, --regrexp-extended | Use ERE in the script. 使用扩展正则表达式 |\n",
    "| -s, --separate | Consider files as separate rather than as a single continuous stream. 针对多个文件的情况，对每个文件单独处理而不将其视为连续的流 |\n",
    "| -u, --unbuffered | Load minimal amounts of data from the input files and flush the output buffers more often. |\n",
    "| -z, --null-data | Separate lines by NUL characters. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two data buffers for `sed`(`sed`中的数据缓冲区)\n",
    "\n",
    "`sed` contains two data buffers:\n",
    "1. __pattern space (模式空间)__\n",
    "2. __hold space (保留空间)__\n",
    "\n",
    "#### Execution cycle (工作周期)\n",
    "1. Reads one line from the input stream, removes any trailing newline, and places it in the pattern space （读入一行，去除换行符，置入模式空间中）.\n",
    "2. Executes the command if the addresses (conditions) are verified. (如果地址或者条件符合，则执行后面的命令)\n",
    "3. When the end of the script is reached, unless the `-n` option is used, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed (除非用了-n选项，否则在所有命令结束后自动将模式空间中的内容输出到标准输出，并在后面自动加入换行符)\n",
    "4. goto 1\n",
    "\n",
    "#### Note (注意)\n",
    "1. Unless special commands (like `D') are used, the pattern space is deleted between two cycles. (除非使用了如`D`的特殊命令，两个工作周期中间模式空间中的内容被自动删除)\n",
    "2. The hold space, on the other hand, keeps its data between cycles (see commands `h', `H', `x', `g', `G' to move data between both buffers). （相反，保留空间中的内容在两个工作周期中被保留）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "### Commands for `sed` SCRIPTS\n",
    "\n",
    "`sed` has many different commands:\n",
    "\n",
    "##### 1. Commands for non-address\n",
    "\n",
    "| Command | Description | Example |\n",
    "| --- | --- | --- |\n",
    "| : label | **label for b and t commands （为命令t和b提供的标签位置）** | : loop |\n",
    "| #comment | The comment extends until the next new line （注释） | |\n",
    "\n",
    "##### 2. Commands for zero- or one-address\n",
    "\n",
    "| Command | Description | Example |\n",
    "| --- | --- | --- | \n",
    "| = | Print the line number of current address | `sed -n '1,~5=' test` |\n",
    "| i \\TEXT   | Insert TEXT before the current address 在当前的地址前面插入内容 | `sed '1~5i \\NEW' test` |\n",
    "| a \\TEXT   | Append TEXT after the current address 在当前的地址后面加入内容 | `sed '1~5a \\NEW' test` |\n",
    "| q [EXIT_CODE] | Quit immediately without any further processing; will print the content space if auto-print is enabled. 立即退出，但同时将模式空间中的内容自动输出，如果没有指定-n选项的话 | `sed -e '1~5p' -e '14q' test` |\n",
    "| Q [EXIT_CODE] | Quit immediately without any further processing 马上退出，且不执行任何自动输出，即便没有使用-n选项 | `sed -e '1~5p' -e '14Q' test` |\n",
    "| r FILENAME | Append the text read from FILENAME 将从文件FILENNAME中读取的内容附加到模式空间中 | `sed -n -e '1～5p;1~5r test2' test` |\n",
    "| R FILENAME | Append a line read from FILENAME 从文件FILENAME中读取一行，附加到模式空间中 | `sed -n -e '1～5p;1~5R test2' test` |\n",
    "\n",
    "##### 3. Commands for 1+-addresses\n",
    "\n",
    "| Command | Description | Example |\n",
    "| --- | --- | --- |\n",
    "| c \\TEXT   | Change the range with TEXT 改变这个范围地址内容  | `sed '1,5c \\NEW' test` |\n",
    "| d | Delete the pattern space and start a new cycle. 删除模式空间中的内容，并开始新的工作周期 | `sed -n -e '1,5d;p' test`  |\n",
    "| D | Delete the first line in the pattern space if the pattern space contains a newline. 删除模式空间中第一行的内容 | `sed -e '1,5{N;D}' test` |\n",
    "| n | Copy the next line to the pattern space. 将模式空间中的内容替换为下一行的内容 | `sed -e 'n;p' test` |\n",
    "| N | Append the next line to the pattern space. 将下一行的内容附加到模式空间后面 | `sed -n -e 'N;p' test` |\n",
    "| g | Copy the hold space to the pattern space. 将模式空间中的内容替换为保留空间中的内容 |                        |\n",
    "| G | Append the hold space to the pattern space. 将保留空间中的内容附加到模式空间后 | `sed -n -e '1!G;h;$p' test3` |\n",
    "| h | Copy the pattern space to the hold space. 将模式空间的内容替换为保留空间的内容 | `sed -n -e '1!G;h;$p' test3`  |\n",
    "| H | Append the pattern space to the hold space. 将模式空间内的内容附加到保留空间内容后面 |                       |\n",
    "| x | Exchange the pattern space and the hold space. 将模式空间内容与保留空间互换 |                    |\n",
    "| b LABEL | Branch to LABEL; if LABEL is omitted, branch to end of script. 跳转到由:LABEL指定的标签，用于循环执行；如不指定标签，跳转到脚本的最后 |        |\n",
    "| t LABEL | If  a  s///  has done a successful substitution since the last input line was  read  and  since the  last t or T command, then branch to LABEL; if LABEL is omitted, branch to end of script. 如果在上一行读入之后或者在执行上一次t或者T命令后，还有成功的替换发生，则跳转到LABEL |             |\n",
    "| T LABEL | If  NO  s///  has done a successful substitution since the last input line was  read  and  since the  last t or T command, then branch to LABEL; if LABEL is omitted, branch to end of script. 如果替换不成功，则跳转；如果么有指定，则跳转到脚本最后 |              |\n",
    "| s/REGEX/REPLACE/ | Substitute the pattern REGEX into REPLACE. 文本替换 |                    |\n",
    "| y/SRC/DEST/  | Transliterate the char in SRC to correponding char in DEST 相当于翻译 | `sed 'y/a-z/A-Z/' test` |\n",
    "| p |   Print the current pattern space. 输出模式空间中的内容  | `sed -n -e '1~5N;p' test` |\n",
    "| P |   Print the first line of the current space. 输出模式空间中的第一行内容 | `sed -n -e '1~5N;P' test` |\n",
    "| w FILENAME | Write the pattern space into FILENAME. 将模式空间中的内容写入文件 |  `sed -n -e '1~5w test3' test`  |\n",
    "| W FILENAME | Write the first line of the pattern space into the FILENAME. 将模式空间中的第一行内容写入文件 |  `sed -e '1~5N;W test3' test` |\n",
    "\n",
    "\n",
    "<font color=\"red\">NOTE</font>: We can add \"!\" before any command, which means ACTION on all the other lines with the exception of the given line.\n",
    "\n",
    "\n",
    "### <font color='blue'>Exercise</font>\n",
    "#### 1. What is the difference between `sed -n -e '1,5{p;N;D}' test` and `sed -n -e '1,5{p;N;d}' test`?\n",
    "#### 2. Here is a complete example of *commify* (Add commas as thousands separator)\n",
    "\n",
    "(1) This is the file `numbers.txt`:\n",
    "```1\n",
    "12\n",
    "123\n",
    "1234\n",
    "12345\n",
    "123456\n",
    "1234567\n",
    "12345678\n",
    "123456789\n",
    "1234567890\n",
    "1234567890.1234\n",
    "+1234567890.1234\n",
    "-1234567890.1234\n",
    "$1234567890.1234\n",
    "```\n",
    "\n",
    "(2) `sed` command:\n",
    "```bash\n",
    "sed -r ':a; s@(^|[^0-9.])([0-9]+)([0-9]{3})@\\1\\2,\\3@g;t a' numbers.txt\n",
    "```\n",
    "\n",
    "#### 3. Rewrite the file so that all the numbers in the file have 2 valid decimal digits. \n",
    "```\n",
    "name marks grade\n",
    "abc 50.5 CB\n",
    "def 45 CC\n",
    "ghhi 55 CA\n",
    "jkl 85 A\n",
    "mno 75.0 BA\n",
    "pqr 77 BA\n",
    "stu 89.50 A\n",
    "```\n",
    "\n",
    "#### 4. There is a file containing 1 to 100, with each on one line. Rewrite the file so that the numbers will be printed on one single line, separated by TAB.\n",
    "\n",
    "#### 5. Tell the difference of the following commands:\n",
    "```bash\n",
    "sed '1,+5!c\\NEWLINE' re/numbers.txt\n",
    "sed '1,~5!c\\NEWLINE' re/numbers.txt\n",
    "sed '1,~5!a\\NEW' re/numbers.txt\n",
    "sed '1,~5!i\\NEW' re/numbers.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zero\r\n",
      "one\r\n",
      "two\r\n",
      "three\r\n",
      "four\r\n",
      "NEW\r\n",
      "NEW\r\n",
      "NEW\r\n",
      "NEW\r\n",
      "NEW\r\n"
     ]
    }
   ],
   "source": [
    "sed '1,~5!c\\NEW' re/numbers.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nine eight seven six five four three two one zero\r\n"
     ]
    }
   ],
   "source": [
    "sed -n -e '1!G;$!h;${s/\\n/ /g;p}' re/numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the explantion for the above command:\n",
    "* For first line, `1h` will copy the pattern space to the hold space.\n",
    "* For the other lines, `G;h` will append the hold space to the pattern space, and then copy back to the hold space. `s/\\n/ /g` will replace all the newlines with a space.\n",
    "* For the final line, `p` will print out the pattern space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zero one two three four five six seven eight nine"
     ]
    }
   ],
   "source": [
    "sed -n -e '1h;1!H;${g;s/\\n/ /g;p}' re/numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the explanation:\n",
    "* For line 1, `1h;1!H` will copy the pattern space to the hold space.\n",
    "* For the other lines, `1!H` will append the pattern space to the hold space.\n",
    "* For the last line `$`, `g` will copy the hold space to the pattern space, and `s/\\n/ /g` will replace all the newlines with a space, and then print out the pattern space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-234,567,890.333\r\n"
     ]
    }
   ],
   "source": [
    "echo \"-234567890.333\" | sed -r ':a; s/\\B[0-9]{3}\\b/,&/; ta'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the explanation:\n",
    "* `:a`: create a label named `a`;\n",
    "* `s/\\B[0-9]{3}\\b/,&/`: match three consecutive digits from the margin to left, insert a comma before\n",
    "* If the last `s///` returns true, branch to the label `a`; branch to the end otherwise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note\n",
    "* If you do want to use the `extended` regular expression, apply `-r` option to `sed` command.\n",
    "* If you do want to use `Perl-style` regular expression (for example, __zero-width assertion__) in sed, please download [ssed](http://sed.sourceforge.net/grabbag/ssed/sed-3.62.tar.gz) and run `ssed` with option `-R`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Reference\n",
    "* [sed FAQ](http://sed.sourceforge.net/sedfaq3.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}