Linux commands cookbook

Linux commands cookbook

50 Most Frequently Used UNIX / Linux Commands
http://www.thegeekstuff.com/2010/11/50-linux-commands/

Write a Simple Script in Shell

Write a for loop.

$ for pod_name in $(kubectl get pods -l app=swag-worker-test -o jsonpath={..metadata.name}); do; kubectl delete pod $pod_name; done
pod "swag-worker-test-67fffcdd5-5hgf3" deleted
pod "swag-worker-test-67fffcdd5-h8jgg" deleted

# you could also write multiple lines
for pod_name in $(kubectl get pods -l app=swag-worker-test -o jsonpath={..metadata.name}); do
    kubectl delete pod $pod_name
done

Write a while true.

# using trap and wait will make your container react immediately to a stop request
$ bash -c "trap : TERM INT; sleep infinity & wait"
# or
$ bash -c "while true; do echo 'I am alive!'; sleep 3600; done"

or

#!/bin/bash
while true; do echo 'I am alive!'; sleep 3600; done

ref:
https://stackoverflow.com/questions/31870222/how-can-i-keep-container-running-on-kubernetes

Set Environment Variables from a File

$ export $(cat .env | grep -v ^# | xargs)

ref:
https://stackoverflow.com/questions/19331497/set-environment-variables-from-file

Switch to Another User

# the latter with "-" gets an environment as if another user just logged in
$ sudo su - ubuntu

Clear the Content of a File

$ echo -n > /var/log/nginx/error.log

ref:
https://unix.stackexchange.com/questions/88808/empty-the-contents-of-a-file

Pipeline stdout with xargs

$ find . -type f -name "*.yaml" | xargs echo
./configmap.yaml ./pvc.yaml ./service.yaml ./statefulset.yaml

$ find . -type f -name "*.yaml" | xargs -n 1 echo
./configmap.yaml
./pvc.yaml
./service.yaml
./statefulset.yaml

$ find . -type f -name "*.yaml" | xargs -n 2 echo
./configmap.yaml ./pvc.yaml
./service.yaml ./statefulset.yaml

$ redis-cli KEYS "*-*-*-*.reply.celery.pidbox" | xargs -n 100 redis-cli DEL

ref:
https://shapeshed.com/unix-xargs/

Set a Timeout for any Command

$ timeout -t 15 celery inspect ping -A app:celery -d [email protected]$(hostname)

Run a One-time Command at a Specific Time

at executes commands at a specified time. You may need to install the "at" package manually.

# install
$ sudo apt-get install at

# start
$ sudo atd

# list jobs
$ atq

$ at 00:05
at> echo "123" > /tmp/test.txt

$ at 00:00 18.1.2017
at> DPS_ENV=production /home/ubuntu/.virtualenvs/dps/bin/python /home/ubuntu/dps/manage.py send_emails > /tmp/send_emails.log

Press Control + D to exit at shell.

ref:
https://www.lifewire.com/linux-command-at-4091646
https://tecadmin.net/one-time-task-scheduling-using-at-commad-in-linux/

Pass Arguments to bash when Executing a script Fetched by curl

$ curl -L http://bit.ly/open-the-pod-bay-doors | bash -s -- --tags docker 

ref:
https://stackoverflow.com/a/25563308/885524
https://github.com/vinta/HAL-9000

Change a File's Modify Time

$ touch -m -d '1 Jan 2006 12:34' tmp
$ touch -m -d '1 Jan 2006 12:34' tmp/old_file.txt

ref:
https://www.unixtutorial.org/2008/11/how-to-update-atime-and-mtime-for-a-file-in-unix/

Delete Old Files under a Directory

$ find /data/storage/tmp/* -mtime +2 | xargs rm -Rf
$ find /data/storage/tmp/* -mtime +2 -exec rm {} \;

ref:
http://stackoverflow.com/questions/14731133/how-to-delete-all-files-older-than-3-days-when-argument-list-too-long

Append String to a File

# append
$ echo "the last line" >> README.md

# replace
$ echo "replace all" > README.md

Rename Sub-folders

$ for f in */migrations/; do mv -v "$f" "${f%???????????}south_migrations"; done

ref:
http://unix.stackexchange.com/questions/220176/rename-specific-level-of-sub-folders

List History Commands

$ export HISTTIMEFORMAT="%Y%m%d %T  "
$ history

Make Permission For A File Same As Another File

$ chmod --reference=file1 file2

Find Computer's Public IP

$ wget -qO- http://ipecho.net/plain ; echo

ref:
http://askubuntu.com/questions/95910/command-for-determining-my-public-ip

Compress and Uncompress Files

$ tar czf media-20151010.tar.gz media/
$ s3cmd put media-20151010.tar.gz s3://goeasytaxi/

# decompress
$ tar -xzf media.tar.gz

$ sudo apt-get install zip unzip
$ zip -j -r deps.zip spark_app/src/deps/
$ zip -r hourmasters.zip hourmasters/
$ scp -r -i ~/hourmasters.pem ssh [email protected]:/home/ubuntu/hourmasters.zip ~/Desktop/

# decompress
$ unzip stork.1.4.zip
$ gzip -d uwsgi.log.*.gz

$ gzip dps.201701171200.sql
$ gunzip dps.201701171200.sql.gz

Count File Lines

$ wc -l filename.txt

$ wc -l *.py

Find Files by Name or Content

$ find / -name virtualenvwrapper.sh

# 在現在的資料夾裡的全部檔案中搜尋字串,會自動搜尋子目錄
$ find . | xargs grep 'string'

$ find . -iname '*something*'

$ find *.html | xargs grep 'share_server.html'

# 搜尋當前目錄及子目錄下的含有 print() 字串的檔案
$ grep -rnw "." -e "print()"

$ grep -rnw "." -e "print()" --include=\*.py

ref:
https://stackoverflow.com/questions/16956810/how-do-i-find-all-files-containing-specific-text-on-linux

Find Directories by Name

$ find . -type d -name "*native*" -print

ref:
https://askubuntu.com/questions/153144/how-can-i-recursively-search-for-directory-names-with-a-particular-string-where

List Files by Date

$ ls -lrt

List Files Opened by a Process

$ lsof | grep uwsgi

$ lsof -i | grep LISTEN
$ lsof -i -n -P | grep LISTEN

Extract Content from a File

$ cat uwsgi.log | grep error

Display Contents of All Files in the Current Directory

$ grep . *
$ grep . *.html

List Used Ports

$ netstat -a

# TCP
$ netstat -ntlp | grep uwsgi

# UCP
$ netstat -nulp

Ping a Port

$ curl -I "10.148.70.84:9200"
$ curl -I "192.168.100.10:80"

$ sudo apt-get install nmap
$ nmap -p 4505 54.250.5.176
$ nmap -p 8000 10.10.100.70
$ nmap -p 5672 10.10.100.82

$ telnet 54.250.5.176 4505

ref:
http://stackoverflow.com/questions/12792856/what-ports-does-rabbitmq-use

Show Network Traffic and Bandwidth

$ tcpdump -i eth0

$ sudo apt-get install tcptrack
$ tcptrack -i eth0

ref:
http://askubuntu.com/questions/257263/how-to-display-network-traffic-in-terminal

List Running Processes

# show all processes
$ pstree -a

# also show pid
$ pstree -ap

# 列出前 10 個最佔記憶體的 processes
$ ps aux | sort -nk +4 | tail

# 列出 mysql 相關的 processes
$ ps aux | grep 'worker process'
$ ps aux | grep uwsgi

# 樹狀顯示
$ ps auxf

# 搜尋 process 並以樹狀結果顯示 parent process
$ ps f -opid,cmd -C python

Kill Processes

# 列出目前所有的正在記憶體當中的程序
$ ps aux

# 匹配字串
$ ps aux | grep "mongo"

# 幹掉它
$ kill PID

# kill all processes matching a name
$ sudo pkill -f runserver

Store User's Input as a Variable

$ read YOUR_VARIABLE_NAME

$ read name
# you type: vinta

$ echo $name
vinta

ref:
https://canred.blogspot.tw/2013/03/read.html

Show Terminal Size

$ stty size
$ echo $LINES && echo $COLUMNS
59 273 

ref:
https://stackoverflow.com/questions/263890/how-do-i-find-the-width-height-of-a-terminal-window

MySQL system error codes

MySQL system error codes

Print all OS error codes and MySQL error codes using the perror command.

$ for i in {1..190..1}; do perror "$i"; done

OS error code   1:  Operation not permitted
OS error code   2:  No such file or directory
OS error code   3:  No such process
OS error code   4:  Interrupted system call
OS error code   5:  Input/output error
OS error code   6:  No such device or address
OS error code   7:  Argument list too long
OS error code   8:  Exec format error
OS error code   9:  Bad file descriptor
OS error code  10:  No child processes
OS error code  11:  Resource temporarily unavailable
OS error code  12:  Cannot allocate memory
OS error code  13:  Permission denied
OS error code  14:  Bad address
OS error code  15:  Block device required
OS error code  16:  Device or resource busy
OS error code  17:  File exists
OS error code  18:  Invalid cross-device link
OS error code  19:  No such device
OS error code  20:  Not a directory
OS error code  21:  Is a directory
OS error code  22:  Invalid argument
OS error code  23:  Too many open files in system
OS error code  24:  Too many open files
OS error code  25:  Inappropriate ioctl for device
OS error code  26:  Text file busy
OS error code  27:  File too large
OS error code  28:  No space left on device
OS error code  30:  Read-only file system
OS error code  31:  Too many links
OS error code  32:  Broken pipe
OS error code  33:  Numerical argument out of domain
OS error code  34:  Numerical result out of range
OS error code  35:  Resource deadlock avoided
OS error code  36:  File name too long
OS error code  37:  No locks available
OS error code  38:  Function not implemented
OS error code  39:  Directory not empty
OS error code  40:  Too many levels of symbolic links
OS error code  42:  No message of desired type
OS error code  43:  Identifier removed
OS error code  44:  Channel number out of range
OS error code  45:  Level 2 not synchronized
OS error code  46:  Level 3 halted
OS error code  47:  Level 3 reset
OS error code  48:  Link number out of range
OS error code  49:  Protocol driver not attached
OS error code  50:  No CSI structure available
OS error code  51:  Level 2 halted
OS error code  52:  Invalid exchange
OS error code  53:  Invalid request descriptor
OS error code  54:  Exchange full
OS error code  55:  No anode
OS error code  56:  Invalid request code
OS error code  57:  Invalid slot
OS error code  59:  Bad font file format
OS error code  60:  Device not a stream
OS error code  61:  No data available
OS error code  62:  Timer expired
OS error code  63:  Out of streams resources
OS error code  64:  Machine is not on the network
OS error code  65:  Package not installed
OS error code  66:  Object is remote
OS error code  67:  Link has been severed
OS error code  68:  Advertise error
OS error code  69:  Srmount error
OS error code  70:  Communication error on send
OS error code  71:  Protocol error
OS error code  72:  Multihop attempted
OS error code  73:  RFS specific error
OS error code  74:  Bad message
OS error code  75:  Value too large for defined data type
OS error code  76:  Name not unique on network
OS error code  77:  File descriptor in bad state
OS error code  78:  Remote address changed
OS error code  79:  Can not access a needed shared library
OS error code  80:  Accessing a corrupted shared library
OS error code  81:  .lib section in a.out corrupted
OS error code  82:  Attempting to link in too many shared libraries
OS error code  83:  Cannot exec a shared library directly
OS error code  84:  Invalid or incomplete multibyte or wide character
OS error code  85:  Interrupted system call should be restarted
OS error code  86:  Streams pipe error
OS error code  87:  Too many users
OS error code  88:  Socket operation on non-socket
OS error code  89:  Destination address required
OS error code  90:  Message too long
OS error code  91:  Protocol wrong type for socket
OS error code  92:  Protocol not available
OS error code  93:  Protocol not supported
OS error code  94:  Socket type not supported
OS error code  95:  Operation not supported
OS error code  96:  Protocol family not supported
OS error code  97:  Address family not supported by protocol
OS error code  98:  Address already in use
OS error code  99:  Cannot assign requested address
OS error code 100:  Network is down
OS error code 101:  Network is unreachable
OS error code 102:  Network dropped connection on reset
OS error code 103:  Software caused connection abort
OS error code 104:  Connection reset by peer
OS error code 105:  No buffer space available
OS error code 106:  Transport endpoint is already connected
OS error code 107:  Transport endpoint is not connected
OS error code 108:  Cannot send after transport endpoint shutdown
OS error code 109:  Too many references: cannot splice
OS error code 110:  Connection timed out
OS error code 111:  Connection refused
OS error code 112:  Host is down
OS error code 113:  No route to host
OS error code 114:  Operation already in progress
OS error code 115:  Operation now in progress
OS error code 116:  Stale NFS file handle
OS error code 117:  Structure needs cleaning
OS error code 118:  Not a XENIX named type file
OS error code 119:  No XENIX semaphores available
OS error code 120:  Is a named type file
OS error code 121:  Remote I/O error
OS error code 122:  Disk quota exceeded
OS error code 123:  No medium found
OS error code 124:  Wrong medium type
OS error code 125:  Operation canceled
OS error code 126:  Required key not available
OS error code 127:  Key has expired
OS error code 128:  Key has been revoked
OS error code 129:  Key was rejected by service
OS error code 130:  Owner died
OS error code 131:  State not recoverable
OS error code 132:  Operation not possible due to RF-kill
OS error code 133:  Memory page has hardware error
MySQL error code 120: Did not find key on read or update
MySQL error code 121: Duplicate key on write or update
MySQL error code 122: Internal (unspecified) error in handler
MySQL error code 123: Someone has changed the row since it was read (while the table was locked to prevent it)
MySQL error code 124: Wrong index given to function
MySQL error code 125: Undefined handler error 125
MySQL error code 126: Index file is crashed
MySQL error code 127: Record file is crashed
MySQL error code 128: Out of memory in engine
MySQL error code 129: Undefined handler error 129
MySQL error code 130: Incorrect file format
MySQL error code 131: Command not supported by database
MySQL error code 132: Old database file
MySQL error code 126: Index file is crashed
MySQL error code 127: Record-file is crashed
MySQL error code 128: Out of memory
MySQL error code 130: Incorrect file format
MySQL error code 131: Command not supported by database
MySQL error code 132: Old database file
MySQL error code 133: No record read before update
MySQL error code 134: Record was already deleted (or record file crashed)
MySQL error code 135: No more room in record file
MySQL error code 136: No more room in index file
MySQL error code 137: No more records (read after end of file)
MySQL error code 138: Unsupported extension used for table
MySQL error code 139: Too big row
MySQL error code 140: Wrong create options
MySQL error code 141: Duplicate unique key or constraint on write or update
MySQL error code 142: Unknown character set used in table
MySQL error code 143: Conflicting table definitions in sub-tables of MERGE table
MySQL error code 144: Table is crashed and last repair failed
MySQL error code 145: Table was marked as crashed and should be repaired
MySQL error code 146: Lock timed out; Retry transaction
MySQL error code 147: Lock table is full;  Restart program with a larger locktable
MySQL error code 148: Updates are not allowed under a read only transactions
MySQL error code 149: Lock deadlock; Retry transaction
MySQL error code 150: Foreign key constraint is incorrectly formed
MySQL error code 151: Cannot add a child row
MySQL error code 152: Cannot delete a parent row
MySQL error code 153: No savepoint with that name
MySQL error code 154: Non unique key block size
MySQL error code 155: The table does not exist in engine
MySQL error code 156: The table already existed in storage engine
MySQL error code 157: Could not connect to storage engine
MySQL error code 158: Unexpected null pointer found when using spatial index
MySQL error code 159: The table changed in storage engine
MySQL error code 160: There is no partition in table for the given value
MySQL error code 161: Row-based binlogging of row failed
MySQL error code 162: Index needed in foreign key constraint
MySQL error code 163: Upholding foreign key constraints would lead to a duplicate key error in some other table
MySQL error code 164: Table needs to be upgraded before it can be used
MySQL error code 165: Table is read only
MySQL error code 166: Failed to get next auto increment value
MySQL error code 167: Failed to set row auto increment value
MySQL error code 168: Unknown (generic) error from engine
MySQL error code 169: Record is the same
MySQL error code 170: It is not possible to log this statement
MySQL error code 171: The event was corrupt, leading to illegal data being read
MySQL error code 172: The table is of a new format not supported by this version
MySQL error code 173: The event could not be processed no other hanlder error happened
MySQL error code 174: Got a fatal error during initialzaction of handler
MySQL error code 175: File to short; Expected more data in file
MySQL error code 176: Read page with wrong checksum
MySQL error code 177: Too many active concurrent transactions
MySQL error code 178: Record not matching the given partition set
MySQL error code 179: Index column length exceeds limit
MySQL error code 180: Index corrupted
MySQL error code 181: Undo record too big
MySQL error code 182: Invalid InnoDB FTS Doc ID
MySQL error code 183: Table is being used in foreign key check
MySQL error code 184: Tablespace already exists
MySQL error code 185: Too many columns
MySQL error code 186: Row in wrong partition
MySQL error code 187: InnoDB is in read only mode
MySQL error code 188: FTS query exceeds result cache memory limit
MySQL error code 189: Temporary file write failure
MySQL error code 190: Operation not allowed when innodb_forced_recovery > 0
MySQL error code 191: Too many words in a FTS phrase or proximity search
MySQL error code 192: Foreign key cascade delete/update exceeds max depth
MySQL error code 193: Required Create option missing
MySQL error code 194: Out of memory in storage engine
MySQL error code 195: Table corrupted
MySQL error code 196: Query interrupted
MySQL error code 197: Tablespace cannot be accessed
MySQL error code 198: Tablespace is not empty
MySQL error code 199: Incorrect file name
MySQL error code 200: Operation is not allowed
MySQL error code 201: Compute generate value failed

ref:
http://man7.org/linux/man-pages/man3/perror.3.html

Redirect Console Output: stdout and stderr

Redirect Console Output: stdout and stderr

File descriptors:

  • 0: stdin
  • 1: stdout
  • 2: stderr
# redirect stdout to a file `test.log`
$ strace uptime > test.log
$ strace uptime 1> test.log

# redirect stderr to a file `test.log`
$ strace uptime 2> test.log

# redirect stderr to stdout
# &1 references the value of the file descriptor 1 (stdout)
$ sh /home/vinta/do_shit.sh > /tmp/do_shit.log 2>&1

# `2> 1` actually means that you redirect stderr to a file named `1`
$ sh /home/vinta/do_shit.sh > /tmp/do_shit.log 2> 1
$ cat 1

# redirect both stderr and stdout to file `test.log`
$ strace uptime &> test.log

# run a command in backgroud
$ long-running-command &

ref:
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html
https://www.cyberciti.biz/faq/redirecting-stderr-to-stdout/
https://www.brianstorti.com/understanding-shell-script-idiom-redirect/