====== Shell Command ======
refer:
{{:linux:linux_shell_scripting_with_bash_2004.pdf|}}\\
Advanced Bash-Scripting Guide:http://tldp.org/LDP/abs/html/\\
Bash script: http://www.linuxconfig.org/Bash_scripting_Tutorial
===== Basic Commands =====
==== man ====
man -a read
=> output:
POSIX Programmers Manual READ(P)
Linux Programmers Manual READ(2)
man -a stat
=> output:
User Commands STAT(1)
Linux Programmer?s Manual STAT(2)
==== uname ====
uname -a
=>Linux sgwp 2.6.17 #3 PREEMPT Fri Apr 4 00:18:16 JST 2008 armv5tel unknown unknown GNU/Linux
==== date ====
* set date
date +%Y%m%d -s "20120120"
date +%T -s "16:00:00"
* display date
date +%Y%m%d#=> display date follow format %Y%m%d
date +%s#=> display UNIX time
* convert unixtime
date -d @1098181096
* sync time:
ntpdate pool.ntp.org
===== file and directory commands =====
==== dirname ====
get directory name from file
==== ls ====
* list derectorys
ls -l | grep '^d' | awk '{ print $9 }'
* list files
ls -1
==== find files in directory====
find . -name filename
==== ln ====
ln [OPTION]... TARGET [LINK_NAME]
Create a link to the specified TARGET with optional LINK_NAME. If LINK_NAME is omitted, a link with the same basename as the TAR-GET is created in the current directory
ssh $i "rm -rf AM;mkdir AM;ln -s /home/ztgame/$1/release/ /home/ztgame/AM"
( -s, --symbolic make symbolic links instead of hard links)
==== du ====
du -sh /db/mysql/
df -ah
df -h
==== lock/unlock file and directory ====
cd /home/admin/conf
lsattr -a
-------------e- ./web
----i--------e- ./.
-------------e- ./..
-------------e- ./dns
-------------e- ./mail
=> directory /home/admin has attribute **i** => can't remove or create new files in this directory with root
And remove this attribute
chattr -i /home/admin/conf/
===== Shell Security =====
==== chmod ====
chmod a+w
=>permission write for all user
===== Job and process, threads commands =====
==== whereis ====
whereis java
=> specific path of process java
whereis iptables
=> iptables: /sbin/iptables /lib/iptables /usr/share/man/man8/iptables.8.gz
==== kill ====
kill `pidof zabbix_agentd`
killall zabbix_agentd
==== netstat ====
netstat -tlnp
(-t:tcp, -l:listening, -n: numberic, -p: programs)
config:
/etc/services
=> presently the policy of IANA to assign a single well-known
The Registered Ports are those from 1024 through 49151
The Dynamic and/or Private Ports are those from 49152 through 65535
==== ps ====
* list all processes with UID,PID,PPID,CMD
ps -eaf | head
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 01:18 ? 00:00:00 /sbin/init
root 2 0 0 01:18 ? 00:00:00 [kthreadd]
root 3 2 0 01:18 ? 00:00:00 [migration/0]
root 4 2 0 01:18 ? 00:00:00 [ksoftirqd/0]
* list all processes with USER,PID,%CPU,%MEM,VSZ,RSS,COMMAND
ps -aux | head
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19232 360 ? Ss 01:18 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 01:18 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 01:18 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S 01:18 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S 01:18 0:00 [migration/0]
root 6 0.0 0.0 0 0 ? S 01:18 0:00 [watchdog/0]
root 7 0.0 0.0 0 0 ? S 01:18 0:08 [events/0]
root 8 0.0 0.0 0 0 ? S 01:18 0:00 [cgroup]
root 9 0.0 0.0 0 0 ? S 01:18 0:00 [khelper]
* Get top 10 processes which sorted by CPU
ps -aux | sort -nk +3 | tail -n 10
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1071 0.1 7.6 710560 38184 ? Sl 01:18 0:57 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
apache 5614 0.3 5.8 275252 29400 ? S 16:26 0:00 /usr/sbin/httpd
apache 5537 0.5 10.6 300316 53632 ? S 16:16 0:03 /usr/sbin/httpd
apache 5553 0.5 11.2 301832 56360 ? S 16:20 0:02 /usr/sbin/httpd
apache 5552 0.6 8.9 290280 44920 ? S 16:20 0:02 /usr/sbin/httpd
apache 5578 0.8 11.2 301840 56296 ? S 16:24 0:01 /usr/sbin/httpd
* Get top 10 processes which sorted by Memory Usage
ps -aux | sort -nk +4 | tail -n 10
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
postfix 5249 0.0 0.6 81364 3336 ? S 15:51 0:00 pickup -l -t fifo -u
root 5250 0.0 0.8 100368 4452 ? Ss 15:52 0:00 sshd: root@pts/0
root 5287 0.0 0.8 100368 4476 ? Ss 15:52 0:00 sshd: root@pts/1
apache 5648 0.0 1.2 255040 6488 ? S 16:29 0:00 /usr/sbin/httpd
root 5141 0.0 1.9 254904 9920 ? Ss 15:38 0:00 /usr/sbin/httpd
apache 5640 0.1 3.1 262836 15744 ? S 16:28 0:00 /usr/sbin/httpd
apache 5647 1.5 7.0 280756 35176 ? S 16:29 0:00 /usr/sbin/httpd
mysql 1071 0.1 7.6 710560 38184 ? Sl 01:18 0:57 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
apache 5614 0.3 8.0 285904 40192 ? S 16:26 0:00 /usr/sbin/httpd
apache 5553 0.5 11.2 301832 56404 ? S 16:20 0:02 /usr/sbin/httpd
==== Check memory information ====
Memory space is divided into memory **used by processes, disk cache, free memory and memory used by kernel**
Some basic parameters in memory information:
* MemTotal — Total amount of physical RAM, in kilobytes.
* **MemFree** — The amount of physical RAM, in kilobytes, left unused by the system.
* **Buffers** — The amount of physical RAM, in kilobytes, used for **file buffers**.
* **Cached** — The amount of physical RAM, in kilobytes, used as cache memory(**Disk Cache**).
* SwapCached — The amount of swap, in kilobytes, used as cache memory.
* **Active** — The total amount of **buffer or page cache memory**, in kilobytes, that is in active use. This is memory that **has been recently used** and is usually not reclaimed for other purposes.
* **Inactive** — The total amount of **buffer or page cache memory**, in kilobytes, that are free and available. This is memory that **has not been recently used** and can be reclaimed for other purposes.
* Active(anon): active memory that is not file backed. This will typically be the higher chunk of active memory on a app server machine which does not have a db
* Active (file): active memory that is file backed. this will typically be the higher chunk of active memory on a data store machine that reads / writes from disk
* HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
* LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
* SwapTotal — The total amount of swap available, in kilobytes.
* SwapFree — The total amount of swap free, in kilobytes.
=== in /proc/meminfo ===
cat /proc/meminfo
output:
MemTotal: 1020400 kB
MemFree: 65064 kB
Buffers: 180148 kB
Cached: 166648 kB
SwapCached: 76 kB
Active: 370004 kB
Inactive: 423848 kB
Active(anon): 183084 kB
Inactive(anon): 264156 kB
Active(file): 186920 kB
Inactive(file): 159692 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 1048568 kB
SwapFree: 1048440 kB
Dirty: 408 kB
Writeback: 0 kB
AnonPages: 447048 kB
Mapped: 19032 kB
Shmem: 184 kB
Slab: 140256 kB
SReclaimable: 118196 kB
SUnreclaim: 22060 kB
KernelStack: 992 kB
PageTables: 5092 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1558768 kB
Committed_AS: 604456 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 7320 kB
VmallocChunk: 34359727504 kB
HardwareCorrupted: 0 kB
AnonHugePages: 143360 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6136 kB
DirectMap2M: 1042432 kB
DirectMap1G: 0 kB
We have the expression:
Active(370004) = Active(anon: 183084) + Active(file:186920)
Inactive(423848) = Inactive(anon:264156) + Inactive(file:159692)
Memtotal = MemFree + Buffers + Cached + Active + Mapped + Shmem + Slab + PageTables + VmallocUsed
=== In top command ===
top
output:
Cpu(s): 4.0%us, 0.5%sy, 0.0%ni, 94.4%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1020400k total, 955336k used, 65064k free, 180148k buffers
Swap: 1048568k total, 128k used, 1048440k free, 166648k cached
The expression:
1020400k total = 955336k used + 65064k free
955336k used = 180148k buffers + 166648k cached + Other Mem(Kernel and App Server Use)
=== Check memory of process ===
cat /proc/1704/status
output:
Name: php-fpm
State: S (sleeping)
Tgid: 1704
Pid: 1704
PPid: 1703
TracerPid: 0
Uid: 99 99 99 99
Gid: 99 99 99 99
Utrace: 0
FDSize: 64
Groups: 99
VmPeak: 329824 kB
VmSize: 329504 kB
VmLck: 0 kB
VmHWM: 127368 kB
VmRSS: 124412 kB
VmData: 118996 kB
VmStk: 88 kB
VmExe: 9920 kB
VmLib: 12832 kB
VmPTE: 476 kB
VmSwap: 0 kB
Threads: 1
SigQ: 0/7826
Explain the parameters:
* FDSize: Number of file descriptor slots currently allocated.
* Groups: Supplementary group list.
* VmPeak: Peak virtual memory size.
* VmSize: Virtual memory size.
* VmLck: Locked memory size (see mlock(3)).
* VmHWM: Peak resident set size ("high water mark").
* VmRSS: Resident set size.
* VmData, VmStk, VmExe: Size of data, stack, and text segments.
* VmLib: Shared library code size.
* VmPTE: Page table entries size (since Linux 2.6.10).
* Threads: Number of threads in process containing this thread.
=== pmap ===
report memory map of a process
* pmap -x PID
pmap -x 6128
Address Kbytes RSS Dirty Mode Mapping
00007f1b20662000 76 0 0 r-x-- zip.so
00007f1b20675000 2044 0 0 ----- zip.so
00007f1b20874000 8 8 8 rw--- zip.so
00007f1b20876000 12 0 0 r-x-- libgpg-error.so.0.5.0
..................
* sort memory map base on Kbytes
pmap -x 6128 | sort -nk +2 | tail -n 10
00007f1b2bd85000 2048 0 0 ----- mod_log_config.so
00007f1b2c99e000 2048 0 0 ----- mod_authz_user.so
00007f1b2d1a9000 2048 0 0 ----- mod_authn_anon.so
00007f1b2d7b7000 2048 0 0 ----- mod_auth_digest.so
00007f1b2e039000 2048 0 0 ----- libdl-2.12.so
00007f1b2e5e6000 2048 0 0 ----- libpthread-2.12.so
00007f1b2efbb000 2048 0 0 ----- libcrypt-2.12.so
00007f1b31cf7000 2380 2352 2352 rw--- [ anon ]
00007f1b27b18000 3268 1664 0 r-x-- libphp5.so
00007f1b31f4a000 123012 122464 122464 rw--- [ anon ]
==== Get all threads of process ====
* Get all threads in linux
ps -efL
* Get all threads of MySQL(base on config **thread_cache_size**):
ps -efL | grep mysql
===== List Open Files for Process =====
==== get open files base on OS information ====
* Step1: get pid of process:
pgrep -f server.gamedangian.jar
=> output: 31497
* Step2: get list of open files
ls -l /proc/31497/fd
==== lsof ====
* install lsof:
yum install lsof
* list of file was opened by process:
* Step1: get pid
pgrep -f server.gamedangian.jar
=> output: 31497
* Step2: get files was opened by process:
lsof -p 31497
==== Get total open files in linux ====
run below command:
cat /proc/sys/fs/file-nr
or run with lsof:
lsof | wc -l
There are two reasons **lsof | wc -l** doesn't count file descriptors. One is that it lists things that aren't open files, such as loaded dynamically linked libraries and current working directories; you need to filter them out. Another is that lsof takes some time to run, so can miss files that are opened or closed while it's running; therefore the number of listed open files is approximate
===== Text Processing =====
==== cat ====
cat > abc.txt
content
#ctrl+D
==== wc ====
# wc -l
==== cut ====
* format: cut -cchars file
* example1:
# who
root console Feb 24 08:54
steve tty02 Feb 24 12:55
# who | cut -c1-8 Extract the first 8 characters
root
steve
* example2:
# cut -d' ' -f 11
# cut -d' ' -f 11-
==== paste ====
Example:
# cat names
Tony
Emanuel
# cat numbers
(307) 555-5356
(212) 555-3456
# paste names numbers
Tony (307) 555-5356
Emanuel (212) 555-3456
==== grep ====
grep -v "#"
=> get content none sign "#"
==== sed ====
Basic syntax:
sed -i 's/original/new/g' file.txt
Explain options:
* -i = --in-place (i.e. save back to the original file)
The command string:
* s = the substitute command
* original = a regular expression describing the word to replace (or just the word itself)
* new = the text to replace it with
* g = global (i.e. replace all and not just the first occurrence)
Below are some basic examples for searching and replaccing string
cat intro
output:
The Unix operating system. Unix system
* Substitute Unix with UNIX:
sed 's/Unix/UNIX/g' intro
=> output:
The UNIX operating system. Unix system
* Substitute Unix with UNIX in file intro:
sed --in-place 's/Unix/UNIX/g' intro
* Substitute Unix with UNIX
sed 's/Unix/UNIX/g' intro
* example with -n option
sed -n '1,2p' intro=> Just print the first 2 lines
* Just print lines containing UNIX:
sed -n '/UNIX/p' intro
* example Deleting Lines
sed '1,2d' intro
=> Delete lines 1 and 2
* Delete all lines containing UNIX:
sed '/UNIX/d' intro
* Delete all characters before string
sed 's/.*No Warranty/No Warranty/g'
==== awk ====
# cat server
1-Ngoa Ho 10.30.31.10
3-Tang Long 10.30.31.38
* example1
cat server |awk '{print $1}'
=> output:
1-Ngoa
3-Tang
cat server |awk '{print $2}'
=> output:
Ho
Long
cat server |awk '{print $3}'
=> output:
10.30.31.10
10.30.31.38
* example2
cat server |awk -F'-' '{print $1}'
output:
1
3
===== Custom Commands =====
==== checksum ====
md5sum, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum
* create checksum for file
md5sum [OPTION]... [FILE]...
* create checksum for text:
echo -n 'helloworld' | md5sum -
==== curl ====
=== basic options ===
-A/--user-agent User-Agent to send to server (H)
-b/--cookie Cookie string or file to read cookies from (H)
-c/--cookie-jar Write cookies to this file after operation (H)
--create-dirs Create necessary local directory hierarchy
--crlf Convert LF to CRLF in upload
--crlfile Get a CRL list in PEM format from the given file
-d/--data HTTP POST data (H)
--data-ascii HTTP POST ASCII data (H)
--data-binary HTTP POST binary data (H)
--data-urlencode HTTP POST data url encoded (H)
--delegation STRING GSS-API delegation permission
--digest Use HTTP Digest Authentication (H)
--disable-eprt Inhibit using EPRT or LPRT (F)
--disable-epsv Inhibit using EPSV (F)
-F/--form Specify HTTP multipart POST data (H)
--form-string Specify HTTP multipart POST data (H)
--ftp-account Account data to send when requested by server (F)
--ftp-alternative-to-user String to replace "USER [name]" (F)
--ftp-create-dirs Create the remote dirs if not present (F)
--ftp-method [multicwd/nocwd/singlecwd] Control CWD usage (F)
--ftp-pasv Use PASV/EPSV instead of PORT (F)
-I/--head Show document info only
-k/--insecure Allow connections to SSL sites without certs (H)
--interface Specify network interface/address to use
Understand about options:
-d/--data HTTP POST data (H)
--data-ascii HTTP POST ASCII data (H)
--data-binary HTTP POST binary data (H)
--data-urlencode HTTP POST data url encoded (H)
--delegation STRING GSS-API delegation permission
--digest Use HTTP Digest Authentication (H)
--disable-eprt Inhibit using EPRT or LPRT (F)
--disable-epsv Inhibit using EPSV (F)
* options -d => default option: --data:
* curl -d
* Or: curl --data
* And other options for postdata:
--data-ascii HTTP POST ASCII data (H)
--data-binary HTTP POST binary data (H)
--data-urlencode HTTP POST data url encoded (H)
--delegation STRING GSS-API delegation permission
--digest Use HTTP Digest Authentication (H)
--disable-eprt Inhibit using EPRT or LPRT (F)
--disable-epsv Inhibit using EPSV (F)
=== examples for curl ===
==== tcpdump ====
* tcpdump command format
tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
[ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
[ -W filecount ] [ -y datalinktype ] [ -Z user ]
[ expression ]
[expression] => proto(ip,ip6,tcp,udp,arp...),host,src host,dst host,port,src port,dst port
* tcpdump examples
tcpdump -i eth0 port 80
tcpdump -i eth0 dst port 80
tcpdump -i lo port 10000 => listen among ports in localhost
tcpdump -i eth0 -w file.dmp port 80
tcpdump -i eth0 -r file.dmp port 80
tcpdump src 10.199.3.129
tcpdump dst 10.30.31.58
tcpdump host 10.30.31.58
tcpdump -v icmp
tcpdump -v "icmp or arp"
==== change timezone ====
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Bangkok /etc/localtime
==== iptables ====
iptables -F
=> Stopping firewall and allowing everyone
iptables-save
==== mount ====
== mount iso ==
* mount iso to directory
mount -t iso9660 -o loop /root/debian-508-i386-DVD-1.iso /mnt/cdrom/
* mount iso to cdrom hardware
mount -t iso9660 -o loop /root/debian-508-i386-DVD-1.iso /media/cdrom
== mount to RAM Disk ==
* step1: create script mountram.sh
cat > /root/scripts/mountram.sh
#!/bin/sh
# Size of your RAM disk (tmpfs)
RAM_DISK_SIZE=1G
MOUNT_POINT=/home/mole/data
SOURCE_DIR=/home/mole/source_real
# Make sure the mount point exists
mkdir -p $MOUNT_POINT;
# Now create the tmp filesystem on mount point
mount -o size=$RAM_DISK_SIZE -t tmpfs tmpfs $MOUNT_POINT
# Copy the source files to the mounted tmpfs filesystem
cp -r $SOURCE_DIR/* $MOUNT_POINT
* step2: create ram disk
echo "none /dev/shm tmpfs defaults 0 0" >> /etc/fstab
* step3: config to autorun script mountram.sh when server start
echo "/root/scripts/mountram.sh" >> /etc/rc.local
==== ldd: print shared library dependencies of binary file ====
ldd /usr/sbin/vsftpd
linux-gate.so.1 => (0x00b1f000)
libssl.so.6 => /lib/libssl.so.6 (0x008e0000)
libwrap.so.0 => /lib/libwrap.so.0 (0x00944000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00991000)
libpam.so.0 => /lib/libpam.so.0 (0x00110000)
libcap.so.1 => /lib/libcap.so.1 (0x00fc0000)
libdl.so.2 => /lib/libdl.so.2 (0x00d9b000)
libc.so.6 => /lib/libc.so.6 (0x003af000)
==== objdump: Check target binary ====
objdump -f mole.vn.register
==== rsync ====
( rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated.
The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.)
* rsync in local
rsync -avz --exclude=".svn" "/home/anhvc/web/9thien.com/" /web/9thien.com/
* rsync to remote host
rsync -avz --exclude=".svn" -e "ssh -p2222" "/home/anhvc/web" 123.30.245.164:~
==== rsync on windows ====
- Step1: Download and install rsync at [[http://www.rsync.net/resources/binaries/cwRsync_3.1.0_Installer.zip|here]]. cwRsync will installed to c:\Program Files (x86)\cwRsync\bin\
- Step2: Add path "c:\Program Files (x86)\cwRsync\bin\" to PATH envionment variable on Windows
- Step3: Check ssh with private-key:
ssh -p1362 -i E:\backup\GoogleDrive\keys\id_rsa root@128.199.236.122
- Step4: Run rsync via ssh to sync data:
rsync -avz --exclude=".svn" -e "ssh -p1362 -i E:\backup\GoogleDrive\keys\id_rsa" "/cygdrive/d/web/www.babies.vn/" root@128.199.236.122:/data/www/www.babies.vn/
- Step5: Chown to apache.apache for /data/www/www.babies.vn/
ssh -p1362 -i E:\backup\GoogleDrive\keys\id_rsa root@128.199.236.122 "chown -R apache.apache /data/www/www.babies.vn/"
- Step6: Fix error rsync
rsync: recv_generator: mkdir failed: Permission denied (13)
Add option for rsync on windows:
--no-p --no-g --chmod=ugo=rwX
==== osgi ====
b
=>list bundles
start ID
=> start bundles
sh
=>shutdown osgi
==== nm: list symbols from object files ====
nm -C -D libhelloworldjni.so
=>
0000054f T JNI_OnLoad
00000568 T JNI_OnUnload
0000050c T Java_HelloWorldJNI_sayGoodbye
00000520 T Java_HelloWorldJNI_sayHelloWorld__
00000534 T Java_HelloWorldJNI_sayHelloWorld(int0_t)
===== Cygwin =====
==== Install Cygwin ====
- Step1: download cygwin setup from https://cygwin.com/install.html
- Step2: Install Cygwin with **default mode** to setup basic packages, and we will install new packages If application must use them
==== Using Cygwin ====
* Go to directory d:\tools on windows:
cd /cygdrive/d/tools