Shell Command

refer: linux_shell_scripting_with_bash_2004.pdf
Advanced Bash-Scripting Guide:http://tldp.org/LDP/abs/html/
Bash script: http://www.linuxconfig.org/Bash_scripting_Tutorial

man -a read
=> output:
POSIX Programmers Manual                   READ(P)
Linux Programmers Manual                   READ(2)

man -a stat
=> output:
User Commands                                STAT(1)
Linux Programmer?s Manual                   STAT(2)

uname -a

⇒Linux sgwp 2.6.17 #3 PREEMPT Fri Apr 4 00:18:16 JST 2008 armv5tel unknown unknown GNU/Linux

• set date

  date +%Y%m%d -s "20120120"  
  date +%T -s "16:00:00"
   

• display date

  date +%Y%m%d#=> display date follow format %Y%m%d
  date +%s#=> display UNIX time

• convert unixtime

    date -d @1098181096

• sync time:

  ntpdate  pool.ntp.org

get directory name from file

• list derectorys

  ls -l | grep '^d' | awk '{ print $9 }'

• list files

  ls -1

find . -name filename

ln [OPTION]... TARGET [LINK_NAME]

Create a link to the specified TARGET with optional LINK_NAME. If LINK_NAME is omitted, a link with the same basename as the TAR-GET is created in the current directory

ssh $i "rm -rf AM;mkdir AM;ln -s /home/ztgame/$1/release/ /home/ztgame/AM"

( -s, –symbolic make symbolic links instead of hard links)

du -sh /db/mysql/ 
df -ah
df -h

cd /home/admin/conf
lsattr -a
-------------e- ./web
----i--------e- ./.
-------------e- ./..
-------------e- ./dns
-------------e- ./mail

⇒ directory /home/admin has attribute i ⇒ can't remove or create new files in this directory with root And remove this attribute

chattr -i /home/admin/conf/

chmod a+w

⇒permission write for all user

whereis java

⇒ specific path of process java

whereis iptables

⇒ iptables: /sbin/iptables /lib/iptables /usr/share/man/man8/iptables.8.gz

kill `pidof zabbix_agentd`
killall zabbix_agentd

netstat -tlnp (-t:tcp, -l:listening, -n: numberic, -p: programs) config: /etc/services ⇒ presently the policy of IANA to assign a single well-known

The Registered Ports are those from 1024 through 49151
The Dynamic and/or Private Ports are those from 49152 through 65535

• list all processes with UID,PID,PPID,CMD

  ps -eaf | head
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  0 01:18 ?        00:00:00 /sbin/init
  root         2     0  0 01:18 ?        00:00:00 [kthreadd]
  root         3     2  0 01:18 ?        00:00:00 [migration/0]
  root         4     2  0 01:18 ?        00:00:00 [ksoftirqd/0]

• list all processes with USER,PID,%CPU,%MEM,VSZ,RSS,COMMAND

  ps -aux | head
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  root         1  0.0  0.0  19232   360 ?        Ss   01:18   0:00 /sbin/init
  root         2  0.0  0.0      0     0 ?        S    01:18   0:00 [kthreadd]
  root         3  0.0  0.0      0     0 ?        S    01:18   0:00 [migration/0]
  root         4  0.0  0.0      0     0 ?        S    01:18   0:00 [ksoftirqd/0]
  root         5  0.0  0.0      0     0 ?        S    01:18   0:00 [migration/0]
  root         6  0.0  0.0      0     0 ?        S    01:18   0:00 [watchdog/0]
  root         7  0.0  0.0      0     0 ?        S    01:18   0:08 [events/0]
  root         8  0.0  0.0      0     0 ?        S    01:18   0:00 [cgroup]
  root         9  0.0  0.0      0     0 ?        S    01:18   0:00 [khelper]

• Get top 10 processes which sorted by CPU

  ps -aux | sort -nk +3 | tail -n 10
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  mysql     1071  0.1  7.6 710560 38184 ?        Sl   01:18   0:57 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
  apache    5614  0.3  5.8 275252 29400 ?        S    16:26   0:00 /usr/sbin/httpd
  apache    5537  0.5 10.6 300316 53632 ?        S    16:16   0:03 /usr/sbin/httpd
  apache    5553  0.5 11.2 301832 56360 ?        S    16:20   0:02 /usr/sbin/httpd
  apache    5552  0.6  8.9 290280 44920 ?        S    16:20   0:02 /usr/sbin/httpd
  apache    5578  0.8 11.2 301840 56296 ?        S    16:24   0:01 /usr/sbin/httpd

• Get top 10 processes which sorted by Memory Usage

  ps -aux | sort -nk +4 | tail -n 10
  USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  postfix   5249  0.0  0.6  81364  3336 ?        S    15:51   0:00 pickup -l -t fifo -u
  root      5250  0.0  0.8 100368  4452 ?        Ss   15:52   0:00 sshd: root@pts/0
  root      5287  0.0  0.8 100368  4476 ?        Ss   15:52   0:00 sshd: root@pts/1
  apache    5648  0.0  1.2 255040  6488 ?        S    16:29   0:00 /usr/sbin/httpd
  root      5141  0.0  1.9 254904  9920 ?        Ss   15:38   0:00 /usr/sbin/httpd
  apache    5640  0.1  3.1 262836 15744 ?        S    16:28   0:00 /usr/sbin/httpd
  apache    5647  1.5  7.0 280756 35176 ?        S    16:29   0:00 /usr/sbin/httpd
  mysql     1071  0.1  7.6 710560 38184 ?        Sl   01:18   0:57 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
  apache    5614  0.3  8.0 285904 40192 ?        S    16:26   0:00 /usr/sbin/httpd
  apache    5553  0.5 11.2 301832 56404 ?        S    16:20   0:02 /usr/sbin/httpd

Memory space is divided into memory used by processes, disk cache, free memory and memory used by kernel Some basic parameters in memory information:

• MemTotal — Total amount of physical RAM, in kilobytes.
• MemFree — The amount of physical RAM, in kilobytes, left unused by the system.
• Buffers — The amount of physical RAM, in kilobytes, used for file buffers.
• Cached — The amount of physical RAM, in kilobytes, used as cache memory(Disk Cache).
• SwapCached — The amount of swap, in kilobytes, used as cache memory.
• Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
• Inactive — The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
• Active(anon): active memory that is not file backed. This will typically be the higher chunk of active memory on a app server machine which does not have a db
• Active (file): active memory that is file backed. this will typically be the higher chunk of active memory on a data store machine that reads / writes from disk
• HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
• LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
• SwapTotal — The total amount of swap available, in kilobytes.
• SwapFree — The total amount of swap free, in kilobytes.

cat /proc/meminfo

output:

MemTotal:        1020400 kB
MemFree:           65064 kB
Buffers:          180148 kB
Cached:           166648 kB
SwapCached:           76 kB
Active:           370004 kB
Inactive:         423848 kB
Active(anon):     183084 kB
Inactive(anon):   264156 kB
Active(file):     186920 kB
Inactive(file):   159692 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1048568 kB
SwapFree:        1048440 kB
Dirty:               408 kB
Writeback:             0 kB
AnonPages:        447048 kB
Mapped:            19032 kB
Shmem:               184 kB
Slab:             140256 kB
SReclaimable:     118196 kB
SUnreclaim:        22060 kB
KernelStack:         992 kB
PageTables:         5092 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1558768 kB
Committed_AS:     604456 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        7320 kB
VmallocChunk:   34359727504 kB
HardwareCorrupted:     0 kB
AnonHugePages:    143360 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6136 kB
DirectMap2M:     1042432 kB
DirectMap1G:           0 kB

We have the expression:

Active(370004) = Active(anon: 183084) + Active(file:186920)
Inactive(423848) = Inactive(anon:264156) + Inactive(file:159692)
Memtotal = MemFree + Buffers + Cached + Active + Mapped + Shmem + Slab +　PageTables　+ VmallocUsed

top

output:

Cpu(s):  4.0%us,  0.5%sy,  0.0%ni, 94.4%id,  1.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1020400k total,   955336k used,    65064k free,   180148k buffers
Swap:  1048568k total,      128k used,  1048440k free,   166648k cached

The expression:

1020400k total = 955336k used + 65064k free
955336k used = 180148k buffers + 166648k cached + Other Mem(Kernel and App Server Use)

cat /proc/1704/status

output:

Name:   php-fpm
State:  S (sleeping)
Tgid:   1704
Pid:    1704
PPid:   1703
TracerPid:      0
Uid:    99      99      99      99
Gid:    99      99      99      99
Utrace: 0
FDSize: 64
Groups: 99
VmPeak:   329824 kB
VmSize:   329504 kB
VmLck:         0 kB
VmHWM:    127368 kB
VmRSS:    124412 kB
VmData:   118996 kB
VmStk:        88 kB
VmExe:      9920 kB
VmLib:     12832 kB
VmPTE:       476 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/7826

Explain the parameters:

• FDSize: Number of file descriptor slots currently allocated.
• Groups: Supplementary group list.
• VmPeak: Peak virtual memory size.
• VmSize: Virtual memory size.
• VmLck: Locked memory size (see mlock(3)).
• VmHWM: Peak resident set size (“high water mark”).
• VmRSS: Resident set size.
• VmData, VmStk, VmExe: Size of data, stack, and text segments.
• VmLib: Shared library code size.
• VmPTE: Page table entries size (since Linux 2.6.10).
• Threads: Number of threads in process containing this thread.

report memory map of a process

• pmap -x PID

  pmap -x 6128
  Address           Kbytes     RSS   Dirty Mode   Mapping
  00007f1b20662000      76       0       0 r-x--  zip.so
  00007f1b20675000    2044       0       0 -----  zip.so
  00007f1b20874000       8       8       8 rw---  zip.so
  00007f1b20876000      12       0       0 r-x--  libgpg-error.so.0.5.0
  ..................

• sort memory map base on Kbytes

  pmap -x 6128 | sort -nk +2 | tail -n 10
  00007f1b2bd85000    2048       0       0 -----  mod_log_config.so
  00007f1b2c99e000    2048       0       0 -----  mod_authz_user.so
  00007f1b2d1a9000    2048       0       0 -----  mod_authn_anon.so
  00007f1b2d7b7000    2048       0       0 -----  mod_auth_digest.so
  00007f1b2e039000    2048       0       0 -----  libdl-2.12.so
  00007f1b2e5e6000    2048       0       0 -----  libpthread-2.12.so
  00007f1b2efbb000    2048       0       0 -----  libcrypt-2.12.so
  00007f1b31cf7000    2380    2352    2352 rw---    [ anon ]
  00007f1b27b18000    3268    1664       0 r-x--  libphp5.so
  00007f1b31f4a000  123012  122464  122464 rw---    [ anon ]

• Get all threads in linux

  ps -efL

• Get all threads of MySQL(base on config thread_cache_size):

  ps -efL | grep mysql

• Step1: get pid of process:

  pgrep -f server.gamedangian.jar

  ⇒ output: 31497

• Step2: get list of open files

  ls -l /proc/31497/fd

• install lsof:

  yum install lsof

• list of file was opened by process:
  • Step1: get pid

    pgrep -f server.gamedangian.jar

    ⇒ output: 31497

  • Step2: get files was opened by process:

    lsof -p 31497

run below command:

cat /proc/sys/fs/file-nr

or run with lsof:

lsof | wc -l

There are two reasons lsof | wc -l doesn't count file descriptors. One is that it lists things that aren't open files, such as loaded dynamically linked libraries and current working directories; you need to filter them out. Another is that lsof takes some time to run, so can miss files that are opened or closed while it's running; therefore the number of listed open files is approximate

cat > abc.txt
content
#ctrl+D

# wc -l

• format: cut -cchars file
• example1:

  # who
  root     console Feb 24 08:54
  steve    tty02   Feb 24 12:55
  
  # who | cut -c1-8       Extract the first 8 characters
  root
  steve

• example2:

  # cut -d' ' -f 11
  # cut -d' ' -f 11-

Example:

# cat names
Tony
Emanuel

# cat numbers
(307) 555-5356
(212) 555-3456

# paste names numbers
Tony    (307) 555-5356
Emanuel (212) 555-3456

grep -v "#" 

⇒ get content none sign “#”

Basic syntax:

sed -i 's/original/new/g' file.txt

Explain options:

• -i = –in-place (i.e. save back to the original file)

The command string:

• s = the substitute command
• original = a regular expression describing the word to replace (or just the word itself)
• new = the text to replace it with
• g = global (i.e. replace all and not just the first occurrence)

Below are some basic examples for searching and replaccing string

cat intro

output:

The Unix operating system. Unix system

• Substitute Unix with UNIX:

  sed 's/Unix/UNIX/g' intro 

  ⇒ output:

  The UNIX operating system. Unix system

• Substitute Unix with UNIX in file intro:

  sed --in-place 's/Unix/UNIX/g' intro

• Substitute Unix with UNIX

  sed 's/Unix/UNIX/g' intro

• example with -n option

  sed -n '1,2p' intro

  ⇒ Just print the first 2 lines

• Just print lines containing UNIX:

  sed -n '/UNIX/p' intro

• example Deleting Lines

  sed '1,2d' intro

  ⇒ Delete lines 1 and 2

• Delete all lines containing UNIX:

  sed '/UNIX/d' intro

• Delete all characters before string

  sed 's/.*No Warranty/No Warranty/g'

# cat server
1-Ngoa Ho 10.30.31.10
3-Tang Long 10.30.31.38

• example1

  cat server |awk '{print $1}'
  => output:
  1-Ngoa
  3-Tang
   
  cat server |awk '{print $2}'
  => output:
  Ho
  Long
   
  cat server |awk '{print $3}'
  => output:  
  10.30.31.10
  10.30.31.38

• example2

  cat server |awk -F'-' '{print $1}'
  output:
  1
  3

md5sum, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum

• create checksum for file

  md5sum [OPTION]... [FILE]...

• create checksum for text:

  echo -n 'helloworld' | md5sum -

-A/--user-agent <string> User-Agent to send to server (H)
-b/--cookie <name=string/file> Cookie string or file to read cookies from (H)
-c/--cookie-jar <file> Write cookies to this file after operation (H)
  --create-dirs   Create necessary local directory hierarchy
  --crlf          Convert LF to CRLF in upload
  --crlfile <file> Get a CRL list in PEM format from the given file
-d/--data <data>   HTTP POST data (H)
  --data-ascii <data>  HTTP POST ASCII data (H)
  --data-binary <data> HTTP POST binary data (H)
  --data-urlencode <name=data/name@filename> HTTP POST data url encoded (H)
  --delegation STRING GSS-API delegation permission
  --digest        Use HTTP Digest Authentication (H)
  --disable-eprt  Inhibit using EPRT or LPRT (F)
  --disable-epsv  Inhibit using EPSV (F)
-F/--form <name=content> Specify HTTP multipart POST data (H)
    --form-string <name=string> Specify HTTP multipart POST data (H)
    --ftp-account <data> Account data to send when requested by server (F)
    --ftp-alternative-to-user <cmd> String to replace "USER [name]" (F)
    --ftp-create-dirs Create the remote dirs if not present (F)
    --ftp-method [multicwd/nocwd/singlecwd] Control CWD usage (F)
    --ftp-pasv      Use PASV/EPSV instead of PORT (F)
-I/--head          Show document info only
-k/--insecure      Allow connections to SSL sites without certs (H)
    --interface <interface> Specify network interface/address to use

Understand about options:

-d/--data <data>   HTTP POST data (H)
  --data-ascii <data>  HTTP POST ASCII data (H)
  --data-binary <data> HTTP POST binary data (H)
  --data-urlencode <name=data/name@filename> HTTP POST data url encoded (H)
  --delegation STRING GSS-API delegation permission
  --digest        Use HTTP Digest Authentication (H)
  --disable-eprt  Inhibit using EPRT or LPRT (F)
  --disable-epsv  Inhibit using EPSV (F)

• options -d ⇒ default option: –data:
  • curl -d <data>
  • Or: curl –data <data>
• And other options for postdata:

    --data-ascii <data>  HTTP POST ASCII data (H)
    --data-binary <data> HTTP POST binary data (H)
    --data-urlencode <name=data/name@filename> HTTP POST data url encoded (H)
    --delegation STRING GSS-API delegation permission
    --digest        Use HTTP Digest Authentication (H)
    --disable-eprt  Inhibit using EPRT or LPRT (F)
    --disable-epsv  Inhibit using EPSV (F)

• tcpdump command format

  tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
                [ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
                [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
                [ -W filecount ] [ -y datalinktype ] [ -Z user ]
                [ expression ]
  [expression] => proto(ip,ip6,tcp,udp,arp...),host,src host,dst host,port,src port,dst port

• tcpdump examples

  tcpdump -i eth0 port 80
  tcpdump -i eth0 dst port 80
  tcpdump -i lo port 10000 => listen among ports in localhost
  tcpdump -i eth0 -w file.dmp port 80
  tcpdump -i eth0 -r file.dmp port 80
  tcpdump src 10.199.3.129
  tcpdump dst 10.30.31.58
  tcpdump host 10.30.31.58
  tcpdump -v icmp
  tcpdump -v "icmp or arp"

rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Bangkok /etc/localtime

iptables -F

⇒ Stopping firewall and allowing everyone

iptables-save

• mount iso to directory

    mount -t iso9660 -o loop /root/debian-508-i386-DVD-1.iso /mnt/cdrom/
    

• mount iso to cdrom hardware

    mount -t iso9660 -o loop /root/debian-508-i386-DVD-1.iso /media/cdrom
    

• step1: create script mountram.sh

  cat > /root/scripts/mountram.sh
  #!/bin/sh
  # Size of your RAM disk (tmpfs)
  RAM_DISK_SIZE=1G
  MOUNT_POINT=/home/mole/data
  SOURCE_DIR=/home/mole/source_real
   
  # Make sure the mount point exists
  mkdir -p $MOUNT_POINT;
   
  # Now create the tmp filesystem on mount point
  mount -o size=$RAM_DISK_SIZE -t tmpfs tmpfs $MOUNT_POINT
  # Copy the source files to the mounted tmpfs filesystem
  cp -r $SOURCE_DIR/*   $MOUNT_POINT

• step2: create ram disk

  echo "none                    /dev/shm                tmpfs   defaults      0 0" >> /etc/fstab

• step3: config to autorun script mountram.sh when server start

  echo "/root/scripts/mountram.sh" >> /etc/rc.local
   

ldd /usr/sbin/vsftpd
linux-gate.so.1 =>  (0x00b1f000)
libssl.so.6 => /lib/libssl.so.6 (0x008e0000)
libwrap.so.0 => /lib/libwrap.so.0 (0x00944000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00991000)
libpam.so.0 => /lib/libpam.so.0 (0x00110000)
libcap.so.1 => /lib/libcap.so.1 (0x00fc0000)
libdl.so.2 => /lib/libdl.so.2 (0x00d9b000)
libc.so.6 => /lib/libc.so.6 (0x003af000)

objdump -f mole.vn.register

( rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.)

• rsync in local

  rsync -avz   --exclude=".svn" "/home/anhvc/web/9thien.com/" /web/9thien.com/

• rsync to remote host

  rsync -avz --exclude=".svn"  -e "ssh -p2222" "/home/anhvc/web" 123.30.245.164:~

1. Step1: Download and install rsync at here. cwRsync will installed to c:\Program Files (x86)\cwRsync\bin\
2. Step2: Add path “c:\Program Files (x86)\cwRsync\bin\” to PATH envionment variable on Windows
3. Step3: Check ssh with private-key:

   ssh -p1362  -i E:\backup\GoogleDrive\keys\id_rsa root@128.199.236.122

4. Step4: Run rsync via ssh to sync data:

   rsync -avz --exclude=".svn"  -e "ssh -p1362 -i E:\backup\GoogleDrive\keys\id_rsa" "/cygdrive/d/web/www.babies.vn/" root@128.199.236.122:/data/www/www.babies.vn/

5. Step5: Chown to apache.apache for /data/www/www.babies.vn/

   ssh -p1362  -i E:\backup\GoogleDrive\keys\id_rsa root@128.199.236.122 "chown -R apache.apache /data/www/www.babies.vn/"

6. Step6: Fix error rsync

   rsync: recv_generator: mkdir failed: Permission denied (13)

   Add option for rsync on windows:

   --no-p --no-g --chmod=ugo=rwX

b
=>list bundles

start ID
=> start bundles

sh
=>shutdown osgi

nm -C -D libhelloworldjni.so ⇒

0000054f T JNI_OnLoad
00000568 T JNI_OnUnload
0000050c T Java_HelloWorldJNI_sayGoodbye
00000520 T Java_HelloWorldJNI_sayHelloWorld__
00000534 T Java_HelloWorldJNI_sayHelloWorld(int0_t)

1. Step1: download cygwin setup from https://cygwin.com/install.html
2. Step2: Install Cygwin with default mode to setup basic packages, and we will install new packages If application must use them

• Go to directory d:\tools on windows:

  cd /cygdrive/d/tools