my notes

User Tools

Site Tools

Trace: • shellcommands
linux:shellcommands

Table of Contents

Shell Command

refer: linux_shell_scripting_with_bash_2004.pdf
Advanced Bash-Scripting Guide:http://tldp.org/LDP/abs/html/
Bash script: http://www.linuxconfig.org/Bash_scripting_Tutorial

Basic Commands

man

man -a read
=> output:
POSIX Programmers Manual                   READ(P)
Linux Programmers Manual                   READ(2)

man -a stat
=> output:
User Commands                                STAT(1)
Linux Programmer?s Manual                   STAT(2)

uname

uname -a

⇒Linux sgwp 2.6.17 #3 PREEMPT Fri Apr 4 00:18:16 JST 2008 armv5tel unknown unknown GNU/Linux

date

  • set date
    date +%Y%m%d -s "20120120"  
date +%T -s "16:00:00"
 
  • display date
    date +%Y%m%d#=> display date follow format %Y%m%d
date +%s#=> display UNIX time
  • convert unixtime
      date -d @1098181096
  • sync time:
    ntpdate  pool.ntp.org

file and directory commands

dirname

get directory name from file

ls

  • list derectorys
    ls -l | grep '^d' | awk '{ print $9 }'
  • list files
    ls -1

find files in directory

find . -name filename

ln

ln [OPTION]... TARGET [LINK_NAME]

Create a link to the specified TARGET with optional LINK_NAME. If LINK_NAME is omitted, a link with the same basename as the TAR-GET is created in the current directory 

ssh $i "rm -rf AM;mkdir AM;ln -s /home/ztgame/$1/release/ /home/ztgame/AM"

( -s, –symbolic make symbolic links instead of hard links)

du

du -sh /db/mysql/ 
df -ah
df -h

lock/unlock file and directory

cd /home/admin/conf
lsattr -a
-------------e- ./web
----i--------e- ./.
-------------e- ./..
-------------e- ./dns
-------------e- ./mail

⇒ directory /home/admin has attribute i ⇒ can't remove or create new files in this directory with root And remove this attribute 

chattr -i /home/admin/conf/

Shell Security

chmod

chmod a+w

⇒permission write for all user

Job and process, threads commands

whereis

whereis java

⇒ specific path of process java 

whereis iptables

⇒ iptables: /sbin/iptables /lib/iptables /usr/share/man/man8/iptables.8.gz

kill

kill `pidof zabbix_agentd`
killall zabbix_agentd

netstat

netstat -tlnp (-t:tcp, -l:listening, -n: numberic, -p: programs) config: /etc/services ⇒ presently the policy of IANA to assign a single well-known 

The Registered Ports are those from 1024 through 49151
The Dynamic and/or Private Ports are those from 49152 through 65535

ps

  • list all processes with UID,PID,PPID,CMD 
    ps -eaf | head
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 01:18 ?        00:00:00 /sbin/init
root         2     0  0 01:18 ?        00:00:00 [kthreadd]
root         3     2  0 01:18 ?        00:00:00 [migration/0]
root         4     2  0 01:18 ?        00:00:00 [ksoftirqd/0]
  • list all processes with USER,PID,%CPU,%MEM,VSZ,RSS,COMMAND
    ps -aux | head
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19232   360 ?        Ss   01:18   0:00 /sbin/init
root         2  0.0  0.0      0     0 ?        S    01:18   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    01:18   0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S    01:18   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    01:18   0:00 [migration/0]
root         6  0.0  0.0      0     0 ?        S    01:18   0:00 [watchdog/0]
root         7  0.0  0.0      0     0 ?        S    01:18   0:08 [events/0]
root         8  0.0  0.0      0     0 ?        S    01:18   0:00 [cgroup]
root         9  0.0  0.0      0     0 ?        S    01:18   0:00 [khelper]
  • Get top 10 processes which sorted by CPU
    ps -aux | sort -nk +3 | tail -n 10
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mysql     1071  0.1  7.6 710560 38184 ?        Sl   01:18   0:57 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
apache    5614  0.3  5.8 275252 29400 ?        S    16:26   0:00 /usr/sbin/httpd
apache    5537  0.5 10.6 300316 53632 ?        S    16:16   0:03 /usr/sbin/httpd
apache    5553  0.5 11.2 301832 56360 ?        S    16:20   0:02 /usr/sbin/httpd
apache    5552  0.6  8.9 290280 44920 ?        S    16:20   0:02 /usr/sbin/httpd
apache    5578  0.8 11.2 301840 56296 ?        S    16:24   0:01 /usr/sbin/httpd
  • Get top 10 processes which sorted by Memory Usage
    ps -aux | sort -nk +4 | tail -n 10
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postfix   5249  0.0  0.6  81364  3336 ?        S    15:51   0:00 pickup -l -t fifo -u
root      5250  0.0  0.8 100368  4452 ?        Ss   15:52   0:00 sshd: root@pts/0
root      5287  0.0  0.8 100368  4476 ?        Ss   15:52   0:00 sshd: root@pts/1
apache    5648  0.0  1.2 255040  6488 ?        S    16:29   0:00 /usr/sbin/httpd
root      5141  0.0  1.9 254904  9920 ?        Ss   15:38   0:00 /usr/sbin/httpd
apache    5640  0.1  3.1 262836 15744 ?        S    16:28   0:00 /usr/sbin/httpd
apache    5647  1.5  7.0 280756 35176 ?        S    16:29   0:00 /usr/sbin/httpd
mysql     1071  0.1  7.6 710560 38184 ?        Sl   01:18   0:57 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
apache    5614  0.3  8.0 285904 40192 ?        S    16:26   0:00 /usr/sbin/httpd
apache    5553  0.5 11.2 301832 56404 ?        S    16:20   0:02 /usr/sbin/httpd

Check memory information

Memory space is divided into memory used by processes, disk cache, free memory and memory used by kernel Some basic parameters in memory information:

  • MemTotal — Total amount of physical RAM, in kilobytes.
  • MemFree — The amount of physical RAM, in kilobytes, left unused by the system.
  • Buffers — The amount of physical RAM, in kilobytes, used for file buffers.
  • Cached — The amount of physical RAM, in kilobytes, used as cache memory(Disk Cache).
  • SwapCached — The amount of swap, in kilobytes, used as cache memory.
  • Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
  • Inactive — The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
  • Active(anon): active memory that is not file backed. This will typically be the higher chunk of active memory on a app server machine which does not have a db
  • Active (file): active memory that is file backed. this will typically be the higher chunk of active memory on a data store machine that reads / writes from disk
  • HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
  • LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
  • SwapTotal — The total amount of swap available, in kilobytes.
  • SwapFree — The total amount of swap free, in kilobytes.

in /proc/meminfo

cat /proc/meminfo

output: 

MemTotal:        1020400 kB
MemFree:           65064 kB
Buffers:          180148 kB
Cached:           166648 kB
SwapCached:           76 kB
Active:           370004 kB
Inactive:         423848 kB
Active(anon):     183084 kB
Inactive(anon):   264156 kB
Active(file):     186920 kB
Inactive(file):   159692 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       1048568 kB
SwapFree:        1048440 kB
Dirty:               408 kB
Writeback:             0 kB
AnonPages:        447048 kB
Mapped:            19032 kB
Shmem:               184 kB
Slab:             140256 kB
SReclaimable:     118196 kB
SUnreclaim:        22060 kB
KernelStack:         992 kB
PageTables:         5092 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1558768 kB
Committed_AS:     604456 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        7320 kB
VmallocChunk:   34359727504 kB
HardwareCorrupted:     0 kB
AnonHugePages:    143360 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6136 kB
DirectMap2M:     1042432 kB
DirectMap1G:           0 kB

We have the expression: 

Active(370004) = Active(anon: 183084) + Active(file:186920)
Inactive(423848) = Inactive(anon:264156) + Inactive(file:159692)
Memtotal = MemFree + Buffers + Cached + Active + Mapped + Shmem + Slab +　PageTables　+ VmallocUsed

In top command

top

output: 

Cpu(s):  4.0%us,  0.5%sy,  0.0%ni, 94.4%id,  1.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1020400k total,   955336k used,    65064k free,   180148k buffers
Swap:  1048568k total,      128k used,  1048440k free,   166648k cached

The expression: 

1020400k total = 955336k used + 65064k free
955336k used = 180148k buffers + 166648k cached + Other Mem(Kernel and App Server Use)

Check memory of process

cat /proc/1704/status

output: 

Name:   php-fpm
State:  S (sleeping)
Tgid:   1704
Pid:    1704
PPid:   1703
TracerPid:      0
Uid:    99      99      99      99
Gid:    99      99      99      99
Utrace: 0
FDSize: 64
Groups: 99
VmPeak:   329824 kB
VmSize:   329504 kB
VmLck:         0 kB
VmHWM:    127368 kB
VmRSS:    124412 kB
VmData:   118996 kB
VmStk:        88 kB
VmExe:      9920 kB
VmLib:     12832 kB
VmPTE:       476 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/7826

Explain the parameters:

  • FDSize: Number of file descriptor slots currently allocated.
  • Groups: Supplementary group list.
  • VmPeak: Peak virtual memory size.
  • VmSize: Virtual memory size.
  • VmLck: Locked memory size (see mlock(3)).
  • VmHWM: Peak resident set size (“high water mark”).
  • VmRSS: Resident set size.
  • VmData, VmStk, VmExe: Size of data, stack, and text segments.
  • VmLib: Shared library code size.
  • VmPTE: Page table entries size (since Linux 2.6.10).
  • Threads: Number of threads in process containing this thread.

pmap

report memory map of a process

  • pmap -x PID
    pmap -x 6128
Address           Kbytes     RSS   Dirty Mode   Mapping
00007f1b20662000      76       0       0 r-x--  zip.so
00007f1b20675000    2044       0       0 -----  zip.so
00007f1b20874000       8       8       8 rw---  zip.so
00007f1b20876000      12       0       0 r-x--  libgpg-error.so.0.5.0
..................
  • sort memory map base on Kbytes
    pmap -x 6128 | sort -nk +2 | tail -n 10
00007f1b2bd85000    2048       0       0 -----  mod_log_config.so
00007f1b2c99e000    2048       0       0 -----  mod_authz_user.so
00007f1b2d1a9000    2048       0       0 -----  mod_authn_anon.so
00007f1b2d7b7000    2048       0       0 -----  mod_auth_digest.so
00007f1b2e039000    2048       0       0 -----  libdl-2.12.so
00007f1b2e5e6000    2048       0       0 -----  libpthread-2.12.so
00007f1b2efbb000    2048       0       0 -----  libcrypt-2.12.so
00007f1b31cf7000    2380    2352    2352 rw---    [ anon ]
00007f1b27b18000    3268    1664       0 r-x--  libphp5.so
00007f1b31f4a000  123012  122464  122464 rw---    [ anon ]

Get all threads of process

  • Get all threads in linux
    ps -efL
  • Get all threads of MySQL(base on config thread_cache_size):
    ps -efL | grep mysql

List Open Files for Process

get open files base on OS information

  • Step1: get pid of process:
    pgrep -f server.gamedangian.jar

    ⇒ output: 31497

  • Step2: get list of open files
    ls -l /proc/31497/fd

lsof

  • install lsof:
    yum install lsof
  • list of file was opened by process:
    • Step1: get pid
      pgrep -f server.gamedangian.jar

      ⇒ output: 31497

    • Step2: get files was opened by process:
      lsof -p 31497

Get total open files in linux

run below command: 

cat /proc/sys/fs/file-nr

or run with lsof: 

lsof | wc -l

There are two reasons lsof | wc -l doesn't count file descriptors. One is that it lists things that aren't open files, such as loaded dynamically linked libraries and current working directories; you need to filter them out. Another is that lsof takes some time to run, so can miss files that are opened or closed while it's running; therefore the number of listed open files is approximate

Text Processing

cat

cat > abc.txt
content
#ctrl+D

wc

# wc -l

cut

  • format: cut -cchars file
  • example1: 
    # who
root     console Feb 24 08:54
steve    tty02   Feb 24 12:55

# who | cut -c1-8       Extract the first 8 characters
root
steve
  • example2: 
    # cut -d' ' -f 11
# cut -d' ' -f 11-

paste

Example: 

# cat names
Tony
Emanuel
# cat numbers
(307) 555-5356
(212) 555-3456
# paste names numbers
Tony    (307) 555-5356
Emanuel (212) 555-3456

grep

grep -v "#"

⇒ get content none sign “#”

sed

Basic syntax: 

sed -i 's/original/new/g' file.txt

Explain options:

  • -i = –in-place (i.e. save back to the original file)

The command string:

  • s = the substitute command
  • original = a regular expression describing the word to replace (or just the word itself)
  • new = the text to replace it with
  • g = global (i.e. replace all and not just the first occurrence)

Below are some basic examples for searching and replaccing string 

cat intro

output: 

The Unix operating system. Unix system
  • Substitute Unix with UNIX: 
    sed 's/Unix/UNIX/g' intro

    ⇒ output: 

    The UNIX operating system. Unix system
  • Substitute Unix with UNIX in file intro: 
    sed --in-place 's/Unix/UNIX/g' intro
  • Substitute Unix with UNIX
    sed 's/Unix/UNIX/g' intro
  • example with -n option
    sed -n '1,2p' intro

    ⇒ Just print the first 2 lines

  • Just print lines containing UNIX: 
    sed -n '/UNIX/p' intro
  • example Deleting Lines
    sed '1,2d' intro

    ⇒ Delete lines 1 and 2

  • Delete all lines containing UNIX:
    sed '/UNIX/d' intro
  • Delete all characters before string
    sed 's/.*No Warranty/No Warranty/g'

awk

# cat server
1-Ngoa Ho 10.30.31.10
3-Tang Long 10.30.31.38
  • example1
    cat server |awk '{print $1}'
=> output:
1-Ngoa
3-Tang
 
cat server |awk '{print $2}'
=> output:
Ho
Long
 
cat server |awk '{print $3}'
=> output:  
10.30.31.10
10.30.31.38
  • example2
    cat server |awk -F'-' '{print $1}'
output:
1
3

Custom Commands

checksum

md5sum, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum

  • create checksum for file 
    md5sum [OPTION]... [FILE]...
  • create checksum for text:
    echo -n 'helloworld' | md5sum -

curl

basic options

-A/--user-agent <string> User-Agent to send to server (H)
-b/--cookie <name=string/file> Cookie string or file to read cookies from (H)
-c/--cookie-jar <file> Write cookies to this file after operation (H)
  --create-dirs   Create necessary local directory hierarchy
  --crlf          Convert LF to CRLF in upload
  --crlfile <file> Get a CRL list in PEM format from the given file
-d/--data <data>   HTTP POST data (H)
  --data-ascii <data>  HTTP POST ASCII data (H)
  --data-binary <data> HTTP POST binary data (H)
  --data-urlencode <name=data/name@filename> HTTP POST data url encoded (H)
  --delegation STRING GSS-API delegation permission
  --digest        Use HTTP Digest Authentication (H)
  --disable-eprt  Inhibit using EPRT or LPRT (F)
  --disable-epsv  Inhibit using EPSV (F)
-F/--form <name=content> Specify HTTP multipart POST data (H)
    --form-string <name=string> Specify HTTP multipart POST data (H)
    --ftp-account <data> Account data to send when requested by server (F)
    --ftp-alternative-to-user <cmd> String to replace "USER [name]" (F)
    --ftp-create-dirs Create the remote dirs if not present (F)
    --ftp-method [multicwd/nocwd/singlecwd] Control CWD usage (F)
    --ftp-pasv      Use PASV/EPSV instead of PORT (F)
-I/--head          Show document info only
-k/--insecure      Allow connections to SSL sites without certs (H)
    --interface <interface> Specify network interface/address to use

Understand about options: 

-d/--data <data>   HTTP POST data (H)
  --data-ascii <data>  HTTP POST ASCII data (H)
  --data-binary <data> HTTP POST binary data (H)
  --data-urlencode <name=data/name@filename> HTTP POST data url encoded (H)
  --delegation STRING GSS-API delegation permission
  --digest        Use HTTP Digest Authentication (H)
  --disable-eprt  Inhibit using EPRT or LPRT (F)
  --disable-epsv  Inhibit using EPSV (F)
  • options -d ⇒ default option: –data:
    • curl -d <data>
    • Or: curl –data <data>
  • And other options for postdata:
      --data-ascii <data>  HTTP POST ASCII data (H)
  --data-binary <data> HTTP POST binary data (H)
  --data-urlencode <name=data/name@filename> HTTP POST data url encoded (H)
  --delegation STRING GSS-API delegation permission
  --digest        Use HTTP Digest Authentication (H)
  --disable-eprt  Inhibit using EPRT or LPRT (F)
  --disable-epsv  Inhibit using EPSV (F)

examples for curl

tcpdump

  • tcpdump command format
    tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]
              [ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]
              [ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]
              [ -W filecount ] [ -y datalinktype ] [ -Z user ]
              [ expression ]
[expression] => proto(ip,ip6,tcp,udp,arp...),host,src host,dst host,port,src port,dst port
  • tcpdump examples
    tcpdump -i eth0 port 80
tcpdump -i eth0 dst port 80
tcpdump -i lo port 10000 => listen among ports in localhost
tcpdump -i eth0 -w file.dmp port 80
tcpdump -i eth0 -r file.dmp port 80
tcpdump src 10.199.3.129
tcpdump dst 10.30.31.58
tcpdump host 10.30.31.58
tcpdump -v icmp
tcpdump -v "icmp or arp"

change timezone

rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Bangkok /etc/localtime

iptables

iptables -F

⇒ Stopping firewall and allowing everyone 

iptables-save

mount

mount iso
  • mount iso to directory
      mount -t iso9660 -o loop /root/debian-508-i386-DVD-1.iso /mnt/cdrom/
  • mount iso to cdrom hardware
      mount -t iso9660 -o loop /root/debian-508-i386-DVD-1.iso /media/cdrom
mount to RAM Disk
  • step1: create script mountram.sh 
    cat > /root/scripts/mountram.sh
#!/bin/sh
# Size of your RAM disk (tmpfs)
RAM_DISK_SIZE=1G
MOUNT_POINT=/home/mole/data
SOURCE_DIR=/home/mole/source_real
 
# Make sure the mount point exists
mkdir -p $MOUNT_POINT;
 
# Now create the tmp filesystem on mount point
mount -o size=$RAM_DISK_SIZE -t tmpfs tmpfs $MOUNT_POINT
# Copy the source files to the mounted tmpfs filesystem
cp -r $SOURCE_DIR/*   $MOUNT_POINT
  • step2: create ram disk
    echo "none                    /dev/shm                tmpfs   defaults      0 0" >> /etc/fstab
  • step3: config to autorun script mountram.sh when server start 
    echo "/root/scripts/mountram.sh" >> /etc/rc.local
 

ldd: print shared library dependencies of binary file

ldd /usr/sbin/vsftpd
linux-gate.so.1 =>  (0x00b1f000)
libssl.so.6 => /lib/libssl.so.6 (0x008e0000)
libwrap.so.0 => /lib/libwrap.so.0 (0x00944000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00991000)
libpam.so.0 => /lib/libpam.so.0 (0x00110000)
libcap.so.1 => /lib/libcap.so.1 (0x00fc0000)
libdl.so.2 => /lib/libdl.so.2 (0x00d9b000)
libc.so.6 => /lib/libc.so.6 (0x003af000)

objdump: Check target binary

objdump -f mole.vn.register

rsync

( rsync is a program that behaves in much the same way that rcp does, but has many more options and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file is being updated. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm described in the technical report that accompanies this package.)

  • rsync in local
    rsync -avz   --exclude=".svn" "/home/anhvc/web/9thien.com/" /web/9thien.com/
  • rsync to remote host
    rsync -avz --exclude=".svn"  -e "ssh -p2222" "/home/anhvc/web" 123.30.245.164:~

rsync on windows

  1. Step1: Download and install rsync at here. cwRsync will installed to c:\Program Files (x86)\cwRsync\bin\
  2. Step2: Add path “c:\Program Files (x86)\cwRsync\bin\” to PATH envionment variable on Windows
  3. Step3: Check ssh with private-key:
    ssh -p1362  -i E:\backup\GoogleDrive\keys\id_rsa root@128.199.236.122
  4. Step4: Run rsync via ssh to sync data:
    rsync -avz --exclude=".svn"  -e "ssh -p1362 -i E:\backup\GoogleDrive\keys\id_rsa" "/cygdrive/d/web/www.babies.vn/" root@128.199.236.122:/data/www/www.babies.vn/
  5. Step5: Chown to apache.apache for /data/www/www.babies.vn/
    ssh -p1362  -i E:\backup\GoogleDrive\keys\id_rsa root@128.199.236.122 "chown -R apache.apache /data/www/www.babies.vn/"
  6. Step6: Fix error rsync
    rsync: recv_generator: mkdir failed: Permission denied (13)

    Add option for rsync on windows: 

    --no-p --no-g --chmod=ugo=rwX

osgi

b
=>list bundles

start ID
=> start bundles

sh
=>shutdown osgi

nm: list symbols from object files

nm -C -D libhelloworldjni.so ⇒ 

0000054f T JNI_OnLoad
00000568 T JNI_OnUnload
0000050c T Java_HelloWorldJNI_sayGoodbye
00000520 T Java_HelloWorldJNI_sayHelloWorld__
00000534 T Java_HelloWorldJNI_sayHelloWorld(int0_t)

Cygwin

Install Cygwin

  1. Step1: download cygwin setup from https://cygwin.com/install.html
  2. Step2: Install Cygwin with default mode to setup basic packages, and we will install new packages If application must use them

Using Cygwin

  • Go to directory d:\tools on windows:
    cd /cygdrive/d/tools
linux/shellcommands.txt · Last modified: 2022/10/29 16:15 by 127.0.0.1