Snippets

20 Feb 2016 • Leave Comments

ABC
disk usage
arithmatic operation
if string comparision
apg
dig
find
Bulk rename
Number range
netstat
ln
Encoding/charset conversion
CR-LF/LF
1. Detect line ending:
2. Conversion
BOM
Remove whitespace
Empty line
newline
read line by line
darkhttpd
SSL cert
curl/wget
mtr/ping
Image crop and keep aspect ratio

ABC

cat -e input.file
sed -n l input.file

disk usage

# disk usage
du -hsc -- file1 dir1
du -hsc -- *
# disk free
df -h

arithmatic operation

echo 5.3025^2*4.871 | bc
bc <<< '100*0.13'
awk "BEGIN {print 100/3.5}"

if string comparision

if [[ "$str1" == "$str2" ]]; then
  echo "$str1 equals to $str2"
elif [[ $str1 == *"$str2"* ]]; then
  echo "$str1 contains $str2"
else
  echo "$str1 has nothing to do with $str2"
fi

Pay attention to quote.

apg

~ # apg -a1 -n1 -m16 -MSNCL -k

dig

~ # dig @dns-server -p 53 www.example.com

find

$ find -L /path/to/search -type f -iname "*filename*"
$ find -L /path/to/search -type f -iname '*filename*'
$ find -L /path/to/search -type f -iname '*.apk' -printf "%f\n" -exec cp -fv '{}' /path/to/copy \;
$ find APPs/ -maxdepth 1 -type f -iname '*.apk' -exec bash -c 'for file; do file="${file##*/}"; mkdir -p ROM/system/preset_apps/"${file%.apk}"; cp APPs/"${file}" ROM/system/preset_apps/"${file%.*}"; done' bash '{}' +
$ find -P history/ -type f -name '*.HEIC' -exec bash -c 'for file; do mv -v "$file" "${file%%.*}.heic"; done' 'bash' {} +
$ find ROM/system/media/wallpaper/ -depth -type f ! -path '*/wallpaper_15.png' -delete
$ find ROM/system/media/wallpaper/ -depth -type d -empty -delete
$ find -L . -type f -iname '*abc*' -exec bash -c 'mv "$0" "${0/abc/def}"' '{}' \;
$ export _pkg_name
$ find -H  -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) -exec bash -c 'for img; do mv "$img" pics/"${_pkg_name}"; done' bash '{}' +
$ find . -type f -name '*.jpg' -exec bash -c 'for f; do mv $f ${f//"$0"}; done' $'\302\240' '{}' +
$ find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw += $4; } END { print games, white, black, draw }'

Dont forget the quotes to prevent shell expansion.
In the fourth example, the random string argument (i.e. bash) preceding {} is a must. It serves as the very first argument for -exec bash command. You can either quote it or not.
Variables (i.e. _pkg_name) should be exported before calling 'find' as it spawns a new process.
-print0 prints the full file name on the standard output, followed by a null character (instead of the newline character that -print uses). This allows file names that contain newlines or other types of white space to be correctly interpreted by programs that process the find output. This option corresponds to the -0 option of xargs.

Sometimes we want to search for directories instead of files. As the very first result is the root directory itself, we may want to omit the 'bash' arbitrary string such that the root directory becomes the $0.

~ $ find -H . -type d -name '*figs*' -exec bash -c 'for d; do rm -rf $d; done' '{}' +
~ $ find -H . -type d -name '*figs*' -exec bash -c 'for d; do rm -rf $d; done' bash '{}' +

The second version will remove current directory as well. Removing the 'bash' string will leaves it untouched.

Bulk rename

a=1
for i in *.jpg; do
  new=$(printf "%04d.jpg" "$a") #04 pad to length of 4
  mv -- "$i" "$new"
  let a=a+1
done

Number range

# -w means equal width
$ seq -w 5 20

netstat

$ netstat -npeatu
$ ss -npeatu

Try l instead of a.

net-tools is deprecated in favor of iproute2. The alternative to netstat is ss.

ln

Soft/symbolic link (symlink) points to file's directory entry while hard link points to file's inode. No matter of symlink or hard link, both are real but special files.

When delete a symlink file, the orginal file is not touched. However, to delete a hard link file, the original file's hard link references count will be decreased by 1. If the final count equal to 0, then the original file is deleted as well.

Encoding/charset conversion

Charset is a set of character entities while encoding is its representation in the terms of bytes and bits. In Enca, the word encoding means the same as representation of text, i.e. the relation between sequence of character entities constituting the text and sequence of bytes (bits) constituting the file.

   $ enca/file filename, detect encoding
   $ enconv filename, convert to *locale*
   $ enconv/enca -x gb2312 filename, convert to specified encoding

enca/enconv is a tool superior to iconv.
It does NOT make backup.

CR-LF/LF

CR: Carriage Return, '\r', '\o015', '\xd'.
LF: Line Feeding, '\n', '\o12', '\xa'.
Some DOS files are ended with CR-LF, while others are LF-CR.

Detect line ending:

$ hexdump -c input.file, detect line ending
$ sed -n l input.file
$ file input.file
$ od -t c input.file

Conversion

app-text/dos2unix:

$ dos2unix/unix2dos -b input.file, '-b' makes a backup

perl:

$ perl -pi -e 's/\r//g' input.file
$ perl -pi -e 's/\n/\r\n/g' input.file

sed:

$ sed -i.bak 's/\r$//g' input.file
$ sed -i.bak 's/$/\r/g' input.file

tr:

$ tr -d '\r' < input.file > output.file, cannot do it the other way round

nano:

Type Ctrl-O and before confirming, type Alt-D (DOS) or Alt-M (Mac) to change the format.

Refer to remove CRs from CR-LF line terminators.

BOM

~ $ sed -i.bak '1s/^\xEF\xBB\xBF//' orig.txt

Remove whitespace

echo:

# `echo' without double quotes; squeeze sequencial spaces within the string to single one
str="   how   are	you   "
echo -e "$str"
echo -e $str
str=`echo $str`

sed:

$ str=" how    are    you   "
$ echo "$str" | sed 's/^[ \t]*//'
$ echo "$str" | sed 's/[ \t]*$//'
$ sed -e 's/[[:space:]]*$//' <<< "$str"
$ sed 's/^[ \t]*//;s/[ \t]*$//' <<< "$str"

awk/gawk:

# even sequencial tabs (maybe other blank characters) are squeezed to single space within string
$ awk '{$1=$1};1' file
$ awk '{$1=$1;print}' file

Empty line

~ $ sed '/^[[:space:]]*$/d' file.txt
~ $ sed '/^\s*$/d' file.txt
~ $ sed 's/^\s*$//g' file.txt   # failure as empty line replaced by a blank line

~ $ grep . file.txt

newline

Replace newline with another character:

~ $ tr '\n' ',' < file.txt

~ $ sed 's/\n/,/g' file.txt     # failure as sed read line but does not include the line terminator - newline
~ $ sed -z 's/\n/,/g' file.txt  # tell sed to use NULL as line terminator, so that newline is included when reading a line

read line by line

#!/bin/bash

input=~/workspace/bash/zh.txt
output=~/workspace/bash/zh.dict.yaml

> "$output"

while IFS=$'\t' read -r left right
do
    line=${left}$'\t'
    for word in $right; do
	if [[ $word == *"ā"* ]]; then
	    tmp=`sed 's/ā/a/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"á"* ]]; then
	    tmp=`sed 's/á/a/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǎ"* ]]; then
	    tmp=`sed 's/ǎ/a/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"à"* ]]; then
	    tmp=`sed 's/à/a/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ē"* ]]; then
	    tmp=`sed 's/ē/e/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"é"* ]]; then
	    tmp=`sed 's/é/e/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ě"* ]]; then
	    tmp=`sed 's/ě/e/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"è"* ]]; then
	    tmp=`sed 's/è/e/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ō"* ]]; then
	    tmp=`sed 's/ō/o/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"ó"* ]]; then
	    tmp=`sed 's/ó/o/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǒ"* ]]; then
	    tmp=`sed 's/ǒ/o/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ò"* ]]; then
	    tmp=`sed 's/ò/o/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ī"* ]]; then
	    tmp=`sed 's/ī/i/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"í"* ]]; then
	    tmp=`sed 's/í/i/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǐ"* ]]; then
	    tmp=`sed 's/ǐ/i/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ì"* ]]; then
	    tmp=`sed 's/ì/i/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ū"* ]]; then
	    tmp=`sed 's/ū/u/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"ú"* ]]; then
	    tmp=`sed 's/ú/u/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǔ"* ]]; then
	    tmp=`sed 's/ǔ/u/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ù"* ]]; then
	    tmp=`sed 's/ù/u/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ǖ"* ]]; then
	    tmp=`sed 's/ǖ/ü/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"ǘ"* ]]; then
	    tmp=`sed 's/ǘ/ü/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǚ"* ]]; then
	    tmp=`sed 's/ǚ/ü/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ǜ"* ]]; then
	    tmp=`sed 's/ǜ/ü/' <<< $word`
	    line=$line${tmp}4' '
	fi
    done
    sed -e 's/[[:space:]]*$//' <<< $line >/dev/null
    echo "$line" >> "$output"
done < "$input"

IFS= option before read command prevents leading/trailing whitespace from being trimmed.
The -r option passed to read command prevents backslash escapes from being interpreted.

Another example:

#!/bin/bash
file="/etc/passwd"
while IFS=: read -r f1 f2 f3 f4 f5 f6 f7
do
        # display fields using f1, f2,..,f7
        printf 'Username: %s, Shell: %s, Home Dir: %s\n' "$f1" "$f7" "$f6"
done <"$file"

Third version:

while read -p "input:" line
do
    echo "$line"
    printf "%s\n" "${line}"
done  < "${1:-/dev/stdin}"

Read from stdin or from a file parameter.

Here is an elegant version, just reading a line:

infd=0 ; [[ "${1}" ]] && : {infd}< "${1}"; read -p "input:" -r line <&"${infd}" # read -u $infd -r line

darkhttpd

$ ./darkhttpd ~/workspace/public_html --pidfile ~/workspace/httpd_traces/httpd.pid --daemon --maxconn 1 --no-listing --log ~/workspace/httpd_traces/access.log --forward example.com:8080 http://www.example.com:8080
or
$ ./darkhttpd ~/workspace/public_html --maxconn 1 --no-listing --log ~/workspace/httpd_traces/access.log --forward example.com:8080 http://www.example.com:8080 &

SSL cert

By openssl s_client:

~ $ openssl s_client -connect chat.freenode.net:6697 </dev/null
~ $ openssl s_client -connect chat.freenode.net:6697 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > freenode.pem

-showcerts show all certificates in the chain, including the CA certificate, otherwise only the server certificate was downloaded.
If the remote server is using SNI (that is, sharing multiple SSL hosts on a single IP address) you will also need to send the correct -servername in order to get the right certificate.

By gnutls-cli:

~ $ gnutls-cli --print-cert chat.freenode.net:6697 </dev/null
~ $ gnutls-cli --print-cert chat.freenode.net:6697 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > freenode.pem
~ $ gnutls-cli --print-cert --save-cert=freenode.pem chat.freenode.net:6697 </dev/null

--sni-hostname serves the same role as -servername of openssl s_client.
gnutls-cli always downloads the full certificate chain.

Examine the certificate downloaded:

~ $ cat freenode.pem; less freenode.pem
~ $ openssl x509 -noout -text -in freenode.pem
~ $ openssl x509 -noout -sha256 -fingerprint -in freenode.pem

curl/wget

Use curl/wget to test web/ftp service

~ $ wget -d http://www.example.com
~ $ wget -S http://www.example.com (Print the headers sent by HTTP servers and responses sent by FTP servers)
~ $ curl -v http://www.example.com
~ $ curl -I http://www.example.com (Fetch the headers only)
~ $ curl -i http://www.example.com (Include the HTTP response headers in the output)
~ $ curl --trace/--trace-ascii test.log http://www.example.com (Enables a full trace dump of all incoming and outgoing data, including descriptive information, to the given output file)
~ $ curl --socks5 127.0.0.1:1080 -v https://www.youtube.com (Test socks5 proxy)
~ $ curl --socks5-hostname 127.0.0.1:1080 -v https://www.youtube.com

Use curl to download a file

~ $ curl -OLv http://www.example.com/file.txt
~ $ curl -Lv -o new_file.txt http://www.example.com/file.txt

mtr/ping

~ $ mtr -rw 12.34.56.78

Image crop and keep aspect ratio

Filter the images:

#!/bin/bash

_input="kwdes.txt"

# Save as UTF-16 txt
# C-x RET r, revert-buffer-with-coding-system, UTF-8
sed -i.bak 's/\xC2\xA0//g' "${_input}" # non-breaking space
sed -i.bak 's/\r//g' "${_input}" # crlf to lf
sed -i.bak 's/[[:blank:]]*$//' "${_input}" # leading white spaces
sed -i.bak 's/^[[:blank:]]*//' "${_input}" # trailing white spaces
sed -i.bak '/^$/d' "${_input}" # empty lines
sed -i.bak -e '1d' "${_input}" # delete the 1st line
wc -l "${_input}"
#mkdir -p $(cut -d$'\t' -f1 "${_input}")
#awk -F'\t' 'FNR==linum { print $1, $4, $5 }' "${_input}"
#awk -F'\t' 'FNR==linum { printf("%s\t%s\t%s\n" $1, $4, $5) }' "${_input}"
#awk -F'\t' 'BEGIN { OFS=FS } FNR==linum { print $1, $2, $3 }' <<< $'a\tb\tc\td\n123\n'

Crop the images:

#!/bin/bash

# https://www.zedge.net/find/wallpapers/keyword, 8 * 1080x1920, com.uphone.wallpaperkeyword_description/keyword_date_num.jpg
# https://wallpaperscraft.com/catalog/keyword/1440x2560/page2
# https://wall.alphacoders.com/search.php?search=Minecraft#sorting_options
# icon_{1024*500,180*120}.jpg, ic_launcher_{512,192,148,96,72,48}.jpg
# magick -verbose input.jpg -trim -resize "1024x500^" -gravity center -extent "1024x500+0+0" +repage output.jpg # -extent -or -crop
# Parameters: line number > 1, selected filename

shopt -s extglob
#shopt -s nullglob

_workspace="${HOME}/WLshare"
cd "${_workspace}"

_input="kwdes.txt"

_linum="$1"
if [[ "${_linum}" < 1 ]]; then
    echo "line number starts from 1"
    exit
fi

_keyword=$(awk -F'\t' -v linum="${_linum}" 'FNR==linum {print $1}' "${_input}")
_keyword="${_keyword//[[:blank:]]/}"
echo "Keyword: ${_keyword}"

_pkg_name=$(awk -F'\t' -v linum="${_linum}" 'FNR==linum {print $4}' "${_input}")
echo "Package name: ${_pkg_name}"

rm -rf "${_keyword}"/*
mkdir -p "${_keyword}"/"${_pkg_name}"

_select="$2"; echo "Select: ${_select}"
_icon=(1024x500 180x120)
for i in "${_icon[@]}"; do
    magick "${_select}" -resize "${i}^" -gravity center -extent "${i}+0+0" +repage "${_keyword}"/"icon_${i%x*}.jpg"
done
_launcher=(512x512 192x192 148x148 96x96 72x72 48x48)
for i in "${_launcher[@]}"; do
    magick "${_select}" -resize "${i}^" -gravity center -extent "${i}+0+0" +repage "${_keyword}"/"ic_launcher_${i%x*}.jpg"
done

cp -f -- +(*.jpg|*.png) "${_keyword}"/"${_pkg_name}" 2>/dev/null
cd "${_keyword}"/"${_pkg_name}"
_size="1080x1920"
_counter=1
for f in *.+(jpg|png) ; do
    if [[ -f "$f" ]]; then
	magick "$f" -resize "${_size}^" -gravity center -extent "${_size}+0+0" +repage "$f"
	mv -f -- "$f" "${_keyword}_"$(date -I)"_${_counter}.jpg"
	_counter=$((_counter + 1))
    fi
done

# { cd ../../; rm -f -- *.+(jpg|png); }
echo 'done!'

ZNHOO Whatever you are, be a good one!