ZNHOO

Migrate Docker Desktop to colima

2024-12-27T00:00:00+00:00

Introduction
Uninstall Docker Desktop
colima Setup
Docker CLI Setup
Kubernetes Setup
References

As required by the company, we have to remove Docker Desktop from our macOS due to the pricing model change. After searching over the web, I found the colima - container runtime a good alternative.

Introduction

CoLiMa stands for Containers on Lima (Linux Virtual Machine). It supports three container runtimes, namely official Docker (default), pure containerd, Incus. In this post, we elect the default runtime.

To put it simple, colima creates a Linux Virtual Machine on macOS based on Hypervisor and Lima, and then spanws the specified runtime process within.

Containers communicate with the VM instead of macOS host. To make resources on the macOS available to containers, please mount them to the VM first. See the SSH Agent Forwarding example.

Uninstall Docker Desktop

Docker Desktop includes both the Docker CLI and Docker engine. After uninstallation, both components are removed, including images, containers, volumes, networks, etc.

Once completed, all Docker objects are gone, inlcuding containers! See https://docs.docker.com/desktop/uninstall/.

colima Setup

Install colima

~ $ brew install colima
...

==> Installing dependencies for colima: lima
==> Installing colima dependency: lima
==> Downloading https://ghcr.io/v2/homebrew/core/lima/manifests/1.0.3
Already downloaded: /Users/jim/Library/Caches/Homebrew/downloads/1433d7017c4773c2360cdd9ae90a6bf68d86b93c83daac1ed445db685427e583--lima-1.0.3.bottle_manifest.json
==> Pouring lima--1.0.3.arm64_sonoma.bottle.tar.gz
🍺  /opt/homebrew/Cellar/lima/1.0.3: 107 files, 205.6MB
==> Installing colima
==> Pouring colima--0.8.1.arm64_sonoma.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /opt/homebrew/etc/bash_completion.d

To start colima now and restart at login:
  brew services start colima
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/colima/bin/colima start -f
==> Summary
🍺  /opt/homebrew/Cellar/colima/0.8.1: 11 files, 5.7MB
==> Running `brew cleanup colima`...

...

Check version.

~ colima version
colima version 0.8.1
git commit: 96598cc5b64e5e9e1e64891642b91edc8ac49d16

runtime: docker
arch: aarch64
client: v27.4.1
server: v27.4.0

~ $ limactl --version
limactl version 1.0.3

Lima is a dependency of the colima, and hence it is also installed.

colima profile

colima has an important concept of profile. Each profile represents the configuration arguments (e.g. CPU architecture) of the Linux virtual machine, namely a colima instance. The colima CLI has a global option --profile, -p that defines the default configuration of colima engines by colima template. For example, we can create two profiles for CPU architectures aarch64 and x86_64 respectively. The default profile name is default which uses the same CPU architecture as the macOS host.

In this post, the terms colima profile, colima instance, colima VM and colima engine could be used interchangeably.

For the list of supported configuration fields, please refer to the config colima section.

Config colima

The $COLIMA_HOME environment variable specifies the directory that holds everything of colima. By default, it points to $HOME/.colima.

21:52:17 jim@Jims-MacBook-Pro ~
$ echo 'export COLIMA_HOME=$HOME/.colima' >> .bashrc

$ export COLIMA_HOME=$HOME/.colima
$ mkdir -p $COLIMA_HOME

The list of supported configurations can be found by colima help start. They can be resources (CPU/memory) constraints, CPU architectures, engines (--runtime), dns, etc. We can even forward environment variables on the host to the VM via --env.

To set the default config of a profile, run colima template -p <profile>. The config file of a profile is located at ${COLIMA_HOME}/_templates/<profile>.yaml. Each CLI option has a corresponding field in this file with detailed explanation. Actually, the configuration of colima is transformed to configurations of lima VM and Docker components.

~ $ colima template -p default --editor nano

~ $ ggrep -E '^[[:space:]]*[-a-z]+' $COLIMA_HOME/_templates/default.yaml

You can fetch a config template at https://gist.github.com/outsinre/b7c8347ff0c0b0053044a9c1b61f4b6e. The default resources assigned to the Linux VM is 2 CPUs, 2 GiB memory and 100 GiB disk space. We want to increase the CPU and memory a bit. Specially, we can set as many CPU (e.g. 8 out 10) as we can, as the OS is smart enough to schedual the CPU between the colima VM and the system. By default, this rule applies to memory as well (e.g. 30 out 36). We call this as ballooning.

Additionally, we want to update more fields. The vmType is set to vzinstead of qemu, mountType is set to virtiofs, and rosetta is turned on. The three configs can boost performance on Apple M1 chip. Optionally, we can enable Kubernetes support.

However, due to a macOS bug, the memory ballooning does not work with vmType: vz. If the macOS experiences performance issue, reduce the memory size or restart colima.

If you experience any network issue, like cannot connecting to public Internet from within containers, cannot pulling images, cannot access to HTTPS URLs, please enable --network-address and/or --network-preferred-route=false. Once enabled, the VM has an reachable IP address 192.168.64.8. See The Virtual Machine's IP is not reachable.

~ $ colima list
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK      RUNTIME    ADDRESS
default    Running    aarch64    8       30GiB     100GiB    docker     192.168.64.8

~ $ curl http://192.168.64.8:32771/anything
~ $ curl http://localhost:32771/anything

Upon colima start, we can customize the default config in three ways.

With the --edit CLI option
With the specific config CLI option (e.g. --dns)
Manually edit the ${COLIMA_HOME}/<profile>/<hostname>.yaml. For example, the hostname of the colima VM of the default profile is colima. It can be customized with the hostname field.

Reference Can config file be used instead of cli flags?.

Start colima

List existing instances.

~ $ colima list
WARN[0000] No instance found. Run `colima start` to create an instance.
PROFILE    STATUS    ARCH    CPUS    MEMORY    DISK    RUNTIME    ADDRESS

Start an instance of the default profile.

~ $ colima start -p default --template default --kubernetes=true
INFO[0000] starting colima
INFO[0000] runtime: docker+k3s
INFO[0001] starting ...                                  context=vm
INFO[0013] provisioning ...                              context=docker
INFO[0014] starting ...                                  context=docker
INFO[0015] provisioning ...                              context=kubernetes
INFO[0015] downloading and installing ...                context=kubernetes
INFO[0109] loading oci images ...                        context=kubernetes
INFO[0118] starting ...                                  context=kubernetes
INFO[0122] updating config ...                           context=kubernetes
INFO[0123] Switched to context "colima".                 context=kubernetes
INFO[0124] done

The colima instance exposes a Docker Unix socket file docker.sock to the macOS host.

~ $ colima list
PROFILE    STATUS     ARCH       CPUS    MEMORY    DISK      RUNTIME    ADDRESS
default    Running    aarch64    4       8GiB      100GiB    docker

~ $ colima status -p default
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: sshfs
INFO[0000] socket: unix:///Users/jim/.colima/default/docker.sock

A new Docker context called colima is created and automatically activated (autoActivate). The associated docker.sock is mapped from the colima VM to the macOS host.

~ $ docker context list
NAME       DESCRIPTION                               DOCKER ENDPOINT                                     ERROR
colima *   colima                                    unix:///Users/jim/.colima/default/docker.sock
default    Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

The original Docker official context called default is still bound to the DOCKER_HOST environment variable. We can use this variable to switch between different engines.

A new Docker builder colima is created and automatically activated. Regarding the error status of the Docker official default builder, see Unix socket file.

~ $ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
colima*      docker
 \_ colima    \_ colima        running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386
default                        error

Cannot load builder default: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

We can also make use of brew services to start colima. But currently, it does not support CLI options. What is worse, it can not create the docker context! The advantage of brew services start is it would regiester the service and autostart at system booting.

# take around 30s
~ $ brew services start colima

~ $ brew services info colima

ssh colima

A SSH config entry is inserted to ${COLIMA_HOME}/ssh_config for the colima VM. Additionally, this entry is also inserted to the macOS SSH config.

~ $ less ${COLIMA_HOME}/ssh_config

~ $ head ~/.ssh/config
Include /Users/jim/.colima/ssh_config

So, we can get into the VM via ssh. Please check the config file for the hostname field.

~ $ ssh <hostname>
# -or-
~ $ ssh -F ${COLIMA_HOME}/ssh_config <hostname>

Alternatively, colima provides the ssh sub-command accepting the profile name. This sub-command preserves the local dir.

21:37:28 jim@Jims-MacBook-Pro ~/misc
$ colima ssh -p default --very-verbose
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
TRAC[0000] cmd int ["limactl" "shell" "--workdir" "/Users/jim/misc" "colima"]

Inspect colima

After getting into the VM, we find many facts.

The VM OS is Ubuntu 24.04. Please check the diskImage filed in the config file.

~ $ ssh colima
Last login: Tue Dec 31 11:54:26 2024 from 192.168.5.2

jim@colima:~$ uname -a
Linux colima 6.8.0-50-generic #51-Ubuntu SMP PREEMPT_DYNAMIC Sat Nov  9 18:03:35 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

jim@colima:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

The colima VM is assigned two IP addresses 192.168.5.1 and 192.168.64.8.

jim@colima:~$ ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:f2:6b:1d brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.1/24 metric 200 brd 192.168.5.255 scope global dynamic eth0
       valid_lft 3596sec preferred_lft 3596sec
    inet6 fe80::5055:55ff:fef2:6b1d/64 scope link
       valid_lft forever preferred_lft forever
3: col0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:55:55:c5:da:42 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.8/24 metric 300 brd 192.168.64.255 scope global dynamic col0
       valid_lft 1796sec preferred_lft 1796sec
    inet6 fd1e:d47e:8ea5:3cab:5055:55ff:fec5:da42/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 2591899sec preferred_lft 604699sec
    inet6 fe80::5055:55ff:fec5:da42/64 scope link
       valid_lft forever preferred_lft forever

jim@colima:~$ ip route
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.1 metric 200
default via 192.168.64.1 dev col0 proto dhcp src 192.168.64.8 metric 300
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.26.0.0/16 dev br-063f6ac19dfc proto kernel scope link src 172.26.0.1
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.1 metric 200
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.1 metric 200
192.168.64.0/24 dev col0 proto kernel scope link src 192.168.64.8 metric 300
192.168.64.1 dev col0 proto dhcp scope link src 192.168.64.8 metric 300

There is a hidden IP address 192.168.5.2 which is actually DNS server both on the macOS host and in the colima VM.

jimgray@colima:~$ ps -eF --forest | grep -i [d]ns
dnsmasq     1277       1  0  3158  2324   0 21:09 ?        00:00:00 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d

jimgray@colima:~$ cat /etc/dnsmasq.d/01-colima.conf
# Generated by Colima
# Do not edit this file manually

address=/host.docker.internal/192.168.5.2
address=/host.lima.internal/192.168.5.2
address=/colima.internal/192.168.5.1
address=/colima/127.0.0.1

server=192.168.5.2

interface=eth0
listen-address=192.168.5.1
bind-interfaces

~ $ sudo lsof -nPR -i tcp:53
COMMAND   PID PPID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 674    1 _mdnsresponder   31u  IPv4 0xd2ba4c44ed2bcfcc      0t0  TCP *:53 (LISTEN)
mDNSRespo 674    1 _mdnsresponder   33u  IPv6 0xd8ba243fde3bf9d9      0t0  TCP *:53 (LISTEN)
~ $ cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver 192.168.5.2

We can access to the SSH port of the colima VM.

~ $ echo | ncat -v 192.168.64.8 22
~ $ echo | ncat -v 192.168.5.2 22

The same user account as the macOS host is created. Attention please; the uid and gid numbers are different.

jim@colima:~$ pwd
/home/jim.linux

jim@colima:~$ id
uid=501(jim) gid=1000(jim) groups=1000(jim),991(docker)

~$ tail -1 /etc/passwd
jim:x:501:1000:Jim Zhan Hu:/home/jim.linux:/bin/bash

The dockerd, containerd, containerd-shim and runc all reside within the VM.

jim@colima:~$ ps -eF --forest | grep [c]ontainerd
root         446       1  0 465190 40872  1 13:58 ?        00:00:11 /usr/bin/containerd
root        1218       1  0 546591 71184  2 13:58 ?        00:00:03 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --host-gateway-ip=192.168.5.2

We check the logs of colima.

# logs of daemons in the VM
jim@colima:~$ sudo journalctl -xfe -u docker
jim@colima:~$ sudo journalctl -xfe -u containerd

# logs of colima
~ $ ls -l $COLIMA_HOME/_lima/colima/*.log
-rw-r--r--  1 jim  staff  12184 Jan  6 20:30 /Users/jim/.colima/_lima/colima/ha.stderr.log
-rw-r--r--  1 jim  staff    167 Jan  6 20:28 /Users/jim/.colima/_lima/colima/ha.stdout.log
-rw-------  1 jim  staff     51 Jan  6 20:28 /Users/jim/.colima/_lima/colima/serialv.log

Data of the colima Engine is stored under ${COLIMA_HOME}/_lima/<hostname>. There are multiple meaningful files colima.yaml (configuration of colima), lima.yaml (configuration of lima) and cloud-config.yaml (user account). Whenever you encounter issues, please inspect these files.

Update colima

When it comes to "update colima", we mean updating either the whole colima or just the Docker runtime part.

To update the Docker runtime part, run colima update -p <profile>.
To update the whole colima (Docker runtime included), refer to Install colima.

Clean colima

Run colima k8s stop and/or colima stop to shutdown the colima instance.

Run colima delete to destroy everything, including containers, images, the ${COLIMA_HOME}/<profile>/<hostname>.yaml config, etc. It simulates a fresh installation.

~ $ colima delete -p default -v
are you sure you want to delete colima and all settings? [y/N]

Since v0.9.0, each container has a separate VM disk for data persistence. The VM disk persists accros colima delete. To remove the VM disk as we pass the --data option.

~ $ colima delete -p default -v --data

colima caches downloaded assets under ~$HOME/Library/Caches/{colima,lima}, we can further prune the cache.

~ $ colima prune -a -v
'/Users/jim/Library/Caches/colima' and '/Users/jim/Library/Caches/lima' will be emptied, are you sure? [y/N] y
INFO[0001] Pruning "/Users/jim/Library/Caches/colima"
INFO[0000] Pruning "/Users/jim/Library/Caches/lima"

Docker CLI Setup

Install Docker CLI

Install Docker CLI. Please do NOT add the --cask CLI option!

~ $ brew install docker

Additionally, we install additional Docker plugins.

~ $ brew install docker-compose docker-buildx

Config Docker CLI

Firstly, let us set the configuration directory via the DOCKER_CONFIG environment variable.

~ $ 17:12:05 jim@Jims-MacBook-Pro ~
$ echo 'export DOCKER_CONFIG=$HOME/.config/docker' >> ~/.bashrc

$ mkdir -p $DOCKER_CONFIG

We configure the Docker CLI let it find the plugins, otherwise docker compose would report 'compose' is not a docker command.

~ $ 17:13:15 jim@Jims-MacBook-Pro ~
$ cat .config/docker/config.json
{
    "auths": {},
    "cliPluginsExtraDirs": [
        "/opt/homebrew/lib/docker/cli-plugins"
    ]
}

Switch Docker Engine

In the Start colima section, the Docker Context and the Docker builder are automatically switched to the colima Engine. But to definitely let Dokcer CLI communicate with the colima Engine, we should explicitly switch to it.

~ $ docker context ls
~ $ docker context use colima
colima
Current context is now "colima"
~ $ less $DOCKER_CONFIG/contexts/meta/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/meta.json

~ $ docker buildx ls
~ $ docker buildx use colima 
~ $ less $DOCKER_CONFIG/buildx/current

This is required if there are multiple engines available on the macOS host.

Verify Docker Setup

Let us start with a simple apline container. From the output, an aarch64 container was created. So, we can say the colima VM is also aarch64. This matches the fact that the macOS is an Apple M1 chip.

~ $ docker run --rm -ti alpine uname -a
Linux 12025f4e1ef2 6.8.0-50-generic #51-Ubuntu SMP PREEMPT_DYNAMIC Sat Nov  9 18:03:35 UTC 2024 aarch64 Linux

Here is an example of a Docker Compose with 7 containers.

~ $ docker compose ls
NAME                STATUS              CONFIG FILES
kong-dev            running(7)          /Users/jim/workspace/biji/archive/kong-dev-compose.yaml

Correspondingly, within the colima engine, there should be 7 containerd-shim-runc processes.

~ $ colima ssh -p default

jim@colima:/Users/jim$ ps -eFH | grep -i [r]unc
root       32809       1  0 309514 14604  3 13:51 ?        00:00:04   /usr/bin/containerd-shim-runc-v2 -namespace moby -id a439998840a3dec6c05c77eba72a7fa2301093e6f166e04a94864a288f91e8cf -address /run/containerd/containerd.sock
root       32886       1  0 309386 13768  3 13:51 ?        00:00:01   /usr/bin/containerd-shim-runc-v2 -namespace moby -id 41227a00a3619d4409a9295112b0b2440227918c96d86809aca4580fe042c714 -address /run/containerd/containerd.sock
root       32925       1  0 309450 13880  1 13:51 ?        00:00:01   /usr/bin/containerd-shim-runc-v2 -namespace moby -id a6770fbd56d55312abfa23b200e57c3c9bd8a7d38c96c95c62aed387882493ea -address /run/containerd/containerd.sock
root       32953       1  0 309514 14640  3 13:51 ?        00:00:04   /usr/bin/containerd-shim-runc-v2 -namespace moby -id 92346757d8056583cac160735a9c6e7b4b802ea161fd722242c3a33ee40b6b06 -address /run/containerd/containerd.sock
root       32961       1  0 309386 14520  0 13:51 ?        00:00:04   /usr/bin/containerd-shim-runc-v2 -namespace moby -id dfee6297c63fac0ddf412dacc925cf4bd7e7508d0cff5be29fd822a9251a84b3 -address /run/containerd/containerd.sock
root       33117       1  0 309450 15132  1 13:51 ?        00:00:04   /usr/bin/containerd-shim-runc-v2 -namespace moby -id 909fed36235360ec776dc52b796bd33cb77b42409200a1415ce9300dd90e9ed4 -address /run/containerd/containerd.sock
root       33691       1  0 309450 13760  2 13:51 ?        00:00:01   /usr/bin/containerd-shim-runc-v2 -namespace moby -id 9a1ae83f551302204f7a2da0fd26ae538afb59e97ece44a0255e44e99ab20e95 -address /run/containerd/containerd.sock

Addtionally, for each container, there are several docker-proxy processes spawned by dockerd when a container is publishing ports, accompanyed by some iptables rules to forward traffic to or from the container within the colima engine.

~ $ colima ssh -p default

jim@colima:/Users/jim$ ps -eFH | grep [1]72.26.0.2
root       32538    1273  0 417873 3712   2 13:51 ?        00:00:00     /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.26.0.2 -container-port 80
root       32567    1273  0 454705 3840   2 13:51 ?        00:00:00     /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32768 -container-ip 172.26.0.2 -container-port 80
root       32586    1273  0 436321 3840   1 13:51 ?        00:00:00     /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32772 -container-ip 172.26.0.2 -container-port 443
root       32616    1273  0 436257 3712   3 13:51 ?        00:00:00     /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32772 -container-ip 172.26.0.2 -container-port 443

~$ sudo iptables -t nat -S
-A POSTROUTING -s 172.26.0.2/32 -d 172.26.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.26.0.2/32 -d 172.26.0.2/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A DOCKER ! -i br-7034e9493ba4 -p tcp -m tcp --dport 32768 -j DNAT --to-destination 172.26.0.2:80
-A DOCKER ! -i br-7034e9493ba4 -p tcp -m tcp --dport 32772 -j DNAT --to-destination 172.26.0.2:443

However, each published port requires a standalone docker-proxy process, consuming quite a lot of memory and CPU! Setting the field userland-proxy to false can solve the performance issue. See section Config colima.

Unix Socket File

From the Start colima section and the Inspect colima section, we know the Docker Unix socket docker.sock is mapped from the colima VM to the macOS host.

+---------------------------------------------------------------------------------------------------------------+
|                                                                                                               |
|                                                                                                               |
|    macOS Host                                                                                                 |
|                                                                                                               |
|                                                                                                               |
|        +-----------------------------------------------+                                                      |
|        |                                               |                                                      |
|        |   colima Engine                               |                                                      |
|        |                                               |                                                      |
|        |                                               |                                                      |
|        |     dockerd containerd containerd-shim runc   |                                                      |
|        |                                               |                                                      |
|        |                                               |                                                      |
|        |         unix:///var/run/docker.sock      -----+---> unix:///Users/jim/.colima/default/docker.sock    |
|        |                                               |                                                      |
|        +-----------------------------------------------+                                                      |
|                                                                                                               |
|                                                                                                               |
+---------------------------------------------------------------------------------------------------------------+

Actually, we can manage the containers directly within the colima VM.

~ $ ssh colima
Last login: Tue Jan 21 22:09:08 2025 from 192.168.5.2

jim@colima:~$ docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT               ERROR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

jim@colima:~$ docker compose ls
NAME                STATUS              CONFIG FILES
kong-dev            running(7)          /Users/jim/workspace/biji/archive/kong-dev-compose.yaml

However, applications on the macOS host can not find the colima engine as they assume the default pathname unix:///var/run/docker.sock. We should configure the DOCKER_HOST Docker environment variable to the colima engine.

Take VSCode for example, we can set the variable in the Docker extension.

For CLI applications, here is an example.

~ $ DOCKER_HOST="unix://${COLIMA_HOME}/default/docker.sock" docker context ls
NAME        DESCRIPTION                               DOCKER ENDPOINT                                     ERROR
colima      colima                                    unix:///Users/jim/.colima/default/docker.sock
default *   Current DOCKER_HOST based configuration   unix:///Users/jim/.colima/default/docker.sock
Warning: DOCKER_HOST environment variable overrides the active context. To use a context, either set the global --context flag, or unset DOCKER_HOST environment variable.

~ $ DOCKER_HOST="unix://${COLIMA_HOME}/default/docker.sock" docker buildx ls
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
colima        docker
 \_ colima     \_ colima        running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386
default*      docker
 \_ default    \_ default       running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386

See FAQ Cannot connect to the Docker daemon.

Multi-platform Build

The default docker build driver prioritizes simplicity but does not support advanced features like multi-platform build, caching, etc.

~ $ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
colima*      docker
 \_ colima    \_ colima        running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386

To support multi-platform build, create a builder with the docker-container build driver that has multi-platform support. This driver would create a dedicated container as the builder backend.

The --use option would switch to the newly created builder, while --bootstrap would start the dedicated container in advanced.

~ $ docker buildx create --use --bootstrap --name multi-platform-builder --node multi-platform-builder --driver docker-container --platform "linux/arm64,linux/amd64"
[+] Building 4.4s (1/1) FINISHED
 => [internal] booting buildkit                                                                                                                                                                                               4.4s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                            4.0s
 => => creating container buildx_buildkit_multi-platform-builder                                                                                                                                                              0.4s
multi-platform-builder

~ $ docker buildx ls
NAME/NODE                    DRIVER/ENDPOINT    STATUS    BUILDKIT   PLATFORMS
multi-platform-builder*      docker-container
 \_ multi-platform-builder    \_ colima         running   v0.18.2    linux/amd64* (+2), linux/arm64*, linux/386
colima                       docker
 \_ colima                    \_ colima         running   v0.17.3    linux/amd64 (+2), linux/arm64, linux/386

~ $ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS          PORTS     NAMES
80b4fa8ad380   moby/buildkit:buildx-stable-1   "buildkitd --allow-i…"   14 seconds ago   Up 14 seconds             buildx_buildkit_multi-platform-builder

Let's build a "linux/amd64" image on the "linux/arm64" machine.

~ $ grep -A1 platforms kong-dev-compose.yaml
72:      platforms:                    # Docker will determine the native platform unless you specify a different value.
73-      - "linux/amd64"               # linux/arm64 | linux/amd64

~ $ $ docker compose -f kong-dev-compose.yaml build kong
[+] Building 0/1s (0/1)                                                                                                                                                                           docker-container:multi-platform-builder
[+] Building 59.2s (7/16)                                                                                                                                                                         docker-container:multi-platform-builder
[+] Building 93.9s (7/16)                                                                                                                                                                         docker-container:multi-platform-builder
 => [kong internal] booting buildkit                                                                                                                                                                                                 3.3s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                                                                   2.9s
 => => creating container buildx_buildkit_multi-platform-builder                                                                                                                                                                     0.4s
 => [kong internal] load build definition from kong-dev-compose.Dockerfile                                                                                                                                                           0.0s
 => => transferring dockerfile: 6.34kB                                                                                                                                                                                               0.0s
 => [kong internal] load metadata for docker.io/library/ubuntu:24.04                                                                                                                                                                 4.3s
 => [kong auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                                                   0.0s
 => [kong internal] load .dockerignore                                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                                      0.0s
 => [kong emmyluadebugger 1/2] FROM docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab                                                                                           1.9s
 => => resolve docker.io/library/ubuntu:24.04@sha256:80dd3c3b9c6cecb9f1667e9290b3bc61b78c2678c02cbdae5f0fea92cc6734ab                                                                                                                0.0s
 => => sha256:8bb55f0677778c3027fcc4253dc452bc9c22de989a696391e739fb1cdbbdb4c2 28.89MB / 28.89MB                                                                                                                                     1.4s
 => => extracting sha256:8bb55f0677778c3027fcc4253dc452bc9c22de989a696391e739fb1cdbbdb4c2                                                                                                                                            0.5s
 => [kong emmyluadebugger 2/2] RUN <<-EOF (set -ex...)                                                                                                                                                                              84.0s
 => => # [ 68%] Building CXX object emmy_debugger/CMakeFiles/emmy_debugger.dir/src/transporter/transporter.cpp.o
 => => # [ 70%] Building CXX object emmy_debugger/CMakeFiles/emmy_debugger.dir/src/debugger/emmy_debugger.cpp.o
 => => # [ 71%] Building CXX object emmy_debugger/CMakeFiles/emmy_debugger.dir/src/debugger/emmy_debugger_manager.cpp.o
 => => # [ 73%] Building CXX object emmy_debugger/CMakeFiles/emmy_debugger.dir/src/debugger/emmy_debugger_lib.cpp.o
 => => # [ 75%] Building CXX object emmy_debugger/CMakeFiles/emmy_debugger.dir/src/debugger/hook_state.cpp.o
 => => # [ 76%] Building CXX object emmy_debugger/CMakeFiles/emmy_debugger.dir/src/debugger/extension_point.cpp.o
 => [kong baseimage  2/11] RUN echo "I am building target platform linux/amd64 on source platform linux/arm64 from base ubuntu:24.04"

...

~ $ docker image inspect 11e108b7f310
...
"Architecture": "amd64",

If we do not have the multi-platform build requirement, then just stick with the default driver for better performance. Read more at multi-platform-docker-build.

SSH Agent Forwarding

In order to reuse the SSH agent on the macOS host, we should complete two settings.

In the section Config clima, we enabled the forwardAgent, --ssh-agent field to forward the SSH agent on the macOS host into the colima VM.

The agent on the macOS host.

~ $ ls -l $SSH_AUTH_SOCK
srw-------  1 jim  staff  0 Feb 28 18:05:19 2024 /var/folders/wc/fnkx5qmx61l_wx5shysmql5r0000gn/T//ssh-lNQDx9E3iDHJ/agent.90068

The forwarded agent in the colima VM.

~ $ ssh colima echo '$SSH_AUTH_SOCK'
/tmp/ssh-MwfhUrSSNj/agent.1141

~ $ colima ssh -p default --very-verbose -- eval ls -l '$SSH_AUTH_SOCK'
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
TRAC[0000] cmd int ["limactl" "shell" "--workdir" "/Users/jim" "colima" "eval" "ls" "-l" "$SSH_AUTH_SOCK"]
srwxrwxr-x 1 jim jim 0 Jan  2 21:11 /tmp/ssh-NY12s8xDgK/agent.1142

Additionally, within the colima VM, a symlink /run/host-services/ssh-auth.sock is created for the forwarded agent.

~ $ colima ssh -p default --very-verbose -- ls -l /run/host-services/ssh-auth.sock
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
TRAC[0000] cmd int ["limactl" "shell" "--workdir" "/Users/jim" "colima" "ls" "-l" "/run/host-services/ssh-auth.sock"]
lrwxrwxrwx 1 jim root 30 Jan  2 21:11 /run/host-services/ssh-auth.sock -> /tmp/ssh-NY12s8xDgK/agent.1142

The VM can resuse the forwarded SSH agent.

~ $ colima ssh -p default -- ssh-add -l
384 SHA256:a2TyZj/tyhwUrs6JGhg//+Zwnpvti1yttted6OgmWtg jim.hu@konghq.com (Kong Dev) (ECDSA)
4096 SHA256:x8gO3GTWyTkJLudIvoKwHZ9Mez0BFktfSc2FZ/jGqPU jim@zhtux (RSA)

Bind mound the forwarded SSH agent to your container.

Containers know nothing about the SSH agent on macOS host, but the one within the colima VM.

~ $ export COLIMA_SSH_AUTH_SOCK=$(ssh colima echo '$SSH_AUTH_SOCK')
# -or-
~ $ export COLIMA_SSH_AUTH_SOCK='/run/host-services/ssh-auth.sock'
   
~ $ docker run --rm --name test-ssh-agent --mount "type=bind,src=$COLIMA_SSH_AUTH_SOCK,dst=$COLIMA_SSH_AUTH_SOCK" -e SSH_AUTH_SOCK=$COLIMA_SSH_AUTH_SOCK nicolaka/netshoot ssh-add -l
384 SHA256:a2TyZj/tyhwUrs6JGhg//+Zwnpvwp9yttted4OgmWtg jim.hu@konghq.com (Kong Dev) (ECDSA)
4096 SHA256:x2gO9GTWyTkJLudIwcFwHZ9Mez0BFktfSc9FZ/jGqPU jim@zhtux (RSA)

If it reports permission error, we should add the write permission chmod a+w to the socket file. See SSH Agent Forwarding.

Mount Volumes

By default, the user home directory and /tmp/colima are bind mounted to the VM.

jim@colima:~$ cat /etc/fstab | grep -E 'colima|Users'
jim@colima:~$ mount | grep -E 'colima|Users'
mount0 on /Users/jim type virtiofs (rw,relatime)
mount1 on /tmp/colima type virtiofs (rw,relatime)

Occasionally, the /tmp/colima mount was borken. We need to manually restart colima.

Since v0.9.0, the /tmp/colima mount is removed

To mount other pathnames from macOS host to containers, we need firstly mount their parent directories to the colima VM! Otherwise, we would receive errors as follows.

otel-collector-1  | 2025-01-07T07:42:50.770341478Z Error: cannot start pipelines: open /tmp/file_exporter.json: is a directory
otel-collector-1  | 2025-01-07T07:42:50.770344603Z 2025/01/07 07:42:50 collector server run finished with error: cannot start pipelines: open /tmp/file_exporter.json: is a directory

Check the mounts field in the config file.

Kubernetes Setup

Regarding how to play with the Kubernetes environment, refer to "biji".

Enable Kubernetes

colima, under the hood, supports Kubernetes via K3s which is a lightweight Kubernetes distribution with small memory and CPU footprint - only 50% percent.

To create a K8s cluster, we can add the 'kubernetes' config field.

~ $ colima template -p default --editor nano
>kubernetes:
>  enabled: true

As K3s server would consume extra CPU and memory. Please create a separate profile for K8s specifically and start it on demand. Alternatively, we can specifiy the --kubernetes=true or --kubernetes=false CLI option. However, the --kubernetes CLI option would overwrite the colima config so that we have to always provide this CLI option.

# enable
~ $ colima start -p default --kubernetes=true

# disable
~ $ colima start -p default --kubernetes=false

Fortunately, colima supports dynamically start/stop the K8s cluster with the kubernetes, k8s sub-command! After we stop the cluster, remember to run docker container prune to remove the associated dangling containers.

~ $ colima status -p default
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/jim/.colima/default/docker.sock
INFO[0000] kubernetes: enabled

~ $ colima k8s stop -p default

~ $ colima status -p default
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/jim/.colima/default/docker.sock

We can customize K3s via the 'k3sArgs' config field. By default, --disable=traefik (Ingress controller) is passed to the k3s server process.

According to the FAQ, colima does not support multiple K8s clusters! But we can run Minikube, Kind or K3d (preferred) with colima as the backend. Alternatively, we can create multiple colima instances to simulate multiple K8s clusters.

Install kubectl

~ $ brew install kubectl

~ $ kubectl version
Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.31.2+k3s1

Config kubectl

The kubectl load config from kubeconfig files --kubeconfig, $KUBECONFIG or ${HOME}/.kube/config in the order of decreasing priority.

~ $ kubectl options

~ $ cat ~/.kube/config

~ $ kubectl config view [--minify]
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:62005
  name: colima
contexts:
- context:
    cluster: colima
    user: colima
  name: colima
current-context: colima
kind: Config
preferences: {}
users:
- name: colima
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

Inspect Kubernetes

The colima K8s cluster is created.

~ $ kubectl config get-clusters
NAME
colima

~ $ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:62005
CoreDNS is running at https://127.0.0.1:62005/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:62005/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

The clima node is created.

~ $ kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
colima   Ready    control-plane,master   14d   v1.31.2+k3s1   192.168.5.1   <none>        Ubuntu 24.04.1 LTS   6.8.0-50-generic   docker://27.4.0

The colima K8s context is created. Generally speaking, a K8s context is a combination of a cluster, a user (auth) and an optional namespace. Similar to Docker Context, we can switch between different K8s contexts by kubectl config use-context.

~ $ kubectl config current-context
colima

~ $ kubectl config get-contexts
CURRENT   NAME     CLUSTER   AUTHINFO   NAMESPACE
*         colima   colima    colima

Within the colima VM, a K3s server process is spawned.

jim@colima:~$ ps -eFF | grep -i [k]3s
root        1794       1 14 1484594 481076 0 20:30 ?       00:05:28 /usr/local/bin/k3s server

jim@colima:~$ sudo journalctl -efx -u k3s.service

We find 3 Docker containers created for different Kubernetes internal components (e.g. CoreDNS Cluster DNS). You will find each container has an accompanying pause container.

# on macOS host
~ $ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS          PORTS     NAMES
b256190c220b   2f6c962e7b83                 "/coredns -conf /etc…"   50 minutes ago   Up 50 minutes             k8s_coredns_coredns-56f6fc8fd7-jqg9q_kube-system_ba370ccf-ea61-4357-a537-bf57ad963b09_0
be7f4f0b1fe3   d3dd7baae2fc                 "local-path-provisio…"   50 minutes ago   Up 50 minutes             k8s_local-path-provisioner_local-path-provisioner-5cf85fd84d-kh7gz_kube-system_7d860118-ebae-4964-a475-8df63e95533e_0
6a1a5c8f7902   5548a49bb60b                 "/metrics-server --c…"   50 minutes ago   Up 50 minutes             k8s_metrics-server_metrics-server-5985cbc9d7-pqbql_kube-system_917fc3ca-e55c-4a8b-9732-3a010a337a16_0
a6d8cade10b8   rancher/mirrored-pause:3.6   "/pause"                 50 minutes ago   Up 50 minutes             k8s_POD_metrics-server-5985cbc9d7-pqbql_kube-system_917fc3ca-e55c-4a8b-9732-3a010a337a16_0
175072564faf   rancher/mirrored-pause:3.6   "/pause"                 50 minutes ago   Up 50 minutes             k8s_POD_coredns-56f6fc8fd7-jqg9q_kube-system_ba370ccf-ea61-4357-a537-bf57ad963b09_0
96a19292498d   rancher/mirrored-pause:3.6   "/pause"                 50 minutes ago   Up 50 minutes             k8s_POD_local-path-provisioner-5cf85fd84d-kh7gz_kube-system_7d860118-ebae-4964-a475-8df63e95533e_0

# within colima VM
jim@colima:~$ ps -eF --forest | grep -A1 [c]ontainerd-shim-runc-v2
root        2444       1  0 309386 13576  2 20:30 ?        00:00:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 96a19292498d036eb229cafba53fab0f5f61736b37c1f7d0379f68b978a38e75 -address /run/containerd/containerd.sock
65535       2509    2444  0   190   384   0 20:30 ?        00:00:00  \_ /pause
root        2445       1  0 309450 13528  2 20:30 ?        00:00:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id a6d8cade10b8e260d6d0217036222e081f646648f7cfe28a14ff988d32942fa9 -address /run/containerd/containerd.sock
65535       2508    2445  0   190   384   0 20:30 ?        00:00:00  \_ /pause
root        2446       1  0 309450 13492  2 20:30 ?        00:00:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 175072564faf2eb02f25385fa8e3fc8615c54999e22fdc4a389c2ad9ca0b17ec -address /run/containerd/containerd.sock
65535       2507    2446  0   190   384   1 20:30 ?        00:00:00  \_ /pause
root        2711       1  0 309450 13896  2 20:30 ?        00:00:03 /usr/bin/containerd-shim-runc-v2 -namespace moby -id be7f4f0b1fe33a4fb2b38d0c5cf7e45a654aef517824652efcd439ea6e360684 -address /run/containerd/containerd.sock
root        2770    2711  0 316386 33920  1 20:30 ?        00:00:02  \_ local-path-provisioner start --config /etc/config/config.json
root        2731       1  0 309450 14292  2 20:30 ?        00:00:03 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 6a1a5c8f790287a28c6d0650096f8caf23815d7c8ba984bf7b7bd4a9dae506f8 -address /run/containerd/containerd.sock
1000        2802    2731  1 320556 59672  0 20:30 ?        00:00:33  \_ /metrics-server --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
root        2751       1  0 309450 13928  2 20:30 ?        00:00:03 /usr/bin/containerd-shim-runc-v2 -namespace moby -id b256190c220bb96392fee9100b55b311900e88ab79acd1ba3f890e5b21a2b8a1 -address /run/containerd/containerd.sock
65532       2795    2751  0 321216 52356  3 20:30 ?        00:00:17  \_ /coredns -conf /etc/coredns/Corefile

Correspondingly, we find the associated 3 deployments and services.

~ $ $ kubectl get deployments -A
NAMESPACE     NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns                  1/1     1            1           14d
kube-system   local-path-provisioner   1/1     1            1           14d
kube-system   metrics-server           1/1     1            1           14d

~ $ kubectl get services -A
NAMESPACE     NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes       ClusterIP   10.43.0.1      <none>        443/TCP                  14d
kube-system   kube-dns         ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP   14d
kube-system   metrics-server   ClusterIP   10.43.39.244   <none>        443/TCP                  14d

Especially, with the default runtime, colima and K3s share container images!

~ $ docker images 'rancher/*'
REPOSITORY                         TAG                    IMAGE ID       CREATED         SIZE
rancher/klipper-helm               v0.9.3-build20241008   128d0eddd2c8   2 months ago    188MB
rancher/local-path-provisioner     v0.0.30                d3dd7baae2fc   3 months ago    51.7MB
rancher/mirrored-library-traefik   2.11.10                43b65488db50   3 months ago    165MB
rancher/mirrored-metrics-server    v0.7.2                 5548a49bb60b   4 months ago    65.5MB
rancher/mirrored-coredns-coredns   1.11.3                 2f6c962e7b83   5 months ago    60.2MB
rancher/klipper-lb                 v0.4.9                 8e8709f8caae   5 months ago    20MB
rancher/mirrored-library-busybox   1.36.1                 7db2ddde018a   19 months ago   4.04MB
rancher/mirrored-pause             3.6                    7d46a07936af   3 years ago     484kB

We also found a Docker network is created.

~ $ docker network ls
NETWORK ID     NAME              DRIVER    SCOPE
7be6cae2031a   k3d-k3s-default   bridge    local

EXTREME CAUTION

As the Docker context and the K8s context share the same colima engine, please pay extreme caution to destruction operations like prune, rm, stop, kill, etc.

References

colima FAQ.

Nginx SSLKEYLOGFILE

2024-12-23T00:00:00+00:00

SSLKEYLOGFILE
sslkeylog.c
References

In this post, I will walk you through a step-by-step tutorial on how to decrypt TLS traffic of Nginx and its variations.

SSLKEYLOGFILE

Environment variable "SSLKEYLOGFILE" points to a pathname storing the TLS (Pre)MasterSecret (Session Secret) that can be used to decrypt TLS traffic when combined with the captured TLS handshake information.

Generally speaking, there are three ways to support "SSLKEYLOGFILE".

Capture TLS traffic on client side.

Clients like mitmproxy (TLS proxy), s_client -keylogfile, cURL, Firefox, Chrome, etc. all natively support "SSLKEYLOGFILE". So, we just need to set this variable before launching the client and before capturing the TLS traffic.
Capture TLS traffic on Nginx side.

Nginx by default does not provide the ability to dump the TLS (Pre)MasterSecret. We have to hack the OpenSSL library.
1. Make use of 3rd-party Nginx modules like nginx-sslkeylog. There is another similar patch available at PATCH SSL: Added SSLKEYLOGFILE key material to debug logging.
2. Preload sslkeylog.c. Compared to method 2.1, it does not require patching and building. It uses LD_PRELOAD to preload a newly built shared library that could dump TLS (Pre)MasterSecret if "SSLKEYLOGFILE" is present.
Make use of GDB to read TLS (Pre)MasterSecret from core dump.
OpenSSL 3.5 onwards support "SSLKEYLOGFILE" via the enable-sslkeylog configure option!

This post focuses on the method 2.2.

sslkeylog.c

In this section, I will build a shared library "libsslkeylog.so" from GitHub repo sslkeylog.c. The shared library can dump TLS (Pre)MasterSecret if it is preloaded to Nginx and the environment variable "SSLKEYLOGFILE" is imported into "nginx.conf".

I will use Nginx's derivation Kong Gateway and Docker image to demonstrate the details.

Compose Dockerfile

The Dockerfile is available at sslkeylogfile.Dockerfile. In the build stage, the script sslkeylog.sh is used to verify the shared library "libsslkeylog.so".

./sslkeylog.sh curl -sI https://www.google.com

The test output from the building process is as follows.

#7 1.555 + ./sslkeylog.sh curl -sI https://www.google.com
#7 1.757 SERVER_HANDSHAKE_TRAFFIC_SECRET e4d298a7b78a9366948f621f7898b69e09b209144647736892987b7e78f869fe a89651be69b7df70508c77d80f5b518b4a3f337168bafe686710492e91edbca484593fb6fa8d7ad268359d14f9c9fb6c
#7 1.799 EXPORTER_SECRET e4d298a7b78a9366948f621f7898b69e09b209144647736892987b7e78f869fe e37bf351ac25efcbe23d444d9aec1449f16d2aceb69b582ec9623b17343bc3f32161a4f2979694890a141ad11e78cfdf
#7 1.799 SERVER_TRAFFIC_SECRET_0 e4d298a7b78a9366948f621f7898b69e09b209144647736892987b7e78f869fe f8035f7c40f19398f09a0159e4c20d868bb688ef2a176fac4f5312458b7faadf4937c2111450995f2597bff706686fd3
#7 1.799 CLIENT_HANDSHAKE_TRAFFIC_SECRET e4d298a7b78a9366948f621f7898b69e09b209144647736892987b7e78f869fe 5a37d7ae206380701b6d6ab5c12ddfe3a35def0acd7e4497e8dfe926ce19a30ed978d34d3de3530b771efdba403e11c9
#7 1.799 CLIENT_TRAFFIC_SECRET_0 e4d298a7b78a9366948f621f7898b69e09b209144647736892987b7e78f869fe 59c8486ac85d3172b21bf18834b49a606616bfa8a881afd5b25287a171f9fd6a394506cc625172af769fdbd7ff88d6cd
#7 1.899 HTTP/2 200
#7 1.899 content-type: text/html; charset=ISO-8859-1
#7 1.899 content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-1uCcl_kqI4AgRQc8nm8x8A' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
#7 1.899 accept-ch: Sec-CH-Prefers-Color-Scheme
#7 1.899 p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
#7 1.899 date: Tue, 24 Dec 2024 03:25:09 GMT
#7 1.899 server: gws
#7 1.899 x-xss-protection: 0
#7 1.899 x-frame-options: SAMEORIGIN
#7 1.899 expires: Tue, 24 Dec 2024 03:25:09 GMT
#7 1.899 cache-control: private
#7 1.899 set-cookie: AEC=AZ6Zc-VxxkKIvZWzk6Etrr2TaoT-NbBl22ugcfxfOObaAIFDKw3slf3ZTA; expires=Sun, 22-Jun-2025 03:25:09 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
#7 1.899 set-cookie: NID=520=gulgYq_BY9y4P9xdlBnruMrbIl078sk3z4lF_A6bXpsvDoA5F6zzsEQLy1dSQOswGfITfIequq5TiTQ6nud7F4PbPXmDHRsTC5TUIUjbpZaX-33Wn8Ebj0xtT_p5zz-p-TofTwPaFTkfRWz2yOrUZdDIbXw-fxZrCkLPWUf400FskSA7Ef9rLsEi47NCb-FS; expires=Wed, 25-Jun-2025 03:25:09 GMT; path=/; domain=.google.com; HttpOnly
#7 1.899 alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
#7 1.899

We will cover more details on the verification script laster on.

Build Custom Image

~ $ BUILDKIT_PROGRESS=plain docker buildx build --no-cache --load -t 'kong-custom:3.9.0.0' -f sslkeylogfile.Dockerfile .

~ $ docker images 'kong-custom'
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
kong-custom   3.9.0.0   b85219c906ce   34 minutes ago   498MB

Decrypt cURL Traffic

Create a container against the custom image and locate the shared library "libsslkeylog.so".

~ $ docker run --rm \
--name sslkeylogfile \
--entrypoint /bin/bash \
-it kong-custom:3.9.0.0

kong@72a118f01128:/$ kong version
Kong Enterprise 3.9.0.0

kong@72a118f01128:/usr/local/kong/lib$ ls -l libsslkeylog.so
-rwxr-xr-x 1 root root 71736 Dec 24 03:25 libsslkeylog.so

Copy the verification script sslkeylog.sh to the directory of the shared library - /usr/local/kong/lib. You can optionally provide "SSLKEYLOGFILE" to specify a pathname, otherwise the dumped (Pre)MasterSecret is printed to the stderr.

kong@72a118f01128:/usr/local/kong/lib$ pwd
/usr/local/kong/lib

kong@72a118f01128:/usr/local/kong/lib$ nano -w sslkeylog.sh
...

kong@72a118f01128:/usr/local/kong/lib$ chmod +x sslkeylog.sh

As stated above, cURL native supports "SSLKEYLOGFILE", without the help of the newly built shared library "libsslkeylog.so". However, the shared library can also dump TLS (Pre)MasterSecret if it is preloaded and "SSLKEYLOGFILE" is set. Hence, we use cURL to verify the shared library works as expected.

Decrypt cURL TLS 1.3

Start traffic capturing.

~ $ docker run --rm --name testsslkeylogfile -it --network container:sslkeylogfile --mount type=bind,src="${HOME}/misc",dst=/tcpdump -d nicolaka/netshoot
8ae6e4e769ae4bb6ecf705f6c96de9f78f74971ce7ec29c329e0516b12af8469

~ $ docker exec -it testsslkeylogfile bash
72a118f01128:~# cd /tcpdump/
72a118f01128:/tcpdump# tcpdump -i any port 443 -Uw sslkeylogfile.pcap
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
^C27 packets captured
27 packets received by filter
0 packets dropped by kernel
72a118f01128:/tcpdump# ls -l sslkeylogfile.pcap
-rw-r--r--    1 root     root          8938 Dec 24 06:47 sslkeylogfile.pcap

Trigger the request.

kong@72a118f01128:/usr/local/kong/lib$ :> /tmp/premaster.txt

kong@72a118f01128:/usr/local/kong/lib$ SSLKEYLOGFILE=/tmp/premaster.txt ./sslkeylog.sh /usr/local/kong-tools/bin/curl -sI --tlsv1.3 https://www.google.com
HTTP/2 200
content-type: text/html; charset=ISO-8859-1
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-DUWj1U9UPGVMwCY29p_R8Q' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
accept-ch: Sec-CH-Prefers-Color-Scheme
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
date: Tue, 24 Dec 2024 06:30:07 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Tue, 24 Dec 2024 06:30:07 GMT
cache-control: private
set-cookie: AEC=AZ6Zc-Xnia3ReOwqLapj8yFlujpG9zCCT-Rq3eBye07oMwzr-WLr1OnluLA; expires=Sun, 22-Jun-2025 06:30:07 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: NID=520=aAOm9hYIhcaWIbS4Mfxf3juDrMpiQJUc8Zs0TTLliXo1wgOdaqmqfaHsrSWvRU8gWzmrR0gwqxBySnfQHrgt7_aS8lIiu1yYnU06APwBYMcv6kFvmCR6w1H8100azBwocZ4ScXG6-NqPj8T13hkMZNn-1Noi4TCdXRfzIZZqsygShH_hQ50vnjGiNdjOkeE; expires=Wed, 25-Jun-2025 06:30:07 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Each TLS 1.3 session needs 5 session secrets to decrypt traffic.

kong@72a118f01128:/usr/local/kong/lib$ cat /tmp/premaster.txt
# SSL key logfile generated by sslkeylog.c
SERVER_HANDSHAKE_TRAFFIC_SECRET 6755ae538171e4100113858b989890d9ddbbb80a5f71180f8c45c5d29220d538 bc84ec978a446acf0b0da5a674b236b75d73ba0bcb949ebb8c73edf127987b8fec8af48c24e4995485c4132ae865ad8e
EXPORTER_SECRET 6755ae538171e4100113858b989890d9ddbbb80a5f71180f8c45c5d29220d538 9b14d21410073a8d7f9f1e8b6885d8240b7cfbf723a709fa538382fd4a4a01d5f8c70752817a5a4b7db5626d8b504e6f
SERVER_TRAFFIC_SECRET_0 6755ae538171e4100113858b989890d9ddbbb80a5f71180f8c45c5d29220d538 c5a31a6c65853d4386c31d2916bfb6e8a4bd900c3195ecb7daf41e613346cf24fdf3bbeca05b4e02f8c898c942bf0d9f
CLIENT_HANDSHAKE_TRAFFIC_SECRET 6755ae538171e4100113858b989890d9ddbbb80a5f71180f8c45c5d29220d538 c05914e620691f1020342c45e1177f113f88281a90885c40f5e5e6f5d102774b0b362c399a589be9174699f7cd63d3e5
CLIENT_TRAFFIC_SECRET_0 6755ae538171e4100113858b989890d9ddbbb80a5f71180f8c45c5d29220d538 1848c3e4cd96112141cafd0a89212e839a258be5b409701ca4e3fe6531e08448de2ea9f95b293a4c0eb091e12446050d

Let's import the dumped (Pre)MasterSecret into the PCAP file.

14:50:44 jim@Jims-MacBook-Pro ~/misc
$ ls -l sslkeylogfile.pcap
-rw-r--r--  1 jim  staff  8938 Dec 24 14:47 sslkeylogfile.pcap

$ docker cp sslkey:/tmp/premaster.txt .
Successfully copied 2.56kB to /Users/jim/misc/.

$ ls -l premaster.txt
-rw-r--r--  1 jim  staff  981 Dec 24 14:47 premaster.txt

$ editcap --inject-secrets tls,premaster.txt sslkeylogfile.pcap sslkeylogfile-dsb.pcap

$ ls -l sslkeylogfile-dsb.pcap
-rw-r--r--  1 jim  staff  10472 Dec 24 15:05 sslkeylogfile-dsb.pcap

Use Wireshark to decrypt and analyze the TLS traffic.

Decrypt cURL TLS 1.2

Decrypting traffic of TLS 1.2 is almost the same as decrypting the traffic of TLS 1.3. I will leave this part to you.

kong@72a118f01128:/usr/local/kong/lib$ :> /tmp/premaster.txt

kong@72a118f01128:/usr/local/kong/lib$ SSLKEYLOGFILE=/tmp/premaster.txt ./sslkeylog.sh /usr/local/kong-tools/bin/curl -sI --tlsv1.2 --tls-max 1.2 https://www.google.com
HTTP/2 200
content-type: text/html; charset=ISO-8859-1
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-zXzn-lr53oUa0iPPjL-jYg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
accept-ch: Sec-CH-Prefers-Color-Scheme
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
date: Tue, 24 Dec 2024 06:35:02 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Tue, 24 Dec 2024 06:35:02 GMT
cache-control: private
set-cookie: AEC=AZ6Zc-XYq2E4RNGSN48i3AiXeBdwUJJsDimdVa_UQSnVCt2Rx6fqu1j1Lg; expires=Sun, 22-Jun-2025 06:35:02 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: NID=520=Vn8bZrsr2eiT_doMDp6KS8Jvo5lXFMX2z-acFRLxWxwYuGfnFWKwdmd8Y8eZV-kgphfnmPqVjW-0689KjGfUmcZVSHsuz-JMFKxN3CIbczHD2VIJqywOBeIZv3ef5IHr0MdcKNvJffcra9mp-X3QFolGZUylIbjkRybTgY14OIUVfZ2pr3tXSAKVKGk2OCE; expires=Wed, 25-Jun-2025 06:35:02 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Pay attention that each TLS 1.2 session needs just 1 session secret to decrypt traffic.

kong@72a118f01128:/usr/local/kong/lib$ cat /tmp/premaster.txt
# SSL key logfile generated by sslkeylog.c
CLIENT_RANDOM 373ad1cfde82cc6ac98ed54f3b5dbfda9eabee9fe49886b1d3c040aa7bef7ddd 6f526e02d325fbaf0e5406ba6f644ba5c8d5a3c8d2fb534913ba47d9ed8f97f5fff36a058f7625cd974d3fea508d2d43

Decrypt Kong Traffic

Before, we start Kong, we need three special settings.

Set the environment variable "SSLKEYLOGFILE".

~ $ export SSLKEYLOGFILE=/usr/local/kong/pre-mastersecret.txt

As we export, it affects all child processes launched in this terminal. Avoid this by prepending the variable to "kong start".

~ $ env SSLKEYLOGFILE=/usr/local/kong/pre-mastersecret.txt kong start

If Kong is deployed with Helm Charts, set it in the Helm values.

customEnv:
  SSLKEYLOGFILE: /usr/local/kong/pre-mastersecret.txt

If Kong is managed by systemd, then insert the the variable to "kong-enterprise-edition.service".

~ $ sudo systemctl edit kong-enterprise-edition.service
### Editing /etc/systemd/system/kong-enterprise-edition.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file
...
Environment=SSLKEYLOGFILE=/usr/local/kong/pre-mastersecret.txt
...

~ $ systemctl daemon-reload

Let Kong inherit the environment variable by inserting env SSLKEYLOGFILE; into "nginx.conf". It is a must as Nginx removes all environment variables inherited from its parent process except the "TZ" variable.

~ $ export KONG_NGINX_MAIN_ENV=SSLKEYLOGFILE
# -or-
~ $ env KONG_NGINX_MAIN_ENV=SSLKEYLOGFILE kong start

If Kong is deployed with Helm Charts, configure it in the Helm values.

env:
  nginx_main_env: SSLKEYLOGFILE

If Kong is managed by systemd, then insert the the variable to "kong-enterprise-edition.service".

~ $ sudo systemctl edit kong-enterprise-edition.service
### Editing /etc/systemd/system/kong-enterprise-edition.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file
...
Environment=KONG_NGINX_MAIN_ENV=SSLKEYLOGFILE
...

~ $ systemctl daemon-reload

Preload the shared library "libsslkeylog.so" for Kong.

~ $ export LD_PRELOAD=/usr/local/kong/lib/libsslkeylog.so

As we export, it affects all child processes launched in this terminal. Avoid this by prepending the variable to "kong start", or use the dynamic loader ld.so or ld-linux.so with the option --preload against "nginx".

~ $ LD_DEBUG="all,unused,statistics" LD_PRELOAD=/usr/local/kong/lib/libsslkeylog.so \
kong start

~ $ LD_DEBUG="all,unused,statistics" ld.so --preload "/usr/local/kong/lib/libsslkeylog.so" \
/usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong/ -c nginx.conf

If Kong is deployed with Helm Charts, do it in the Helm values.

customEnv:
  LD_PRELOAD: /usr/local/kong/lib/libsslkeylog.so

If Kong is managed by systemd, then insert the the variable to "kong-enterprise-edition.service".

~ $ sudo systemctl edit kong-enterprise-edition.service
### Editing /etc/systemd/system/kong-enterprise-edition.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file
...
Environment=LD_PRELOAD=/usr/local/kong/lib/libsslkeylog.so
...

~ $ systemctl daemon-reload

Create Kong Container

For a demostration purpose, I will start Kong in DB-less mode. Pay attention to the three special settings.

~ $ docker run --rm \
--name sslkeylogfile \
-e "KONG_DATABASE=off" \
-e "KONG_LICENSE_DATA=${KONG_LICENSE_DATA}" \
-e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 http2 ssl" \
-p 8000-8001:8000-8001 -p 8443:8443 -d \
\
-e "SSLKEYLOGFILE=${SSLKEYLOGFILE:-/usr/local/kong/pre-mastersecret.txt}" \
-e "KONG_NGINX_MAIN_ENV=SSLKEYLOGFILE" \
-e "LD_PRELOAD=/usr/local/kong/lib/libsslkeylog.so" \
\
kong-custom:3.9.0.0

~ $ docker ps -f 'name=sslkeylogfile'
CONTAINER ID   IMAGE                 COMMAND                  CREATED              STATUS                        PORTS                                                            NAMES
bcf77d685ed8   kong-custom:3.9.0.0   "/entrypoint.sh kong…"   About a minute ago   Up About a minute (healthy)   8002-8004/tcp, 0.0.0.0:8000-8001->8000-8001/tcp, 8443-8447/tcp   sslkeylogfile

~ $ docker exec -it sslkeylogfile bash
kong@57d808a5eb3f:/$ ps -eFH | grep [n]ginx
kong         1     0  1 134701 98936  1 08:38 ?        00:00:01 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
kong      2549     1  1 669837 129140 2 08:38 ?        00:00:01   nginx: worker process
...

Check if the three special settings are correctly configured.

kong@3ce3b73ed92d:/$ env | grep 'SSLKEYLOGFILE\|KONG_NGINX_MAIN_ENV\|LD_PRELOAD'
SSLKEYLOGFILE=/usr/local/kong/pre-mastersecret.txt
KONG_NGINX_MAIN_ENV=SSLKEYLOGFILE
LD_PRELOAD=/usr/local/kong/lib/libsslkeylog.so

kong@57d808a5eb3f:/$ grep -F env /usr/local/kong/nginx.conf
env SSLKEYLOGFILE;

Check if Kong correctly preloads the shared library.

kong@57d808a5eb3f:/$ ls -l /usr/local/kong/lib/libsslkeylog.so
-rwxr-xr-x 1 root root 71736 Dec 24 03:25 /usr/local/kong/lib/libsslkeylog.so

kong@3ce3b73ed92d:/$ lsof -nP -a -p 1,2550 | grep ssl
nginx      1 kong  mem       REG              254,1             2635191 /usr/local/kong/lib/libssl.so.3 (path dev=0,216)
nginx      1 kong  mem       REG              254,1             2642476 /usr/local/kong/lib/libsslkeylog.so (path dev=0,216)
nginx   2550 kong  mem       REG              254,1             2635191 /usr/local/kong/lib/libssl.so.3 (path dev=0,216)
nginx   2550 kong  mem       REG              254,1             2642476 /usr/local/kong/lib/libsslkeylog.so (path dev=0,216)

# even affect "ls" as the variable LD_PRELOAD is exported
kong@3ce3b73ed92d:/$ ldd /usr/bin/ls | grep ssl
  /usr/local/kong/lib/libsslkeylog.so (0x0000ffffa1420000)

Load the sample runtime declarative config to Kong.

~ $ curl -sS localhost:8001/ | jq -r .configuration.database
off

~ $ curl -sS 'http://localhost:8001/config?check_hash=1&flatten_errors=1' --data-binary '@sslkeylogfile.yml' | jq -r .

~ $ curl -sS http://localhost:8001/config | jq -r .config

Decrypt Kong TLS 1.3

The process is almost the same as that in Decrypt cURL TLS 1.3.

Start traffic capturing.

~ $ docker run --rm --name testsslkeylogfile -it --network container:sslkeylogfile --mount type=bind,src="${HOME}/misc",dst=/tcpdump -d nicolaka/netshoot
~ $ docker exec -it testsslkeylogfile bash

3ce3b73ed92d:~# cd /tcpdump/
3ce3b73ed92d:/tcpdump# tcpdump -i any port 8443 or port 443 -Uw sslkeylogfile.pcap
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
^C37 packets captured
62 packets received by filter
0 packets dropped by kernel

Trigger the request.

~ $ docker exec -it sslkeylogfile bash

kong@3ce3b73ed92d:/$ /usr/local/kong-tools/bin/curl -ksI --tlsv1.3 https://localhost:8443/sslkeylogfile
HTTP/2 200
content-type: text/html; charset=ISO-8859-1
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-G9tWrQ1duP-4WMfkwjlddA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
accept-ch: Sec-CH-Prefers-Color-Scheme
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
date: Tue, 24 Dec 2024 10:06:07 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Tue, 24 Dec 2024 10:06:07 GMT
cache-control: private
set-cookie: AEC=AZ6Zc-Xf9dWoag6LIHV2mWOAVhSHlj1p3KOjwiQY-m0IiYgyatxWDPvZKQ; expires=Sun, 22-Jun-2025 10:06:07 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: NID=520=HL3HxiTphQw9L0EG5eWpwW51t-egXIhozTYSFQf8bHQF_nt8JV93JiJjR8N8JrPrdOjYHrFTDpEKNgdKr7gvA8vE1zr99-73Egy7JBex_SX_BDbntCBGkKUrZ8pbtKIMg0BajyqcQoAtpzWXFLEcwAmuxnx7IZ1XnDFSKF_MriNXtJUO0ZgiRvr8VTtlYlY; expires=Wed, 25-Jun-2025 10:06:07 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
x-kong-upstream-latency: 321
x-kong-proxy-latency: 9
via: 1.1 kong/3.9.0.0-enterprise-edition
x-kong-request-id: 6728f746f230033fab49d41f197f4258

Let's import the dumped (Pre)MasterSecret into the PCAP file.

~ $ docker cp sslkeylogfile:/usr/local/kong/pre-mastersecret.txt .
Successfully copied 4.61kB to /Users/zachary/misc/.

~ $ editcap --inject-secrets tls,pre-mastersecret.txt sslkeylogfile.pcap sslkeylogfile-dsb.pcap

Use Wireshark to decrypt and analyze the TLS traffic.

References

golang

2024-11-12T00:00:00+00:00

Installation
1. macOS
GOPATH

The terminologies "Go" and "golang" might be used interchangeably throughout this post.

Installation

macOS

We can install golang via brew or officially built binary package.

To install or upgrade golang with brew.

~ $ brew install go|golang

~ $ type go
go is hashed (/opt/homebrew/bin/go)
~ $ go version
~ $ go env

~ $ brew upgrade go|golang

For the second method, just download and click the binary (e.g. go1.23.3.darwin-arm64.tar.gz). It will prompts us to install golang in the system directory /usr/local/go, for all users. For update, just repeat the same process.

~ $ type go
go is hashed (/usr/local/go/bin/go)

GOPATH

The variable GOPATH is a pathname for installing extra modules beyond the official ones, usually residing in the user's home. It is also the place where we develop and test personal modules.

~ $ go env GOPATH
/Users/zachary/go

~ $ echo $GOPATH

~ $ ls go/bin/
go-wrk

Interestingly, it is not a Shell environment variable, but exclusive to golang! Upon installation, the golang toolchain would automatically configure it, including the GOPATH/bin. We can manually configure it for Shell reference.

# ~/.bash_profile

export GOPATH="${HOME}/go"
PATH="${GOPATH}/bin:$PATH"

Actually, golang maintains its own environment variables via go env. These variables are stored at $HOME/.config/go/env (Linux) or $HOME/Library/Application Support/go/env (macOS).

Here is an example.

~ $ go env GOPRIVATE

~ $ go env -w GOPRIVATE='github.com/kong/,github.com/Kong/'

~ $ cat ~/Library/Application\ Support/go/env
GOPRIVATE=github.com/kong/,github.com/Kong/

GDB and Core Dump

2024-07-25T00:00:00+00:00

This is a tutorial on dynamically analyzing and exploiting objects with GDB (GNU DeBugger). Without explicit notice, the object type is Core Dump. The term "object" and the term "executable" are used interchangably in this post.

Please playaround with the code in the Appendix section.

Limitations
GDB Extensions
GDB Configuration
Objects with Debug Info
Core Dump
1. Verify Core Dump
Quickstart
GDB Commands
More Tools
1. hexdump
2. strings
3. readelf
4. objdump
Memory
Nginx
Appendix

Limitations

GDB currently does not support Apple Arm chipset (e.g. M1). To bypass this blocker, we can run GDB within Docker container on M1.

However, attaching a tracee process (e.g. Nginx worker) to the gcore or gdb tracer process within containerized environment is limited by the kernel.yama.ptrace_scope kernel parameter. To resolve this limitation, we have two solutions as follows.

For a vanilla Docker container, we should add the --privileged option. Alternatively, add the --cap_add SYS_PTRACE option and run gcore or gdb with sudo.
For K8s deployment, refer to the Generate core dump within K8s Environment tutorial.

GDB Extensions

gef (GDB Ehanced Features) is a Python plugin for vanilla GDB, supporting both x86 (32/64) and Arm (AArch 32/64). It comes with just a single Python script file and requires no dependencies except Python. Howeveer, some gef commands depends on a few other tools like file, readelf, nm, and ps.

Its successor pwndbg is more actively maintained. peda, on the other hand, is almost deprecated, and the last commit is from 4 years ago.

For any extension, we can load it via GDB configuration at startup, or load it interactively after startup.

gef

Before using gef, ensure the following tools are available in your system.

file
readelf
nm
ps
python3

Additionally, ensure the locale is set to UTF-8, otherwise gef reports "UnicodeEncodeError". See https://github.com/hugsy/gef/issues/195.

We can download gef from the official repo.

ubuntu@ip-172-31-9-194:~/workspace/ git clone https://github.com/hugsy/gef

ubuntu@ip-172-31-9-194:~/workspace$ cd gef/
ubuntu@ip-172-31-9-194:~/workspace/gef$ ls
LICENSE  README.md  docs  gef.py  mkdocs.yml  scripts  tests

The only file required is the gef.py script. We can load gef.py on the fly.

ubuntu@ip-172-31-9-194:~/misc$ gdb -q
(gdb) source ~/workspace/gef/gef.py
GEF for linux ready, type `gef' to start, `gef config' to configure
93 commands loaded and 5 functions added for GDB 12.1 in 0.00ms using Python engine 3.10
gef➤  

Alternatively, load gef automatically on GDB startup.

ubuntu@ip-172-31-9-194:~/misc$ echo 'source /path/to/gef.py' >> ~/.config/gdb/gdbinit

ubuntu@ip-172-31-9-194:~/misc$ gdb -q
GEF for linux ready, type `gef' to start, `gef config' to configure
93 commands loaded and 5 functions added for GDB 12.1 in 0.00ms using Python engine 3.10
gef➤

At any moment, we can use the gef context command to show registers, stack, code (disassembly), code, frames, etc.

gef➤  context

From the figure above, the stack is growing from high address to low address.

In addition to standard GDB commands, gef offers enhanced commands. Use the gef command to list all enhanced commands.

pwndbg

pwngdb is more powerful than its predecessor gef. However, it is a standard Linux package (e.g. .rpm). In order not to pollute the system LFS, we install the portable tarball that contains everything needed. The portable tarball even includes a built-in gdb.

~ $ cd /tmp

~ $ curl -LO https://github.com/pwndbg/pwndbg/releases/download/2025.04.18/pwndbg_2025.04.18_arm64-portable.tar.xz

~ $ sudo tar -xJvf pwndbg_2025.04.18_arm64-portable.tar.xz -C /opt/

To launch pwngdb, we just run the /opt/pwndbg/bin/pwndbg Shell script.

~ $ cat /opt/pwndbg/bin/pwndbg

~ $ /opt/pwndbg/bin/pwndbg
pwndbg: loaded 188 pwndbg commands and 39 shell commands. Type pwndbg [--shell | --all] [filter] for a list.
pwndbg: created $rebase, $base, $hex2ptr, $argv, $envp, $argc, $environ, $bn_sym, $bn_var, $bn_eval, $ida GDB functions (can be used with print/break)
------- tip of the day (disable with set show-tips off) -------
If your program has multiple threads they will be displayed in the context display or using the context threads command
pwndbg>

Inpsecting the process, we found pwngdb use the ld-linux dynamic loader to launch GDB with a specialized gdbinit.py.

~ $ ps aux | grep [g]db
kong      410273  1.1  1.0 480768 82600 pts/0    Sl+  14:14   0:00 /opt/pwndbg/lib/ld-linux-aarch64.so.1 /opt/pwndbg/exe/gdb --quiet --early-init-eval-command=set auto-load safe-path / --command=/opt/pwndbg/exe/gdbinit.py

openresty-gdb-utils

openresty-gdb-utils is another GDB extension to debug OpenResty programs, including Nginx, ngx_lua, LuaJIT, etc.

Regarding installation, configuration and commands, please refer to the official page.

nginx.gdb

nginx.gdb is personal GDB script with a few functionalities.

GDB Configuration

To show the GDB installation, pass --configuration option.

(kong-dev) kong@kong-ee:/kong-ee$ gdb --configuration
This GDB was configured as follows:
   configure --host=aarch64-linux-gnu --target=aarch64-linux-gnu
             --with-auto-load-dir=$debugdir:$datadir/auto-load
             --with-auto-load-safe-path=$debugdir:$datadir/auto-load
             --with-expat
             --with-gdb-datadir=/usr/share/gdb (relocatable)
             --with-jit-reader-dir=/usr/lib/gdb (relocatable)
             --without-libunwind-ia64
             --with-lzma
             --with-babeltrace
             --without-intel-pt
             --with-xxhash
             --with-python=/usr (relocatable)
             --with-python-libdir=/usr/lib (relocatable)
             --with-debuginfod
             --with-curses
             --without-guile
             --without-amd-dbgapi
             --enable-source-highlight
             --enable-threading
             --enable-tui
             --with-system-readline
             --with-separate-debug-dir=/usr/lib/debug (relocatable)
             --with-system-gdbinit=/etc/gdb/gdbinit
             --with-system-gdbinit-dir=/etc/gdb/gdbinit.d

("Relocatable" means the directory can be moved with the GDB installation
tree, and GDB will still find it.)

To configure GDB, we have two types of configuration files.

~/.config/gdb/gdbearlyinit which is checked before any other GDB configuration file.
```
# similar to option '-q'
set startup-quietly on
```
This kind of configuration can be replaced by CLI options.

~/.config/gdb/gdbinit which is the primary configuration file.

Automatically load gef

source /home/ubuntu/workspace/gef/gef.py

Automatically load openresty-gdb-utils.

directory /home/ubuntu/workspace/openresty-gdb-utils
   
py import sys
py sys.path.append("/home/ubuntu/workspace/openresty-gdb-utils")
   
source luajit20.gdb
source ngx-lua.gdb
source luajit21.py
source ngx-raw-req.py
set python print-stack full

This kind of configuration can be replanced by interactive commands after GDB startup.

Objects with Debug Info

Without debug information, GDB cannot do symbol translation for names (e.g. variables and functions).

When compiling an executable, we can add the -ggdb3 option to gcc to include debug information. Additionally, we can add the -O0 option to suppress the target machine code optimization from compiler.

ubuntu@ip-172-31-9-194:~/misc$ cat test-gdb.c
#include <stdio.h>

int f(int a, int b)
{
        int sum;
        sum = a + b;
        return sum;
}

int main(int argc, char *argv[])
{
        printf("number of args: %d\n", argc);
        printf("arg input: %s\n", argv[1]);

        int x = 1;
        int y = 2;
        int s = f(x, y);

        printf("%d\n", s/0);
}

ubuntu@ip-172-31-9-194:~/misc$ gcc -Wall -Wextra -ggdb3 -O0 -std=c11 -o test-gdb.out test-gdb.c
test-gdb.c: In function ‘main’:
test-gdb.c:19:25: warning: division by zero [-Wdiv-by-zero]
   19 |         printf("%d\n", s/0);
      |                         ^

ubuntu@ip-172-31-9-194:~/misc$ file test-gdb.out
test-gdb.out: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8bfff35a0f503e12c7fa1f0ffed67597fcf3fb0c, for GNU/Linux 3.2.0, with debug_info, not stripped

The -ggdb3 and -O0 options improve debug experience, but the program would be less optimized and performant. Do not do this for production.

Core Dump

This post focuses on Core Dump object. It is the memory dump of a running process.

There are multiple ways to get a Core Dump. For example, we can make use of the command gcore to dump the memory. Alternatively, we can configure the system to automatically generate a Core Dump when a program crash. Please read "biji" for details.

Please also read the tutorial on How to generate Core Dump within K8s environment.

Verify Core Dump

After receiving a Core Dump file, we must validate it is legitmate.

ubuntu@ip-172-31-9-194:~/misc$ file core_new.2436
core_new.2436: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'nginx: worker prnginx: worker process', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/local/openresty/nginx/sbin/nginx', platform: 'x86_64'

Additionally, we can inspect the logical and physical disk size to see if it is sparse file.

ubuntu@ip-172-31-9-194:~/misc$ ll -hs core_new.2436
747M -rw-r--r-- 1 ubuntu ubuntu 3.0G Jul 31 03:44 core_new.2436

ubuntu@ip-172-31-9-194:~/misc$ du -hsc -- core_new.2436
747M    core_new.2436
747M    total
ubuntu@ip-172-31-9-194:~/misc$ du -hsc --apparent-size -- core_new.2436
3.0G    core_new.2436
3.0G    total

~ $ stat core_new.2436
  File: core_new.2436
  Size: 3114838104      Blocks: 1528400    IO Block: 4096   regular file
ubuntu@ip-172-31-9-194:~/misc$ bc <<< '(1528400 * 512)/1024/1024'
746

Why is Core Dump file a spare file? Core dump is a dump of the virtual memory space, but process usually do not use the entire virtual space. There are unmapped space ranges for two purposes.

Reserved but unmapped space.
Gaps between different segments (e.g. stack, heap) are never mapped.
Memory fragmentation.
Modern 64-bit systems have enormous theoretical address spaces. A process would not use all of them.

GDB understands the sparse Core Dump. Although Core Dump is often sparse, it is not always guaranteed.

Quickstart

Command gdb is an interactive Shell capable of inspecting execution details of a process at a certain point, like dereferencing an uninitialized or NULL pointer (address 0x0).

With the option --nx, gdb skips the configuration.

~ $ gdb --nx
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb)

In this section, we will describe multiple ways to make use of the GDB.

Launch a Process

With the option --args, you can pass to the program arguments at the end of command line.

~ $ gdb [--args] /path/to/program [arglist]

# -or- interactively
~ $ gdb
(gdb) file /path/to/program
(gdb) run [arglist]

Trace a Process

With the option -p, you can omit the binary. We can generate a Core Dump file on the fly with gcore.

# with binary
~ $ gdb /path/to/program <pid>

# no binary; must have '-p'
~ $ gdb -p pid

# generate core dump
(gdb) help gcore
generate-core-file, gcore
Save a core file with the current state of the debugged process.
Usage: generate-core-file [FILENAME]
Argument is optional filename.  Default filename is 'core.PROCESS_ID'.
(gdb) gcore
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.1231557

(gdb) shell ls -lhs core.1231557
9.9M -rw-rw-r-- 1 ubuntu ubuntu 9.9M Aug 15 14:15 core.1231557

Load a Core Dump

With the option -c, you can omit the binary.

# binary available
~ $ gdb /path/to/program /path/to/core.pid

# no binary; must have '-c'
~ $ gdb -c /path/to/core.pid

# -or- interactively
(gdb) file /path/to/program
(gdb) core-file /path/to/core.pid

GDB Commands

At any moment, we can type the special command help for help.

(gdb) help

(gdb) help internals

(gdb) help disassemble

The following is a list of commonly used interactive GDB commands.

Keyboard ENTER key repeats the last command.
run starts or restart the program. The argument list can be provided immediately like run -a -x y.

A program either runs successfully, or runs into issues. When the program is running in the middle, we can stop it via shortcut Ctrl-C or gdb command signal SIGINT.

backtrace shows the call stack. With the option -full, it also prints local variables.

(gdb) backtrace
#0  0x0000ffffa7a7bd74 in __GI_epoll_pwait (epfd=32, events=0xaaab04fa8760, maxevents=512, timeout=492, set=0x0) at ../sysdeps/unix/sysv/linux/epoll_pwait.c:40
#1  0x0000aaaadbfe4ac0 in ngx_epoll_process_events (cycle=0xaaab04fa10e0, timer=492, flags=1) at src/event/modules/ngx_epoll_module.c:800
#2  0x0000aaaadbfcf65c in ngx_process_events_and_timers (cycle=0xaaab04fa10e0) at src/event/ngx_event.c:258
#3  0x0000aaaadbfe10b4 in ngx_worker_process_cycle (cycle=0xaaab04fa10e0, data=0x0) at src/os/unix/ngx_process_cycle.c:793
#4  0x0000aaaadbfdccb8 in ngx_spawn_process (cycle=0xaaab04fa10e0, proc=0xaaaadbfe0fc8 <ngx_worker_process_cycle>, data=0x0, name=0xaaaadc2ba400 "worker process", respawn=-3) at src/os/unix/ngx_process.c:199
#5  0x0000aaaadbfdfa98 in ngx_start_worker_processes (cycle=0xaaab04fa10e0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:382
#6  0x0000aaaadbfdefc4 in ngx_master_process_cycle (cycle=0xaaab04fa10e0) at src/os/unix/ngx_process_cycle.c:135
#7  0x0000aaaadbf91b2c in main (argc=5, argv=0xffffd729e868) at src/core/nginx.c:387

pwndbg has a more powerful context command to show more information.

frame. Stack Frame is an important concept in GDB. It represents a function call and occupies an entry in the call stack.

Take the output from step 3 for example, the #0 is the innermost frame (most recent function), while #7 is the outermost frame (the main function). We call the frame #0 the current frame. It is the base of a lot of GDB commands.

A frame contains the function name, arguments, local variables, source code line, etc. Specially, it includes the memory address at which the function is executing, namely the address where the code is mapped onto, recorded in the register $pc. Please be noted that, it is not the stack address.
```
frame N           # Select frame number N
up                # Move to the caller frame (up the call stack)
down              # Move to the callee frame (down the call stack)

info frame       # Show detailed info about current frame
info locals      # Show local variables in current frame
info args        # Show function arguments of current frame
   
print var        # Print value of variable in current frame
set var=value    # Set value of variable in current frame
```

disassemble shows the assembly code. By default, it shows the code surrounding the register $pc, namely the current frame. With the option /s, it also shows the source code.

(gdb) disassemble /s 0x0000aaaadbfe4ac0
Dump of assembler code for function ngx_epoll_process_events:
src/event/modules/ngx_epoll_module.c:
785     {
   0x0000aaaadbfe4a38 <+0>:     stp     x29, x30, [sp, #-128]!
   0x0000aaaadbfe4a3c <+4>:     mov     x29, sp
   0x0000aaaadbfe4a40 <+8>:     str     x0, [sp, #40]
   0x0000aaaadbfe4a44 <+12>:    str     x1, [sp, #32]
   0x0000aaaadbfe4a48 <+16>:    str     x2, [sp, #24]
   
786         int                events;
787         uint32_t           revents;
788         ngx_int_t          instance, i;
789         ngx_uint_t         level;
790         ngx_err_t          err;
791         ngx_event_t       *rev, *wev;
792         ngx_queue_t       *queue;
793         ngx_connection_t  *c;
794
795         /* NGX_TIMER_INFINITE == INFTIM */
796
797         ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
   0x0000aaaadbfe4a4c <+20>:    ldr     x0, [sp, #40]
   0x0000aaaadbfe4a50 <+24>:    ldr     x0, [x0, #16]
   0x0000aaaadbfe4a54 <+28>:    ldr     x0, [x0]
   0x0000aaaadbfe4a58 <+32>:    bl      0xaaaadc195650 <ngx_http_lua_kong_get_dynamic_log_level>
--Type <RET> for more, q to quit, c to continue without paging--
   0x0000aaaadbfe4a5c <+36>:    and     x0, x0, #0x80
   0x0000aaaadbfe4a60 <+40>:    cmp     x0, #0x0
   0x0000aaaadbfe4a64 <+44>:    b.eq    0xaaaadbfe4a88 <ngx_epoll_process_events+80>  // b.none
   0x0000aaaadbfe4a68 <+48>:    ldr     x0, [sp, #40]
   0x0000aaaadbfe4a6c <+52>:    ldr     x1, [x0, #16]
   0x0000aaaadbfe4a70 <+56>:    ldr     x4, [sp, #32]
   0x0000aaaadbfe4a74 <+60>:    adrp    x0, 0xaaaadc2ba000
   0x0000aaaadbfe4a78 <+64>:    add     x3, x0, #0xac0
   0x0000aaaadbfe4a7c <+68>:    mov     w2, #0x0                        // #0
   0x0000aaaadbfe4a80 <+72>:    mov     x0, #0x8                        // #8
   0x0000aaaadbfe4a84 <+76>:    bl      0xaaaadbf946fc <ngx_log_error_core>
   
798                        "epoll timer: %M", timer);
799
800         events = epoll_wait(ep, event_list, (int) nevents, timer);
   0x0000aaaadbfe4a88 <+80>:    adrp    x0, 0xaaaadc661000 <week+16>
   0x0000aaaadbfe4a8c <+84>:    add     x0, x0, #0xb90
   0x0000aaaadbfe4a90 <+88>:    ldr     w4, [x0]
   0x0000aaaadbfe4a94 <+92>:    adrp    x0, 0xaaaadc6a0000 <ngx_processes+47632>
   0x0000aaaadbfe4a98 <+96>:    add     x0, x0, #0xa48
   0x0000aaaadbfe4a9c <+100>:   ldr     x1, [x0]
   0x0000aaaadbfe4aa0 <+104>:   adrp    x0, 0xaaaadc6a0000 <ngx_processes+47632>
   0x0000aaaadbfe4aa4 <+108>:   add     x0, x0, #0xa50
   0x0000aaaadbfe4aa8 <+112>:   ldr     x0, [x0]
   0x0000aaaadbfe4aac <+116>:   mov     w2, w0
--Type <RET> for more, q to quit, c to continue without paging--q
Quit

The objdump has similar capabilities.

list shows only source code. With the argument ., it shows code surrounding the current frame.

gef➤  frame 1
#1  0x0000aaaaad984ac0 in ngx_epoll_process_events (cycle=0xaaaab6b09130, timer=0x134, flags=0x1) at src/event/modules/ngx_epoll_module.c:800
800         events = epoll_wait(ep, event_list, (int) nevents, timer);
gef➤  list .
795         /* NGX_TIMER_INFINITE == INFTIM */
796
797         ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0,
798                        "epoll timer: %M", timer);
799
800         events = epoll_wait(ep, event_list, (int) nevents, timer);
801
802         err = (events == -1) ? ngx_errno : 0;
803
804         if (flags & NGX_UPDATE_TIME || ngx_event_timer_alarm) {

To show the source code pathname.

(gdb) info sources
/home/kong/.cache/bazel/_bazel_kong/ee741459d483c56c5059256f49a7a414/execroot/_main/bazel-out/aarch64-fastbuild/bin/build/kong-dev/openresty/nginx/sbin/nginx:
(Full debug information has not yet been read for this file.)
   
/home/kong/.cache/bazel/_bazel_kong/ee741459d483c56c5059256f49a7a414/execroot/_main/bazel-out/aarch64-fastbuild/bin/external/openresty/openresty.build_tmpdir/build/nginx-1.25.3/src/core/nginx.c,
/home/kong/.cache/bazel/_bazel_kong/ee741459d483c56c5059256f49a7a414/execroot/_main/bazel-out/aarch64-fastbuild/bin/external/openresty/openresty.build_tmpdir/build/nginx-1.25.3/src/os/unix/ngx_files.h,
/home/kong/.cache/bazel/_bazel_kong/ee741459d483c56c5059256f49a7a414/execroot/_main/bazel-out/aarch64-fastbuild/bin/external/openresty/openresty.build_tmpdir/build/nginx-1.25.3/src/core/ngx_log.h,

break toggle a breakpoint and pauses the program at the specified line.

Use list or disassemble to identify the code line you are interested in.

(gdb) break ngx_epoll_module.c:800
Breakpoint 2 at 0xaaaadbfe4a88: file src/event/modules/ngx_epoll_module.c, line 800.
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
2       breakpoint     keep y   0x0000aaaadbfe4a88 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:800
(gdb) delete breakpoints 2
(gdb) info breakpoints
No breakpoints, watchpoints, tracepoints, or catchpoints.

You can define a conditional breakpoint like break 15 if var > 10.

watch monitors a variable and pauses the program when the variable is modified.

(gdb) info variables -t ngx_cycle_t
All defined variables with type matching regular expression "ngx_cycle_t" :
   
File ../echo-nginx-module-0.63/src/ngx_http_echo_filter.c:
28:     static volatile ngx_cycle_t *ngx_http_echo_prev_cycle;
   
File ../headers-more-nginx-module-0.37/src/ngx_http_headers_more_filter_module.c:
111:    static volatile ngx_cycle_t *ngx_http_headers_more_prev_cycle;
   
File ../ngx_lua-0.10.26/src/ngx_http_lua_module.c:
71:     static volatile ngx_cycle_t *ngx_http_lua_prev_cycle;
   
File src/core/ngx_cycle.c:
21:     volatile ngx_cycle_t *ngx_cycle;
   
File src/os/unix/ngx_process_cycle.c:
73:     static ngx_cycle_t ngx_exit_cycle;
   
(gdb) watch ngx_cycle
Hardware watchpoint 5: ngx_cycle
   
(gdb) info watchpoints
Num     Type           Disp Enb Address            What
5       hw watchpoint  keep y                      ngx_cycle

continue resumes the program until crash, the next breakpoint or exit.
1. next executes the next statement.
2. step steps into funtion call while next does not.
  
  stepi steps over machine instruction instead of source code statement.
3. finish completes the current function call and then pause.

print shows values of variables.

(gdb) print ngx_cycle
$12 = (volatile ngx_cycle_t *) 0xaaab04fa10e0
    
(gdb) print * $12
$13 = {conf_ctx = 0xaaab04fa2580, pool = 0xaaab04fa10a0, log = 0xaaab04fa10f8, new_log = {log_level = 6, file = 0xaaab04fa15f0, connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0,
  files = 0x0, free_connections = 0xffff705cbb78, free_connection_n = 15840, modules = 0xaaab04fa2cb0, modules_n = 89, modules_used = 1, reusable_connections_queue = {prev = 0xaaab04fa1180, next = 0xaaab04fa1180}, reusable_connections_n = 0, connections_reuse_time = 0,
  listening = {elts = 0xaaab055dbf30, nelts = 16, size = 296, nalloc = 20, pool = 0xaaab04fa10a0, old_elts = 0xaaab055d45d0}, paths = {elts = 0xaaab04fa1530, nelts = 5, size = 8, nalloc = 10, pool = 0xaaab04fa10a0, old_elts = 0x0}, config_dump = {elts = 0xaaab04ffdd40,
    nelts = 7, size = 24, nalloc = 8, pool = 0xaaab04fa10a0, old_elts = 0xaaab04ffde30}, config_dump_rbtree = {root = 0xaaab04fc3850, sentinel = 0xaaab04fa1248, insert = 0xaaaadbfa0258 <ngx_str_rbtree_insert_value>}, config_dump_sentinel = {key = 0, left = 0x0,
    right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0xaaab04fa1278, part = {elts = 0xaaab04fa15f0, nelts = 7, next = 0x0}, size = 40, nalloc = 20, pool = 0xaaab04fa10a0}, shared_memory = {last = 0xaaab051b0240, part = {
      elts = 0xaaab04fa1940, nelts = 1, next = 0xaaab04fc4a60}, size = 88, nalloc = 1, pool = 0xaaab04fa10a0}, connection_n = 16384, files_n = 0, connections = 0xffff7057f010, read_events = 0xffff703fe010, write_events = 0xffff7027d010, old_cycle = 0x0, conf_file = {
    len = 58, data = 0xaaab04fa1480 "/kong-ee/bazel-bin/build/kong-dev/kong/servroot/nginx.conf"}, conf_param = {len = 0, data = 0xaaab04fa14f0 ""}, conf_prefix = {len = 48, data = 0xaaab04fa13a0 "/kong-ee/bazel-bin/build/kong-dev/kong/servroot/"}, prefix = {len = 48,
    data = 0xaaab04fa13e0 "/kong-ee/bazel-bin/build/kong-dev/kong/servroot/"}, error_log = {len = 14, data = 0xaaab04fa1440 "logs/error.log"}, lock_file = {len = 64, data = 0xaaab055b4330 "/kong-ee/bazel-bin/build/kong-dev/kong/servroot/logs/nginx.lock.accept"},
  hostname = {len = 7, data = 0xaaab04fa2c70 "kong-ee"}, intercept_error_log_handler = 0x0, intercept_error_log_data = 0x0, entered_logger = 0}

x shows values referenced by an memory address.

0x0000ffffe2a53890│+0x0000: 0x0000ffffe2a538d0  →  0x0000ffffe2a53950  →  0x0000ffffe2a539a0  →  0x0000ffffe2a539d0  →  0x0000ffffe2a53a40  →  0x0000ffffe2a53a80  →  0x0000ffffe2a53b90         ← $x29, $sp
0x0000ffffe2a53898│+0x0008: 0x0000aaaaad984ac0  →  <ngx_epoll_process_events+0088> str w0,  [sp,  #64]
0x0000ffffe2a538a0│+0x0010: 0x0000ffffe2a54028  →  0x0000ffffe2a54dcf  →  "nginx: worker process"
0x0000ffffe2a538a8│+0x0018: 0x0000000000000005
0x0000ffffe2a538b0│+0x0020: 0x0000aaaaadffde88  →  0x0000aaaaad931400  →  <__do_global_dtors_aux+0000> stp x29,  x30,  [sp,  #-32]!
0x0000ffffe2a538b8│+0x0028: 0x0000aaaaad931574  →  <main+0000> sub sp,  sp,  #0x320
 
gef➤  x/a 0x0000ffffe2a538a0
0xffffe2a538a0: 0xffffe2a54028
gef➤  x/a 0xffffe2a54028
0xffffe2a54028: 0xffffe2a54dcf
 
gef➤  x/s 0xffffe2a54dcf
0xffffe2a54dcf: "nginx: worker process"
 
gef➤  print (char *) 0x0000ffffe2a54dcf
$5 = 0xffffe2a54dcf "nginx: worker process"

info is very useful.
record combined with *reverse-* (e.g. *reverse-step*) can execute the program in reverse order. Therefore, there is no need to re-run the program to inpsect previous execution contexts.
shell invokes a shell command, like shell ll.

Similar to Bash, you can check historical GDB commands in ~/.gdb_history.

For sophiscated manual, please refer to the official documentation. It you are using pwndgb, refer to cheatsheet.

More Tools

Apart from GDB, we have a bunch of other tools to inspect objects.

To make it simple, we can even use vim to inspect the file.

hexdump

hexdump can dump file contents in various formats (e.g. hexadecimal), including customized format with the option --format.

The following example dumps the contents in ASCII format.

~ $ hexdump -C core.pid | less 

000140b0  f0 b8 1d 00 00 00 00 00  0a 00 00 00 00 00 00 00  |................|
000140c0  00 5f 49 54 4d 5f 64 65  72 65 67 69 73 74 65 72  |._ITM_deregister|
000140d0  54 4d 43 6c 6f 6e 65 54  61 62 6c 65 00 5f 5f 67  |TMCloneTable.__g|
000140e0  6d 6f 6e 5f 73 74 61 72  74 5f 5f 00 5f 49 54 4d  |mon_start__._ITM|
000140f0  5f 72 65 67 69 73 74 65  72 54 4d 43 6c 6f 6e 65  |_registerTMClone|
00014100  54 61 62 6c 65 00 63 72  79 70 74 5f 72 00 6c 75  |Table.crypt_r.lu|
00014110  61 5f 70 75 73 68 66 73  74 72 69 6e 67 00 6c 75  |a_pushfstring.lu|
00014120  61 5f 73 65 74 65 78 64  61 74 61 00 6c 75 61 5f  |a_setexdata.lua_|
00014130  67 65 74 66 69 65 6c 64  00 6c 75 61 4c 5f 70 75  |getfield.luaL_pu|
00014140  73 68 72 65 73 75 6c 74  00 6c 75 61 5f 78 6d 6f  |shresult.lua_xmo|
00014150  76 65 00 6c 75 61 5f 72  65 73 75 6d 65 00 6c 75  |ve.lua_resume.lu|
00014160  61 4c 5f 61 64 64 6c 73  74 72 69 6e 67 00 6c 75  |aL_addlstring.lu|
00014170  61 4c 5f 6f 70 74 6e 75  6d 62 65 72 00 6c 75 61  |aL_optnumber.lua|
00014180  5f 6e 65 77 75 73 65 72  64 61 74 61 00 6c 75 61  |_newuserdata.lua|
00014190  5f 73 65 74 66 69 65 6c  64 00 6c 75 61 5f 73 65  |_setfield.lua_se|
000141a0  74 74 6f 70 00 6c 75 61  5f 70 75 73 68 73 74 72  |ttop.lua_pushstr|
000141b0  69 6e 67 00 6c 75 61 6f  70 65 6e 5f 66 66 69 00  |ing.luaopen_ffi.|
000141c0  6c 75 61 5f 67 65 74 69  6e 66 6f 00 6c 75 61 5f  |lua_getinfo.lua_|
000141d0  69 73 6e 75 6d 62 65 72  00 6c 75 61 4c 5f 63 68  |isnumber.luaL_ch|
000141e0  65 63 6b 6c 73 74 72 69  6e 67 00 6c 75 61 5f 6e  |ecklstring.lua_n|
000141f0  65 78 74 00 6c 75 61 5f  74 6f 74 68 72 65 61 64  |ext.lua_tothread|
00014200  00 6c 75 61 5f 67 63 00  6c 75 61 5f 63 72 65 61  |.lua_gc.lua_crea|

However, all the non-printable characters are shown as dots in the ASCII column.

~ $ for ((i=0; i<=255; ++i)); do printf "\x$(printf %x $i)"; done | hexdump -C
00000000  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000010  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000020  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000030  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000040  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000050  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000060  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000070  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000080  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000090  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000000a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000000b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000000c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000000d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000000e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000000f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000100

The following examples show that the object contains a lot of zeros.

~ $ hexdump -C 0x577af3685000-0x577af8547000.bin | less
00003b90  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00003ba0  00 00 00 00 00 00 00 00  91 00 00 00 00 00 00 00  |................|
00003bb0  30 8c 68 f3 7a 57 00 00  30 8c 68 f3 7a 57 00 00  |0.h.zW..0.h.zW..|
00003bc0  a0 8c 68 f3 7a 57 00 00  06 00 00 00 00 00 00 00  |..h.zW..........|
00003bd0  31 37 32 2e 31 36 2e 30  2e 31 30 3a 35 33 00 00  |172.16.0.10:53..|
00003be0  00 00 00 00 00 00 00 00  98 8a 68 f3 7a 57 00 00  |..........h.zW..|
00003bf0  40 8c 68 f3 7a 57 00 00  00 00 00 00 00 00 00 00  |@.h.zW..........|
00003c00  40 72 9b 6b 0b 73 00 00  40 8b 68 f3 7a 57 00 00  |@r.k.s..@.h.zW..|
00003c10  31 37 32 2e 31 36 2e 30  2e 31 30 00 00 00 00 00  |172.16.0.10.....|
00003c20  02 00 00 35 ac 10 00 0a  00 00 00 00 00 00 00 00  |...5............|
00003c30  00 00 00 00 00 00 00 00  61 00 00 00 00 00 00 00  |........a.......|
00003c40  0e 00 00 00 00 00 00 00  d0 8b 68 f3 7a 57 00 00  |..........h.zW..|
00003c50  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00003c60  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00003c70  88 8b 68 f3 7a 57 00 00  10 00 00 00 00 00 00 00  |..h.zW..........|
00003c80  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00003c90  00 00 00 00 00 00 00 00  91 00 00 00 00 00 00 00  |................|
00003ca0  20 8d 68 f3 7a 57 00 00  20 8d 68 f3 7a 57 00 00  | .h.zW.. .h.zW..|
00003cb0  90 8d 68 f3 7a 57 00 00  06 00 00 00 00 00 00 00  |..h.zW..........|
00003cc0  20 8c 68 f3 7a 57 00 00  10 00 00 00 00 00 00 00  | .h.zW..........|
00003cd0  0e 00 00 00 00 00 00 00  e0 8c 68 f3 7a 57 00 00  |..........h.zW..|
00003ce0  31 37 32 2e 31 36 2e 30  2e 31 30 3a 35 33 00 00  |172.16.0.10:53..|
00003cf0  00 00 00 00 00 00 00 00  e8 8b 68 f3 7a 57 00 00  |..........h.zW..|
00003d00  30 8d 68 f3 7a 57 00 00  00 00 00 00 00 00 00 00  |0.h.zW..........|
00003d10  f8 31 98 69 0b 73 00 00  f8 8b 68 f3 7a 57 00 00  |.1.i.s....h.zW..|
00003d20  00 00 00 00 00 00 00 00  61 00 00 00 00 00 00 00  |........a.......|

strings

Though hexdump shows ASCII characters, it is not easy to extract the ASCII column. GNU strings, on the other hand, can dump printable characters and helps identify large strings in the Core Dump.

The option -t prefixes the string with the offset in the object file. It is useful show the distribution and density of string.

~ $ strings -t d /path/to/core.pid > core.pid.density

~ $ strings [-n 4] /path/to/core.pid > core.pid.ascii

~ $ sort -o core.pid.ascii.sorted core.pid.ascii

~ $ uniq -c core.pid.ascii.sorted > core.pid.ascii.sorted.uniq

~ $ sort -nrk1,1 -o core.pid.ascii.sorted.uniq.sorted core.pid.ascii.sorted.uniq

However, non-printable characters are not included in the output.

readelf

readelf is the architecture independent but objdump can dump source code.

Specially, gef supports print ELF info.

gef➤  elf-info

Regarding more examples of readelf, please see "biji".

objdump

readelf cannot dump source code as objdump does. Here is an example.

~ $ objdump --source --source-comment=txt test-gdb.out

...

0000000000001149 <f>:
txt#include <stdio.h>
txt
txtint f(int a, int b)
txt{
    1149:       f3 0f 1e fa             endbr64
txt     int sum;
txt     sum = a + b;
    114d:       8d 04 37                lea    (%rdi,%rsi,1),%eax
txt     return sum;
txt}
    1150:       c3                      ret

...

GDB command disassemble and list can also dump source code and assembler code.

Memory

We can make use of top to show the general memory utilization.

~ $ top -o +RES -p $(pgrep -d',' nginx)

Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.9 us,  1.8 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7841.2 total,   1885.2 free,   4823.7 used,   1392.9 buff/cache
MiB Swap:   1024.0 total,    105.4 free,    918.6 used.   3017.5 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
37937 kong      20   0 1393704 414628  11904 S   0.0   5.2  14:36.72 nginx
37936 kong      20   0 1023736 108848    896 S   0.0   1.4   0:00.00 nginx

Memory mappings represents how virtual memory space is mapped to RAM. It tells us how the memory is used. GDB and gef both can show memory mappings. Of the three commands below, maintenance info sections only show the mappings of different memory sections of the ELF. It corresponds to only a few starting lines of vmmap and info proc mappings that include dyanmically allocated memory (e.g. heap/stack).

(gdb) maintenance info sections

gef➤  vmmap

(gdb) info proc mappings

Without GDB, we can show the memory mappings of a running process with pmap.

~ $ pmap -X $(pgrep nginx) > all-$(date -Iseconds).pmap

Core dumps captured at different timestamps reveal newly allocated chunks. Moreover, we can use hexdump, strings and even the vim to see what is inside the new chunks. How to identify newly allocated chunks?

The following are two excerpts from two Core Dumps. We will use this data as an example to show the process.

# Core Dump 1
[14]      0x577ae51df000->0x577ae521a000 at 0x00002a40: load1 ALLOC LOAD READONLY HAS_CONTENTS
[15]      0x577ae54fc000->0x577ae54ff000 at 0x0003da40: load2 ALLOC LOAD READONLY HAS_CONTENTS
[16]      0x577ae54ff000->0x577ae551e000 at 0x00040a40: load3 ALLOC LOAD HAS_CONTENTS
[17]      0x577ae551e000->0x577ae5550000 at 0x0005fa40: load4 ALLOC LOAD HAS_CONTENTS
[18]      0x577ae63cd000->0x577ae66a0000 at 0x00091a40: load5 ALLOC LOAD HAS_CONTENTS
[19]      0x577ae66a0000->0x577af3685000 at 0x00364a40: load6 ALLOC LOAD HAS_CONTENTS
[20]      0x730b65ac0000->0x730b671c1000 at 0x0d349a40: load7 ALLOC LOAD HAS_CONTENTS

# Core Dump 2
[14]      0x577ae51df000->0x577ae521a000 at 0x00002a78: load1 ALLOC LOAD READONLY HAS_CONTENTS
[15]      0x577ae54fc000->0x577ae54ff000 at 0x0003da78: load2 ALLOC LOAD READONLY HAS_CONTENTS
[16]      0x577ae54ff000->0x577ae551e000 at 0x00040a78: load3 ALLOC LOAD HAS_CONTENTS
[17]      0x577ae551e000->0x577ae5550000 at 0x0005fa78: load4 ALLOC LOAD HAS_CONTENTS
[18]      0x577ae63cd000->0x577ae66a0000 at 0x00091a78: load5 ALLOC LOAD HAS_CONTENTS
[19]      0x577ae66a0000->0x577af8547000 at 0x00364a78: load6 ALLOC LOAD HAS_CONTENTS
[20]      0x730b64d1d000->0x730b671c1000 at 0x1220ba78: load7 ALLOC LOAD HAS_CONTENTS

The 19th mapping in Core Dump 2 has expanded its ending address than the Core Dump 1. We can calculate the increased size in bytes.

~ $ bc <<< 'obase=16; ibase = 16; 0X577AF8547000 - 0X577AF3685000'
4EC2000

~ $ bc <<< 'obase=10; ibase = 16; 0X577AF8547000 - 0X577AF3685000'
82583552

Now let us extract the newly allocated memory from Core Dump 2 and save to the local disk.

(gdb) dump memory ~/misc/0x577af3685000-0x577af8547000.bin 0x577af3685000 0x577af8547000

With the extracted ranged memory dump, we can resort to More Tools.

Please check "whatever" and "kong-dev" for more details. You are strongly recommended to read Memory Leak (and Growth) Flame Graphs.

Nginx

(gdb) p ngx_cycle->pool
$1 = (ngx_pool_t *) 0x577ae63f3ff0
(gdb) p * ngx_cycle->pool
$2 = {d = {last = 0x577ae63f7ff0 "", end = 0x577ae63f7ff0 "", next = 0x577ae63fd020, failed = 6}, max = 4095, current = 0x577ae643f5f0, chain = 0x0, large = 0x577ae64c52d8, cleanup = 0x577ae64c5980, log = 0x577ae63f4058}
(gdb)

src/ngx_http_lua_socket_udp.c
 -> ngx_http_lua_socket_udp_setpeername
   ->     r = ngx_http_lua_get_req(L);
      ->     host.data = ngx_palloc(r->pool, len + 1);

(gdb) p * (ngx_http_upstream_resolved_t *) 0x577af3685060
$4 = {host = {len = 3471771947605571377, data = 0x33353a30312e <error: Cannot access memory at address 0x33353a30312e>}, port = 0, no_port = 96185581326184, naddrs = 96185581326496, addrs = 0x0, sockaddr = 0x0, socklen = 97, name = {len = 14,
    data = 0x577af3685060 "172.16.0.10:53"}, ctx = 0x0}

Appendix

Copied from How to look at the stack with gdb.

#include <stdio.h>
#include <stdlib.h>

int main() {
    char stack_string[10] = "stack";
    int x = 10;
    char *heap_string;

    heap_string = malloc(50);

    printf("Enter a string for the stack: ");
    gets(stack_string);
    printf("Enter a string for the heap: ");
    gets(heap_string);
    printf("Stack string is: %s\n", stack_string);
    printf("Heap string is: %s\n", heap_string);
    printf("x is: %d\n", x);
}

LuaRocks Tutorial

2024-03-01T00:00:00+00:00

LuaRocks luarocks
Manual Installation
1. Build lua
2. Build luarocks
rocks
rocks servers
config
Create Rocks
Upload Rocks
1. Upload to Official Server
2. Upload to Custom Server
Install Rocks

LuaRocks luarocks

"LuaRocks" refers to the project name, while "luarocks" and "luarocks-admin" refer to CLI tool.

Manual Installation

Build lua

This sections show how to manually compile Lua 5.1.

# fetch source
~ $ curl -O https://www.lua.org/ftp/lua-5.1.4.tar.gz
~ $ tar -xf lua-5.1.4.tar.gz

# customization options
~ $ make echo
~ $ make pecho
~ $ make lecho

# determine the platform compatiblity
~ $ make

# make
~ $ make linux
~ $ make -n install

# install
~ $ sudo make install
~ $ /usr/local/bin/lua -v

Build luarocks

In this section, we will demonstrate how to install Luarocks from source.

Firstly, we install the dependency lib lua-devel as below. We can avoid this if we have manually built Lua.

[ec2-user@ip-172-31-16-59 tmp]$ sudo yum install lua-devel

Now compile Luarocks:

[ec2-user@ip-172-31-16-59 ~]$ cd /tmp

[ec2-user@ip-172-31-16-59 tmp]$ wget https://luarocks.org/releases/luarocks-3.9.0.tar.gz
[ec2-user@ip-172-31-16-59 tmp]$ tar -xzpvf luarocks-3.9.0.tar.gz
[ec2-user@ip-172-31-16-59 tmp]$ cd luarocks-3.9.0/
[ec2-user@ip-172-31-16-59 luarocks-3.9.0]$ ./configure --with-lua-include=/usr/include
[ec2-user@ip-172-31-16-59 luarocks-3.9.0]$ make
[ec2-user@ip-172-31-16-59 luarocks-3.9.0]$ sudo make install

[ec2-user@ip-172-31-16-59 luarocks-3.9.0]$ luarocks search lua-cassandra

[ec2-user@ip-172-31-16-59 luarocks-3.9.0]$ sudo -i
[root@ip-172-31-16-59 ~]# echo $PATH

# .bash_profile
PATH=$PATH:$HOME/bin:/usr/local/bin

[root@ip-172-31-16-59 ~]# exit
[ec2-user@ip-172-31-16-59 luarocks-3.9.0]$ sudo -i luarocks -h

For more details regarding configuring Luarocks, please check the section config.

rocks

The term rock refers to a ZIP archive in the ".rock" extention, containing two kinds of source files as follows.

.rockspec. We can generate a .rockspec template quickly according to write_rockspec.
Source code, mainly .lua and may include .c.

There are 3 types of rocks as shown below. They differ mainly in the format of the source code part. Please read the official doc for more details.

Source Rock (.src.rock). The source code is ZIPped as it is. It is usually uploaded to rocks servers by luarocks upload or luarocks-admin.

The next two types are for installation purpose.
Binary Rock (.<system-arch>.rock). This installation type include .c source code but compiled to the platform-specific C module.

Can be installed directly to the specific platform.
Pure-Lua Rock (.all.rock). This installation type contains only Lua code. The ZIP organizes the source rock according to the rock tree structure.

It can be installed directly independent of platforms, but does not guarantee running everywhere. The Lua code might depend on external FFI C modules or os.execute(<cmd>).

Please refer to the section Create Rocks on how to create rocks.

rocks servers

We have two kinds of Rock Repository.

rocks servers, can be an URL (e.g. Github repo) or a local pathname (local server). For examle, the official (default) rocks server URL is https://luarocks.org.

We can upload rocks to and download rocks from rocks servers

~ $ root@0d3fe341fc88:/# luarocks install luacov --server https://kong.github.io/kongrocks-dev/rocks/
Installing https://kong.github.io/kongrocks-dev/rocks/luacov-0.16.0-1.src.rock

luacov 0.16.0-1 depends on lua >= 5.1 (5.1-1 provided by VM: success)
luacov 0.16.0-1 depends on datafile (0.10-1 installed: success)
luacov 0.16.0-1 is now installed in /usr/local (license: MIT)

rocks trees, is a local pathname, to where we install rocks.

A rock repository contains three types of files.

rock
.rockspec: optional as it is already in the rock.
manifest file.

config

To show current config.

# list all
~ $ luarocks config

# list only one entry
~ $ luarocks config rocks_servers

Here is an example to customize rocks_servers.

-- ~/.luarocks/config-5.4.lua
rocks_servers = {
    "https://dummy:" .. os_getenv("GITHUB_TOKEN") .. "@raw.githubusercontent.com/Kong/kongrocks/main/rocks",  -- internal rocks server
    "https://luarocks.org",    -- official rocks server
    "https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/",
    "https://luafr.org/luarocks/",
}

Please refer to official config file and Emacs LSP example.

Create Rocks

We create a rock using luarocks pack.

Firstly, make sure zip is available.

~ $ type zip

There are two methods listed as follows.

Given a .rockspec file, we create a Source Rock based on source code downloaded.

~ $ luarocks pack ./lua-resty-session/lua-resty-session-4.0.5-1.rockspec

~ $ ls
lua-resty-session-4.0.5-1.src.rock
   
~ $ file lua-resty-session-4.0.5-1.src.rock
lua-resty-session-4.0.5-1.src.rock: Zip archive data, at least v2.0 to extract, compression method=deflate

Given a module installed in a rocks tree, we can create a Binary Rock or a Pure-Lua Rock.

From the example below, we find dkjson is a Pure-Lua Rock (without C modules).
```
~ $ luarocks list dkjson
~ $ luarocks show dkjson

# version "2.6-1" is optional; default to latest.
~ $ luarocks pack [--verbose] dkjson [2.6-1]

~ $ ls
dkjson-2.6-1.all.rock

~ $ file dkjson-2.6-1.all.rock
dkjson-2.6-1.all.rock: Zip archive data, at least v2.0 to extract, compression method=deflate
```
Apart from luarocks pack, we can also use luarocks build --pack-binary-rock to create a Binary Rock (with C modules) or a Pure-Lua Rock. This is a better way as it does not require the rock to be installed.

Upload Rocks

Upload to Official Server

Luarocks upload uploads both the .rockspec and the Source Rock to the official rocks_server. We must provide the .rockspec file. If the Source Rock is not provided, it would be create on the fly.

Offer API key on CLI.

# switch to temporary directory
16:11:41 zachary@Zacharys-MacBook-Pro ~/workspace/resty-redis-cluster
$ cd ~/misc/

16:15:30 zachary@Zacharys-MacBook-Pro ~/misc
$ luarocks upload --api-key xxxyyyzzz ~/workspace/resty-redis-cluster/kong-redis-cluster-1.3.0-0.rockspec

Alternatively, configure API key in the upload config.

11:11:00 zachary@Zacharys-MacBook-Pro ~
$ cat ~/.luarocks/upload_config.lua
key = "xxxyyyzzz"
server = "https://luarocks.org"

# switch to temporary directory
16:11:41 zachary@Zacharys-MacBook-Pro ~/workspace/resty-redis-cluster
$ cd ~/misc/

16:15:30 zachary@Zacharys-MacBook-Pro ~/misc
$ luarocks upload ~/workspace/resty-redis-cluster/kong-redis-cluster-1.3.0-0.rockspec

We can optionally, remove the temporary Source Rock.

16:15:30 zachary@Zacharys-MacBook-Pro ~/misc
$ rm kong-redis-cluster-1.3.0-0.src.rock

Verify the new rock is available from the official rocks server.

16:16:29 zachary@Zacharys-MacBook-Pro ~/misc
$ luarocks search kong-redis-cluster [1.3.0]

To upload rocks to custom rocks servers, please refer to luarocks-admin.

Upload to Custom Server

luarocks upload uploads only Source Rock to only the official rocks server. We can upload any type of rocks to any custom rocks servers, by luarocks-admin.

Manuall method.

Copy the .rockspec to the custom rocks server.

~ $ git clone git@github.com:Kong/kongrocks.git
~ $ git checkout -b FTI-5247
~ $ cd ~/workspace/kongrocks

~ $ cp ~/workspace/ce2ee/distribution/kong-openid-connect/kong-openid-connect-2.5.6-1.rockspec ./rocks/

Create Source Rock and copy it to the custom rocks server.

~ $ cd ~/misc
~ $ luarocks pack ~/workspace/ce2ee/distribution/kong-openid-connect/kong-openid-connect-2.5.6-1.rockspec
   
~ $ ls
kong-openid-connect-2.5.6-1.src.rock
   
~ $ cp ~/misc/kong-openid-connect-2.5.6-1.src.rock ./rocks/

This is optional but highly recommended.

Create Binary Rock or Pure-Lua Rock and copy it to the custom rocks server.

We pass the option --pack-binary-rock to luarocks build to skip rock installation.

~ $ cd ~/misc
   
~ $ luarocks build --pack-binary-rock ~/workspace/ce2ee/distribution/kong-openid-connect/kong-openid-connect-2.5.6-1.rockspec
# -or-
~ $ luarocks build --pack-binary-rock kong-openid-connect-2.5.6-1.src.rock

~ $ ls
kong-openid-connect-2.5.6-1.all.rock
   
~ $ cp ~/misc/kong-openid-connect-2.5.6-1.all.rock ./rocks/

This is optional as Binary Rock or Pure-Lua Rock is mainly for installation to a specific platform. You are recommended to skip this part.

Update manifest of the custom rocks server.
```
~ $ luarocks-admin make-manifest ./rocks/
```
A rocks server has only one manifest file to record all available .rockspec and rocks.
Sync the change from local repository to remote repository.
```
~ $ git add -A
~ $ git push -u origin head
```

The manual method described above is fairly flexibile. However, we can make use of luarocks-admin add and luarocks-admin remove as they automatically update the manifest file. The limitation is they handle only one rock at a time. See example below.

~ $ cd ~/misc

~ $ luarocks pack kong-openid-connect-2.5.6-2.rockspec
~ $ luarocks build --pack-binary-rock kong-openid-connect-2.5.6-2.src.rock
~ $ ls
kong-openid-connect-2.5.6-2.all.rock    kong-openid-connect-2.5.6-2.rockspec    kong-openid-connect-2.5.6-2.src.rock

~ $ cd ~/workspace/kongrocks
~ $ luarocks-admin add --server ~/workspace/kongrocks/rocks/ ~/misc/kong-openid-connect-2.5.6-2.rockspec
~ $ luarocks-admin add --server ~/workspace/kongrocks/rocks/ ~/misc/kong-openid-connect-2.5.6-2.src.rock
~ $ luarocks-admin add --server ~/workspace/kongrocks/rocks/ ~/misc/kong-openid-connect-2.5.6-2.all.rock

~ $ git status

Pay attention that, we explicitly set --server to overwrite the rocks_servers in config file.

Install Rocks

We install a Binary Rock or Pure-Lua Rock to the rocks tree on local host.

We can either manually add .lua file to LUA_PATH, or invoke the following commands.

luarocks build compiles Binary Rock or Pure-Lua Rock based on Source Rock downloaded from rocks servers, and then install it.
luarocks make differs from luarocks build in that it compiles Binary Rock or Pure-Lua Rock from the source code in local dev environment. This is very useful when when we want to debug the code.
luarocks install installs a local Binary Rock or Pure-Lua Rock file, or falls back to luarocks build when given a .rockspec file.

See https://github.com/luarocks/luarocks/wiki/luarocks#overview-of-the-difference-between-make-build-install-and-pack.

Golden Image

2024-02-19T00:00:00+00:00

golden image or master image, in the software industry, refers to a base Docker image built from scratch. Once build, no further changes are allowed.

Based on the golden image, we generate distributable images that are ready to be deployed in production environment. Distributable images may include programming language runtime, development tools, system administration tools, etc.

Customers may further customize golden images or distributable images in accord with their platforms.

The statements above apply to Linux packages (e.g. RPM) as well.

Emacs LSP

2023-01-20T00:00:00+00:00

LSP
Emacs eglot
LSP Servers

LSP

Language Server Protocol (LSP) is originally from Mircosoft and now develops to a standard protocol.

LSP works in a client/server mode. In this post, the client is usually package of IDE (e.g. Emacs), while the server is specific to programing language (e.g. C/C++).

To configure Emacs with LSP support, we firstly, install the eglot in Emacs, then install a language-specific LSP server on your host, and might do some setup in init.el depending on the LSP server requirement.

Emacs eglot

Emacs supports LSP by its built-in package eglot or 3rd-party package lsp-mode.

eglot has less features but requires less dependencies. What is more, it is official!

To install the eglot client, just run M-x package-install RET eglot RET.
To start eglot for a LSP.
1. Manually run M-x eglot.
2. Automatically by (add-hook 'foo-mode-hook 'eglot-ensure).
To turn off eglot and LSP.
1. Manually run M-x eglot-shutdown.
2. Automatically by (setq eglot-autoshutdown t).

LSP Servers

C/C++ clangd

Install the C/C++ LSP server clangd on Arch Linux.

~ $ sudo pacman -S clang

~ $ clangd --version

Make sure the value of variable eglot-server-programs contains c-mode and c++-mode. All the languages major mode listed here in is enabed by default.

Please open up a C/C++ source file and run M-x eglot. You will find LSP server clangd is automatically launched on your host, as the child process of Emacs. To shutdown the LSP server, just run M-x eglot-shutdown.

However, we'd like to automatically launch the LSP server upon a C/C++ source file is loaded. Add the following contents init.el.

(require 'eglot)
(add-hook 'c-mode-hook 'eglot-ensure)
(add-hook 'c++-mode-hook 'eglot-ensure)

lua-lsp

deprecated; see lua-language-server

This section assumes we are in Arch Linux system with mutliple Lua versions installed (e.g. extra/lua53 and extra/lua54).

To install lua-lsp, we should install Lua package manager luarocks first.

~ $ sudo pacman -S luarocks

Default Luarocks config (just a Lua script) is located under /etc/luarocks. We can inspect the full configuration of Lua 5.3 as below.

~ $ luarocks --lua-version 5.3 config --scope user
config_files = {
   nearest = "/etc/luarocks/config-5.3.lua",
   system = {
      file = "/etc/luarocks/config-5.3.lua",
      found = true
   },
   user = {
      file = "/home/outsinre/.luarocks/config-5.3.lua",
      found = true
   }
}
--
-- skipped
--
rocks_trees = {
   {
      name = "user",
      root = "/home/outsinre/.luarocks"
   },
   {
      name = "system",
      root = "/usr"
   }
}

rocks_trees refers the the root dir storing rocks contents. To avoid conflicts, we'd better configure different rocks tree for different Lua versions. Take Lua 5.3 for example, we will configure its rocks tree to ~/.luarocks53.

Let's change the current user's rocks tree for Lua 5.3.

~$ mkdir -p $HOME/.config/luarocks/
~$ cp /etc/luarocks/config-5.3.lua .config/luarocks/

~ $ vim .config/luarocks/config-5.3.lua
>{ name = "user", root = home .. "/.luarocks53" };

~ $ luarocks --lua-version 5.3 config --scope user

Finally, we install lua-lsp by explicitly provide the rocks tree.

# please omit option '--local'
~ $ luarocks --lua-version 5.3 --tree ~/.luarocks53 --server=http://luarocks.org/dev install lua-lsp

Now, we add the lua-lsp binary to PATH.

# ~/.bash_profile

eval $(luarocks --lua-version 5.3 path)

The built-in method above is provided by Luarocks but it exports LUA_PATH and LUA_CPATH as well. We may just want to set "PATH".

~ $ nano -w ~/.bash_profile
>PATH=$HOME/.luarocks53/bin:$PATH

To automatically launch lua-lsp, add the following lines to init.el.

(require 'eglot)
(add-hook 'lua-mode-hook 'eglot-ensure)

Repeat above steps for other Lua versions.

lua-language-server

lua-lsp has not been updated for years, and so we swtich the lua-language-server.

Let's install "lua-language-server" manually per-user.

~ $ wget https://github.com/LuaLS/lua-language-server/releases/download/3.6.19/lua-language-server-3.6.19-linux-x64.tar.gz

~ $ mkdir -p ~/.local/share/lua-language-server

~ $ tar -xzvf lua-language-server-3.6.19-linux-x64.tar.gz -C ~/.local/share/lua-language-server

We cannot add lua-language-server to "PATH" directly, instead we must create a wrapper, namely ~/bin/lsp-lua.

#!/usr/bin/env bash

exec "${HOME}/.local/share/lua-language-server/bin/lua-language-server" "--configpath=${HOME}/.local/share/lua-language-server/settings.lua" "$@"

Now let's have a try.

~ $ lsp-lua --version
3.6.19

lua-language-server by default is not added to eglot. So, we should manually set it up in init.el.

(add-to-list 'eglot-server-programs
             '(lua-mode "~/bin/lsp-lua"))
(add-hook 'lua-mode-hook 'eglot-ensure)

Restart Emacs and open a Lua file, eglot would automatically invokes ~/bi/lsp-lua to launch lua-language-server.

Before closing this section, let's configure lua-language-server a bit. Remember that, ~/bin/lsp-lua has specified the option --configpath to configurations from a Lua file settings.lua as follows. Plese read Settings for a complete list of parameters.

return {
    Lua = {
        ["runtime.version"] = "Lua 5.1",
        completion = {
            callSnippet = "Both",
            displayContext = 1,
            keywordSnippet = "Both",
        },
        ["hint.enable"] = true,
        ["format.enable"] = true,
    }
}

To format current buffer, just run M-x: eglot-format-buffer.

OpenResty

2023-01-02T00:00:00+00:00

Grab Sources
Verify Sources
Build Binary
Quickstart
macOS

This post presents the way to manually build OpenResty on archlinux. For a quick playaround, try with Docker image.

Grab Sources

Download sources.

# source
14:47:22 outsinre@zhtux ~/misc$ curl -O https://openresty.org/download/openresty-1.21.4.1.tar.gz

# signature
14:47:33 outsinre@zhtux ~/misc$ curl -O https://openresty.org/download/openresty-1.21.4.1.tar.gz.asc

Verify Sources

Import and sign the public PGP key of Yichun Zhang A0E98066

14:47:33 outsinre@zhtux ~/misc$ gpg --recv-key A0E98066

# locally sign the public key
15:02:08 outsinre@zhtux ~/misc$ gpg --edit-key A0E98066
gpg (GnuPG) 2.2.40; Copyright (C) 2022 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


pub  rsa2048/B550E09EA0E98066
     created: 2013-02-18  expires: never       usage: SC
     trust: unknown       validity: unknown
sub  rsa2048/9C86AC14130DD88B
     created: 2013-02-18  expires: never       usage: E
[ unknown] (1). Yichun Zhang (agentzh) <agentzh@gmail.com>

gpg> lsign 1

pub  rsa2048/B550E09EA0E98066
     created: 2013-02-18  expires: never       usage: SC
     trust: unknown       validity: unknown
 Primary key fingerprint: 2545 1EB0 8846 0026 195B  D62C B550 E09E A0E9 8066

     Yichun Zhang (agentzh) <agentzh@gmail.com>

Are you sure that you want to sign this key with your
key "Zhu Cac (gnupg-rsa-4096) <zhucac@outlook.com>" (38DA5BA4AABFC8FA)

The signature will be marked as non-exportable.

Really sign? (y/N) y

gpg> save

Verfiy sources.

15:07:44 outsinre@zhtux ~/misc$ gpg --verify openresty-1.21.4.1.tar.gz.asc openresty-1.21.4.1.tar.gz
gpg: Signature made Tue 17 May 2022 03:29:41 AM CST
gpg:                using RSA key B550E09EA0E98066
gpg: Good signature from "Yichun Zhang (agentzh) <agentzh@gmail.com>" [full]

Build Binary

Extract sources.

15:17:29 outsinre@zhtux ~/misc$ tar -xzpvf openresty-1.21.4.1.tar.gz
15:17:41 outsinre@zhtux ~/misc$ cd openresty-1.21.4.1

15:19:31 outsinre@zhtux ~/misc/openresty-1.21.4.1$ ls -la --color=auto
total 116
drwxrwxr-x  5 outsinre outsinre  4096 May 16  2022 .
drwxr-xr-x  6 outsinre outsinre  4096 Jan  2 15:17 ..
-rw-rw-r--  1 outsinre outsinre 22924 May 16  2022 COPYRIGHT
-rw-rw-r--  1 outsinre outsinre  8974 May 16  2022 README-windows.txt
-rw-rw-r--  1 outsinre outsinre  4689 May 16  2022 README.markdown
drwxrwxr-x 46 outsinre outsinre  4096 May 16  2022 bundle
-rwxrwxr-x  1 outsinre outsinre 51290 May 16  2022 configure
drwxrwxr-x  2 outsinre outsinre  4096 May 16  2022 patches
drwxrwxr-x  2 outsinre outsinre  4096 May 16  2022 util

Configure the build. We enable the debugging log and dry run. Make sure perl 5.6.1+, libpcre, libssl is available on the system.

21:21 outsinre@zhtux ~/misc/openresty-1.21.4.1$ ./configure --help
54:41 outsinre@zhtux ~/misc/openresty-1.21.4.1$ ./configure --with-debug --with-pcre-jit --dry-run

54:41 outsinre@zhtux ~/misc/openresty-1.21.4.1$ ./configure --with-debug --with-pcre-jit

Build from sources.

16:16:43 outsinre@zhtux ~/misc/openresty-1.21.4.1$ make
16:16:43 outsinre@zhtux ~/misc/openresty-1.21.4.1$ sudo make install

Set Env. Program openresty is a symbolic link to program nginx.

# .bash_profile

export OPENRESTY_PREFIX=/usr/local/openresty
export PATH=$OPENRESTY_PREFIX/bin:$OPENRESTY_PREFIX/nginx/sbin:$PATH

Quickstart

Check availability.

16:31:37 outsinre@zhtux ~$ openresty -V
nginx version: openresty/1.21.4.1
built by gcc 12.2.0 (GCC)
built with OpenSSL 3.0.7 1 Nov 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_ssl_module

OpenResty ships with program resty (written with Perl) that can execute OpenResty Lua script.

16:32:16 outsinre@zhtux ~$ resty -e 'local var = 1; print(var)'
1

To quickly open a simple HTTP server, follow OpenResty Getting Started.

macOS

Check OpenResty brew tap.

Install.

~ $ brew tap openresty/brew
~ $ brew install openresty-debug

~ $ type openresty

Export Env.

# ~/.bash_profile

export OPENRESTY_PREFIX=$(brew --prefix openresty/brew/openresty-debug)
PATH="$OPENRESTY_PREFIX/nginx/sbin:$OPENRESTY_PREFIX/luajit/bin/:$PATH"

GHA

2023-01-01T00:00:00+00:00

Workflow
Runner
Variables
Persist Output
Tokens and Secrets
Security

Workflow

Workflows are YMAL files located under the .github/workflows directory of your GitHub repo.

A workflow has one or more jobs. Each job has one or more steps that execute on a runner. Each step is a user-defined Shell script or well-known actions like actions/checkout. Workflows can be triggered by one or more events.

Here is an quick illustration of core concepts.

Check workflow examples at https://github.com/outsinre/kong/tree/master/.github/workflows.

Please refer to Understanding GitHub Actions.

Runner

Each GitHub-hosted runner is a new virtual machine (VM) hosted by GitHub with the runner application and other tools preinstalled, and is available with Ubuntu Linux, Windows, or macOS operating systems. When you use a GitHub-hosted runner, machine maintenance and upgrades are taken care of for you.

Each workflow job has its own standalone runner and specifies the types of runners by runs-on.

See more details about runner at Using GitHub-hosted runners.

Variables

Variables can be categorized into 2 types:

Default variables.

Default environment (Shell) variables.

      
${{ env.GITHUB_REPOSITORY }}
      
$GITHUB_REPOSITORY

Default context variables.
```
      
${{ github.repository }}
      
```

Custom variables.
1. Custom environment (Shell) variables defined via the env key.
```
env:
  my_var: foo
```
2. Custom configuration variables defined at the "Settings" of organization, repository, or environment level.

To access variables, we use context in the form as follows. Most commonly used contexts are github and env.

${{ context.property }}

# example
${{ env.GITHUB_REPOSITORY }}

Specially, the env context has the following two forms:

Workflow syntax as mentioned above.
Shell syntax, like $GITHUB_REPOSITORY. This form is usually used within the run key.

The difference between the two forms is that the workflow syntax is interpolated before the workflow jobs are sent to a runner, while the Shell syntax is interpolated at runtime.

Persist Output

We can share values among steps of a job, or even among jobs.

To share values among steps of a job, please use GITHUB_ENV.

GITHUB_ENV is an environment variable storing the pathname of a environment file on the runner. We can access it via Shell syntax in run steps and workflow syntax in non-Shell context. See Using the env context to access environment variable values.
```
   
# define in a step
echo "MY_VAR=myValue" >> $GITHUB_ENV

# reference in another step
${MY_VAR}
# -or-
${{ env.MY_VAR }}
   
```
However, in the well-know actions/github-script, we access the environment variables via process.env.my_var.

To share values among jobs, please try the following methods.

Use environment variable GITHUB_OUTPUT to declare the output of a job. Similar to GITHUB_ENV above, but it can only be accessed with workflow syntax.

      
# define output1 in a job
run: echo "my_var=hello" >> "$GITHUB_OUTPUT"
output1: ${{ steps.<step-id>.outputs.my_var }}

# reference output1 in another job
${{needs.<job_id>.outputs.output1}}
      

Of course, GITHUB_OUTPUT can also be used to share values among steps of a job.

      
# reference output1 in another step within the same job
${{ steps.<step-id>.outputs.output1_var }}

Storing workflow data as artifacts.

Tokens and Secrets

GitHub tokens, namely Personal Access Tokens, are opaque strings (credentials) defined for your GitHub account at https://github.com/settings/tokens.
GitHub secrets are names declared at the "Settings" of organization, repository, or repository environment level. Any values (e.g. JSON string) can be assigned to secrets. But to operate the repo, we assign tokens to secrets.

Once set, secrets are available to workflows via $.

There is a special secret $ which is automatically issued by GitHub when a workflow is triggered, and you don not need to create this explicitly. It is different from the personal access tokens. Refer to Automatic token authentication .

Refer to Using secrets in GitHub Actions.

Security

Kong does not care about security.

SELinux

2022-05-15T00:00:00+00:00

Utilities
Status
Modes
Audit Logs
Boolean Values
Ports

This post only serves as a quick reference for SELinux. Read whatever for the design and principle behind.

Utilities

Make sure the following packages are installed.

~ $ dnf provides /usr/sbin/sestatus
policycoreutils
~ $ dnf list policycoreutils

~ $ dnf repoquery --provides /usr/sbin/getenforce
libselinux-utils
~ $ dnf list libselinux-utils

~ $ dnf provides /usr/sbin/semanage
policycoreutils-python-utils
~ $ dnf list policycoreutils-python-utils

Status

~ $ getenforce
Enforcing

~ $ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actua

~ $ seinfo
Statistics for policy file: /sys/fs/selinux/policy
Policy Version:             31 (MLS enabled)
Target Policy:              selinux
Handle unknown classes:     allow
  Classes:             132    Permissions:         464
  Sensitivities:         1    Categories:         1024
  Types:              4938    Attributes:          252
  Users:                 8    Roles:                14
  Booleans:            335    Cond. Expr.:         380
  Allow:            110525    Neverallow:            0
  Auditallow:          161    Dontaudit:         10253
  Type_trans:       241591    Type_change:          87
  Type_member:          35    Range_trans:        5781
  Role allow:           37    Role_trans:          420
  Constraints:          72    Validatetrans:         0
  MLS Constrain:        72    MLS Val. Tran:         0
  Permissives:           0    Polcap:                5
  Defaults:              7    Typebounds:            0
  Allowxperm:            0    Neverallowxperm:       0
  Auditallowxperm:       0    Dontauditxperm:        0
  Ibendportcon:          0    Ibpkeycon:             0
  Initial SIDs:         27    Fs_use:               34
  Genfscon:            108    Portcon:             642
  Netifcon:              0    Nodecon:               0

Modes

SELinux has three modes:

enforcing: security policy is enforced.
permissive: security policy is disabled but log events.
disabled: SELinux daemon is not loaded at all.

We can changed the modes permanently by /etc/selinux/config and reboot. Alternaitvely, we can temporarily switch between enforcing and permissive on CLI.

# enforcing
~ $ sudo setenforce 1

# permissive
~ $ sudo setenforce 0

Audit Logs

If SELinux denies an access (e.g. a file), the denial event is logged to /var/log/audit/audit.log. Sometimes, we call it "Access Vector Cache (AVC) denial".

The log may reveal what is the cause of our applications.

Boolean Values

SELinux have many built-in boolean values.

List boolean values by getsebool.

~ $ getsebool -a | grep [h]ttpd_can_network_connect
httpd_can_network_connect ---> off

We can set a boolean value by setsebool.

~ $ setsebool -P httpd_can_network_connect 1
# -or-
~ $ setsebool -P httpd_can_network_connect=1

Ports

Each port number, together with protocol, belong to a perticular service (i.e., label). For example, TCP port 22 is for SSH service (TCP).

List all services.

~ $ sudo semanage port -l
...
ephemeral_port_t               tcp      32768-60999
ephemeral_port_t               udp      32768-60999
...
ssh_port_t                     tcp      22
...
unreserved_port_t              sctp     1024-65535
unreserved_port_t              tcp      61000-65535, 1024-32767
unreserved_port_t              udp      61000-65535, 1024-32767

There are two special services, namely ephemeral_port_t (32768-60999) and unreserved_port_t (1024-32767,61000-65535).

~ $ seinfo --portcon 32768

Portcon: 3
   portcon sctp 1024-65535 system_u:object_r:unreserved_port_t:s0
   portcon tcp 32768-60999 system_u:object_r:ephemeral_port_t:s0
   portcon udp 32768-60999 system_u:object_r:ephemeral_port_t:s0

~$ seinfo --portcon 32767

Portcon: 3
   portcon sctp 1024-65535 system_u:object_r:unreserved_port_t:s0
   portcon tcp 1024-32767 system_u:object_r:unreserved_port_t:s0
   portcon udp 1024-32767 system_u:object_r:unreserved_port_t:s0

By default, ephemeral ports are allowed but unreserved ports are disallowed. To allow an unreserved port, we just need to add it to a specific other services.

For example, here is the default allowed HTTP ports:

~ $ sudo semanage port -l | grep [h]ttp_port_t
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

To enable/disable a port, we add/delete it from a type and protocol combination.

# add a port
~ $ sudo semanage port -a -t http_port_t -p tcp 32767

# delete a port
~ $ sudo semanage port -d -t http_port_t -p tcp 32768

systemd

2021-07-19T00:00:00+00:00

systemd manager
Hierarchy of Control Groups
systemctl
1. systemd drop-in
journalctl
User Unit

systemd manager

Systemd (system daemon) is a manager capable of managing both services and the Linux system.

The daemon locates at /lib/systemd/systemd, and /sbin/init is just a symbolic link:

~$ ll /sbin/init
lrwxrwxrwx 1 root root 22 May 18 18:32 /sbin/init -> ../lib/systemd/systemd

At the very boot of the system, systemd runs as the PID 1, namely the system manager instance:

~$ ps -1
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:24 /sbin/init

When a user login, a user manager instance is started for services of that particular account by user@.service:

~ $ systemctl cat user@.service

~ $ id -u
~ $ systemctl show user@1000.service

~ $ systemctl status user@1000.service

~ $ ps -eF | grep -i [s]ystemd
jim     572       1  0  4081  9208   7 Jun20 ?        00:00:00 /usr/lib/systemd/systemd --user

Hierarchy of Control Groups

.slice controls the resources
User processes may be started by:
1. user@.service unit.
2. manual startup, like sshd, displayer manager (e.g. initx/gdm), etc.

~ $ systemd-cgls

Control group /:
-.slice
├─user.slice
│ └─user-1000.slice
│   ├─user@1000.service
│   │ ├─app.slice
│   │ │ ├─dconf.service
│   │ │ │ └─10610 /usr/lib/dconf-service
│   │ │ ├─thunar.service
│   │ │ │ └─47699 /usr/bin/Thunar --daemon
│   │ │ ├─at-spi-dbus-bus.service
│   │ │ │ ├─ 816 /usr/lib/at-spi-bus-launcher
│   │ │ │ ├─2094 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
│   │ │ │ └─2098 /usr/lib/at-spi2-registryd --use-gnome-session
│   │ │ └─dbus.service
│   │ │   ├─  648 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
│   │ │   └─47703 /usr/lib/xfce4/xfconf/xfconfd
│   │ └─init.scope
│   │   ├─572 /usr/lib/systemd/systemd --user
│   │   └─573 (sd-pam)
│   └─session-1.scope
│     ├─   550 login -- jim
│     ├─   579 -bash
│     ├─   589 /usr/bin/ssh-agent -s
│     ├─   604 /bin/sh /usr/bin/startx
│     ├─   622 xinit /home/jim/.xinitrc -- /home/jim/.xserverrc :0 vt1 -keeptty -auth /tmp/serverauth.RW8Bv770dV
│     ├─   623 /usr/lib/Xorg -nolisten tcp -nolisten local -keeptty :0 vt1 -keeptty -auth /tmp/serverauth.RW8Bv770dV vt1
│     ├─   642 awesome
│     ├─   666 xautolock -detectsleep -time 3 -locker slock -corners 00+0 -cornerdelay 5 -notify 5 -notifier notify-send -u critical -t 10000 -- 'LOCKING screen in 5 seconds'
│     ├─   675 fcitx
│     ├─   680 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --config-file /usr/share/fcitx/dbus/daemon.conf
│     ├─   684 /usr/bin/fcitx-dbus-watcher unix:abstract=/tmp/dbus-y7OCHgrxGl,guid=98e8dc5f3eed24ce2b4a183c60cf475a 680
│     ├─  2091 emacs --daemon
│     ├─ 16367 goldendict
│     ├─ 94741 /usr/lib/firefox/firefox
│     ├─ 94901 /usr/lib/firefox/firefox -contentproc -childID 2 -isForBrowser -prefsLen 210 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─ 94935 /usr/lib/firefox/firefox -contentproc -childID 3 -isForBrowser -prefsLen 5036 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─ 95053 /usr/lib/firefox/firefox -contentproc -parentBuildID 20210623174607 -prefsLen 6216 -prefMapSize 239063 -appdir /usr/lib/firefox/browser 94741 true rdd
│     ├─114937 /usr/lib/firefox/firefox -contentproc -childID 19 -isForBrowser -prefsLen 9718 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─128993 /usr/lib/firefox/firefox -contentproc -childID 50 -isForBrowser -prefsLen 11032 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─155942 /usr/lib/firefox/firefox -contentproc -childID 64 -isForBrowser -prefsLen 11034 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─157055 /usr/lib/firefox/firefox -contentproc -childID 69 -isForBrowser -prefsLen 11034 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─161533 /usr/lib/firefox/firefox -contentproc -childID 70 -isForBrowser -prefsLen 11035 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─165017 st
│     ├─165018 /bin/bash
│     ├─165033 tmux new -s arch
│     ├─165035 tmux new -s arch
│     ├─165036 -bash
│     ├─165051 ssh ora1
│     ├─165855 -bash
│     ├─165867 -bash
│     ├─165879 ssh ora2
│     ├─166174 /usr/lib/firefox/firefox -contentproc -childID 95 -isForBrowser -prefsLen 11070 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─166322 /usr/lib/firefox/firefox -contentproc -childID 97 -isForBrowser -prefsLen 11070 -prefMapSize 239063 -parentBuildID 20210623174607 -appdir /usr/lib/firefox/browser 94741 true tab
│     ├─167307 /usr/lib/thunderbird/thunderbird
│     ├─177691 -bash
│     ├─177791 man systemd
│     ├─177801 less
│     ├─177869 man user@.service
│     ├─177879 less
│     ├─177932 -bash
│     ├─178007 systemd-cgls
│     └─178008 less
├─init.scope
│ └─1 /sbin/init
└─system.slice
  ├─systemd-udevd.service
  │ └─431 /usr/lib/systemd/systemd-udevd
  ├─polkit.service
  │ └─94871 /usr/lib/polkit-1/polkitd --no-debug

systemctl

# show installed units on the system
~ $ systemctl list-unit-files [pattern]

# show all units in memory
~ $ systemctl
# filter units in memory
~ $ systemctl --type=help
~ $ systemctl --state=help
~ $ systemctl --type=service --state=running [list-units [pattern]]

# reload all installed unit files (e.g. nginx.service)
~ $ systemctl daemon-reload
# reload daemon config (e.g. nginx.conf)
~ $ systemctl reload [pattern]

# view unit configuration
# mostly default from 'systemd' config
~ $ systemctl show [pattern]
# view unit file
~ $ systemctl cat [pattern]

# manage a unit
~ $ systemctl enable UNIT
~ $ systemctl enable PATHNAME
~ $ systemctl status/start/stop/restart/disable/mask UNIT
~ $ systemctl enable name@sub.service

# dependency
~ $ systemctl [--reverse] list-dependencies UNIT
~ $ systemctl --with-dependencies [--reverse] status PATTERN

systemd drop-in

It is not unusual when we do not want to modify the shipped unit file but want customization.

For a particular unit foo.service, systemd supports merging drop-in files *.conf stored under the drop-in directory foo.d. Drop-in directories can be located in /etc/systemd/system, /usr/lib/systemd/system or /run/systemd/system.

We can manually create the drop-in directory and then add or remove drop-in files. If the drop-in directory is empty, then the unit would be cleared!

Alternatively, systemctl edit automatically create and/or remove the drop-in directory and drop-in files. After editing, it would call systemctl daemon-reload automatically.

The example belows add a few environment variables to the unit file.

~ $ systemctl edit kong-enterprise-edition

~ $ systemctl cat kong-enterprise-edition
...

# /etc/systemd/system/kong-enterprise-edition.service.d/override.conf
[Service]
Environment=KONG_NGINX_DAEMON=off
Environment=KONG_DATABASE=off

If somehow, we want to update the ExecStart, we must firstly prepend an empty directive like the following. Otherwise, there would be two conflicted ExecStart.

# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375 --containerd=/run/containerd/containerd.sock

We can optionally add --full, --force or --runtime to systemctl edit. For details, please refer to man page of systemctl.

If somehow, we want to discard all drop-in directories and drop-in files, then run the command as follows.

~ $ systemctl revert kong-enterprise-edition
Removed /etc/systemd/system/kong-enterprise-edition.service.d/override.conf.
Removed /etc/systemd/system/kong-enterprise-edition.service.d.

journalctl

~ # journalctl -xfe
~ # journalctl -u nginx

User Unit

User unit should be under ~/.config/systemd/user/.

Firstly, let user unit know the user PATH. At the end of ~/.bash_profile, put the following line.

# ~/.bash_profile
systemctl --user import-environment PATH

Check if user unit know the PATH

~ $ systemctl --user show-environment

Now enable and start user unit.

~ $ systemctl --user daemon-reload

~ $ systemctl --user status weechat-headless.service
~ $ systemctl --user enable weechat-headless.service

~ $ systemctl --user start weechat-headless.service

Public Cloud Infrastructure

2021-07-18T00:00:00+00:00

General Components
Oracle
AWS
1. amazon-linux-extras
Aliyun

This will presents a general setup on public cloud platform (e.g. AWS) to launch a web server.

General Components

Pay attention to the charging items:

Compute instance, like EC2.
1. Running time.
VPC (Virtual Private Cloud) Network
1. Elastic IP.
2. Firewall like Security Group (zone) and Compute iptables.
3. Bandwidth or traffic.
Storage, like EBS (Elastic Block Storage).
1. Compute Storage
2. Image storage

Oracle

THINK TWICE before rebooting a instance or detach block volume from a instance. Check Instance Configuration for details.

Glossary

Oracle Cloud Infrastructure (OCI):

Tenancy. Subscription account, the root account included.
User. A login user account, including the Oracle federated OracleIdentityCloudService.
Compartment. A group of resources for different projects, like Finance compartment, Logging compartment etc. Think of compartment as Linux Cgroup or Tess cluster.

See link for all resources within the root compartment.
Tag namespace. A group of tags applied to resources. Think of tag namespace as Tess namespace - just another dimension of resources organization.

Always-free

Always free resources does not limit running time or network traffic:

2 instances.
2 public IPs. Can be changed for free.
200 GB volume storage (boot volume + block volume) plus 5 volume backups.
2 VCN (Virtual Cloud Network).
Cloud Shell. Cloud Shell is a browser-based VM and separated from resournces above (e.g. 50 GB standalone storage). We can manage OCI instances through the Cloud Shell.

Apart from Cloud Shell, we also have 'Web Shell'. Web Shell is a browser-based interface to a particular instance, usually for management.

Instance Configuration

Create an instance
1. Select resources with "Always Free Eligible" label.
2. Select Oracle Linux is recommended. CentOS is deprecated.
3. Instance shape is "VM.Standard.E2.1.Micro".
4. Upload SSH public key.
Configure VCN. Each VCN includes a VNIC (Virtual Network Interface Card) attached to the instance. The core part of VCN and VNIC is:
1. Subnet for internal network.
2. Public IP address. Public IP assigned may be blocked, but we can always got a new one free of charge at: Compute - Instances - Instance Details - Attached VNICs - VNIC Details - IPv4 Addresses.
3. Security List (firewall). Add "ingress rules" to the default "security list". By default, only port 22 on ingress is allowed. Leave the "Stateless" alone, otherwise you need to add matching "egress rules". By default, ICMP is blocked; SSH is allowed. Probably, we should enable 80/443.
Block Volume. By default, each instance is assigned 50 GB boot volume. However, each account can have up to 200 GB volume.
1. Create a 50 GB block volume.
2. Attach the block volume. When you "terminate" an instance, the block volume attached remain. Be careful over free quota!
  
  Step 3 and step 4 below is dangerous! Mount the attached block volume only on demand.
3. Login and run the iSCSI connect commands. Attention that the block volume use network connection:
```
~ $ sudo lsof -nP -i tcp:3260
~ $ sudo fdisk -l /dev/sdb
```
4. Either mount the device somewhere (e.g. "/opt") or extend it to root LVM.
  
  To mount the new device by "/etc/fstab", remember to add _netdev and nofail options, as attached block volumes require network access upon boot.
CAUTION: before rebooting the instance or detaching the block volume, please revert step 4 and step 3 above, otherwise we lost contact to our instance, even Web/Cloud Shell would not help!
Optionally, disable OCI plugins, like the Custom Logs Monitoring. OCI plugins communicate with oracle-cloud-agent.service on instance to collect performance metrics, install OS updates, perform other instance management tasks, etc.

Repeat above steps for another free instance. By default, both instances share the same VCN, subnet and security list.

Linux Configuration

If you are unware of the following actions, just skip it.

Disable or enable Orable Cloud Agent.

~ $ systemctl stop oracle-cloud-agent
~ $ systemctl disable oracle-cloud-agent
   
~ $ systemctl stop oracle-cloud-agent-updater
~ $ systemctl disable oracle-cloud-agent-updater

Disable or enable rpcbind.

~ $ systemctl stop rpcbind.service
~ $ systemctl stop rpcbind.socket
~ $ systemctl stop rpcbind.target
   
~ $ systemctl disable rpcbind
~ $ systemctl disable rpcbind.socket

Optionally disable cockpit.socket for Web Shell.

You are recommended to leave it open for online instance debug.

As CentOS is EOL. We should change the mirrorlist.

~ $ sudo sed -i.bak 's/mirrorlist=/#mirrorlist=/g' /etc/yum.repos.d/CentOS-*
~ $ sudo sed -i.bak 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
   
# fix librepo issue
~ $ sudo yum install python3-librepo

From now on, Oracle Linux is recommended.

Firewall. By default, firewalld is enabled and most ports are disallowed! At least enable 80 and 443.

~ $ sudo firewall-cmd --permanent --zone=public --add-service=http --add-service=https
~ $ sudo firewall-cmd --reload
~ $ sudo firewall-cmd --list-all

To allow a port, we should be both enable it in VCN and on instance.

Block Volume

It is not recommended to extend the block volume into existing LVM as the reverting process is complicated! When rebooting the instance or detaching the block volume, we must revert steps below. See Instance Configuration.

Firstly, prepare the disk partition:

~ $ sudo fdisk -l

~ $ sudo parted -a opt /dev/sdb

(parted) mklabel gpt
(parted) unit s
(parted) print free
(parted) mkpart primary xfs 0% 100%
(parted) name 1 blockvolume
(parted) print free
(parted) quit

~ $ sudo fdisk -l
~ $ sudo blkid

Then, we make a filesystem on the block volume as below. This step is optional as the we will use this partition as LVM, and the filesystem created will be erased!

~ $ sudo mkfs.xfs /deb/sdb1
~ $ sudo lsblk
~ $ sudo blkid

Let's inspecting existing LVM. There is only LV created /dev/centosvolume/root storage of which we will extend soon.

# LVM inspection
~ $ sudo pvscan -v ; sudo pvdisplay -v
  PV /dev/sda3   VG centosvolume    lvm2 [<39.06 GiB / 0    free]
  Total: 1 [<39.06 GiB] / in use: 1 [<39.06 GiB] / in no VG: 0 [0   ]

~ $ sudo vgscan -v ; sudo vgdisplay -v
  Found volume group "centosvolume" using metadata type lvm2
  
~ $ sudo lvscan -v ; sudo lvdisplay -v
  ACTIVE            '/dev/centosvolume/root' [<39.06 GiB] inherit

Creating PV:

~ $ sudo pvcreate /dev/sdb1
WARNING: xfs signature detected on /dev/sdb1 at offset 0. Wipe it? [y/n]: y
  Wiping xfs signature on /dev/sdb1.
  Physical volume "/dev/sdb1" successfully created.

~ $ sudo pvdisplay -v

Add the new PV to existing VG:

~ $ sudo vgextend -v centosvolume /dev/sdb1
  Wiping signatures on new PV /dev/sdb1.
  Archiving volume group "centosvolume" metadata (seqno 2).
  Adding physical volume '/dev/sdb1' to volume group 'centosvolume'
  Volume group "centosvolume" will be extended by 1 new physical volumes
  Creating volume group backup "/etc/lvm/backup/centosvolume" (seqno 3).
  Volume group "centosvolume" successfully extended

~ $ sudo vgdisplay -v

Add the new to existing LV:

~ $ sudo lvextend -v --extents +100%FREE /dev/centosvolume/root
  Converted 100%FREE into at most 12799 physical extents.
  Archiving volume group "centosvolume" metadata (seqno 3).
  Extending logical volume centosvolume/root to up to 89.05 GiB
  Size of logical volume centosvolume/root changed from <39.06 GiB (9999 extents) to 89.05 GiB (22798 extents).
  Loading table for centosvolume-root (253:0).
  Suspending centosvolume-root (253:0) with device flush
  Resuming centosvolume-root (253:0).
  Creating volume group backup "/etc/lvm/backup/centosvolume" (seqno 4).
  Logical volume centosvolume/root successfully resized.

~ $ sudo vgdisplay -v
  --- Volume group ---
  VG Name               centosvolume
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               89.05 GiB
  PE Size               4.00 MiB
  Total PE              22798
  Alloc PE / Size       22798 / 89.05 GiB
  Free  PE / Size       0 / 0
  VG UUID               1xOpAZ-6Idw-y2Oa-ELis-d338-8VgM-xyz

  --- Logical volume ---
  LV Path                /dev/centosvolume/root
  LV Name                root
  VG Name                centosvolume
  LV UUID                2r1j88-PO5l-Fqrj-oeGz-T8Ez-TMSV-xyz
  LV Write Access        read/write
  LV Creation host, time localhost, 2021-05-25 17:18:43 +0000
  LV Status              available
  # open                 1
  LV Size                89.05 GiB
  Current LE             22798
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

  --- Physical volumes ---
  PV Name               /dev/sda3
  PV UUID               NnWbVT-DtBY-XUMh-iFnD-JQZD-bXuJ-xyz
  PV Status             allocatable
  Total PE / Free PE    9999 / 0

  PV Name               /dev/sdb1
  PV UUID               yYxOYl-xxHv-EBOf-3jTw-O34v-BmmQ-xyz
  PV Status             allocatable
  Total PE / Free PE    12799 / 0

From output above, we find the LV size is expanded to over 89 GiB. However, the XFS filesystem does not about this storage expansion. Let's do it again:

~ $ df -h
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       361M     0  361M   0% /dev
tmpfs                          403M     0  403M   0% /dev/shm
tmpfs                          403M   41M  362M  11% /run
tmpfs                          403M     0  403M   0% /sys/fs/cgroup
/dev/mapper/centosvolume-root   40G   11G   29G  27% /
/dev/sda2                     1014M  328M  687M  33% /boot
/dev/sda1                      100M  6.9M   93M   7% /boot/efi
tmpfs                           81M     0   81M   0% /run/user/1000

~ $ sudo xfs_growfs /dev/centosvolume/root
meta-data=/dev/mapper/centosvolume-root isize=512    agcount=4, agsize=2559744 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=10238976, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=4999, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 10238976 to 23345152

~ $ df -h
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       361M     0  361M   0% /dev
tmpfs                          403M     0  403M   0% /dev/shm
tmpfs                          403M   41M  362M  11% /run
tmpfs                          403M     0  403M   0% /sys/fs/cgroup
/dev/mapper/centosvolume-root   90G   11G   79G  13% /
/dev/sda2                     1014M  328M  687M  33% /boot
/dev/sda1                      100M  6.9M   93M   7% /boot/efi
tmpfs                           81M     0   81M   0% /run/user/1000

If the root filesystem is ext4, then use resize2fs instead.

AWS

See "AWS_EC2.docx" and "kong-on-aws.md".

amazon-linux-extras

amazon-linux-extras is a mechanism to gurantee the stability of fresh packages on Amazon Linux 2. Currently, only a small number of packages are supported.

It's built upon YUM repository. Whenever a new package is installed through 'amazon-linux-extras', it add a corresponding YUM repository for the package.

Firstly, make sure amazon-linux-extras is installed.

~ $ type amazon-linux-extras

~ $ sudo yum install amazon-linux-extras

~ $ amazon-linux-extras --help

Packages supported by 'amazon-linux-extras' are called topic. List available topics.

~ $ amazon-linux-extras list

Enable a topic (YUM repository). The output will give you instructions how to install the topic.

# Postgres14 client topic
~ $ sudo amazon-linux-extras enable postgresql14

~ $ sudo yum repolist

Install a topic.

~ $ sudo yum clean metadata
~ $ sudo yum install postgresql14

To make it easier, we can enable and install a topic at once.

~ $ sudo amazon-linux-extras install postgresql14

Aliyun

to-do

Linux Time

2021-03-03T00:00:00+00:00

VirtualBox guest OS relies on VirtualBox guest additions to synchronizes time with host. Hence leave this part after VirtualBox guest additions.

In an operating system, the time (clock) is determined by four parts: time value, time standard, time zone, and Daylight Saving Time (DST) if applicable. Especially, RTC is just a bare value on board without any other information attached. It's the time managment tools that control time standard, time zone and DST.

timedatectl is used to control system date and time while hwclock is for hwardware time (RTC). The use of timedatectl requires an active dbus. Therefore, it may not be possible to use this command under a chroot (i.e. during installation). In such cases, you can revert back to the hwclock command or wait for booting into the new system.

As stated before, dual boot with Windows should set Linux system to treat RTC as localtime. If this installation resides in virtual machine, then leave it default.

Check time:

[root@host ~ #] timedatectl set-ntp true
[root@host ~ #] timedatectl status

Here is an example of timedatectl output:

                      Local time: Thu 2017-09-21 16:08:56 CEST
                  Universal time: Thu 2017-09-21 14:08:56 UTC
                        RTC time: Thu 2017-09-21 14:08:56
                       Time zone: Europe/Warsaw (CEST, +0200)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

Don't be confused. Both Local time and Universal time are system time but with different time standards. The third one (RTC) is hardware time value. Obviously, RTC in this example is treated as Universal Time, which can be verified by the last line - RTC in local TZ: no

Set timezone:

[root@host ~ #] ln -sf /usr/share/zoneinfo/Asia/Chongqing /etc/localtime (set system time zone manually)
# or
[root@host ~ #] timedatectl set-timezone Asia/Chongqiong

From the past experience, it's better to set time first and then time zone.

Set time:

[root@host ~ #] hwclock [--systohc | --hctosys] [--utc | --localtime]
[root@host ~ #] timedatectl set-local-rtc 0/1

Tmux

2020-10-24T00:00:00+00:00

Overview
Configuration
Try Tmux
Tmux Sub-commands
Tmux Panes Synchronization
Tmux Command Key Bindings

Overview

Tmux mainly comprises the server daemon and client. The server daemon manages multiple sessions. When a session is created, the server daemon is lauchned automatically. The client displays a session by attach-session. The server and client are separate Tmux processes. They communicate through the socket under $TMPDIR, like "/tmp/tmux-xyz".

A session can create multiple windows like terminal TABs. Each window occupy the whole screen area and can further be splitted into multiple panes (rectangle area on screen). Each window or pane is a separate pseudo terminal.

We can regard Tmux Window and Pane as Emacs Frame and Window.

Configuration

Tmux supports separate configurations to server, session, window or pane by set-option (set).

set -s configures the server.
set configures a specific session.
set -w (alias setw ) configures a specific window.
set -p configures a specific pane. By default, pane inherits the configuration of window if there is not specific config.

Except the server, we can add extra -g to configures the default options. Usually, we add -g in ~/.tmux.conf. The following is a sample:

# /etc/tmux.conf
# ~/.tmux.conf

# copy mode
#set -gw mode-keys vi
set -gw mode-keys emacs

# scrollback
set -g history-limit 100000
set -g mouse on

# bring ssh-agent to tmux
set -g update-environment "SSH_ASKPASS SSH_AUTH_SOCK SSH_AGENT_PID SSH_CONNECTION http_proxy HTTP_PROXY https_proxy HTTPS_PROXY ftp_proxy FTP_PROXY all_proxy ALL_PROXY no_proxy NO_PROXY"

To reflect the newest configuration file update, just source it.

Another interesting repo tmux-colors-solarized.

Try Tmux

Start new session:

~ $ tmux new-session -s session-name

To put Tmux background, we detach by Ctrl-b d shortcut.

List and kill session:

~$ tmux ls
~$ tmux kill-session -t session-name

Attatch to session:

# attach to latest detached session
~ $ tmux a | at | attatch

# attach to a speciic session
~ $ tmux a -t session-name

Tmux Sub-commands

Tmux has many sub-commands, like new-session, kill-session and attach-session above. Use tmux list-commands to show all supported sub-commands.

Many sub-commands has built-in aliases, like new for new-session. We can also just type a few leading characters of a sub-command as long as no ambiguity incurred. For example, attach-session can be expressed as just a single character a or a simple word att.

Once attached to a session, we can use key bindings to trigger sub-commands within the session. We call those key bindings command key bindings. Command key bindings share the same prefix key Ctrl-b. For example, to create a window for the session, use Ctrl-b c.

We can also manually input sub-commands within a session. Use Ctrl-b : to get into command mode, and then input the sub-command. For instance, Ctrl-b : source-file ~/.tmux.conf reloads Tmux configuration on the fly. If you mis-typed a sub-commands, then Ctrl-u and try again. To get out of command mode, just press ESC.

Tmux Panes Synchronization

Sometimes, we'd like to run the same commands in multiple servers. For such requirements, we have ansible/fabric/pssh etc. Alternatively, tmux support panes synchronization, namely synchronize the commands in one pane to others panes in the same window!

To make use this feature, just Ctrol-b : setw synchronize-panes on/off.

Tmux Command Key Bindings

Apart from command mode, Tmux has the copy mode by Ctrol-b [. Copy mode is used to navigate within shell screen in casef we don't have X (i.e. scroll up). Copy mode has its own key bingdings. By default, Tmux uses Emacs-like key bindings and gets out ESC. You can change to Vi style and get out by ENTER.

Check supported key bindings of copy mode by tmux list-keys -T copy-mode (without -T, list all Tmux key bindings). You will find that only part of Emacs or Vi key bindings are supported.

The following are some general Tmux key bingdings and the corresponding sub-commands.

Session operation:

: command mode
[ copy mode

s  list sessions
$  rename a session

Window operation:

c  create window
w  list windows
n  next window
p  previous window
f  find window
&  kill window

Pane operation:

%  vertical split
"  horizontal split

o  swap panes
x  kill pane
+  break pane into window (e.g. to select text by mouse to copy)
-  restore pane from window
⍽  space - toggle between layouts
q (pane numbers will show up on screen. When the numbers show up, press the relevant key, switching to that pane)
{ (Move the current pane left)
} (Move the current pane right)
z toggle pane zoom

Misc operation:

d  detach
t  big clock
?  list shortcuts

macOS

2020-10-12T00:00:00+00:00

Introductory
Shortcuts
Package Installation
General Setup
Shell Setup
gh
Docker Desktop
Homebrew
1. Homebrew Tap
  1. Personal Tap
2. brew edit
UTM

Introductory

macOS (Mac OS X or OS X) is a proprietary OS derived from Berkeley Software Distribution (BSD), and therefore is a Unix-like system. The main difference between macOS and other Unix-like distributions is that macOS is designed with desktop for personal PC.

Shortcuts

# Screenshot
shift + command + 3/4/5

# Lock Screen
ctrl + command + q

Package Installation

Homebrew.
App Store.
Jamf Self Service.

General Setup

touchpad tap with 1 click
Rime
Firefox
iTerm2
https://emacsformacosx.com/
ssh; load-ssh-agent

Shell Setup

Set "bashrc" and "bash_profile" only after the following:

brew install coreutils
type gdate ; type date

brew install xz

brew install bash
type -a bash

cat >>/etc/shells <<EOL
> /opt/homebrew/bin/bash
> EOL

chsh -s /opt/homebrew/bin/bash

openssl version -a
brew install openssl@1.1
/opt/homebrew/Cellar/openssl\@1.1/1.1.1o/bin/openssl version -a
PATH="/opt/homebrew/opt/openssl@1.1/bin:$PATH"

brew install tmux nmap gnupg

# for emacs markdown-preview
brew install pandoc
mkdir -p "$HOME/.local/bin"
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
ln -vs $(type pandoc) "$HOME/.local/bin/markdown"

brew install libpq
PATH="/opt/homebrew/opt/libpq/bin:$PATH"

brew install btop
mkdir -p ~/.config/btop

PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"

gh

brew install gh

Create a GITHUB_TOKEN and then run gh auth login with the token.

Docker Desktop

Mac with Apple silicon
For compatibility, run softwareupdate –install-rosetta.

Homebrew

After installing Homebrew, we can optionally update mirror to speed up package fetching.

Everything of Homebrew is located under $(brew --prefix) (e.g., /opt/homebrew). Especially, the prefix of a formulae is brew --prefix <formulae>.

Get help info.

~ $ brew commands

~ $ brew help install

~ $ brew config

~ $ brew doctor

brew supports installing both CLI and GUI formulae.

CLI
```
~ $ brew install <formulae>
```
GUI
```
~ $ brew install [--cask] <formulae>
```
Option --cask is optinal and brew cask is no longer a supported command.

Sometimes, a formulate offers extra installation options (e.g. modules). For example, we can install OpenResty with --with-slice to include the corresponding modules.

~ $ brew options openresty
--with-iconv
        Compile with ngx_http_iconv_module
--with-postgresql
        Compile with ngx_http_postgres_module
--with-slice
        Compile with ngx_http_slice_module

Update formulae.

~ $ brew update

~ $ brew outdated

~ $ brew upgrade

Latest Homebrew uses API to install formulae, so it must fetch a file from https://formulae.brew.sh/api/formula.json. The file size is over 18M and so may report timeout issue as below.

11:46:09 zachary@Zacharys-MacBook-Pro ~/workspace/resty-redis-cluster$ brew search openresty
curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 5003 milliseconds with 0 bytes received
curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 5002 milliseconds with 0 bytes received
Error: Failure while executing; `/opt/homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/3.6.20-137-g12f3fb8\ \(Macintosh\;\ arm64\ Mac\ OS\ X\ 13.2\)\ curl/7.86.0 --header Accept-Language:\ en --fail --pr
ogress-bar --max-time 5 --retry 3 --location --remote-time --output /Users/zachary/Library/Caches/Homebrew/api/formula.json --time-cond /Users/zachary/Library/Caches/Homebrew/api/formula.json --compressed --silent https://formulae.brew.sh/api/formula.json` exited with 28.
 Here's the output:
curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 5003 milliseconds with 0 bytes received
curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
curl: (28) Operation timed out after 5002 milliseconds with 0 bytes received

Either set a proxy or disable API by HOMEBREW_NO_INSTALL_FROM_API=1.

Homebrew Tap

A tap is a repository from where brew retrieves packages. By default, brew assumes a Github repository.

Official tap list.

~ $ brew tap
homebrew/cask
homebrew/core

Say we have a tap named as user/repo, the repository is https://github.com/<user>/homebrew-<repo> (homebrew prefix). Take homebrew/core for example, the Github URL is https://github.com/homebrew/homebrew-core. The tap is cloned to local disk under directory $(brew –repository)/Library/Taps.

We can add/remove 3rd-party taps on demand.

# assume Github repo
~ $ brew tap <user/repo>
# -or-
~ $ brew tap <user/homebrew-repo>

# Other repo with specific URL
~ $ brew tap <user/repo> <URL>

# remove
~ $ brew untap user/repo

Once added, we can install formulae as in Homebrew.

However, brew tries to install a formulae from official taps first. In case of duplicate formulae names among taps, We can install a formulae from a specific 3rd-party tap.

~ $ brew install user/repo/formulae

Personal Tap

This section shows how to create a 3rd-party tap.

Step 1. Create an empty Github repo as the personal tap, namely homebrew-wrk2. The repo name must follow pattern homebrew-<foo> (homebrew prefix). We will put our formulaes into directory "Formula".

Now, let's add the personal tap to Homebrew.

~ $ brew tap outsinre/wrk2

~ $ brew tap
homebrew/cask
homebrew/core
outsinre/wrk2

~ $ cd $(brew --repository outsinre/wrk2)
~ $ git status

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ tree
.
├── Formula
└── README.md

1 directory, 1 files

~ $ brew tap --eval-all outsinre/wrk2

Step 2. Prepare code release.

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git branch -a
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git checkout wrk2-a64-support
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git diff head~1

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git checkout master
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git merge wrk2-a64-support

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git tag -l
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git tag -a 1.0.0 -m "release 1.0.0" HEAD
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git tag -l
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git show -s 1.0.0

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git push origin head
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git push origin 1.0.0

Go to Github and create release 1.0.0.

Step 3. Create formulae.

~ $ brew create --tap outsinre/wrk2 https://github.com/outsinre/wrk2/archive/refs/tags/1.0.0.tar.gz

Command above puts the newly formulae into our local personal tap outsinre/wrk2. If we don't offer the --tap option, the formulae would, by default, be placed into the official tap (i.e. homebrew/core).

An editor is popped up to edit the new formulae. Please follow wrk.rb and wrk2.rb.

Step 4. Audit the new formulae's adherence to Homebrew house style.

# validate tap
~ $ brew tap --eval-all outsinre/wrk2

# validate formula
~ $ brew audit --strict --online outsinre/wrk2/wrk2

# validate formula more strictly
~ $ brew audit --new outsinre/wrk2/wrk2

outsinre/wrk2/wrk2:
  * 2: col 9: Description shouldn't start with an article.
  * 7: col 3: `license` (line 7) should be put before `head` (line 6)
  * 24: col 12: Ambiguous splat operator. Parenthesize the method arguments if it's surely a splat operator, or add a whitespace to the right of the `*` if it should be a multiplication.
Error: 3 problems in 1 formula detected

Update formulae according to audit results.

~ $ cd $(brew --repository outsinre/wrk2)
~ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ nano -w wrk2.rb

Step 5. Sync local tap with Github.

~ $ cd $(brew --repository outsinre/wrk2)

~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git status
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git add .
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git commit -m "add formulae wrk2"
~ $ /opt/homebrew/Library/Taps/outsinre/homebrew-wrk2$ git push origin head

Step 6. Install the new formulae.

~ $ brew install outsinre/wrk2/wrk2

If a new release is out. We should firstly backup and remove $(brew --repository outsinre/wrk2)/Formula/wrk2.rb and then repeat steps 2 ~ 6 above. Mostly, we only need to update the sha256 and url part.

However, we can also install the from the latest source code if head is defined the formulae.

~ $ brew install --HEAD outsinre/wrk2/wrk2

References:

brew edit

Sometimes, we want to change (version, url, etc.) formulas locally in place. For example, we do not want to wait for upstream update.

The following steps demostrate how to upgrade OpenResty version locally.

Remove the formulae in advance.

~ $ brew uninstall openresty

Edit formula "openresty" in tap "openresty/brew". We change "VERSION" from "1.21.4.1" ro "1.21.4.2rc1".

~ $ brew tap
openresty/brew

~ $ brew edit openresty/brew/openresty
> VERSION = "1.21.4.2rc1"

~ $ brew --repository openresty/brew
/opt/homebrew/Library/Taps/openresty/homebrew-brew

~ $ cd $(brew --repository openresty/brew)

15:12:31 zachary@Zacharys-MacBook-Pro /opt/homebrew/Library/Taps/openresty/homebrew-brew$ git diff
diff --git a/Formula/openresty.rb b/Formula/openresty.rb
index f25142d..46262e2 100644
--- a/Formula/openresty.rb
+++ b/Formula/openresty.rb
@@ -3,7 +3,7 @@ require 'etc'
 class Openresty < Formula
   desc "Scalable Web Platform by Extending NGINX with Lua"
   homepage "https://openresty.org"
-  VERSION = "1.21.4.1".freeze
+  VERSION = "1.21.4.2rc1".freeze
   revision 2
   url "https://openresty.org/download/openresty-#{VERSION}.tar.gz"
   sha256 "0c5093b64f7821e85065c99e5d4e6cc31820cfd7f37b9a0dec84209d87a2af99"

However, we should also update "sha256" of the new version. We can manually download the tarball and calculate "sha256" locally. Alternatively, we let brew calculate it as follows.

~ $ brew fetch --build-from-source openresty/brew/openresty

==> Fetching openresty from openresty/brew
==> Downloading https://openresty.org/download/openresty-1.21.4.2rc1.tar.gz

Downloaded to: /Users/zachary/Library/Caches/Homebrew/downloads/ad1460c199727f56f390f2dfd99817dc8d6bdedd539e3b8fac4efa6eb0519a27--openresty-1.21.4.2rc1.tar.gz
SHA256: ee96bbcf119abe5f1fc90461dd4674bd9397aa3db5544139578d9ace81983fdb
Warning: Formula reports different sha256: 0c5093b64f7821e85065c99e5d4e6cc31820cfd7f37b9a0dec84209d87a2af99

The new "sha256" is printed to stdout. Let's update accordingly.

~ $ brew edit openresty/brew/openresty

15:16:25 zachary@Zacharys-MacBook-Pro /opt/homebrew/Library/Taps/openresty/homebrew-brew$ git diff
diff --git a/Formula/openresty.rb b/Formula/openresty.rb
index f25142d..afdf97c 100644
--- a/Formula/openresty.rb
+++ b/Formula/openresty.rb
@@ -3,10 +3,10 @@ require 'etc'
 class Openresty < Formula
   desc "Scalable Web Platform by Extending NGINX with Lua"
   homepage "https://openresty.org"
-  VERSION = "1.21.4.1".freeze
+  VERSION = "1.21.4.2rc1".freeze
   revision 2
   url "https://openresty.org/download/openresty-#{VERSION}.tar.gz"
-  sha256 "0c5093b64f7821e85065c99e5d4e6cc31820cfd7f37b9a0dec84209d87a2af99"
+  sha256 "ee96bbcf119abe5f1fc90461dd4674bd9397aa3db5544139578d9ace81983fdb"

   option "with-postgresql", "Compile with ngx_http_postgres_module"
   option "with-iconv", "Compile with ngx_http_iconv_module"

We can validate the update according to instructions in section Personal Tap.

If everything is ok, we install the new version.

~ $ brew install openresty/brew/openresty

One side effect is that, local edits may conflicts with future upstream updates!

UTM

UTM is an open source VMM supporting iOS and OSX. What is more, it supports Apple ARM64!

Installation.
Gallery.
After installation, remove and re-add the USB (DVD) drive, otherwise the boot menu break from time to time.
Follow tutorial UTM Ubuntu 20.04.

UTM uses davfs2 to share files with host OS. Here is setup on guest OS:

# clipboard sharing, mouse integration
~ $ apt install spice-vdagent

# file shareing
~ $ apt install spice-webdavd davfs2

# group
~ $ sudo usermod -a -G davfs2 kong
~ $ sudo reboot

# /etc/fstab
http://127.0.0.1:9843   /media/webdav   davfs   _netdev,noauto,noexec,user,uid=kong,gid=kong    0       0

~ $ mkdir -p /media/webdav
~ $ mount /media/webdav
~ $ ls -al ~/.davfs2/

# auto password
# ~/.davfs2/secrets
/media/webdav   kong    kong

~ $ umount /media/wevdav
~ $ mount /media/wevdav

If just use Web browser (e.g. VM has X), then no need to install davfs2.

TeXLive

2020-08-11T00:00:00+00:00

texmf
Overview
Installation
Update
Additional Packages and Fonts

texmf

You will see name texmf everywhere. It stands for TEX and METAFONT (an old language Knuth used to design fonts).

Overview

To setup a TeX system, you need to do finish two tasks:

Install TeX by a distribution, like TeXLive, MiKTeX etc.
Manage TeX system by built-in texconﬁg or tlmgr of TeXLive, the later of which is far more powerful.
1. Install, update, uninstall, backup individual TeX packages.
2. Change installation options like paper size and source location.
You can regard tlmgr as package manager similar to dnf, emerge etc.
Leave TeX installation in a standalone directory. Do not spread it across the LFS.

Installation

OS Package Manager

Installation from package manager is a minimal install and may lack some fundamental TeX packages (error-prone). So, it's not recommended.

CentOS

By default, the XeTeX engin is not installed. Hence, put it on command line too:

~ $ dnf install texlive texlive-xetex

archlinux

~ $ pacman -S texlive-most texlive-lang biber

Net Installer

Net Installer is recommended by TeXLive, but limited by local bandwidth and is prone to network abrupt.

The advantage by net installer is you get the latest updates.

DVD ISO

It's recommended to do a full install from ISO.

ISO contains everything needed to run TeX system and is a fully tested release.

Preparation

Spare at least 10G disk space.
Download the ISO (around 5G).
texmf trees shoulf follow TeX Directory Structure (TDS). A tree usually includes the following top-level sub-directories:
```
tex/
fonts/
metafonts/
metapost/
bibtex/
scripts/
doc/
source/
implementation/
program/
```
Of of all the top-level sub-directories, tex/ is where TeX macros located, with pathname as texmf/tex/format/package/. format can be amstex, latex, plain, texinfo.

Various texmf trees.

TEXMF, TEXDIR, TEXMFROOT (TeXLive release):
  /usr/local/texlive/{2020,2021}

TEXMFDIST, TEXMFMAIN (official TeX tree):
  /usr/local/texlive/2020/texmf-dist

TEXMFLOCAL (system-wide customized tree):
  /usr/local/texlive/texmf-local
TEXMFHOME (per-user customized tree):
  ~/texmf

TEXMFSYSVAR (system-wide runtime cache data):
  /usr/local/texlive/2020/texmf-var
TEXMFVAR (per-user runtime cache data):
  ~/.texlive2020/texmf-var

TEXMFSYSCONFIG (system-wide modiﬁed conﬁguration data):
  /usr/local/texlive/2020/texmf-config
TEXMFCONFIG (per-user modiﬁed conﬁguration data):
  ~/.texlive2020/texmf-config

These TeX variables can be overriden by system environment variables like TEXLIVE_INSTALL_PREFIX and TEXLIVE_INSTALL_TEXDIR. Check install-tl --help for details.

Installation from ISO

~ # mkdir -p /mnt/iso; cd /mnt/iso
~ # mount /path/to/texlive.iso .

# text mode
~ # per ./install-tl

# GUI mode
~ # per ./install-tl -gui

Leave the default settings (full installation) if you dunno what you are doing. There is an option:

create symlinks in standard directories

Please leave it it uncheck, which otherwise may ruin your system bin directories.

It may take half an hour to complete the installation.

TeX environment variables for Unix

There may exist multiple TeX distributions installed alongside. So, it's better to create a current version symlink.

~ # cd /usr/local/texlive/
~ # ln -sv 2020 current 

Mainly, adjust the search path.

PATH=/usr/local/texlive/current/bin/x86_64-linux:$PATH; export PATH
MANPATH=/usr/local/texlive/current/texmf-dist/doc/man:$MANPATH; export MANPATH
INFOPATH=/usr/local/texlive/current/texmf-dist/doc/info:$INFOPATH; export INFOPATH

Check Installation Scheme

tlmgr will reveal everythin about the setup.

~ # tlmgr --help
~ # tlmgr conf

~ # kpsewhich -a texmf.cnf
~ # kpsewhich -var-value=TEXMFHOME

~ # kpsewhich foo.sty
~ # kpsewhich bar.cls

Update

It is highly recommened not to update TeX distribution in place. Instead, you'd better install a new distribution version. Alternatively, please update or install individual packages into TEXMFLOCAL or TEXMFHOME.

Where should we get updates from?

By default, the

after install, set CTAN as source for package updates

option is checked upon installation.

But we can manually set the update repository afterwards:
```
~ # tlmgr option repository http://mirror.ctan.org/systems/texlive/tlnet
```
This tells tlmgr to use a nearby CTAN mirror for future updates.

How to update?

# GUI
~ # tlmgr -gui

# Text
~ # tlmgr update -all -dry-run
~ # tlmgr update -list

~ # tlmgr update -all
~ # tlmgr -repository /local/mirror/tlnet install collection-xetex

~ # tlmgr show collection-latexextra

Additional Packages and Fonts

Additional new or updated packages and fonts are expected to be in TEXMFLOCAL (system-wide) or TEXMFHOME (per-user, RECOMMENDED), independent of TeX distributions.

For changes under TEXMFLOCAL, we should update the search path for Kpathsea, as TEXMFLOCAL is prefixed with double !!in texmf.cnf.

~ # mktexlsr [/usr/local/telive/texmf-local/]
# -or-
~ # mktexlsr [$(kpsewhich -var-value=TEXMFLOCAL)]

~ # kpsewhich foobar.sty

mktexls acutally just creates a file named ls-R under the specified directory.

Without the directory argument, mktexlsr tries updates all ls-R for the installation.
You can also use texhash as it is just a symlink to mktexlsr.

Kickstart

2020-07-25T00:00:00+00:00

Kickstart

Kickstart

Outline

Kickstart 支持在内网设定装机服务器，所有需要装机的设备连接服务器，实现从网络启动，实现自动化装机。本文介绍简化版用例，基于 Kickstart 定制 ISO 文件，实现自动装机。

主要定制包括：

自动装机。
增、删、改内置 rpm 包。
定制系统参数，如 DNS, 网络、启动服务等。

这部分在主系统装完后操作。
装完自动重启。

Preparation

所有的操作在同版本 CentOS 下进行。例如要定制 CentOS 7.8 的 ISO, 则最好在 CentOS 7.8 系统下操作。

yum install rsync createrepo genisoimage isomd5sum pykickstart

genisoimage 替换老旧的 mkisofs.
isomd5sum (implantisomd5) 用于给 ISO 打 md5 值。
pykickstart (ksvalidator) 用于验证 ks.cfg 语法。

Procedures

1 Mount Official 7.8 ISO image

挂载官方 ISO 文件。

lsblk
ll /dev/cdrom

mkdir -p /mnt{cdrom,iso}
mount /dev/cdrom /mnt/cdrom

2 Customize ISO Contents

定制 ISO 内容。

mkdir -p /mnt/iso
rsync -a /mnt/cdrom /mnt/iso

rm -rf /mnt/iso/repodata/*

cp /path/to/my-pkg.rpm /mnt/iso/Packages/
yum install -y --downloadonly --downloaddir=/mnt/iso/Packages/ pkg1 pkg2 pkg3

清空 repodata 目录，后面会重建。
本例主要添加额外安装包到 Packages 目录。

3 Customize comps.xml

对 ISO 改动后，需要重新生成 comps.xml. comps.xml 是 XML 文件，有 2 个大 tag, 即 'group' 与 'environment'. group 定义 rpm 包的集合，里面的 rpm 包紧密相关，而 environment 是 group 的集合，里面包含的 group 全部安装进系统。例如，CentOS 的 'minimal' environment 里，默认只有 core 这个 group. 在系统中通过 'dnf group install' 安装 group 与 environment 集合。

本例中，主要是往 minimal environment 里增加内置 rpm 包，所以需要把这些新的 rpm 包名加到 comps.xml 里。

mkdir -p /mnt/iso/douyu
cp /mnt/cdrom/repodata/*c7-minimal-x86_64-comps.xml /mnt/iso/douyu/c7-minimal-x86_64-comps.xml

# emacs /mnt/iso/douyu/c7-minimal-x86_64-comps.xml
#      <packagereq type="default">my-pkg</packagereq>
#      <packagereq type="default">pkg1</packagereq>
#      <packagereq type="default">pkg2</packagereq>
#      <packagereq type="optional">pkg3</packagereq>

comps.xml 定制说明：

添加 pkg.rpm 包时，需同时添加其依赖包，否则无法安装。
为添加的 rpm 包定义 group.
1. 可以直接修改 'core' group.
2. 可以仿照 core 自定义一个新 group.
本例是直接修改 core. 如果是自定义 group, 则在 environment 环境后，添加 <groupid>my-group</groupid>, 表示把自己的 group 包括进对应的 environment 里，以便安装。没有添加进 environment 里的 group 是不会安装的。
为新 rpm 包添加 'packagereq' tag.

type 参数为 'default' 表示默认会安装（默认打勾），'mandatory' 表示必须要安装，不可取消打勾，'optional' 表示可选安装（默认末打勾）。

4 Generate ks.cfg

Kickstart 配制文件生成方案：

Kickstart 提供了图形界面的配制工具 system-config-kickstart, 可先用图形界面生成、更新 ks.cfg 模版，再作细节优化。
先用图形界面手动安装一遍操作系统。安装完毕，anaconda 会在 /root 下生成 anaconda-ks.cfg 的文件，记录本次手动安装过程所对应的 Kickstart 配制文件。

两种方案中，前者可定制的参数比较全面，后者因为是实操作，更可靠。需要注意的是，如果第 2 种方法是在 VM 里手动安装，则可能存在一些参数与实际物理机安装不同。例如 /root 下的 anaconda-ks.cfg 文件里，往往在分区参数上，分区大小是固定的（part / --fstype="xfs" --size=7167），需要改成自适应模式。

生成 ks.cfg 后，先验证语法：

mkdir -p /mnt/iso/douyu
cp /path/to/ks.cfg /mnt/iso/douyu/

ksvalidator /mnt/iso/douyu/ks.cfg

ks.cfg 主要作用是代替手动安装的步骤，例如语言、键盘、时区、分区等。
%post ... %end 表示安装完毕，重启前的收尾工作。
%packages ... %end 指定安装需要安装的包。
1. 以 @^ 开头的表示安装 environment 里所有的 group.
2. 以 @ 并开表示单独安装某个 group 里的包。
3. 直接以包名开头，表示安装某个包。

ks.cfg 模版参考 '7.8\7.8-ks_auto3.cfg'.

生成 ks.cfg 后，修改启动项模版 legacy BIOS isolinux.cfg 与 EFI grub.cfg.

Legacy BIOS 启动：

# isolinux/isolinux.cfg

timeout 60

label linux
  menu label ^KS Install CentOS 7.8
  kernel vmlinuz
  menu default
  append initrd=initrd.img inst.stage2=hd:LABEL=CentOS7 inst.ks=cdrom:/douyu/ks.cfg quiet

超时改成 60 ms 而非默认的 600 ms.
主要修改地方是，指定 ks.cfg 位置。
inst.stage2 是安装完毕后，硬盘启动位置 label

'hd:LABEL=' 全部改成统一的字符串，可随意设定，但 MUST be the SAME AS THAT of genisoimage -V BELLOW.
inst.ks 是 Kickstart 加载配制文件的位置。

本例中，ks.cfg 直接存于 ISO 文件，实际中可放于其它地方，如 U 盘、硬盘分区、网络地址等。
把此 menu 改成默认的，即加 'menu default' 行。同时注释其它 menu 的 'menu default' 行。官方 ISO 的 'menu default' 在 'label check' menu 上。

参考 '7.8\isolinux.cfg'.

EFI grub.cfg 启动：

# EFI/BOOT/grub.cfg

set default="0"

menuentry 'KS Install CentOS 7' --class fedora --class gnu-linux --class gnu --class os {
	linuxefi /images/pxeboot/vmlinuz inst.stage2=hd:LABEL=CentOS7 inst.ks=cdrom:/douyu/ks.cfg quiet
	initrdefi /images/pxeboot/initrd.img
}

KS 启动项放在菜单第 1 位，而 set default=0 表示默认启动第 1 个菜单。
EFI grub.cfg 部分与 legacy BIOS 部分没有什么区别，主要是增加 ks.cfg 位置。
'hd:LABEL=' 同上。

这里虽然生成了 EFI 的 boot menu, 但是 ks.cfg 里没有创建对应的 ESP 分区，所以是实际无法以 EFI 启动安装。可以先手动用 UEFI 安装一次，再找修改，请参考 UEFI Kickstart centos.org 和 Kickstart Example for EFI-based system

7 Generate repodata

至此，各项准备工作完毕，开始执行关键命令。首先基于 comps.xml 生成 repodata 目录。repodata 目录包含 ISO 里所有文件的索引，comps.xml 等。

cd /mnt/iso

createrepo -g douyu/c7-minimal-x86_64-comps.xml .
ll repodata/

diff douyu/c7-minimal-x86_64-comps.xml repodata/*c7-minimal-x86_64-comps.xml

createrepo 会复制一份 comps.xml 进 repodata 目录，因此自定义的 comps.xml 不要放进 repodata.

8 Generate new ISO image

生成新 ISO 镜像。

cd /mnt/iso

genisoimage -v -joliet-long -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -cache-inodes -T -eltorito-alt-boot -e images/efiboot.img -no-emul-boot -V 'CentOS7' -o /home/jim/workspace/CentOS-7-dy.iso /mnt/iso/

genisoimage 的参数顺序非常重要，不能随变改动。
Legacy BIOS 启动参数，对应 isolinux.cfg.

-b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -cache-inodes -T
UEFI 启动参数，对应 EFI grub.cfg.

-e images/efiboot.img -no-emul-boot.
Legacy BIOS 和 UEFI 被 -eltorito-alt-boot 隔开。

二者皆附带 -no-emul-boot.
最重要的是 -V 参数后的标签必须与上面 isolinux.cfg 或 grub.cfg 里的 'hd:LABEL=' 相同。

9 Hybrid ISO

上面的制作的 ISO 只能以 CD 模式启动，Hybrid 模式 ISO 可同时用于制作 Usb stick 以硬盘模式启动。

The isohybrid utility modifies a an ISO 9660 image generated with mkisofs, genisoimage, or compatible utilities, to be bootable as a CD-ROM or as a hard disk.

isohybrid -v /home/jim/workspace/CentOS-7.8-dy.iso

10 Embed MD5 into ISO

ISO 文件里有一个 section 为空，没有有效内容，因此正好用来填充该 ISO 其它有效内容的 MD5 值。implantisomd5 把 MD5 值注入 ISO 中，会修改 ISO 的内容。对应地，可以用 checkisomd5 命令检验 ISO 文件完整性。

implantisomd5 /home/jim/workspace/CentOS-7.8-dy.iso

checkisomd5 /home/jim/workspace/CentOS-7.8-dy.iso

implantisomd5 只计算原始有效内容的 MD5 值，不是计算整个 ISO 的 MD5 值。同一 ISO 只能注入一次 MD5.

Install New ISO Image

制作完毕 ISO 后开始安装。

如果安装的初始界面出现 +[drm:vmw_host_log [vmwgfx]] *ERROR* Failed to send host log message 日志，是因为 VirtualBox 里 Display 的驱动设置问题，应从 VMSVGA 换成 VBoxVGA, 参考 https://unix.stackexchange.com/q/502540.
安装过程中，可以用 Alt+F1 进主界面，Alt+F2 进 Shell 界面。在 Shell 界面，可以查看 /tmp/{packaging,anaconda}.log 日志，也可以从 /run/install/repo 下找到外部文件。

Reference

https://o-my-chenjian.com/2017/11/20/DIY-A-CentOS7-System/

Todos

https://superuser.com/a/359247 https://superuser.com/a/359247
openssl passwd /etc/shadow hash 原理。
ks.cfg 里如果包含 'auth', 需要 AppStream 里的 'authselect-compat' package. BZ#1640697. 查看 CentOS 8 的 Release Notes.

Terminal Shell Sed

2019-11-30T00:00:00+00:00

I was aimed to be an ICT expertise. However, still today, I don't quite get the basics of ICT origin. This post will help clarify the relations between Terminal, Shell and Sed.

This is long story. Let's begin with Terminal. At the very beginning, the only interface with the giant (as big as a house) computer was punched tape along with an human being operator. Afterwards, terminal was created allowing individuals to interact with computer. Terminal is a physical device like the following one:

Usually a computer allows multiple terminals online. Each terminal is connected to input devices (e.g. keyboard) and output devices (e.g. monitor screen). Hence, a terminal is more like a serial interface/port between the computer and its peripheral devices. Sometimes, we call the physical terminal as console.

As time flies, standalone terminals were outdated and virtual terminal emerged with smaller computer - Personal Compuer (PC). PC has only one screen and one keyboard embedded but allows switching (Ctrl-Alt-Fn) between multiple (6 by default, vt1 - vt6) virtual terminals. Each virtual terminal has a kernel device (/dev/tty1-6) that provides a means of input and output. It is a software terminal instead of a physical one. Most often, a virtual terminal has a login manager in front before an account can interact with his default login shell (i.e. POSIX Bash).

Apart from virtual terminal, we have terminal emulator when X window presents. Terminal emulator is similar to a virtual terminal but managed by an X server instead of directly by the kernel.

We also have pseudo terminal (pts/xy) that is created and owned by a applications like sshd, terminal emulator (e.g. xterm) etc. A pseudo terminal is a device (e.g. file /dev/pts/04), actually consisting of the master side and the slave side, though they appear to a single device name.

When we ssh to a remote server, we establish a secure TCP connection to the remote sshd daemon. We run a remote application through SSH, by default, which is a Shell like /bin/bash if we does not explicitly provide a command. The remote application needs a terminal for I/O, like STDIN/STDOUT/STDERR. So sshd creates a pseudo terminal for the application (e.g. Bash). sshd is associated with the master, while the remote application is associated with the slave. Here is simple data flow:

|        local     |           Internet          |             remote            |
|                  |                             |                               |
| terminal --- ssh | --------------------------- | sshd - master -- slave - bash |
|                  |                             |                               |
|                  |                             |                               |

After that, ssh/sshd transfer data between local terminal and remote pseudo terminal. Without the slave, how would the Bash process get its STDIN/STDOUT/STDERR? Similarly, terminal emulator (e.g. xterm) also creates pseudo terminal upon startup. The emulator connects to the master side while user processes connect to the slave side. The local terminal above probably is a pseudo terminal of terminal emulator nowadays.

Sometimes you may hear of "attach STDIN/STDOUT" (like docker attach). It actually means creating a pipe between local terminal (e.g. xterm) and the remote terminal (e.g. docker container).

Now let's move on to Shell. Terminal is where input and output happen (like typing program names), but Shell is a job manager (desktop manager does the same thing). Nowadays, OS (Linux, Unix etc.) schedules multiple processes concurrently, namely mutli-tasking support. Shell is the multi-tasking interface with end users, capable of starting, stopping, suspending, resuming etc. jobs. Upon login, the default Shell is ready for interaction.

With Shell, we can suspend the current job to background with Ctrl-Z. On the contratry, fg put the first background job foreground running again. Alternatively, bg send the job running background instead of suspending it. To start a program running in background, we can append & to program like program-name &. Different Shells may have different syntax to manage jobs.

Job is distinct from process, but both of them relate to program. A program is a static executable binary while a process is a running program. Job is only meaningful to Shell. Jobs refer to a pipeline of processes that run interactively. For example, ls | head launches two processes but the pipeline defines a single job. Therefore, a job of Shell corresponds to a process group of OS. Shell assign each job a job ID apart from process PID assigned by the OS. To list existing jobs, just run jobs -l like:

user@tux ~ $ sleep 300 &
[1] 21810
user@tux ~ $ sleep 301 &
[2] 21811
user@tux ~ $ sleep 302 &
[3] 21812
user@tux ~/workspace/outsinre.github.com $ jobs -l
[1]  21810 Running                 sleep 300 &
[2]- 21811 Running                 sleep 301 &
[3]+ 21812 Running                 sleep 302 &

Numbers in brackets are job IDs while numbers that follow job IDs are PID. More about job control, please refer to Manual 7.1 Job Control Basics .

BTW, a daemon does not belong to job as it _detach_es from Shell and gets out of Shell management. Process suspended or running in background belong to job. Putting a job background allows starting another job as the Shell is _release_d.

Finally, we go to sed. It is just a sample of the many programs managed by Shell. Nothing more required to clarify.

Refer to the TTY demystified and Using pseudo-terminals (pty) to control interactive programs.

sudo

2019-04-13T00:00:00+00:00

sudo Front End
Installation
Policy Syntax
Configure Policy
Run Command
1. Preserve Env

sudo Front End

sudo allows a permitted user to execute a command as the superuser (by default) or another user (if provided), as specified by a security policy.

sudo is just a front end and depends on a plugin architecture to load different security policies. We can enable plugins in /etc/sudo.conf as below. The default plugin is sudoers.

[root@ip-172-31-34-111 ~]# grep ^Plugin /etc/sudo.conf
Plugin sudoers_policy sudoers.so
Plugin sudoers_io sudoers.so

Once a plugin is enabled, we can configure its security policies. For example, security polices of plugin sudoers is located in file /etc/sudoers or /etc/sudoers.d.

Third party developers can distribute their own plugins and security policies to work seamlessly with the sudo front end. This post only talks about plugin sudoers.

Installation

[root@host ~]# pacman -S sudo

List sudoers policy of a user.

[root@host ~]# sudo -ll -U <user>

Policy Syntax

Check the SUDOERS FILE FORMAT section in the man page of suoers(5).

sudoers policy is prescribed with Extented Backus-Naur Form (EBNF). Do not despair if you are unfamiliar with EBNF; it is fairly simple and take the chance to study it.

An EBNF definition looks like the following.

symbol ::= definition | alternate1 | alternate2

Special characters ? (optional, 0 or 1), * (zero or more), and + (one or more) mean the same thing as regex. Single quotes denote verbatim character string as opposed to a defined symbol (variable) name. The rest part of EBNF of sudoer is left to yourself.

The sudoers file is composed of two types of entries: alias and user specification. The advantage of alias is to group multiple items together and assign a meaningful label. There exist four kinds of aliases, namely 'User_Alias', 'Runas_Alias', 'Host_Alias' and 'Cmd_Alias'. When there exist only few accounts and hostnames concerned, alias is optional.

When a user is matched by multiple entries, they are applied in order but only the last match takes effect. Let's forget about EBNF and alias part (not the focus of this post). The following is a sample of user specification. It is divided into two groups by the = separator. The left side defines the object while the right side defines the actions.

USER HOSTNAME = (RUNAS_USER:RUNAS_GROUP) COMMANDS

USER is the primary key, for which this entry is designated.

It can be a user name (e.g. jim, user ID (#1000), group name (%wheel), group ID (%#10) etc.
HOSTNAME defines the location where this user specification takes effect. The sudoers file can be shared among multiple systems.

This field can be hostname, IP address, network range etc. Attention please; the loopback interface, namely locahost or 127.0.0.1 will be ignored by sudoers.
'(RUNAS_USER:RUNAS_GROUP) COMMANDS' refers to the combination of target user/group and target commands. They comprise an integrated component.
1. '(RUNAS_USER:RUNAS_GROUP)' is an optional field. By default, it is assumed to be '(root:root)'. Usually, 'RUNAS_GROUP' is optional and defaults to that of 'RUNAS_USER'.
2. 'COMMANDS' specifies the list of commands can be executed. By default, a user must authenticate himself by password before executing the command. We can prepend 'COMMANDS' with 'NOPASSWD:'.

Multiple components are separated by comma. Special value 'ALL' matches everything (e.g. any hostnames).

Entry below means on debain host, account jim can run /bin/ls as acount operator; he can also run /bin/kill and /usr/bin/apg as root.

jim debian = (operator) NOPASSWD: /bin/ls, (root) /bin/kill, /usr/bin/apg

Let us see a more aggressive config.

jim ALL=(ALL:ALL) NOPASSWD: ALL

Configure Policy

To modify sudoers policy, please use visudo.
Personally, I'd like to put per-user policy file under directory /etc/sudoers.d/. Usually, we name the file after the user account in question.

To make the directory loaded, add the #includedir directive in /etc/sudoers. As per the man page, sudo will read each policy file in /etc/sudoers.d, skipping filenames that contain a dot '.' ro end with tilde '~'.

Below is an example.

[root@host ~]# visudo -f /etc/sudoers
[root@host ~]# visudo -f /etc/sudoers.d/user
#
gray	localhost	=	(jim) ALL
%wheel	ALL		=	(root) /bin/iptables

Once finished editting, use the option -c to verify syntax.

[root@host ~]# visudo -c -f /etc/sudoers
[root@host ~]# visudo -c -f /etc/sudoers.d/user

Then check the result:

[root@host ~]# sudo -ll -U <user>

Run Command

# run as root
[user@host ~]$ sudo <command>

# run as specific user
[userhost ~]$ sudo -u <target-user> <command>

If the command to ran is to edit a protected file, sudoedit is preferred than sudo vim.

Preserve Env

Sometimes it is useful to bring current Envs to target user (e.g. root) and commands, so that we don't bother to set again. There are multiple methods to achieve this.

Set an Env in current user.

[ec2-user@ip-172-31-34-111 ~]$ export MY_VAR=123

[ec2-user@ip-172-31-34-111 ~]$ echo $MY_VAR
123

[ec2-user@ip-172-31-34-111 ~]$ printenv | grep MY_VAR
MY_VAR=123

Three methods as below.

Globally set by plugin sudoers. By default, /etc/sudoers enables option env_reset to reset Envs so that commands are ran in a new, minimal environment, including TERM, PATH, HOME, MAIL, SHELL, LOGNAME, USER, USERNAME, etc.

We can add to the default list new Envs by option env_keep and/or env_check like below. Envs in env_keep will be preserved while env_check will remove Envs that is unsafe. Unsafe Envs refer to those Envs that contain character / or %.
```
Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
   
Defaults    env_keep += "MY_VAR1 MY_VAR2"

Defaults    env_check = "MY_VAR1 MY_VAR2"
```
Alternatively, we can even prefix env_reset with ! to disable the default behaviour, so that all Envs are preserved. But this is not recommended.
```
Defaults !env_reset
```
Setting by security policies is globally available and applies to all scenarios.

Refer to 'Command environment' section of sudoers man page.
Set sudo option -E, --preserve-env[=list] to sudo. We can restrict Envs by opt-arg =list.

This method is convenient, especially for user-defined Envs.

Set directly on CLI.

[ec2-user@ip-172-31-34-111 ~]$ sudo [env] MY_VAR=$MY_VAR printenv | grep MY_VAR
MY_VAR=123

The env is optional but suggested to be present when preserving default Envs like 'PATH'.

[ec2-user@ip-172-31-34-111 ~]$ sudo printenv | grep ^PATH
PATH=/sbin:/bin:/usr/sbin:/usr/bin

[ec2-user@ip-172-31-34-111 ~]$ sudo env "PATH=$PATH" printenv | grep ^PATH
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin

This method is stupid but powerful.

Before closing this section, please be cautious that bringing current Envs to target user/command is dangerous. For example, sudo by default enables option secure_path to restrict the PATH as below, so that only secure PATH is included. That is the reason we need to add env when preserving PATH in method 3 above.

[root@ip-172-31-34-111 ~]# grep secure_path /etc/sudoers
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

Of course, if you'd like, updating secure_path policy is also fine.

ZooKeeper and Kafka

2019-04-11T00:00:00+00:00

ZooKeeper ABCs
Quorum
Kafka ABCs
Deployment
ZooKeeper Replicated Mode
Kafka

ZooKeeper ABCs

Kafka replies on and is the client of ZooKeeper.

ZooKeeper is a Distributed Coordination Service for Distributed Applications (i.e. Kafka), relieving them from coordination but focusing on high-level synchronization, configuration maintenance, and groups and naming.

ZooKeeper defines a hierarchical namespace similar to the structure and a Linux filesystem like the figure below. Each node in the tree is named as a path starting with a forward slash '/' (the root node). In ZooKeeper parlance, a tree node is called znode.

As ZooKeeper is designed to coordinate data (like metadata, configuration, location etc.), so the data stored in each znode is usually small in the byte to kilobyte range. ZooKeeper keeps the data in memory for high throughput and low latency. Meanwhile, snapshots of the in-memory data is kept in a persistent storage (disk) along with transaction logs. Transaction logs refer to logs of clients' read and write operations.

Like the applications it coordinates, ZooKeeper itself is also distributed and replicated across a cluster of hosts - ZooKeeper ensemble. The in-memory data and persistent data are replicated among the ZooKeeper servers.

The servers that make up the ZooKeeper ensemble must all know about each other (by configuration file). ZooKeeper performs heartbeats test (a tiny packet) periodically to check connection and availability of servers.

At any give moment, a client (a coordinated application) connects (TCP connection) to a single ZooKeeper server, and switch to another one if the current TCP connection breaks. Read requests are served locally while requests to change the state of data (write requests) are all forwarded to a particular server called leader. The rest servers are called followers (similar to partitions leader and followers of Kafka). Followers replicate write results from the leader to keep synchronized.

Write synchronization is the core of ZooKeeper.

The underlying implementation takes care of replacing the leader upon failure and syncing followers with the leader.

Quorum

According to Wikipedia, a quorum is the minimum number of members of a deliberative assembly necessary to conduct the business of that group. A quorum requirement is protection against totally unrepresentative action in the name of the body by an unduly small number of persons. A quorum is often more than half of the total number.

ZooKeeper is functional only when a quorum of the servers are available. Recall that all writes requests are forwarded to the leader in ZooKeeper. Upon successive write, the result is synced to followers. If less than a quorum of the servers are synched on the write, ZooKeeper was nonfunctional as 'write synchronization is the core'.

Take 4 ZooKeeper servers for example, a quorum is 3 (> 4/2) which allows 1 server failure and writes should be synced among 3 servers. If there are only 3 servers, the quorum is 2 (> 3/2) which is able to allow 1 server failure as well. So the additional 4th server does not bring in any improvement in terms of stability. However, it benefits load balancing. But what if the number is increased to 5? This requires 3 (> 5/2) servers to form a quorum, the same as a 4-server ZooKeeper. However, the system allows 2 server failure and is more fault-tolerant. Usually, either 3 or 5, not 4 - choose an odd number, namely 1, 3, 5, etc.

ZooKeeper elects the leader from at least of a quorum of the servers. Therefore, proper choice of the number of servers is essential for deployment. If there is only 1 server, it forms the standalone mode, otherwise, it is replicated mode.

A basic rule:

At least a quorum of ZooKeeper servers are online (active);
At least a quorum of active ZooKeeper servers are synched on a 'write' operation.

Read more about quorum at why-zookeeper-on-odd-number-nodes?

Kafka ABCs

Kafka is a distributed streaming platform.
1. Streaming: real-time messaging pipeline similar to Logstash and different from Elasticsearch that is a static storage.
2. Distributed: cluster with fault-tolerance and load balancing.
Kafka receives data from producers while streaming it to consumers (also called subscribers). In this post, words read and consume are used interchangeably; write and produce are used interchangeably. However, read and write emphasize internal actions while produce and consume emphasize interactions with external applications.
Kafka and Logstash are both message pipeline.

However, Kafka is more powerful. Basically, Kafka is a cluster while Logstash runs in standalone mode.

They can co-operate: each Logstash instance connects to a Kafka server (also called broker) in the cluster. Logstash instances are not aware of each other.

Kafka uses the term record or message while Elastic Stack and Flume use event, namely the data.

Each record comprises timestamp, metadata other key-value pairs.

{"@timestamp":"2019-04-17T09:45:22.361Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.0.0","topic":"var-logs"},"message":"127.0.0.1 - - [17/Apr/2019:09:45:19 +0000] \"GET /index.html HTTP/1.1\" 200 3700 \"-\" \"curl/7.29.0\" \"-\"","log":{"offset":505,"file":{"path":"/var/log/nginx/access.log"}},"input":{"type":"log"},"ecs":{"version":"1.0.0"},"host":{"name":"76595710480d","architecture":"x86_64","os":{"platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","kernel":"3.10.0-693.el7.x86_64","codename":"Core"},"containerized":true,"hostname":"76595710480d"},"agent":{"ephemeral_id":"8909f3d3-8037-4288-bf12-7f22e821181b","hostname":"76595710480d","id":"0c54a8df-1b11-421a-b91b-98a18b4d1ff0","version":"7.0.0","type":"filebeat"},"cloud":{"instance":{"id":"i-00000472","name":"apple-dev.novalocal"},"machine":{"type":"8C16G100G"},"availability_zone":"nova","provider":"openstack"}}

Topic, Partition and Commit Log

Records are classified into topics: categories of records. We can regard the topic as a label assigned to a group of relevant records (i.e. Nginx logs). Records of a particular topic are stored in one or more (determined at topic creation) partitions with individual records replicated in one or more partitions (determined at topic creation).

A partition is an ordered and immutable sequence of records with new records continuously appended to, similar to an array disallowing random write.

Write in order while read in a free style.

Multiple partitions act as a parallelism to speed up producing and consuming. Additionally, the capacity of a partition may be limited by the Kafka server hold it. So multiple partitions accomplish storage scalability. Typically, partititons reside on dedicated high speed I/O devices like SSD.

Records in partitions form a structured commit log. In the context of Kafka, "log" refers to topic data, NOT the log of Kafka itself. The relevant configuration directive is log.dir. In this post, we call them commit log and broker log separately.
Offset

A record is uniquely identified by the index within a partition, which is called offset - the only metadata retained by each consumer. For sequential consumption, the offset is reduced one by one. Consumers are independent of each other. We can use command line tail -F to examine the topic records without interfering in what is and what can be consumed.

Records sent by a particular producer to a particular topic partition are appended in the order they are sent. However Kafka only guarantees such order on a per-partition base. The order of records among consumers, producers and/or partitions are not forseeable. If that is the desired outcome, then configure only one partition for a topic and only one consumer instance per consumer group (discussed below).
Persistent Storage

Kafka can be configured to hold records for a retention period of time. During that period, a record is available for consumption, after which it is discarded to free up space.

Consumption does not mean to eat up records but just read. Whether to get rid of a record depends on the retention configuration. The read/write performance is constant with respect to storage size so storing data for a long time is not a problem.
Distribution

Each partition is replicated across a configurable number of Kafka brokers - the replication factor for fault tolerance. Of the replicated brokers, one is the leader and the rest are followers (recall the leader and follower of Zookeeper). At any given moment, the leader broker handles both readding and writing operation (different from that of Zookeeper) while the rest brokers only replicate the leader.

A leader has to maintain a list of followers that are in-sync replicas (ISR). An ISR means the follower has fully caught up with the leader, namely synced in time. If a leader fails, one of the ISR followers becomes the new leader.

From the perspective of a Kafka broker, it holds a share of existing partitions and acts as a leader for some of its partitions and a follower for others.

Therefore, load is well balanced within the cluster.
Producer

A producer publish data to the records of its choice and decide the partition to which a record is assigned to. The assignment can be done in a round robin fashion to load balance or based on key values.
Consumer Group

A topic can have zero or more consumers which are also assigned to labels, forming consumer groups (also called subscriber group). That is to say, all consumers subscribing to a topic are divided into groups. That makes a sense as there may exist multiple entities interested in the same topic, with each entity forms a subscriber group. Consumer instances within a group can be separate threads, processes or machines.

Partitions of a topic are effectively load-balanced over the consumer instances of a subscribing group, in a granularity of partitions. Each instance exclusively consumes zero or more of the partitions at any given time. If there are more instances than partitions, then some of them may be in idle state, consuming nothing. In the other way around, each partition is broadcasted in a granularity of subscribing groups.

In the figure below, there are 4 partitions (P0 - P3) belonging to a particular topic (replication is left out for simplifition). From the figure, each partition is broadcasted to both group A and group B. Within either group, partitions are exclusively distributed.

Subscribing groups achieve multi-subscriber support (with the help of persistent storage), namely scalability. Within each group, Kafaka achives parallelism and load balance. The points are:
1. Broadcast among groups.
2. Exclusive distribution among consumer instances of a group.
A consumer group can be esteemed as a logical subscriber. In terms of scalability, load balance, and fault tolerance:
1. Groups can also leave and join the system.
2. Instances can dynamically join (get a share from other instances) and/or leave (i.e. die) a group (its partitions are transferred to others).
By the way, consumer group has a unique identifier that is created by Java consumer API, not by ZooKeeper or Kafka producer.
Early versions of Kafka depends on external Zookeeper. Recent Kafka has built Zookeeper into release packages. More aggressively, Kafka (>= 2.8.0) can run without Zookeeper, with self-managed quorum (e.g. Kafka Raft).

Deployment

This post will deploy a ZooKeeper cluster and Kafka cluster with 3 Docker containers for testing purpose. All the 3 containers reside on the same physical host.

Container	IP	Servers
logger1	172.17.0.2	ZooKeeper 1, Kafka 1
logger2	172.17.0.3	ZooKeeper 2, Kafka 2
logger3	172.17.0.4	ZooKeeper 3, Kafka 3

References

Prerequisites

Docker CentOS 7;
Java 8;
ZooKeeper MUST be ran before Kafka.

Docker

Account - logger

Create an account 'logger' in both the host and containers. The name 'logger' is chosen to represent ZooKeeper and Kafka respectively. Its home directery is the main place to share critical configuration files. This is optional as we can create the new account within containers directly. It is listed just as a practice.

First create the account in the physical host. We'd better keep the user ID and group ID consistent in case of permission conflicts. In this post, we set both IDs to 555. In Linux practice, manually created user ID and group ID are suggested to be equal to or larger than 1000.

root@tux ~ # groupadd -g 555 logger
root@tux ~ # useradd -ms /bin/bash -u 555 -g logger logger
root@tux ~ # passwd logger
root@tux ~ # id logger
root@tux ~ # su - logger

Create a shared directory between the host and containers:

root@tux ~ # mkdir -p /var/opt/logger
root@tux ~ # chown -R logger:logger /var/opt/logger

Latest CentOS 7

root@tux ~ # docker search centos
root@tux ~ # docker pull centos
root@tux ~ # docker image ls

Dockerfile

In the Dockerfile, we pass a default password for 'logger', which should be changed after docker attach or docker exec.
Enable EPEL and IUS, and install a few tools (iproute2 vs net-tools).

FROM centos:latest
RUN groupadd -g 555 logger ; useradd -ms /bin/bash -u 555 -g logger logger ; echo "logger:12345678" | chpasswd
RUN yum install -y epel-release https://centos7.iuscommunity.org/ius-release.rpm
RUN yum install -y sudo iproute most nmap-ncat java-1.8.0-openjdk
CMD ["/bin/bash"]

Remember to change password upon login.

Build Image

The Dockerfile is read from STDIN (dash).

logger@tux ~ $ sudo docker build -t centos-7.6:logger - < Dockerfile
logger@tux ~ $ sudo docker image ls

Create Container

Create an alias in /etc/hosts for the host IP by --add-host, making connections to the host easier.
Use the default 'bridge' networking mode; run as the default 'root'.

logger@tux ~ # HOSTIP="$( awk -F $'/|[[:space:]]+' '{print $4}' < <(ip -4 -o address show scope global dev eth0) )"

logger@tux ~ $ sudo docker run --name logger1 -d -it --mount type=bind,source=/var/opt/logger,target=/var/opt/logger -w /var/opt/logger --net bridge --add-host host-eth0:${HOSTIP} --user root:root centos-7.6:logger bash

Container Setup

By default, containers's hostname is set to the value of their IDs. So 'localhost' in sudoers does not work as expected.

root@tux ~ # docker container ls
root@tux ~ # docker attach logger1
root@container ~ # passwd logger
# <new password>

root@container ~ # visudo [-c] -f /etc/sudoers.d/logger
# logger $(hostname) = (root) ALL

root@container ~ # visudo -cf /etc/sudoers.d/logger
root@container ~ # sudo -ll -U logger

Switch to the new account:

root@container ~ # su - logger

logger@container ~ $ echo 'PS1="[\u@\h-logger1 \W]\$ "' >> ~/.bashrc ; source ~/.bashrc

Insert logger1 to PS1, making it easier to differentiate containers on shell prompt.

Java 8

Install JRE:

logger@container-logger1 ~ $ yum repolist
logger@container-logger1 ~ $ java -version

logger@container-logger1 ~ $ yum search java-1.8.0-openjdk
logger@container-logger1 ~ $ sudo yum install java-1.8.0-openjdk.x86_64
logger@container-logger1 ~ $ java -version

System property value can be configured dynamically on command line by java -Dproperty=value. property is a Java variable while value overrides the counterpart set in configuration files. If value is a string with blanks, then quote it.

ZooKeeper Replicated Mode

The installation is quite easy: just download and extract the tarball. Stable versions are recommended! As of writing this post, stable 3.4.14 is available.

Install ZooKeeper to /opt:

logger@container-logger1 ~ $ curl -O /var/opt/logger https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/stable/zookeeper-3.4.14.tar.gz
logger@container-logger1 ~ $ sudo tar -xzpvf /var/opt/logger/zookeeper-3.4.14.tar.gz -C /opt/
logger@container-logger1 ~ $ cd /opt
logger@container-logger1 ~ $ sudo ln -sv zookeeper-3.4.14 zookeeper
logger@container-logger1 ~ $ sudo chown -R logger:logger zookeeper/
logger@container-logger1 ~ $ cd zookeeper/conf/

Next, we focus on the configuration. A sample configuration file is located at zookeeper/conf/zoo_sample.cfg. We are interested in the following directives:

tickTime: basic unit in milliseconds; heartbeats interval - the tick. The minimum timeout value will be twice the 'tickTime' and the maximum timeout value is 10 times.
initLimit: timeout for servers in quorum to connect to the leader.
syncLimit: timeout between sending a update request to and receiving acknowledgement from a leader.

The two directives are not required in standalone mode.
dataDir: persistent storage for both snapshots of in-memory data and trasaction logs.
dataLogDir: a dedicated directive that defines a different location to store transactions logs.

A separate physical storage device for transaction logs can significantly improve performance of ZooKeeper.
clientPort: the port to listen for client (Kafka brokers in this case) connections. By default, it is 2181.

clientPortAddress is the address to listen for client connections. By default, it is '0.0.0.0'.
autopurge.snapRetainCount: when doing storage cleanup, only retain the most recent version. No less than 3!
autopurge.purgeInterval: cleanup interval in hour unit.
server.X=hostname:port1:port2:

Specify ZooKeeper servers where X is an integer between 1 and 255, being the unique identifier within the ensemble. The first port number port1 (default to 2888) is for followers to connect to the leader; the next port2 (default to 3888) is for leader selection.

Standalone mode does not require the 'server.X' directive, but can also specify a single line like 'server.1=localhost:2888:3888'.

If multiple servers run on a single host, choose separate ports, directroies etc. The name tick is the basic unit in measuring timeout. When setting up ZooKeeper in standalone mode (1 server only), 'initLimit' and 'syncLimit' are ignored as there is no synchronization requirement. Standalone mode is handy when testing development.

The configuration file uses .properties format, and can be any name, but often we use conf/zoo.cfg (check bin/zkEnv.sh). Here is a sample:

tickTime=2000                             # 1 tick = 2000 ms = 2s
initLimit=5                               # 5 ticks
syncLimit=2                               # 2 ticks
dataDir=/var/opt/logger/z1-snapshots      # persistent in-memory snapshots
dataLogDir=/var/opt/logger/z1-txnLogs     # persistent transaction logs
clientPort=2181
autopurge.snapRetainCount=3               # keep lastest 3 versions
autopurge.purgeInterval=2                 # cleanup every 2 hours

4lw.commands.whitelist=*                  # new in 3.5.3, enable all four-letter commands

server.1=logger1:2888:3888                # communication within Zookeeper cluster
server.2=logger2:2888:3888                # port 2888 used to contact leader
server.3=logger3:2888:3888                # port 3888 used to elect leader

In the example above, the server hostname 'logger1', 'logger2' and 'logger3' is their /etc/hosts aliases, otherwise use IP addresses.

According to the configuration example, create 'dataDir', 'dataLogDir' and create container aliases.

logger@container-logger1 ~ $ mkdir -p /var/opt/logger/z1-{snapshots,txnLogs}

logger@container-logger1 ~ $ sudoedit /etc/hosts
# -or-
logger@container-logger1 ~ $ cat >> /etc/hosts <<EOF
> 12.34.56.71 logger1
> 12.34.56.72 logger2
> 12.34.56.73 logger3
> EOF
logger@container-logger1 ~ $ ping logger1

We are very close to start ZooKeeper now. Before that, file myid under 'dataDir' must be created to define the unique identifier in accord with 'server.X'. Upon startup, a ZooKeeper server gets its identifier by looking up myid. The only content of this file will be interger X:

logger@container-logger1 ~ $ echo '1' > /var/opt/logger/z1-snapshots/myid

For standalone mode, if directive 'server.X' is omitted, 'myid' should be omitted either.

ZooKeeper Startup

ZooKeeper ships with startup scripts bin/zkServer.sh which sources bin/zkEnv.sh to set runtime variables. Have a read at the two scripts before executing. Here is an excerpt from 'zkServer.sh':

nohup "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

Add 'zookeeper/bin' to 'PATH':

logger@container-logger1 ~ $ sudo chmod -x /opt/zookeeper/bin/*.cmd
logger@container-logger1 ~ $ echo 'PATH=/opt/zookeeper/bin:${PATH}' >> ~/.bashrc ; source ~/.bashrc
logger@container-logger1 ~ $ zkServer.sh
logger@container-logger1 ~ $ zkServer.sh print-cmd

From 'zkEnv.sh', the Java max heap size is set to 1000m by variable 'ZK_SERVER_HEAP' and 'ZK_CLIENT_HEAP', reflected by java -Xmx option. An overwhelming heap size will seriously degrade ZooKeeper performance due to frequent disk swap. The minimal guarantee is that heap size is smaller than memory size.

ZooKeeper Logging

Basically, ZooKeeper uses log4j 1.2 as its logging infrastructure. Its configuration file should either be in the working directory (where 'zkServer.sh' is invoked; not the directory where 'zkServer.sh' resides), or inclued in JAVA 'CLASSPATH'. The default log4j.properties file resides in conf/ directory which is accessible from 'CLASSPATH' as defined in 'zkEnv.sh'.

Variable 'ZOO_LOG4J_PROP' defines the log level and appender (output destination) of Log4j. By default, they are 'INFO' and 'CONSOLE' respectively in 'zkEnv.sh'. About the priorities of different log levels, read the post 'log4j'. Logs appended to 'CONSOLE' actually go to STDERR and STDOUT.

Meanwhile, 'zkServer.sh' redirects STDERR and STDOUT to "${ZOO_LOG_DIR}/zookeeper.out" and runs in background. Variable 'ZOO_LOG_DIR' is set to '.' (the working directory). If it is ran under the home directory '~', zookeeper.out will be created there. Therefore, logs of ZooKeeper are printed to file zookeeper.out in the working directory.

Besides, there are other appenders like 'ROLLINGFILE' and 'TRACEFILE'. Each appender has their own properties defined in 'log4j.properties'. The advantage of 'ROLLINGFILE' is that logs are rotated by property 'MaxFileSize' and property 'MaxBackupIndex'. Multiple appenders can be turned on or off concurrently. For example, both 'CONSOLE' and 'ROLLINGFILE' can be turned on. If 'CONSOLE' is turned off, 'zookeeper.out' is always empty.

Hence, start ZooKeeper with specified 'ZOO_LOG_DIR' and 'ZOO_LOG4J_PROP':

ZooKeeper logs is located at /var/opt/logger/z1-logs.
Choose 'ROLLINGFILE' appender.

logger@container-logger1 ~ $ mkdir -p /var/opt/logger/z1-logs
logger@container-logger1 ~ $ ZOO_LOG_DIR="/var/opt/logger/z1-logs" ZOO_LOG4J_PROP="INFO,ROLLINGFILE" zkServer.sh print-cmd
# -or-
logger@container-logger1 ~ $ echo 'export ZOO_LOG_DIR="/var/opt/logger/z1-logs" ZOO_LOG4J_PROP="INFO,ROLLINGFILE"' >> ~/.bashrc ; source ~/.bashrc
logger@container-logger1 ~ $ zkServer.sh print-cmd

Now, the setup of the first ZooKeeper server is completed. We will commit logger1 on which logger2 and logger3 is created.

logger@container-logger1 ~ $ ^P-^Q
logger@tux ~ $ sudo docker stop logger1

logger@tux ~ $ sudo docker commit -a 'logger' -m "copy logger1" logger1 centos-7.6:logger1
logger@tux ~ $ sudo docker image ls

Recall that, each container has a hostname alias associated with a fixed IP in /etc/hosts. So always start the containers in the exact order: logger1, logger2 and logger3, such that the Docker daemon assign IPs in order.

Start and create the containers in order:

logger@tux ~ $ sudo docker start logger1

logger@tux ~ $ sudo docker run --name logger2 -d -it --mount type=bind,source=/var/opt/logger,target=/var/opt/logger -w /var/opt/logger --net bridge --add-host host-eth0:${HOSTIP} --user root:root centos-7.6:logger1 bash
logger@tux ~ $ sudo docker run --name logger3 -d -it --mount type=bind,source=/var/opt/logger,target=/var/opt/logger -w /var/opt/logger --net bridge --add-host host-eth0:${HOSTIP} --user root:root centos-7.6:logger1 bash

Attach to logger2 aned logger3, and adjust the following items:

/etc/sudoers.d/zk
~/.bashrc
conf/zoo.cfg ; mkdir -p /var/opt/logger/z2-{logs,snapshots,txnLogs}
echo '2' > /var/opt/logger/z2-snapshots/myid
zkServer.sh print-cmd

Now, it is time to start the servers. A customized configuration file can be supplied to the command line upon starting (i.e. for debugging) to override the default conf/zoo.cfg:

logger@container-logger1 ~ $ zkServer.sh start [/path/to/zoo.cfg]

logger@container-logger1 ~ $ zkServer.sh status

However, it may report:

Error contacting service. It is probably not running.

This means the ZooKeeper culster are not yet established (i.e. leader election ongoing) though each server is already running. Then try other methods:

logger@container-logger1 ~ $ ps -eF | grep '[z]ookeeper'
logger@container-logger1 ~ $ tail -F ~/var/opt/logger/z1-snapshots/zookeeper.log

zookeeper.log prints many useful information like configuration pathname, hosts aliases resolution, autopurge arguments, tick and timeout, peer election process etc. Specially, there is a line:

Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/opt/logger/z1-txnLogs/version-2 snapdir /var/opt/logger/z1-snapshots/version-2

datadir refers to the location where transaction logs are stored; snapdir refers to that of in-memory snapshots. Please pay attention that these two name is different than those in zoo.cfg. datadir corresponds to directive 'dataLogDir' while 'snapdir' corresponds to directive 'dataDir'. Read PurgeTxnLog API.

ZooKeeper Status

logger@container-logger1 ~ $ zkServer.sh
logger@container-logger1 ~ $ pgrep -u logger -P 1 -ac -f 'java.*\/home\/logger\/zookeeper\/bin'
logger@container-logger1 ~ $ zkServer.sh status

# zkServer.sh status
# ZooKeeper JMX enabled by default
# Using config: /opt/zookeeper/bin/../conf/zoo.cfg
# Mode: follower

The 'Mode' line tells if a server is a leader or a follower.

Apart from the script wrapper 'zkServer.sh', ZooKeeper can respond to a predefined set of four-letter commands. We issue the commands to servers' (TCP/UDP) sockets via telnet or nc, at the client port (default 2181):

logger@container-logger1 ~ $ rpm -q nmap-ncat.x86_64
logger@container-logger1 ~ $ nc localhost 2181 <<< 'dump'
logger@container-logger1 ~ $ echo 'srvr | stat | envi' | nc localhost 2181

From ZooKeeker 3.5.3 onwards, by default only 'srvr' is enabled. We should configure property '4lw.commands.whitelist' to turn on desired commands.

Another wrapper script 'zkCli.sh' connects to servers and perform filesystem-like operations:

logger@container-logger1 ~ $ zkCli.sh -server localhost:2181 help     # print help message

logger@container-logger1 ~ $ zkCli.sh -server localhost:2181          # interactive ZooKeeper shell

# [zk: localhost:2181(CONNECTED) 0] help                         # print help message
# [zk: localhost:2181(CONNECTED) 0] ls /                         # list the 'root' znode
# [zk: localhost:2181(CONNECTED) 0] create /zk_test my_data      # creat a new znode directory
# [zk: localhost:2181(CONNECTED) 0] ls /
# [zk: localhost:2181(CONNECTED) 0] set /zk_test my_new_data
# [zk: localhost:2181(CONNECTED) 0] get /zk_test
# [zk: localhost:2181(CONNECTED) 0] delete /zk_test
# [zk: localhost:2181(CONNECTED) 0] ls /
# [zk: localhost:2181(CONNECTED) 0] quit

ZooKeeper Storage

For a long running production system, persistent storage must be managed manually or automatically by autopurge mentioned above.
Recall that logs of ZooKeeper itself is rotated automatically by 'ROLLINGFILE'.

To manually purge history snapshots and related transaction logs:

logger@container-logger1 ~ $ cd /opt/zookeeper/
logger@container-logger1 ~ $ java -cp zookeeper-3.4.14.jar:lib/slf4j-api-1.7.25.jar:lib/slf4j-log4j12-1.7.25.jar:lib/log4j-1.2.17.jar:conf org.apache.zookeeper.server.PurgeTxnLog <datadir> <snapdir> -n <count>

datadir ('dataLogDir') and snapdir ('dataDir') can be found either from zookeeper.log, in zoo.cfg, or by the four-letter command conf.
-n <count> means to keep the last 'count' snapshots and transaction logs, which should be no less than 3 conventionally.
Make sure the correct classpath is provided.
Create a cron job.

Actually, ZooKeeper provides a built-in 'zkCleanup.sh' script to wrap the Java command:

logger@container-logger1 ~ $ zkCleanup.sh --help

'zkCleanup.sh' accepts the same arguments as above and makes cron jobs much eaiser. However, ZooKeeper 3.4.0 onwards already supports autopurge, why not just use the built-in feature?

Stop ZooKeeper

logger@container-logger1 ~ $ zkServer.sh stop

ZooKeeper Systemd Unit

Read the post Zookeeper install on CentOS 7 first.

Zookeeper Advanced Tips

Dynamic Reconfiguration

From 3.5.0 onwards, directives 'clientPort' and 'clientPortAddress' can be integrated into the 'Server.X' directive as:

# server.X = <address1>:port1:port2[:role];[<client port address>:]<client port>

server.1 = 12.34.56.78:2888:3888;2181
server.2 = 12.34.56.78:2888:3888:participant;2181
server.3 = 192.168.0.105:2888:3888:observer;12.34.56.78:2181

Dynamic Reconfiguration means to update confiuration on the fly without restarting ZooKeeper servers.

Observer Server

As mentioned earlier, at least a quorum of the active servers are synched on a 'write' operation. However, this requirement make it hard to scale the number of ZooKeeper servers and/or the number of clients as write sync takes time and resources.

ZooKeeper now introduces a new type of server called observer to improve scalability, distinguished from the common participant servers. Observers are almost the same as participant followers except that they does not participate in write sync and only receive the sync results.

Therefore, we can increase the number of observers as much as we like without harming the write sync performance. To specify an observer server, the configuration looks like:

peerType=observer
server.1=localhost:2888:3888:observer;2181

Kafka

Kafka Installation

Like that of ZooKeeper, the installation process of Kafka is quite straightforward, download and untar.

A kafka distribution is named as 'kafka_scalaVersion-kafkaVersion.tgz`. Each Kafka version may have multiple scala disbributions, and vice versa. As of the post, the Scala version is 2.12 while Kafka version is 2.2.0.

logger@container-logger1 ~ $ curl -O /var/opt/logger/ https://www-eu.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
logger@container-logger1 ~ $ sudo tar -xzpvf /var/opt/logger/kafka_2.12-2.2.0.tgz -C /opt/
logger@container-logger1 ~ $ cd /opt
logger@container-logger1 ~ $ sudo ln -sv kafka_2.12-2.2.0 kafka
logger@container-logger1 ~ $ sudo chown -R logger:logger kafka/
logger@container-logger1 ~ $ ll kafka/config

Kafka Configuration

Now configure server.properties:

# similar to ZooKeeper 'myid'; a unique integer
broker.id=1

# comma-separated listeners
listeners=PLAINTEXT://0.0.0.0:9092,EDGE://1.2.3.4:9093,USA://5.6.7.8:9094
# advertised listners
advertised.listeners=PLAINTEXT://0.0.0.0:9092,EDGE://1.2.3.4:9093,USA://5.6.7.8:9094
# listner for inter-broker communication
inter.broker.listener.name=PLAINTEXT
# listener name mapping to security prototol
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,EDGE:PLAINTEXT,USA:PLAINTEXT

# commit logs location; high speed SSDs
#log.dir=/tmp/kafka-logs
# default to 'log.dir'
log.dirs=/home/zk/kafka/commitLogs

# default partitions per topic
num.partitions=3

# replication factor for internal topics like "__consumer_offsets"
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=2

# ZooKeeper servers
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
zookeeper.connection.timeout.ms=3000

listeners

Listeners are what interfaces a Kafka broker bind to.

The directive is a list of comma-separated listeners. Each listener is actually a TCP socket on the broker and is associated with a name, hostname/IP, and port.
```
# listeners = listener_name://my.host.name:port
```
Specify hostname as '0.0.0.0' to bind to all interfaces. Leave hostname empty to bind to the default interface. A hostname can be resolved to multiple IPs to do load balance. If this directive is not set, it defaults to 'PLAINTEXT://0.0.0.0:9092'.

A listener should be mapped to a security protocol by its name for connection authentication, and thus often named after a security protocol:
```
$ name to security protocol map
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
```
So we cannot define two listeners with the same name.
```
# error: two listners with the same name
listeners=PLAINTEXT://192.168.1.100:9092,PLAINTEXT://12.34.56.78:9092
```
Multiple listeners MUST have different ports.
advertised.listeners

Advertised listeners are what interfaces a client (producer/consumer) connects to.

A listener is firtly published to the ZooKeeper cluster, then spreaded to all other Kafka brokers, and finally to clients. The limitation is that a listener can be advertised only once.
1. Communication within the cluster brokers and can be achieved through any defined listeners. Clients get the cluster metadata from any defined listeners as well.
2. However, read and/or write connections are only done through advertised listeners. Hence, usually we advertise listeners with public IP.
```
listeners=PLAINTEXT://0.0.0.0:9092
   
advertised.listeners=PLAINTEXT://12.34.56.78:9092
```
The design of multiple listeners defined and/or advertised gives a fair amount of flexibility. For example, we can advertise a listener for internal clients while another one for external clients. Additionally, a separate advertised listener for replicated partition traffic is desired. Directive 'inter.broker.listener.name' can be set to a listener for traffic between brokers like:
```
inter.broker.listener.name=PLAINTEXT
```
If not properly advertised, a client may fail to produce or consumer data. For example, if a listener is bound to '0.0.0.0:9092' but only advertise the LAN address '192.168.1.100:9092', clients could still initiate a connection to the broker through '12.34.56.78:9092' (public IP), fetching metadata about brokers. When it comes to read and/or write data, clients can only contact the advertised addresses enclosed in metadata. That demonstrates the difference between reachable socket and reachable service.

We CANNOT advertise a listener with '0.0.0.0'.
'log.dirs' is a list comma-separated locations to store commit logs. If not set, it defaults to 'log.dir' that is a single location. 'log.dir' defaults to /tmp/kafka-logs.
ZooKeeper listens for Kafka connections at port 2181; Kafka listens at port 9092.

Start Kafka

Before starting brokers, create directories and change 'PATH':

logger@container-logger1 ~ $ mkdir -p /var/opt/logger/k1-logs
logger@container-logger1 ~ $ echo 'PATH=/opt/kafka/bin:${PATH}' >> ~/.bashrc ; source ~/.bashrc

Kafka has a bunch of built-in scripts, of which 'kafka-server-start.sh' and 'kafka-run-class.sh' are responsible for starting the service. In the scripts, there is a special variable 'base_dir' setting the base directory (recall 'ZOO_LOG_DIR' of ZooKeeper) that defaults to Kafka installation directory.

Similar to ZooKeeper, Kafka also utilize 'Log4j' to trace broker logs affected by environment variables 'LOG_DIR', and 'KAFKA_LOG4J_OPTS', and Java property 'kafka.logs.dir'

# Script 'kafka-server-start.sh':
# Log4j property file
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"

# Script 'kafka-run-class.sh'
LOG_DIR="$base_dir/logs"

From the definition, broker logs default to kafka/logs where Kafka is installed, including 'server.log', 'state-change.log' and 'controller.log' (recording leader election).

We can just leave brokers logging to the defauts. OK, start the server:

logger@container-logger1 ~ $ kafka-server-start.sh
# USAGE: /opt/kafka/bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*

# explictly provide 'server.properties'
logger@container-logger1 ~ $ kafka-server-start.sh -daemon /opt/kafka/config/server.properties

-daemon option put Kafka into background.
Unlike ZooKeeper, Kafka requires explicit pathname of property file.

--override changes default properties. For example, to change the location of broker logs, either modify property 'kafka.logs.dir' or environment variable 'LOG_DIR'

logger@container-logger1 ~ $ mkdir -p /var/opt/br1-logs
logger@container-logger1 ~ $ LOG_DIR=/var/opt/br1-logs kafka-server-start.sh ...
# -or-
logger@container-logger1 ~ $ kafka-server-start.sh ... --override kafka.logs.dir=/var/opt/br1-logs

Repeat the above setup for the other two Kafka servers.

Kafka Status

We can verify if the server was started correctly through ZooKeeper dump:

logger@container-logger1 ~ $ ps -eF | grep '[k]akfka'
logger@container-logger1 ~ $ ss -npelt
logger@container-logger1 ~ $ pgrep -u logger -P 1 -ac -f 'java.*\/home\/logger\/kafka\/bin'
logger@container-logger1 ~ $ tail -F /var/opt/br1-logs/server.log

logger@container-logger1 ~ $ nc localhost 2181 <<< "dump"

Here is a sample output:

SessionTracker dump:
Session Sets (4):
0 expire at Sat Feb 21 00:11:04 UTC 1970:
0 expire at Sat Feb 21 00:11:06 UTC 1970:
0 expire at Sat Feb 21 00:11:08 UTC 1970:
3 expire at Sat Feb 21 00:11:10 UTC 1970:
        0x2010197720b0000
        0x301019c5afd0000
        0x301019c5afd0001
ephemeral nodes dump:
Sessions with Ephemerals (3):
0x301019c5afd0000:
        /brokers/ids/2
0x301019c5afd0001:
        /brokers/ids/3
0x2010197720b0000:
        /controller
        /brokers/ids/1

From the excerpt, we know ZooKeeper create an individual znode for each Kafka server, within the namespace tree. To further investigate:

logger@container-logger1 ~ $ zkCli.sh -server localhost:2181 ls /

# [cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]

logger@container-logger1 ~ $ zkCli.sh -server localhost:2181 get /brokers/ids/1
# [1, 2, 3]

Stop Kafka

When a broker is gracefully stopped, it may take advantage of:

Commit logs are synced to disk to avoid logs recovery upon restart.

Log recovery taks time as checksum validation is required for all entries in the tail of the log.
Partitions this broker is the leader for are synchronized to followers, which significantly increase the process of new leader election - leadership migration.

This requires that the topic replication factor is greater than 1 (i.e. --replication-factor=2), otherwise this single leader would sync the partitions nowhere and commit logs would be unavailable! A replication factor greater than 1 is always what we want!

Another requirement, is directive controlled.shutdown.enable is set to true (default).

To gracefully stop a broker:

logger@container-logger1 ~ $ kafka-server-stop.sh

The script does not accept any options and just sends 'SIGTERM' signal to the Kafka process. So don't try to pass --help as it inevitably stops the service.

Stop Kafka before ZooKeeper;
It takes time to stop Kafka cluster, so wait for while!

Topic Creation

Kafka provides a simple console producer and consumer script, quite handy for test.

Before Kafka version 0.9, built-in scripts require the option --zookeeper to specify one or more ZooKeeper servers to connect to as offset data is stored within ZooKeeper. The offset data is now moved into Kafka itself (the built-in __consumer_offset topic). So --zookeeper option is depcreated in favor of option --bootstrap-server using brokers.

Firstly, create a test topic:

logger@container-logger1 ~ $ kafka-topics.sh
logger@container-logger1 ~ $ kafka-topics.sh --zookeeper logger1:2181,logger2:2181 --list

logger@container-logger1 ~ $ kafka-topics.sh --bootstrap-server logger1:9092,logger2:9092,logger3:9092 --create --replication-factor 2 --partitions 3 --topic test-topic
logger@container-logger1 ~ $ kafka-topics.sh --bootstrap-server logger2:9092 --list
logger@container-logger1 ~ $ kafka-topics.sh --bootstrap-server logger2:9092 --describe --topic test-topic

# Topic:test-topic        PartitionCount:3        ReplicationFactor:2     Configs:segment.bytes=1073741824
#         Topic: test-topic       Partition: 0    Leader: 2       Replicas: 2,3   Isr: 2,3
#         Topic: test-topic       Partition: 1    Leader: 3       Replicas: 3,1   Isr: 3,1
#         Topic: test-topic       Partition: 2    Leader: 1       Replicas: 1,2   Isr: 1,2

--replication-factor defines the number of replicas.

It is a must!
--partitions overrides 'num.partitions' in 'server.properties'.
The topic has 3 partitions.
From the output of --describe, replicas in all followers are in-sync replicas.

Console Producer and Consumer

Next, we launch a consumer on the topic:

logger@container-logger1 ~ $ kafka-console-consumer.sh
logger@container-logger1 ~ $ kafka-console-consumer.sh --bootstrap-server logger2:9092 --topic test-topic --from-beginning

No explicit consumer group is created here. By default, 'kafka-console-consumer.sh' will create a random group ID through Java consumer API.

Then start a producer with --broker-list:

logger@container-logger2 ~ $ kafka-console-producer.sh --help
logger@container-logger2 ~ $ kafka-console-producer.sh --broker-list logger1:9092,logger2:9092,logger3:9092 --topic test-topic

# hello, world

Now everything typed in the producer console will be automatically printed in the consumer console. Check partitions:

logger@container-logger1 ~ $ zkCli.sh -server localhost:2181 get /brokers/topics/test-topic/partitions/1

Use ^C to stop the consumer and producer.

Consumer Group

Let's play around consumer group.

Firstly, we want a list of existing consumer groups either by ZooKeeper interface or by Kafka interfaces.

logger@container-logger1 ~ $ zkCli.sh -server localhost:2181 ls /consumers

# < 0.9: consumers contact ZooKeeper
logger@container-logger1 ~ $ kafka-consumer-groups.sh --zookeeper logger1:2181  --list
# >= 0.9: consumers contact Kafka directly by Java consumer API
logger@container-logger1 ~ $ kafka-consumer-groups.sh --bootstrap-server logger1:9092 --list

logger@container-logger1 ~ $ kafka-consumer-groups.sh --bootstrap-server logger1:9092 --describe --group consumer-group-name

Next, we try to create an explicit consumer group. A consumer can join an existing group (implicit default included) or create a new one:

--group consumer-group-name1

This is the most obvious option.

--consumer.config filename;

# /path/to/filename
group.id=consumer-group-name2

--consumer-property group.id=consumer-group-name3

These two options are done through direct Java property.
If the provided name exsit, then the new consumer join the group, otherwise the group would be created.

logger@container-logger1 ~ $ kafka-console-consumer.sh --bootstrap-server logger1:9092 --topic test-topic --group consumer-group-12345
logger@container-logger1 ~ $ kafka-console-consumer.sh --bootstrap-server logger1:9092 --topic test-topic --consumer.config consumer-group-12346
logger@container-logger1 ~ $ kafka-console-consumer.sh --bootstrap-server logger1:9092 --topic test-topic --consumer-property group.id=consumer-group-12347

logger@container-logger2 ~ $ kafka-consumer-groups.sh --bootstrap-server logger2:9092 --describe --group consumer-group-12345

Kafka Security

Post introduction-to-apache-kafka-security gives a clear outline on Kafka security mechanisms. It recommends SASL/SCRAM method (Zookeeper only supports SASL/DIGEST-MD5).

Kafka uses Java Authentication and Authorization Service (JAAS) for both server side security setup and client side authentication.

There are two ways to provide JAAS data (e.g., SASL).

Configuration property string in .properties like sasl.jaas.config=. This is the recommended method.

Below is an example. Please make sure the trailing semi-colon is present.
```
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="(username)" password="(password)";
```
Built-in CLI clients can automatically detect the security setup. If not, add option --command-string to kafka-topics.sh, or option --producer.config to kafka-console-producer.sh.

JAAS file. For example, on server side, we can pass the static file as below.

export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/kafka_jaas.conf"

Dynamic Update Mode

From Kafka 1.1 onwards, directives could be updated/overriden dynamically. The Broker's Dynamic Update Mode column may be:

read-only: requires restart for update;
per-broker: can be updated for each broker on the fly;
cluster-wide: can be updated for all brokers; may also be updated for an individual broker.
Some directives can be changed dynamically per-topic, like the directive 'num.partitions'.

To get a list of supported directives on command line:

logger@container-logger1 ~ $ kafka-configs.sh --help

To check current broker configuration:

logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-name 2 --describe

Change number of cleanup threads for commit logs:

# per-broker
logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-name 2 --alter --add-config log.cleaner.threads=2
logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-name 2 --describe

To apply the change to all brokers, use option --entity-default:

# cluster-wide
logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-default --alter --add-config log.cleaner.threads=2
logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-default --describe

To revert the dynamic modification and restore the static value configured:

logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-name 2 --alter --delete-config log.cleaner.threads
logger@container-logger1 ~ $ kafka-configs.sh --bootstrap-server logger1:9092 --entity-type brokers --entity-name 2 --describe

A directive may be configured at multiple levels in the follow order of precedence:

Dynamic per-broker;
Dynamic cluster-wide;
Static in 'server.properties';
Kafka default.

Log4j

2019-04-11T00:00:00+00:00

ABCs

ABCs

There are three core components of Log4j:

Logger: log level priority.

From highest to lowest:
```
OFF > FATAL > ERROR > WARN > INFO > DEBUG > TRACE > ALL
```
1. The lower a level is, the more detailed is the log.
2. When a particular log level is set, logs of higher levels are also printed while logs of lower levels are not. Specailly, 'OFF' turns off log completely while 'ALL' prints everything.
3. Use 'ERROR' or 'WARN' to avoid demanding log storage in practice.
Appender: determine destinations of log output, console, files, remote socket servers, Apache Flume, JMS, remote UNIX Syslog daemons, and various database APIs etc.

Often, log to either console or file.
Layout: log format.

Filebeat

2019-04-11T00:00:00+00:00

Installation

Installation

Flume

2019-04-10T00:00:00+00:00

ABCs
Configuration
Starting an Agent

ABCs

flume [c] an inclined channel for conveying water (as for power); a ravine or gorge with a stream running through it.

Apache Flume is a distributed system for collecting, aggregating and moving log data from many different sources to a centralized data store.

Require Java Runtime Environment (JRE) - 1.8 or later.
A Flume event is byte payload with optional string atrributes - also called Flume data.
A Flume agent is a JVM process consisting of three components: sources, channels, and sinks.
When a source receives an event, it stores it into one or more channels. The channel keeps the event until it is consumed by a Flume sink.

Channels are passive buffers between sources and sinks: neither send nor pull events proactively.
Multiple agents can be connected to form a multi-hop events flow. For example, an avro-typed source can recive external events from an avro-typed sink, using RPC.

Multi-agent flow:

Consolidation:

Multiplexing:

Configuration

Usually, Flume configuration files end with .properties suffix.

Always keep in mind that, an agent may have multiple sources, multiple channels, and multiple sinks. How these multiple objects are configured to cooperate are important!

Each source, channel and sink has a type property.

For example, a source type may be exec; a channel type may be memory; a sink type may be avro.

Other properties are set accordingly.

A Flume source can have an interceptors property. Interceptors are used to tranform recevied events before putting them onto a channel.

For example, an interceptor may be chosen to translate raw event data into Json format.

my_agent.sources.r12a.type = exec
my_agent.sources.r12a.channels=c12
my_agent.sources.r12a.command=tail -F /var/log/server.log
my_agent.sources.r12a.logStdErr = true
my_agent.sources.r12a.batchSize = 100
my_agent.sources.r12a.interceptors=log2Json
my_agent.sources.r12a.interceptors.log2Json.type=Flume.log2Json$Builder

Flume.log2Json$Builder is an external Java class.

Multiple sinks can be grouped (sink group) to achieve load_balance or failover from one sink to another, with the help of sink processor.
```
my_agent.sinkgroups=g12a
my_agent.sinkgroups.g12a.sinks = k12a k12b k12c k12d k12e k12f k12g k12h
my_agent.sinkgroups.g12a.processor.type = load_balance
my_agent.sinkgroups.g12a.processor.backoff = true
my_agent.sinkgroups.g12a.processor.selector = round_robin
my_agent.sinkgroups.g12a.processor.selector.maxTimeOut = 3000
```
If backoff is enabled, the sink processor will backlist sinks that fail, removing them for a given timeout. When the timeout ends, if the sink is still unresponsive, the timeout is increased exponentionally.

Each source and sink should be assigned to a channel.

my_agent.sources.r12a.channels = c12
my_agent.sources.r12b.channels = c12
my_agent.sinks.k12a.channel = c12
my_agent.sinks.k12b.channel = c12
my_agent.sinks.k12c.channel = c12
my_agent.sinks.k12d.channel = c12
my_agent.sinks.k12e.channel = c12
my_agent.sinks.k12f.channel = c12
my_agent.sinks.k12g.channel = c12
my_agent.sinks.k12h.channel = c12

Starting an Agent

Flume distribution brings in a shell script under its bin directory: flume-ng. We should specify the agent name defined in configuration file.

~ # bin/flume-ng --help
~ # bin/flume-ng agent -c conf -f conf/flume-conf.properties -n $agent_name 

Elastic Stack

2019-04-10T00:00:00+00:00

Top-down Architecture
Installation
Logstash
Filebeat

Top-down Architecture

Elastic Stack is called ELK previously. Due new products like Beats and X-Pack, the acronym is changed accordingly.

Kibana

The frontend of the stack: search and virtualize data like charts and tables.
Elasticsearch

Jason-based search engine (based on Lucene): index, analyze and store data as Json format.

From the figure above, we know that Elasticsearch is the heart of the stack. We will find below that the Elastic Stack repository (for CentOS) is called elasticsearch.
Logstash

stash [vt] to store in a usually secret place for future use

Data collection pipeline: unify and normalize data. The name comprises 'log' and 'stash'. 'log' means the input (source data) while 'stash' means output (store the data). Between the input and output, there is the optional filter component. Logstash has now evolved into a more general tool. It can even receive data from and/or send data to Kafka (another FOSS project).

Logstash will format the data in Json format and send out to Elasticsearch (the stash).
Beats

beat [c] A stroke or blow. In music, a beat is a unit of measurement.

Agent: ships different types of data from edge devices to Logstash or directly to Elasticsearch.

There are different kinds of beats desginated for different kinds of data. For example, Filebeat is to collect log files (i.e. access.log and error.log) with modules. A Nginx module collects Nginx log files; a MySQL module collects MySQL log files. Another beat is Metricbeat that collects system-level and/or service-level performance metrics like CPU and memory usage for the operating system. It also ships with modules for popular services such as Nginx, MySQL etc.
Elastic Stack Features

Formerly known as X-Pack - addons: a pack of extensions to the stack, like authentication, monitoring & altering, reporting, Elasticsearch SQL, and machine learning (abnormality detection, forecasting, relation graph).

For relation graph, pay attention to the difference between relevance and popularity. Mostly, relevance is what we desire. All colleagues use Google search in a team does not mean they are relevant as per Google: it is just becuase Google is popular.

Installation

Reference

Prerequisites

Java 8
Use the same version across the entire stack.
Follow the order: Elasticsearch, Kibana, Logstash, Beats etc.
Each component can be installed by justing extracting tarballs or through official 'yum' repository.

Repo

Import key:

user@tux ~ $ sudo rpmkeys --import https://packages.elastic.co/GPG-KEY-elasticsearch

Add the repository:

user@tux ~ $ cat >> /etc/pki/rpm-gpg/elastic-stack.repo <<EOF
> [elastic-7.x]
> name=Elastic repository for 7.x packages
> baseurl=https://artifacts.elastic.co/packages/7.x/yum
> gpgcheck=1
> gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
> enabled=1
> autorefresh=1
> type=rpm-md

Search for a package:

user@tux ~ $ yum search elasticsearch kibana

Logstash

Congfiguration file format is writen in a format similar to Json.

Filebeat

Filebeat support various outputs but we usually just send the events directly to Elasticsearch or to Logstash for additional processing. Events can also be sent to products out of the Elastic Stack like Kafka.
Filebeat uses backpressure-sensative protocol to do congestion control, adjusting the speed of reading log entries in accord with Logstash or Elasticsearch staus.
'inputs' generate 'event's in Json format; 'processors' filters events; 'output' ships events out.

'field' refers to the key part of an event 'key: value' pair.

Install Filebeat

user@tux ~ $ yum search filebeat
user@tux ~ $ yum install filebeat
user@tux ~ $ rpm -ql filebeat

The package includes a sysvinit script, namely /etc/init.d/filebeat. In the 'Filebeat Start' section, we will discuss how to utlize this script for Sytemd startup.

Beats YAML

This applies to all beats applications.

Configuration is writen in YAML format, demanding strict indentation. All settings are structured using dictionaries and lists. Please use the same number of spaces and avoid tabs for dictionary pairs and list items. The structured form collapses to a namespaced naming method.

Dictionary is represented with key: value pairs in the same indentation level, with the ':' separator followed by a space or newline. Multiple pairs can be put on the same line in an abbreviated form:

# Enclosed by braces
person: {name: "John Doe", age: 34, country: "Canada"}

A list item takes the form: - , namely a dash followed by a space. All list items of a list should be at the same indetation level. In the following example, 'colors' is a dictionary 'key' with the value part a list consisted of three items.

colors:
  - Red
  - Green
  - Blue

Multiple list items can also be abbreviated as:

# Enclosed by square brackets
colors: ["Red", "Green", "Blue"]

Whenever a semicolon appears followed by a space, it is a dictionary pair. Whenever a dash followed by a space, it is a list item.

Configuration Notice:

Please use single quotes for regex and pathnames that containing spaces or special characters.
System environment variables and configuration variables (defined within YML itself) are referenced in the form ${VAR:default-value}. Environment variables are expanded before YAML parsing. So we CANNOT refer to system environment variables within YAML files of 'Live Reloading' as those files are reloaded periodically but after the main YAML parsing.

The command line -E option can be used to override variables like -E key=value.
Aside environment and configuration variable reference, event field can be referenced in the form %{[field-name]:default-value}. Pay attention please, the prefix symbol is %. Event fields are accessed using field references [field-name]. This is called Format String (sprintf).

Filebeat Configuration

By default, configuration files are installed to /etc/filebeat:

user@tux ~ $ ll /etc/filebeat/

> total 300
> -rw-r--r-- 1 root root 219620 Apr  5 22:11 fields.yml
> -rw-r--r-- 1 root root  71973 Apr  5 22:11 filebeat.reference.yml
> -rw------- 1 root root   7745 Apr  5 22:11 filebeat.yml
> drwxr-xr-x 2 root root   4096 Apr 18 08:06 modules.d

All configuration files are writen in YAML format. For pre-defined fields, check the installed 'fields.yml'. 'filebeat.reference.yml' is the reference sample. 'filebeat.yml' is the default configuration. 'modules.d' contains built-in configurations for modules like Nginx, Apache etc. Filebeat supports Golang glob syntax. Especially, globstar can be enabled by directive 'recursive_glob.enabled' (default to 'true').

The following are hands-on configuration practice:

Paths

# default to the location of the Filebeat binary
path.home: "/opt/filebeat"

# configuration location
path.config: ${path.home}

# data location
path.data: ${path.home}/data

# Agent logs
path.logs: ${path.home}/logs

# By yum repo
path.home: /usr/share/filebeat
path.config: /etc/filebeat
path.data: /var/lib/filebeat
path.logs: /var/log/filebeat

'path.home' is usually set to the installation location.

Global and General options
```
# Filebeat's global options
filebeat.registry.path: ${path.data}/registry

# General options that are supported by all Elastic beats
name: "filebeat-log-project"
tags: ["project-X", "nginx"]

fields_under_root: false
fields: {project: "X", instance-id: "574734885120952459"}
```
The registry file stores the state and location information that Filebeat uses to track where it was last reading. If a relative path is passed, it is considered relative to the data path. It defaults to ${path.data}/registry.

The general 'fields_under_root' field applies to field 'fields' that is added to the event.

Logging

logging.level: info
#logging.selectors: ["*"]
logging.to_files: true
logging.to_syslog: false
logging.json: false
logging.files:
  path: ${path.home}/logs
  name: filebeat.log
  rotateeverybytes: 10485760
  keepfiles: 7
  permissions: 0644

Log settings for the Filebeat agent itself. If the 'path' is a relative pathname, it would be relative to the 'path.home'.
Log options can be modified on command line directly. For example, -d '*' enables debug level for all components.

HTTP Server

http.enabled: true
http.host: localhost
http.port: 5066

Filebeat has a built-in HTTP server, exposing internal metrics.

~$ curl http://localhost:5066/?pretty
~$ curl http://localhost:5066/stats?pretty

Live Reloading

Like many other services, Filebeat can load configurations from external files. In 'filebeat.yml', we can set values of key 'filebeat.inputs' and key 'filebeat.modules' from extra YAML files.

For inputs:
```
filebeat.config.inputs:
  enabled: true
  path: inputs.d/*.yml
  reload.enabled: true
  reload.period: 10s     
```
Each file matched by the 'path' glob must contain a list of one or more inputs.

For modules:
```
filebeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s
```
If the subcommand filebeat modules is used to disable or enable modules, then 'path' MUST point to a location called modules.d. Similarly, each matched globbing must contain a list of one or more module definitions (i.e. - module: nginx).

However, live reloading is limited sometimes. For example, if a system environment variable is modified or defined, the process wound not know about it unless restarted.

Modules and Inputs

Filebeat reads logs from either 'filebeat.inputs' or 'filebeat.modules'. 'filebeat.inputs' is mainly used to read arbitrary log files while modules read common service logs with predefined settings that is quite handy.

filebeat.modules:
  - module: nginx
    access:
      enabled: true
      var.paths: /var/log/nginx/access.log
    error:
      enabled: true
      var.paths: /var/log/nginx/error.log

filebeat.inputs:
  - type: log
    enabled: true
    paths:
  - /var/log/myJson.log
    scan_frequency: 10s
    exclude_files: ['.gz$','~$']

    fields_under_root: true
    fields:
      hostname: "${HOSTNAME:-}"

    json.keys_under_root: true
    json.overwrite_keys: false

    json.message_key: "hostHeader"
    include_lines: ['www.bing.com','www.example.com']
    exclude_lines: ['a-shit-hostHeader','b-shit-HostHeader']

'paths' must be unique among multiple inputs. If more than one inputs harvests the same file, it would lead to unexpected behavior.
Only log entries whose 'hostHeader' value matching 'include_lines' will be harvested.
'include_lines' is applied before 'exclude_lines' regardless of the order they appear.
Read more about modules configuration.

Processors

Processor is probably one of the most important general option. A processor filters input logs by adding/dropping/modifying fields and/or events. It resembles 'exlude_files', 'include_lines' and 'exclude_lines' which are applied upon inputs. Processors, on the other hand, applies to fields/events after inputs and before put to outputs.

An event can go through a chain of processors in the exact order they are defined, before sending out.
```
processors:
  - include_fields:
      fields: ["cpu"]
  - drop_fields:
      fields: ['@timestamp', '@metadata', "log", "tags", "input", "ecs", "host", "agent", "message"]
  - drop_event:
      when:
        equals:
          http.code: 200
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 1
      target: ""
      overwrite_keys: true
```
More about processors, please check the 'Filebeat Processors' section below.
Outputs

Here is an excerpt of outputing to Kafka:
```
output.kafka:
  enabled: true
  hosts: ["logger1:9092", "logger2:9092", "logger3:9092"] # initial broker seeds to retrieve metadata of Kafka and Zookeeper

  topic: "cdn-sre"
  #topics:

  partition.round_robin:
    reachable_only: true
    group_events: 3
  version: '2.0.0'
  codec.json:
    pretty: false
    escape_html: false
  worker: 3
  compression: gzip
  required_acks: 1
  max_message_bytes: 1000000
  client_id: ${HOSTNAME:localhost}
```
1. The 'hosts' provides only the initial list of Kafka listeners from which to fetch the whole cluster metadata, including the full list of advertised.listeners where events are published to.
2. Besides the 'topic' directive, there is also a 'topics' sub-directive that filters events and publish them to different Kafka topics. It is a useful feature is not constrained by the one and only one output rule. That is to say, 'output.kafka' can publish events to multiple topics.
3. 'group_events' and 'worker' can be set the same value.
4. 'max_message_bytes' should be equal to or less than the Kafka broker's 'message.max.bytes'
Like the console consumer/producer of Kafka, Filebeat can output events into console and/or files.
```
output.file:
  enabled: true
  codec.json:
    pretty: true
    escape_html: false
  path: ${path.logs}
  filename: filebeat.out
  rotate_every_kb: 10000
  number_of_files: 7

output.console:
  enabled: true
  codec.json:
    pretty: true
    escape_html: false
```
Beats support one and only one output. If you desire multiple outputs, then send to Logstash or Kafka first.

Filebeat Pre-start

This applies to all beats applications.

On POSIX systems, configuration files are subject to ownership and permission checks:

The owner of configuration files must be the user that is running the beat applications or the root account.
The permissions of configuration files permit 'write' only by the owner itself which can be 0644 or 0600.
The permission and owner checks can be disabled temperarily on command line by --strict.perms=false.

Filebeat supports a set of commands like help, test, export, modules etc. We can use test to check configuration syntax:

user@tux ~ $ beatname help test
user@tux ~ $ beatname test config -c /path/to/beatname1.yml

# verify if Filebeat can connect the outputs:
user@tux ~ $ beatname test output -c /path/to/beatname1.yml

By default, all built-in modules are disabled, the module command can manage modules on the fly, similar to the mechanism of 'filebeat.config.inputs':

user@tux ~ $ filebeat modules list
user@tux ~ $ filebeat modules enable nginx
user@tux ~ $ filebeat modules disable nginx

Command export exports current config to STDOUT without comments:

user@tux ~ $ filebeat export config

Start Filebeat

Firstly enable Filebeat on boot:

user@tux ~ $ sudo chkconfig --add filebeat
user@tux ~ $ sudo chkconfig --list filebeat

chkconfig creates corresponding symlinks under /etc/rc.x/ according to the Chkconfig Header. /usr/lib/systemd/system-generators/systemd-sysv-generator of Systemd will then auto-create the unit file according to LSB Header.

Start Filebeat:

user@tux ~ $ sudo service filebeat status
user@tux ~ $ sudo service filebeat start

We cannot provide filebeat options and arguments by /etc/init.d/filebeat script. Alternatively, we can manually start the service:

# run in foreground with '-e' option
user@tux ~ $ sudo /usr/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat/
user@tux ~ $ sudo /path/to/filebeat -e -c /path/to/filebeat.yml

# run in background; enable 'NOPASSWD:' or use '-b' option
user@tux ~ $ nohup sudo -b ./filebeat -c apple/filebeat.yml > logs/nohup.out 2>&1 &
user@tux ~ $ sudo -b nohup ./filebeat -c apple/filebeat.yml > logs/nohup.out 2>&1 &

The order of 'sudo' and 'nohup' does not matter in terms of 'SIGHUP'

But it makes a difference which part the 'sudo' is applied to.
The -e option is useful for debugging in that it logs to STDERR and disables syslog/file output.

Stop Filebeat

If Filebeat is launched as a system service, stop it via the system service management functionality.
If Filebeat is invoked in the console, stop it by Ctrl-c or 'SIGTERM' signal.

Filebeat Processors

The libbeat library provides processors for:

reducing the number of exported fields;
enhancing events with additional metadata;
performing additional processing and decoding.

For example, to drop messages beginning with 'DBG:':

processors:
  - drop_event:
    when:
      regex:
        message: '^DBG:'

A processor consists of a name, an optional when condition and a set of parameters:

processors:
- <processor_name>:
    when:
      <condition>
    <parameters>

- <processor_name>:
    when:
      <condition>
    <parameters>

'processor_name' is a one of the pre-defined processor name.
when is a filter condition.
'when' and 'parameters' must be defined under the namespace of 'processor_name' which is a list item. The abbreviated form is - <processor_name>: {when: {<condition>},<parameters>}. Consequently, they are indented further compared with 'processor_name'.
1. It a list item.
2. The item value is a dictionary 'key: value' pair.
3. The value of the pair comprises lists and dictionaries.

Processors are applied in the order they are defined and are valid at different levels:

At the top-level (root level) in 'filebeat.yml' for all collected events;
Under a specific input applied only to that particular input. When to drop universal fields (i.e. 'agent' and 'ecs'), 'drop_fields' should be defined in the top level.

Decode Json

If a log entry is in Json format, Filebeat will escape the Json object like this:

{ "outer": "value", "inner": "{\"data\": \"value\"}" }

From the sample outpout, the field 'inner' has a Json object escaped. To decode a Json field, means removing the escaping behaviour.

Here is a hands-on schema on how to correctly decode collected Json logs under the field of 'message':

- type: log
  # ...

  exclude_files: ['.gz$','~$']

  json.keys_under_root: true
  json.overwrite_keys: false
  json.message_key: "host" # apply line and multiline filtering in accord with the specified Json key
  include_lines: ['www.example.com']
  exclude_lines: ['shit']
  # ...

processors:
- decode_json_fields:
    fields: ["message"]
    process_array: false
    max_depth: 1
    target: ""
    overwrite_keys: false
- drop_fields:
    fields: ['@timestamp', '@metadata', "log", "tags", "input", "ecs", "host", "agent", "message"]

Json Decoding can be configured at either 'inputs' level or 'processors' level.
Regarding per-input level, as long as any one of the 'json.keys_under_root', 'json.overwrite_keys' and 'json.message_key' directives appear, Json decoding is applied.
1. 'json.keys_under_root' places the decode Json at the root level.
2. Decoded 'key: value' pairs will overwrite built-in pairs that share the same key if 'json.overwrite_keys' is enabled.
3. Json decoding happens before line (i.e. 'exclude_lines') filtering and multiline filtering. Hence, line and multiline filtering does not affect the original Json event as it is transformed.
  
  The 'json.message_key' specifies a decoded Json key and applies line and multiline filtering to the event according the key value.
'decode_json_fields' is a global processor that applies to all inputs, with the 'fields' directive pointing to event fileds that require Json decoding.
1. 'target: ""' is equivalent to 'json.keys_under_root'.
2. 'overwrite_keys' and 'json.overwrite_keys' are equivalent as well.
'@metadata' and '@timestamp' cannot be dropped even they appear in in the list. Filebeat requires '@metadata' when sending out events (i.e. to Logstash).

A particular setup flow is like 'Filebeat - Kafka - Logstash'. We filter out the two fields in Logstash.
IMPORTANT: Filebeat refuses to launch on some systems when either 'overwrite_keys' or 'json.overwrite_keys' is enabled. Check issue 6381 and issue 1378. Therefore, it is recommended to leave those two directives disabled.

Gitlab Jenkins

2019-04-03T00:00:00+00:00

ABCs
Installation
QA

ABCs

Jenkins is a Continuous Integration (CI) utility. Among others, which is mostly used to automate building, testing, delivering or deploying packages upon sources pushed to the mainlin repository.
Gitlab requires Webhooks.
Jenkins requires gitlab-plugin or Gitlab Webhook Plugin.
Generally speaking, the Webhook defined in Gitlag repository will call the Gitlab Webhook Plugin associated with Jenkins server which in return performs the build.

Read Continuous Integration with Jenkins and GitLab.

Installation

Choose the Long-term Support (LTS).
Java is located under '/home/shibin.luo/collectflume/jdk1.8.0_112/bin/java'.

Yum

~ # sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
~ # sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
~ # sudo yum install jenkins

https://medium.com/@teeks99/continuous-integration-with-jenkins-and-gitlab-fa770c62e88a

QA

ccssh5 BGP-GZ-b-3gb
Build status feedback
Jenkins master/slave.
Java

rxvt-unicode

2019-03-12T00:00:00+00:00

Installation
Daemon Mode
Configuration

Installation

[root@host ~ #] pacman -S rxvt-unicode

[root@host ~ #] echo 'x11-terms/rxvt-unicode xft perl 256-color' >> /etc/portage/package.use/urxvt
# Add to '/etc/portage/patches/x11-terms/rxvt-unicode-9.21/' the patch 'secondary-wheel.patch'
[root@host ~ #] emerge -avt x11-terms/rxvt-unicode

Daemon Mode

Like Emacs, daemon mode speeds up startup speed.

Put the following code line into auto-start script or X setting.

urxvtd --quiet --opendisplay --fork

To launch client side, just urxvtc.

If the daemon is terminated, all running clients would be terminated as well.
By default, the local listening socket is located at ~/.urxvt/urxvtd-hostname.

Configuration

Add to ~/.Xresources:

#include ".Xresources.d/urxvt"

Then customize options there. To configure Solarized color theme, clone the resources, the include the pathname.

# .Xresources.d/urxvt

#include "Xresources.dark"

X

2019-03-11T00:00:00+00:00

X
Xorg
Video Driver
1. Hardware Acceleration
KDE
Awesome - Window Manager
Xinit
xserverrc
xinitrc
Automatic startx on login
Touchpad
XKB
SysRq

X

X - window server
1. Xorg
2. Wayland
3. ~/.Xdefaults. Deprecated but used by ancient applications like urxvt. Now replaced by ~/.Xresources. X resources for applications in the form key: value.
4. ~/.Xmodmap. Deprecated by XKB (either through setxkbmap or directly in Xorg conf).
Video Driver
1. Basic GPU driver
2. Hardware Acceleration
Window Manager (WM) - manage GUI windows
1. Awesome
2. i3
Display Manager (DM) - control graphical login and start WM/DE
1. SDDM
2. GDM
3. ~/.xsessionrc. Config of DM (i.e. choose WM).
4. ~/.xprofile. General config of X. Equivalent of xinitrc.
xinit/startx - manually start WM/DE
1. ~/.xinitrc.
  1. General config of X. Equivalent of xprofile.
  2. Set WM.
2. ~/.xserverrc. Start X server.
Desktop (DE) = WM + DM + more
1. Xfce
2. KDE Plasma
3. Gnome

Xorg

[root@host ~]# pacman -S xorg-server
[root@host ~]# ls -al /usr/bin/X
/usr/bin/X: symboli link to /usr/bin/Xorg

X is the generic name of X Window System display server. The binary X is a symbolic link to the real implementation - Xorg.

Video Driver

Firstly, identify vedio cards:

[root@host ~]# lspci | grep -e VGA -e 3D

The command returns:

Intel HD Graphics 520
NVIDIA GeForce 920M

So the computer has an integrated Intel video card and a discrete NVIDIA video card respectivelly: it is called NVIDIA Optimus. We have several choices to deal with NVIDIA Optimus: turn off one of the video card, use Bumbelee, use Nvidia-xrun etc. Nvidia-xrun is recommended:

Nvidia-xrun is a utility to allow Nvidia optimus enabled laptops run X server with discrete nvidia graphics on demand. This solution offers full GPU utilization, compatibility and better performance than Bumblebee.

If this Arch Linux is a VirtualBox guest, then install VirtualBox guest additions instead.

Check Xorg video driver.

Hardware Acceleration

There are two libraries that support hardware acceleration, namely VA-API and VDPAU. Most applications use the VA-API but we can install both libraries.

For applications that use the VA-API library:

~ $ sudo pacman -S libva-mesa-driver
~ $ vainfo

For applications that use the VDPAU library:

~ $ sudo pacman -S mesa-vdpau
~ $ vdpauinfo

Test:

~ $ mpv --hwdec=auto <video_filename>

KDE

Currently, KDE sucks! This section only shows how to install KDE plasma.

~ $ sudo pacman -S xorg-server
~ $ reboot

~ $ sudo pacman -S plasma-meta
~ $ sudo pacman -S sddm
~ $ systemctl enable sddm

~ $ reboot

It seems that sddm is quite slow to launch Plasma. We need to increase system entropy by haveged and/or rng-tools:

~ $ sudo pacman -S rng-tools
~ $ sudo systemctl enable rngd
~ $ sudo systemctl start rngd

~ $ sudo pacman -S haveged
~ $ systemctl enable haveged
~ $ systemctl start haveged

It might relate to a kernel option unset. However, such hack uses pseudo-random to increase entropy and may leave your system in risks.

Awesome - Window Manager

[root@host ~]# pacman -S awesome
[user@host ~]$ awesome

# E: awesome: main:656: cannot open display (error 5)

Awesome is just a Window Manager. We need Displayer Manager or xinit/startx to launch X server before Window Manager.

Xinit

We don't need a Display Manager.

[root@host ~]# pacman -S xorg-xinit

startx is a wrapper script of xinit that provides some basic settings:

[root@host ~]# cat /usr/bin/startx

xserverrc

xserverrc is the file to start X server, sourced by xinit:

#!/bin/sh

# https://git.edevau.net/onkelbeh/GentooRepository/commit/9d68e491c96541f7cbf3d5766597906561bbf512
if [ -z "${XDG_VTNR}" ]; then
  [[ -t 0 && "$(tty)" == "/dev/tty1" && "$USER" != "root" && -z "$DISPLAY" ]] && exec /usr/bin/X -nolisten tcp -nolisten local -keeptty "$@" vt7
else
  [[ -t 0 && "${XDG_VTNR}" -eq 1 && "$USER" != "root" && -z "$DISPLAY" ]] && exec /usr/bin/X -nolisten tcp -nolisten local -keeptty "$@" vt${XDG_VTNR}
fi

When using a DM, the X server is running as root. The X server log is located at /var/log/Xorg.0.log. However, with xinit, X server is run under normal account. The X server log is located at ~/.local/share/xorg/Xorg.0.log.

The last argument vtXY is the virtual terminal that Xorg will use. By default, X will use the one where it is started.
Xorg -keeptty will redirect log of xinit iteself to Xorg.0.log as well. Check man Xserver and man Xorg for more options.

xinitrc

~/.xinitrc defaults to /etc/X11/xinit/xinitrc. This file is used to launch the DE/WM.

[user@host ~]$ cp /path/to/xinitrc.bak ~/.xinitrc

# ~/.xinitrc

#exec $command
exec awesome

Commands after exec won't be executed as it replace the current shell. If any other commands are required, put them before exec line.

That is enough, just type startx.

Add the following code to ~/.bash_profile:

# auto startx
if shopt -q login_shell; then
  if [ -z "${XDG_VTNR}" ]; then
    [[ -t 0 && "$(tty)" == "/dev/tty1" && "$USER" != "root" && -z "$DISPLAY" ]] && exec startx 2>&1 | tee "$HOME"/.startx.log
  else
    [[ -t 0 && "${XDG_VTNR}" -eq 1 && "$USER" != "root" && -z "$DISPLAY" ]] && exec startx 2>&1 | tee "$HOME"/.startx.log
  fi
fi

We cal also append Xorg options to startx directly like startx -- -keeptty.

Touchpad

xorg-server, by default, depends on and uses xf86-input-libinput for all input devices:

[user@tux ~]$ fgrep -e "Using input driver " ~/.local/share/Xorg/Xorg.0.log
[user@tux ~]$ libinput list-devices
[user@tux ~]$ man 5 xorg.conf ; man 4 libinput

Before configuring Xorg, glimpse at /usr/share/X11/xorg.conf.d and read the INPUTCLASS SECTION of man 5 xorg.conf.

To configure Touchpad, we copy the skeleton from 40-libinput.conf:

[root@tux #]$ cp /usr/share/X11/xorg.conf.d/40-libinput.conf /etc/X11/xorg.conf.d/30-touchpad.conf

Identifier is any reasonable string specific to the device configured. I choose the exact device name from Xorg log.
Turn on tap to click.
Two-finger click simulates mouse middle buttion.
Three-finger click simulates mouse right button.
When mouse and touchpad events are detected simutaneously, ignore that of mouse.

Section "InputClass"
        Identifier "DLL06F2:00 06CB:75DA Touchpad"
        MatchDevicePath "/dev/input/event*"
        MatchIsTouchpad "on"
        Driver "libinput"
                Option "Tapping" "on"
                Option "TappingButtonMap" "lmr"
                Option "ClickMethod" "clickfinger"
                Option "NaturalScrolling" "true"
                Option "ScrollMethod" "twofinger"
EndSection

Section "InputClass"
        Identifier "Touchpad ignore duplicates"
        MatchDevicePath "/dev/input/mouse*"
        MatchIsTouchpad "on"
        MatchOS "Linux"
                Option "Ignore" "on"
EndSection

XKB

Post archlinux uses /etc/vconsole.conf and/or loadkeys to set keyboard layout for virtual terminal. When it comes to X server, we utilize XkbOptions of Xorg.conf or setxkbmap in .xinitrc.

Basically, I wanna switch 'Caps_Lock' and 'Control_L', or even turn off 'Caps_Lock'.
Check /usr/share/X11/xkb/rules/base.lst for keyboard model (i.e. latitude) and layout (i.e. us).
Run the following command to see similar options:

[root@tux #]$ grep -E "(ctrl|caps):" /usr/share/X11/xkb/rules/base.lst

By Xorg.conf:

# /etc/X11/xorg.conf.d/10-keyboard.conf

Section "InputClass"
        Identifier "Keyboard switch Ctrl and CapsLk"
        MatchIsKeyboard "on"
        Option "XkbOptions" "ctrl:nocaps"
EndSection

By setxkbmap (overrides settings in Xorg.conf) in ~/.xinitrc or ~/.xprofile.

setxkbmap -model latitude -layout us -option ctrl:swapcaps

SysRq

Check Gentoo SysRq.

Emacs C/C++ IDE

2018-12-03T00:00:00+00:00

Tag
Semantics and Completion
GNU BLOBAL
Installation
GLOBAL tag database
Usage
1. Multiple matches
Code Completion
1. Header file completion
References

This tutorial demonstrates the process of setting up a C/C++ development environment within Emacs. It mainly involves GNU GLOBAL and ggtags.

Tag

Tag is just an index file storing language objects from project sources, which allows these objects to be quickly and easily located and queried by text editors or other utilities.

In regard to tag environment, we should differentiate concepts of command and format. etags generates 'TAGS' files understood by Emacs while ctags generate 'tags' files for Vi.

'tags' format contains far richer metadata than 'TAGS' format that cannot be interpreted by Emacs. In spite of that, both formats are more or less the same.

Emacs package has built-in etags and ctags commands. There exist other standalone package providing these commands, such as universal-ctags and exuberant-ctags(deprecated).

Nowadays, ctags from universal-ctags can generate TAGS format with -e option. Hence, we can concentrate on ctags. Besides, ctags supports more languages.

Another package cscope that is more powerful than ctags is, in reality, dead. Originally, it encompasses much more features, especially when C and C++ are concerned. Moreover, it has its own tag database unlike plain-text format. You will find, from this tutorial, that cscope is silimar to GNU GLOBAL but support fewer languages.

Semantics and Completion

Tag only cares about sources navigation. Upon writing code, we want semantic parser to understand source objects and then achieve auto completion.

There exists a host of auto-completion packages, like company-mode.

GNU BLOBAL

GNU GLOBAL is an universal source code tagging system across diverse platforms and environments, such as Emacs, Vim, Bash Shell, various web browsers.

Through the command global, we can easily locate various objects, such as functions, macros, structs, classes etc. The gtags command is similar to etags and ctags, but is different from them in the following two points:

Independence of any editor;
Capability to treat both definition and reference;
Tag database support.

GLOBAL supports 6 langages as C, C++, Yacc, Java, PHP4 and assembly by built-in semantic parser. Actually, GLOBAL brings along gtags-cscope support which, though outdated, can replace the default parser when parsing C/C++.

To support a wider range, we can combine GLOBAL, Pygments, etags/ctags plug-in parser. As mentioned above, ctags is a superset of etags. Ideally, we prefer ctags of universal-ctags over the built-in one of Emacs.

Installation

Universal Ctags

Before make GLOBAL, we should build universal-ctags ctags. The goal is to replace the default /usr/bin/ctags from Emacs.

Then supply

./configure --prefix=<PREFIX> --with-exuberant-ctags=/usr/local/bin/ctags
# -or-
./configure --prefix=<PREFIX> --with-universal-ctags=/usr/local/bin/ctags

argument, telling GLOBAL where universal-ctags locates.

I won't discuss building universal-ctags in this post and maybe get back to topic later on. Instead, the default GLOBAL built-in parser is enough for C/C++.

Emerge GLOBAL

Since GLOBAL is independent of editors, it requires imtermediate bridges. To interact with Emacs, it has gtags.el. We should enable the emacs USE:

root@tux ~ # echo 'dev-util/global emacs' >> /etc/portage/package.use/global (opt)
root@tux ~ # emerge -avt dev-util/global

Actually, Emacs also has etags.el for default etags and ctags.el for default ctags.

ggtags

However, gtags.el is quite primitve, we would like the improved version ggtags. So just install ggtags.el through ELPA list-packages.

After installing ggtags, add to init.el:

(require 'ggtags)
(add-hook 'c-mode-common-hook
          (lambda ()
            (when (derived-mode-p 'c-mode 'c++-mode 'java-mode 'asm-mode)
              (ggtags-mode 1))))

Another replacement is helm-gtags. I will leave it to yourself.

pygments

In order to support syntax highlighting, we need pygments. It can be installed through package manager or within Python virtual environment.

After installing pygments, we also require an intermediate plugin for GLOBAL, namely Pygments Plug-in Parser for GNU GLOBAL.

Since 6.3.2, GLOBAL includes it by default. For older version, we should install it manually.

GLOBAL tag database

A tag is a name of an object/entity in source code. An object can be a variable, a method definition, an include-operator …

A tag contains information like name of the entity, location in the source code, and which source file it belongs to.

GNU GLOBAL makes use of three tag databases:

GTAGS	definition database
GRTAGS	reference database
GPATH	path name database

A definition of a tag is where the tag is implemented (not declaration). For example, a function definition is the block where the function is actually implemented.

On the contrary, a reference of a tag is where it is used in a source tree, but not where it is defined.

Usage

Up to now, we have almost finished the installation and configuration process. Next, we will examine how these components collaborate with each other to achieve an intelligent development environment.

The most commonly used command is ggtags-find-tag-dwim bound to M-., which find a tag by context. It will jump to a reference if the tag at point is a definition. Rather, it will jump to a definition if the tag at point is a reference. The the tag at point is a include header, it jumps to that header file.

Other useful functions are:

ggtags-find-definition	find definition tags
ggtags-find-reference	find reference tags
ggtags-find-other-symbol	find tags that have no definitions
ggtags-find-tag-regexp	find definition tags by regexp
ggtags-query-replace	do a query & replace in all files found in a search
ggtags-find-file	find a file from all the files indexed by `gtags`
ggtags-visit-project-root	open the project root in `dired`

Multiple matches

When a search find multiple matches, a new buffer named ggtags-global is created and ggtags-navigation-mode is enabled to faciliate entry selection by following commands:

M-n	move to the next match
M-p	move to the previous match
M-}	move to the next file
M-{	move to the previous file
M-=	move to the file where navigation session starts
M-<	move to the first match
M->	move to the last match
C-M-s, M-s s	use `isearch` to find a match
M-,	abort and go back to the location where search was started

Code Completion

Install company-mode and add to init.el:

(require 'company) ; optional
(add-hook 'after-init-hook 'global-company-mode)

By default, company-clang backend to retrieve standard libraries (/usr/include/) but not our project files. We can make use of company-complete that encompasses company-clang.

(setq company-backends (delete 'company-semantic company-backends)) ; optional
(define-key c-mode-map  [(tab)] 'company-complete)
(define-key c++-mode-map  [(tab)] 'company-complete)

By this snippet, we use 'company-complete' which calls 'company-clang' on demand while completes project entities. The very first reference suggests deleting company-semantic from company-backends. However, as far as I know, it is unneccessary.

Then we resort to GNU Emacs Manual - Per-Directory Local Variables, creating a special file .dir-locals.el under our project root. This file is usually used to define directory-local variables.

Emacs will read it when it visits any file in that directory or any of its subdirectories, and apply the settings it specifies to the file' s buffer.

We set company-clang-arguments in this file, specifying the project header files' location, namly the include path.

((nil . ((company-clang-arguments . ("-I/home/<user>/project_root/include1/"
                                     "-I/home/<user>/project_root/include2/")))))

That is enough! company-clang is able to retrieve completion candidates from our project sources.

Header file completion

company-c-headers helps complete #include statements by searching header filenames.

After installation, add to init.el:

(add-to-list 'company-backends 'company-c-headers)

By default, company-c-headers only covers C headers under /usr/include/ and /usr/local/include/. To support C++ headers, we should:

(add-to-list 'company-c-headers-path-system "/usr/include/c++/4.8/")

Simiarly, to complete projects header files, set company-c-headers-path-user in .dir-locals.el.

References

MTP

2018-07-18T00:00:00+00:00

MTP (Media Transfer Protocol) is a protocol to allow the transfer of files to external devices. It is provided by several programs, most of them depending on FUSE.

From both the iOS and Android posts, we resort to Thunar (with gvfs support). In this post, I will show manually how to manage MTP device (Windows phone, Android, and even Creative Zen player) files. Irrespective of methods adopted, almost all packages implementing MTP depend on libmtp and/or FUSE.

Android doesn't support mounting your phone as a USB mass storage device anymore (from Android 3.0 onwards). Instead, put device into MTP mode and then access to device through MTP package. Take Android for instance, upon connected to computer, the device would prompts you to choose the mode, usually writing "charging only", "transmit files" and "tansmit photos". Just select the 2nd option.

There are a bunch of MTP packages like mtpfs, go-mtpfs, jmtpfs etc. Specially, gphotofs is a FUSE-based file system for interfacing with digital cameras using the gphoto2 application.

mtpfs is almost orphaned. Here I will take jmtpfs for example. Please make sure:

Relevant Udev rule is created for your device like that in Android post.
Also check if user belongs to plugdev group.

root@tux / # emerge -avt jmtpfs
user@tux ~ $ mkdir ~/mnt
user@tux ~ $ jmtpfs -h; jmtpfs -l
user@tux ~ $ jmtpfs ~/mnt
user@tux ~ $ ls ~/mnt
user@tux ~ $ fusermount -u ~/mnt

That's all!

To simplify the process by regular mount, we create jmtpfs entry in /etc/fstab.

Firstly, make symbolic similar to mount.fuse:

root@tux ~ # ln -sv /usr/bin/jmtpfs /usr/bin/mount.jmtpfs

Then, add to /etc/fstab:

# /etc/fstab

jmtpfs /home/user/run/motog fuse noauto,noatime,nodev,user,uid=1000,gid=1000 0 0

Of the options, user means any other accounts besides root can mount the entry.

Finally:

user@tux ~ $ mount ~/run/motog
user@tux ~ $ ls ~/run/motog

Interestingly, you will find even root cannot access to /home/user/run/motog, which is FUSE security restriction! What is more, umount is not allowed. Check man page of mount.fuse. To allow other accounts and/or root to read/write device files and umount, we should turn on 'user_allow_other' option in FUSE configuration /etc/fuse.conf:

# /etc/fuse.conf

# Allow non-root users to specify the 'allow_other' or 'allow_root'
# mount options.
#
user_allow_other

Append 'allow_other' or 'allow_root' to mounting options. Before this takes effect, remember to reboot system:

# /etc/fstab

jmtpfs /home/user/run/motog fuse noauto,noatime,nodev,user,allow_root,uid=1000,gid=1000 0 0

Up to now, I have demonstrated mounting MTP devices and file permissions thereof. To detach the device, most libmtp based packages resort to fusermount:

user@tux ~ $ fusermount -u ~/run/motog

If 'allow_other' or 'allow_root' is enabled, then use umount:

user@tux ~ $ umount ~/run/motog

Bitcoin Whitepaper

2018-06-13T00:00:00+00:00

Bitcoin

p2p timestamp system

chronological cryptography vs trust on 3rd party institution;
solve double-spending issue;
depends on the fact that honest nodes collectively outpace attacker nodes;
proof-of-work;

transaction

we mine a block (and get coin reward) istead of a coin; bitcoins are created each time a user discovers a new block.
coin is defined as a set of transactions digitally signed; payee can verify the payer and cannot verify whether it's the earliest transaction, thus cannot exclude double-spending;
all transactions are publicly announced;
a transaction has inputs (several) and outputs (at most 2). Inputs are existing outputs from other transactions for you. Hence, transaction actually takes outputs for you and then generate new outputs, one of which is for payee and the other is for change. The remaining (sum of inputs minus sum of outputs) is Bitcoin fee.
we need 'change' because outputs cannot be partially spent. You can only spend your output as a whole, which explains why we could have 'change' output.
wallet (account balance) of a peer (identified by private key) is derived from all transaction outputs for his public key, namely 'pay-to-pubkey-hash' transactions.
To get the concept of transactoin, read https://bitcoin.stackexchange.com/a/20623 and https://bitcoin.stackexchange.com/q/736

block reward = block subsidy + transactions fees

timestamp server

another hash on the whole transaction block;
each item in the block consists of a transaction and timestamp thereof; transaction items in a block are usually unrelated.
payee can verify it is the earliest and not a double-spending transaction. kick out double-spending transaction.
announce the hash and timestamp together.

proof-of-work

how to hash? simple sha-256 is not enough! Add nonce to find a special hash value that starts with a number of zero bits. The number is increased overtime (more and more difficult).
proof-of-work is where our 'mining resources' are invested and represented by 'minining resources' invested.
proof-of-work is deliberately designed to avoid double-spending such that attacker cannot catch up.
Nodes compete to confirm a transaction, namely compete to tally!

p2p networking process

new transaction is simply hashed and signed before announcement.
a block (local copy) comprises a set of transaction items, previous hash, nounce. received transaction along with timestamp are inserted into the transaction set.
difficult hash computational work for block, namely proof-of-work.
success! a peer announces hash and the block.
other peers verify and accept the block.
other peers create a new block by replacing the 'Prev Hash' field and announce it.

Conceptually, there is only one valid (of course longest) blockchain over the network. But techniquely, there exist mutiple blockchains simutaneouly due to node joining/leaving, network latency, forks etc.

double-spending

Timestamp is acutally not critical element since attacker node can forge local time. Instead, the longest blockchain is accepted while other candidates are discarded.
Currently, user confirm a transaction if it buried over 6 (included) blocks. 6 blockes are enough to avoid catching up by powerful attacker. If an attacker double-spends coins by signing second transaction immediately (say one to Alice and another to Bob), only one of them (say that to Bob) is accepted when it is buried in a longer blockchain.
It is not really the Proof of Work which prevents double spends but rather the blockchain itself which prevents double spends. The Proof of Work is just one aspect of the blockchain.
Freshly-mined coins cannot be spent for 100 blocks. It is advisable to wait some additional time for a better chance that the transaction will be propagated by all nodes. Some older bitcoin clients won't show generated coinsas confirmed until they are 120 blocks deep.

Blockchain forks might cause oprphan blocks. Suppose a block owner spends the subsidy and rewards (newly coins), and get 6 confirmations. However, 6 is too small to guarantee that the new block would finally merged into mainnet. Maybe this block will be orphaned and newly coins disappear as well.

The key is that transaction with coinbase coins input has no source! We cannot track back! For transaction with old coins input, we can track back to other transaction output. Even though the block container become orphan, it will be confirmed by other block afterwards.

https://bitcoin.stackexchange.com/q/61385

incentive

the first transaction in a block is a special transaction that starts a new coin owned by the creator of the block; the transaction set is empty.
transaction fees; fee is given to the peer that achieve 'proof-of-work' - the one that find a block. Bitcoin fee is fixed independently of the amount of coins.
attacker with more mining resources probably chooses to be honest as that would earn more coins than stealing.

privacy

peer public key is anonymous though transactions are public.

Economy

Bitcoin amount limit would fail in deflation spiral. But that is not a problem as Bitcoin cannot be global currency.
Currently, miners mine bitcoins assuming price increase in the future.
People use bitcoins to bypass censorship.

中文

账本，区块链，blockchain;
矿池，asic hwardware node;
挖矿，mine, proof-of-work, hash with leading zeros;

todos

asic hardware;
testnet for fun;
p2p timestamp? what if fake node time?
centralized Bitcoin exchange?
how many bits of a hash is required?

reference

Bitcoin: A Peer-to-Peer Electronic Cash System
https://en.bitcoin.it/wiki/Controlled_supply
https://bitcoin.org/en/how-it-works
https://bitcoin.org/en/you-need-to-know

Awesome

2018-06-10T00:00:00+00:00

rc.lua
X terminal emulator
Autostart
Menu
Theme
wallpaper
Transparency
Screenshot
slock
1. xautolock
st terminal emulator

Awesome is a lightweight window manager and yet can be highly cusotmized to your own needs. This post mainly focuses on awesome configuration at ~/.config/awesome/rc.lua.

rc.lua

~ $ mkdir -p .config/awesome
~ $ cp /etc/xdg/awesome/rc.lua ~/.config/awesome/

We use Super-s to show key bindings.

X terminal emulator

st and lilyterm are recommended in that they are lightweight and supports UTF8 and CJK by nature.

-- rc.lua

terminal = "st"

Check syntax:

user@tux ~ # awesome -k

Then press Ctrl-Super-r to reload rc.lua.

Autostart

Desktop (XFCE, KDE etc.) has many autostart .desktop files in /etc/xdg/autostart/ or ~/.config/autostart/ to launch default applications on startup (i.e. input method, power manager etc.). Awesome requires manual settings:

Here is an example of VBoxClient-all autostart with awesome. Create ~/.config/awesome/autostart.sh:

#!/usr/bin/env bash

function run {
  if ! pgrep $1 ;
  then
    $@&
  fi
}

run VBoxClient-all
run fcitx-autostart

Make it executable:

[user@host ~]$ chmod +x ~/.config/awesome/autostart.sh

Check autostart.sh:

[user@host ~]$ ~/.config/awesome/autostart.sh

You can add any other programs to autostart by appending run executable [--arguments] to the end of autostart.sh.

Finally, add to the end of ~/.config/awesome/rc.lua:

-- awful.spawn.with_shell("~/.config/awesome/autostart.sh")
awful.spawn.with_shell(awful.util.getdir("config") .. "/autostart.sh")

Please be noted that, awesome restart does not restart scripts in autostart.sh.

-- rc.lua

myawesomemenu = {
   { "hotkeys", function() hotkeys_popup.show_help(nil, awful.screen.focused()) end },
   { "manual", terminal .. " -e man awesome" },
   { "edit config", editor_cmd .. " " .. awesome.conffile },
   { "aw restart", awesome.restart },
   { "aw quit", function() awesome.quit() end },
}

-- systemd-logind allows unpriviledged users to shutdown system
shutdownmenu = {
   { "hibernate", "systemctl hibernate" },
   { "suspend", "systemctl suspend" },
   { "reboot", "systemctl reboot" },
   { "shutdown", "systemctl poweroff" },
}

appsmenu = {
   { "emacs", "emacs" },
   { "thunar" , "thunar" },
   { "firefox", "firefox" },
   { "chrome", "chromium" },
   { "telegram", "telegram-desktop" },
}

settingsmenu = {
   { "sound", "st -e alsamixer" },
}

mymainmenu = awful.menu({ items = { { "awesome", myawesomemenu, beautiful.awesome_icon },
                                    { "shutdown", shutdownmenu },
                                    { "settings", settingsmenu },
                                    { "applications", appsmenu },
                                    { "files", "thunar" },
                                    { "editor", "emc" },
                                    { "browser", "firefox" },
                                    { "terminal", terminal }
                                  }
                        })

Trigger menu bar by Mod-p binding and then type the application keywords.

Theme

First, we create per-user theme directory and copy default theme there where we will do customization.

user@tux ~ $ mkdir -p ~/.config/awesome/themes/
user@tux ~ $ cp -a /usr/share/awesome/themes/default/ ~/.config/awesome/themes/

Afterwards, change beautiful.wallpaper value to the new directory in rc.lua:

-- rc.lua

-- beautiful.init(gears.filesystem.get_themes_dir() .. "default/theme.lua")
beautiful.init(awful.util.getdir("config") .. "/themes/default/theme.lua")
-- or
beautiful.init(awful.util.get_configuration_dir() .. "/themes/default/theme.lua")

For example, we can set useless gap to 5:

-- rc.lau

beautiful.useless_gap = 5

wallpaper

To set a wallpaper, change theme.wallpaper value in theme.lua:

-- theme.lua
theme.wallpaper = "~/.config/awesome/themes/default/wallpaper.png"

Alternatively, we can set wallpaper in rc.lua before function set_wallpaper(s):

-- rc.lua

beautiful.wallpaper = awful.util.get_configuration_dir() .. "sub-path/to/wallpaper.png"

Transparency

For wibox transparency, append transparency value (00 - FF) to bg_normal in theme.lua. 00 means totally transparent while FF is not transparent.

-- theme.lua

--theme.bg_normal     = "#222222"
theme.bg_normal     = "#22222200"

Alternatively, append bg = beautiful.bg_normal .. "0" to s.mywibox.

-- rc.lua

s.mywibox = awful.wibar({ position = "top", screen = s, bg = beautiful.bg_normal .. "00" })

Screenshot

root@tux / # emerge -avt scrot
user@tux ~ $ scrot -s -q50 -cd3 -t20

Add to the globalkeys array:

-- rc.lua

awful.key({ }, "Print", nil, function () awful.spawn("scrot -s -q50 -e 'mv $f ~/Desktop/ 2>/dev/null'") end,
          {description = "take screenshot", group = "launcher"}),
-- or
awful.key({ }, "Print", function () awful.util.spawn_with_shell("scrot -s -q50 -e 'mv $f ~/Desktop/ 2>/dev/null'", false) end,
          {description = "take screenshot", group = "launcher"}),

Pay attention to the nil part, otherwise we could not take screenshot with -s.

slock

root@tux / # echo 'x11-misc/slock savedconfig' >> /etc/portage/package.use/slock
root@tux / # emerge -avt slock (Additionally, I have add in the *dpms* patch.)

Set key binding:

-- rc.lua

awful.key({ modkey }, "l", function () awful.util.spawn("slock", false) end,
          {description = "lock screen"}),
awful.key({ }, "XF86ScreenSaver", function () awful.util.spawn("slock", false) end,
          {description = "lock screen"}),

XF86ScreenSaver means Fn + F2 key combination. To determine the key name, just press relevant key or binding in Emacs GUI or resort to sev, showkey xmodkey etc.

To prevent from virtual terminal switching or killing X, use the following Xorg config:

# 30-slock.conf
Section "ServerFlags"
    Option "DontVTSwitch" "True"
    Option "DontZap"      "True"
EndSection

Otherwise, screenlock is meaningless.

xautolock

To automatically lock screen after a period of inactivity, we use xautolock:

root@tux / # emerge -avt xautolock

We create a script autostart script ~/opt/bin/xautoslock.sh:

#!/bin/sh

exec xautolock -detectsleep \
     -time 3 -locker "slock" \
     -corners "00+0" -cornerdelay 10 \
     -notify 10 \
     -notifier "notify-send -u critical -t 10000 -- 'LOCKING screen in 10 seconds'"

Then, assign executable permission (chmod +x xautoslock.sh) before appending it to autostart.sh:

run ~/opt/bin/xautoslock.sh

st terminal emulator

Firstly, we enable savedconfig:

root@tux / # echo 'x11-terms/st savedconfig' >> /etc/portage/package.use/st

st is quite basic and lack features like copy/paste, scrollback etc. We should apply a few local patches by /etc/portage/patches before they are merged into portage:

# make sure the slot number is correct
root@tux / # mkdir -p /etc/portage/patches/x11-terms/st-0.7

If the directory made does not contain version and/or slot specifier (/etc/portage/patches/x11-terms/st/), then everytime you emerge (i.e. update) the package, patches are definitely applied, which would fail if new versions in portage already apply the patches, causing patches to be applied twice.

Grab patch files from st official page and put them into directory just created. clipboard, scrollback, spoiler, visualbell, xresources and solarized are recommended. solarized-both-0.8.1 has bug and please use solarized-dark-0.8.1 instead.

Sometimes, a patch must be applied before another, we can rename the patch files with number prefix as patches are apllied in lexicographical order.

Then step into the package's ebuild directory and check patches:

root@tux / # cd /us/portage/x11-terms/st
root@tux / # ebuild st-0.7.ebuild clean prepare

We get the following log:

* st-0.7.tar.gz BLAKE2B SHA512 size ;-) ...
>>> Unpacking source...
>>> Unpacking st-0.7.tar.gz to /var/tmp/portage/x11-terms/st-0.7/work
>>> Source unpacked in /var/tmp/portage/x11-terms/st-0.7/work
>>> Preparing source in /var/tmp/portage/x11-terms/st-0.7/work/st-0.7 ...
* Applying st-clipboard-0.7.diff ...
* Applying st-scrollback-0.7.diff ...
* User patches applied.
* Checking existence of //etc/portage/savedconfig//x11-terms/st-0.7 ...
* found //etc/portage/savedconfig//x11-terms/st-0.7
* Building using saved configfile //etc/portage/savedconfig//x11-terms/st-0.7
>>> Source prepared.

We find a line User patches applied. Bt this patching method, there is no need to update ebuild as long as it is EAPI6 and/or include eapply_user instruction.

st behaviour could be further adjusted from savedconfig. By default, a copy will be saved as /etc/portage/savedconfig/x11-terms/st-0.7. For example, we can change default shell, termname, font, etc. To avoid re-compiling upon configuration update, we apply xresources patch that supports Xresources.

If a patch happended to update saveconfig USE, and later on you removed the patch. The compiling would be probably fail! Either remove saved config or take back the patch.

Finally, just emerge!

As of 0.7, the scrollback patch ignores saved config and report histsize undeclared. Either bump to new slot or remove saved config file.

To change termname to xterm-256color:

# ~/.Xresources

st.termname: xterm-256color

ENG phonetics

2018-06-06T00:00:00+00:00

References
Tips
IPA
发音过程
Vowel vs. Consonant
长短元音符号
Syllable 音节

References

BBC pronunciation
english phonology
iowa phonetics
Cambrige: an international pronunciation course
American Accent Training
https://v.qq.com/x/page/w0146n8467c.html

Tips

Open Open Open your mouth;

It means not only begin to practice, but keep your mouth and tongue movement to certain extent, speaking loud.
Check dictionary.
Film and TV series are not a proper guide. Use standard prounciation materials.
Improve intonation.

IPA

The International Phonetic Alphabet (IPA) is an alphabetic system of phonetic notation based primarily on the Latin alphabet.

The IPA is designed to represent only those qualities of speech that are part of oral language: phones, phonemes, intonation and the separation of words and syllables.

IPA symbols are composed of one or more elements of two basic types, letters and diacritics （变音符号）. English letter ⟨t⟩ may be transcribed in IPA with a single letter, [t], or with a letter plus diacritics, [t̺ʰ], depending on how precise one wishes to be. Often, slashes are used to signal broad （宽式注音） or phonemic transcription （音位注音，也称严式注音）; thus, /t/ is less specific than, and could refer to, either [t̺ʰ] or [t], depending on the context and language.

1888 年 8 月第一套国际音标符号诞生，这套符号设立的原则是：每一个独立的音都有一个独立的符号相对应，相同的符号在任何语言中都表示同一个发音。

发音过程

气流 airstream: 从肺部出来；
发声 phonation: 喉部（声带）决定清音和浊音；
口鼻 oro-nasal: 气流进入口腔还是鼻腔，形成鼻音和口腔音；
发音 articulation: 通过舌的运动改变口腔的形状，发出各种不同的口腔音。

结合这四个步骤，我们可以对发音进行不同的分类。

Vowel vs. Consonant

语音一般分为辅音 (consonant) 和元音 (vowel) 两大类，元音和辅音的根本区别在于气流是否受阻。注意是“受阻”不是“完全没有”。发辅音时，声道紧闭或声道变窄使气流无法排出，或排出时会产生能够辨别出的摩擦。发元音时气流可以相对不受阻碍地从口腔或鼻腔中排出而不会产生类似的现象。

A vowel is a sound such as the ones represented in writing by the letters 'a', 'e' 'i', 'o' and 'u', which you pronounce with your mouth open, allowing the air to flow through it.

A consonant is a sound such as 'p', 'f', 'n', or 't' which you pronounce by stopping the air flowing freely through your mouth.

因此，在描述辅音音姿的时候，特别需要考虑的是气流在口腔中受阻的位置和方式，而描述元音时，需要考虑的主要是舌在口腔中的位置。

从理论上讲,一个音段要么是元音，要么就是辅音。但像 yet 中的 [j] 和 wet 中的 [w] 实际上是完完全全的元音，只是它们出现在辅音的位置上。我们一般用术语“半元音 (semi-vowel)”来描述这些既非元音又非辅音，而是处于中间状态的语音。

We can also divide phonation into voiceless (清音：声带不颤动) and voiced （浊音：声带颤动）。对应的我们有“清辅音”和“浊辅音”。对于元音，所有的都是“浊元音”。我们还可以根据是否“送气（aspirated)”来区分发音，如在 peak 中 /p/ 送气，用 [pͪ] 表示，在 speak 中 /p/ 不送气，用 [p] 表示。peak 的严式注音为 [pͪik], 宽式注音为 [pik], 严式注音包括了对送气现象的表述。

长短元音符号

英语词 fit 和 feet 中间的元音是用长度符号 [ː] 来区分的（即 fit [fit], feet [fiːt]），表示为 [iː, i; uː, u; ɔː, ɔ; ɜː, ɜ]. 现在有了独立的短音符号 [i, ɪ; u, ʊ; ɔ, ɒ; ɜ, ə]，每一对的前者是短音，后者是长音。有字典为了便于区别，同时标注长度符号和新短音符号 [iː, ɪ; uː, ʊ; ɔː, ɒ; ɜː, ə].

Syllable 音节

元音和辅音都是指的单个发音，称为“音段（segment）”，当音段组合成音节（syllable）。通常把音节分成两个部分：韵基（rime）和节首（onset）, 韵基中的元音是节核（nucleus）, 后面的辅音称作节尾（coda）。

所有音节必须有节核,但不是所有的音节都有节首和节尾。没有节尾的音节叫做“开音节（open syllable）”, 有节尾的音节叫做“闭音节（closed syllable）”。

在词的层面上，一个词在单独提及的时候（即单词本身没有在句子里）称为“引用形式（citation form）”，这时至少有一个音节是重读的。也就是说，如果某个词是单音节词，那么这个词的引用形式必须是重读的，如不定冠词 a，单独说这个词的时候应该说 [eɪ]，而不是 [ə]. 虚词一般都有重读和轻读两种形式,在短语和句子的层面上一般采取轻读。

在多音节的词中,至少有一个音节是重读音节。在比较长的词中（一般指 4 个音节以上）,还会有两个重读音节。

Bitcoin

2018-05-22T00:00:00+00:00

Bitcoin Core Installation
Mode
Configuration
Daemon/server
RPC
Testing

This post only focuses only on Bitcoin daemon deployment on Linux. To get Bitcoin outline, concepts, principles, read NextCloud Note.

Bitcoin Core Installation

I choose to install daemon instead of QT GUI.

root@tux ~ / emerge -avt net-p2p/bitcoind net-p2p/bitcoin-cli
root@tux ~ / less /etc/{bitcoin/bitcoin.conf,init.d/bitcoind,conf.d/bitcoind}

Mode

Full node (mainnet)

A full node is a program that fully validates transactions and blocks. Running in full node comes with certain costs (i.e. harward requirements). To store the full blockchain, you need around 145 gigabytes of disk space, 2 gigabytes of RAM, broadband Internet etc.

Mostly, you should keep your computer running at least 6 hours continuously without interruption.
Prune mode (mainnet)

Prune mode save diskpace by parameter --prune=<n> which prunes (delete) old blocks. By default, 0 disables pruning blocks; 1 allows manual pruning via RPC; >550 = automatically prune block files to stay under the specified target size in MiB.
Testnet

Unlike mainnet, the testnet is an alternative blockchain, only to be used to testing. Coins in testnet are separate and distinct from actual bitcoins, and are never supposed to have any value.

This allows application developers or bitcoin testers to experiment, without having to use real bitcoins or worrying about breaking the main bitcoin chain. What is more, a raw recruit can play around with testnet and switch to real Bitcoin blockchain afterwards.

The size of testnet blockchain is not trivial. As of January 2018, it is was 14GB, containing data for about 6 years worth of testnet activity.

We can combine Testnet and Prune together.
Regression Test Mode (Regtest)

To further simplify the testing process, we can use local Regtest mode, where interaction with random peers and blocks is unnecessary or unwanted. It lets you instantly create a brand-new private block chain with the same basic rules as testnet — but one major difference: you choose when to create new blocks, so you have complete control over the environment. Unlike previous modes, block generation is pretty fast!

Configuration

Default ports as of 0.16.0:

	mainnet	testnet	regtest
P2p	8333	18333	18444
RPC	8332	18332	18443

Here is a configration for Regtest mode:

# /etc/bitcoin/bitcoin.conf

#
# Mode
#

regtest=1
#testnet=1

#
# Network
#

#proxy=127.0.0.1:1080

#listen=1

#https://github.com/bitcoin/bitcoin/blob/master/doc/tor.md
#onion=127.0.0.1:9050
listenonion=0

bind=127.0.0.1
bind=192.168.56.100
#port=18444

addnode=192.168.56.101
#addnode=192.168.56.101:18444

#connect=192.168.56.101
#connect=192.168.56.101:18444

#
# RPC
#

server=1

#https://github.com/bitcoin/bitcoin/tree/master/share/rpcauth
rpcauth=myuser:ff164b6f16a012debdadc24e7f7da$eb5427acefdcf06e0cf4f070568297e93a29bd0e16f28bac6a6c9dce5a0d08fa
#rpcuser=myuser
#rpcpassword=AbfGbK7qUbpLh3lLtgMKD3xKFUupXgC3BSsf98X2jrI=

rpcallowip=127.0.0.1
rpcallowip=::1
rpcallowip=192.168.0.0/24
rpcbind=127.0.0.1
#rpcport=18443

#
# Miscellanous
#

#prune=550
txindex=1

prune and txindex are incompatible. Turn on prune in mainnet to save disk space.
rpcauth is recommended over rpcuser/rpcpassword. It generates random password if we do not offer one.
```
user@tux ~ # /path/to/rpcauth.py <username> [<password>]
```
rpcauth is only used by server side. Please write down generated rpcpassword pair as it will be used by client side.
Difference between addnode and connect.

addnode adds a node to the list of nodes to connect while connect only connects to this very node, ignoring other nodes.
connect= and proxy= will disable listen=1, rejecting incoming connections.
We can connect the Bitcoin network through Tor. A bit further, we set local .onion address so that our own Bitcoin node is reachable through Tor.

Daemon/server

root@tux ~ / rc-service bitcoind start
root@tux ~ / ss -npelt
root@tux ~ / ll /var/lib/bitcoin/.bitcoin/
root@tux ~ / gpasswd -a username bitcoin

Bitcoin is running in daemon mode in the background.
We should add username to bitcoin group, otherwise bitcoin_cli cannot run with username account.
From /etc/init.d/bitcoind we find critical parameters are -conf and -datadir.

Usually, we put -conf under -datadir.
Bitcoin-qt disables -server by default and we can access to RPC through Help -> Debug window -> Console.

RPC

Unlike the server side, client side bitcoin_cli does not recognize rpcauth. To authenticate itself, bitcoin_cli read rpcpassword from configuration file or command line option.

# enable bitcoin group without re-login.
user@tux ~ # newgrp - bitcoin

We could tell it to share configuration with server:

user@tux ~ # bitcoin-cli -conf=/etc/bitcoin/bitcoin.conf [-rpcuser=user -rpcpassword=password] help
# generate bitcoin.conf symlink
user@tux ~ # mkdir -p ~/.bitcoin; ln -sv /etc/bitcoin/bitcoin.conf ~/.bitcoin/bitcoin.conf
user@tux ~ # bitcoin-cli [-rpcuser=user -rpcpassword=password] help

Alternatively, we set separate client configration file without symlink. You are recommended to use this method:

# ~/.bitcoin/bitcoin.conf

regtest=1
rpcuser=myuser
rpcpassword=mypassword
#rpcport=18443

For client options, check man page.

Testing

In this section, I will discuss more Bitcoin details with Regtest mode.

Info

user@tux ~ # bitcoin-cli help
user@tux ~ # bitcoin-cli help getblockchaininfo
user@tux ~ # bitcoin-cli getblockchaininfo | getwalletinfo | getnetworkinfo | getpeerinfo
user@tux ~ # bitcoin-cli logging "[\"all\"]"

Block

user@tux ~ # bitcoin-cli getblockhash 50
user@tux ~ # bitcoin-cli getblock <returned hash> [0 | 1 | 2]

To examine a block information, we should first get its hash value. To get the block hash, we use height vlue which means distance from the genesis block.

Mining

Mining is just generating in Regtest mode.

user@tux ~ # bitcoin-cli generate 101
user@tux ~ # bitcoin-cli getbalance

A Regtest blockchain with 101 blocks is created, which takes less than a second. Each confirmed block reward is 50 bitcoins (no transaction fees). However, a block must have 100 confirmations before that reward can be spent, so we generate 101 blocks to get access to the coinbase transaction from block 1.

user@tux ~ # bitcoin-cli generate 1
user@tux ~ # bitcoin-cli getbalance

Peer

Thgough we use Regest locally, connecting to peer is necessary when you want to test transactions. To do this, we just file run another Regtest instance locally with different port, conf and datadir.

If you'd like, the extra Regtest instance could be moved into virtual machine as long as Hostonly network is correctly configured. Please be noted that, launch virtual machine before host Regtest instance which otherwise does not know about Hostonly interface IP.

In order that two nodes notice each other, we should use parameter addnode/connect or command line option thereof.

user@tux ~ # bitcoin-cli getaddednodeinfo | getpeerinfo

Transaction

Once we have established connection to peers, we can send bitcoins around. Fristly, we should generate a new address to receive payments.

user@tux ~ # bitcoin-cli getnewaddress
user@tux ~ # bitcoin-cli listreceivedbyaddress 1 true

Send 10 bitcoins to the address using sendtoaddress RPC command which automatically selects an unspent transaction output (UTXO) from which to spend the satoshis. To check UTXO, we use listunspent.

user@tux ~ # bitcoin-cli getbalance
user@tux ~ # bitcoin-cli listunspent
user@tux ~ # bitcoin-cli sendtoaddress $NEW_ADDRESS 10.00 "first spending" "send to myself" true

Examine transaction

user@tux ~ # bitcoin-cli gettransaction <transaction hash returned> true
user@tux ~ # bitcoin-cli getrawtransaction <transaction hash returned> true
user@tux ~ # bitcoin-cli decoderawtransaction <hex string returned>
user@tux ~ # bitcoin-cli listunspent 0
user@tux ~ # bitcoin-cli listreceivedbyaddress 1 true
user@tux ~ # bitcoin-cli listtransactions "" 2

Confirm a transaction

Up to now, the transaction is not yet confirmed! We should mine 100 blocks to confirm it.

user@tux ~ # bitcoin-cli generate 100

If the input is from coinbase transaction, you can't spent it until it get 100 confirmations, avoiding blockchain fork issue. Hence, 1 confirmation is not enough!

Similary, we can send bitcoins the other Regtest node.

Stop daemon

We can stop the daemon:

user@tux ~ # bitcoin-cli stop

However, we'd better use OpenRC or Systemd service.

Docker Newbie

2018-04-21T00:00:00+00:00

Architecture
1. Docker Engine
  1. Docker Desktop
Image Container and Registry
Installation
Start daemon
1. docker.sock
Docker Context
CLI Sample
Pull Image
Launch Containers
Runtime metrics
Data Share
1. ENV Variables
2. SELinux
Nginx Container Sample
1. Get into Container
Networking Drivers
exec and shell
Custom Image
1. Docker Commit
2. Docker Build Dockerfile
Share Images
Docker Compose
Troubleshooting

Architecture

In general, Docker works in client-server mode illustrated in the figure below.

Server-side Docker Engine, also called Docker Daemon, is the dockerd that manages the containers, images, volumes, networks, etc.

The daemon serves requests by RESTful API over local Unix socket or over remote network interface.
Client-sdie Docker CLI are docker, docker-compose, etc.

CLI options and arguments are consolidated and transformed to REST API. We can use cURL to manage Docker objects.

The term "Docker" most of the time means the overall architecture.

Docker Engine

Most of the time, Docker Daemon is merely the daemon part. But occasionally, it may refer to the collection of Docker daemon, the Docker REST API and the Docker client. In this section, we focus on the daemon.

Docker Engine relies on Linux kernel under the hood, like the namespace (containers isolation), cgroup (resources management), libraries, etc. So, by "container", we actually mean "Linux container". But for convenience, we just call it "container".

At the very beginning, Docker Engine is an integrated piece of daemon including all capabilities to manage containers, images, volumes, networks, etc.

+--------------+            +-----------------+        +----------+   
|              | REST API   |  Docker Engine  |        |+----------+  
|  docker CLI  +----------->|                 +------->+|+----------+ 
|              |            |     dockerd     |         +|+----------+
+--------------+            +-----------------+          +|container+|
                                                          +----------+

Later on (2017), in order to expand its adoption and add neutrality and modularity, the core capabilities are donated to CNCF as a seperate daemon containerd (i.e. docker-containerd). The Docker Engine only focuses on developer experience like serving login, build, inspect, log, etc.

                                                       +--------------+
+--------------+            +-----------------+        |              |       +----------+
|              | REST API   |  Docker Engine  |  gRPC  | containerd   |       |+----------+
|  docker CLI  +----------->|                 +------->|     +------+ +------>+|+----------+ 
|              |            |     dockerd     |        |     | runc | |        +|+----------+
+--------------+            +-----------------+        |     +------+ |         +|container+|
                                                       +--------------+          +----------+

In the meantime, the Open Container Initiative (OCI) standardizes the containerd. According to the standard, the responsibility of creating containers was removed from containerd in faviour of runtime (e.g. runc). The interaction with Linux namespace and Linux cgroup has shifted from containerd to runtime. We call containerd the high-level runtime, managing container lifecycle, images, volumes, network, etc.

                                                       +--------------+
+--------------+            +-----------------+        |              |       +----------+
|              | REST API   |  Docker Engine  |  gRPC  | containerd   |       |+----------+
|  docker CLI  +----------->|                 +------->|     +------+ +------>+|+----------+ 
|              |            |     dockerd     |        |     | runc | |        +|+----------+
+--------------+            +-----------------+        |     +------+ |         +|container+|
                                                       +--------------+          +----------+

With OCI, everyone can build his own containerization system. Nowadays, there exist multiple OCI runtimes. In order to support those runtimes, Docker inserts a new component between containerd and runtime, namely the containerd-shim (e.g. docker-containerd-shim). The containerd-shim invokes the runtime (e.g. runc) to create the container. Once the container is created, the runtime exits and the lifecycle management is handed over to containerd and container-shim.

Some of the runtimes are fully compatibile with docker-containerd-shim like the youki, while others are not like the Wasmtime. Runtimes compatibile with docker-containerd-shim can be a drop-in replacement for runc. Runtimes incompatibile with docker-containerd-shim must implements their own shim according to shim API.

                                                                                                   container lifecycle
                                                                                   +--------------------------------------------+
                                                                                   |                                            v
                                                                                   |                   +---------------+   +---------+
                                                                                   |                   | runtime youki +-->|container|
                                                                                   |                   |    (exit)     |   +---------+
                                                                                   |                   +---------------+
                                                                                   v                      ^
+--------------+            +-----------------+        +--------------+    +-----------------+            |
|              |  REST API  |  Docker Engine  |  gRPC  |  High-level  |    |                 |   image    |                     .
|  docker CLI  +----------->|                 +------->|   runtime    +--->| containerd-shim +------------+                     .
|              |            |     dockerd     |        |  containerd  |    |                 |   bundle   |                     .
+--------------+            +-----------------+        +--------------+    +-----------------+            |
                                                                                   ^                      v
                                                                                   |                   +--------------+
                                                                                   |                   | runtime runc |    +---------+
                                                                                   |                   |    (exit)    +--->|container|
                                                                                   |                   +--------------+    +---------+
                                                                                   |                                            ^
                                                                                   +--------------------------------------------+
                                                                                                   container lifecycle

More and more middle layers are added to the architecture, making it too complicated. podman, on the other hand, is much more simpler as follows. podman talks directly to the runtime, without dockerd, conainerd or containerd-shim.

podman CLI -> runtime runc -> containers

We can actually create and run containers directly with runtime runc. It is out of the scope of this post, please read official runc page.

You are strongly recommended to read https://stackoverflow.com/q/46649592/2336707. The following output is a demonstration in my dev environment. Two containers were created and they are the child processes of the containerd-shim-runc-v2.

ubuntu@ip-172-31-9-194:~/misc$ docker run --name httpbin -P -d kennethreitz/httpbin
4bd1077052750a2a7552e4347bcbba483d47f2555b89874606c0b04b93f7c2dc

ubuntu@ip-172-31-9-194:~/misc$ docker run --rm -itd ubuntu
18be920d5a998ceee5f438ae43c7d1171fec6d34d4742c66611ff7a63f9d6a68

ubuntu@ip-172-31-9-194:~/misc$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                                       NAMES
a598c760b207   ubuntu                 "/bin/bash"              29 minutes ago   Up 29 minutes                                               ubuntu
4bd107705275   kennethreitz/httpbin   "gunicorn -b 0.0.0.0…"   48 minutes ago   Up 48 minutes   0.0.0.0:32768->80/tcp, [::]:32768->80/tcp   httpbin

ubuntu@ip-172-31-9-194:~/misc$ ps  -eF --forest
# dockerd
root         530       1  0 570701 82248  3 Dec23 ?        00:00:29 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root       15033     530  0 436282 3968   0 Dec27 ?        00:00:00  \_ /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 32768 -container-ip 172.17.0.2 -container-port 80
root       15040     530  0 399416 3840   0 Dec27 ?        00:00:00  \_ /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 32768 -container-ip 172.17.0.2 -container-port 80
# containerd
root         357       1  0 468968 48524  3 Dec23 ?        00:02:22 /usr/bin/containerd
# container httpbin
root       15063       1  0 309542 13748  0 Dec27 ?        00:00:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 4bd1077052750a2a7552e4347bcbba483d47f2555b89874606c0b04b93f7c2dc -address /run/containerd/containerd.sock
root       15083   15063  0 21495 24480   0 Dec27 ?        00:00:05  \_ /usr/bin/python3 /usr/local/bin/gunicorn -b 0.0.0.0:80 httpbin:app -k gevent
root       15109   15083  0 32493 33912   3 Dec27 ?        00:00:08      \_ /usr/bin/python3 /usr/local/bin/gunicorn -b 0.0.0.0:80 httpbin:app -k gevent
# container ubuntu
root       15363       1  0 309542 13780  3 Dec27 ?        00:00:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id a598c760b2071f4951d35d255d3669ce3e9877534b36efcd21e2dfcb8727b136 -address /run/containerd/containerd.sock
root       15383   15363  0  1147  3840   2 Dec27 pts/0    00:00:00  \_ /bin/bash

Docker Desktop

In order to run containers on Window and macOS, a Linux virtual machine is required to host dockerd, containerd, containerd-shim, runtime, etc. The Docker CLI remains on the host.

As such, Docker provides the Docker Desktop. Docker Desktop is an all-in-one (including GUI) software and gives us a uniform expiernce accross Linux, Window and macOS.

Especially, Docker Desktop creates the Linux virtual machine for us, so that we are not bothered on this. On Window and macOS, Docker Desktop utilizes native virtualization framework (Hyper-V and WSL 2 of Windows; Hypervisor of macOS) to boost performance. On Linux system, Docker Desktop is not a must. However, if we choose it, the Virtual Machine is still created.

Image Container and Registry

Dockers comprises image, container and registry.

Image is static, readonly and a minimal root filesystem bundle. There are many highly qualified base iamge from official registry like nginx, redis, php, python, ruby etc. Especially, we have ubuntu, centos, etc. that are just OS minimal bare bones (like Gentoo stage tarball).

An image consists of multiple incremental layers that are defined by a Dockerfile. Correspondingly, we call the image storage layer storage based on Union Filesytem (FS). Recall that booting USB stick also uses Union FS. The most adopted Union FS are overlay2.
Container is a set of processes with added isolation (Linux namespace) and resource management (Linux cgroup).

It is created on top of a base image with an additional layer storing running but volatile data. We can think of image and container as class and object in Object-oriented programming.
Registry is store where users publicize, share and download repostitory. The default registry is docker.io or registry-1.docker.io with a frontend website Docker Hub.

Repository, on the other hand, actually refers to name of an image (e.g. ubuntu). We can specify version of a repository by a tag (label) like ubuntu:16.04 (colon separator). The default tag is latest.

The naming of an image follows the format as follows.

registry.fqdn[:port]/[namespace/]repository[:tag | @<image-ID>]

Default registry can be ommitted. Others like quay.io must be provided.
The namespace part means a registered account in the registry. It can be an individual user name or an organization name like "kong".
repository is the default name of an image like "ubuntu" and "kong-gateway".
tag is a string. Default to latest.
image id comprises a SHA256 digest like @sha256:abea36737d98dea6109e1e292e0b9e443f59864b.

Specifying an image by tag can always gets the latest updates. For example, every time a patch release is released (e.g. kong/kong:3.4.5), kong/kong:3.4 refers to that patch version 3.4.5 with a different digest. On the other hand, an image specified by digest always is pinned to that specific image. See Pin actions to a full length commit SHA for security concerns.

Installation

We only install Docker CE version. For Windows and MacOS, Docker Desktop includes both Docker and Docker Compose. On Linux, it is highly recommended to install Docker and Docker Compose by official repo, or install by downloaded packages.

On Amazon Linux 2, we can install Docker by package manager, and then install Docker Compose by downloading binary file manually.

Install Docker by package manager:

~ $ sudo yum update -y

# CentOS
~ $ sudo yum install docker

# Amazon Linux 2 - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-container-image.html
~ $ sudo amazon-linux-extras install docker

~ $ docker version

In order to run docker as a normal user, add the account to docker group:

~ $ sudo usermod -aG docker <username>

~ $ reboot

Install Docker Compose manually:

~ $ sudo mkdir -p /usr/local/lib/docker/cli-plugins
~ $ sudo curl -sSL https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/lib/docker/cli-plugins/docker-compose

~ $ sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
~ $ docker compose version

~ $ PATH="/usr/local/lib/docker/cli-plugins:$PATH"
~ $ docker-compose version

Start daemon

Start Docker Engine.

~ $ sudo systemctl enable docker
~ $ systemctl status docker
~ $ sudo systemctl start docker

~ $ docker info
~ $ docker ps
~ $ docker compose ls

All data is located in the /var/lib/docker directory.

ubuntu@ip-172-31-9-194:~/misc$ sudo -E PATH=$PATH ls /var/lib/docker/
buildkit  containers  engine-id  image  network  overlay2  plugins  runtimes  swarm  tmp  volumes

docker.sock

The daemon listens for RESTful API requests via either an Unix socket file at /var/run/docker.sock or an IP address. By default only the Unis socket file is enabled. See Docker Context for enabling the IP socket.

We can communicate with the daemon directly with cURL according to the Docker API spec.

~ $ curl --unix-socket /var/run/docker.sock --no-buffer http://localhost/events
~ $ curl --unix-socket /var/run/docker.sock http://localhost/version
~ $ curl --unix-socket /var/run/docker.sock http://localhost/images/json | jq
~ $ curl --unix-socket /var/run/docker.sock http://localhost/containers/json | jq

We can also create containers inside another container by mounting the socket file, as long as docker CLI is available.

# on host use the special 'docker' image
~ $ docker run --name docker-sock --rm -it -v /var/run/docker.sock:/var/run/docker.sock docker sh

# within docker-sock
/ # docker run --name docker-in-docker --rm -it ubuntu bash

# within docker-in-docker
root@978d8704d548:/#

On the host terminal, we can retrieve the nested containers

~ $ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS           NAMES
978d8704d548   ubuntu    "bash"                   5 seconds ago    Up 4 seconds                    docker-in-docker
a3b93da63e90   docker    "dockerd-entrypoint.…"   17 seconds ago   Up 16 seconds   2375-2376/tcp   docker-sock

We can also get all containers from within docker-sock.

/ # docker ps -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED         STATUS                       PORTS                NAMES
a3b93da63e90   docker                   "dockerd-entrypoint.…"   4 minutes ago   Up 4 minutes                 2375-2376/tcp        docker-sock
0dadcbdeb804   862614378b4c             "docker-entrypoint.s…"   13 months ago   Exited (0) 13 months ago                          eloquent_jones
f3f0faf83ac5   docker/getting-started   "/docker-entrypoint.…"   14 months ago   Exited (255) 13 months ago   0.0.0.0:80->80/tcp   romantic_colden

However, mounting docker.sock would make your host vulnerable to attack as Docker daemon within the container is ran as root.

Docker Context

Recall that Docker works in client-server mode, where the client connects to the server via RESTful API. As a developer, it is not unusual that we have different environment of different purposes such as development environment, staging environment, buildx environment, production environment, etc.

For a single docker CLI to communicate with different Docker Engines, we have Docker Context. A context is a profile recording the information of a Docker Engine like the IP address. To swtich between Docker Engines, just use docker context CLI.

By default, only the local Unix socket is enabled. The example below enables IP socket listening and but only binds to localhost for demo purpose. To expose the Docker Engine on public Internet, please follow the guide at security concern.

~ $ sudo systemctl editedit docker.service
# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375 --containerd=/run/containerd/containerd.sock

~ $ sudo systemctl daemon-reload

~ $ sudo systemctl restart docker

~ $ ps -eFH | grep [d]ockerd
root       24927       1  0 552268 77412  3 07:29 ?        00:00:00   /usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375 --containerd=/run/containerd/containerd.sock

~ $ curl -sS http://localhost:2375/containers/json | jq -r '.[].Names'
[
  "/httpbin"
]

CLI Sample

~ $ fgrep -qa docker /proc/1/cgroup; echo $?                    # check if it is within a docker or on the host

~ $ docker info                                                 # display the outline of docker environment
~ $ docker image/container ls [-a]                              # list images/containers
~ $ docker [image] history                                      # show layers of an image
~ $ docker inspect [ name | ID ]                                # display low-level details on any docker objects
~ $ docker logs <container>

Here is the full list of docker CLI: Docker CLI.

Pull Image

It is highly recommended to pull the docker/getting-started image, run and visit http://localhost.

~ $ docker search -f is-official=true ubuntu                    # search only official image

~ $ docker pull ubuntu:16.04                                    # specify a tag
~ $ docker pull ubuntu@sha256:<hash>                            # specify an image ID

~ $ docker images

docker search search docker images from registries defined in /etc/container/registries.conf.

Unfortunately, it does not suport tags or IDs. Instead, go to the registry website or check third party tool DevOps-Python-tools.
We can offer docker pull a tag (i.e. 'ubuntu:latest') or an image ID.
When pulling an image without its digest, we can update the image with the same pull command again.

On the contrary, with digest, the image is fixed and pinned to that exact version. This makes sure you are interacting with the exact image. However, upcoming security fixes are also missed.

To get image digest, we either go to the official registry, use images --digests, or even inspect command.
Any any time, Ctrl-C terminates the pull process.
Docker support proxy configuration when feteching the images.

Launch Containers

docker run equals docker create plus docker start. Usually, we just docker run.

Syntax.

docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Example:

~ $ docker run --name centos-5.8 \
-d \
-it \
--rm \
--mount type=bind,src=/home/jim/workspace/,dst=/home/jim/workspace/,ro \
-w /home/jim/workspace/ \
--net host
-u $(id -u):$(id -g) \
7a126f3dba08 \
bash

#
root@docker ~ # cat /etc/os-release
root@docker ~ # exit 13
root@docker ~ # echo $?

When we run an image, a container is created with an extra layer of writable filesystem.
To be compatible with AMD64/ARM64, we can add the --platform linux/x86_64 or --platform linux/arm64. Check multi-platform-docker-build.

This also applies to docker build.
By default, the root process of a container (PID 1), namely the CMD/ENTRYPOINTWITH is started in the forground mode. The host terminal is attached to the process's STDOUT/STDERR, but not STDIN. So we can see the output (error message included) of the root process as follows:
```
~ $ docker run -t --rm ubuntu ls /
bin   dev  home  media  opt   root  sbin  sys  usr
boot  etc  lib   mnt    proc  run   srv   tmp  var
   
~ $ docker run --rm ubuntu ls /
bin
boot
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
```
We can use multiple -a options to control the attachment combinations of STDIN/STDOUT/STDERR. To following example, we can even input to the root process as long as its STDIN is open. Refer to confused-about-docker-t-option-to-allocate-a-pseudo-tty and Attach to STDIN/STDOUT/STDERR.
```
~ $ docker run -a stdin -a stdout ...
```
If we want to start the process in background mode, namely the detach mode, then add the -d option. Containers runs in this mode will print the container ID and release host terminal immediately. So we cannot input to the root process or see the output or error message: STDIN/STDOUT/STDERR detached. If the root process exits, then the container exits as well. So we cannot do like this:
```
~ $ docker run -d -p 80:80 my_image service nginx start
```
As the root process service exits immediately after nginx is started. The next example will keep the container running as the tail command persists:
```
~ $ docker run -d ubuntu bash -c "tail -f /dev/null"
```
-t allocates a pseudo-TTY connected for the root process, especially useful when the process is an interactive Shell. The -i option forces the process's STDIN to be open and runs the container interactively, so we can input some data to the process directly, which works even when -d is present. Here is an example:
```
echo test | docker run -i ubuntu cat -
test
```
The two options are usually used together for Shell: -t creates a pseudo-TTY, while -i makes the STDIN of the pseudo-TTY open for data input.
```
# STDIN not open by default, so keeps waiting for input
~ $ dockder run -t --rm ubuntu bash
   
# can input/output, but no standalone STDOUT, reuse that of host terminal
~ $ docker run -i --rum ubunty bash
   
# standalone stdin/stdout/stderr, also interactive for input
~ $ docker run -it --rum ubunty bash
```
--rm automatically remove the container when it exits.
Check Data Share for --mount.
-w lets the root process running inside the given directory that is created on demand.
--net, --network connects the container to a network. By default, it is bridge. Details are discussed in later sections.
-u, --user runs the root process as a non-root user. Attention that, the username is that within the container. So the image creator should create that name in Dockerfile.
bash overrides the CMD/ENTRYPOINT instructions of the image.

Here is a note about the different options:

Please also follow this post Cannot pipe to docker run with stdin attached.

Runtime metrics

Please see https://docs.docker.com/engine/containers/runmetrics/.

Docker containers can read from or write to pathnames, either on host or on memory filesystem - to share data. There are three storage types:

Volume, Data Volume, or Named Volume.

By default, running data of a container is layered on top of the image used to create it. A volume decouples that data from both the host or the container. Just think of a Windows partition or removable disk drive.

Volumes are managed by Docker and persist. Data within can be shared among multiple containers, as well as between the host and a container.

Beofre you restart a docker compose project, please run "docker volume prune -a", otherwise history data might interrupt new containers.
Bind Mount.

Bind-mount a file or directory in the host to a file or directory in the container. The target can be read-only or read-write. For example, bind host /etc/resolv.conf to a container, sharing name servers.

Attention please; to bind-mount a file, please provide the absolute path, otherwise the dest pathname in the container might be a directory! Check How to mount a single file in a volume.
tmpfs Mount.

Needless to say, tmpfs is a memory filesystem that let container stores data in host memory.

We use option --volume , -v and --mount to share data between containers and hosts. --mount is recommended as it support all 3 kinds of data sharing and is more verbose. --volume will be deprecated soon.

If the file or directory on the host does not exist. --volume and --mount behaves differently. --volume would create the pathname as a directory, NOT a file, while --mount would report error.

On the other hand, if the target pathname already exists in the container, both options would obsecure contents over there. This is useful if we'd like to test a new version of code without touching the original copies.

ENV Variables

To pass environment variables to containers, we can:

-e, --env applies only to docker run. This method reveal sensitive values in Shell history. We can firstly export the variable on CLI, then pass --env VAR without the value part.
–env-file (default $PWD/.env) applies both to docker run and docker compose. This fits when there are a lot of variables to pass in.
docker compose can pick up a few compose-specific variables from CLI, so just export it.
CLI variables can also be for substitution in compose file.

SELinux

When bind-mount a file or mount a directory of host, SELinx policy in the container may restrict access to the shared pathname.

Temporarily turn off SELinux policy:

~ $ su -c "setenforce 0"
~ $ docker restart container-ID

Adding a SELinux rule for the shared pathname:

~ $ chcon -Rt svirt_sandbox_file_t /path/to/

Pass argument :z or :Z to --volume option:
```
-v /root/workspace:/root/workspace:z
```
Attention please, --mount does not support this.
Pass --privileged=true to docker run.

However, this method is discouraged as privileged containers bring in security risks. If it is the last resort, first create a privileged container and then create a non-priviledged container inside.

Privilege permissions can have fine-grained control by --cap-add or --cap-drop, which is recommended!

Nginx Container Sample

~ $ docker run --name webserver \
-d \
--net host
--mount type=bind,source=/tmp/logs,target=/var/log/nginx \
-p 8080:80 nginx

~ $ docker container ls
~ $ docker container logs webserver
~ $ docker container stop/kill webserver
~ $ docker container start webserver
~ $ docker container rm webserve          # remove one or more container (even running instances)
~ $ docker container prune                # remove all stopped container

-p maps host port 8080 to container port 80 that is already bound to host Nginx process.
Check the Dockerfile, there is a line telling how Nginx should be started:
```
CMD ["nginx", "-g", "daemon off;"]
```
The --mount type is a Bind Mount directory.
Visit the Nginx container page at http://host-ip:8080.
stop attempts to trigger a graceful shutdown by sending the standard POSIX signal SIGTERM, whereas kill just kills the process by sending SIGKILL signal.

Get into Container

Exec a simple command in container:

~ $ docker exec webserver sh -c 'echo $PATH'
~ $ docker exec webserver ps

Sometimes, we want to exec the command in the background when we do not care about its output or need not any input:

~ $ docker exec -d webserver touch /tmp/test.txt

If we want to interactively and continuously control the container:

~ $ docker exec -it webserver bash

Apart from docker exec, docker attach <container> is also recommended.

This command attaches the host terminal's STDIN, STDOUT and STDERR files to a running container, allowing interactive control or inspect as if the container was running directly in the host's terminal. It will display the output of the ENTRYPOINT/CMD process.

For example, a container can be shutdown by C-c shortcut, sending the SIGINT signal (identical to SIGTERM) to the container by default. Rather, C-p C-q detaches from the container and leave it running in the background again.

If the process is running as PID 1 (like /usr/bin/init), it ignores any signal and will not terminate on SIGINT or SIGTERM unless it is coded to do so.

Networking Drivers

The Docker's networking subsystem is pluggable, using drivers. Docker provides multiple built-in networks, based on which we can define custom networks.

Below is a simple explanation:

Bridge

The default driver if none is given upon run, providing network isolation from the outside. It is to bridge traffic among multiple containers and the host. Check the image below, docker0 is the bridge interface.

If we define a custom bridge network, containers within can communicate with each other by alias or name, otherwise they can only communicate by IP addresses.

Pay attention please; this is different from the bridge mode of VMWare or VirtualBox (real Virtual Machine). VMWare and VirtualBox's bridge is deployed directly on the host's interface and appears to be a real physical device parallel to the host and can be connected to from within LAN directly.
Host

Share the host's networking directly without isolation. However, LAN devices cannot differntiate between containers and the host as there is not individual IP addressed assigned to containers. The host mode is preferred when the service exposes a port publicly like Nginx servers.

To use host network, just add network_mode: host to Dockder compose file, and but must remove the ports mappings as containers share the same network as the host. Alternatively, add --network=host option to docker run.

Unfortunately, host mode does not work on macOS.
Overlay

Overlay connects multiple Docker daemons together, creating a distributed network among multiple Docker daemon hosts. This network sits on top of (overlays) the host-specific networks, allowing containers connected to it.
macvlan

As the name implied, maclan assignes a MAC address to a container, making it be a physical device on the same network as the host - counterpart of VMWare/VirtualBox's 'bridge'.
none

Disable networking.

DNS

Docker Destkop has multiple built-in DNS servers as follows.

When resolving containers within the the same Docker network, the internal DNS within dockerd is utilized. However, resolving hostnames outside of the Docker network, would be forwarded to host via CoreDNS.

Please read How Docker Desktop Networking Works Under the Hood regarding how Docker Destop achieves DNS, HTTP Proxy, TCP/IP stack, Port Forwarding, etc. network features via vpnkit.

host.docker.internal

Occasionally, we want to access to host services from within containers.

If the container is booted with host network, then use localhost or 127.0.0.1.
If the container is booted with bridge network, then use host.docker.internal. Depending on the platform, we might need a bit setup.
1. On macOS, host.docker.internal is intuitively supported.
2. On Linux, we should manually add host.docker.internal:host-gateway to docker run --add-host or to extra_hosts of docker compose. This would add an entry in "/etc/hosts".

The hostname "host.docker.internal" is used only for connection, so please set the correct "Host" header. See example below.

~$ docker exec -it alpine sh

/ # curl -v http://localhost/anything --connect-to localhost:80:host.docker.internal:80
* processing: http://localhost/anything
* Connecting to hostname: host.docker.internal
* Connecting to port: 80
*   Trying 192.168.65.254:80...
* Connected to host.docker.internal (192.168.65.254) port 80
> GET /anything HTTP/1.1
> Host: localhost
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.21.4.2rc1
< Date: Sat, 05 Aug 2023 06:45:34 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: close
<
{"message":"hello, world"}
* Closing connection

Self-define Network

Docker automates network configuration upon container startup. We can customize that on purpose.

~ $ docker network ls
~ $ docker network create -d bridge mynet
~ $ docker run -it --name ubt1604 -v /src/hostdir:/opt/condir:rw --network mynet ubuntu:16.04
~ $ docker network inspect mynet

Use the --net or --network option. To the 'host' networking driver, just pass --net host option to docker run.

SSH Agent Forwarding

In order not to set up a new SSH environment within containers, we can forward SSH agent on host to container.

For Docker Desktop:

~ $ docker run --rm -it -u root \
--mount "type=bind,src=/run/host-services/ssh-auth.sock,target=/run/host-services/ssh-auth.sock,ro" \
-e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" \
--entrypoint /bin/bash kong/kong-gateway:latest

root@3442a4bc63cd:/# ssh-add -l

For Docker engine:

~ $ docker run --rm -it -u root \
--mount "type=bind,src=$SSH_AUTH_SOCK,target=/run/host-services/ssh-auth.sock,ro" \
-e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" \
--entrypoint /bin/bash kong/kong-gateway:latest

root@3442a4bc63cd:/# ssh-add -l

It may report permission issue. We should add the w (write) permission to the socket file. Do it on the host and/or within the container. The example below shows the socket file disallows w by "others" that the "kong" account belongs to. See macOS SSH agent forwarding not working any longer.

# in the container
kong@66cbbf96f403:/$ ssh-add -l
Error connecting to agent: Permission denied

kong@66cbbf96f403:/$ ls -l $SSH_AUTH_SOCK
srwxrwxr-x 1 501 ubuntu 0 Jan  2 13:11 /run/host-services/ssh-auth.sock

~ $ sudo chmod o+w /run/host-services/ssh-auth.sock

Attention that, if you are SSH into Linux VPS from macOS, the SSH agent might be forwarded to the Linux VPS, depending on the SSH config on macOS. This is totally a different topic. Containers in the Linux VPS has no access to the forwarded macOS SSH agent, and we should launch a new one in the Linux VPS.

link

The legacy communication method is --link. Docker copies information (e.g. ENV Variables) from source container to receipt (target) container, and provides network access from receipt container to source container.

Take docker run --name web --link postgres:alias ... for example, the newly created web is receipt container and the existing postgres is source container.

--link is amost deprecated as we can use other features to accomplish the same functionalities. For example, for info copy, use ENV Variables or data share. For network communication, just follow Networking Drivers and Self-define Network.

Here is an example:

## source container

13:47:23 zachary@Zacharys-MacBook-Pro ~$ docker run --name postgres -e HELLO=world -e POSTGRES_HOST_AUTH_METHOD=trust -d postgres:14
25732d58f238b1d3e83fc52ea3c8b91f75290385066e6541d616e68eecd6cfdd

13:47:39 zachary@Zacharys-MacBook-Pro ~$ docker exec -it postgres bash
root@25732d58f238:/# echo $HELLO
world


## receipt/target container

13:48:11 zachary@Zacharys-MacBook-Pro ~$ docker run --name ubuntu -it --link postgres:db ubuntu bash

# src in hosts
root@8e95191dba0e:/# cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.3      db 25732d58f238 postgres
172.17.0.4      8e95191dba0e

# src ENVs
root@8e95191dba0e:/# env | grep DB_ | sort
DB_ENV_GOSU_VERSION=1.14
DB_ENV_HELLO=world
DB_ENV_LANG=en_US.utf8
DB_ENV_PGDATA=/var/lib/postgresql/data
DB_ENV_PG_MAJOR=14
DB_ENV_PG_VERSION=14.4-1.pgdg110+1
DB_ENV_POSTGRES_HOST_AUTH_METHOD=trust
DB_NAME=/ubuntu/db
DB_PORT=tcp://172.17.0.3:5432
DB_PORT_5432_TCP=tcp://172.17.0.3:5432
DB_PORT_5432_TCP_ADDR=172.17.0.3
DB_PORT_5432_TCP_PORT=5432
DB_PORT_5432_TCP_PROTO=tcp

# src network
root@8e95191dba0e:/# ping db
PING db (172.17.0.3) 56(84) bytes of data.
64 bytes from db (172.17.0.3): icmp_seq=1 ttl=64 time=0.210 ms
64 bytes from db (172.17.0.3): icmp_seq=2 ttl=64 time=0.452 ms
64 bytes from db (172.17.0.3): icmp_seq=3 ttl=64 time=0.136 ms

Attention please; --link is one-way link only. Info is transferred from source containers to receipt containers but source containers know nothing about receipt containers. To achieve bi-directional communication, please use network.

exec and shell

Usually, in the end of image, we have three kinds of instruction:

RUN. Creates a new layer of image. Usually used to install package. It's recommended to install multiple packages in a single RUN instruction so that we have less image layers.
CMD. Set the default command to run when run a container.
ENTRYPOINT. run a container as a command, so that we can provide extra arguments when run.

Refer to RUN, ENTRYPOINT and CMD for more details.

Each instruction has two kinds of running forms:

shell form.
```
<instruction> cmd arg1 arg2 ...
```
By default, /bin/sh will be used to run the cmd as:
```
/bin/sh -c 'cmd arg1 arg2 ...'
```
It is the preferred form of the RUN instruction to install package in the image and exit.
exec form.

The cmd is ran directly without any Shell involvement, which is the preferred form of CMD and ENTRYPOINT as we usually launch a daemon background within the container. No need to maintain a Shell process.
```
<instruction> ["cmd", "arg1", "arg2", ...]
```

If we want to run the cmd with Bash, we can use the exec form but explicitly invoke /bin/bash.

<instruction> ["/bin/bash", "-c", "cmd", "arg1", "arg2", ...]

Refer to exec form or sh form for more details.

When there are multiple CMD or ENTRYPOINT instructions inherited from different image layers, only that of the topmost layer is respected! We can use docker container inspect to show the instructions and their forms. For example, nginx image has CMD ["nginx", "-g", "daemon off;"].

We can pass custom commands and arguments when invoking docker run, which will override the CMD instruction and arguments thereof. If there exists the ENTRYPOINT instruction in exec form, then custom arguments would be appended to the ENTRYPOINT cmd. By default, ENTRYPOINT exec form will take extra arguments from CMD instruction in shell form. Custom arguments when docker run will override those in the CMD instruction. ENTRYPOINT in shell form would ignore custom arguments from CMD or docker run.

We can override ENTRYPOINT and/or CMD as follows.

# docker run --entrypoint /path/to/cmd <image> -a arg1 -b arg2 arg3

~ $ docker run --entrypoint /bin/bash -it nginx

~ $ docker run --rm --entrypoint /bin/bash kong/kong-gateway:latest -c "kong version"
Kong Enterprise 3.6.1.0

# docker run --entrypoint '' <image> /path/to/cmd -a arg1 -b arg2 arg3

~ $ docker run --rm --entrypoint '' -it kong/kong-gateway:latest /bin/bash -c "kong version"
Kong Enterprise 3.6.1.0

Here is an illustration between CMD and ENTRYPOINT:

Refer to Dockerfile reference for more details.

Custom Image

Docker Commit

~ $ docker exec -it webserver bash
#
root@docker ~ # echo '<h1>Hello, Docker!</h1>' > /usr/share/nginx/html/index.html
root@docker ~ # exit
#
~ $ docker diff webserver
~ $ docker container commit -a 'jim' -m 'change front page' webserver nginx:v2
~ $ docker image ls nginx
~ $ docker history nginx:v2

We modified container layer storage. Use diff to check details.
Visit the Nginx container page again.
commit records modification as a new layer and create a new image.

Avoid commit command as miscellaneous operations (garbage) are recorded either. As discussed in "Data Share" section, we can make use of Volume, Bind Mount, or tmpfs, or resort to 'Build Image by Dockerfile' section below.
To verify the new image:
```
~ $ docker run --name web2 -d -p 8081:80 --rm nginx:v2
```
The --rm tells to remove the container upon exit.
We can inspect the target image, and will find "ContainerConfig" and "Config". They are almost identical.

"ContainerConfig" is the config of current container from within this image is committed, while the "Config" is the exact configuration of the image. Pay attention to the Cmd parts. If we build by Dockerfile, then they looks different. The ContainerConfig this is the temporary container spawned to create the image. Check what-is-different-of-config-and-containerconfig-of-docker-inspect.

Docker Build Dockerfile

From commit example above, we can create new image layer but many negligible commands like ls, pwd, etc. are recourded as well.

Similar to Makefile, Docker uses Dockerfile to define image with specified instructions like FROM, COPY, RUN etc. Each instruction creates a new intermediate layer and a new intermediate image. In order to minimize number of layers and images, we'd better merge instructions as much as possible.

In this section, we use Dockerfile to create image nginx:v2.

~ $ mkdir mynginx
~ $ cd mynginx
~ $ vim Dockerfile
#
FROM nginx
RUN echo '<h1>Hello, Docker!</h1>' > /usr/share/nginx/html/index.html

Just two lines! FROM imports the base image on which we will create the new layer. If we do not want any base image, use the special null image scratch.

Now we build the image. Please refer to Docker Build Overview to check the difference between docker buildx and docker build. To put it simple, docker build is wrapper of docker buildx with default arguments.

~ $ BUILDKIT_PROGRESS=plain docker buildx build --no-cache --load -t nginx:v3 -t nginx -f Dockerfile .
#
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM nginx
 ---> b175e7467d66
Step 2/2 : RUN echo '<h1>Hello, Docker!</h1>' > /usr/share/nginx/html/index.html
 ---> Running in d5baea5c6341
Removing intermediate container d5baea5c6341
 ---> 18cc3a3480f0
Successfully built 18cc3a3480f0
Successfully tagged nginx:v3

The -t option of docker build actually refers to name of the target image. We can leave the tag part to use the default latest. We can also supply multiple tags. From example above, the second tag is the default latest.

If we specify a name that exists already, we detach that name from an existing image and associate it with the new image. The old image still exist and can be checked the "RepoTags" field of docker inspect <image-id>.
During the building process, both an intermediate container (d5baea5c6341) and image (18cc3a3480f0) is created for 'RUN' instruction.

The intermediate container defines a new layer which is then committed to create a new image. Afterwards, the intermediate container is removed, but the intermediate image is kept.
The trailing dot means the current directory is the building context directory. It is also the default location of the Dockerfile. Sometimes, we exclude some context files from the durectory by the .dockerignore file, as below:
```
~ $ echo ".git" >> .dockerignore
```
Docker sends all files within context directory to remote Docker engine (daemon). Image can be built without a context directory if we don't have any supplementary files.
```
~ $ docker build -t nginx:v3 - < /path/to/Dockerfile
```
The hypen character cannot be omitted!
If there exist multiple CMD/ENTRYPOINT instructions from different layers, only that of the topmost layer will be executed upon container start. All the rest CMD/ENTRYPOINT are overriden.

After the building, we can run nginx:v3 image:

~ $ docker run --name web3 -d -p 8081:80 --rm nginx:v3

Apart from builing a new docker image for the web server, we can utilize 'Data Share' to attach a Volume or Bind Mount to the base docker image. Build the web server within the attached storage instead.
Sometimes, we may want to remove the cache of build, which can be accomplished by docker builder prune -a.
Here is another Dockerfile instance:
```
FROM centos:latest
RUN [/bin/bash, -c, 'groupadd -g 1000 username ; useradd -ms /bin/bash -u 1000 -g 1000 username; echo "username:1C2B3A" | chpasswd']
CMD ["/bin/bash"]
```
1. Recall that in section 'Run an Image', -u username:groupname requires that the username and groupname exist when creating the image. Change the account password immedately after the container is created as the initial password is explicitly written in the Dockerfile.
2. By default, the 'RUN' instruction uses /bin/sh form. It is replaced by /bin/bash in this example.
  
  Also, multiple relevant shell commands are merged into one single 'RUN' instruction. We can also the split the command by line continuation like:
```
RUN /bin/bash -c 'useradd -ms /bin/bash -u 1000 -g 1000 username ; \
echo "username:1C2B3A" | chpasswd'
```
Some commands may have ask interactive questions. Please check DEBIAN_FRONTEND=noninteractive.
See multi-platform build.

Refer to Best practice for writing Dockerfile.

We can share our own docker image to a registry (e.g. docker.io) by docker push.

Suppose we have a nginx image got from Docker hub, and want to re-push it to Docker Hub under a personal account.

Pull official image. If we do not specify a tag, the latest image is pulled.

~ $ docker pull nginx
Using default tag: latest
...

~ $ docker login -u myaccount

Push to my personal account. If we do not specify a tag, the image is tagged to latest.

~ $ docker tag <nginx|sha256> myaccount/nginx

~ $ docker push myaccount/nginx
Using default tag: latest
...

If we want to use a different registry rather than the default docker.io, then add the registry to the new tag as well as follows.

~ $ docker tag <nginx|sha256> myregistry.com:5000/myaccount/nginx

~ $ docker push myregistry.com:5000/myaccount/nginx

We can assign mutiple tages to the image.

~ $ docker tag <nginx|sha256> myaccount/nginx:2.0.0
~ $ docker push myaccount/nginx:2.0.0

~ $ docker tag <nginx|sha256> myaccount/nginx:2.0
~ $ docker push myaccount/nginx:2.0

~ $ docker tag <nginx|sha256> myaccount/nginx:2
~ $ docker push myaccount/nginx:2

We can re-assign the latest tag to a new version.

~ $ docker tag myaccount/nginx:2.1.0 myaccount/nginx:latest
~ $ docker push myaccount/nginx:latest

If the imange is multi-platform (e.g. AMD64 and ARM64) capable, we have to repeat the pull, tag and push for each platform via option --platform. The more advanced tool regctl takes care of all platforms with just one command. Please read 1, 2 and 3.

~ $ regctl registry login
~ $ regctl registry config

~ $ regctl image manifest kong/kong-gateway:latest
~ $ regctl image inspect kong/kong-gateway:latest

~ $ regctl image manifest kong/kong-gateway:3.4.1.0

# pull, tag and push
~ $ regctl image copy kong/kong-gateway:3.4.1.0 kong/kong-gateway:latest
~ $ regctl image copy kong/kong-gateway:3.4.1.0-ubuntu kong/kong-gateway:latest-ubuntu

Apart from pushing to a registry, we can just share the image bundle.

~ $ docker images 'kong-wp'

~ $ docker image save -o kong-wp-3501.tar kong-wp:3.5.0.1

# for file transmission
~ $ tar -cJvf kong-wp-3501.tar.xz kong-wp-3501.tar
~ $ tar -xJvf kong-wp-3501.tar.xz

~ $ docker image load -i kong-wp-3501.tar

Pay attention that image bundle is different from the OCI container bundle. See What is the difference between save and export in Docker?.

Docker Compose

The compose project name by default is named after PWD. The name of containers share the same prefix (i.e. name of the project).

We can share compose configurations between files and/or projects by including other compose files or extending a service from from another service of another compose file. Check "biji/archive/kong-dev-compose.yaml".

When bind-mount a file, pay attention to provide the absolute path. Check data share.

In Docker Compose file, we can also use buildx to build an image from Dockerfile. Alternatively, we can also run multiple commands.

Troubleshooting

We can attach an ephemeral container to an existing container for troubleshooting purpose. The ephemeral container would share the target container's Linux namespaces.

For example, to debug network issues, we can make use of netshoot. The netshoot container has a world of built-in network troubleshooting tools like nmap, tcpdump, etc. We just need to attach the netshoot container to the target container's network namespace.

~ $ docker run --name netshoot \
--rm \
--network container:<target-name|target-ID> \
--mount type=bind,src=./data/,dst=/data \
-itd nicolaka/netshoot

~ $ docker exec -it netshoot zsh

# capture packets of target container
~ # tcpdump -i eth0 port 6379 -w /data/redis.pcap

For simple Linux utilities, we just busybox.

~ $ docker run --name test \
--rm \
--net=container:opentelemetry-otel-collector-1 --pid=container:opentelemetry-otel-collector-1 \
-it busybox:1.36

/ # ps aux
PID   USER     TIME  COMMAND
    1 10001     1:12 /otelcol-contrib --config /etc/otelcol-contrib/config.yaml
  212 root      0:00 sh
  218 root      0:00 ps aux

Policy Routing

2018-03-26T00:00:00+00:00

Policy Routing
Terminology
Principle
Iproute2 Tutorial
Launch WireGuard
Relation Between Ipset and Iptables
Create Ipset
Iptables Set MARK On IPset
Iproute2 Rule and Table On the MARK
Set Up Dnsmasq to Update Ipset
Iptables NAT by MASQUERADE
Local Init Script
Refs

This post is the successor of WireGuard.

Policy Routing

Policy routing could be accomplished based on bare IPs or domains. WireGuard post subscribes and updates IP sets periodically and adds relevant routes, which lacks flexibility as most IPs of the WireGuard set are not blocked by ISP. All traffic, by default, is routed to WireGuard except that of the CN set. The CN set can be called WireGuard blacklist. Another blacklist routing schema based on dnsmasq is dnsmasq-china-list.

Rather, WireGuard whitelist maintains a much smaller IP set of blocked domains - blocked IP set. Only route traffic of blocked set to WireGuard table while leaving the rest to the main table. All traffic, by default, is routed as usual.

Usually, each IP set corresponds to a domain set. For instance, the blocked IP set mainly derives from the blocked domain set though ISP may decide to add on individual IPs to the list occasionally.

In a nutshell, blacklist, whitelist and their sub-list can combine to realize policy routing as long decent routing rules and/or tables are configured. However, all the following configuration steps only involve whitelist set. To support other sets, just introduce extra routing tables and rules.

Terminology

A routing rule decides which routing table to look up.
A routing table is a collection of routes that refer to entries that direct packets.
Iproute2, Iptables, Ipset and Dnsmasq together do smart routing, namely policy routing.

Principle

Dnsmasq serves as a smart DNS service, maintaining domain sets: blocked domain set, non-blocked domain set, and sub-sets.
A relevant kernel Ipset is created in respond to each domain set.
Dnsmasq updates Ipsets on the fly.
Iptables set MARK on Ipsets.
Iproute2 rules and tables route packets based on Iptables MARK.

Iproute2 Tutorial

iproute2 is successor of the ancient ifconfig, providing userspace utilities for TCP / IP networking and traffic control in Linux, including routing, interfaces, tunnels, and traffic control.
Predefined identifiers (number and string name pairs) are located under /etc/iproute2/. An item can be either referenced by its number or name. For example, table 254 main is where we update routes mostly.
A metric or preference is to set preference among routes or rules with equal specificity.

Check routing rules and routes:

root@tux ~ # man [ ip-route | ip-rule ]
root@tux ~ # ip rule list
root@tux ~ # ip route [ list table main ]

Create routing rules and tables:

root@tux ~ # ip -4 rule add not fwmark 0xca6c pref 10 table 0xca6c
root@tux ~ # ip rule add to 223.255.236.0/22 pref 20 table 254

New routing tables are created on demand (i.e. not exist)
pref specifies rule preference.

Add routes:

root@tux ~ # ip route add 1.0.1.0/24 via 192.168.0.1 dev wlan0 proto dhcp src 192.168.0.100 metric 30
root@tux ~ # ip route add 8.8.8.8/32 dev wlan0 pref 40

via means the gateway.
dev followed by the outgoing interface.
pref and metric are the same thing: priority/preference.

Launch WireGuard

wg-quick generates Iproute2 rules and tables to route all traffic through WireGuard except that of endpoint. To support different sets, we should carefully configure Iproute2.

The following setup is an example of WireGuard whitelist set. In spite of wg-quick method in prior post, manual configuration is preferred in this post.

01-wg0.start:

#!/bin/sh

ip link add wg0 mtu 1420 type wireguard
ip link set dev wg0 up
ip addr add 10.0.0.2/24 dev wg0
wg setconf wg0 /etc/wireguard/wg0.conf

The script is updated as this post progresses. Similarto the preceding post, set this as local.d Init service.

Relation Between Ipset and Iptables

"IP set" is a framework inside the Linux kernel, which can be administered by userspace utility ipset. Depending on the type, an IP set may store IP addresses, networks, (TCP/UDP) port numbers, MAC addresses, interface names or combinations of them in a way, which ensures lightning speed when matching an entry against a set.

Linux kernel must enable IP set and then net-firewall/ipset is required. IP set can be enabled by IP_SET and relevant sub-options such as IP_SET_HASH_IP, IP_SET_HASH_NET and IP_SET_LIST_SET. Alternatively, enable USE=modules of net-firewall/ipset.

An IP set based on blocked domains will be updated on the fly. To make use of the IP set, please enable Netfilter (Iptables) SET match extension (NETFILTER_XT_SET) and MARK target (NETFILTER_XT_MARK).

Create Ipset

Installation:

# USE=modules is not enabled.
root@tux ~ # emerge -avt ipset
root@tux ~ # rc-update add ipset default

Now we can create kernel IP set with ipset command:

root@tux ~ # ipset -! create gfwlist hash:net
root@tux ~ # ipset list [ gfwlist ]

By default, ipset will throw an error errors when creating sets or adding elements that do exist or when deleting elements that don't exist. -!, -exist will ignore the error.

An IP set called 'gfwlist' is created but it's an empty set without any IP elements. Let's add a few entries:

root@tux ~ # ipset -! add gfwlist 8.8.8.8; ipset list
root@tux ~ # ipset -! add gfwlist 8.8.4.4; ipset list
root@tux ~ # ipset -! del gfwlist 8.8.4.4; ipset list

We want all IPs associated with blocked domains be added to 'gfwlist' set, which is impossible without the help of dnsmasq as IPs vary frequently. That is to say, 'gfwlist' set should be updated frequently as well. Especially, we want to add clean DNS servers (i.e. 8.8.8.8) to 'gfwlist' at first.

Save current state

root@tux ~ # rc-service ipset save
# -or-
root@tux ~ # ipset save > /var/lib/ipset/rules-save
# Launch
root@tux ~ # rc-service ipset start; ipset list

By default, ipset service will load prior set on startup while save current set on stop.

Iptables Set MARK On IPset

Having Kernel IP set created and preliminarily updated, we assign MARK target to 'gfwlist' set with Iptables.

# Create a new chain
root@tux ~ # iptables -t mangle -N GFWLIST

# OUTPUT to GFWLIST
root@tux ~ # iptables -t mangle -C OUTPUT -j GFWLIST || iptables -t mangle -A OUTPUT -j GFWLIST
# Set mark 51820 on packets of 'gfwlist' IP set.
root@TUX ~ # iptables -t mangle -A GFWLIST -m set --match-set gfwlist dst -j MARK --set-mark 51820

# Only required when peer A serves as a gateway as well.
root@tux ~ # iptables -t mangle -C [ FORWARD | PREROUTING ]  -j GFWLIST || iptables -t mangle -A [ FORWARD | PREROUTING ] -j GFWLIST (opt)
root@tux ~ # sysctl -w net.ipv4.ip_forward=1 (opt)

mangle table is used for specialized packet alteration.
A new chain GFWLIST is created for 'gfwlist' IP set for better isolation.
-C checks whether a rule exists or not.
When a packet hits FORWARD chain, it has been routed already. Instead, PREROUTING is hit before routing.

Iproute2 Rule and Table On the MARK

Now that 'gfwlist' IP set is marked by Iptables, we will create a routing rule and table for the mark. Before that we can define a string identifier for the table.

# Number and name identifier
root@tux ~ # echo "51820	gfwlist" >> /etc/iproute2/rt_tables

# Create routing rule and table: traffic of 'gfwlist' set is routed through 'gfwlist' table
root@tux ~ # ip rule add fwmark 51820 table gfwlist

# Set routing table: going out through WireGuard.
root@tux ~ # ip route add default dev wg0 table gfwlist

Identifier of Iproute2 table (51820 gfwlist) needn't to be the same as that of Iptables mark (51820).

Recall that, we've manually added 8.8.8.8 to 'gfwlist' set. Instead, add a new route in the main table though it is somewhat less mart.

root@tux ~ # ip route add 8.8.8.8 dev wg0 table main

01-wg0.start:

#!/bin/sh

# Bring up WireGuard link
ip link add wg0 type wireguard
ip link set mtu 1420 dev wg0
ip addr add 10.0.0.2/24 dev wg0
wg setconf wg0 /etc/wireguard/wg0_wg.conf
ip link set up dev wg0

# To route server's internal IP
ip route add 192.168.58.0/24 dev wg0
# -or-
#ip rule add 192.168.58.0/24 table gfwlist

# To route 'gfwlist' IP set
ip rule add fwmark 51820 table gfwlist
ip route add default dev wg0 table gfwlist

In this case, the internal IP of client and server fall into different network block, namely 10.0.0.0/24 and 192.168.58.0/24. As mentioned in the prior post, explicit route should be added on both sides.
This is almost the final version. As 192.168.58.0/24 is an LAN block, we'd better leave it in the main table.

Set Up Dnsmasq to Update Ipset

The key success of policy routing depends on the accuracy of 'gfwlist' IP set. So far, it is almost empty besides a few clean DNS elements. This is where Dnsmasq comes to help!

Dnsmasq provides Domain Name System (DNS) forwarder, Dynamic Host Configuration Protocol (DHCP) server, router advertisement and network boot features for small computer networks, created as free software. Here, we use Dnsmasq to provide smart DNS service locally and update 'gfwlist' IP set on demand.

Installation:

root@tux ~ # echo 'net-dns/dnsmasq -dhcp dnssec' > /etc/portage/package.use/dnsmasq
root@tux ~ # emerge -avt net-dns/dnsmasq
root@tux ~ # rc-update add dnsmasq default

Dnsmasq is presented with predefined blocked domains. Dnsmasq resolves them into IPs upon receiving DNS query.
Dnsmasq now has builtin Ipset support and the resolved IPs can be added to 'gfwlist' IP set automatically.

Configuration Files

All options within the default /etc/dnsmasq.conf are commented out. We can leave that as a reference but load configuration from separate files.

Firstly, we add a command line --conf-dir to /etc/conf.d/dnsmasq:

# Loads all files with the suffix .conf; ignore ther others

DNSMASQ_OPTS="--user=dnsmasq --group=dnsmasq --conf-dir=/etc/dnsmasq.d/,*.conf"

A better idea is appending that option to /etc/dnsmasq.conf so that resolvconf will pick it up:

root@tux ~ # echo >> /etc/dnsmasq.conf
root@tux ~ # echo 'conf-dir=/etc/dnsmasq.d/,*.conf' >> /etc/dnsmasq.conf

root@tux ~ # mkdir -p /etc/dnsmasq.d

Configure Dnsmasq

Dnsmasq will be the sole DNS server in the system in the following set up. Please prevent dhcpcd from overriding /etc/resolv.conf:

# /etc/dhcpcd.conf

nohook resolv.conf

Then we create /etc/dnsmasq.d/dns.conf:

# lo only
bind-interfaces
listen-address=127.0.0.1
# lo and eth2
#bind-dynamic
#interface=eth2

# DNS port
port=53

# dbus
enable-dbus

# dnssec
dnssec
conf-file=/usr/share/dnsmasq/trust-anchors.conf
#dnssec-check-unsigned

# Upstream nameservers (i.e. from ISP)
resolv-file=/etc/dnsmasq.d/resolv.dnsmasq

# hosts addon
addn-hosts=/etc/dnsmasq.d/hosts.dnsmasq

dnssec-check-unsigned would drop DNS replies if upstream DNS servers does not support DNSSEC. Up to now, I cannot find any ISP DNS servers with DNSSEC support. Hence, dnssec-check-unsigned cannot resolve unblocked domains from upstream nameservers.

If we want Dnsmasq to support newly created interface on demand, we can use bind-dynamic:

# lo only
#bind-interfaces
#listen-address=127.0.0.1

# lo and eth2
bind-dynamic
interface=eth2

This would implicitly include lo.

Update Blocked Domains and Ipset

As discussed earlier, we should firstly fuel Dnsmasq with blocked domains. Here is a predefined and updated blocked domain list: gfwlist2dnsmasq.

root@tux ~ # git clone --depth=1 https://github.com/cokebar/gfwlist2dnsmasq
root@tux ~ # ./gfwlist2dnsmasq.sh -h
root@tux ~ # ./gfwlist2dnsmasq.sh -d 8.8.8.8 -p 53 -s gfwlist -o /etc/dnsmasq.d/gfwlist.conf

The accompanying script gfwlist2dnsmasq.sh automatically fetches newest blocked domains and generates corresponding Dnsmasq configuration file without user intervention. Then, set cron job to update 'gfwlist' set periodically:

#!/bin/sh

/opt/gfwlist2dnsmasq/gfwlist2dnsmasq.sh -d 8.8.8.8 -p 53 -s gfwlist -o /etc/dnsmasq.d/gfwlist.conf

A new dnsmasq configuration file which only contains server and ipset pairs like:

server=/030buy.com/8.8.8.8#53
ipset=/030buy.com/gfwlist

server=/0rz.tw/8.8.8.8#53
ipset=/0rz.tw/gfwlist

dnsmasq resolves 030buy.com and 0rz.tw through 8.8.8.8:53, which would be routed through WireGuard.

So far, Dnsmasq is done! However, we can't even dig blocked domains.

Iptables NAT by MASQUERADE

To be clear, we should enable MASQUERADE or SNAT as post linux as gateway router writes. Please be noted policy routing on the client side do not require IP forwarding or Iptables FORWARD.

Please refer to the Netfilter flow chart:

Locally generated packets are routed twice before going out. The first routing is exercized before entering the Iptables flow as routing decision. The second routing happens within the flow as "reroute check".
Routing will not modify any field of packet!
We can examine Iptables logs at different steps of the flow chart.

Let us have a look at Iptables log on -t nat OUTPUT chain:

root@tux ~ # iptables -t nat -A OUTPUT -m mark --mark 51820 -j LOG --log-level debug
root@tux ~ # dmesg -w
root@tux ~ # dig www.google.com

Here is an log sample:

[20083.739224] IN= OUT=wlan0 SRC=192.168.0.100 DST=8.8.8.8 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=10293 DF PROTO=UDP SPT=38374 DPT=53 LEN=40 MARK=0xca6c 
[20083.739408] IN= OUT=wlan0 SRC=192.168.0.100 DST=23.45.67.89 LEN=124 TOS=0x00 PREC=0x00 TTL=64 ID=14878 PROTO=UDP SPT=12345 DPT=54321 LEN=104 

The first line. The packet DST is 8.8.8.8 with MARK as 0xca6c, which means this is the original DNS query packet. The routing decision successfully matched the default route in main table. Afterwards, the packet was set MARK by -t mangle OUTPUT chain. Then reroute check routed the packet against gfwlist rule and table according to the the MARK. This packet went to wg0 and we get the log at -t nat OUTPUT.
The second line. The original DNS packet was encapsulated into a new WireGuard UDP with DST 23.45.67.89 (server's public IP). The new packet was then sent to server.
IN is empty as local generated packets do have associated any input interface.

As mentioned earlier, routing does not change a packet. However, the OUT and SRC of the first packet should be changed to wg0 and 10.0.0.1 (internal IP of client A) after reroute check and before encapsulation by WireGuard. It looks as though the packet were generated by wg0. This can be achieved by Iptables NAT. That is to say, the first packet should be transferred from:

# Before 'reroute check'.
OUT=wlan0 SRC=192.168.0.100 DST=8.8.8.8

# After 'reroute check' and before encapsulation.
OUT=wg0 SRC=10.0.0.1 DST=8.8.8.8 MARK=0xca6c

Without Iptables NAT support, reply packets from the server would be abandoned. Suppose the VPN server receives the second line and replies with DST 192.168.0.100 and DPT 12345. The reply packet is then passed to wg0 in accord with DPT where WireGuard listens for traffic. Upon receiving the packet, wg0 abandons it immediately as DST does not match 10.0.0.1 - malformed packet.

Iptables NAT usually is executed in -t nat POSTROUTING chain with MASQUERADE or SNAT target.

root@tux ~ # iptables -t nat -A POSTROUTING -o wg0 -m mark --mark 51820 -j SNAT --to-source 10.0.0.1
# -or-
root@tux ~ # iptables -t nat -A POSTROUTING -o wg0 -m mark --mark 51820 -j MASQUERADE

We can also examine Iptable logs in other tables and chains in accordance with the figure above. For example, before -t mangle OUTPUT the first packet is not marked. After -t nat POSTROUTING, the first packet is modified accordingly.

Up to now, policy routing with WireGuard whitelist ('gfwlist' Ipset) is done! We can add extra Ipsets and relevant Iproute2 routing rules and tables. Of course, new Ipsets should also be marked by Iptables.

Local Init Script

01-wg0.start:

#!/bin/sh

# Bring up WireGuard link
ip link add wg0 mtu 1420 type wireguard
ip link set dev wg0 up
ip addr add 10.0.0.2/24 dev wg0
wg setconf wg0 /etc/wireguard/wg0.conf

# ip route
ip route add 192.168.58.0/24 dev wg0
# -or-
#ip rule add 192.168.58.0/24 table gfwlist

# ip rule
ip rule add fwmark 51820 table gfwlist
ip route add default dev wg0 table gfwlist

01-wg0.stop:

#!/bin/sh

# del gfwlist table
ip rule del table gfwlist

# ip route
ip route del 192.168.58.0/24 dev wg0
# -or-
#ip rule del 192.168.58.0/24 table gfwlist

# Remove WireGuard
ip link del dev wg0 type wireguard

The VPN Proxy section in prior post, all traffic is routed through wg0. So it presents multiple methods (i.e. suppress_prefixlength 0 from wg-quick) such that traffic to the server's public IP is routed through the main table instead.

The scripts above do not require any of those methods as 'gfwlist' does not include that IP. More specifically, as long as Ipset is used, we can put the IP into a set (i.e. CN set) routed by main table. At the same time, make sure it is not covered by Ipsets routed through wg0.

Refs

WireGuard

2018-03-20T00:00:00+00:00

Preamble
Installation
Schema
Create WireGuard Interface
1. Assign WireGuard IP
WireGuard Key pair
Configure WireGuard Interface
Persistent Configuration
WireGuard Link Up
Check WireGuard Route
Allow endpoint in Iptables
1. Final ping
Set Server B as a Gateway
LAN Access
VPN Proxy
Policy Routing Based on Bare IPs
1. IP set
  1. Static Route Method
  2. Geoip method
2. Improve Local DNS Servers
wg-quick
Booting
Notes
Reference

Preamble

WireGuard claim to utilizes state-of-the-art cryptography (Noise protocol framework, Curve25519, ChaCha20, Poly1305, BLAKE2, SipHash24, HKDF etc.) while provide a simple construction.

It requires a virtual network interface and an IP assigned. The interface is associated with a public/private key pair of which the public part serves as the identity within WireGuard network. WireGuard encapsulates IP packets (application data payload) over UDP and forwards packets based on Cryptokey Routing- a UDP VPN.

Originally, WireGuard is integrated into Linux kernel (module or built-in tree). Nowadays, it's under heady development to be cross-platform and widely deployable. The 6th reference (NAT-to-NAT VPN with WireGuard) is worth reading.

Installation

To install as an external module:

root@tux ~ # emerge -avt wireguard openresolv
root@tux ~ # modinfo wireguard

Optionally, we can build WireGuard as an internal kernel module or as built-in with module-src USE.

openresolv is needed if wireguard (by wg-quick) defines custom DNS.

Schema

Before everything else, we should grasp the conceptual overview.

Peer	Client A	Server B
External IP	192.168.0.123	23.45.67.89
UDP port	48574	39814
Internal IP	10.0.0.1/24	10.0.0.2/24

External IP (i.e. eth0) is given by ISP or Wi-Fi router, without which you cannot reach the Internet. On the other hand, internal IP manually assigned to WireGuard interface is privately valid only within WireGuard network. UDP port associated with external IP is where WireGuard service listens for traffic.

WireGuard does not assume server side or client side. The peer initiates connections is regarded as the client. For terminology consistency, peer B with public IP is the server while peer A censorshipped by ISP is the client. In this post, peer A sites behind a Wi-Fi router (NAT).

Clearly, Peer A, on the initiative, opens connections to peer B establishing a point-to-point tunnel. With a few arguments adjustment, peer A reaches the whole remote block 10.0.0.0/24 such as devices without WireGuard but IP falling into that range. A step further, peer A can route all its traffic through B to the Internet.

Firstly, we start off by establishing a point-to-point link. Without explicit notice, the configuration steps should be executed on both sides.

Create WireGuard Interface

Create Wireguard interface with Iproute2, which automatically loads wireguard module:

root@tux ~ # modprobe -v wireguard (opt)
root@tux ~ # ip link add dev wg0 type wireguard
root@tux ~ # ip link

Assign WireGuard IP

Unless you have a really good reason, please assign internal IPs within the same subnet for client and server sides. Otherwise, you want extra routes on both sides.

root@tux ~ # ip addr add 10.0.0.1/24 dev wg0
root@tux ~ # ip addr

WireGuard Key pair

Permission:

root@tux ~ $ umask 077
root@tux ~ $ mkdir -p /etc/wireguard/keys; cd /etc/wireguard/keys/

Private/public keys:

root@tux ~ $ wg genkey > private-key; cat private-key
root@tux ~ $ wg pubkey < private-key > public-key; cat public-key

Optionally, this can be done all at once:

root@tux ~ $ wg genkey | tee private-key | wg pubkey > public-key

Configure WireGuard Interface

Arguments can be loaded from file.

root@tux ~ # wg setconf wg0 /etc/wireguard/wg0.conf
# -or-
root@tux ~ # wg set wg0 listen-port 48574 private-key /etc/wireguard/private-key
root@tux ~ # wg set wg0 peer <public-key of peer> [persistent-keepalive 25] [endpoint 23.45.67.89:39814] allowed-ips 10.0.0.2/32,10.0.100.2/24
root@tux ~ # wg [showconf wg0]

Public key should be base64 format like V7g3kxzLATJ6edBybau1IrE3FOgLHajxxFfMZ+QOUyE=.
To traverse NAT or firewall, persistent-keepalive is a must for peers behind NAT or firewall.

For instance, client A sites behind NAT (i.e. Wi-Fi router), persistent-keepalive sends packets to other peers periodically to keep NAT mapping open.
endpoint is the remote peer's external yet public IP and UDP port.

Only client sides that makes the initial connection require endpoint. It's recommended to leave this argument out when setting server sides because they update endpoint values by examining from where correctly authenticated packets originates. Meanwhile, clients behind NAT or dial-up do not even have fixed public IPs.
allowed-ips is a list of comma-separated IP ranges to which WireGuard traffic can be sent and from which WireGuard traffic can be recevied.

For a simple point-to-point connection, it should be a peer's internal IP. To reach the whole LAN network, we set it to 10.0.0.0/24.

The catch-all 0.0.0.0/0 and ::/0 match all IPv4/6 addresses, routing all local traffic through WireGuard. This is useful if we want to set up transparent proxy on client sides.

Whatever network ranges we choose, the peer's internal IP must be covered.
Multiple peers can be added.

Persistent Configuration

Setup above is versatile across reboots. To be permanent, arguments can be loaded from file on boot. By default, configuration file locates under /etc/wireguard/.

To save and load configuration:

root@tux ~ # wg showconf wg0 > /etc/wireguard/wg0.conf
root@tux ~ # wg setconf wg0 /etc/wireguard/wg0.conf

Here is an example of point-to-point VPN link:

[Interface]
ListenPort = 48574
PrivateKey = EOpLmUJ08uwH2z2NpBwJ2upWzB5Tn36nhQvlXccAFnk

[Peer]
PublicKey = 7nXZYaqyXuVdW0RwHauMJuW81axAM9hSavY9JIJsZFU=
AllowedIPs = 10.0.0.2/32
Endpoint = 23.45.67.89:39814

WireGuard Link Up

root@tux ~ # ip link set up dev wg0
root@tux ~ # ip link; ip addr
root@tux ~ # ss -npelu

A WireGuard UDP socket is created.

Check WireGuard Route

When internal IP is assigned, a system route is added automatically for that IP range like:

10.0.0.0/24 dev wg0 proto kernel scope link src 10.0.0.1

If a remote peer's internal IP (be covered by allowed-ips) belongs to a different subnet (i.e. B's is 192.168.58.1/32), they cannot connect to each other unless a special route is created on both sides.

# client A
root@tux ~ # ip route add 192.168.58.0/24 dev wg0 proto kernel scope link
# server B
root@tux ~ # ip route add 10.0.0.0/24 dev wg0 proto kernel scope link

For strict control, the range could be narrowed down to the internal IP address like:

# client A
root@tux ~ # ip route add 192.168.58.1/32 dev wg0 proto kernel scope link

Allow endpoint in Iptables

Please make sure the endpoint is accepted by firewall.

root@tux ~ # iptables -A OUTPUT -d 23.45.67.89/32 -p udp -m udp --dport 39814 -m conntrack --ctstate NEW -j ACCEPT
root@tux ~ # iptables -A INPUT -s 23.45.67.89/32 -p udp -m udp --sport 39814 -m conntrack --ctstate NEW -j ACCEPT

Final ping

By far, a point-to-point link is established. Peer A and B can communicate with each other.

Set Server B as a Gateway

For client A's visit to remote LAN or even the Internet, server B should be set as a gateway involving the following steps:

IP forwarding and optional Proxy ARP.
Iptables FORWARD.
Iptables NAT.

IP Forwarding

root@tux / # sysctl -w net.ipv4.ip_forward=1

Usually, IP forwarding is not needed for communication within LAN. However, if peer A want to reach 10.0.0.123 without WireGuard setup, it must depend on peer B's forwarding functionality. This is because A does not have direct Ethernet connection with 10.0.0.123.

Needless to say, to reach the Internet through server B, this is a must.

Proxy ARP on Server Side

Optionally, enable Proxy ARP on wg0. However, it seems this step could be omitted.

# Permanent across reboots
root@tux ~ # echo 'net.ipv4.conf.wg0.proxy_arp=1' > /etc/sysctl.d/25-proxy_arp.conf

For the current session:

root@tux ~ # sysctl -p /etc/sysctl.d/25-proxy_arp.conf
# -or-
root@tux ~ # sysctl -w net.ipv4.conf.all.proxy_arp=1

Allow IP Forwarding through Iptables

root@tux ~ # iptables -A FORWARD -i wg0 -o eth0 -j ACCEPT

We assume eth0 is the remote interface that has Internet access.
You may also choose to ACCEPT -i eth0 -o wg0. For example, server B initiates connection to client A.

Set up Iptables NAT

root@tux ~ # iptables -t nat -A POSTROUTING -i wg0 -o eth0 -s 10.0.0.0/24 -j SNAT --to-source <IP-of-eth0>

Similarly, you may also enable -i eth0 -o wg0.

LAN Access

In order to reach the whole remote block "10.0.0.0/24* including devices without WireGuard, client A should set allowed-ips to cover the LAN range.

VPN Proxy

All client A's traffic goes through the WireGuard tunnel: to set up VPN routing as a proxy.

Please be noted that all configurations in this section are done on client A.

allowed-ips Catch All

Similar to LAN Access section above, client A should firstly set its allowed-ips to cover the catch-all block: 0.0.0.0/0 and/or ::/0.

Update Client A Routes

To route all local traffic through the tunnel, the default route in main table should be bypassed as it catches all traffic. Specifically, all traffic except that of endpoint (server B's public IP) goes to wg0. The key is to route traffic to wg0 before the default route entry, such that only traffic to the server is routed through the main table.

Of all the methods mentioned below, wg-quick is robust to peer roaming (endpoint change) as it does not require explicit endpoint routing to be added.

Replace the default Route

user@root ~ # ip route del default
user@root ~ # ip route add default dev wg0
user@root ~ # ip route add 23.45.67.89/32 via 192.168.0.1 dev wlan0

Override the default Route

Override the default with two more specific rules that add up to the default but match before the default. We split the default into 0.0.0.0/1 and 128.0.0.0/1.

user@root ~ # ip route del default (optional)
user@root ~ # ip route add 0.0.0.0/1 dev wg0
user@root ~ # ip route add 128.0.0.0/1 dev wg0
user@root ~ # ip route add 23.45.67.89/32 via 192.168.0.1 dev wlan0

Rule-based routing

The two methods above operates on route entries of the main routing table. Here, new routing rules and tables are created.

Notice: a routing rule is set for a specific routing table, namely to determine which routing table is consulted when routing. Without explicit priority value, new rules are prepended to old ones.

# Traffic to server's public IP is routed on the main table.
user@root ~ # ip rule add to 23.45.67.89/32 lookup main pref 30

# All the rest is routed through WireGuard table.
user@root ~ # ip rule add to all lookup 51820 pref 40

# Create the default route for the new table.
user@root ~ # ip route add default dev wg0 table 51820
# -or-
user@root ~ # ip route add 0.0.0.0/0 dev wg0 table 51820

Routing table 51820 is created for wg0. A routing table can be referenced by its number or string name.
pref means priority of a routing table.
The keyword default can be replaced with 0.0.0.0/0 that resembles the catch-all block of allowed-ips.

wg-quick method

All the three methods above requires explicit route for server's public IP. However, such configuration is subject to peer roaming where public IP varies without prior notice. wg-quick utilizes an improved but obscure rule-based method: fwmark and suppress_prefixlength.

# Mark all traffic of *wg0*.
user@root ~ # wg set wg0 fwmark 51820

# Not marked traffic is routed through WireGuard table.
user@root ~ # ip -4 rule add not fwmark 51820 table 51820

# Create the default route for the new table.
user@root ~ # ip -4 route add 0.0.0.0/0 dev wg0 table 51820

# Suppress routing tables that the prefix length of the longest match
# is <= 0 in the main table.
# Namely skip the default entry (prefix length is 0).
user@root ~ # ip -4 rule add table main suppress_prefixlength 0

The resulting routing rules look like:

from all lookup local 
from all lookup main suppress_prefixlength 0
not from all fwmark 0xca6c lookup 51820 
from all lookup main 
from all lookup default

32764: specific routes in the main table except the default is respected.
32765: all the rest goes to table 51820.
32766: traffic to the server goes to the main table including the default.

suppress_prefixlength

Recall that an IP can be divided into the network part and the host part. Prefix length means the length of the network part. For example, the prefix length of 192.168.0.100/24 is 24 while that of 12.34.56.78/32 is 32.

Say we have a rule from all lookup table 42 suppress_prefixlength 16. When network stack hits the rule, it looks up routes in table 42 against destination IP as it would do to rule from all lookup table 42. When the longest match is found in table 42, it checks the route entry's prefix length. If the prefix is longer than 16 bits, it proceeds as usual. Otherwise (shorter than or equal to 16 bits), network stack go to check the next routing rule. That is to say, prefix length (of longest match) less than or equal to 16 is ignored (suppressed).

Here is a more detailed explanation. Suppose table 42 contains two entries:

# prefix 8
10.0.0.0/8 dev eth0
# prefix 24
10.1.1.0/24 dev eth1

And the destination IP is 10.1.1.1. Obviously, the 2nd entry matches (24 > 16 > 8) and routes the packet to eth1. If 10.2.1.1, then the 1st entry has the longest match but prefix is less than or equal to 16 bits. Hence table 42 does not route the packet.

suppress_prefixlength wants to set a minimal network prefix threshold for routes! The longer a prefix is, the more favored the route entry is: to avoid routes that are too general. Specially, threshold 0 just excludes the default entry (i.e. default via 192.168.0.1 dev eth0) as the prefix length is 0 (0.0.0.0/0). Therefore suppress_prefixlength 0 above suppresses or skips the default entry in the main (254) table.

Procedures Illustrated

Now let's go through an example to see how a packet is routed through rules of wg-quick:

dig @8.8.8.8 www.bing.com sends DNS query to 8.8.8.8.
Network stack hits rule 32764 and checks main table.

The default entry matches 8.8.8.8/32 but its prefix length is 0. So it's ignored.

Suppose there are not any other routes.
Network stack checks the next rule 32765.

The DNS packet is not marked as 51820 and successfully routed to 51820 table, going to wg0.
WireGuard received the DNS query packet.

It encrypts the packet and forms a new UDP packet with destination IP as endpoint (server's public IP). This newly generated packet has fwmark 51820.
Network checks rule 32764 to route the new UDP packet.

Similary the default entry is ignored.
Network stack checks rule 32765.

Due to fwmark 51820, table 51820 is ignored either.
Network stack checks rule 32766.

It's the main table again. This time there is no suppress_prefixlength 0 limit. The WireGuard UDP packet is routed to default entry of main table, going out!

Policy Routing Based on Bare IPs

In VPN Proxy section, traffic with fwmark is routed through the main table while that without the mark is routed through wg0 table. Therefore, all traffic by default goes through WireGuard. Such all-in-one routing method is somewhat inefficient.

For example, we can route some traffic (i.e. to www.baidu.com) through the main table while some (i.e. to www.block.com) through wg0 table. To achieve smart routing, we can mark former part while leaving the rest alone.

To set fwmark for a packet, we can resort to Iptables and IP set with the help of dnsmasq like this post. Obviously, there is no need to modify the routing rules and tables above.

The two methods followed are somewhat less elegant but deserves attention. They are mainly based on bare IPs.

IP set

Firstly, we divide IPs into two sets. Traffic of one goes to WireGuard while the other not. IPs are from APNIC Delegated List.

CN set:

user@tux ~ # wget -O- 'http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest' | awk -F\| '/CN\|ipv4/ { printf("%s/%d\n", $4, 32-log($5)/log(2)) }' > CNip.txt

non-CN set:

user@tux ~ # wget -O- 'http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest' | awk -F\| '/\|ipv4\|/ && ! /\|CN\|/ && ! /\|\*\|/ { printf("%s/%d\n", $4, 32-log($5)/log(2)) }' > nonCNip.txt

We can set a cron job to update IP set.

Static Route Method

One obvious method is adding all IPs of non-CN set to server peer's allowed-ips while removing the catch-all 0.0.0.0/0. wg-quick will create the relevant routes in main table like:

1.0.128.0/17 dev wg0 proto kernel scope link src 10.0.0.1

However, that would fluff the allowed-ips list and increase maintainance burden.

Similarly we can also create routes in main table for IPs of CN set like:

1.0.32.0/19 via 192.168.0.1 dev wlan0 proto dhcp src 192.168.0.123 metric 304

I will choose the 2nd method as:

root@tux ~ # xargs -a /etc/wireguard/CNip.txt -I'{}' ip route add '{}' via 192.168.0.1 dev wlan0 proto dhcp src 192.168.0.123 metric 304
# -or-
root@tux ~ # wget -O- 'http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest' | awk -F\| '/CN\|ipv4/ { printf("%s/%d\n", $4, 32-log($5)/log(2)) }' | xargs -I'{}' ip route add '{}' via 192.168.0.1 dev wlan0 proto dhcp src 192.168.0.123 metric 304

Don't worry about route entries if you manually bring down WireGuard.
We have src in the route entries, which would fail if wlan0 or wg0 do not have a IP the time we add routes (i.e. Wi-Fi router down).

To simplify things, just remove src part. If you'd like, remove metric and proto as well.

Read more on chnroutes and chinaroute路由表更新命令

Geoip method

Instead of IP set, we can make use of geopip extension of iptables. This is much more simpler.

root@tux ~ # iptables -t mangle -A OUTPUT -o wlan0 -m geoip --dst-cc CN -j MARK --set-mark 51820

Recall the ip rule above, packets with 51820 mark will be routed by main table as usual.

Improve Local DNS Servers

Policy routing based on IPs highly depends on correct DNS resolving when visiting a domain. Accordingly, a custom clean DNS server (i.e. 8.8.8.8) other than the one provided by ISP (prone to DNS poisoning) is required. This, in turn, would degrade performance when visiting domestic domains as custom DNS servers such as 8.8.8.8 usually return IPs geographically far.

Consequentially, smart DNS service that resolves domestic domains as usual (geographically nearby) but censorshipped domains correctly. There exist such smart DNS servers around. Just make sure traffic to it goes to WireGuard! In other words, organize IPs of smart DNS servers into the non-CN set. The goal is to set system DNS servers such that they resolve domestic domains into IPs of CN set while foreign domains into IPs of non-CN set.

Alternatively, a smart DNS server can be set locally such like ChinaDNS. An far better method is illustrated in iproute2, iptables, dnsmasq

wg-quick

wg-quick is just a wrapper script of commands of ip and wg above to faciliate the interface and WireGuard setup.

# wg-quick [ up | down | save ] [ CONFIG_FILE | INTERFACE ]
root@tux ~ # wg-quick save wg0
root@tux ~ # wg-quick up wg0

Please make sure /etc/wireguard/INTERFACE.conf file exist.

Attention: the configuration file of wg-quick introduces a few extra arguments (i.e. internal IP, DNS) to format understood by wg in order to configure additional attributes. wg-quick handles the values that it understands, and then passes the rest directly to wg setconf. Especially, wg-quick update system route table automatically.

Here is an client example:

[Interface]
ListenPort = 48574
Address = 10.0.0.1/24
DNS = 8.8.8.8
SaveConfig = true
PrivateKey = EOpLmUJ08uwH2z2NpBwJ2upWzB5Tn36nhQvlXccAFnk

[Peer]
PublicKey = 7nXZYaqyXuVdW0RwHauMJuW81axAM9hSavY9JIJsZFU=
AllowedIPs = 10.0.0.2/32
Endpoint = 23.45.67.89:39814

You can see that "Address", "DNS" and "SaveConfig" items are added. The DNS defined by wg-quick will replace local ones with help of resolvconf.

Here is an server example:

[Interface]
Address = 10.0.0.2/24
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 39814
PrivateKey = [SERVER PRIVATE KEY]

[Peer]
PublicKey = [CLIENT PUBLIC KEY]
AllowedIPs = 10.0.0.1/32  # Client's internal IP.

Booting

Gentoo wg-quick

Here I use local.d startup script. Please make sure the scripts are executable.

Up: /etc/local.d/01-wireguard.start:

#!/bin/sh

echo "Starting WireGuard ..."
/usr/bin/wg-quick up wg0

echo "Adding CNip routes in background ..."
xargs -a /etc/wireguard/CNip.txt -I'{}' ip route add '{}' via 192.168.0.1 dev wlan0 proto dhcp src 192.168.0.123 metric 304 >/dev/null 2>&1 &

The route part can be moved to /etc/dhcpcd.exit-hook as well. But on my system, dhcpcd reports:

Error: Device for nexthop is not up

From the message, wlan0 is not up before route entries added.

Down: /etc/local.d/01-wireguard.stop:

#!/bin/sh

echo "Stopping WireGuard ..."
/usr/bin/wg-quick down wg0

echo "Removing CNip routes in background ..."
xargs -a /etc/wireguard/CNip.txt -I'{}' ip route del '{}' via 192.168.0.1 dev wlan0 proto dhcp src 192.168.0.123 metric 304 >/dev/null 2>&1 &

By default, the local service will silence all output. Setting rc_verbose=yes will cause it to show which scripts were run and their output, if any.

# /etc/conf.d/local
rc_verbose=yes

Check rc_verbose in /etc/rc.conf.

systemd wg-quick

root@tux ~ # systemctl enable wg-quick@wg0
root@tux ~ # systemctl start wg-quick@wg0

cron job

Set a cron job to update IP set daily.

root@tux ~ # crontab -u root -e
#
9  14  * * *     root    wget -O- 'http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest' | awk -F\| '/CN\|ipv4/ { printf("%s/%d\n", $4, 32-log($5)/log(2)) }' > /etc/wireguard/CNip.txt

Alternatively, we can set a script file (executable) into /etc/cron.daily/.

#!/bin/sh

#/etc/cron.daily/CNip

wget -O- 'http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest' | awk -F\| '/CN\|ipv4/ { printf("%s/%d\n", $4, 32-log($5)/log(2)) }' > /etc/wireguard/CNip.txt

Notes

wg-quick up process:

[#] ip link add wg-quick type wireguard
[#] wg setconf wg-quick /dev/fd/63
[#] ip address add 192.168.57.1/24 dev wg-quick
[#] ip link set mtu 1420 dev wg-quick
[#] ip link set wg-quick up
[#] resolvconf -a wg-quick -m 0 -x
[#] wg set wg-quick fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg-quick table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0

wg-quick down process:

[#] wg showconf wg-quick
[#] ip -4 rule delete table main suppress_prefixlength 0
RTNETLINK answers: Address family not supported by protocol
Dump terminated
RTNETLINK answers: Address family not supported by protocol
Dump terminated
[#] ip link delete dev wg-quick
[#] resolvconf -d wg-quick

Reference

VPN versus proxy

2018-03-19T00:00:00+00:00

VPN and proxy both can, more or less, serve as a man in the middle to target network. VPN is meant to establish private (encryption) connection to target network over the Internet. Proxy is just a traffic relay though developers may decide to implement security enhancement. Therefore, VPN can be an alternative to proxy for security concern.

Here is a simple flow illustration:

VPN: application - VPN client (encrypt traffic) - the Internet - VPN server (decrypt traffic) on target network

Proxy: application - proxy client (opt) - proxy server - target network

VPN as proxy: application - VPN client (encrypt traffic) - the Internet - VPN server (decrypt traffic) - target network

Proxy

With proxy, traffic appears to come from somewhere else hiding your IP from target network.
HTTP proxy is only designed for web traffic while SOCKS proxy is indifferent to the type of traffic (HTTP, FTP, BT etc.) that passes through it.

SOCKS is slower than HTTP as it require more overhead.
HTTP proxying has a different usage model in mind, the CONNECT method allows for forwarding TCP connections like SOCKS proxy.

HTTPS proxy requires the CONNECT method.
SOCKS works on TCP mostly. But SOCKS5 provides a means for UDP packets to be forwarded.

Security

As mentioned above, proxy by nature, does not protect traffic security.
BT over proxy would reveal your IP. Try BTGuard instead.

VPN

VPN encrypt connection to provide private communication by design.
VPN creats a virtual interface at the operating system level, and the VPN connection captures the entire network connection of the device it is configured on.

Proxy, on the other hand, needs enabled on application basis.
Encryption and decryption incur much more overhead than SOCKS proxy, let alone HTTP proxy.
UDP VPN (i.e. WireGuard) is faster than TCP VPN (OpenVPN) but less reliable.

Eng

2018-03-07T00:00:00+00:00

for ease of review, this post is moved to cloud.

A report from Gizmodo says that Google is partnering with the United States Department of Defense and building drone software.

partner with
We had to admire his prowess as an oarsman/his rowing prowess.

prowess: Someone's prowess is their great skill at doing something. (FORMAL)
A pilot scheme or a pilot project is one which is used to test an idea before deciding whether to introduce it on a larger scale.
If one group of people leapfrogs into a particular position or leapfrogs someone else, they use the achievements of another person or group in order to make advances of their own.
A patio is an area of flat blocks or concrete next to a house, where people can sit and relax or eat.
A bartender is a person who serves drinks behind a bar. (AM; in BRIT, use barman, barmaid)
ox/beef; pig/pork; lamb/sheep/mutton.

If you beef up something, you increase, strengthen, or improve it.
The oldest type of proxy server, HTTP proxies are designed expressly for web-based traffic.

An express command or order is one that is clearly and deliberately stated.
If you eavesdrop on someone, you listen secretly to what they are saying.

eavesdropper.

The government illegally eavesdropped on his telephone conversations…
A toll is a total number of deaths, accidents, or disasters that occur in a particular period of time. (JOURNALISM)

There are fears that the casualty toll may be higher.
There are a variety options to do this, and if you're a seasoned sysadmin you probably already have a favourite way.

You can use seasoned to describe a person who has a lot of experience of something. For example, a seasoned traveller is a person who has travelled a lot.
A reciprocal action or agreement involves two people or groups who do the same thing to each other or agree to help each another in a similar way. (FORMAL)

They expected a reciprocal gesture before more hostages could be freed…
Something, especially food, that is kosher is approved of or allowed by the laws of Judaism.
Someone who is agile can move quickly and easily. If you have an agile mind, you think quickly and intelligently.
In finance, a portfolio is the combination of shares or other investments that a particular person or company has.

A case for carrying papers or drawings or maps; usually leather.
It is more than a piece of technology and orchestration.
The topography of a particular area is its physical shape, including its hills, valleys, and rivers.

The geography of a place is the way that features such as rivers, mountains, towns, or streets are arranged within it.

It is important to remember that topographically closer does not inherently mean geographically closer, though this is often the case.
The penultimate thing in a series of things is the last but one. (FORMAL)
If someone or something aggravates a situation, they make it worse.

If someone or something aggravates you, they make you annoyed. (INFORMAL)
The mileage in a particular course of action is its usefulness in getting you what you want.
[find | consider | see] [it | him] [to be | [as] being] [ very | rather] [appropriate | amusing | strange inappropriate]

to be and being means different context.
reverse course; impose tariffs on; in response to; rescind sanctions against;
If a government or a group of people in power rescind a law or agreement, they officially withdraw it and state that it is no longer valid. (FORMAL)
A slew of things is a large number of them. (mainly AM)
If you wield a weapon, tool, or piece of equipment, you carry and use it.

If someone wields power, they have it and are able to use it.
If a group of people or things fan out, they move forwards away from a particular point in different directions.
Some concerns have been raised that …
ledger, tally, bookkeeping/bookkeeper
to = (in order) to = (so as) to = so that;
If a boyfriend or girlfriend stands you up, they fail to keep an arrangement to meet you. (INFORMAL)

stand sb up; no show no call;
A contingency is something that might happen in the future (= possibility, eventuality).

A contingency measure/plan is one that is intended to be used if a possible situation actually occurs.

in case of contingency

Windows

2018-01-01T00:00:00+00:00

Prevent Win10 1803 auto-restart
Security
Drivers
Windows Embedded Standard 7
1. Add more features
Activation
1. MAK
2. Key Management Server (KMS)

Prevent Win10 1803 auto-restart

Run gpedit.msc as administrator;
Computer Configuration - Administration Template - Windows Update
Configure Automatic Updates - Enabled

Choose '4 - Auto download and schedule the install'
No auto-restart with logged on users for shceduled automatic updates installations - Enabled

Security

WannaCry / Wcry / WannaCrypt

Turn off SMB v1 and optionally v2 and/or v3. Then check by:

netstat -na | find "LISTENING" | find ":445 "

Especially, Windows XP SP3 requires patch Windows XP SP3 安全更新程序 (KB4012598).
Anything related to Intel Management Engine (IME) or Automatic Management Technology (AMT) should be avoid.
Turn off Windows Defender (scan disk) with gpedit.msc.

Drivers

Unlike Widnows 8 onwards, Windows 7 ISO lack drivers (i.e. Ethernet, Wireless etc.). Upon installation, it probably cannot connect the Internet.

Windows Embedded Standard 7

Installation tips:

Use a Template - Thin Client.
Modify Drivers & Modify Features.
Automatically detect devices.
Untick 'Resolve optional dependencies' otherwise final OS Footprint would be 3.4G around. Leave 'Resolve Dependencies' to the last step of package selection, which otherwise would pulls uncessary leftover.
.Net Framework, tick everything. Miscellaneous applications depend on .NET framework.
Application Support, untick 'MSMQ'.
Boot Environments, 'Enhanced write filter boot Environment' to 'Windows Boot Environment'.
Browsers, keep IE 8.
Data Access and Storage, untick 'Windows Data Access Components - SQL'.
Data Integrity, leave everything unticked.
Devices and Printers, untick 'Printing Utilities and Management'.
Diagnostics, leave 'Common Diagnostics Tools' and optionally 'User' ticked (Windows Task Manager).
Embedded Enabling Features, untick everything and optionally tick 'Registry Filter' (for regedit).
Fonts, tick 'Simplified Chinese Fonts'.
Graphics and Multimedia, untick 'Windows Media Player 12'.
International, leave everything unticked.
Internet Information Service - IIS, leave everything unticked.
Management, leave it alone but optionally untick 'Windows Update User Interface'.
MediaCenter, leave everything unticked.
Networking, untick everything and tick 'Network and Sharing Center', and optionally 'Wireless Networking'. With regards to 'Wireless Networking' part, VirtualBox does not support wirless adapters (only simulates Ethernet), which means you cannot add or see wirless adapter in Windows Device Manager. However, you can attach a USB Wi-Fi stick to guest OS and connect directly to Wi-Fi router.
Remote Connections, untick 'Remote Desktop Connection'.
Security, untick everything.
System Services, leave it alone.
User Interface, untick 'Help', 'Microsoft Speech API', and 'Accessibility'.
Resolve Dependencies. Choose 'Unbranded Startup Screens', 'Windows Boot Environment', 'Standard Windows USB Stack' and 'Windows Explorer Shell' (a MSUT). Some previously unticked feature might be ticked again as a dependency of some other features.

Add more features

Suppose you have unticked an feature (i.e. IE 8), and want to add it back after installation, do the following:

Find the feature package cabinet file from ISO, like DS/Packages/FeaturePack/x86~winemb-ie-explorer~~~~6.1.7600.16385~1.0/WinEmb-IE-Explorer.cab.
Use Pkgmr or DISM command line to install the cabinet file. dism is far more powerful than what you think. For example, you can use it to check a cabinet file information.

# Do it in an elevated command prompt.
DISM /?
Pkgmgr /?
DISM /Online /Add-Package /PackagePath:"C:\Users\Brink\Desktop\WinEmb-IE-Explorer.cab"
Pkgmgr /ip /m:C:\Users\Brink\Desktop\WinEmb-IE-Explorer.cab
# restart system as requested

Activation

MAK

Windows XP Professional VOL SP3 x86

MRX3F-47B9T-2487J-KWKMF-RPWBY
Windows Emebedded 7 Standard x86

XGY72-BRBBT-FF8MH-2GG8H-W7KCW MPMVY-PP762-WWVBC-83RXJ-2H7RH GJVTR-C4WQ6-BKRH3-DRFFH-J83DM

Key Management Server (KMS)

KMS activation is not permanent.

A Microsoft Key Management Server (KMS) is a legitimate service offered under Microsoft Volume Activation 2.0 solution which is used to activate volume licensed Microsoft products. It works with minimal administration intervention and allows automated activation of Microsoft products (i.e. Windows OS, Office, Visio etc.).

With KMS activation method, the client (Microsft products) contacts verification server periodically to renew license. Each activation holds the license for 180 days. I will not go into details on KMS mechanism.

Here, I will set up a self-hosted KMS server :) based on FOSS reverse engineering code. Before that, please review a few notes on KMS:

KMS activates only Volume (VOL) licensing Microsoft products.

'Windows Enterprise' is certainly VOL licensed while 'Windows Professionl' use either VOL license or retail license. Office versions downloaded from Microsoft MSDN and/or Technet are non-VOL. It's recommended to download VOL licensing editions from itellyou.cn.

MAK activation method (one key forever) makes life easier on the premise that you succeed in finding a MAK key.
Where to put the KMS server side?

Intuitively, we can deploy it on Windows system to be activated. However, local server may be compromised by Windows updates or anti-virus software. Beginning with Windows 8.1 the KMS server must be a different computer than the client. You cannot use vlmcsd on the same computer where you want to activate a product. If you have only one computer, you can run vlmcsd in a virtual machine. It's better to put the server side on a alway-on devices like VPS.

KMS emulator

Among the other things, vlmcsd in C and py-kms in Python2 are the most popular KMS emulators. They can run on almost any platforms like Windows, Cygwin, Linux, Android, OpenWrt, Unix, BSD etc. Here, I choose vlmcsd on CentOS 7 as an example.

To set up KMS emulator on Windows system, you'd better use py-kms.

Firstly, build binaries from source.

root@tux / # cd /opt/
root@tux opt # git clone --depth=1 https://github.com/Wind4/vlmcsd.git
root@tux opt # cd vlmcsd
root@tux opt # man man/vlmcsd.7
root@tux opt # make
root@tux opt # chown -R nobody: /opt/vlmcsd
root@tux opt # cd bin
root@tux opt # vlmcsd -h
root@tux opt # vlmcs -h

The author also provides pre-compiled binaries on Github relase page.

Launch KMS server
```
root@tux / # /opt/vlmcsd/bin/vlmcsd -g nobody -u nobody -o3 -m1 -t15 -d -lsyslog -Dev
root@tux / # /opt/vlmcsd/bin/vlmcsd -i /opt/vlmcsd/etc/vlmcsd.ini
   
root@tux / # journalctl -xeft vlmcsd
```
The options -D and -e are useful for terminal debugging.

We can load configuration file by the option -i and run vlmcsd in background, such that we can send the "HUP" signal to vlmcsd process without restarting.
```
# man man/vlmcsd.ini.5

User = nobody
Group = nobody
PublicIPProtectionLevel = 3
MaxWorkers = 1
ConnectionTimeout = 15
DisconnectClientsImmediately = TRUE
LogFile = syslog
LogVerbose = TRUE
```
Command line options take precedence over the respective configuration line in the .ini file. For example, -k (do not disconnect clients) on command line overrides 'DisconnectClientsImmediately = TRUE'.

About configuration options, please check etc/vlmcsd.ini.
Firewalld

Make sure the relevant port are accessible from outside.
Test KMS server locally
```
user@tux ~ # journalctl -xeft vlmcsd

user@tux ~ # /opt/vlmcsd/bin/vlmcs 127.0.0.1:1688 -v -e  # print examples
user@tux ~ # /opt/vlmcsd/bin/vlmcs -v -x        # print supported Windows and Office versions
user@tux ~ # /opt/vlmcsd/bin/vlmcs -v -l 35     # activate Windows 7 Enterprise
   
user@tux ~ # /opt/vlmcsd/bin/vlmcs vlmcsd.example.com:1234 -v -l 35     # check if firewall allows public access
```
1. vlmcs tests against a KMS server that can be a vlmcsd emulator or a real Microsoft KMS.
2. A Microsoft KMS sends correct activation messages only if it detects a certain minimum of clients (25 for Windows client OSses, 5 otherwise) on the network. This is Microsoft's futile attempt to prevent running a KMS server in a home environment. Use the -n argument to charges a KMS server. The vlmcsd emulator is always fully charged, so I set MaxWorkers to 1. If -n is larger than 1, vlmcsd will report RPC error.
3. The above vlmcs client runs in the same host as vlmcsd. Repeat the test from your PC to see what happens.

Systemd unit (/etc/systemd/system/vlmcsd.service)

[Unit]
Description=KMS emulator
After=network.target

[Service]
Type=forking
PermissionsStartOnly=true
LimitNOFILE=4096
   
ExecStart=/opt/vlmcsd/bin/vlmcsd -i /opt/vlmcsd/etc/vlmcsd.ini

[Install]
WantedBy=multi-user.target

There is no point in leaving this service always online as long as you remember to activate the product every 180 days.

GVLK

KMS recognizes product type by General Volume License Key (GVLK). If you accidentally ignore GVLK during installation or entered the some other keys (i.e. invalid MAK key) afterwards, please restore Windows GVLKs (a.k.a KMS Client Setup Key), Office GVLKs. Open an elevated command prompt on the client Windows:

For Windows:

slmgr
slmgr /ipk <GVLK>

For Office:

cd C:\Program Files\Microsoft Office\Office16
cscript ospp.vbs
cscript ospp.vbs /inpkey:<GVLK>

Choose an approprate GVLK in accord with reference link above. Do *not fill GVLK in product itself. Always use the command line!

slmgr is for Windows OS and usually resides in the system32 directory. ospp.vbs is for Office 2010/2013/2016. To use it, we should first change the current directory Office's installation. slmgr.vbs and cscript ospp.vbs without parameters print help message.

You'll have to install a volume license (VL) version of Office. Office versions downloaded from MSDN and/or Technet are non-VL. If you happened to install a retail licensing product (i.e. OEM, Home, and Ultimate Windows), then in all likelihood KMS is out of your luck. To check Windows version on command line:

wmic os get caption

There exist some inofficial GVLKs for retail licensing Windows, which can be found in vlmcsd.7 man page. Hence, why not attempt to convert Windows to a KMS client as stated above? The only difference is that those inofficial GVLKs hold each activation for 45 or 30 days.

About Office, if you happened to install a retail version (i.e. Office 16 Pro Plus retail, Visio included), there exists a script to help you convert the retail version to VOL version. If you have tried to input different MAK keys before the conversion or the product brings in a default MAK key, you should first remove them before KMS activation.

cscript ospp.vbs /dstatus
cscript ospp.vbs /unpkey:<last-5-digits>

You are highly recommended to install and activate Visio 2016 Pro before other office components. inpkey automatically assigns the keys to Office or Visio.

Configure a client

For Windows OS, skms means set KMS:

slmgr.vbs /skms <kms-server[:tcp-port]>

For Office, it is optional if Windows OS already sets that.

cd C:\Program Files\Microsoft Office\Office16
# -or-
cd C:/Program Files(x86)/Microsoft Office/Office16
#
cscript ospp.vbs /sethst:<kms-server>
cscript ospp.vbs /setprt:<tcp-port>

Activate a product

After telling a client the KMS address, Windows contacts it for activation on demands. To activate Windows system immediately, right click Computer, select Properties and activate there. Alternatively, do it on command line:

slmgr.vbs /ato
slmgr.vbs /xpr
slmgr.vbs /dlv

Similarly for Office:

cscript ospp.vbs /act
cscript ospp.vbs /dstatus

References

Linux as Gateway Router

2017-12-29T00:00:00+00:00

Principal
Set Device Gateway
Set the Box
Reference

Principal

Suppose a Linux box such as a personal computer has two networking cards (i.e. eth0 and eth1). One card (i.e. eth0) connects to the Internet by whatever means like PPoE dial-up, ADSL, static IP etc. The other one (i.e. eth1) stays within a LAN block.

Nowadays, it's common that a networking card supports two inferfaces.

The Linux box can serve as a gateway/router for other devices (i.e. virtual machines using NAT mode) LAN with Iptables support. To achieve this functionality:

Enable packet forwarding.
Iptables FORWARD support.
Iptables NAT support.

Actually, what this post does is to set up the routing functionality on Linux boxes, simulating a router.

A layer 3 router maintains a table mapping IPs to ports while a layer 2 switch maintains a table mapping mac addresses to ports. However, a switch learns mac address and port pair by broadcasting packets at first while a router usually needs manual configuration. After a table entry is learned, the switch formards upcoming packet the exact port instead of broadcasting.
A Wi-Fi router comprises of two parts, namely a wireless access point and a routing table. The access point handles traffic among WLAN devices while the routing table fowards traffic in and out.

IP Forwarding at Layer 3

To server as a gateway, the Linux box must forward a packet not destined for local interfaces (i.e. neither eth0 nor eth1). In this post, such packets are named as "foreign.

If the box receives a packet with destination address not configured locally, it could either drop or forward it. It is intuitive to drop a strange packet. Alternatively, the box (i.e. router) may choose to forward the packet to some interface letting the packet traverse it as a router usually resides across between multiple network blocks.

forwarding is a synonym for routing meaning forwarding a foreign packet according to routing tables similar to routing locally generated packets. In other words, packet forwarding is routing tables' business and has nothing to do with Iptables.

Read more at IP Forwarding = when and why is this required?. It is basically for inter-network (except for VLAN router) packet transmission and not for that of inter-interface. If two interfaces are on the same network then IP forwarding is not involved (rare case though). Additionally, if one is virtual interface, IP forwarding is not needed either.

For example, traffic between two devices on the same network does not require forwarding/routing at layer 3. Instead, it is handled at layer 2 by switches, access points of Wi-Fi router, or direct Ethernet wire.

The box should enable net.ipv4.ip_forward.

Iptables FORWARD

Once forwarded, the foreign packet will be examined by FORWARD chain of Iptables. Please be noted, keyword FORWARD is not meant to forward a packet but apply Iptables rules against the forwarded packets from routing tables.

The box should allow forwarded packets to go outside by -j ACCEPT.

Iptables NAT

Before a foreign packet is forwarded outside, the box should also modify the source IP to that of the outgoing interface (i.e. eth0), namely to set up NAT by Iptables.

After examined by Iptables FORWARD, the source IP remains of the original one (that of the foreign device). If it is left alone, the destination device would attempt to send back packets directly to the foreign device without intervention of the box.

Iptables NAT support will maintain an kernel mapping between the source IP and that of outgoing interface, such that destination device can reply to the box instead.

-j MASQUERADE or -j SNAT should be enabled.

Set Device Gateway

Set LAN devices' gateway to IP of eth1 on the box.

For Linux device, this can be done with ip route or route command line. Say LAN block is 192.168.10.0/24 while 192.168.10.1 is IP of eth1 of the box.

root@tux / # ip route add default 192.168.10.0/24 via 192.168.10.1 dev eth1
# -or-
root@tux / # route add 192.168.10.0 netmask 255.255.255.0 gw 192.168.10.1 dev eth1

It is not unusual that home devices' (i.e. smartphone) default gateway is set to 192.168.0.1 (IP of Wi-Fi router).

Set the Box

Enable IP forwarding.

It simply determines behaviour when receiving a packet for another network. Without forwarding, the packet is dropped. With forwarding, routing tables are consulted to determine to which interface it should be forwarded.

Permanent across reboots:
```
root@tux / # echo 'net.ipv4.ip_forward=1' > /etc/sysctl.d/20-ip_forward.conf
```
For current session:
```
root@tux / # sysctl -w net.ipv4.ip_forward=1
# -or-
root@tux / # sysctl -p /etc/sysctl.d/20-ip_forward.conf
# Check
root@tux / # cat /proc/sys/net/ipv4/ip_forward
```
Enable Iptables FORWARD

Please be noted locally generated packets are not examined by FORWARD chains.
```
root@tux / # iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT
```
This FORWARD rule lets forwarded packets go outside.
Enable Iptables NAT

Turn on MASQUERADE NAT on the box. Generally, MASQUERADE is required when many LAN private IPs' traffic sits behind and goes through a public single IP (i.e. Wi-Fi router), namely a many-to-one relationship.

Needless to say, Iptables NAT should be accomplished by -t nat table.
```
root@tux / # iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE (poor NAT)
# or
root@tux / # iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source <IP of eth0> (better NAT)
```
Read more on Wireshark post.

Reference

Wireshark

2017-12-12T00:00:00+00:00

Installation
Permission
Tutorials
Android

Installation

root@tux / # echo 'net-analyzer/wireshark androiddump' >> /etc/portage/package.use/wireshark
root@tux / # emerge -avt wireshark

Permission

Running Wireshark in root account is dangerous. We can add normal user account into wireshark group.

root@tux / # gpasswd -a username wireshark

You should log out and log in back to take effect. newgrp log in to a new group in current shell:

user@tux ~ $ newgrp wireshark
user@tux ~ $ wireshark-gtk

Then launch Wireshark from terminal directly.

Tutorials

to-dos

Android

It's recommended to install terminal emulator app such as Termux.

androiddump

androiddump as explained by the term name, is used to capture Android system logcat messages and networking flow. Before the capture carried out, you should provide the following:

Android SDK tools (i.e. adb) are added to PATH variable (for logcat capture);
Android tcpdump on a rooted device (for networking flow). Device vendor may have enclosed tcpdump in Android image.

tcpdump captures Android traffic and then pass it to Wireshark on your PC. androiddump serves as the bridge between tcpdump and Wireshark.
Obviously, your device should be in Developer Debugging mode and recognized by SDK tools.

tcpdump requires root permission (rooted device), exposing to maleware attack.

NAT gateway

Set PC as gateway router to your device. Then use Wireshark to capture PC Wi-Fi interface. Without explicit pointing out, I use router and gateway interchangeably in this section.

Here is my case. Linux and Android connect to the same Wi-Fi as follows:

Device	IP
Linux	192.168.0.150
Android	192.168.0.151
Router	192.168.0.1
WAN	dial-up

By default Linux and Android uses gateway 192.168.0.1. We can check route by:

user@tux ~ $ route -n
user@tux ~ $ ip route

Firstly, we set Android Wi-Fi gateway to Linux's IP 192.168.0.150. Do this in Android Wi-Fi setting. Thus all Android Wi-Fi data routes through Linux.

Then on Linux, we should configure NAT for Android traffic, forwarding Android traffic to the real Wi-Fi gateway 192.168.0.1.

root@tux / # sysctl -w net.ipv4.ip_forward=1
root@tux / # iptables -A FORWARD -i wlan0 -j ACCEPT
root@tux / # iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE (poor NAT)
# or
root@tux / # iptables -t nat -A POSTROUTING -o wlan0 -j SNAT --to-source 192.168.0.150 (better NAT)

NAT requires IP forwarding that, by default, is disabled in Linux boxes.
MASQUERADE replaces the sender's (Android device 192.168.0.151 here) address by the router's (Linux PC 192.168.0.150 here) address.

As the router here is Linux PC, we'd better use SNAT istead. Bascially, we use SNAT for static router IP, improving Iptables performance. However, MASQUERADE is a must when the router connects to the Internet with PPoE dial-up (different WAN IP each dial-up).
This is a special case of the reference as there is only one Linux interface (wlan0) for both LAN and WAN communication. Please read more details via LinuxTutorialIptablesNetworkGateway.

We can omit the MASQUERADE or SNAT part because Wi-Fi router can reach Android device directly without Linux wlan0 NAT support.

To check the setup, make sure Android connect to network (ping in Termux). Further more, we can check Iptables counts:

root@tux / # iptables -L -nv [-t nat]

Android outgoing traffic flow is as follows:

Android -> Router NAT -> Linux NAT -> Router NAT -> The WAN Internet

Router NAT handles LAN communication while Linux NAT handles Android traffic mapping. For incoming traffic, just reverse the arrow.

Finally, Wireshark can capture Android traffic on PC. We can print out only Android traffic by filtering the Android IP:

ip.addr == 192.168.0.151

If you choose to omit the MASQUERADE or SNAT part, Wireshark can only caputre traffic from Android while miss traffic to it. Meanwhile, the reverse traffic flow resembles the usual:

the WAN Internet -> Router NAT -> Android

Proxy

Set up a proxy (i.e. OpenVPN, Privoxy HTTP etc.) on PC and proxy device traffic through.

Alternatives

tPacketCapture app based on Android VPN service.
AndroidHttpCapture app for HTTP(s) capture.
We can use Charles and Fiddler for HTTP(s) capture.

Solutions based on VPN service can capture transmission layer packets. Most tools above only capture HTTP(s) layer traffic.

Above all, the most convenient method is running tshark (Wireshark terminal counterpart) on Wi-Fi router (i.e. OpenWrt).

archlinux

2017-11-28T00:00:00+00:00

Installation guide
1. USB Stick
2. USB Booting
3. Time and Network
4. BIOS GPT scheme
  1. Partitioning
    1. Swap
  2. Filesystem
  3. Mount
5. UEFI GPT scheme
  1. Disk Preparation
  2. Erase Disk
  3. GPT Table
  4. Partitioning
  5. ESP Filesystem
  6. LUKS
  7. LVM
  8. Filesystem
  9. Mounting
6. Mirrors
7. base group
8. fstab
9. Copy Keyfile
10. Chroot
11. Time
12. Localization
13. Network
14. Initramfs by mkinitcpio
15. root password
16. Boot loader - Grub
17. Reboot
General Settings
X
Time
pacman
1. AUR
Xterm
1. Copy and Paste
Alacritty
Resolution
1. Virtual terminal
2. Xorg
Fcitx
WPS
ALSA

Installation guide

Arch Linux has dropped support for i686 platforms. Only x86_64 now!
Download the latest ISO image from download link.
Install necessary packages. At least wpa_supplicant as the base group does not include that.

USB Stick

dd command will be used to create the bootable stick as it is more reliable and also the recommended method on the Wiki page. dd erases the whole USB stick and treats it as a hard drive.

Without explict notice, we assume the USB drive to be /dev/sdb. dd command is effective but dangerous as well. It take effects immediately without confirmation prompt. Therefore, please make sure you select the exact of= argument.

root@archiso / # fdisk -l; blkid; lsblk -f; findmnt
root@archiso / # dd bs=1M if=/path/to/archlinux.iso of=/dev/sdb status=progress oflag=sync

The optimal block size bs argument is determined by various factors (both hardware and software) on the system. Empirically, 1M or 4M is good enough.

To restore the USB stick as normal data storage, we should remove the ISO 9660 filesystem signature before re-partitioning and re-formating the USB drive by wipefs command. It only erase signatures of filesystem, raid or partition-table while leave filesystem itself untouched.

root@archiso / # wipefs --all /dev/sdb

USB Booting

Disable secure boot in BIOS setting as most Linux distributions do not buy digital certifits from Verisign.

If the USB stick refuses to load, then try the dd command once more. If directory /sys/firmware/efi/efivars exists, then it's booted in UEFI mode, otherwise it is in legacy BIOS mode.

By default, Arch ISO uses Zsh shell. To log all commands, set in ~/.zshrc the following to empty string:

HISTSIZE=-1
HISTFILESIZE=-1

And then run source .zshrc.

Within the Live system, we use Alt-arrow or Ctrl-Alt-Fx to switch virtual console and read installation guide there.

Time and Network

Once booted into the live system, check network connection and rectify system clock:

root@archiso ~ # ping www.archlinux.org
root@archiso ~ # timedatectl set-ntp true; timedatectl set-timezone Asia/Shanghai; timedatectl status

For wired/wireless network that require captive login or user agreement, refer to Using command line to connect to a wireless network with an http login and Connecting to a Wireless network with a captive portal. Also, please check network.

BIOS GPT scheme

Grub bootloader.
BIOS/GPT partitioning.
Single root partition plus extra swap file.

This booting schema is only used with VMs in VirtualBox.

Partitioning

Firstly, to identify disk devices:

root@archiso ~ # fdisk -l
root@archiso ~ # lsblk

For "Grub, BIOS/GPT" scheme, we must create the BIOS boot partition to hold Grub core.img. Around 1 MiB (2048 sectors) is enough. Partition number can be in any position order but has to be on the first 2 TiB of the disk. This partition should be flagged as bios_grub for parted, ef02 for gdisk, or select BIOS boot and partition type 4 for fdisk.

On the other hand, "Grub, BIOS/MBR" scheme uses post-MBR gap (after the 512B MBR and before the first partition) to store core.img. Usually this gap is 31 KiB. For complex modules in core.img, this space is limited. That can be resolved by pushing forward starting sector of the first parititon. For instance, partitioning tool can align at MiB boundaries, thus leaving enough post-MBR gap. So, it eliminates the bother to create BIOS boot partition.

Here is the partitioning steps:

root@archiso ~ # parted -a optimal /dev/sda
(parted) help
(parted) mktable gpt
(parted) unit s
(parted) print free
(parted) mkpart primary 0% 2047s        # 1MiB 2047s
(parted) set 1 bios_grub on
(parted) mkpart primary 2048s 100%
(parted) print free
(parted) quit

Tell parted to optimally align at 1MiB boundaries.
By default, GPT's free sector starts at 34s (equally 0%). Make BIOS boot partition be the very first parition which starts at 34s spanning to 2047s. parted reminds:

Warning: The resulting partition is not properly aligned for best performance. Ignore/Cancel?

Choose Ignore as this partition will not be regularly accessed. Performance issues can be disregarded though this is out of GPT alignment specifications.
The rest space is assigned to root starting at 2048s.

Swap

There is not any other separate partitions except that a swap file will be created. Swap partition bears NO advantages over swap file but could be shared among different systems. On the other hand, we can easily resize and plug/unplug a swap file on-the-fly.

Leave the swap file part after booting into new system.

Filesystem

There is no need to create file system for the partition bios_grub. Hence, just create an ext4 on the single root parition:

root@archiso ~ # mkfs.ext4 /dev/sda2

Mount

root@archiso ~ # mount /dev/sda2 /mnt

As there is only the root parition, we would not create any other mount points under /mnt, neither /home nor /boot.

UEFI GPT scheme

Almost all modern systems take UEFI booting schema. The old BIOS legacy mode discussed above is almost deprecated. To set the PC booting in only UEFI mode in BIOS setting (disable legacy BIOS). Also make sure the ISO is booted in UEFI mode.

Disk Preparation

If LVM, system encrytion (LUKS), or RAID is desired, do it now.
ESP could be any of FAT12, FAT16, FAT32 and VFAT (preferred).

We will erase partition or the whole drive before partitioning and formating.

Erase Disk

This section only covers normal mechanical disk drives. For SSDs, it is a different story.

We want to securely erase the disk by overwritting the entire drive with random data. There are a host of general methods available. But I choose the dm-crypt specific method here. Optionally, you could either erase just a single partition (i.e. /dev/sdXY) or the complete drive (i.e. /dev/sdX).

We first create a temporary dm-crypt container:

# erase a partition
root@archiso ~ # cryptsetup open /dev/sda1 --type plain --key-file /dev/urandom to_be_wiped
# erase a disk
root@archiso ~ # cryptsetup open /dev/sda --type plain --key-file /dev/urandom to_be_wiped

--key-file /dev/urandom is used as the encryption key. The container device is located at /dev/mapper/to_be_wipded. We can verify its existence by lsblk.

Next, we fill the container with zeros without the need of /dev/urandom:

root@archiso ~ # dd bs=1M if=/dev/zero of=/dev/mapper/to_be_wiped status=progress

For a large (i.e. a multi-terabyte disk) drive or partition, this may take hours or even a day.

Finally, close the temporary container:

root@archiso ~ # cryptsetup close /dev/mapper/to_be_wipded
root@archiso ~ # lsblk

GPT Table

Use parted to create GPT table:

root@archiso ~ # parted -a opt /dev/sda
(parted) mklabel gpt

Partitioning

For future compatability and scability, create an extra BIOS boot partition (for Grub). Additionally, I want the /boot partition also encrypted. This setup really makes the system complicated. Anyway, the practice and security improvement deserve it!

Here is the partitioning steps:

root@archiso ~ # parted -a opt /dev/sda
(parted) unit s/MiB
(parted) p free
(parted) mkpart primary 30s/40s 2047s
(parted) toggle 1 bios_grub
(parted) mkpart primary fat32 2048s/1MiB 551MiB
(parted) toggle 2 esp/boot
(parted) mkpart primary 551MiB 751MiB
(parted) mkpart primary 751MiB 100%
(parted) print free

As the Grub BIOS boot partition is not regularly accessed, there is no need to follow rigid alignment rules. It can just start at 34s (no alignment) or 40s (minimal alignment).

GRUB embeds its core.img into this partition
It is recommended. to create the ESP with 550MiB.
When the unit used if MiB, the end of part 2 and the start of part 3 are both 551MiB. If using s unit, their relation should be +1.

Up to now, the partitions layout looks like:

/dev/sda1: BIOS boot
/dev/sda2: ESP
/dev/sda3: boot LUKS
/dev/sda4: '/', '/home', and '/swap' LUKS

ESP Filesystem

root@archiso ~ # mkfs.vfat -F32 /dev/sda2
root@archiso ~ # parted /dev/sda p free ; fdisk -l /dev/sda

BIOS boot partition shall not be formated or mounted. Just leave it alone.
Both sda3 and sda4 will be encrypted by LUKS, so there is no need to create filesystem at this stage.

LUKS

As Grub does not support LUKS2, use LUSK1 (--type luks1) on /dev/sda3 for /boot.

Firstly, we create a passphrase-protected file by gpg (namely arch-luks.gpg) that will be used to to encrypt the LUKS container.

root@archiso ~ # export GPG_TTY=$(tty)
root@archiso ~ # dd if=/dev/urandom bs=8388607 count=1 | gpg -v --symmetric --cipher-algo AES256 --armor --output ~/arch-luks.gpg

Why bs=8388607 count=1? Because the maximum key file size is 8192KiB. From the empirics of Gentoo installation, we should decrease the maximum value by 1.

Before anything else, make a backup of the key file (i.e. to a USB stick), as otherwise the Live environment would lose it upon reboot.

Next, we create the LUKS container:

root@archiso ~ # gpg --decrypt ~/arch-luks.gpg | cryptsetup luksFormat -v --type luks1 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --key-file - --use-random /dev/sda3
root@archiso ~ # gpg --decrypt ~/arch-luks.gpg | cryptsetup luksFormat -v --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --key-file - --use-random /dev/sda4

luksFormat does not format the device, but sets up the header part of the LUKS container. Specially it encrypts the master key and embeds it into the header.

Master key is randomly choosen upon container creation and, encrypted and embedded in the header part.

It is the master key's job to encrypt the data on partition. Key file or passphrase is only used to encrypt/decrypt the master key.

For safety, add a fallback passphrase in addition to the key file. It always asks for an existing passphrase or key file before adding a new one.

root@archiso ~ # cryptsetup luksAddKey -v --key-file arch-luks --iter-time 5000 /dev/sda3 --key-slot 0
root@archiso ~ # cryptsetup luksAddKey -v --key-file arch-luks --iter-time 5000 /dev/sda4 --key-slot 0

Apart from luksAddKey, we can also luksRemoveKey, luksKillSlot etc.

We can use luksDump command to check the LUKS header:

root@archiso ~ # cryptsetup luksDump /dev/sda3
root@archiso ~ # cryptsetup luksDump /dev/sda4

It's better to backup the LUKS header:

root@archiso ~ # cryptsetup luksHeaderBackup /dev/sda3 --header-backup-file ~/sda3-luks-header.img
root@archiso ~ # cryptsetup luksHeaderBackup /dev/sda4 --header-backup-file ~/sda4-luks-header.img

Of course, we have a counterpart command luksHeaderRestore to restore header file. Once backuped, the the LUKS header may be deleted from the container. Details, refer to Arch wiki.

Once created, it is time to open the containers. The first open command, uses the fallback passphrase while the second uses key file.

root@archiso ~ # cryptsetup open /dev/sda3 cryptboot
root@archiso ~ # cryptsetup --key-file ~/arch-luks open /dev/sda4 cryptlvm
root@archiso ~ # lsblk
root@archiso ~ # ls -al /dev/mapper/

The open command asks for key file or passphrase to decrypt the master key in the LUKS header. The decrypted master key is then used to decrypte the LUKS container. The decrypted containers are now available at /dev/mapper/.

LVM

LUKS containers are ready for LVM containers on which we then create Arch Linux filesystem.

At this stage, we only care about /dev/mapper/cryptlvm as there is no LVM requirement on /dev/mapper/cryptboot.

root@archiso ~ # pvcreate /dev/mapper/cryptlvm ; pvdisplay -v -m
root@archiso ~ # vgcreate volgrp /dev/mapper/cryptlvm ; vgdisplay -v
root@archiso ~ # lvcreate --size 16G volgrp --name swap
root@archiso ~ # lvcreate --size 60G volgrp --name root
root@archiso ~ # lvcreate --size 250G volgrp --name home ; lvdisplay -v -m
root@archiso ~ # ls -al /dev/{mapper,volgrp}

Create a physical volume /dev/mapper/cryptlvm on top of the LUKS container.
Create a volume group volgrp and add the physical volume into it. We can add more, if multiple physical volumes exist.
Create logical volumes within volgrp.

For futher compatibility, leave some volgrp space unallocated. Size of logical volumes can be ajusted on the fly.

Created logical volmes' symbolic links reside in two equivalent locations, namely /dev/mapper/ and /dev/volgrp/.

Filesystem

Format the logical volumes.

root@archiso ~ # mkswap /dev/volgrp/swap
root@archiso ~ # mkfs.ext4 /dev/volgrp/root
root@archiso ~ # mkfs.ext4 /dev/volgrp/home

Recall that, we have not create filesystem for the decrypted cryptboot yet. Just create filesystem on top LUKS container directly without LVM.

root@archiso ~ # mkfs.ext2 /dev/mapper/cryptboot
root@archiso ~ # lsblk

Mounting

root@archiso ~ # blkid
root@archiso ~ # swapon /dev/volgrp/swap
root@archiso ~ # mount /dev/volgrp/root /mnt
root@archiso ~ # mkdir /mnt/home; mount /dev/volgrp/home /mnt/home
#
root@archiso ~ # mkdir /mnt/boot; mount /dev/mapper/cryptboot /mnt/boot
#
root@archiso ~ # mkdir /mnt/efi; mount /dev/sda2 /mnt/efi
#
root@archiso ~ # lsblk

Before mounting, we should use blkid to check relevant partition and filesystem are correctly prepared.
LUKS boot partition can be mounted directly as we have already create the filesystem on it.

Mirrors

Get a state-of-the-art copy from mirrorlist generator:

curl -vo mirrorlist https://www.archlinux.org/mirrorlist/?country=all&protocol=http&protocol=https&ip_version=4&ip_version=6

Edit /etc/pacman.d/mirrorlist and place geographically closest mirrors on top.

base group

# base
root@archiso ~ # pacstrap /mnt base base-devel
# optional
root@archiso ~ # pacstrap /mnt nano

This will install all packages from base and base-devel to the root partition. Around 200 MiB packages will be downloaded and 700 MiB disk space consumed. Ignore the warning on locale failure that will be handled after chroot.

base and base-devl are meta packages. Check Meta package and package group.
The base group does not even has a text editor. So we install nano explicitly here. Other groups or individual packages can also be appended to the pacstrap command. However, we'd better install only necessary packages at this stage. Of course, we can install packages within chroot by pacman or when the OS is completely installed.

fstab

root@archiso ~ # genfstab -U /mnt >> /mnt/etc/fstab
root@archiso ~ # cat /mnt/etc/fstab

-U and -L use UUID and label respectively.

Attention that, the separate cryptboot container is also included in /etc/fstab. To support automount of /boot partition after booting, we can resort to /mnt/etc/crypttab:

# /mnt/etc/crypttab
cryptboot	/dev/sda2	/efi/arch-luks	cipher=aes-xts-plain64:sha512,size=512

Copy Keyfile

Before chrooting, copy the key file to ESP partition.

root@archiso ~ # cp ~/arch-luks.gpg /mnt/efi/

Chroot

root@archiso ~ # arch-chroot /mnt

Much simpler than that of Gentoo. The shell prompt becomes [root@archiso /#] and the default shell becomes Bash.

To log all commands, set the following in ~/.bashrc empty string or negative integer:

# ~/.bashrc
HISTSIZE=-1
HISTFILESIZE=-1

Time

[root@archiso /#] ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# On VirtualBox
[root@archiso /#] hwclock -w/--systohc

Localization

Mainly, we will modify two files /etc/locale.gen and /etc/locale.conf.

Uncomment the following lines from /etc/locale.gen:

en_US.UTF-8 UTF-8
zh_CN.UTF-8 UTF-8
zh_CN.GB18030 GB18030

Generate the locale:

[root@archiso / #] grep '^[^#]'/etc/locale.gen

[root@archiso / #] locale-gen
[root@archiso / #] locale -a

[root@archiso / #] echo LANG=en_US.UTF-8 >> /etc/locale.conf

Here we set system-wide locale to en_US.UTF-8. For per-user locale, leave it to ~/.config/locale.conf.

To set customized keymap permanently, edit (create) /etc/vconsole.conf:

[root@archiso / #] echo 'KEYMAP=emacs' >> /etc/vconsole.conf

Check /usr/share/keymaps/i386/qwerty for predefined keymaps. Switch to a keymap temporarily:

root@archiso ~ # loadkeys emacs

Attention, this only affects keyboard layout in virtual console. X keyboard layout is discuessed here.

Network

Set hostname:

# Create /etc/hostname
[root@archiso / #] echo "myhostname" >> /etc/hostname

Update /etc/hosts:

127.0.0.1	localhost
::1		localhost
127.0.1.1	myhostname.localdomain	myhostname

The third line in the request uses 127.0.1.1 instead of 127.0.0.1. It does not matter which one is used, actually. But hostname resolution provide some information.

Install NetworkManager:

[root@archiso / #] pacman -S networkmanager

NetworkManager has built-in DHCP client, so we don't need to install dhcpcd. Aslo, it will pull in wpa_supplicant. To configure wpa_supplicant.conf, check Gentoo Installation.

Initramfs by mkinitcpio

# /etc/mkinitcpio.conf
MODULES=(vfat)
HOOKS=(base udev autodetect modconf block filesystems keyboard fsck keymap encrypt lvm2)

hook	info
keymap	needed as custom keymap (emacs) is set in vconsole.conf.
encrypt	LUKS
lvm2	LVM

Add vfat module, otherwise initramfs fails load the key file.
This is a busybox based initramfs configuration. It is different from systemd init booting process, which is the next phase.
Use mkinitpico -H <hook-name> to check hooks info. Use mkinitcpio -L to list available hooks.
Remove consolefont hook as it is unnecessary.

Before creating the new initramfs, we should install lvm2, otherwise mkinitcpio cannot find the hook:

[root@archiso / #] pacman -S lvm2

Now we will re-create the initramfs to include the new hooks, though pacstrap above already created it upon installation of linux package (pulled in by the base group).

[root@archiso / #] mkinitcpio -P

mkinitcpio will install generate two images, a default and a fallback that skips the autodetect hook thus including a full range of mostly-unneeded modules. Obviously, the fallback image is much bigger in size than the default one as it attempts to pull in as many modules as possible. Therefore, fallback image generation usually reports missing firmware. If the system does not have such hardware, it can be safely ignored. Otherwise, just install the relevant modules like 'wd719x-firmware'.

root password

[root@archiso / #] passwd

Boot loader - Grub

[root@archiso / #] pacman -S grub efibootmgr intel-ucode

efibootmgr is used by the GRUB installation script to write boot entries to NVRAM.
intel-ucode is used to apply CPU stability and security updates during boot.

Since the boot partition is encrypted by LUKS, so we should enable encryption option for Grub.

# /etc/default/grub
GRUB_ENABLE_CRYPTODISK=y

This option is used by grub-install to generate core.img. Then we set the kernel parameters so that initramfs can unlock (using encrypt hook) the encrypted swap, root, home partitions. For the arguments format, check mkinitcpio -H encrypt before editing /etc/default/grub.

# /etc/default/grub

# or PARTUUID
GRUB_CMDLINE_LINUX="cryptdevice=UUID=of-sda4:cryptlvm cryptkey=UUID=of-sda2:auto:/arch-luks resume=UUID=of-volgrp-swap root=UUID=of-volgrp-root"
# or /dev/mapper/volgrp-root
GRUB_CMDLINE_LINUX="cryptdevice=/dev/sda4:cryptlvm cryptkey=/dev/sda2:vfat:/arch-luks resume=/dev/volgrp/swap root=/dev/mapper/volgrp-root"

It is error-prone to input UUID/PARTUUID without GUI, we can replace UUID with the device pathname.
Please make sure the key file is located under ESP partition.

If you forget the copy the key file before choorting, just exit the chroot, copy key file, and re-enter.
Kernel parameter

resume is used to suspend to disk. root is optional as long as grub-mkconfig is used to generate the boot menu.
mkinitpico by default, doe not support gpg hook. Therefore, we cannot use a gpg-procted key file unless mkinitcpio-gnupg is adopted. However, using a gpg-protected key file does not add security level compared to passphrase as the protected key file is also protected by passphrase. Hence, we use file arch-luks directly.

Once the configured, we can install the bootloader to ESP partition:

[root@archiso / #] grub-install --target=x86_64-efi --efi-directory=/efi --bootloader-id=GRUB --recheck --modules="part_gpt"

Attention that, there is no /dev/sda argument as that is for legacy BIOS booting.

Then, generate boot menu:

[root@archiso / #] pacman -S os-prober
[root@archiso / #] grub-mkconfig -o /boot/grub/grub.cfg

In order for Grub to find Windows 10 system, please turn off Fast Startup. Also make sure os-prober* is installed. By the way, /boot and /efi must also be mounted.

If the grub-mkconfig hangs there, probably read LVM need access to /run/lvm under new root and use the suggested hack.

root@archiso ~ # mkdir /mnt/hostlvm
root@archiso ~ # mount --bind /run/lvm /mnt/hostlvm
root@archiso ~ # arch-chroot /mnt
root@archiso ~ # ln -s /hostlvm /run/lvm

The following is for BIOS/GPT schema:

[root@archiso / #] pacman -S grub intel-ucode
[root@archiso / #] grub-install --target=i386-pc /dev/sda
[root@archiso / #] grub-mkconfig -o /boot/grub/grub.cfg

Although we install x86_64 Arch Linux, the --target should be i386-pc due to BIOS booting scheme.

Reboot

[root@archiso / #] cp .bash_history /path/to/usb-stick/bash.log (optionally as it remains after reboot but maybe overriden)
[root@archiso / #] exit/Ctrl-D
root@archiso ~ # cp .zsh_history /path/to/usb-stick/zsh.log
root@archiso ~ # umount -R /mnt
root@archiso ~ # reboot

We'd better unmount all partitions under /mnt to determine busy partitions and diagnose with fuser.

General Settings

Check Post-installation.

Enable NetworkManager

# optional on desktop as networkmanager will detect Wi-Fi interface and ask for Wi-Fi password
[root@host ~]# wpa_passphrase ssid psk > /etc/wpa_supplicant/wpa_supplicant-wlan0.conf
   
[root@host ~]# systemctl enable/start NetworkManager

New user account
Swap file

Optional if swap partition is created.
```
[root@host ~]# dd bs=1M count=1024 if=/dev/zero of=/swapfile
[root@host ~]# chmod 600 /swapfile
[root@host ~]# mkswap /swapfile
[root@host ~]# swapon /swapfile
[root@host ~]# swapon --show
[root@host ~]# free -h
```
Finally, addan entry to /etc/fstab:

/swapfile none swap defaults 0 0
1. We must use the exact swap file path instead of UUID or Label.
2. To cease manual operation bother, try systemd-swap that automates swap management.

X

Time

Refer to Linux Time.

pacman

/etc/pacman.conf sets almost everything regarding pacman, like database, cache, log, repos etc.

pacman has two data warehouses:

main repo. the synced database that stores package metadata (pacman -S).
```
~ $ tar -tzpvf /var/lib/pacman/sync/core.db
```
local repo. the locally installed packages (pacman -Q).

[root@host ~ #] pacman -Ss regex                       # search for pkg
[root@host ~ #] pacman -Si pkg                         # show pkg info
[root@host ~ #] pacman -Sg grp                         # show group info
[root@host ~ #] pacman -S pkg1 pkg2                    # install pkg1 and pkg2

[root@host ~ #] pacman -Sy archlinux-keyring           # upgrade keyring first
[root@host ~ #] pacman -Syu                            # roll the system
[root@host ~ #] pacman -Syu pkg                        # roll the system and install pgk
                                                       
[root@host ~ #] pacman -Qs pkg                         # search for locally installed pkg 
[root@host ~ #] pacman -Qi pkg                         # show locally installed pkg info
[root@host ~ #] pacman -Qe                             # list explicitly installed pkgs
                      
[root@host ~ #] pactree pkg                            # show dependency

[root@host ~ #] pacman -F /path/to/file                # pathname in main repo
[root@host ~ #] pacman -Qo /path/to/file               # pathname in local repo
                                                       
[root@host ~ #] pacman -D --asdeps pkg                 # mark a package as non-explicitly installed
[root@host ~ #] pacman -D --asexplicit pkg             # mark a package as explicitly installed
                                                       
[root@host ~ #] pacman -R pkg                          # remove pkg but leave dependencies alone
[root@host ~ #] pacman -R grp
[root@host ~ #] pacman -Rs pkg                         # remove pkg and orphan dependencies; report error: argument '-' specified with empty stdin when no orphans

[root@host ~ #] pacman -Qtd                            # query orphaned packages
[root@host ~ #] pacman -Qtdq | pacman -Rns -           # remove orphaned packages

[root@host ~ #] paccache -ruk0                         # clean downloaded packages
[root@host ~ #] pacman -Scc                            # more aggressive

AUR

Use wps-office-cn as an example.

# get the PKGBUILD file
~ $ git \
-c 'url.https://aur.archlinux.org/wps-office-cn.git.insteadOf=https://aur.archlinux.org/wps-office-cn.git' \
clone --depth=1 \
https://aur.archlinux.org/wps-office-cn.git

# build the package
~ $ cd wps-office-cn$
~ $ makepkg -sirc

# install the package
~ $ sudo pacman -U wps-office-cn-11.1.0.11698-1-x86_64.pkg.tar.zst

wpspdf relies on libtiff5.

To upgrade the package, repeat the above procedures.

Xterm

Install xterm package. Add the following into ~/.Xresources:

XTerm.termName: xterm-256color
XTerm.vt100.locale: true
XTerm.vt100.metaSendsEscape: true
XTerm.vt100.reverseVideo: true

XTerm.vt100.backarrowKey: false
XTerm.ttyModes: erase ^?

XTerm.bellIsUrgent: true

XTerm.vt100.faceName: DejaVu Sans Mono:style=Book:antialias=true
XTerm.vt100.faceNameDoublesize: WenQuanYi WenQuanYi Bitmap Song
XTerm.vt100.faceSize: 10

XTerm.vt100.translations: #override \n\
    Ctrl Shift <Key>C: copy-selection(CLIPBOARD) \n\
    Ctrl Shift <Key>V: insert-selection(CLIPBOARD)

To take these settings into effect immediately:

[root@host ~ #] xrdb -merge ~/.Xresources
# -or-
[root@host ~ #] xrdb ~/.Xresources

xrdb without the -merge option will reload .Xresources, replacing current settings.

Copy and Paste

X11 has two kinds of buffers, namely PRIMARY and CLIPBOARD.

To copy/paste to/from the CLIPBOARD , we select (highlight) text, press Ctrl-C and Ctrl-V. Selected text will be copied to PRIMARY buffer automatically. To paste from PRIMARY, press the middle mouse button. Shift-insert also pastes from PRIMARY, but support only by terminal emulators. Implicitly, PRIMARY buffer lives a short period and will be replaced with new selection.

Literally, we use paste and insert, buffer and selection interchangeably.

Buffer/Selection	In/Copy	Out/Paste/Insert
PRIMARY	select	MiddleMouse/Shift-Insert
CLIPBOARD	select & Ctrl-C	Ctrl-V

The actual buffer and key shortcut used are determined by X application. When it comes to Xterm, selection goes to PRIMARY by default. However, we can set selection to CLIPBOARD by Ctrl-MiddbleMouse and enable Select to Clipboard. Alternatively, we can adjust ~/.Xresources:

# ~/.Xresources
XTerm.vt100.selectToClipboard: true/false

Such settings will abandom the PRIMARY buffer. Instead, we add copy/paste shortcuts:

# ~/.Xresources
XTerm.vt100.translations: #override \n\
    Ctrl Shift <Key>C: copy-selection(CLIPBOARD) \n\
    Ctrl Shift <Key>V: insert-selection(CLIPBOARD)

Notice that each key binding line must be separated by \n\.

Alacritty

xterm is really outdated, use Alacritty on Arch Linux.

However, Alacritty sets ENV variable TERM to alacritty which would break remote shell (e.g. SSH) like below.

WARNING: terminal is not fully functional

For general compatibility, we can configure Alacritty and set ENV TERM to xterm-256color.

# /usr/share/doc/alacritty/example/alacritty.yml
# /.config/alacritty/alacritty.yml

env:
  TERM: xterm-256color

Resolution

Virtual terminal

VirtualBox might fail to detect correct screen resolution for virtual terminal (console). Similar to Android-x86 post, we should add custom kernel parameter through Grub bootloader.

In Grub menu, press e, and then F2. You are now in command line prompt, type vbeinfo to list possible resolutions. The one prefixed with * is the default. Choose a desired one (i.e. 1366x768) and press ESC. Append video=1366x768 to linux line and press F10. If none of the listed values meet requirement, we can try vboxmanage setextradata:

~ $ VBoxManage setextradata archlinux "CustomVideoMode1" "1300x730x24"

The newly created value will be listed by vbeinfo. To make the desired resolution permanent, we can edit /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet video=1300x730"

We can also set Grub's resolution by:

GRUB_GFXMODE="1366x768x24" (for Grub menu itself)

Do not forget to update the Grub menu:

[root@host ~ #] grub-mkconfig -o /boot/grub/grub.cfg

Xorg

Most of the time, Xorg resolution works out of box. Command line tool xrandr and Xorg.conf can be used to set Xorg resolution.

Query resolution:

user@host ~ $ xrandr

The entry with star * means current resolution while with + means default preferred resolution. Set resolution:

user@host ~ $ echo $DISPLAY
user@host ~ $ xrandr --display :0 --output VGA-1 --mode 1366x768

We can put the commands into ~/.xinitrc or ~/.config/awesome/autostart.sh. Alternatively, create Xorg.conf /etc/X11/xorg.conf.d/10-resolution.conf:

Section "Monitor"
  Identifier "VGA-1"
  Modeline "1368x768_60.00"   85.25  1368 1440 1576 1784  768 771 781 798 -hsync +vsync
  Option "PreferredMode" "1368x768_60.00"
EndSection

Section "Screen"
  Identifier "Screen 0"
  Monitor "VGA-1"
  DefaultDepth 24
  SubSection "Display"
    Depth 24
  EndSubSection
EndSection

You can get the Identifier value from xrandr output.

Fcitx

Check Fcitx:

user@host ~ $ pacman -S fcitx-im fcitx-configtool

Then, append run fcitx-autostart into ~/.config/awesome/autostart.sh and relaunch awesome.

If some QT applications cannot input Chinese, ensure the following packages are available.

user@host ~ $ pacman -Qi fcitx-qt5
user@host ~ $ pacman -Qi fcitx-qt6

WPS

WPS on Linux has two version, namely wps-office-cn and wps-office. The CN version has much more features!

~ $ git clone https://aur.archlinux.org/wps-office-cn.git
~ $ cd wps-office-cn
~ $ makepkg
~ $ sudo pacman -U wps-office-cn-11.1.0.10702-1-x86_64.pkg.tar.zst

To display math symbols, we need to install AUR ttf-wps-fonts.

After closing WPS, the background process refuses to exit:

~ $ sudo chmod -x /usr/lib/office6/wpsoffice
~ $ sudo chmod -x /usr/lib/office6/wpscloudsvr

However, remove the excution permission may disallow WPS login. It's a trade-off.

ALSA

Assume the system has multiple audio cards, we may have adjust the default card.

First, identify the card list.

~ $ aplay -l

~ $ cat /proc/asound/cards

Then, update ~/.config/alsa/asoundrc.

defaults.pcm.card 0
defaults.ctl.card 0

Re-login and try alsamixer and speaker-test.

QQ robot

2017-11-11T00:00:00+00:00

Some APIs has beeb shut down and commands related to QQ number would fail!

QQBot written with Python, is a conversation robot base on Tencent's SmartQQ. This post describes QQbot on Gentoo PC.

Installation

# within virtualenv
~ $ pip install qqbot
~ $ qqbot -h

At the very first run, it may throw exception complaining lack of sqlite3 module. Make sure python is built with relevant USE:

~ # echo 'dev-lang/python sqlite' >> /etc/portage/package.use/python
~ # emerge -av1 dev-lang/python:3.6

Startup

Help

Firstly, check basic parameters:

~ $ qqbot -h

Especially, pay attention to --daemon --bench --qq.

First run

~ $ qqbot
# -or-
~ $ qqbot -q <qq number>

For the very first run, a QRcode pops up. Use QQ mobile app to scan for authorization. -q attempts to load login cookies locally first and resorts to QRcode if that fails.

In order that QRcode image pops up automatically, gvfs-open or shotwell must be available under Linux system, which should not be a problem as long as gvfs package is installed. We can, however, manually open the QRcode located under ~/.qqbot-tmp/*.png.

Once login, a QQbot-term service is launched at port 8188 for interaction. Open a new terminal and:

~ $ qq help
~ $ qq list [ buddy | group | discuss ]
~ $ qq list buddy [ name | nick | mark | qq ] = xxx
~ $ qq list buddy name:like:xxx
~ $ qq list group-member <qq group>

For more operations, refer to official README.MD section 3.

Aternatively, all operations can be done through HTTP API. For example, open http://localhost:8188/list/buddy. We just separate each argument by /. This is pretty useful when terminal output jumbers disorderly.

Configuration

Check ~/.qqbot-tmp/v2.x.conf, and add a section for specific QQ account:

    "jim" : {
        "termServerPort" : 8188,
        "qq" : "123456789",
        "daemon" : True,
	"plugins" : [],
	"pluginsConf" : {},
    },

If there are multiple QQ accounts, please assign different termServerPort.
daemon mode releases terminal immediately after login. stout and stderr are redirected to daemon-$qq.log.
plugins list overides the default value.

With that configuration block, launch QQbot by -u:

~ $ qq stop
~ $ qqbot -u jim

User configuration section name instead of QQ number after -u.

Plugins

Pre-built plugins (i.e. qqbot.plugins.sample ) are installed into site-packages/qqbot/plugins reference to whom should be prefixed with qqbot.plugins..

To be simple, put personal plugin myplugin.py under ~/.qqbot-tmp/plugins directory. You can load a plugin on-the-fly (hot-plug way) or by telling QQbot to load it on startup. Reference to personal plugin is the script filename.

hot-plug method.

~ $ qq plugins (list loaded)
~ $ qq plug qqbot.plugins.test
~ $ qq plugins
~ $ qq unplug myplugin

Automatic plug

Put your desired plugin name to "plugins" : ['autohello',] list.

qqbot.plugins.miniirc

To be able to talk with IRC client:

~ $ qq plug qqbot.plugins.miniirc

Then launch Weechat:

/server add qqbot localhost/6667
# set qqbot nick; change sasl_mechanism to plain; turn off ssl and ssl_verify;
/connect qqbot
/list
/join #group
/query buddy-name

The minirrc plugin is limited and recognizes only QQ name. For instance, you cannot query a QQ number.

V2

2017-10-20T00:00:00+00:00

Arch
Installation
Server Configuration
Client Configuration
Windows
1. Global Proxy
macOS
Refs

Arch

The flowing table illustrates the network layers of a typical V2 deployment.

Firefox                                youtube.com
|                                      |
Socks V2 in                            Freedom V2 out
|                                      |
VMess V2 out                           VMess V2 in
|                                      |
Websocket V2 L6                        Websocket V2 L6
|                                      |
TLS V2 L5                              TCP V2 L5
|                                      |
HTTP =========== Cloudflare ========== HTTP Nginx proxy_pass
|                                      |
TLS                                    TLS
|                                      |
TCP                                    TCP
.                                      .
.                                      .
.                                      .
> .................................... <

Encryption between Nginx and backend V2 can be abandoned (TCP V2 L5) as the traffic only circulates on local host.
Cloudflare now provides free websocket caching service. Therefore, we can turn on Cloudflare CDN to protect our V2 server. However, we cannot camouflage the Host header when Cloudflare CDN is turned on.

Installation

Download and extract binary release.

~ # unzip /path/to/v2ray.zip -d /opt/v2ray/
~ # mkdir -p /opt/v2ray/log

As we will run the service as account nobody, so make sure the directory and files onwership is correct.

~ # chown -R nobody: /opt/v2ray/

Server Configuration

V2 supports multiple configuration formats like Json, YAML, etc. By default, Json format is used. Specially, V2 Json allows comments (//foo or /* bar */).

Proxy by Nginx Location

location /V2 {
    proxy_redirect off;
    proxy_pass http://127.0.0.1:10086;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
}

A side effect when browsing '/V2', brwoser reports "400 Bad Request" as the backend V2 is a websocket server instead of a bare HTTP server. We may also receive "502 Bad Gateway" as the backend V2 is not started yet. We resolve the issue by proxy_intercept_errors as follows.

proxy_intercept_errors on;
error_page 400 502 = /;

The error_page directive does internal redirect. This method is always preferred. A more detailed method would be as below.

proxy_intercept_errors on;

error_page 400 404 /404.html;
location = /404.html {
    internal;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
    internal;
}

The internal directives specifies that a given location can only be used for internal redirects.

Proxy by HTTP Header

Unlike section Proxy by Nginx Location, we can proxy_pass based on customized HTTP headers under the '/' location, using the if directive. This method requires V2 client to be 2.40.2 onwards.

ssl_certificate      /etc/letsencrypt/live/www.example.com/fullchain.pem;
ssl_certificate_key  /etc/letsencrypt/live/www.example.com/privkey.pem;

location / {
    root /var/www;
  
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
  
    if ( $http_host = "shit" ) {
        proxy_pass http://127.0.0.1:10086;
}

The template above uses 'Host' header to redirect requests to backend V2. The customized Host header must not be the website domain. Otherwise, all normal HTTP traffic would go to V2. Usually, we set it to a public domain like www.amazon.com. As you will, it can be any arbitrary strings, like shit.

Modify client configuration accordingly.

"wsSettings": {
  "path": "/",
  "headers": {
    "Host": "shit"
  }
}

Manually manipulate the 'Host' header is not recommended as it is crucial for vhost matching. If V2 supports self-defined header, then it is far more desirable. By the way, it rerquires the evil if directive. Actually, we can also redirect based on the 'Upgrade' header:

if ( $http_upgrade = "websocket" ) {
    proxy_pass http://127.0.0.1:10086;
}

This method still uses the evil if directive.

SELinux and Firewall

On servers with SELinux on, V2ray may report websocket: bad handshake. By default, SELinux does not permit Nginx operations, such as proxying to upstream locations or communicating with other processes through sockets.

Check SELinux audit log.

~ # grep 'denied.*nginx' /var/log/audit/audit.log

Turn SELinux bool value.

~ # getsebool -a | grep httpd_can_network_connect

~ # setsebool -P httpd_can_network_connect 1

Turn on V2 port if it is not ephemeral ports.

~ # semanage port -l | grep [h]ttp_port_t

~ # semanage port -a -t http_port_t -p tcp 10086

Finally, make sure the cloud instance allows at least 443 traffic.

Verify Configuration

~ # /opt/v2ray/v2ray test -c /opt/v2ray/config.json

~ # sudo -u nobody -g nobody -- /opt/v2ray/v2ray -config /opt/v2ray/config.json

~ # ss -nplt

~ # tail -F /opt/v2ray/log/{access,error}.log
~ # tail -F /var/log/nginx/{access,error}.log

Systemd

From now on, V2 provides built-in Systemd unit file. V2 will run as nobody instead of root.

~ # systemctl daemon-reload

~ # systemctl enable /opt/v2ray/systemd/v2ray.service
~ # systemctl start v2ray.service

# -or-
~ # systemctl enable /opt/v2ray/systemd/v2ray@.service
~ # systemctl enable v2ray@4.27.4
~ # systemctl start v2ray@4.27.4

Client Configuration

As TLS encryption is adopted, extra Vmess encryption is unnecessary unless Cloudflare CDN is adopted. Details refer to Cloudflare CDN section.

sniffing

This option is used by inbound or inbounddetour to retrieve domains upon receiving IP packets. Usually, users connect to a domain instead of an IP. Most of the time V2 inbound or inbounddetour receives the domain (as DNS query first) such that relevant traffic is directed in accord with domain rules of the routing section.

However, it happens that a client may connect to an IP directly (i.e. Telegram). What is worse, IP packets can bypass domain rules.

Additionally, if a user forgets to select:

Proxy DNS when using SOCKS v5

in Firefox setting, DNS queries would be sent to local DNS service and then IP packets go to V2. Obviously, local DNS replies are usually poisoned.

Therefore, sniffing comes to help such that it siniffs domains from IP packets, reset the connection, and route against domain rules. DNS poision issue is resolved as well.

From the discussion, we find routing based on IP packet sniffing is more reliable and robust. In the meantime, dokodemo transparent proxy redirects IP packets based on Iptables.

DNS

V2 has built-in DNS support, mainly for two purposes:

Support the routing rules based on domainStrategy setting.

For example, in order to match an IP rule in routing, the target domain must be resolved to IP first by the dns section. That IP will only be used to re-reroute the request, namely deciding which outbound to use!
DNS service for the freedom outbound.

When a target domain is routed to the freedom outbound, it should be directly accessed. V2ray will consult the dns section to resolve the domain. Once got the host IP to connect to, it should be routed again in the routing section. This is a little bit confusing!

Apart from the the built-in DNS support, V2ray has a dns outbound that can be used to serve as a open DNS service, similar to Dnsmasq.

Disable "Proxy DNS when using SOCKS v5" in Firefox.
At V2ray client side:
1. Set up a dokodemo-door inbound at port 53, receiving DNS query.
2. Set up a dns outbound.
3. Set up a routing rule, routes the query from dokodemo-door inbound to the dns outbound.
4. The dns outound intercept DNS queries and redirect the queries to the dns section.

We cal also combine V2ray and Dnsmasq. Queries are first sent to local Dnsmasq, then to V2ray dokodemo-door.

Domain List

V2ray has built-in domain list, like geosite.dat and geoip.dat, which can be used in the routing section and the dns section to achieve smart proxy.

An enhanced version can be found at v2ray-rules-dat.

Besides, V2ray can also load domain list from external files with prefix ext, like h2y.dat. Similar to the built-in domain list, reference to external list can be:

"ext:h2y.dat:ad"

Cloudflare CDN

Turn on CDN to hide server IP from censorship and probably accelerate speed.

Cloudflare supports caching websocket traffic for free accounts. To enable Cloudflare CDN on V2, just click the orange symbol on DNS record.

You might wait for a while before the DNS setting comes into effect. Check by dig your.domain.com and find the returned IP be that of Cloudflare's CDN servers.

A few notes:

Cloudflare's free CDN service actually degrade instead of accelerate your service.

This can be caused by ISP dropping IP packets to its servers. What's worse, Cloudflare may do this on purpose for paid service.
Once enabled, we must reset Customized HTTP header to the real domain name instead of shit.

Otherwise, V2 client cannot establish TLS session with Cloudflare as CDN servers does not have a shit vhost.
VMess encryption must be enabled besides TLS, thus Cloudflare cannot see VMess payload.

You'd better not use V2 do confidential communication such as Email, Login etc.

Vmess and Clock

Vmess depends on system clock to do authorization, so make sure the clock delta between client and server fall within 90s.

Windows

Global Proxy

To enable global Socks proxy on Windows, the address part in "Windows Settings" should be http://socks=127.0.0.1. Alternatively and recommended, use the "Internet Options", "Connections", "LAN settings", "Proxy Server", "Advanced" and "Bypass proxy server for local addresses".

Alternatively, use 3rd-party tools like Clash as the client side of V2. Whatever methods you choose, make sure to bypass the remote domain and VPS IP.

However, Windows UWP app like Mail, Calendar etc. have their own sandbox (namely App Container) and does not respect system proxy, including the "Microsoft Store" itself. The container prevents App from connecting to system's loopback interface, such that Wireshark cannot capture UWP traffic and UWP does not use global proxy. Clash supports UWP loopback access and enjoys the proxy.

It is recommended to use Windows' built-in CheckNetIsolation.exe.

PS C:\Users\jim> CheckNetIsolation.exe -?
PS C:\Users\jim> CheckNetIsolation.exe LoopbackExempt -?
PS C:\Users\jim> CheckNetIsolation.exe LoopbackExempt -s

PS C:\Users\jim> CheckNetIsolation.exe LoopbackExempt -a -n=<package-family-name>
PS C:\Users\jim> CheckNetIsolation.exe LoopbackExempt -a -p=<sid>

How to find out an App's "Package Family Name"?

PS C:\Users\jim> Get-AppxPackage > lst  # list full list
PS C:\Users\jim> Get-Content .\lst.md | Select-String -Pattern "wechat"
PS C:\Users\jim> Get-Content .\lst.md | Select-String -Pattern ".*windowscommunicationsapps.*"  # Mail and Calender

macOS

V2raya deployed by Homebrew.

Originally, V2raya depends on V2ray, so we should install "v2ray" first. However, V2raya now automatically installs the dependency "v2ray5" (version 5).

# install v2ray (deprecated)
~ $ brew install v2ray

# install v2raya (automatically pulls V2ray5)
~ $ brew tap v2raya/v2raya
~ $ brew install v2raya/v2raya/v2raya

# install homebrew-services
~ $ brew services list

# start v2raya
~ $ brew services start v2raya
~ $ brew services info v2raya

Go to http://localhost:2017 for setup. For example, update geoip.dat and geosite.dat at "/opt/homebrew/Cellar/v2ray/4.45.2/share/v2ray". Then, go to "V2raya - Settings - Traffic Splitting Mode of Rule Port - Routing A - Configure", append the following line:

domain(geosite:category-ads-all) -> block

The resulted V2 config is located at ~/.config/v2raya/config.json. We can add any routing rules on demand.

Currently, V2raya on macOS does not support transparent proxy. We can set macOS system proxy to use V2raya. Attention please; since RoutingA (Advanced) is used, the Socks and Http inbound port are 20173 and 20172 respectively.

Refs

Let's Encrypt

2017-10-10T00:00:00+00:00

ACME and Certbot
1. Certbot Configuration
Nginx Template
Obtain a Certificate
Manage a Certificate
Deploy Certificates
ECDSA Certificate and RSA Certificate

ACME and Certbot

In order to manage Let's Encrypt certificates, we need a client tool that supports the RFC 8555 Automated Certificate Management Environm (ACME) protocol. ACME clients communicate with servers of Let's Encrypt using ACME protocol, to automate the following jobs:

Generate key pair.
Create and send Certificate Signing Request (CSR).
Domain Control Validation (DCV). Challenge ownership of domain or web server.
Automatically renew certificates.

Let's Encrypt officially recommends the Certbot client.

~ # dnf --enablerepo ol8_developer_EPEL search certbot
~ # dnf --enablerepo ol8_developer_EPEL install certbot
~ # dnf repoquery -l certbot

~ # certbot -h all
~ # certbot certificates # List existing certificates

To run certbot, we supply a subcommand like certonly (obtain a certificate), install (update vhost), run (both) etc. Whenever possible, try with --dry-run first.

If no subcommand is given, then run is assumed. A subcommand accepts different types of plugins, namely authenticator plugins and installer plugins. The general usage looks like:

certbot subcommand --plugin-name ...

The certonly subcommand and install subcommand accept authenticator plugins and installer plugins respectivelly, while the run subcommand accepts plugins belonging to both types.

authenticator plugins challenge you whether you are eligible for a certificate by verifying the domain owership. installer plugins automatically modify web server's configuration with specified certificate. Most of the time, we firstly use authenticator plugins to obtain a certificate and then manually update configurations of the web server. We don't use installer plugins/

Examples of authenticator plugins are --webroot and --standalone. There does not exist plugins exclusively belonging to the installer type. However, --apache and --nginx belong to both types of authenticator and installer, and can be used with the run subcommand to both obtain and install a certificate.

The following table simply presents their relationship:

subcommand	type	plugin
certonly	authenticator	webroot, standalone, dns-cloudflare
install	installer	n/a
run (default)	both	apache, nginx

Certbot Configuration

By default, the configuration is:

~ $ cat /etc/letsencrypt/cli.ini
preconfigured-renewal = True

A few custom configs:

# /etc/letsencrypt/cli.ini

# TOS
agree-tos = true

# cert notification
email = alice@example.com

# ECC key by default
key-type = ecdsa
elliptic-curve = secp384r1

# RSA key size
rsa-key-size = 4096

# renewal hook
deploy-hook = systemctl reload nginx.service

Nginx Template

A simple Nginx HTTP template.

server {
	listen       80 default_server;
	listen       [::]:80 default_server;
	server_name  _;
	root         /usr/share/nginx/html;

	# Load configuration files for the default server block.
	include /etc/nginx/default.d/*.conf;

	location / {
	}

	error_page 404 /404.html;
		location = /40x.html {
	}

	error_page 500 502 503 504 /50x.html;
		location = /50x.html {
	}
}

Obtain a Certificate

ACME challenge is a complex process, you'd better turn off CDN caching before applying for a certificate.

Webroot Authenticator

~ # certbot -h certonly
~ # certbot -h webroot

~ # certbot certonly --webroot -w /var/www/example.com -d example.com,www.example.com -w /var/www/b.example.com -d b1.example.com -d b2.example.com --email "name@example.com" --dry-run

As the name implies, certonly only obtains certificates but does not install them. When requesting a certificate for multiple domains, each domain will use the most recently specified --webroot-path, -w. The --email option is to receive notification like expiration message. Multiple domain names can be separated by comma in the -d argument or by individual -d arguments. Always append the --dry-run before any real operation.

Use the --webroot authenticator if you have full control over the running web server and the domain. Let's Encrypt's ACME server tells the Certbot client to write unique files under the root of your web server (i.e. /usr/share/nginx/html/). This step is challenging you whether you do own the web server (i.e. write access). The file URL takes the form:

http://domain/.well-known/acme-challenge/<file>

Afterwards, the ACME server tries to fetch the URL, verifying you own the domain. Therefore, make sure the domain name is finally resolved to the web server IP and the web server is running on HTTP 80 (NOT HTTPS 443). Though, the webroot authenticator use HTTP to challenge the ownership, the security of Certbot itself is guranteed by ACME protocol. We call this kind of challenge http-01.

You can cover your web server with CDN (i.e. Cloudflare), as the challenge method is to download the unique file. However, if the CDN enables HSTS, then temporarily turn it off by removing the caching capability in DNS settting.

Once downloaded, the ACME server also compares the file hashes of the fetched copy with its local store.

Standalone Authenticator

The --standalone authenticator is usually used when the host where you run certbot is not the one you would like to host your web server.

~ # certbot -h standalone
~ # certbot certonly --standalone -d www.example.com,blog.example.com --dry-run

It starts a temporary standalone web server to talk to Let’s Encrypt. Therefore, it does not verify web server. You must make sure port 80 is available. You may have to turn down existing web servers to release port 80.

Recall that --webroot challenge domain onwership by http-01, GETting an unique URL. Then how does --standalone challenge the domain onwership? By http-01 too!

Though ACME protocol supports challenge with tls-01 by verifying the temporarily self-generated certificate, but Certbot only implements HTTP 80. If you have chosen to use tls-o1, then you must make sure the domain name is directly resolved to the host IP where you apply for certificates certbot client. If the domain name is resolved to the host IP, it means you manage the domain. You cannot cover your domain with CDN, otherwise the ACME server would got the CDN's certificate instead of the temporary one by Certbot.

DNS Authenticator for Wildcard Certificate

With the certbot client, the only way to obtain a wildcard certificate from Let's Encrypt is using DNS Plugins. DNS plugins belong to the authenticator type but challenge you by DNS protocol.

Basically, a DNS plugin uses a API token from the DNS platform to first add a TXT record and then remove that record, such that you are proved to own the domain name. Therefore, the DNS authenticator does not interfere in web server.

For each DNS platform, certbot provodes a corresponding DNS plugin. DNS plugins are not installed by default.

Take Cloudflare for example. Firstly, let's install DNS plugin:

~ $ sudo dnf --enablerepo ol8_developer_EPEL install python3-certbot-dns-cloudflare

Then create API token for the DNS plugin at API Token. Please configure the IP whitelist. We test the token:

~ $ curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
>      -H "Authorization: Bearer xxxxxxxxxxxxxyyyyyyyyyyyyyzzzzzzzzzzzzz" \
>      -H "Content-Type:application/json"

~ $ curl -X GET "https://api.cloudflare.com/client/v4/zones/" \
>      -H "Authorization: Bearer xxxxxxxxxxxxxyyyyyyyyyyyyyzzzzzzzzzzzzz" \
>      -H "Content-Type:application/json"

The token is used for both cert creation and renewal. Store the token somewhere:

~ # mkdir -p ~/root/.secrets/certbot
~ # chmod 700 ~/root/.secrets/certbot

~ # cat > /root/.secrets/certbot/cloudflare.ini <<EOF
> # Cloudflare API token used by Certbot
> dns_cloudflare_api_token = xxxxxxxxxxxxxyyyyyyyyyyyyyzzzzzzzzzzzzz
> EOF

~ # chmod 600 /root/.secrets/certbot/cloudflare.ini

Finally create wildcard certificate:

~ # certbot certonly --dry-run \
--dns-cloudflare \
--dns-cloudflare-credentials /root/.secrets/certbot/cloudflare.ini \
--dns-cloudflare-propagation-seconds 60 \
--domains example.com,*.example.com \
--cert-name example.com \
--key-type ecdsa \
--elliptic-curve secp384r1 ; echo

Manage a Certificate

No matter which plugins you use, the generated certificates are placed under /etc/letsencrypt/. Also the arguments used to generate the certificates are stored alongside for latter renwal. The directory tree looks like:

# tree /etc/letsencrypt/
/etc/letsencrypt/
├── accounts
│   ├── acme-staging-v02.api.letsencrypt.org
│   │   └── directory
│   │       └── 74cfecb0fe45ec846ae6cc76b6ba0b44
│   │           ├── meta.json
│   │           ├── private_key.json
│   │           └── regr.json
│   └── acme-v02.api.letsencrypt.org
│       └── directory
│           └── 34ec5a2656cb495bb9c7b74755d5dfad
│               ├── meta.json
│               ├── private_key.json
│               └── regr.json
├── archive
│   └── example.com
│       ├── cert1.pem
│       ├── chain1.pem
│       ├── fullchain1.pem
│       └── privkey1.pem
├── cli.ini
├── csr
│   └── 0000_csr-certbot.pem
├── keys
│   └── 0000_key-certbot.pem
├── live
│   ├── README
│   └── example.com
│       ├── cert.pem -> ../../archive/example.com/cert1.pem
│       ├── chain.pem -> ../../archive/example.com/chain1.pem
│       ├── fullchain.pem -> ../../archive/example.com/fullchain1.pem
│       ├── privkey.pem -> ../../archive/example.com/privkey1.pem
│       └── README
├── renewal
│   └── example.com.conf
└── renewal-hooks
    ├── deploy
    ├── post
    └── pre

18 directories, 20 files

You will find a certificate has different version, namely fullchain.pem, chain.pem and cert.pem. Use fullchain.pem whenever possible as it is a combination of chain.pem and cert.pem. Use chain.pem for OCSP stapling as that does not require the leaf cert.

First, list existing certificates:

~ # certbot certificates
~ # certbot certificates --cert-name example.com

It's not unusual that different certificates share common FQDNs. From the output, you will find each certificate has a name. By default, it's named afer the first FQDN from option --domains. We can explicitly set a certificate name with option --cert-name.

Change a Certificate's Domains

Add new domains to an existing certificate with argument --expand.

~ # certbot certonly --expand --cert-name example.com --webroot -w /var/www/log.example.com -d blog.example.com --dry-run

But the it is recommended to abandon the --expand option such that you can either add or remove domains by supplying a complete new list of domains to the -d option.

~ # certbot certonly --cert-name example.com --webroot -w /var/www/log.example.com -d blog.example.com --dry-run

After the above operation, the certificate example.com only contains domain name blog.example.com. It actually just creates a new certificate with the same name!

Renew a Certificate

Let's Encrypt certificates expire after 90 days and can be renewed if a certificiate expires in less than 30 days. Renewing a certificate is actually to generate a new identical copy of the original certificate, using the same arguments which they are created with. The only difference is the expiration date is updated.

Renewal just create a new certificate with the same name! So we can just re-run the command used to obtain the certificate, as in section Obtain a Certificate.

But sometimes, we need fine control over renewal. Certbot stores a renewal configuration for each certificate:

~ # realpath example.com.conf
/etc/letsencrypt/renewal/example.com.conf

A certificate can be manually renewed or automatically renewed.

Manual Renewal

To interactively renew all of your certificates, invoke the renew subcommand.

~ # certbot -h renew
~ # cerbot renew --deploy-hook "systemctl reload nginx.service" --dry-run
~ # cerbot renew --deploy-hook /path/to/hook-script.sh --dry-run

The --deploy-hook will execute a program if and only if the certificate is successfully renewed. You can add the hook to its renewal configuration file.

# /etc/letsencrypt/renewal/example.com.conf
 
[renewalparams]
deploy_hook = systemctl reload nginx.service

You can also put the hook into /etc/letsencrypt/renewal-hooks/deploy/. For example, here is a hook script:

#!/usr/bin/env bash

systemctl reload nginx.service

Or more generally, you can put the hook in Certbot Configuration file.

Once hooks is configured, just run:

~ # cerbot renew --dry-run

Renewal of standalone certificates requires port 80 to be available. In other words, Nginx should not listen on HTTP 80. If that's the case, you can stop Nginx first. Alternatively, we use --pre-hook and --post-hook arguments.

~ # cerbot renew --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx" --dry-run

Attention please; unlike the --deploy-hook hook, the --pre-hook and --post-hook will be executed no matter of sucessful or failed renewal. Similarly, you can put the hooks into configuration files or script files.

Automatic Renwal

We can automate the process by crontab or systemd timer.

Crontab:

# /var/spool/cron/root

~ # crontab -u root -e
~ # crontab -u root -l

30 0,12 * * * /usr/bin/certbot renew --quiet

Systemd timer:

~ # systemctl list-unit-files --type timer
~ # systemctl status certbot-renew.timer
~ # systemctl enable certbot-renew.timer
~ # systemctl start certbot-renew.timer
~ # systemctl list-timers -all

The certbot-renew.timer would invoke certbot-renew.service automatically.

Backup the Account and Certificates

Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt/. You should make a secure backup of this folder now. This configuration directory will also contain certificates and private keys obtained by Certbot so making regular backups of this folder is ideal.

Say we have obtained a certificate and configured auto-renewal on server A, but want to use the same certificate on server B. How to achive this?

Basically, we have two methods.

Write a deploy-hook to rsync renewed certificate and key to server B.
Start from scratch on server B to obtain new certificate with the same procedure in this post. Don't worry about conflicts as obtaining a new certificate justs create a brand new one.

The only concern is the number limit of renewal per-certificate. Currently, Let's Encrypte allows 5 renewal per month for an individual certificate.

Deploy Certificates

You can update web server's SSL vhost configuration accordingly once the certificate is ready as below. We can follow Mozilla SSL Config Recommendation to get a secure and comptabile template.

server {
    listen       443 ssl http2 default_server;
    listen       [::]:443 ssl http2 default_server;
    server_name  www.example.com;
    root         /usr/share/nginx/html;

    ssl_certificate "/etc/letsencrypt/live/www.example.com/fullchain.pem";
    ssl_certificate_key "/etc/letsencrypt/live/www.example.com/privkey.pem";
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  10m;
    ssl_ciphers PROFILE=SYSTEM;
    ssl_prefer_server_ciphers on;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
    }

    error_page 404 /404.html;
        location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
        location = /50x.html {
    }
}

Then reload the web server:

~ # nginx -t
~ # systemctl reload nginx

Additionaly, you can redirect all HTTP traffic to HTTPS:

return 301 https://$server_name$request_uri;

This does not influence certificate renewal but may hinder new certificate application.

If everything goes as expected, turn back on CDN coverage.

ECDSA Certificate and RSA Certificate

As of version 1.10, Certbot supports ECDSA certificates.

When it comes to certificates and Signature Algorithms, we categorize them into Elliptic Curve Digital Signature Algorithm (ECDSA) certificate and RSA certificate. ECDSA certificate is also named Elliptic Curve Cryptography (ECC) cerficiate.

The two kinds of certificate differ in the type of public/private keys. When generating a key pair, we can choose ECDSA algorithm or RSA algorithm. In return, the key algorithm affecrs almost any security operations, like encryption, signature etc.

ECDSA outweighs RSA in two ways:

Under certain level of security (in bits), ECDSA has much shorter key length.
Less CPU computation and networking load.

Therefore, it is always recommended to obtain a ECDSA certificate as it vastly reduce the time taken to perform TLS handshake and load web pages faster.

For a long, Let's Encrypt does not support ECDSA certificate. However, the situation changed recdently (2020-09).

To obtain a certificate from Let's Encrypt, we have to resort to other ACME clients other than Certbot. I choose acme.sh.

When managing certificates with acme.sh, please switch to root instead of the sudo, su method.

Firstly, we install acme.sh:

~ # curl https://get.acme.sh | sh

~ # ll ~/.acme.sh/
~ # crontab -u root -l

Then, obtain an ECDSA certificate by Webroot Authenticator:

~ # acme.sh --issue -w /usr/share/nginx/html/ -d blog.example.com --keylength ec-256

The parameter --keylength ec-256 specifies the length of ECDSA key.

How do I know if the certificate is ECDSA-based or RSA-based? Just check the "Public Key Algorithm" field, and make sure the field value is id-ecPublicKey, as follows:

~ # openssl x509 -inform pem -noout -text -fingerprint -md5 < ~/.acme.sh/blog.example.com_ecc/blog.example.com.cer

You may find that, the public key is just 256-bit long.

By default, everthing about acme.sh is placed under ~/.acme.sh/ by default. This directory is only for internal usage. To make use of the certificate, we should install the cert to Apache/Nginx etc.

The following Nginx configuration is error-prone:

ssl_certificate /root/.acme.sh/blog.example.com_cc/fullchain.cer;
ssl_certificate_key /root/.acme.sh/blog.example.com_cc/blog.example.com.key;

Nginx will report permission error:

~ # systemctl reload nginx

[emerg] BIO_new_file ... (SSL: error:0200100D:system library:fopen:Permission denied:fopen('/root/.acme.sh/blog.example.com_ecc/fullchain.cer','r')

Instead, execute --install-cert <domain> that will copy the certificate and key into another directory, like:

~ # mkdir -p /etc/acme.sh/blog.example.com_ecc

~ # acme.sh --install-cert --ecc -d blog.example.com --cert-file /etc/acme.sh/blog.example.com_ecc/cert.pem --key-file /etc/acme.sh/blog.example.com_ecc/key.pem --fullchain-file /etc/acme.sh/blog.example.com_ecc/fullchain.pem --reloadcmd "systemctl reload nginx.service"

The destination can be anywhare but the -d demands domain name. Attention please; the --ecc option tells acme.sh to copy ECC certificate instead of RSA certificate, by looking for a directory with _ecc suffix like blog.example.com_ecc.

acme.sh maintains two copies of a certificate, one for internal usage, one for webserver. On the contrary, certbot use symblic!

The --reloadcmd is critical to tell Nginx reload renewed certificates. Check ~/.acme.sh/en.example.com_ecc/en.example.com.conf , we will find the reload command is encoded by base64:

Le_ReloadCmd='__ACME_BASE64__START_c3lzdGVtY3RsIHJlbG9hZCBuZ2lueC5zZXJ2aWNl__ACME_BASE64__END_'

Finally, update Nginx to make use of the copy of the certificate and key under /etc/acme.sh/blog.example.com/.

动态称球

2017-10-01T00:00:00+00:00

问题描述

有N个同规格的球，现发现其中有一个坏了，重量和其它N-1个球有稍微区别。现给一个没有砝码的灵敏天平，请通过最少次的称量，找出坏球，并判别是轻了还是重了。

数学模型

给球编号，1,2, … , N.
假设需要的最少次数是n，那么N和n之间的关系是：

n = ⌈ log₃(2N+3) ⌉ 或 N = ⌊ (3ⁿ - 3)/2 ⌋ (N >= 4)
1. 这里出现两个特别的数字2和3. 2是因为每一个球有两个可能性，即它是坏球且比其它标准球轻，或它是坏球且比其它标准球重。3是因为每次称量会有三种结果，即左边重（右边轻），右边重（左边轻）或平衡。
2. 我们说最少次数n，其实不是没次称量都要n次，因为称量过程是随机的，我们没法预测下一次称量的结果。但如果算法合理，最复杂的情况是n次；反之，算法不合理，最复杂情况不低于n次。
3. 假设n不变，球数加1，即N+1个球，那么可以保证找出坏球，但不能保证识别出轻重。

称量分析

称球的过程其实可以用决策树或矩阵方程表述。

每次称量前，把待称量的球（假设K个，第一次为N）分成三堆，并确保上天平的两堆球数相同，也即：第一堆（左堆）⌈ K/3 ⌉个球，第二堆（右堆）⌈ K/3 ⌉个球，第三堆（未上称）K - 2⌈ K/3 ⌉个球。

这样的划分的结果是，不上天平的那堆球数少。
前面说过，每次称量有三种可能，每种可能可以推导出新的信息：
1. 左边重：坏球肯定在天平上的两堆里。坏球在左边并比标准球重，或坏球在右边且比标准球轻。
2. 右边重：坏球肯定在天平上的两堆里。坏球在左边并比标准球轻，或坏球在右边且比标准球重。
3. 平衡：坏球在未上称的第三堆，暂时无法判断轻重。
每次称量前，我们需要算出：
1. 如何根据上次称量结果，找出本次待称量的是哪些球。
  
  假设上次称量结果为前两种情况（左边重或右边重，坏球在天平上），那么本次待称量的是天平上的球。假设上次称量的球数为K'，那么本次K = 2⌈ K'/3 ⌉. 如果上次称量结果为平衡，那么本次待称量即未上天平的第三堆，K = K' - 2⌈ K'/3 ⌉.
  
  后面会分析到，“实际称量球数是待称量球的一个子集（<= K）”。
2. 分堆。如何决定一个编号球被分到哪一堆，或者说每一堆里都有哪些编号球。
  
  决定了本次待称量的球，如何划分这些球才是核心问题，我们在下节讲解。

三堆

第一次称量没有上次参考。待称量的是所有的N个球。只需按照编号顺序划分为：

堆	球
左	1 ~ ⌈ N/3 ⌉
右	(⌈ N/3 ⌉ + 1) ~ 2⌈ N/3 ⌉
否	(2⌈ N/3 ⌉ + 1) ~ N

“否”表示未上天平的第三堆。第一次的分堆随意，只需要保证堆的数量即可。下面来考虑更一般的情况。

假设本次待称量的球已经确定，这些球除了编号这个属性外，还额外增加了一个新标签，即前面节第二条所述三种推理。那么如何描述新信息呢？

本次待称量的球可以用u、v两个子集合表示，注意这里的u、v集合不是分堆，而是用来辅助分堆。我们也可以说u、v里的球具有u、v标签。

u里的球，要么是标准球，要么是坏球并且比标准球重。
v里的球，要么是标准球，要么是坏球并且比标准球轻。

u、v的定义可以反过来，但不影响结果。最简单的u、v集合出现在第一次称量之后：

第一次称量左重，则左堆为u，右边堆为v。
第一次称量右重，则右堆为u，左边堆为v。

这时| u | = | v | = ⌈ N/3 ⌉. 但更一般情况如下：

u、v有可能为空集；
| u | ≠ | v |更常见；
经过几轮称量后，堆里会同时含有u、v中的球，这是为了便于排除更多的球。

假设上次称量的三堆如下：

堆	u', v'	数量
左	u₁', v₁'	⌈ K'/3 ⌉
右	u₂', v₂'	⌈ K'/3 ⌉
否	v' 余下	K' - 2⌈ K'/3 ⌉

表格用逗号来分隔u/v集合，便于识别。

如果左重，那么u = u₁'，v = v₂'；u₂'、v₁'被排除；
如果右重，那么u = u₂'，v = v₁'；u₁'、v₂'被排除；
如果平衡，那么u = ∅，v = v' - v₁' - v₂'引入一个标准球（左右堆里全是）。为何这里需要引入标准球？因为本次v' - v₁' - v₂'集合球标签单一，引入标准球寻求更多差异。

轻特别注意如何排除部分“待称量球”，“实际称量球数是待称量球的一个子集（<= K）”。

显然，对于第一次称量没有前次参考，u = v = ∅.

K = | u | + | v |. 那么把u、v里的球分成三堆呢？

把u集合的球平均分配到第一、二堆。如果不够⌈ K/3 ⌉，则
从v集合平均往第一、二堆放球到满。
把v里余下的球分到第三堆。明显不存在v集合取完后，还没有填完第一、二堆的情况。

至此分球完毕，用表格表示如下：

堆	u, v	数量
左	u₁, v₁	⌈ K/3 ⌉
右	u₂, v₂	⌈ K/3 ⌉
否	v余下	K - 2⌈ K/3 ⌉

开始本次称量。

引入标准球
1. u = ∅，或v = ∅，或 u = v = ∅，此时引入标准球替换一个待称量球，寻取差异。被替换的待称量球放在第三堆。
2. 一般发生在上次称量平衡时。但第一次称量时明显u = v = ∅. u = ∅, v ≠ ∅不常见，多出现在| u' | 和 | v' | 非常小时（如< =3）。
3. 如果 | u | 或 | v | 非常小（如3），根本不引入标准球。

N = 1、2、3

N <= 3的情况特殊，这里特别说明。

N = 1，不需要称，坏球肯定就那一个。
N = 2，没有参考，无法称量，
N = 3，n = 2。1号球左边，2号球右边，称量一次后用三号球替换2号球。
1. 左重，左重，1号坏球并且比标准球重；
2. 左重，平衡，2号坏球并且比标准球轻；
3. 左重，右重，不可能；
4. 平衡，左重，3号坏球并且比标准球轻；
5. 平衡，平衡，不可能；
6. 平衡，右重，3号坏球并且比标准球重；
7. 右重，左重，不可能；
8. 右重，平衡，2号球坏并且比标准球重；
9. 右重，右重，1号球号并且比标准球轻；

N = 4

n = ⌈ log₃(2*4+3) ⌉ = 3.

第一次称量，如上分析，没有参考，所以简单按编号分堆：

u' = v' = ∅.
⌈ K/3 ⌉ = ⌈ N/3 ⌉ = ⌈ 4/3 ⌉ = 2

堆	u, v	数量
左	1 2	2
右	3 4	2
否	∅	0

分析下次称量u、v集合：

左重，u = { 1, 2 }, v = { 3, 4 }.
右重，u = { 3, 4 }, v = { 1, 2 }.
平衡，不可能。

第二次称量：

K = | u | + | v | = 4，比较特殊K = K'. ⌈ K/3 ⌉ = ⌈ 4/3 ⌉ = 2

一左重：

堆	u, v	数量
左	1, 3	2
右	2, 4	2
否	∅	0

或一右重：

堆	u, v	数量
左	3, 1	2
右	4, 2	2
否	∅	0

注意，这次不是按照顺序分堆。逗号左边是u集合，右边是v集合。

根据第一、二次称量，分析下次称量u、v集合，实际有4重情况：

一左重，二左重，u = { 1, 2 } ∩ { 1, 3 } = { 1 }，v = { 3, 4 } ∩ { 2, 4 } = { 4 }.
一左重，二右重，u = { 1, 2 } ∩ { 2, 4 } = { 2 }，v = { 3, 4 } ∩ { 1, 3 } = { 3 }.
一左重，二平衡，不可能。
一右重，二左重，u = { 3, 4 } ∩ { 3, 1 } = { 3 }，v = { 1, 2 } ∩ { 4, 2 } = { 2 }.
一右重，二平衡，不可能。
一右重，二右重，u = { 3, 4 } ∩ { 4, 2 } = { 4 }，v = { 1, 2 } ∩ { 3, 1 } = { 1 }.

第二轮称量后，排除两个球，剩下两个球。

第三次称量：

经过第二次称量，u、v cardinality为1, K = 2 < 4属于特殊情况。我们不需要继续称量分析，而是直接引入标准球。被排除的两个球肯定是标准球，随便引入一个替换u、v中的球上天平即可。

堆	u, v	数量
左	1,	1
右	, 4 > 2	1
否	∅	0
堆	u, v	数量
左	2,	1
右	, 3 > 1	1
否	∅	0
堆	u, v	数量
左	3,	1
右	, 2 > 4	1
否	∅	0
堆	u, v	数量
左	4,	1
右	, 1 > 3	1
否	∅	0

上表重，分别引入了2、1、4、3标准球作为参考：

左重，1是坏球并比标准球重；平衡，4是坏球并比标准球轻；右重不可能。
左重，2是坏球并比标准球重；平衡，3是坏球并比标准球轻；右重不可能。
左重，3是坏球并比标准球重；平衡，2是坏球并比标准球轻；右重不可能。
左重，4是坏球并比标准球重；平衡，1是坏球并比标准球轻；右重不可能。

N = 12

这个过程非常复杂，在纸上演练吧。

firewalld

2017-09-20T00:00:00+00:00

firewalld and iptables
mask iptables service
Install
Status
Runtime and Permanent
service and port
Adding ports
Adding services
Create/modify services
Disallow Ports
Finally
Refs

firewalld and iptables

Firewalld stack
1. iptables CLI -> iptables daemon -> kernel netfilter -> xtables/nftables. We should know the difference between a service and shell command in terms of iptables.
2. firewall-cmd CLI -> firewalld daemon -> iptables/nft CLI. Before firewalld 0.6.0, the underlying is iptables, not it becomes nft. Here is an illustration of before 0.6.0:
  
  At the very top, sits the GUI tool.
Specially, firewalld introduces zone to defines the level of trust for a connection, IP source or interface, which resembles Microsoft Windows firewall. Rules are created within a zone. A zone is bound to one or more interfaces, IP sources or interfaces, but a connection, interface or IP source can only be part of one zone.

mask iptables service

Switch to firewalld.

~ # systemctl stop iptables-services
~ # sysremctl disable iptables-services
~ # systemctl mask iptables-services

Install

~ # yum install iptables firewalld
~ # systemctl enable firewalld
~ # systemctl start firewalld
~ # systemctl status firewalld

Don't break SSH connection as the default SSH port in default zone is 22 that is usually changed by administrator, otherwise you could no longer SSH into server.

Status

~ # firewall-cmd --reload
~ # firewall-cmd --state

# print zone names
~ # firewall-cmd --get-zones
# print all zone details
~ # firewall-cmd --list-all-zones
# default is 'pulbic'
~ # firewall-cmd --get-default-zone
# zones that has bindings
~ # firewall-cmd --get-active-zone

# print all service names
~ # firewall-cmd --get-services
# print a service detail
~ # firewall-cmd --info-service https

# print objects in a zone
~ # firewall-cmd --zone=public --list-services
~ # firewall-cmd --zone=public --list-ports
~ # firewall-cmd --zone=public --list-interfaces
~ # firewall-cmd --zone=public --list-sources
# print all objects in a zone
~ # firewall-cmd --zone=public --list-all
~ # firewall-cmd --info-zone public

Default zone is public to which unmatched traffic would be applied.

For use in public areas. You do not trust the other computers on networks to not harm your computer. Only selected incoming connections are accepted.
We activate a zone (--get-active-zones) by binding a network interface, source IP address range(s) or ports to it.

As firewalld sites on top of iptables, we can manipulate underlying iptables rules with firewall-cmd directly with the --direct option.

~ # firewall-cmd --direct --get-all-chains
~ # firewall-cmd --direct --get-chains nat

~ # firewall-cmd --direct --get-all-rules
~ # firewall-cmd --direct --get-rules nat

Attention please; do not use iptables command to manipulate firewalld rules as that would make things complicated.

Runtime and Permanent

When updating firewalld rules, option --permanent does not affect runtime rules but write to disk for accross boot.

When changing something, do not add --permanent option and add --timeout option to have a test, especially when you change the SSH port or disallow a port.
After that either execute the update again with the --permanent option or run firewall-cmd --runtime-to-permanent.
Reload rules into runtime if step 1 is skipped.

service and port

Services are pre-defined well-known ports like http, https etc.

Check /usr/lib/firewalld/services XML definitions. You shouldn't edit those files.

To edit a servce (i.e. change ssh port):

Copy /usr/lib/firewalld/services/ssh.xml to /etc/firewalld/services/; change port there.

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>SSH</short>
  <description>Secure Shell (SSH) is a protocol for logging into and executing commands on remote machines. It provides secure encrypted communications. If you plan on accessing your machine remotely via SSH over a firewalled interface, enable this option. You need the openssh-server package installed for this option to be useful.</description>
  <port protocol="tcp" port="12345"/>
</service>

Alternatively, adding port directly to default zone.

Adding ports

~ # firewall-cmd --zone=public --add-port=12345/tcp
~ # firewall-cmd --reload (opt)

~ # firewall-cmd --permanent --zone=public --add-port=12345/tcp

Take effect immediately at rumtine without reload.
As we have the runtime command at first, no --reload is required.
Take effect accross reload.
--zone can be ommited unless you want to change other zones.

Adding services

 ~ # firewall-cmd --permanent --zone=public --add-service=http --add-service=https

Create/modify services

Only a permanent service can be created.

~ # firewall-cmd --permanent --new-service-from-file=/path/to/service.xml --name=myservice (using an existing service)
# or
~ # firewall-cmd --permanent --new-service=myservice (create an empty service)
~ # firewall-cmd --info-service=myservice

Modify it after creation:

~ # firewall-cmd --permanent --service=myservice --set-description=description
~ # firewall-cmd --permanent --service=myservice --set-short=description
~ # firewall-cmd --permanent --service=myservice --add-port=portid[-portid]/protocol
~ # firewall-cmd --permanent --service=myservice --remove-port=portid[-portid]/protocol
~ # firewall-cmd --info-service=myservice

Adding the new service to a zone:

~ # firewall-cmd --permanent --zone=public --add-service=myservice

I think the easiest way is to copy an existing service XML to /etc/firewalld/services/myservice.xml and edit that file directly.

Disallow Ports

# drop zone
# test
~ # firewall-cmd --zone=drop --add-port=12345/tcp --timeout 5m
# permanent
~ # firewall-cmd --permanent --zone=drop --add-port=12345/tcp

# default zone
# test
~ # firewall-cmd --zone=public --remove-port=12345/tcp --timeout 5m
# permanent
~ # firewall-cmd --permanent --zone=public --add-port=12345/tcp

By default, the drop zone is not activated.

Finally

~ # firewall-cmd --reload

Refs

32-bit Android-x86 5.1-rc1

2017-08-25T00:00:00+00:00

This post describe customzing android-x86-5.1-rc1.iso and using tips. Regarding the installation, please refer to virtualbox post.

Customization

Download android-x86-5.1-rc1.iso.

virtual/cdrtools offers many tools to operates on ISO image, including mkisofs creating bootable ISO.

~ $ file android-x86-5.1-rc1.iso
~ $ isoinfo -pd -debug -i android-x86-5.1-rc1.iso
~ $ isoinfo -l -i android-x86-5.1-rc1.iso
~ $ iso-info -l -d ERROR -i android-x86-5.1-rc1.iso

Mount ISO.
```
~ # mkdir -p /mnt/android-x86
~ # mount -t loop /path/to/android-x86-5.1-rc1.iso /mnt/android-x86
```
Get system.sfs which is a Squashfs (compressed read-only for Linux) that containing the Android /system partition - our customzation target in this post.

Mount Squashfs.

~ # mkdir -p /mnt/squashfs
~ # mount -t squashfs -o loop /mnt/android-x86/system.sfs /mnt/squashfs
# or
# Make sure *sys-fs/squashfs-tools* is installed.
~ # unsquashfs [-ll] -d /mnt/squashfs /mnt/android-x86/system.sfs

There exists only one file system.img which is the image of original /system mountpoint of Android-x86. That is to say, the author first create Android /system partition as image and then compress it as Squashfs.

Mount image.
```
~ # mkdir -p /mnt/img
~ # mount -t ext4 -o loop /mnt/squashfs/system.img /mnt/img
# Either
~ # cp -a img /home/username/workspace/ (rsync -a img /home/username/workspace/)
~ # chown -R username: /home/username/workspace/img/
~ $ cd /home/username/workspace/img
# or
~ # cd /mnt
~ # cp squashfs/system.img .
~ # make temp_img
~ # mount -t ext4 -o loop,rw system.img temp_img
```
1. We cannot modify /mnt/img since it's directly mounted from read-only Squashfs.
2. Do NOT add trailing slash to /mnt/img when rsync.
Do customization here under /home/username/workspace/img. For example, remove LatinIME, SetupWizard etc. etc. Especially for build.prop:
```
ro.product.locale.language=zh
ro.product.locale.region=CN
# ro.product.cpu.abilist64=x86_64, arm64-V8A,x86,armeabi-v7a, armeabi
dalvik.vm.heapstartsize=8m
dalvik.vm.heapgrowthlimit=150m
dalvik.vm.heapsize=256m
# ro.com.google.gmsversion=5.1_r2
# ro.com.google.clientidbase=android-asus
# ro.com.google.clientidbase.ms=android-asus
# ro.com.google.clientidbase.am=android-asus
# ro.com.google.clientidbase.gmm=android-asus
# ro.com.google.clientidbase.yt=android-asus
# ro.setupwizard.mode=OPTIONAL (DISABLED, or REQUIRED)
dalvik.vm.execution-mode=int:fast
ro.sf.lcd_density=130
```
1. For 64-bit, arm64-V8A precdes x86 in abilist.
2. DPI can be changed on-the-fly; please read the Tips part below.
system/app:

BasicDreams/ Bluetooth/ Browser/ Calculator/ Calendar/ Camera2/ CaptivePortalLogin/ ConfigUpdater/ DeskClock/ Email/ Exchange2/ Galaxy4/ Gallery2/ Gmail2/ GoogleC* HTMLViewer/ HoloSpiralWallpaper/ KeyChain/ LatinIME/ LiveWallpapers* Music2/ NotePad/ PhaseBeam/ PicoTts/ PrintSpooler/ QuickSearchBox/ RSSReader/ SoundRecorder/ UserDictionaryProvider/ VisualizationWallpapers/ YouTube/

system/priv-app:

CalendarProvider/ Contacts* FusedLocation/ GmsCore/ Google* ManagedProvisioning/ MusicFX/ Phonesky/ TSCalibration2/ WallpaperCropper/

/fonts: all NotoSans*.ttf except NotoSansSymbols-Regular-Subsetted.ttf.

usr/idc: refer to Double Mouse below.

Customization finished. Then we should re-pack the ISO file.
Do NOT unmount.

If you choose to the alternative method (mounting system.img file as rw), do not umount now. Upon custiomization finished, intuitively, we should umount ~/workspace/img, which is fine but far from perfect. For example, we cannot resize (read the next step -l 700M) image although we removed bunch of built-in apks.

What we should do is to creating a new image.

Require furtuer verification.
make_ext4fs and simg2img

To continue the following procedures, we must manually compile make_ext4fs and/or simg2img from Android-x86-5.1-rc1 source. Binaries got from web would probably be troublesome. For example, make_ext4fs comes out of AOSP 6.0.1 source, make_ext4fs compiled from which would cause Android-x86-5.1-rc1 losing network and SuperSU.

When creating image file, make_ext4fs sets file permissions in accord with core/include/private/android_filesystem_config.h irrepsective of that in working directory. Therefore, file permissions is fixed in make_ext4fs binary. That is the reason we must compile make_ext4fs from the correct Android source of your image.

To compile make_ext4fs we should get source of core, extras and additionally zlib and libselinux. Android-x86 fetch the later two from official AOSP
```
~ $ mkdir tools
# remove everything except *core/{include/private,libsparse}*
~ $ git clone --depth=1 -b lollipop-x86 https://git.code.sf.net/p/android-x86/system_core core
# remove everything except *extras/ext4_utils*
~ $ git clone --depth=1 -b lollipop-x86 https://git.code.sf.net/p/android-x86/system_extras extras
~ $ git clone --depth=1 -b android-5.1.1_r30 https://android.googlesource.com/platform/external/zlib zlib
~ $ git clone --depth=1 -b android-5.1.1_r30 https://android.googlesource.com/platform/external/libselinux libselinux
```
Makefile:
```
SELIB = libselinux
ZLLIB = zlib/src
SPLIB = core/libsparse
COLIB = core
MALIB = extras/ext4_utils

all:
    $(MAKE) -C $(SELIB)/src all
    $(MAKE) -C $(ZLLIB) all
    $(MAKE) -C $(SPLIB) libsparse.a
    $(MAKE) -C $(MALIB) all
    mv $(MALIB)/make_ext4fs make_ext4fs

clean:
    $(MAKE) -C $(SELIB)/src clean
    $(MAKE) -C $(ZLLIB) clean
    $(MAKE) -C $(SPLIB) clean
    $(MAKE) -C $(MALIB) clean
```
Then we get make_ext4fs binary. For simg2img, just grab one from web since it's compatibable.
1. make_ext4fs
2. simg2img
3. android_img_repack_tools which is recommended.
4. Howto: compile mkbootimg/mkbootfs/make_ext4fs on OS X
5. UnPack / rePack android img with Ubuntu simg2img/make_ext4fs

New image

# Create *sparse* format image
~ $ make_ext4fs -T 0 -l 700M -s -a system system_sparse.img img
# Convert Android sparse image to raw image.
~ $ simg2img system_sparse.img system_raw.img (If`-s` supplied to *make_ext4fs*)
~ $ file system_sparse.img system_raw.img;
~ $ mv system_raw.img system.img

You are recommended to omit -s argument and simg2img as sparse image format is for Android but Android-x86 (Android-x86 uses Linux RAW image).

'l': new smaller image size.
's': sparse mode to compress image file.
'a': mount point at Android.
Check the new image file size, it's exactly 700M.
'T': time 0 means 1970-1-1.

New Squashfs

We cannot mount Squashfs as rw. Must create new Squashfs like image above.
```
~ $ mksquashfs system.img system.sfs -force-uid 1001 -force-gid 1001
```
Re-pack BootableCDsForBIOSAndUEFI

We cannot mount ISO as rw. Though we can use Archive manager to modify ISO contents, bootable information gets lost meanwhile, resulting in ISO unbootable.
```
~ # rsync -a --exclude="system.sfs" /mnt/android-x86 /home/username/workspace/ (cp -a android-x86 temp_android-x86)
~ # chown -R username: /home/username/workspace/android-x86
~ $ cp -a system.sfs android-x86/
~ $ cd android-x86/
~ $ mkisofs -A "Android-x86-5.1-rc1" -V "Android-x86-5.1-rc1" -volset "Android-x86-5.1-rc1" -input-charset utf-8 -o ../android-x86.iso -JlrTUv -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -b boot/grub/efi.img -no-emul-boot ./
```
1. Use -r instead of -R. -r means that we want to use Rockridge extensions. The -R command line option also achieves this but in a less satisfactory way. -R maintains the UID and GID of each file's owner. The same UID and GID will more than likely be meaningless once the files are transported over to another system, so ownership of all files in the ISO image is set to root:root if we use -r instead of -R. Also, -R preserves file permissions, including the write flags. Having any of the write flags set for a file on a read-only media doesn't make much sense, so -r resets it in the filesystem image.
2. -U may need removed.
3. New mkisofs version abandons -e option.
4. Use -m and -x now works identically to exlucde leftovers.
5. Android-x86-5.1-rc1 (32 bits) ISO file does not have efi.img.
Refs

Tips

By switching to virtual terminal (F1 - F6), we get root shell.
If you are away from VB, the right menu (between the right Alt and right Ctrl keys), ESC, left, right, up and down keys wakes up Android-x86 from powersave mode.

Hostonly

hostonly might not able to ping guest while the reverse direction works. Probably, check both the host and guest arp table.
1. I previously set dhcpcd.conf with noarp, resulting outdated arp table (also try arp -d/a).
2. Dunno why, hostonly networking is unreliable, ocassionally cannot ping.

adb

~ # adb tcpip 5555 (on android-x86)
~ # adb -e connect 192.168.56.101:5555
~ # adb root
~ # adb -e connect 192.168.56.101:5555
~ # adb shell

Shutdown

# https://sourceforge.net/p/android-x86/kernel/ci/kernel-4.4/tree/kernel/reboot.c
# Alt-F1
~ # reboot -f/p (shutdown)
~ # reboot (reboot)

Reset

# Alt-F1
~ # rm -R /data

DPI

~ $ su
~ # wm density
~ # wm density 200 && reboot

Adjust the value as you wish.

Double Mouse

To remove the duplicate mouse issue, firstly, set VM mouse device to usbtablet:

~ $ vboxmanage modifyvm Android-x86 --mouse usbtablet

You may found one of the mouse changes to draggable circle. Then, we further create an Input Device Configuration file for the usbtablet.

Get into Android terminal:

~ $ su
~ # cd /system/usr/idc
~ # dumpsys input | less

From the output of dumpsys input you can easily find the Virtualbox Usbtablet device however the devicetype is pointer instead of correct touchscreen. What we do is to create a configureation under /system/usr/idc to tell Android how to use the device correctly.

The configureation file can be grabbed from Android-x86-4.4-r5.iso system (/system/usr/idc/GenericTouch.idc). If you want to skip all that trouble, here is it:

# Copyright (C) 2011 The Android-x86 Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Basic Parameters
touch.deviceType = touchScreen
#touch.orientationAware = 1

# Touch Size
touch.touchSize.calibration = default

# Tool Size
touch.toolSize.calibration = default

# Pressure
touch.pressure.calibration = default

# Size
touch.size.calibration = default

# Orientation
touch.orientation.calibration = none

device.internal = 1

We should change the filename to Vendor_XXXX_Product_XXXX_Version_XXXX.idc as directed, where XXXX comes out of dumpsys input output.

Put the newly created file to /system/usr/idc. Attention, make sure the file has read permission.

~ # chmod 644 Vendor_XXXX_Product_XXXX_Version_XXXX.idc

Skip SetupWizard

The first time Android starts up, you might fail to bypass Google account setup even Wi-Fi setup is skipped. This was the VM NAT does connect to the Internet but Android think it's Wi-Fi, and meanwhile Google was blocked, which make Android try to connect to the fake Wi-Fi endlessly. The solution:

Click the upleft, upright, downright, and downleft corners of Android's effective launcher (not your computer screen) sequentially. That's it.

Another way is turn off NAT temporarily.

~ $ vboxmanage controlvm Android51 setlinkstate1 off

Resolution

Set a proper resolution value make Android-x86 run far more smoothly.

Create temporary Grub menu

During booting, e enters edit mode; e edits selected kernel line; append vga=ask; enter; b to boot; "ENTER" to see available resolution; scan to include self-defined value (next section below). Each resolution has hex number prefix. For example, 800x600 responds to 0x314 (decimal 788).

Input one of the hex prefix to test. If none of the default resolution fits reuirement, define one by vboxmanage.

Self-defined resolution

~ $ VBoxManage setextradata Android-x86 "CustomVideoMode1" "405x720x16"
~ $ VBoxManage setextradata Android-x86 "CustomVideoMode2" "800x600x16"
~ $ VBoxManage setextradata Android-x86 "CustomVideoMode2" (remove this entry)

For Android 6.0 and above, use 32-bit color instead of 16.
Add more by CustomVideoMode2, CustomVideoMode3 etc.
This command will update *.vbox file.
Type scan to show self-defined value and relavent hex prefix.

Debug mode

During boot, entering Grub debug mode,
```
~ # mount -o remount,rw /mnt
~ # cd /mnt/grub
~ # vi menu.lst
```
Add a boot entry, adding vga=desired-decimal-prefix or UVESA_MODE=desired-resolution to kernel line.

vga value in menu.lst should be decimal while in temporary boot edit mode be hex.
Use wm size checks resolution.
wm overscan 3,4,3,4 simulates mobile edge margins.

Nextcloud

2017-08-05T00:00:00+00:00

Nextcloud is open source platform serving file (private cloud disk), contacts, calendar synchronization ans sharing. By Nextcloud, you control your information, being not subject to privacy leak.

Requirements -LNMP

L: CentOS 7 64-bit;
N: Nginx;
M: MariaDB;
P: Php-FPM.

Notes

Administration work can be done:

Web UI;
occ;
Edit configuration file manually.

MariaDB

~ # yum install mariadb mariadb-server
~ # systemctl status mariadb
~ # systemctl start mariadb
~ # systemctl enable mariadb

MariaDB configuration

Create password for MariaDB root user:

~ # mysql_secure_installation
Enter current password for root (enter for none): # just ENTER
Set root password? [Y/n] Y
New password: 
Re-enter new password: 
Remove anonymous users? [Y/n] Y
Disallow root login remotely? [Y/n] Y
Remove test database and access to it? [Y/n] Y
Reload privilege tables now? [Y/n] Y

With the MariaDB root password just set, now we can login to the mysql shell to create a database and a user for Nextcloud.

~ # mysql -u root -pPassword
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 5.5.52-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> help

Now creating database and user with SQL within SQL shell:

create database nc_db
# or
create database nc_db default character set utf8mb4 collate utf8mb4_unicode_ci;
create user nc_user@localhost identified by 'password';
grant all privileges on nc_db.* to nc_user@localhost identified by 'password';
flush privileges;
show databases;
show tables from nc_db;
select user,host,password,plugin from mysql.user;
show grants for nc_user@localhost;
quit;

Create a database and user for Nextcloud service. Grant all privileges of the database to the user.
If the user account does exist and you provide the IDENTIFIED BY clause, the user's password will be changed.

So use the same 'password' value in case of losing it.

Php-fpm

Make sure webtatic repository is enabled and check webtatic packages.

~ # yum --enablerepo=webtatic install php70w-fpm php70w-common
~ # yum --enablerepo=webtatic install php70w-ctype php70w-dom php70w-gd php70w-iconv php70w-json php70w-libxml php70w-mbstring php70w-posix php70w-simplexml php70w-xmlreader php70w-xmlwriter php70w-zip php70w-zlib
~ # yum --enablerepo=webtatic install php70w-mysql php70w-curl php70w-intl php70w-mcrypt php70w-opcache php70w-pecl-apcu
~ # php -v/i
~ # php -m | grep -i apcu

Some Php modules (i.e. php-70w-curl) does not exist but enclosed by another module (i.e. php70w-common).
opcache module won't be installed without explicit pointing out, which would cause APCu memory exhausted issue.

Php-fpm configuration

Edit Php-fpm configuration file:

# /etc/php-fpm.d/www.conf

user = nginx
group = nginx

listen = 127.0.0.1:9000

env[HOSTNAME] = $HOSTNAME
env[PATH] = /usr/local/bin:/usr/bin:/bin
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp

Next, create a new directory for the session path in the /var/lib/ directory, and change the owner to the nginx user:

~ # mkdir -p /var/lib/php/session
~ # chown nginx:nginx -R /var/lib/php/session/

Launch nginx and php-fpm:

# php-fpm
~ # systemctl status php-fpm
~ # systemctl enable php-fpm
~ # systemctl start php-fpm
# nginx
~ # systemctl status nginx
~ # systemctl enable nginx
~ # systemctl start nginx

There existed an error when starting php-fpm:

Aug 05 05:52:29 localhost php-fpm[7613]: [05-Aug-2017 05:52:29] NOTICE: PHP message: PHP Fatal error:  PHP Startup: apc_shm_create: shmget(0, 67108864, 914) failed: Invalid argument. It is possible that the chosen SHM segment size is higher than the operation system allows. Linux has usually a default limit of 32MB per segment. in Unknown on line 0
Aug 05 05:52:29 localhost php-fpm[7613]: [05-Aug-2017 05:52:29] NOTICE: PHP message: PHP Fatal error:  PHP Startup: apc_shm_attach: shmat failed: in Unknown on line 0

That was caused by shm size (32M) of CentOS 7 kernel much smaller than that (64M) of Php APC user cache (APCu) setting. Kernel's shm value can be verified by

~ # sysctl -a | grep -i shm
# kernel.shmmax = 33554432
# kernel.shmall = 2097152
# kernel.shmmni = 4096
# or
~ # cat /proc/sys/kernel/shmmax

The upper bond here is 33554432 Bytes (32M). However, the value of APCu shm size is:

# /etc/php.d/apcu.ini
# The size of each shared memory segment with M/G suffix.
apc.shm_size=64M

So we can either cut down APCu shm size or increase that of kernel:

~ # sysctl -w kernel.shmmax=67108864
# To be permanent across reboot:
# vi /etc/sysctl.d/15-shmmax.conf
kernel.shmmax = 67108864
~ # sysctl -f /etc/sysctl.d/15-shmmax.conf

Install Nextcloud tarball

~ # wget https://download.nextcloud.com/server/releases/nextcloud-12.0.1.tar.bz2
~ # yum install bzip2
~ # tar -xjpvf nextcloud-12.0.1.tar.bz2 -C /usr/share/nginx/html/
~ # chown nginx:nginx -R /usr/share/nginx/html/nextcloud/

Create data folder of Nextcloud outside of web root:

~ # mkdir -p /opt/nextcloud-data
~ # chown nginx:nginx -R /opt/nextcloud-data/

Nginx configuration for Nextcloud

~ # cd /etc/nginx/conf.d/
~ # vim nextcloud.conf

Refer to Nginx Configuration:

Check server_name, root, ssl_certificate and ssl_certificate_key etc. directives.
If the host posesses IPv6 address, add directives listen [::]:80; and listen [::]:443 ssl http2;.

# /etc/nginx/conf.d/nextcloud.conf

upstream php-handler {
    server 127.0.0.1:9000;
    #server unix:/var/run/php5-fpm.sock;
}

server {
    listen 80;
    listen [::]:80;
    server_name cloud.example.com;
    # enforce https
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name cloud.example.com;

    ssl_certificate "/etc/letsencrypt/live/cloud.example.com/fullchain.pem";
    ssl_certificate_key "/etc/letsencrypt/live/cloud.example.com/privkey.pem";

    # Add headers to serve security related headers
    # Before enabling Strict-Transport-Security headers please read into this
    # topic first.
    # add_header Strict-Transport-Security "max-age=15768000;
    # includeSubDomains; preload;";
    #
    # WARNING: Only add the preload option once you read about
    # the consequences in https://hstspreload.org/. This option
    # will add the domain to a hardcoded list that is shipped
    # in all major browsers and getting removed from this list
    # could take several months.
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # Path to the root of your installation
    root /usr/share/nginx/html/nextcloud/;

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    # The following 2 rules are only needed for the user_webfinger app.
    # Uncomment it if you're planning to use this app.
    #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
    #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
    # last;

    location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }
    location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

    # set max upload size
    client_max_body_size 512M;
    fastcgi_buffers 64 4K;

    # Enable gzip but do not remove ETag headers
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    # Uncomment if your server is build with the ngx_pagespeed module
    # This module is currently not supported.
    #pagespeed off;

    location / {
        rewrite ^ /index.php$uri;
    }

    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
        deny all;
    }
    location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
        deny all;
    }

    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param HTTPS on;
        #Avoid sending the security headers twice
        fastcgi_param modHeadersAvailable true;
        fastcgi_param front_controller_active true;
        fastcgi_pass php-handler;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
    }

    location ~ ^/(?:updater|ocs-provider)(?:$|/) {
        try_files $uri/ =404;
        index index.php;
    }

    # Adding the cache control header for js and css files
    # Make sure it is BELOW the PHP block
    location ~ \.(?:css|js|woff|svg|gif)$ {
        try_files $uri /index.php$uri$is_args$args;
        add_header Cache-Control "public, max-age=15778463";
        # Add headers to serve security related headers (It is intended to
        # have those duplicated to the ones above)
        # Before enabling Strict-Transport-Security headers please read into
        # this topic first.
        # add_header Strict-Transport-Security "max-age=15768000;
        #  includeSubDomains; preload;";
        #
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header X-Robots-Tag none;
        add_header X-Download-Options noopen;
        add_header X-Permitted-Cross-Domain-Policies none;
        # Optional: Don't log access to assets
        access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg)$ {
        try_files $uri /index.php$uri$is_args$args;
        # Optional: Don't log access to other assets
        access_log off;
    }
}

Let's Encrypt Certificate

We should has SSL certificate to domain cloud.example.com.

~ # certbot certonly --cert-name cloud.example.com --webroot -w /usr/share/nginx/html/ -d cloud.example.com --dry-run

Launch Nextcloud

After uploading the configuration to /etc/nginx/conf.d/nextcloud.conf, check syntax and reload nginx:

~ # nginx -t
~ # systemctl reload nginx
# or
~ # nginx -s reload

Enhancements

Nextcloud supports loading configuration parameters from multiple files. You can add arbitrary files ending with *.config.php in the config/ directory, for example you could place your email server configuration in email.config.php.

During configuration, some variables deserve special attention: Php memory_limit, APCu shm size, etc.

Reset admin password.

Nextcloud installation wizard does not ask for password confirmation. If you cannot remember the correct password, please reset it.
```
~ # su -s /bin/bash -c "php /usr/share/nginx/html/nextcloud/occ user:resetpassword admin-bob" nginx
```

HTTP Strict Transport Security - HSTS

# /etc/nginx/conf.d/nextcloud.conf
# https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;

OPcache

On the admin panel, you might recived message:

The PHP Opcache is not properly configured. For better performance we recommend ↗ to use following settings in the php.ini

First make sure Php opcache module is installed by:
```
~ # php -m | grep -i opcache
~ # yum install php70w-opcache
```
Then update /etc/php.d/opcache.ini as instructed:
```
# /etc/php.d/opcache.ini
   
opcache.enable=1
opcache.enable_cli=1
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000
opcache.memory_consumption=128
opcache.save_comments=1
opcache.revalidate_freq=1
```
Updates to Php *.ini configuration file require reloading php-fpm service.
Php memory limit

memory_limit in default /etc/php.ini is 128M. However per-site value of Nextcloud in .user.ini is 512M that is the whole available amount. To prevent Php from exhausting too much memory, set to 128M instead.
```
# /usr/share/nginx/html/nextcloud/.user.ini
upload_max_filesize=100M
post_max_size=100M
memory_limit=128M
```
1. Per-site Php setting overrides that of global under /etc.
2. It does not require php-fpm reloading.
3. Make sure upload_max_filesize and post_max_size value is smaller than memory_limit.
Local user (APCu) and Distributed cache (Redis)

Php APCu module is installed previously, then enable it in config.php:
```
# /usr/share/nginx/html/nextcloud/config/personal.config.php
'memcache.local' => '\OC\Memcache\APCu',
```
1. Updates to *.php requires NO service reloading (just refreshing web page) except installing new Php modules.
2. As I use nextcloud for private useage, distributed cache (Memcache/Redis) is not a must.

Redis as file locking backend

Though Redis as distributed cache is not a must, however Redis cache can also act as Transactional File Locking backend. File locking is enabled by default, using the database locking backend. This places a significant load on your database.

Using memcache.locking relieves the database load and improves performance.

~ # yum install redis php70w-pecl-redis
~ # systemctl status redis
~ # systemctl enable redis
~ # systemctl start redis

Add the following to config.php:

# /usr/share/nginx/html/nextcloud/config/personal.config.php

'filelocking.enabled' => true,
'memcache.locking' => '\OC\Memcache\Redis',
'redis' => [
  'host' => 'localhost', // can also be a unix domain socket: '/tmp/redis.sock'
  'port' => 6379,
  'timeout' => 0.0,
  'password' => 'dunno', // Optional, if not defined no password will be used.
  'dbindex' => 0, // Optional, if undefined SELECT will not run and will use Redis Server's default DB Index.
],

For enhanced security, please set a password for Redis cache. Reload php-fpm and refresh web page.

Disable thumbnailer.

By default, thumbnailer is enabled for images, mp3, etc., which however impacts overhead on system resources. Without previews, Nextcloud runs fairly smoothly.
```
# /usr/share/nginx/html/nextcloud/config/personal.config.php

'enable_previews' => false,
```
The side effect of disabling previews is Gallery app no longer show any media files:

No media files found

Why not disable Gallery as well?

Email notification.

TLS (STARTTLS) with 587, SSL (SSL/TLS) with 465.

# /usr/share/nginx/html/nextcloud/config/personal.config.php

"mail_smtpmode" => "smtp",
"mail_smtpsecure" => 'tls',
"mail_smtpport" => 587,
"mail_smtphost" => "smtp-mail.outlook.com",
"mail_domain" => "outlook.com",
"mail_from_address" => "valid-alias",
"mail_smtpauth" => true,
"mail_smtpauthtype" => "LOGIN",
"mail_smtpname" => "email@outlook.com",
"mail_smtppassword" => "email-password",
"mail_smtptimeout" => 30,

Outlook only accept valid mail_from_address owned by mail_smtpname. Fake mail_from_address or mail_domain cannot authenticate to the Outlook SMTP server.
If a malware or SPAM scanner is running on the SMTP server it might be necessary that you increase the mail_smtptimeout value.

4-byte UTF-8 support

Read No FILE_FORMAT and backup database before going on!

maintenance mode

~ # su -s /bin/bash -c "php ./occ maintenance:mode --on" nginx

InnoDB

/etc/my.cnf.d/4-byte.cnf
[client]
default-character-set = utf8mb4

[mysql]
default-character-set = utf8mb4

[mysqld]
innodb_large_prefix = 1
innodb_file_format = barracuda
innodb_file_per_table = 1
character-set-client-handshake = 0
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

Restart Mariadb
```
~ # systemctl restart mariadb
```

4-byte UTF-8

~ # mysql -u nc_db -pPassword

Modify database:

ALTER DATABASE nc_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
use nc_db;
# For each table in nc_db:
ALTER TABLE <table_name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
# For each column in <table_name>:
ALTER TABLE <table_name> MODIFY <column_name> VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci;

mysql.utf8mb4 in config.php

su -s /bin/bash -c "php ./occ config:system:set mysql.utf8mb4 --type boolean --value=true" nginx
su -s /bin/bash -c "php ./occ maintenance:mode --off" nginx
su -s /bin/bash -c "php ./occ maintenance:repair" nginx
su -s /bin/bash -c "php ./occ maintenance:mode --on" nginx

Attention: turn off maintenace mode before repair since it requires apps be loaded.

Data folder

The previous setup use a separate data directory /opt/nextcloud-data for files. If you'd like to switch to another location (i.e. /usr/local/data), use php ./occ files:scan --all.

Firstly, move data folder to the new location:

~ # mv /opt/nextcloud-data /usr/local/data
~ # chown -R nginx:nginx /usr/local/data

Afterwards, tell Nextcloud where the new location is:

# vi /usr/share/nginx/html/nextcloud/config/personal.config.php
'datadirectory' => '/usr/local/data',

Finally, refresh Nextcloud by occ:

~ # su -s /bin/bash -c "php ./occ files:scan --all" nginx

Attention: with the help of occ files:scan --all, we can modify (add, remove files) data directory directly without the interfence of Nextcloud web or clients. For example, you have a big video file on server, and have to fetch the video file to local PC and then synchronize through Nextcloud client. Alternatively, just copy the video file to /opt/nextcloud-data/username/files/Documents and run the occ command above.

DAV

CardDAV

App Contacts 1.5.3 by default does not create default address-book upon installation. Before creating or importing any contacts, an address-book must be created at first.
Importing address book from file works only partially because import breaks process limit in shared hosting environment.

Contact could not be created.

Further checking web server log, I found this:

10 worker_connections are not enough while connecting to upstream, client: 192.168.0.100, server: cloud.example.com, request: "PUT /remote.php/dav/addressbooks/users/jim/test-50/12345678-f873-4f3b-861d-6a6b5b78e5a5.vcf HTTP/1.1", upstream: "fastcgi://127.0.0.1:9999", host: "cloud.example.com"
1. If you have a big contacts vcard file to be imported, cut it into smaller blocks (i.e. every 100 contacts) and import one by one.
2. Increase web server's allowed concurrent connections, take Nginx for example:
```
# /etc/nginx/nginx.conf

events {
worker_connections 1024;
}
```
  Set back to original value once imported.
contact birthdays with no year given are shown with (*1604) in calendar.

The contacts app is using VCARD 3.0 at the moment.

CalDVA

Calendar app may remind incompatible ics format, then import without specifying an existing calendar and use New Calendar instead.
Add or subscribe to 中国农历.

Currently (as of Nextcloud 12), subscribed iCal web link would not be synced to mobile (i.e. Davdroid).
1. Use ICSdroid to subscribe the original link or to Nextcloud's subscription link.
2. As 中国农历 writes, use the tool to generate lunar for incoming years, which won't consume too much disk space. Afterwards, import that as a new calendar instead of a subscription.
3. Wait for Nextcloud 14.
Obviously, the 2nd method is favorable.

DAVDroid / Etar

When adding Davdroid account, select,

Contact group method: groups are per-contact categories

as Nextcloud Contact app does not support VCARD 4.0.

This is NOT true when DAVDroid upgrades to 1.11.4.1-ose and choose,

Groups are separate VCards
Davdroid does not sync at all. For synchronization, must turn on autostart for Davdroid in system setting.
If Davdroid address book or calendar does not show up stock Contacts/Calendar apps, try True Contacts and Etar instead.

Etar has, by default, a special calendar called 'Contact Birthday nobody@gmail.com' which show local contacts' birthday. *Please disable it's synchronization under 'CALENDARS TO SYNC', otherwise it incurs duplicate birthday events.

Maintenance

   ~ $ su -s /bin/bash -c "php ./occ dav:sync-birthday-calendar" nginx
   ~ $ su -s /bin/bash -c "php ./occ dav:sync-system-addressbook" nginx

For example, you accidently delete the default 'Contact birthdays' subscription, the first command will get it back.

Notes

Evernote has now gone too far, leaving its users behind. I want to just abandon it. But by far, neither Nextcloud official Notes app (I use this one) nor unofficial Nextnotes support importing Evernote notes.

However, we have QOwnNotes as provisional tool to transmitting Evernotes to Nextloud.

Install QOwnNotes and Nextcloud on your system.
Install and enable QOwnNotesAPI (optional)
Add Nextcloud account to QOwnNotes client. (optional)
IMPORTANT! Set QOwnNotes note folder path to be that of Nextcloud.
Export Evernote notebook to ENEX format .enex.
Import the .enex in QOwnNotes.
Sync the notes in Nextcloud client.

The 2nd and 3rd steps are unneccessary since we won't use QOwnNotes after the transmission. Pay attention to the role of QOwnNotesAPI. Normal note files are handled by Nextloud while note transhes and versions (on the server side) are handled by QOwnNotesAPI separately.

Bug

A bug of Notes app in web interface

By default, the first line of a note will be the filename. Maybe, the Notes app in web interface cannot create the right file name with full shape punctuators or Chinese characters.

The current solution is to:

Disable Notes app;
Backup and remove the Files - Notes folder;

Specifically remove the troublesome new note file.
Re-enable Notes app;
Apply No. 125 patch.
Move back the Notes folder
Do a occ files:scan --all and reload Notes app.

To further fix broken activity app, we should remove the troublesome activity records:

mysql -u nc_db -pPassword
select activity_id,type,app,subject,subjectparams,file,link from nc_db.oc_activity order by activity_id DESC limit 10;
delete from nextcloud.oc_activity where activity_id>2300;

Adjust the activity_id range.

News

After enabling News app, you receives warnings like:

Ajax or Web cron mode detected! Your feeds will not updated!

By default, Nexcloud Ajax is used. For news to be updated automatically, we should use system cron service. If you don't care about automatic news fetch, just ignore that.

Firstly, install cronie service:

~ # yum install cronie
~ # systemctl enable crond
~ # systemctl start crond
~ # cat /etc/crontab

Then, we add cron job for user nginx:

~ # crontab -u nginx -e
*/15  *  *  *  * php -f /var/www/nextcloud/cron.php
~ # crontab -u nginx -l
~ # cat /var/spool/cron/nginx

Finally, select the option Cron in the admin menu for background jobs. If left on AJAX it would execute the AJAX job on every page load.

As the reference pointed out, we can use systemd timer as well. It's all your choice.

Tips

The Android app requires autostart and storage permissions to sync feeds.

Backup

Do it before upgrading.

Mainly, we should backup folders and database.

folders

The config folder, i.e. config.php, personal.config.php etc.
The data folder. The most important part.
The theme folder.
During upgrading, the Nextcloud installation folder is backed up automatically. If you trust it, that's fine.

~ # rsync -avzP --delete nextcloud/ nextcloud-dirbkp_`date +"%Y%m%d"`/ 
~ # rsync -avzP --delete nextcloud-data/ nextcloud-data-dirbkp_`date +"%Y%m%d"`/ 

Optionally, we can synchroize data to PC just in case.

database

# backup
mysqldump --single-transaction -h localhost -u nc_user -pPassword nc_db > nextcloud-sqlbkp_`date +"%Y%m%d"`.bak
# restore
drop database nc_db
create database nc_db
# -or-
create database nc_db default character set utf8mb4 collate utf8mb4_unicode_ci";
~ # mysql -h localhost -u nc_user -pPassword nc_db < nextcloud-sqlbkp.bak

There is no space between -p and Password.

Update

Firstly, updater checks environment.

updater enables maintenance mode automatically.
Secondly, upgrade does the real update.
Both tools could be executed through the web interface and/or command line.

Web-based updater would ask you if Keep maintenance mode active. Command line upgrade and web-based upgrade require active (enabled) and inactive (disabled) maintenance mode respectively.
We may execute upgrade twice: one for Nextcloud itself and the other for apps.

Here, I use command line for the whole update process.

~ # cd /usr/share/nginx/html/nextcloud/
~ # su -s /bin/bash -c 'php ./updater/updater.phar' nginx
# if errors like "The following extra files have been found: .user.ini.bak", remove those files. Re-run:
~ # su -s /bin/bash -c 'php ./updater/updater.phar' nginx
# Should the "occ upgrade" command be executed? [Y/n] Y
# Keep maintenance mode active? [y/N] N
~ # su -s /bin/bash -c 'php occ status' nginx
~ # su -s /bin/bash -c 'php occ upgrade' nginx (optionally)
~ # su -s /bin/bash -c 'php ./updater/updater.phar' nginx

Please execute occ upgrade when prompted by occ status, which is usually required when new app versions are available.

miantenance mode

maintenance:mode locks the sessions of logged-in users and prevents new logins. This is the mode to use for upgrades. Check the mode by:

# "maintenance" value in 'config.php' is updated accordingly
~ # su -s /bin/bash -c "php ./occ list" nginx
~ # su -s /bin/bash -c "php ./occ maintenance:mode" nginx
~ # su -s /bin/bash -c "php ./occ maintenance:mode --on/off " nginx

Domain switch

Suppose you would moving from cloud.old.com to cloud.new.com, then remember to modify trusted_domains and overwrite.cli.url in config/config.php.

   'trusted_domains' =>
   array (
     0 => '192.168.0.29',
     1 => 'cloud.new.com',
   ),
   'overwrite.cli.url' => 'https://cloud.new.com',

This can be done by occ as well:

~ # su -s /bin/bash -c "php ./occ config:system:get trusted_domains" nginx
~ # su -s /bin/bash -c "php ./occ config:system:set trusted_domains 1 --value=cloud.new.com" nginx
~ # su -s /bin/bash -c "php ./occ config:system:get overwrite.cli.url" nginx
~ # su -s /bin/bash -c "php ./occ config:system:set overwrite.cli.url --value=https://cloud.new.com" nginx

Refs

simple-obfs

2017-05-07T00:00:00+00:00

ABC

As ptproxy is almost dead and shadowsocks advances further, we require a newer traffic obfuscation tool, where simple-obfs comes to sight.
simple-obfs is similar to Tor Pluggable Transport (i.e. obfs4proxy) in disguising abnormal traffic as daily HTTP(s), thus free from firewall interference.
This post prefers standalone mode.
The author does not provide information on configuration file format.

Please check jconf_t * read_jconf (const char *file) function within src/jconf.c.

Build

I will show how to build binary in Gentoo. Before that, make sure dependency packages are installed.

~ $ cd ~/opt/
~ $ git clone --depth=1 https://github.com/shadowsocks/simple-obfs
~ $ cd simple-obfs/
~ $ git submodule update --init --recursive
~ $ ./autogen.sh
~ $ ./configure [--prefix=/home/username/opt] && make
~ $ make install prefix=~/opt
# -or-
~ # cp src/{obfs-local,obfs-server} /opt/bin

By default, binaries, doc, man are installed into /usr/local/{bin,doc,man}. configure --prefix=/foo/bar changes to /foo/bar.
1. Do not put tailing slash to --prefix= argument.
2. prefix must be absolute path.
To be simple, obfs-local and obfs-server binaries are enough. Just copy to PATH.

Finally, to clean letfovers:

~ $ cd ~/opt/simple-obfs
~ $ make clean
~ $ make distclean
~ $ git status

Server unit

[Unit]
Description=simple-obfs server
Requires=shadowsocks-libev.service
After=network.target shadowsocks-libev.service

[Service]
Type=simple
PermissionsStartOnly=false
User=nobody
Group=nobody
LimitNOFILE=4096
ExecStart=/usr/local/bin/obfs-server -c /etc/shadowsocks-libev/obfs-server.json

[Install]
WantedBy=multi-user.target

Server json

{
    "server":"0.0.0.0",
    "server_port":1235,
    "dst_addr":"127.0.0.1:1236",
    "timeout":100,
    "obfs":"tls",
    "failover":"www.bing.com",
    "fast_open":true
}

Listen on port 1235 for obfs-client connection;
Forward traffic to shadowsocks on port 1235;
Obfuscation method is http or tls.
Fail over to public domain or personal web server.

Client initd

#!/sbin/openrc-run
#
# 'start-stop-daemon' arguments resides before '--exec' while
# "${OBFS_COMMAND}" arguments after two succesive dashes '--'.
#
# Drop to 'nobody' user.

USER="nobody"

OBFS_COMMAND="/opt/bin/obfs-local"
OBFS_PIDFILE="/var/run/simple-obfs.pid"
OBFS_CONFILE="/etc/shadowsocks-libev/obfs-local.json"

depend() {
  after dhcpcd
}

start() {
  if [ "${RC_CMD}" = "restart" ];
  then
    ebegin "Restarting obfs-local"
    eend $?
  fi

  ebegin "Starting obfs-local"

  # start-stop-daemon -S -x "${OBFS_COMMAND}" -u "${USER}" -b -m -p "${OBFS_PIDFILE}" -- -c "${OBFS_CONFILE}"
  start-stop-daemon -S -x "${OBFS_COMMAND}" -- -c "${OBFS_CONFILE}" -a "${USER}" -f "${OBFS_PIDFILE}"

  eend $?
}

stop() {
  ebegin "Stopping obfs-local"

  start-stop-daemon -K -p "${OBFS_PIDFILE}"

  eend $?
}

Client json

{
    "server":"vps-ip",
    "server_port":1235,
    "local_address":"127.0.0.1",
    "local_port":1234,
    "timeout":100,
    "obfs":"tls",
    "obfs_host":"www.bing.com",
    "fast_open":true
}

Notice that server and server_port remain unchanged irrespective of client or server side.

Notes

Make sure Iptables allows relevant ports and IPs.
On Android, first install simple-obfuscation app. Afterwards, Change profile port to that of obfs-server. Obviously, you should set Configure part to relevant values.

You may got unknown plugin obfs-local error after the exact first run. Please turn on autostart permission.

Nginx

2017-04-11T00:00:00+00:00

Nginx Installation
Nginx Overview
Control Signal
Configuration Tips
if else
Number of Connections
Catch-all Virtual Server

Nginx Installation

~ # dnf install nginx
~ # systemctl enable nginx
~ # systemctl status nginx

Nginx Overview

Starter
Nginx has one master process and several worker processes.
1. The main purpose of master process is to read and evaluate configurations, maintain worker processes, manage TCP connections etc.
2. Worker processes serve client requests. The number of worker processes can be configured and is subject to system resources like number of CPU cores, storage size of HDDs, load pattern etc.
Nginx employs event-based model and OS-dependent mechanisms to efficiently distribute requests among worker processes.

Control Signal

Test configuration:

~ # nginx -t

Start Nginx:

~ # nginx -p </path/to/prefix>

We can control Nginx with the nginx -s <signal> option:

reload - reloading Nginx configuration
reopen - log file rotation
quit - gracefully shutdown Nginx. Stop only all current requests are served.
stop - fast shutdown.

We can also control Nginx by sending signal to the master process directly. For example, kill -HUP <master-pid> is an equivalent of nginx -s reload. By the way, systemctl reload nginx is just a wrapper of nginx -s reload.

Configuration Tips

Do not Repeat Yourself.

Set directives at their broadest applicable context.
Modular configuration. Different configuration files or directories can be integrated into nginx.conf by the include directive.

Here is the common list of configuration modules.
1. sites-available
2. sites-enabled - symbolic links to sites-available
3. default.d
4. modules.d
5. conf.d
6. stream.d

if else

Nginx if does not support logical AND or OR, so we have to add a particular if directive for each condition. For logical OR, that is enough. But for logical AND, we also set a variable and then concatenate different flags from those if directives. Then, a last if directive is used to check the variable value, as below.

if ($request_uri = /) {
  set $con root;
}

if ($host ~* example.com) {
  set $con "${con}+example.com";
}

if ($http_cookie !~* "auth_token") {
  set $con "${con}+no_auth_token";
}


if ($con = "root+example.com+no_auth_token") {
  return 403;
}

Number of Connections

By default, a single worker process allows 1024 connections simutaneously. We can litmit the number by:

events {
    worker_connections 50;
}

Catch-all Virtual Server

# catch-all 'default_server' vhosts

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    server_name _;
    root /usr/share/nginx/html;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
	    # Use "$host" instead of "$server_name" unless you have special needs.
        #return 301 https://$server_name$request_uri;
        return 301 https://$host$request_uri;
    }
}

# Settings for a TLS enabled server.

server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    server_name _;
    root /usr/share/nginx/html;

    # Let's Encrypt will start offer wildcard certs beginning 2018
    ssl_certificate "/etc/letsencrypt/live/example.com/fullchain.pem";
    ssl_certificate_key "/etc/letsencrypt/live/example.com/privkey.pem";
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout  10m;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
        return 404;
    }

    error_page 404 /40x.html;
    location = /40x.html {
        internal;
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        internal;
    }

}

All HTTP requests are redirected (301) to HTTPS.
Trigger 404 code, telling end users that the request is not expected.
Attention to server_name: _;. It is not a must.

Android Linux

2017-04-11T00:00:00+00:00

Outline

Adb

PC tool that sends commands to Android system over cable or Wi-Fi. Commands are:
1. Adb sub-commands like shell, install, push, sideload etc.
2. Android system commands like pm, am etc. by adb shell.
Fastboot

Similar to Adb, Fastboot connects to Android bootloader - Fastboot by cable. As Android bootloader does not provide built-in commands, Fastboot supports only sub-commands of itself like flash etc.

Fastboot tool VS Fastboot bootloader.
TWRP

Android system includes the tiny Recovery and complete Android ROM (i.e. MIUI). TWRP is an extension to stock Recovery with more functionalities. Mostly, we use it to flash 3rd party ROMs.

Possible TWRP sub-commands are wipe, install etc. by adb shell twrp.

You could change TWRP version if flashing failed.

Thunar

~ # echo "gnome-base/gvfs mtp" >> /etc/portage/package.use/gvfs
~ # emerge -av1 gvfs

Logout and login to take effect.

TWRP

Set rm -rf to avoid re-formating EXT4 filesystem unpon wiping.
After wiping and before flashing, keep all (at least /system) partitions umounted and disable MTP, which otherwise would report system, data etc. partitions busy errors.
Error. If you receive error like:
```
Unable to mount /data
```
Resort to:
1. Adanced Wipe, Repair or Change File System;
2. Reboot TWRP.
3. bootloader emergency recovery wipe;
4. adb sideload.
Instead of flashing TWRP to device, we can temporarily boot device with TWRP image.

ROM

If previous ROM is official Android 6.0 and new ROM is 3rd party Android 7.0/7.1, you'd better firstly flash an official 7.0/7.1 development version (a big so-called "底包").
To support F2FS data partition:
1. 3rd party recovery (i.e. TWRP) should support creating F2FS filesystem (formating data and cache partitions). Don't format system partition as F2FS.
  
  Formatting data partition to F2FS would wipe the internal storage too.
2. ROM must run on F2FS systems.

Android Debug Bridge (adb)

Android Debug Bridge (adb) is a versatile command-line tool that lets you communicate with a device (an emulator or a connected Android device).

The adb command facilitates a variety of device actions, such as installing and debugging apps, and it provides access to a Unix shell that you can use to run a variety of commands on a device.

It is a client-server program that includes three components:

A client, which sends commands. The client runs on your development machine. You can invoke a client from a command-line terminal by issuing an adb command.
A daemon (adbd), which runs commands on an Android device. The daemon runs as a background process on each device.
A server, which manages communication between the client and the daemon. The server runs as a background process on your development machine.

Installation

From Gentoo portage:

~ # emerge -avt dev-util/android-tools
# Android platform tools (adb, fastboot, and mkbootimg)
# Logout and login back to take effect

Current stable is 1.0.32. For an up-to-date version (recommended), download here, which is a standalone version ready for terminal (just extract the zip):

~ $ cd ~/opt
~ $ unzip ~/Downloads/platform-tools-latest-linux.zip
~ $ ./adb
~ $ ./fastboot

Connection to device

There are two ways to pair an Android device with your PC, namely over USB cable and over Wi-Fi.

In order to let computer recognizes the Android device, we should enable:

Developer options;
USB debugging;
Fastboot Unlock (i.e. Xiaomi virgo).

Over USB cable

~ $ ./adb start-server/kill-server
~ $ ./adb devices [-l]
~ $ ss/netstat -npelt

adb devices launches adb start-server on demand. When the server starts, it binds to local TCP port 5037 and listens for commands sent from adb clients - all adb clients use port 5037 to communicate with the adb server.

adb client - adb server - adb daemon on Android

Example output:

List of devices attached
afc75758               device usb:1-1.2 product:NX531J model:NX531J device:NX531J

The serial number is afc75758 while device represents connection state.

If adb prints:

List of devices attached
afc75758	unauthorized

Unlock your device and confirm PC RSA key (for first connection). This security mechanism protects user devices because it ensures that USB debugging and other adb commands cannot be executed unless you're able to unlock the device and acknowledge the dialog.

If adb complains:

List of devices attached
afc75758               no permissions (udev requires plugdev group membership); see [http://developer.android.com/tools/device.html] usb:1-1.2

Possible methods:

Add current user account into plugdev group.
Run adb under root account.
Switch the USB connection type from Charge only to Transmit files (MTP) or Transmit photos (PTP) from device drop-down menu. Alternatively, set Select USB configuration in Developer options.

Follow the given link to create a udev rule your device:

# lsusb
Bus 001 Device 035: ID 19d2:abcd ZTE WCDMA Technologies MSM

lsub prints the device vendor ID (19d2) and product ID (abcd). If /lib/udev/rules.d/69-libmtp.rules (media-libs/libmtp) does NOT include your device, then

# /etc/udev/rules.d/51-android.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="19d2", ATTRS{idProduct}=="abcd", GROUP="plugdev", MODE="0660", SYMLINK+="NX531J-%k"
# or
SUBSYSTEM=="usb", ATTRS{idVendor}=="19d2", ATTRS{idProduct}=="abcd", OWNER="jim", MODE="0600", SYMLINK+="NX531J-%k"

Finally, trigger the rules:

~ # gpasswd -a username plugdev (optional)
~ # udevadm control -R/--reload (optional)
~ # udevadm trigger

Pay attention to how GROUP and OWNER relates to MODE.

We might create a new rule for fastboot (bootloader) uses a different idProduct number.
Different ROM gives differents idProduct number.

Over Wi-Fi

Connection over Wi-Fi needs some initial setup over USB. Connect your Android device and adb host computer to a common Wi-Fi network accessible to both (i.e. WLAN).

~ $ ./adb tcpip 5037
~ $ ./adb connect device_ip_address
~ $ ./adb devices -l

Connection over Wi-Fi is slow in speed and less secure subjecting to sniffer.

adb sub-commands

~ $ adb [-d | -e | -s serial_number] command

One of the most special is shell:

~ $ ./adb shell
# get into Android Linux system
shell@NX531J:/ $ ls /system/bin
shell@NX531J:/ $ netstat -lt

If you are connecting over Wi-Fi, netstat will show the TCP socket which is owned by adb shell. To tell the ADB daemon return to listening over USB and closing the socket:

~ $ ./adb usb

Other commands are pull/push/backup for file transmission and backup/restore. Pull/push is pretty fast to transmit files but may the device a hot potato - high temprature.

You should also pay attention to remount/install/uninstall/shell for file updating. Apart from traditional shell commands, there are two special ones, namely am (activity manager) and pm (package manager).

Lastly, the most common command is:

~ $ reboot [bootloader|recovery|sideload|sideload-auto-reboot]

As stated in the Outline section, we can use adb shell twrp install /path/to/zip/ROM/on/android to flash a new ROM.

logcat

logcat is a pretty useful tool to collect system logs which can be filtered by parameters to focus on specific app.

~ $ ./adb shell
shell@NX531J:/ $ logcat --help
# or
~ $ ./adb logcat --help

For example, we want logs of Amaze:

~ $ ./adb shell ps | grep -i amaze
# get the PID
~ $ ./adb logcat | grep -F $PID
~ $ ./adb logcat -s *:E

Flashing

Fastboot is Android device's bootloader that resembles Grub in loading kernel.
1. A system can be a tiny Recovery system or a complete ROM system. That is to say, Recovery and ROM are of equal objects in viewpoint of Fastboot.
2. Device and PC must be connected cable line. This is the reason we call it "线刷" (i.e. fastboot update update.zip). Here is a reference 命令行下的fastboot刷机.
Recovery is a tiny runnable Android system. As explained above, we can connect PC to device recovery over cable line or Wi-Fi.

Usually Recovery is used to do "卡刷" (PC involvement).
fastboot and adb commands communicate with fastboot bootloader and recovery/ROM system respectively.
The ROM (zip) format of "线刷" and "卡刷" is probably different.

Check your vendor's instruction.
After wiping partitions (data, cache, system, etc.), you'd better reboot TWRP.

fastboot TWRP

fastboot can flash sparse img (recovery, data, system, and even self-made ones) to relevant Android partitions. To make fastboot recognize your device,

Let the device get into fastboot mode.
```
# get into fastboot mode
~ $ ./adb reboot bootloader
```
Another way is to use combination keys mostly by pressing power key and volume down key simultaneously. Details refer to your device documentation.
Create a udev rule.

When the device is at fastboot mode, the product ID changes accordingly. We should add another udev rule like for adb.

boot/flash

~ $ ./fastboot devices -l
~ $ ./fastboot boot recovery /path/to/TWRP.img (recommended)
~ $ ./fastboot [-i 0x19D2 ] flash recovery /path/to/TWRP.img

getvar
```
~ $ ./fastboot getvar all
```

The difference between flash and boot is that:

flash do flash the TWRP to recovery parition for permanent effect.
boot temporarily boots the device with TWRP, which resembles a LiveCD environment.

adb sideload ROM

As of version 2.3, TWRP now supports adb sideload mode. ADB sideload is a different ADB mode that you can use to push and install a zip using one command from your computer. Don't mix adb sideload with fasboot flash ("线刷"), though they both work over cable connection.

adb sideload requires reliable USB cable connection.

Let your device be in sideload. You either do it in TWRP menu or by adb command line from your computer:
```
~ $ ./adb reboot sideload/sideload-auto-reboot
~ $ ./adb devices
```
You will get:
```
List of devices attached
afc75758	sideload
```
Flash the zip ROM
```
~ $ ./adb sideload /path/to/rom.zip
```

ROM custiomization

Before flashing official zip ROM, delete builit-in promotional APKs to save storage. Alternatively, you have to delete them by root apps after flashing.

To add back a previously deleted app, push both the apk and odex. install does NOT help in case of apk with separate odex.
```
~ $ ./adb remount
~ $ ./adb push /path/apk /system/app/
```
1. You may remount before pushing to /system/app.
2. Check newly pushed files permission mode.
3. You can also make a zip ROM to install just one app. Details refer to 努比亚z11精简包+系统程序中英文对照表.

To further delete a built-in apk,

~ $ ./adb shell pm list packages | grep -i example
~ $ ./adb pm uninstall nubia.example.cn
~ $ ./adb reboot recovery
~ $ ./adb shell
shell@NX531J:/ $ find / -iname '*example*'
shell@NX531J:/ $ find / -iname '*example*' -delete

Sometimes deleted apps reappear somehow. They are restored by Android cache partition. Wipe cache gets rid of that.

Add personal apps to ROM

Put your apk (i.e. Example) into /system/app/Example.
If apk depends on system lib (check /META-INF/com/google/android/updater-script), do a symlink to /system/lib.
Lastly, check files modes.

以下再以另一个例子来说明如何内置带有库（LIB）的软件。

我以来电通为例子

在电脑上用RAR打开“来电通.APK”。发现它是带有LIB目录的。

进入它，并把那两个SO文件拉出来，是的就是拉。然后放到手机的TF卡上。我一般喜欢放到个文件夹中。

这里记得把来电通的中文名改成一个你喜欢的英文名。比如说我的改成LDT1024.apk(如果文件名是英文，可以不不改掉。)

在RE中按住MENU 多选，选两个SO文件。复制它们到SYSTEM/LIB中。

同理，再回去把LDT1024. apk 拷到SYSTEM/APP中。

重启。接着你进去会发现，程序那里多出来了两个来电通的东东。一个是主程另一个来电通拔号。

apktool

To install apktool, at least java 1.7 is installed. Afterwards, just follow Install Instructions.

Put the two downloaded files into home directory ~/opt/apktool instead of /usr/bin.
```
~ $ cd ~/opt/apktool
~ $ chmod +x apktool*
~ $ ./apktool
```
Linux wrapper script is optional but we have to type a long command:
```
~ $ cd ~/opt/apktool
~ $ java -jar apktool.jar
```
Framework resources

Every Apktool release contains internally the most up to date AOSP framework at the time of the release. This allows you to decode and build most apks without a problem. However, manufacturers add their own framework files in addition to the regular AOSP ones. To use apktool against these manufacturer apks you must first install the manufacturer framework files.

Framework files are usually a few built-in apks under /system/framework. Nubia has two built-in framework files, namely framework-nubia-res.apk and framework-res.apk. We just need the former one for Apktool brings along an up-to-date framework-res.apk.
```
~ $ ./apktool if -t nubia /path/to/framework-nubia-res.apk 
```
1. -t nubia sets a tag for the installed framwork which should be added to decode, which otherwise would report Can't find framework resources for package of id.
2. By default, it's installed to ~/.local/share/apktool/framework.
Decompile/decode and build back

After decoding, we can find useful information on the app under res/values-zh-rCN/string.xml.
By default /tmp is mounted as noexec, which would cause apktool build error like:
```
brut.common.BrutException: could not exec: [/tmp/brut_util_Jar_1093671213752336150.tmp
```
We either remount /tmp as exec or refer to aapt. Firstly, extract directory prebuilt/aapt from apktool.jar to somewhere (i.e. ~/opt/apktool). Then use -a / --aapt parameter.
```
~ $ ./apktool b -a ./aapt/linux/64/aapt Test-apk/
```

Apps

ConnectBot

Connectbot freezes after login? No response with black screen? Please enable Start shell session in host edit.
LatinIME

By default, I have removed default AOSP input method /system/app/LatinIME from official ROM in favor of Google Pinyin Input. Meanwhile I choose require PIN/password to start device when setting lock screen PIN/password. In such case, I cannot boot the device since there is no stock input method to fill in PIN/password.

When confronted with the issue, I can temporarily adb push AOSP LatinIM into system or use physical OTG keyboard.

After booting the device, change lock screen method to pattern instead of PIN/password, which does not depend on stock LatinIME.

To be verified: may be solved by adding google-pinyin.apk into ROM /system/app or /system/priv-app/, thus flashing google-pinyin as stock IME.
OsmAnd+

obf maps data hidden link: obf link 1, obf link 2, and obf link 3. Generated download link is as:

http://dl5.osmand.net/download.php?standard=yes&file=Hong-kong_asia_2.obf.zip

Hidden Faq link.

boot/recovery img manipulation

Read mkbootimg/unpackbootimg. You'd better use android_system_core version.

Refs

CentOS RHEL

2017-04-05T00:00:00+00:00

Fedora RHEL CentOS
Module
rpm yum dnf
1. rpm
2. yum
3. dnf
Repositories
rpmbuild
1. Extracting RPM
locale
SSH
Create User Account
Pip/Virtualenv
SS

Fedora RHEL CentOS

Fedora is a Linux distribution released under GPL.
Red Hat, Inc. copy the source code from Fedora, make modifications, release it as Red Hat Enterprise Linux (RHEL) distribution.

Due to GPL, Red Hat, Inc. is obligated to disclose RHEL source code. But Red Hat can make money from its professional after-sale services.
CentOS (Community ENTerprise OS) is a Linux distribution built from the source of RHEL directly without any modification, with the intention of making CentOS binary compatible with RHEL such that library versions the same and binaries work on RHEL also work on CentOS.
1. The main purpose of CentOS is to make RHEL available to pulbic users, benefitting from Red Hat's professional Linux service.
2. Remove Red Hat, Inc. logo.
Fedora - RHEL - CentOS.

Module

Module is a new packaging feature brought in from RHEL 8.

Package

Traditional .rpm file.
Group and Environment

A large set of packages as a whole for specific system categories like X window, desktop, security, development, etc.
Module is a collection of a few packages, as a logical unit for a specific project, like Php and Nginx.
1. Stream is the upstream version.
2. Version is the build version.
3. Profile is the sub-functionality of a module, similar to Gentoo's tag.

rpm yum dnf

The relation between rpm (Redhat Package Manager) command and yum (Yellow dog Updater, Modified) command is analogous to dpkg and apt-get. dnf is the new version of yum with performance improvement and module feature. From RHEL 8 onwards, dnf replaces yum as the default manager.

Basically, yum is more intelligent than rpm. rpm wants to know the exact location of target .rpm file while yum only needs the package name. yum also takes care of dependencies. However, rpm is somehow a lightweight tool with concise output.

Check 'SPECIFYING PACKAGE NAMES' in the man page of yum to check how to pass globs arguments. The full name of a .rpm package is as follows:

# <name>-<version>-<release>.<os>.<arch>.rpm

vim-enhanced-7.4.160-4.el7.x86_64
# name:     vim-enhanced
# version:  7.4.160
# release:  4
# os:       el7
# arch:     x86_64
# suffix:   rpm

centos-release-7-4.1708.el7.centos.x86_64
# name:     centos-release
# version:  7
# release:  4.1708
# os:       el7.centos
# arch:     x86_64

Package version is also called major version, indicating the upstream source version from developers. Package release is also called minor version, indicating the final .rpm file by RHEL compliling/building/patching. Especially, RHEL may add customized patches or bug fixes to the major version, building a new .rpm package to the repository.
os may be el6 (rhel6), centos6, el5, suse11 etc.

By design, it is not possible to install difference versions of the same package alongside. The workaround is to make the new version a different package. For example, python2 and python3 are actually different package names.

rpm

rpm manages package file directly, maintaining a database of package information located under /var/lib/rpm/. It does not care too much the package version but the RPM file you provided.

Query installed packages:

~ # rpm -qa 'globbing'
~ # rpm -qa | grep 'regex'

~ # rpm -qi pkg; rpm -qip pkg.rpm                     # query pkg information
~ # rpm -qR pkg; rpm -qRp pkg.rpm                     # dependencies

~ # rpm -ql pkg; rpm -qlp pkg.rpm                     # files in a package
~ # rpm -qf /path/to/file                             # package that owns the file

Verify. List discrepancies between installed files and files recorded in metadata from local RPM database.

~ # rpm -V pkg; rpm -Vp pkg.rpm

Kering:

~ # rpm --import /path/to/key.pub
~ # rpmkeys --import http://www.example.com/key.pub

Once imported, all public keys in the keyring can be managed as a package. Commands applied to a package also apply to a key like query, erase, information etc.

~ # ls /etc/pki/rpm-gpg/

~ # rpm -qa gpg-pubkey*

~ # rpm -qi gpg-pubkey-db42a60e

Signature. To verify a RPM file is not corrupted (e.g. due to network transmission) and correctly signed (by developer).

# verify MD5
# ok
rpm -Kv --nosignature kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm:
    Header SHA256 digest: OK
    Payload SHA256 digest: OK

# verify both MD5 and signature
# MD5: ok
# signature: key not found
~ $ rpm -Kv kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm:
    Header V4 RSA/SHA256 Signature, key ID 1a62ff7c: NOKEY
    Header SHA256 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA256 Signature, key ID 1a62ff7c: NOKEY

# get signing key ID from RPM file
~ $ rpm -qip kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm | grep Signature
warning: kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm: Header V4 RSA/SHA256 Signature, key ID 1a62ff7c: NOKEY
Signature   : RSA/SHA256, Sat 01 Jul 2023 04:11:14 AM UTC, Key ID 998dff461a62ff7c

~ $ rpm -qp --queryformat '%{NAME}-%{VERSION}-%{RELEASE} %{SIGPGP:pgpsig}\n' kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0-1 RSA/SHA256, Sat 01 Jul 2023 04:11:15 AM UTC, Key ID 998dff461a62ff7c

# import the signing key
~ $ sudo rpm --import https://docs.konghq.com/assets/keys/2023-rpm.asc

# list GPG key in RPM database
~ $ rpm -qa gpg-pubkey*
gpg-pubkey-be1229cf-5631588c
gpg-pubkey-fd431d51-4ae0493b
gpg-pubkey-1a62ff7c-639852e2
gpg-pubkey-d4082792-5b32db75

# examine the GPG key
~ $ rpm -qi gpg-pubkey-1a62ff7c-639852e2
Name        : gpg-pubkey
Version     : 1a62ff7c
Release     : 639852e2
Architecture: (none)
Install Date: Mon 03 Jul 2023 05:45:24 AM UTC
Group       : Public Keys
Size        : 0
License     : pubkey
Signature   : (none)
Source RPM  : (none)
Build Date  : Tue 13 Dec 2022 10:24:34 AM UTC
Build Host  : localhost
Relocations : (not relocatable)
Packager    : Bazel Testing
Summary     : gpg(Bazel Testing)
Description :
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: rpm-4.14.3 (NSS-3)
...

# verify again
~ $ rpm -Kv kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm:
    Header V4 RSA/SHA256 Signature, key ID 1a62ff7c: OK
    Header SHA256 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA256 Signature, key ID 1a62ff7c: OK

Install:

~ # rpm -ivvh --test /path/to/pkg.rpm      # dry run
~ # rpm -ivh /path/to/pkg.rpm              # install
~ # rpm --reinstall -vh /path/to/same.rom  # reinstall

~ # rpm -Uvh /path/to/pkg.rpm              # upgrade
~ # rpm -Fvh /path/to/pkg.rpm              # refresh
~ # rpm -Fvh --force /path/to/old.rpm      # downgrade

The path can be an ftp/http URL.
-i, --install is the general form of installing a package.
-U, --upgrade is the combination of -i, --install and -e. It firstly install a package and then erase earlier versions if present.

-U does not require an earlier version is installed, so it can be a replacement of -i.
-F, --freshen is similar to -U but mandates an earlier version is already installed.
--force, --replacepkgs, --replacefiles and --oldpackage do the same thing.

Uninstall/Erase:

~ # rpm -evv pkg                     # uninstall a pkg

yum

The main configuration of YUM is /etc/yum.conf and specific repository configuration files are placed in /etc/yum.repos.d/. Within the main configuration file, cachedir defines the location of cached .rpm packages (defaults to /var/cache/yum/$basearch/$releasever).

By default, of the built-in repositories, only 'CentOS-Base.repo' is enabled.

$releasever indicates the main version. For example, the main version of CentOS 5.8 is 5.
$arch indicates the architecture like x8_64, i386/i586/i686.
$basearch is similar to $arch, but is base architecture: either x86_64 or i386.

Cache:

~ $ yum help [sub-command]

~ # yum clean [ all | packages | metadata | expire-cache | rpmdb | plugins ]
~ # yum makecache

After any edits of repo files, in order to clear any cached information, and make sure the changes are immediately recoginized, as root run yum clean all.

Query:

~ # yum history

~ # yum list [all | installed | available | updates ] [pkg]
~ # yum info [all | installed | available | updates ] [pkg] # more details

~ # yum search pkg
~ # yum --showduplicates search pkg    # show all versions

~ # yum repolist [--all | --enabled | --disabled]
~ # yum check-update
~ # yum upgrade [pkg]

~ # yum provides /path/to/file             # rpm -qf
~ # repoquery -f /path/to/file
~ # repoqeury --provides /path/to/file
~ # repoqeury --whatprovides /path/to/file

~ # repoquery -l pkg

~ # repoquery --deplist pkg
~ # repoquery --whatdepends pkg

update is synonym to upgrade.
list –updates is almost the same as check-update. As the command form implies, check-update is useful in Shell script while list –updates is for humans on the command line.

Please pay attention, their exit status code difference.
Make sure yum-utils is installed to use repoquery.

Install:

~ # yum [-y] install pkg1 pkg2

# will install dependency automatically
~ # yum install /path/to/pkg.rpm

~ # yum reinstall pkg1 pkg2

~ # yum upgrade pkg1 pkg2

~ # yum remove pkg1 pkg2

To install packages for group or environment set:

~ # yum group summary
~ # yum group [ list | info |install | upgrade | remove | mark ] [grp]

dnf

dnf is compatible with but more powerful than yum. One of the newest feature provided is module.

~ # dnf module -h
~ # dnf module list nginx
~ # dnf module info nginx

# NAME:STREAM:VERSION:CONTEXT:ARCH/PROFILE
~ # dnf module info nginx:1.16:8010020191122190044:cdc1202b:x86_64/common
~ # dnf module install nginx:1.14/common

The purpose of module is to manage package versions and profiles in an integrated way. The following two methods is equivalent:

~ # dnf module install nginx
~ # dnf install nginx

The only difference is that dnf module can provide better control when selecting pakage stream, version, arch etc.

The following example shows how to control repo with dnf (applying to yum as well):

~ $ dnf help repolist 
~ $ dnf repolist [--all | --enabled | --disabled]

~ $ sudo dnf config-manager --add-repo /etc/yum.repos.d/oracle-epel-ol8.repo

~ $ grep enabled /etc/yum.repos.d/oracle-epel-ol8.repo
~ $ sudo dnf config-manager --set-enabled ol8_developer_EPEL
~ $ duso dnf config-manager --set-disabled ol8_developer_EPEL

~ $ dnf --enablerepo=repo1,repo2 -disablerepo=repo3,repo4 install pkg

Repositories

Of the 3rd party repositories, IUS and Remi (both depend on EPEL) is recommended over Webtatic.

~ # yum repolist [all|enabled|disabled]

~ # yum-config-manager --add-repo <repo-url>
~ # yum update -y

~ # yum repolist [all|enabled|disabled]
~ # yum-config-manager --disable <repo-id>

~ # yum-config-manager --enable <repo-id>
~ # yum update -y

To remove a repo, just remove the file within /etc/yum.repos.d.

EPEL not from RHEL

EPEL (Extra Packages for Enterprise Linux) is open source and free community based repository project from Fedora team which provides 100% high quality add-on software packages.

~ # yum info epel-release
~ # yum install epel-release

To specify repository when installing package:

~ # yum --enablerepo=epel install nginx

The --enablerepo option overides the permanent option setting in the /etc/yum/*.repo files for only the current command. --disablerepo does the opposite for enabled repos.

Inline with Upstream Stable (IUS)

~ # rpm -Uvh https://centos7.iuscommunity.org/ius-release.rpm

Read IUS Usage for more information.

Webtatic (DEPRECATED)

The Webtatic Yum repository is a CentOS/RHEL repository containing updated web-related packages like git.

~ # rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm

CentOS-5.8

CentOS-5 reached the end of its lifecycle as of March 31, 2017. The official baseurl is deprecated and moved to a backup location Vault. Packages there is no longer maintained or upgraded, and may suffer from security risks.

However, we sometimes have to maintain ancient releases for whatever reasons. To install packages through yum, point repositories to the new location: either manually edit /etc/yum.repos.d/CentOS-Base.repo or install centos-release-5-8.el5.centos.x86_64.rpm file.

For each section (i.e. Base) Comment out both the mirrorlist= and baseurl= lines.
Add a new baseurl line:
```
baseurl=http://vault.centos.org/5.8/os/$basearch/
```
1. Replace /os/ part approriately for other sections like /updates/.
2. It is highly recommended to replace 5.8 with 5.11, upgrading the system to CentOS 5.11
Alternatively, install the corresponding repository package:
```
root@tux ~ # wget http://vault.centos.org/5.11/os/x86_64/CentOS/centos-release-5-11.el5.centos.x86_64.rpm
root@tux ~ # rpm -Uvh centos-release-5-11.el5.centos.x86_64.rpm
```
For other repositories like EPEL, Webtatic etc., find their archive URL and repeat the same procedures.

Update

root@tux ~ # rpm -q centos-release; cat /etc/redhat-release; cat /etc/centos-release
root@tux ~ # yum clean all; yum makecache
root@tux ~ # yum list updates; yum update
root@tux ~ # reboot
root@tux ~ # yum install epel-release; yum install git

Here is an example of manually edited Centos-Base.repo:

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=http://vault.centos.org/5.11/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#released updates 
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
baseurl=http://vault.centos.org/5.11/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
baseurl=http://vault.centos.org/5.11/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
baseurl=http://vault.centos.org/5.11/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
baseurl=http://vault.centos.org/5.11/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

Upgrade System

~ # dnf check-update
~ # dnf upgrade kernel
~ # dnf upgrade
~ # reboot

rpmbuild

For how to build RPM package, refer to "logging/filebeat-zookeeper-kafka".

Refer to How to create a Linux RPM package.

Extracting RPM

Upon receving a .rpm file, we can extract the contents with rpm2cpio and cipo:

~ $ rpm2cpio /path/to/pkg.rpm | cpio -idmv [--no-preserve-owner] [--no-absolute-filenames -D dst] '*resty*'
~ $ find .

-i: extract; -t list instead of extract.
-d: make directories dst and others on demand.
-m: preserve modification time.
-v: verbose.
--no-preserve-owner: replace the ownership with current user.
--no-absolute-filenames: remove the root dir prefix /, and replace it with -D.
*resty* is a Shell globbing pattern to extract only specific files. By default, extract all files.

locale

Redhad-based distributions places locale customization in /etc/locale.conf. Check man locale.conf.

SSH

~ # ssh-add
~ # ssh-copy-id -p 12345 root@12.23.56.78
# add an entry to ~/.ssh/config

Create User Account

Check useradd.

~ # ssh-copy-id -p 12345 username@12.34.56.78
# add an entry to ~/.ssh/config

Pip/Virtualenv

~ # yum install python-pip python-virtualenv
~ # pip install -U pip

SS

~ # mkdir -p ~/opt/pyvenv2.6
~ # virtualenv --system-site-packages ~/opt/pyvenv2.6
~ # pip install git+https://github.com/shadowsocks/shadowsocks.git@master
# or
~ # wget https://github.com/shadowsocks/shadowsocks/archive/master.zip
~ # pip install master.zip

Server Json

~ # mkdir -p /etc/shadowsocks
~ # vi /etc/shadowsocks/server.json

Fill in the fileds without any comments:

{
    "server":"::0",
    "server_port":3900,
    "local_address":"127.0.0.1",
    "local_port":1080,
    "password":"mypassword",
    "timeout":300,
    "method":"aes-256-cfb",
    "fast_open":true,
    "workers":2
}

TCP Fast Open

~ # vi /etc/sysctl.d/10-tcp-fast-open.conf
# net.ipv4.tcp_fastopen = 3
~ # sysctl -p /etc/sysctl.d/10-tcp-fast-open.conf

Unfortunately CentOS 7 does not satisfy lower kernel bound - 3.7.1.
It should be enabled on cleint and server simutaneously.

A quick test

~ # ssserver -c /etc/shadowsocks/server.json --user nobody -d start -vv

Systemd Unit

Please reinstall Shadowsocks into system-wide location. The above virtualenv version was just a test.

# vi /etc/systemd/system/shadowsocks.service

simple Type

[Unit]
Description=Shadowsocks Server
After=network.target

[Service]
Type=simple
PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /run/shadowsocks
ExecStartPre=/usr/bin/chown nobody:nobody /run/shadowsocks
ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody
ExecStart=/usr/bin/ssserver -c /etc/shadowsocks/server.json
Restart=on-abort
User=nobody
Group=nobody
UMask=0027

[Install]
WantedBy=multi-user.target

Please be noted that:

simple Type is used so that Shadowsocks does not forking itself so that daemon mode is handed over to systemd.
1. Do not add -d start/stop/restart arguments to ExecStart
2. No PIDFile and --pid-file required. Read more on doesn't make sense to set PIDFile= by simple services.
You may find a special line with kcptun. Neglect it now for it will be discussed later on.
ref1 and ref2.

forking Type

[Unit]
Description=Shadowsocks Server
After=network.target

[Service]
Type=forking
PermissionsStartOnly=true
PIDFile=/run/shadowsocks/ss.pid
ExecStartPre=/usr/bin/mkdir -p /run/shadowsocks
ExecStartPre=/usr/bin/chown nobody:nobody /run/shadowsocks
ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody
ExecStart=/usr/bin/ssserver -d start -c /etc/shadowsocks/ss.json --pid-file /run/shadowsocks/ss.pid --log-file /run/shadowsocks/ss.log
Restart=on-abort
User=nobody
Group=nobody
UMask=0027

[Install]
WantedBy=multi-user.target

kcptun

kcptun estabilishs a KCP tunnel to speed up TCP connection by encapsulating TC packets within UDP flooding. It may introduce twice or even triple traffic depending on arguments choosen. You are advised to use it only in WI-FI environment.

To add kcptun support, just insert a line into Systemd service file:

ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody

Attention to trailing & of -c argument.

Virtual Private Network (VPN)

2017-03-20T00:00:00+00:00

ABCs

VPN is a tunneling protocol (TP) designed to facilitate travel employees' networking connection back to office (i.e. headquarter).

However VPN can do more than that. For example, we use VPN to bypass firewall, ISP censorship etc. We call such VPN as Personal VPN.
There are many VPNs implementations over the world to choose from, which includes but are not limited to:

PPTP, L2TP/IPSec, SSTP, OpenVPN, etc. Among of them, OpenVPN is the most widely deployed but rather complicated.

Wireguard emerges as a state-of-the-art VPN technology based on DH key exchange and Poly1305 encryption.
Logically, most VPNs locate at link layer (layer 2) offering services to IP layer (layer 3). On the other hand VPN itself sits on top of TCP/UDP layer. Almost all VPNs create virtual interface at system level to which kernel transmits application IP packets.

VPN service either resides in kernel (i.e. WireGuard kernel module or built-in tree) or run as a daemon at userspace space. Obviously, the former has greater performance.

Upon receving IP payload, VPN encapsulates application's IP payload into VPN TCP or UDP packets which in turn are encapsulated into the Internet IP packets again.
Generally, UDP VPN (i.e. WireGuard) is faster than TCP VPN (OpenVPN) but less reliable.

PPTP

Point-to-Point Tunneling Protocol (PPTP). A specification for PPTP was published in July 1999 as RFC 2637.

PPTP has many known security issues, and it's likely the NSA (and probably other intelligence agencies) are decrypting these supposedly "secure" connections. That means attackers and more repressive governments would have an easier way to compromise these connections.

PPTP is common and easy to set up. PPTP clients are built into many platforms, including Windows, Android etc. That's the only advantage, and it's not worth it. It's time to move on.

In Summary: PPTP is an obselete, old and vulnerable, although integrated into common operating systems and easy to set up. Stay away!

L2TP/IPsec

L2TPv3 was proposed as standard RFC 3931 in 2005.

Layer 2 Tunneling Protocol (L2TP) is a tunneling protocol used to support virtual private networks (VPNs) or as part of the delivery of services by ISPs. It does not provide any encryption or confidentiality by itself. Rather, it relies on an encryption protocol that it passes within the tunnel to provide privacy.

Since L2TP is a VPN protocol that doesn't offer any encryption, it's usually implemented along with IPsec encryption. This is a slower solution than OpenVPN. The traffic must be converted into L2TP form, and then encryption added on top with IPsec. It's a two-step process.

SSTP

Secure Socket Tunneling Protocol (SSTP) was introduced in Windows Vista Service Pack 1. It's a proprietary Microsoft protocol, and is best supported on Windows. It may be more stable on Windows because it's integrated into the operating system whereas OpenVPN isn't — that's the biggest potential advantage. Some support for it is available on other operating systems, but it's nowhere near as widespread.

OpenVPN

Unlike PPTP and L2TP (proposed to RFC as standards), OpenVPN is actively maintained by OpenVPN Technologies, Inc.. Apart from the free OpenVPN Community Edition (CE), OpenVPN Technologies, Inc. provides proprietary solution - OpenVPN Access Server (AS). The proprietary OpenVPN AS is almost the same to free OpenVPN CE, with advantage of easy configuration and deployment (web interface).

OpenVPN uses open-source technologies like the OpenSSL encryption library and SSL v3/TLS v1 protocols. It can be configured to run on any port, so you could configure a server to work over TCP port 443.

OpenVPN support isn't integrated into popular desktop or mobile operating systems. Connecting to an OpenVPN network requires extra installation - either a desktop application or a mobile app. Yes, you can even use mobile apps to connect to OpenVPN networks on Apple's iOS.

Client

Android: OpenVPN for Android and OpenConnect (open source; no root requirement).
iOS: OpenVPN Connect (closed source).
GNU/Linux: net-vpn/openvpn.

Create PKI

Username/password authentication is discouraged for sake of anonymity and privacy.

To set up OpenVPN server/client, we need to create Public Key Infrastructure (PKI).

OpenVPN relies on a bidirectional authentication strategy, so the client must authenticate the server's certificate and in parallel, the server must authenticate the client's certificate. This is accomplished by the 3rd party's signature (the CA) on both the client and server certificates. Once this is established, further checks are performed before the authentication is complete.

CA

A public master Certificate Authority (CA) certificate and a private key thereof. CA is the root of trust web.

(ca.key, ca.pub, ca.crt).
Server/Client key

A separate public certificate and private key pair (hereafter referred to as a certificate) for each server and each client.

(server.key, server.pub. server.crt); (client.key, client.pub, client.crt).
Other files

tls-auth key file; crl-verify pem file; dh pem file etc.
An .ovpn file (can be imported) simplifies configuration.

Security

2017-03-10T00:00:00+00:00

In this post, security refers to cryptography, authentication, signing, password management, confidential communication and implementation thereof.

Prerequisite

Security -> Privacy -> Anonymity.

Security (of data)

When talking about security, we refer to securing the data contents, which must meet three basic prerequisite: confidentiality, availibility, and integrity. To guarantee data security, people encrypt data, store data in private place, etc.

Please be noted data security is a static state of data management, serving as the mean to data privacy.
Privacy (of data)

Unlike static state of data security, privacy concerns utilizing secure data to ensure only authorized users can access it while unauthorized user cannot.
Anonymity (of user)

Anonymity means disconnection between user identify and Internet activity. Identity is user's electronic representation, mainly consisting of computer mac address, browser fingerprint, email account, etc. ISP and end web servers can easily trace your identity by sniffing Internet communication.

Without security and privacy, anonymity is impossible.

PGP - Encryption

Open Pretty Good Privacy (OpenPGP)

OpenPGP is an IETF standard proposed by PGP inc.
PGP

PGP is a proprietary software in accord with OpenGPG standard, previously own by PGP inc. now acquired by Symantec Corp.

PGP software comes before OpenPGP standard. In order to let PGP software thrive, PGP inc. proposed OpenPGP standard (PGP-complicant) to IETF.
GNU Privacy Guard (GnuPG)

GnuPG is another OpenGPG implementation free of distribution that obeys GNU General Public License (GPL).
There are other PGP-compliant software such as GoAnywhere OpenPGP Studio.

SSH - Transmission

Secure Shell (SSH)

SSH is a cryptographic network protocol for operating network services securely over an unsecured network. The best known example application is for remote login to computer systems by users.

SSH was designed as a replacement for Telnet and for unsecured remote shell protocols such as the Berkeley rlogin, rsh, and rexec protocols.
OpenBSD Secure Shell (OpenSSH)

OpenSSH started as a fork of the free SSH program, developed by Tatu Ylönen; later versions of Ylönen's SSH were proprietary software, offered by SSH Communications Security. OpenSSH was first released as part of the OpenBSD operating system in 1999.

OpenBSD is a free and open source Unix-like computer operating system descended from Berkeley Software Distribution (BSD), a Research Unix derivative developed at the University of California, Berkeley.
SFTP/SCP etc.

SFTP/SCP are secure version of FTP/CP.

Cryptography library

GnuTLS and OpenSSH are different implementations of SSL/TLS/DTLS protocols.

GnuTLS

GnuTLS is a secure communications library implementing the SSL, TLS and DTLS protocols and technologies around them.

GnuPG depends on GnuTLS library.

OpenSSL

OpenSSL is a toolkit implementing SSL v2/v3 and TLS protocols with full-strength cryptography world-wide.

OpenSSH is a program depending on OpenSSL the library, specifically OpenSSH uses the libcrypto part of OpenSSL.

Summary

OpenGPG/SSH - protocol standard.
GnuPG/OpenSSH - implementaion.
Gnutls/OpenSSL - underlying library.

Public/Private key

Losing sole possession of your private key is catastrophic.

GnuPG

Public key encrypts message to you. Private key decryptes message recevied; signs document in your name.
The term key usually refers to key pair which is a coulple of public key (--list-keys) and private key (--list-secret-keys).
By default, two key pairs are generated, of which one pair is called master key pair for Signing and Certifying while another one is subkey pair for Encrypting.

When addkey, a new subkey pair will be generated.
Each subkey pair can be identified by hexadecimal --fingerprint --fingerprint, keyid (low 64 bits of fingerprint), uid, email address, etc.

OpenSSH

Check key information by ssh-keygen -lv and ssh-add -l.
Use ssh-copy-id to distribute public key to remote server. It simply appends the public key file contents to remote ~/.ssh/authorized_keys.
SSH key does NOT hold uid. On the contrary, email and notes are called comments which can be changed by ssh-keygen [-o] -c.

avplayer.org

2017-03-09T00:00:00+00:00

Update procedure:

# ssh-add
# ssh fyzhuji
# cd /srv/wiki/wikidata
# edit my.file
# git add my.file
# git commit -m "edit my file"

That's all! No need to push since updating on server.

CN2 VPS

2017-02-16T00:00:00+00:00

Virtualization

OpenVZ/XEN/KVM

OpenVZ

OpenVZ is a half virtualized technology at OS-level with limitations. Each VPS is actually a guest operating system container and shares the same underlying host operating system but still operates as secure, isolated environment.
1. We cannot make any kernel level modifications inside the VPS. The VPS's on the host share the host kernel.
2. Host can easily be over-subscribed. Hosting providers can and do host 20+ OpenVZ virtual machines on a system with 12 GB of ram. Each VPS has 1 GB of ram "assigned" to it. Not all systems use 1 GB all the time, so the system "robs Peter to pay Paul" when a VPS does need all its memory. It will allow the VPS to access its assigned amount of ram. However if none is available due to other VPS's using it, contention can get bad and things slow down.
3. Openvz is better when you want to run a mass of linux VMs and need to be able to flexibly allocate resources between them. Since the systems are all using the same file system, kernel and memory space, a container dosen't need memory for its exclusive use.

XEN

OpenVZ is generally oversells to take advantages of unused resources, while XEN cannot, thus slightly more costly and they call XEN VDS (Virtual Dedicated Server) rather than VPS (Virtual Private Server). If you running gameserver for example, XEN is a MUST! KVM won't be as fast as XEN.

KVM

Your host OS has to be Linux but it supports Linux, Windows, Solaris, and BSD guests. It runs on x86 and x86-64 systems with hardware supporting virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, that provides the core virtualization infrastructure and a processor specific module.

ChinaTelecom 163 to CN2

ChinaNetNextCarryingNetwork (CN2)

Isolated from the old ChinaNet/China163, CN2 is ChinaTelecom's newly built highly qualified backbone to serve coporate clients.
MTR - My TraceRoute, an Excellent network diagnostic tool

MTR represents an evolution of traceroute with a greate data sample. It combines the functions of the traceroute and ping programs in one network diagnostic tool.

Because MTR provides an image of the route traffic takes from one host to another, you can think of it as a directional tool. Furthermore, the route taken between two points on the Internet can vary a great deal based on location and the routers that are located upstream of you. For this reason it is often recommended that you collect MTR reports in both directions for all hosts that are experiencing connectivity issues, or as many hosts as possible.
```
~ $ mtr -rtw [destination_host]
```
Bi-directional CN2

It's really important to mtr from VPS to PC since that's the daily traffic flow direction!

An example of mtr output:

zhtux ~ # mtr -rw 220.173.118.215
Start: Thu Feb 16 20:25:53 2017
HOST: zhtux           Loss%   Snt   Last   Avg  Best  Wrst StDev
|-- 192.168.0.1      0.0%    10    3.0   2.1   1.1   6.2   1.5
|-- hz-dns90         0.0%    10    4.8   4.0   2.5   7.5   1.3
|-- 172.16.203.253   0.0%    10    3.9   4.5   3.5   6.4   0.8
|-- hz-dns90         0.0%    10    3.7   3.8   1.9   8.1   2.1
|-- hz-dns90         0.0%    10    6.0  12.1   5.2  40.8  10.8
|-- hz-dns90        20.0%    10   23.7   8.6   3.0  23.7   6.9
|-- 101.4.114.229   50.0%    10    6.0  15.6   6.0  32.9  10.7
|-- 101.4.112.37     0.0%    10   26.2  31.1  23.3  59.3  10.2
|-- 101.4.117.34    10.0%    10   18.8  27.8  18.8  43.9   8.6
|-- 101.4.118.154    0.0%    10   20.6  25.5  20.6  34.4   4.5
|-- 202.97.15.237    0.0%    10   57.1  58.5  43.7  72.2  10.7
|-- 202.97.46.109   60.0%    10   84.3  81.8  73.0  92.3   8.4
|-- 202.97.77.30    60.0%    10  101.2  92.6  84.5 101.2   7.8
|-- ???             100.0    10    0.0   0.0   0.0   0.0   0.0
|-- 220.173.118.215  0.0%    10   97.3  95.9  84.1 105.3   7.9

If a node has 100% loss rate, it probably disallows ping.
Usually, IP nodes beginning with 59.43 represents CN2 line.

Mobile Phone specifications

2017-02-05T00:00:00+00:00

Gonna buy a mobile phone? Ha, you are probably trapped by vendor inspiring presenation! What invented you are not the reality. To pick up a reasonable mobile phone, please examinate specifications.

Summary

Hardware is not everything!

iPhones' is relativly inferior to counterparts but the outcome is better.
You'd best buy a black version to hide black margin.
Phone at a discount might be substantially repaired ones.
Though most phones are produced by FOXCONN, final qualification determined by meta equipment, quality control, design etc.
Huawei's operating system is designed to prevent users from rooting! Think twice before upgrading.
Vivo/Oppo does not deserve the price.

Vivo/Oppo/OnePlus share the owner. Oneplus focuses on users overseas.

Chipset/SoC

SoC consists of CPU, GPU, Baseband processor (BP, BBP) etc, of which BP is the most paramount, responsible for telecommunication.

Qualcomm Snapdragon (CPU) is better than MediaTek (MTK) Helio.

Most processor vendors develop both CPUs and GPUs and tend to combine their own CPU/GPU in chipset (SoC). For example, Snapdragon CPUs are usually accompanied by Qualcomm Adreno GPU.
CPU model outshines number of Cores.
Pay attention to frequency, i.e. Snapdragon 820 1.8GHz VS 2.15GHz.

Each core may differ in frequency, say 6 cores of 1.5GHz while 2 another of 2.00GHz.
14nm/20nm.
GPU
1. Qualcomm Adreno (embeded into Qualcomm SoC);
2. ARM Mali;
3. PowerVR SGX (used by many Apple product);
4. NVIDIA Tegra;
BP is built-in or external? Support LTE user equipment category 6 (CAT6) / CAT7?

RAM/ROM

RAM - memory.
1. LPDDR3/4
ROM - flash disk where OS resides.
1. eMMc5.0/5.1/UFS 2.0/UFS 2.1
micro-SD card extension.

Battery

Charger. QC3.0/2.0/VOOC
1/2/3A

Wireless

80211b/a/g/n/ac wave1/ac wave2 (2-stream mu-mimo).

Camera

f/1.8, 2.0, 2.2
Optical Image Stabilization (OIS)
Phase Detection Auto Focus (PDAF)

Sensor

NFC
Ultra-red

Socket

Type-C USB 2.0/3.0/3.1
OTG

GTK recently-used.xbel

2017-01-23T00:00:00+00:00

GTK2/3 library creates ~/.local/share/recently-used.xbel for applications to keep their recently visited files, revealing user privacy! To disable recently-used.xbel:

Create directory with the same name.
```
~ $ rm .local/share/recently-used.xbel*
~ $ mkdir .local/share/recently-used.xbel
```
This is enough! The next configurations are alternatives by changing GTK2/3 configurations.

GTK3 setting.ini

~ $ echo -e "[Settings]\ngtk-recent-files-max-age=0\ngtk-recent-files-limit=0" >> ~/.config/gtk-3.0/settings.ini

GTK2 gtkrc-2.0

~ $ echo "gtk-recent-files-max-age=0" >> ~/.gtkrc-2.0

Refs:

Thinkpad X220 upgrading

2017-01-12T00:00:00+00:00

Thinkpad X220 is somewhat ancient, purchased in 2012. I am gonna improve memory and disk. There is a variety of different memory and disk alternatives. Before payment, think twice!

Specs

Model: 4290NL7 ISA bridge: QM67

Memory

4GB (1*4GB), 8GB max
PC3-10600 1333MHz DDR3 SDRAM
Base sizes: 1GB, 2GB, 4GB
Non-parity
Dual-channel capable
Two 204-pin SO-DIMM sockets

Disk

Primary Drive Bay: 2.5" wide, 7mm high, SATA3 or SSD.

Shipped with SATA2 320GB/7200rpm disk.
mSATA Slot: SATA2, compatible with all newer mSATA SSDs.
SATA2: 3Gbps; SATA3: 6Gbps.
The 2.5-inch bay can take slim 7mm hard drives and SSDs while the mSATA slot, located under the keyboard, can only take specialized mSATA SSDs.
Wireless PCI Express Mini Card for WWAN shares the same motherboard slot on board. They cannot be used simultaneously - sSATA or WWAN.

mSATA derived from mPCIe (mini PCI express) is an obsolete standard. System made after 2012 does not use neither of them any more. M.2 is the newer replacement.

Static electricity prevention

Static electricity, although harmless to you, can seriously damage computer components and options. Improper handling of static-sensitive parts can cause damage to the part.

Touch a metal table or a grounded metal object (i.e. faucet). This action reduces any static electricity from your body. The static electricity could damage the card.

Upgrading

Memory

A new 4GB strip.
Disk
1. Though SATA2 is almost half the speed of SATA3, it does NOT make much difference to daily operation without intensive I/O.
2. New cheap mSATA SSD now that expensive one cannot be fully exploited due to SATA2 limitation. What's more, SSD is not for data storage.

SSD

4k alignment;
4k random read/write performance (IOPS) is critical performance reference;
Check SMART data by official tool or CrystalDiskInfo;
Turn on AHCI in BIOS;

Theme

2016-11-23T00:00:00+00:00

Scheme
Xfce
Guake
Emacs

Scheme

x11-themes/zuki-themes-3.20 is compatible with both GTK2 and GTK3.
1. Make sure xfce USE is enabled to include relevant Xfwm4 part.
2. (opt) Get a copy of chrome/userChrome.css into Firefox's profile like ~/.mozilla/firefox/53rwll2m.default/chrome/userChrome.css.
papirus-icon-theme for Icons. The advantage of papirus-icon-theme is supporting hardcode-tray support, especially for Thunar icons.

Just copy Papirus and Papirus-Dark to ~/.local/share/icons/. Remember to gtk-update-icon-cache ~/.loca/share/icons/{Papirus,Papirus-Dark}.
Solarized is a palette theme mainly for editor (Emacs/Vim) and terminal emulator (Guake).

Xfce

Theme are divided into 5 different themes:

GTK2/3 theme: Applications/Settings/Apperance/Style. Rendering GTK toolkit like buttons, textboxes, drawing canvas, etc - everything within the border and title bar of a window.
Window Manager theme (Xfwm4): Applications/Settings/Window Manager/Style. The borders, title bar, maximize/minimize buttons, etc.
Icon theme (Adwaita): Applications/Settings/Apperance/Icons.
Notification theme: Applications/Settings/Notifications. Notification popup.
Cursor theme: Applications/Settings/Mouse and Touchpad/Theme.

All of these components play a part in determining the look of Xfce Desktop as you see it on the screen.

Check /usr/share/themes and /usr/share/icons for theme files.
Themes can be configured by command gconftool-2 either.
(deprecated) Consistent look for GTK2 and GTK3.

By default, x11-themes/gtk-engines-xfce:0 (slot 0 for GTK2 themes) is installed. We emerge it explicitly to pull in GTK3 themes (slot 3).
```
# emerge -avt x11-themes/gtk-engines-xfce
```
Term theme engine refers to preliminary themes that mainly serve as dependencies of other themes. For example, many excel themes (i.e. Arc-theme and Greybird) depend on gtk-engines-murrine.

The xfce devs dropped support for gtk3-themes in the default theme (gtk-engines-xfce), because the theming api broke so much in the releases that they couldn't keep the pace. now it's the job of the distributor to provide a common theme
Rude GTK3

GTK3 updates always break existing themes. The developers just ignore themes compatibility!

>=x11-libs/gtk+-3.20 breaks existing Xfce GTK3 themes (i.e. Firefox mouseover fails to display, scroolbar disappers, Ctrl-o no borders etc.).

Since many packages depend on 3.20, we have to either choose a compatible GTK3 theme (i.e. Greybird require many dependencies) or remove GTK3 theme from system (only slot 0). What's worse, Xfce's default GTK2 Raleigh theme is incompatible with some GTK3 applications.

Themes should be updated regularly to remain compatible with the GTK3 library installed on system.
Zuki - a GTK3-compatible theme

To manually configure consistent desktop look, choose a theme that covers as many aspects of those five categories as possible, cares about GTK version 2 and 3, and updates frequently to follow GTK3 library.
(deprecated) Missing icons

xfce-base/xfce4-meta depends on virtual/freedesktop-icon-theme. The lastest virtual/freedesktop-icon-theme ebuild has been changed to prefer x11-themes/adwaita-icon-theme (by default) over x11-themes/gnome-icon-theme. But the former does not contain icons for Xfce4 Desktop.
```
# emerge -avtn x11-themes/gnome-icon-theme
```
Since papirus-icon-theme is used, we leave Xfce's default icon themes alone.

Guake

Without explicit note, I use terminal and terminal emulator interchangeably.

Palette
1. Most terminals support palette setting in Preferences/Appearance.
2. If the built-in palette fails to meet your requirement, try separate palette manually like guake-colors-solarized.
3. Usually, choose solarized-dark.
TERM

Applications (Emacs Vim etc.) to run in terminal may depend on variable TERM to correctly apply palette.

Guake supports 256 colors (ancient terminals have only 16 colors). However due to this bug support true colors in terminal, Guake fails to set TERM (xterm) as xterm-256color. This can be verified by tput colors (8 foreground colors and 8 background colors) and M-x: list-colors-display in Emacs.

It's terminal emulator's job to set TERM on startup before any shell init scripts are loaded. But before the bug is resolved, we must resort to shell.
```
# ~/.bashrc
if [[ -e /usr/share/terminfo/x/xterm-256color ]] ; then
    case $TERM in (rxvt-unicode|screen|tmux|xterm) TERM=$TERM-256color;; esac
fi
```
Possible TERM values can be found unser /usr/share/terminfo.
dircolors-solarized

After changing to solarized-dark palette, ls output displays ugly. To fix that, apply Solarized color setup to ls output by dircolors - solarized palette for ls command.
```
~ $ cd ~/opt
~ $ git clone --depth=1 https://github.com/seebi/dircolors-solarized
~ $ ln -sv ~/opt/dircolors-solarized/dircolors.ansi-dark ~/.dircolors
~ $ echo 'eval `dircolors ~/.dircolors`' >> ~/.bashrc
# Uncheck *Allow bold font* in Preferences
```
There are four available color theme, namely dircolors.{256dark,ansi-universal,ansi-dark,ansi-light}.
1. If the terminal does not support Solarized palette (rarely), choose dircolors.256color which simulates Solarized by terminal's built-in 256 colors, resulting in an approximate Solarized theme. It must be a 256-color terminal and TERM is correctly set.
2. dircolors.ansi-universal is smart enough to work both for 16 colors and 256 colors.
3. If you are sure about the terminal colors, choose dircolors.ansi-{dark,light}. They are optimized version.
Summary
1. Guake has 256 colors;
2. Wrong TERM value due to bug (updated in ~/.bashrc);
3. Support Solarized palette in Preferences;
4. Choose dircolors.ansi-{dark,light}.

Emacs

emacs-color-theme-solarized

~ $ cd ~/.emacs.d/site-lisp
~ $ git clone --depth=1 https://github.com/sellout/emacs-color-theme-solarized

(add-to-list 'custom-theme-load-path (expand-file-name "site-lisp/emacs-color-theme-solarized" user-emacs-directory))
;; By default it's 16, but requires terminal emulator palette set to `solarized' and
;; make sure `TERM={xterm,screen,rxvt-unicode}-256color'. Otherwsie set to
;; 256 and got degraded but approximate `solarized' theme
;(setq solarized-termcolors 256)
(load-theme 'solarized t nil)
;; `solarized-light' in GUI mode; `solarized-dark' in terminal
(add-hook 'after-make-frame-functions
      (lambda (frame)
        (let ((mode (if (display-graphic-p frame) 'light 'dark)))
      (set-frame-parameter frame 'background-mode mode)
      (set-terminal-parameter frame 'background-mode mode))
        (enable-theme 'solarized)))

Fix

Currently, there is a 6-year bug not yet fixed, namely Emacs solarized in terminal with true 256 color is not correctly displayed.

If you encounter such issue, either strip -256color from TERM value or set (setq solarized-termcolors 256) in init.el. We can also set individual TERM value for Emacs like this:
```
TERM=${TERM%-256color} emacsclient -t -a "" "$@"
```
No matter which method is chosen, we have to degrade to 8-bit color in terminal before the bug is fixed!

Encoding

2016-11-22T00:00:00+00:00

Terminology

Code point: the raw code defined by people (Unicode, 区位码).
Cdoe unit: code after encoding.
Fixed witdth/length encoding: without transformation. Code points are directly indexable. Finding the Nth code point in a sequence of code points is a constant time operation.

For example. read two bytes each time.
Variable width/length encoding: MUST has transformation. Efficient space usage .

Read byte by byte.
Big/Little Endian (BE/LE):
1. Decides whether the bigger part (MSB, high bytes) or little part (low bytes) is stored first.
2. Firstly stored bytes actually resides on low disk/memory address.
Byte order mark (BOM). Depending on BE or LE, a byte read may represent different characters. BOM is inserted into the very beginning to differentiate BE and LE.
1. UTF-8: EF BB FF. BOM is not a must, but help mark it's UTF-8.
2. UTF-16/UCS-2 BE: FE FF.
3. UTF-16/UCS-2 LE: FF FE.
4. UTF-32/UCS-4 BE: 00 00 FE FF　　
5. UTF-32/UCS-4 LE: FF FE 00 00

Unicode/UCS

At the very beginning, there were two independent institutes deciding to desgin a international coding system, namely ISO (ISO 10646 project, UCS) and unicode.org (Unicode). Finally they together merged their desgin and chosen Unicode as the scheme name.

16 planes: U+0000 ~ U+10FFFF, at most 21 bits.
Basic Multilingual Plane (BMP): U+0000 ~ U+FFFF.
Includes almost all possible characters in the world.

UTF-16 vs. UCS-2

UTF-16 is downward compatibility with UCS-2.
UCS-2 is fixed width (16 bits) encoding scheme while the successor UTF-16 is variable width (sequence of 16-bit words).
UCS-2 covers code points in BMP from U+0000 to U+FFFF with any transformation.
Apart from the BMP, UTF-16 covers other planes with the help of surrogate code points (from U+D800 - U+DFFF) and results in a pair of 16-bit words, together called surrogate pair.
1. surrogate code point in Unicode has no corresponding characters. They are for furture extension.
2. So UTF-16 covers all Unicode planes from U+0000 to U+10FFFF with ether one 16-bit word or a pair of 16-bit word.
3. For BMP, Unicode = UCS-2 = UTF-16 (code point = code unit).
To decode a UTF-16/UCS-2 code unit, read two bytes or four bytes. One byte or three bytes is meaningless.
UCS-2 is obsolete.

UTF-32/UCS-4

UTF-32 and UCS-4 is identical.
Fixed-length encoding with exactly 32 bits. There are many leading zero bits.
Without any transformation. Each 32-bit value in UTF-32 represents one Unicode code point and is exactly equal to that code point's numerical value.

ANSI

You may encounter it in Windows.
It's not a concrete encoding, but scheme used Windows to select the correct encoding which is transparent to users.

If the Windows system is English, then the encoding behind is codepage cp437, while Chinese corresponds to cp936 (namely GBK).

国家标准信息交换用汉字编码，简称国标码（GB2312-80标准）

所有的汉字编码都兼容ASCII。更多细节参考外码？内码？

区位码：把汉字列成一94行94列的表，每行称区，每列称位。区位座标定位一个汉字，如“汉”的区位码是十进制2626，十六进制1A1AH，表示“汉”在表格的第26行第26列。
1. 区位码是编码设计的草稿。默认用十进制表示。
2. 2字节表示一个汉字。
国标码：区位码+2020H，得到国家标准编码GB2312（GB表示“国标”）.“汉”的GB2312码是1A1AH+2020H=3A3AH.

国标码是国家标准，不可乱改。注意，这里国标码最高位是0.
GB2312区位码和国标码都是人认识的编码，计算机是不知道的，通常称之为“（机）外码”。

字面意思是在计算外实用的码。
（机）内码：国标码+8080H或区位码+A0A0H，得到汉字在计算机内存储的编码。“汉”的机内码是3A3AH+8080H=BABAH.

把国标码的最高位置1，即得到机内码。ASCII字节最高位为0，计算机很容易区分汉字和英语。
计算机存储的是GB2312的机内码，而不是国标码。这与ASCII、UTF-8等直接存储编码不同。
Unicode和国标码不兼容，具体说是Unicode的code point和GB2312国标码之间没有换算关系。

GBK GB18030

GB2312的扩展，GB2312、GBK（K表示“扩展”）到GB18030向下兼容。
不同之处：
1. 后面的包含更多的汉字；
2. GBK GB18030只要求机内码第一字节开头置1（只有第一字节加80H），但兼容部分国标码两字节都必须是1开头。这不影响对GBK、GB18030的解码，只要遇到开头是1的，就接着都下一字节，合并解码。
从GBK、GB18030开始，机内码就是国标码，不需要什么区位码、国标码、到机内码转换。

所以通常叫GB2312为区位码，GBK为机内码，以示这个微妙地区分。
现在的PC平台必须支持GB18030，对嵌入式产品暂不作要求。所以手机、MP3一般只支持GB2312。
从编程角度看，GB2312、GBK到GB18030都属于“双字节字符集” (DBCS)，并且都是大端先存（BE）。属于Fixed Width Encoding的一种.
中文Windows的缺省内码是GBK，可以通过GB18030升级包升级到GB18030。不过GB18030相对GBK增加的字符，普通人很难用到，通常我们还是用GBK指代中文Windows内码。

Filesystem

2016-11-20T00:00:00+00:00

This post talks about filesystem operations under Linux environment.

Filesystem

Filename is a string encoded and stored in fileystem.
Linux kernel first decodes filename stored on disk (more specific, in filesystem on disk), then encodes it again, and finally submit to user space applications (like ls, thuar etc.).
When it comes to reading filenames on disk,
1. Windows NT kernel reads two bytes at a time. Everything in Windows NT kernel is UCS-2.
2. Linux kernel reads byte by byte, which means it recoginizes only variable length encoding streams, single byte encoding (i.e. cp437) included.
  
  The possible encoding streams are listed in Native Language Support (NLS) under File Systems in make menuconfig. But be careful, this NLS (in kernel space) is different from that one (nls USE and sytem locale) in user space.

FAT

FAT is not a separate filesystem, but a common part of the MSDOS, UMSDOS and VFAT.
1. Mount options of FAT applies to MSDOS, UMSDOS and VFAT either.
2. Attention, they are Linux filesystem drivers, not the original Windows FAT filesystems.
You might encounter two confusing options namely codepage and iocharset. Without proper setting, you would see garbled filenames.
1. To be simple, codepage encodes (on creation) and decodes (on display) shortname while iocharset should be the system locale.
2. No matter of codepage or iocharset, it's used by kernel instead of application.
3. The codepage for display should be the same as that of creation, otherwise it's decoded as garbage.
Details are a long story as follows:

shortname and longname

Shortname is a concept belongs to MSDOS only.
1. Looks like 8.3. The former is at most 8-byte long and the extension occupies 3 bytes.
2. Case insensitive. Actually only allow upper case characters.
Upon VFAT, long filename is supported, abandoning that two limitations.

In order to be compatible with MSDOS, each filename has a shortname version. We call the original one as longname. That is to say, VFAT stores two filenames in filesystem like:

longname; shortname, codepage
Shortname is encoded as codepage (i.e. 936 for Simplified Chinese), while longname is encoded as Unicode (NOT UTF-8).

Actually, most modern fileystems encode filenames as Unicode.

Mount

When it comes to mount, we talk about mounting FAT (MSDOS VFAT) under Linux.

codepage option. Kernel uses it to decode shortname and then translates it into Unicode. Longname is Unicode by default.
iocharset option. Kernel uses it to encode the Unicode shortname (MSDOS) or longname (VFAT). Then the encoded stream is passed to use space.

There is a special iocharset value utf8. You should set it in a separate way (discussed next).
Upon receiving the stream, application decodes it with the system locale (nls in user space).

Set the correct value

Suppose we have a VFAT fileystem initialized in Windows GBK sysetem. Now it will be mounted and used under Linux.

Shortname's codepage is cp936 while longname is Unicode.
Set codepage=936 mount option (without prefix cp).
1. If you want to see the shortname, use -t msdos instead of -t vfat. Meanwhile, iocharset is ignored.
2. Actually, we rarely use MSDOS nowadays. If you don't care, just leave it the default (in kernel).
iocharset= depends on system locale xx_YY.ZZ:
1. If ZZ is Chinese encoding like GB2312, GBK, GB18030, then set iocharset=cp936 (with prefix cp).
2. If ZZ is UTF-8, DO NOT set iocharset=utf8! Instead, use utf8 alone while keeping the default iocharset value.
If iocharset=utf8, the kernel vfat module allows lower case shortname which conflicts with Windows's tradition.

The kernel will throws a warning on such a case.
Remember the relevant NLS is compiled as Y or M.
The default codepage and iocharset (maybe utf8) value can be set in kernel.

NTFS

Shortname has nothing to do with NTFS. There is not such mount option as codepage.
The iocharset option of NTFS has changed to nls.
Similarly, if nls=utf8, you can use ntf8 alone.

There does not exist lower/upper case filename issue with nls=utf8.

HTTPd

2016-11-15T00:00:00+00:00

Lightweight HTTPd

Python

$ cd ~/workspace/webui-aria2
$ python3 -m http.server 6801 &
or
$ python2 -m SimpleHTTPServer 6801 &

Really primitive. Use it if you are lazy :).

darkhttpd;

It's pretty handy for temporary serving a static web page with multiple extended feature than Python's.
```
~ # darkhttpd /var/public_html --chroot --daemon --no-listing --uid nobody --gid nobody --maxconn 1 --log /var/public_html/traces.log
~ $ darkhttpd ~/opt/public_html --daemon --no-listing --maxconn 1 --log ~/opt/public_html/traces.log
```
1. Default port under root is 80 while 8080 for normal accounts. We can take vantage of iptable's REDIRECT module of nat table.
  
  If darkhttpd is running on remote host (i.e. real deployment), use PREROUTING chain:
```
# iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
```
  Loopback devices (like localhost) do not use the PREROUTING chain. If darkhttpd is running on localhost (i.e. for test), use OUTPUT chain:
```
# iptables -t nat -I OUTPUT -d 127.0.0.1 -p tcp --dport 80 -j REDIRECT --to-ports 8080
```
  Hence, there is no need to specify 8080 port in web URL.
2. Only root can and must chroot and drop privileges.
lighttpd;
1. Suffer from memory leak?
2. Support CGI/FastCGI.
nginx;
1. Good at static web page.
2. Popular than lighttpd?

CNAME vs URL Forward

A CNAME record is carried out on DNS server, irrelevant to server (SSH, HTTP, FTP etc.). That is to say, the CNAME url is not hosted on your servers at all.

The CNAME translation process is transparent to browse client, without user end involvement.
URL Forward is usually done on web server.

The server examines the host field of HTTP header, and return back the target URL to browser client if that matches a Redirect/Forward rule.
URL Forward is divided into two categories:
1. By terminology, Forward and Redirect are used interchangeably, depending on the context. But usually, Forward is a broader concept, meaning Redirect or Frame.
2. URL Redirect (i.e. 301, 302 Unmasked) is what we usually refer to. The browser gets a HTTP 3XX status code and moves permanently to the new site. Broswer address bar is updated to the target URL.
3. URL Frame (Cloaking/Masking) is similar but keep the original fetched web page. The new HTTP response will be embedded as a frame so that the visitor will always see the original URL in the address bar.
Platforms like Cloudclare, Freenom etc. support URL Forward nowadays.

Refs

Bootloader

2016-10-27T00:00:00+00:00

UEFI (also named as EFI) firmware on system board is the primary bootloader.
1. sys-boot/efibootmgr application is not a bootloader; it is a tool to interact with the UEFI firmware and update its settings.
2. Enable CONFIG_EFI_VARS so that the system firmware can be manipulated via efibootmgr.
Grub/LILO etc. are secondary bootloaders with more flexible extensions.
1. Secondary bootloader is not a requirement, even for multiple boot entries.
The kernel must have specific options enabled to be directly bootable by the system's UEFI firmware (CONFIG_EFI and CONFIG_EFI_STUB).
1. Be sure to read though the EFI stub kernel article before continuing.
2. To reiterate, efibootmgr is not a requirement to directly boot an UEFI system. The Linux kernel itself can be booted immediately.
3. Additional kernel command-line options can be built-in to the Linux kernel (CONFIG_CMDLINE).
4. Even an initramfs can be 'built-in' to the kernel, which is not recommended. To support initramfs directly, passing a parameter to the primary UEFI bootloader with the help of efibootmgr.
5. For additional kernels or Windows, enable CONFIG_EFI_VARS. Then create another UEFI entries by efibootmgr.

Keyboard layout

2016-10-14T00:00:00+00:00

ABCs

Emacs users are subject to Emacs pinky. Common solution is to switch Caps and Ctrl or turn off Caps.

Caps is recommended to turned off as it's only used occasionally. What's more, we can use Shift to shout.
Setting for virtual termnial is different from that of X, and should be set respectively.
In this post, I assume Intel PC and QWERTY keybord.

Virtual Terminal (OpenRC)

In Virtual Terminal, without specific clarification, term layout refers to keymap.

Keyboard layout can be found under /usr/share/keymaps. Keymap compressed in .gz format. Depending on the hardware, they are organized in sub-directories.

For Intel QWERTY keyboard, the default keymap is i386/qwerty/us.map.gz.

Among other keymaps, there is a special include directory offer supplemental keymap extensions.
There are two import parameters in /etc/conf.d/keymaps, namely keymap and extended_keymaps.

keymap=us sets keyboard layout to us.map.gz while extended_keymaps is a comma-listed extensions.

In this post, we will ignore extended_keymaps.
To switch Caps/Ctrl or turn off Caps, we can modiy the keycode in keymap (decompressed). Before that, let's examine the us.map format:
```
include "linux-with-alt-and-altgr"
keycode  29 = Control
keycode  58 = Caps_Lock
keycode  97 = Control
```
1. include extends the current keymap.
2. keycode 29 = Control means the Ctrl key is pressed which also applies to 58 and 97.
3. Specially 97 means the right Ctrl key while 29 is left Ctrl key.
Switch Caps and left Ctrl:
```
keycode  29 = Caps_Lock
keycode  58 = Control
```
Just swap the keycode for Control and Caps_Lock. Yeah, right, you can swap 29 and 97 (right Control) as well.
Turn off Caps:
```
keycode  58 = Control
```
Although modification the original default us.map.gz is handy, create a new keymap is preferrable.

Create caps-off.map:
```
include "us.map"
keycode  58 = Control
```
Remember to compress the new keymap:
```
$ gzip caps-off.map
# mv caps-off.map.gz /usr/share/keymaps/i386/qwerty/
```
Then set keymap=caps-off in /etc/conf.d/keymaps.
Finally restart keymaps service:
```
# rc-service keymaps restart
```
Interestingly, someone already compose a customized /usr/share/keymaps/i386/qwerty/emacs.map.gz.

I have chosen to use this one keymap=emacs. It includes much more customization for Emacs.
Attention: changing /etc/conf.d/keymaps directly affects only Virtual Terminal.

X

Aside from changing keymaps, X supports XKB in /usr/share/X11/xkb/rules/base.lst instead of raw keycode.

X KeyBoard extension, or XKB, defines the way keyboards codes are handled in X, and provides access to internal translation tables. It is the basic mechanism that allows using multiple keyboard layouts in X.
(deprecated) x11-apps/xmodmap is a somewhat low level tool, similar to OpenRC /etc/conf.d/keymaps.

xmodmap can temporarily change keymaps on command line (Terminal Emulator). However, for persistent effect, create a configuration file ~/.Xmodmap:
```
!
! Swap Caps_Lock and Control_L
!
remove Lock = Caps_Lock
remove Control = Control_L
keysym Control_L = Caps_Lock
keysym Caps_Lock = Control_L
add Lock = Caps_Lock
add Control = Control_L
```
Insert xmodmap ~/.Xmodmap command to ~/.xinitrc, ~/.Xresources, ~/.xprofile (Gnome Display Manager) or whatever. Alternatively, we can add it to desktop's autostart list. Most desktops (Xfce4, Gnome, or KDE) supports such setting.
XKB command line tool x11-apps/setxkbmap is a relativelly high level tool. If present, it resets /etc/conf.d/keymap and xmodmap settings.

Edit ~/.xinitrc or ~/.xprofile and call setxkbmap from there.
```
/usr/bin/setxkbmap -option "ctrl:swapcaps"
or
/usr/bin/setxkbmap -option "ctrl:nocaps"
```

XKB in Xorg configuration

Without installing setxkbmap package, we can use XKB rules in /etc/X11/xorg.conf.d/10-keyboard.conf:

Section "InputClass"
    Identifier      "keyboard-all"
    MatchIsKeyboard "on"
    #Option  "XkbOptions"    "ctrl:swapcaps"
    #Option  "XkbOptions"    "terminate:ctrl_alt_bksp"
    Option  "XkbOptions"    "ctrl:nocaps"
EndSection

Restart X to take effect.

Refs

DISQUS on GitHub page

2016-09-11T00:00:00+00:00

DISQUS is a centralized excel comments management tool. This post introduces basically how to add disqus plugin into a GitHub page generated by Jekyll.

DISQUS

First of all, you should go to disqus and signup.
Add your site to DISQUS. You can do this on the admin panel. DISQUS might direct you there automatically.
You should fill in some basic information like disqus shortname to identify your GitHub site, GitHub site url etc. In the community setting, Comments Count Link, change 0 Comments to Leave Comments. In the advanced setting part, you'd best fill in the trusted domain.

Finally you will get a piece of code depends on site platform. Since there is no default Jekyll platform, you choose universal code almost like this:

   
<div id="disqus_thread"></div>
<script>
    /**
 *  RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS.
 *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables
 */

    var disqus_config = function () {
    this.page.url = '{{ page.url | prepend: site.url }}';  // Replace PAGE_URL with your page's canonical URL variable
    this.page.identifier = '{{ page.id }}'; // Replace PAGE_IDENTIFIER with your page's unique identifier variable
    };

    (function() {  // REQUIRED CONFIGURATION VARIABLE: EDIT THE SHORTNAME BELOW
    var d = document, s = d.createElement('script');

    s.src = '//EXAMPLE.disqus.com/embed.js';  // IMPORTANT: Replace EXAMPLE with your forum shortname!

    s.setAttribute('data-timestamp', +new Date());
    (d.head || d.body).appendChild(s);
    })();
</script>
<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript>

<script id="dsq-count-scr" src="//EXAMPLE.disqus.com/count.js" async></script>
   

Pay attention to:
1. Edit and uncomment the disqus_config part. Replace PAGE_URL and PAGE_IDENTIFIER with your own Jekyll page variable.
2. Replace the EXAMPLE part with the disqus shortname just created for your GitHub site. That's all!

Embedding

After updating the code snippet, you will embed the code into your GitHub site source code.
Insert the code snippet between {% if layout.comments %} and {% endif %} of newly created _includes/disqus.html

Insert {% include disqus.html %} into _layouts/default.html after the {{ content }} part. The consequent code is like:

<div class="container content">
     
  {{ content }}
  {% include disqus.html %}
     
</div>

Insert a YAML Front Matter comments: true line into _layouts/page.html and _layouts/post.html respectively.
{% if layout.comments %} could be {% if page.comments %} depending on your needs. The latter requires comments: true on each post YAML Front Matter to turn on DISQUS comments. But the former seems not able to disable comments for a specific post. Details refer to Jekyll variable.
Up to now, you would have create a new file disqus.html, updated default.html, page.html and post.html. That's all!

Comments Counter

Besides comments, you can display the number of comments on a post.
Insert:
```
<script id="dsq-count-scr" src="//EXAMPLE.disqus.com/count.js" async></script>
```
to the end of _includes/disqus.html but before the last line {% endif %}. The EXAMPLE should be replaced your real disqus shortname.
Next insert:
```
   
{% if layout.comments %} • <a href="{{ site.url }}{{ page.url }}#disqus_thread">Leave Comments</a>{% endif %}
   
```
where you would like to show the comments counter. Usually, it should be _layouts/post.html. You can also insert it into index.html to show counters for each post on the front page of your site.

The key is the trailing #disqus_thread which is the DIV id of the very first snippet code above. The layout in {% if layout.comments %} might be changed based on your previous choice.

That's all!

Censorship

Seemingly DISQUS does not show up due to GFW block. To get this solved, you must:

Enable HTTPS for your site url. If you did not set a customized domain, then good luck. The original default GitHub page is accessed by HTTPS. Otherwise, you must set your own HTTPS which usually needs payment service.
Use a proxy to visit your HTTPS site.

Refs

Intel Graphics

2016-09-10T00:00:00+00:00

It's long-lasting panic to handle Intel HD3000 graphics corruption, tearing, judder, glitch etc. Desktop GUI get stuck without any responding. MPV playback spreads dots all over. Worsely, switches between virtual terminal and X freeze the desktop applications.

Intel DDX

It is almost deprecated but has better performance. We should manually install the official relevant Intel driver:

root@tux ~ # emerge -avt x11-drivers/xf86-video-intel

Then configure Xorg to adopt it:

# /etc/X11/xorg.conf.d/20-intel.conf

Section "Device"
        Identifier        "Intel Graphics"
        Driver        "intel"
#       Option        "AccelMethod"        "uxa"
        Option        "AccelMethod"        "sna"
#       Option        "TearFree"        "true"
        Option        "DRI"        "2"
#       Option        "DRI"        "3"
EndSection

This driver is just maintained but no new features or bug fixes available.

Modesetting DDX

This is now the default driver on newer Intel graphics chipsets for Gentoo. It is more generic and actively developed. As of x11-base/xorg-drivers-1.19, this has become the default for Gentoo.

Firstly, enable global glamor USE:

# /etc/portage/make.conf

USE="glamor"
VIDEO_CARDS="intel i965"

Then, Xorg configuration:

# /etc/X11/xorg.conf.d/20-modesetting.conf

Section "Device"
    Identifier  "Intel Graphics"
    Driver      "modesetting"
    Option      "AccelMethod"    "glamor"
    Option      "DRI"            "3"
EndSection

Note, if both 20-intel.conf and 20-modesetting.conf are defined in /etc/X11/xorg.conf.d/, the X server will attempt to load the files in alpha-numeric order.

Xorg arguments

startx -- vt7

Arguments after the two dashes are passed to Xorg server. Default OpenRC Xinit configuration is located under /etc/X11/xinit. However it fails to set the correct virtual terminal (i.e. vt7) to start X, resulting in X freezes upon switches between X and virtual terminal.

Alternatively, vt7 can be passed to X server directly in ~/.xserverrc:

exec /usr/bin/X -nolisten tcp "$@" vt7

ThinkPad Buttons

https://wiki.gentoo.org/wiki/ACPI/ThinkPad-special-buttons

Screen brightness

# /etc/default/grub

acpi_osi="!Windows 2012"
# -or-
acpi_osi="Linux"

acpi_backlight="video"
# -or-
acpi_backlight="vendor"
# -or-
acpi_backlight="native"

At the time of writing this post, correct combination is acpi_osi="!Windows 2012" acpi_backlight="video".

Disk Partitioning and Filesystem

2016-05-27T00:00:00+00:00

Hard Disk Drive (HDD)
Firmware and Interface
Partitioning table
Booting scheme
Partitioning tool
Sector
Filesystem
Bootable USB stick
Operating System Limitations
Summary
Manually creating Windows UEFI bootable USB stick
1. Windows 7 Notes

Hard Disk Drive (HDD)

In terms of transmission method:

Parallel: IDE/ATA/PATA; SCSI.
Serial: Serial ATA/SATA; Serial Attached SCSI/SAS.

In terms of application:

SCSI/SAS: workstation, RAID.
ATA/SATA: PC.

In terms of features:

ATA/SATA: hotplug; cheap; warm-friendly.
SCSI/SAS: stable; higher performance.

Firmware and Interface

Firmware refers to underlying hardware as 'UEFI Firmware' or 'BIOS Firmware'. Nowadays, almost all PCs bring in UEFI Firmware.
At the very beginning of booting, we press ESC/F2/F12 and get into 'Firmware interface' as System BIOS, ROM BIOS or PC BIOS which is also named to BIOS Setting.

Partitioning table

GPT
MBR (MSDOS)
MAC
BSD

Booting scheme

Legacy (BIOS) booting
UEFI booting.
UEFI Firmware supports either UEFI booting or Legacy booting, which is called Compatibility Support Module (CSM).

Partitioning tool

To create partition table and partitions

parted
fdisk
diskpart on Windows

Both fdisk and parted are partitioning utilities. fdisk is well known, stable, and recommended for the MBR partition layout while parted was one of the first Linux block device management utilities to support GPT partitions.

Sector

Read Storage I/O Alignment and Size first! It is imperative to differentiate the concepts of physical sector and logical sector, especially on partitions alignment.

On the hardware level, we have sectors with 512B (0.5KiB) in the old days. In order to improve disk scability and abilities of error check, new disk drives with 4096B (4KiB) sectors are inovated. It is nonnegligible that there exist gaps between physical sectors, data within which could be utilized to detect disk error and recovery partial data.

For backward compatibility, we have physical sector and logical sector, which is called advanced format feature. Many ancient operating systems does not support reading sectors with 4KiB. Consequently, operating systems still read and write logical sectors of 512B (0.5KiB) while disk driver operates on physical sectors in 4096B (4KiB). Here is an example output of parted command. It makes sense that partition size is presented with logical sectors, when unit is selected to _s_ector.

Model: ATA HITACHI HTS72323 (scsi)
Disk /dev/sda: 625142448s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Obviously, 1 physical sector corresponds to 8 logical sectors. We can check both values through Linux runtime sysfs:

/sys/block/sdX/queue/physical_block_size
/sys/block/sdX/queue/logical_block_size

/sys/block/sdX/alignment_offset          # 0
/sys/block/sdX/sdXY/alignment_offset     # 0

/sys/block/sdX/queue/optimal_io_size     # 1048576 = 1024 * 1024 = 1MiB
/sys/block/sdX/queue/minimum_io_size

fdisk can also reports optimal_io_size and minimum_io_size. To get authorative results, we resort to the disk official page by checking disk model first:

/sys/block/sdX/device/model

For the purpose of disk performance, please align the logical partition table addresses to actual physical blocks on the disks. Make sure partitions start at boundaries of:

(optimal_io_size + alignment_offset) / physical_block_size = 256 physical sectors
# -or-
(minimal_io_size + alignment_offset) / physical_block_size

Alignment does not care about the ending boundary at all. That is to say, space gap may exist between partitions.

Minimal Alignment

Keep in mind, on software layer (both OSes and partitioning tools), sector is equivalent to logical sector. We sometimes call partition boundaries on software level as logical sector address (LBA).

We are suggested to create partitions at the exact boundary of physical blocks, namely oders of 4096 (8s) like 40s, 48s, etc.

For instance. I want to create a Grub GPT/BIOS boot partition. Although alignment is not critical to this partition (not regularly accessed), I present it here to illustrate the rules above.

Usually, a Grub BIOS boot partition takes around 1MiB = 2048s. On GPT disks, the very first usable sector is 34s since the size of the EFI label is usually 34 sectors (0s - 33s).

So we can choose the starting sector to be 40s and the ending point be 2047s (included). Please be noted, here we use 2047s as the ending sector such that the next partition can start at 2048s. Hence, a properly aligned partition ends at multiple 8s minus 1.

root@tux / # parted -a opt /dev/sda
(parted) unit s
(prated) mkpart primary 40s 2047s
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel?
(parted) Ignore
(parted) align-check opt 1
(parted) align-check min 1

Now that the the starting point and ending point is set appropriately, why does parted still reports warning? Because -a opt tells the partition to align optimally. minimal tells to align precisely at multiple physical sectors. Please set unit to KiB, MiB, or GiB (power of 2, orders of 1024): require exact unit.

Use minimum alignment as given by the disk topology information. This and the opt value will use layout information provided by the disk to align the logical partition table addresses to actual physical blocks on the disks. The min value is the minimum alignment needed to align the partition properly to physical blocks, which avoids performance degradation.

Optimal Alignment

By default, parted uses optimal alignment at inexact megabytes:

Use optimum alignment as given by the disk topology information. This aligns to a multiple of the physical block size in a way that guarantees optimal performance.

The optimal uses inexact units like K, M or G (power of 10, oders of 1000). It aligns at a more general level, usually in MB and allows +/-500KB (500MB for GB unit) adjustment automatically, like mkpart primary fat32 1MB 201MB. Specially, we can set unit %. Hence start a partition by percentile is appreciated like mkpart primary fat32 0% 551MB.

The resulting boundary is determined by optimal_io_size (check the formula above). If the that value is zero, then conform to 1MiB. Therefore, in real practice, just use the exact units or percentiles, as alignment is guranteed.

Alignment Summary

LVM partitions comply with with the same alignment rules above.
Interestingly, HD manufactures use order of 1000 to present disk size as that would make the number larger.
In math, 1MB is actually 1M Bytes while 1GiB is 1Gi Bytes.
We use unit _s_ector to create small partitions while MiB/GiB for large partitions.

4 KB 扇区磁盘上的 Linux：实用性建议; What's the point of hard drives reporting their physical sector size?

Filesystem

'Format' refers to create filesystem on a disk parition.

NTFS
FAT 12/16/vfat/32
EXT 2/3/4, XFS
BTRFS

EFI System Partition (ESP) (including bootable USB stick) must be a FAT (FAT32 is recommended) filesystem. Now most Linux distributions use xfs instead of ext4.

Bootable USB stick

dd (rude but robust)

The ISO is installed to the whole USB stick as if it were a hard drive. This make USB stick unacessible to OS file manager due to boot and/or esp flag.
Rufus (reliable and versatile tool)
diskpart (Windows)
Ubuntu Disk Creater
Manual copy.

For UEFI booting, just copy (i.e. rsync) ISO contents to a FAT32 USB partition without altering other partitions that can be used as data storage.

For Windows bootable USB stick, sometimes we encounter lack of the exact USB driver from the ISO, resulting in installation failure in the early phase. You are recommended to change the USB stick (i.e. a newer with USB 3.0 port). Optionally, you can get a driver copy and place it a location that is accessible during installation (i.e. within another USB stick).

Operating System Limitations

Usually, MBR and BIOS (MBR + BIOS), and GPT and UEFI (GPT + UEFI) go hand in hand without any compatibility issue. However, that matching is a must for Windows and Mac while Linux is more flexible.
For Dual boot with Windows, turn off Windows 'Fast Reboot' and 'Secure boot' in BIOS setting.

What's more, then stick to the matching rule unless they are installed separately on two disks.
UEFI requires the firmware and operating system loader (or kernel) to be size-matched; for example, a 64-bit UEFI firmware implementation can load only a 64-bit operating system (OS) boot loader or kernel.

Summary

The final booting scheme depends on:

Firmware: UEFI/BIOS Firmware;
BIOS setting: uefi/Legacy/CSM booting;
Partitioning: GPT/MBR;
Operating System (and boot loader thereof).

Manually creating Windows UEFI bootable USB stick

This tutorial shows how to manually create a Windows 8.1 bootable USB stick under Linux. parted does NOT ask for confirmation on each command like fdisk.

On Windows, we use diskpart. Alternatively, format the USB to FAT32 and copy all ISO files to the USB - quite simple!

Create partition table - GPT

# parted -a optimal /dev/sdb
(parted) mktable/mklabel gpt
(parted) unit s
(parted) print free

mktable/mklabel ERASEs the whole disk data!

Create a primary partition
```
(parted) mkpart primary fat32 0% 5GiB
(parted) print free
(parted) align-check opt 1
(parted) name 1 Win81USB
(parted) print free
```
1. The fat32 argument is optional at this step. It may be specified to set an appropriate partition ID. parted does not format filesystem.
2. 0% is better for start of the very first partition alignment. Similarly, 100% is better for end of the very last partition alignment.
3. 5 GiB is enough for Windows 8.1 ISO. The remaining space can be used to store data (i.e. create ext4 partition).
4. align-check checks partition alignment. optimal/opt guarantees optimal disk performance. Try to use unit s (sector) when mkpart for better disk performance.
(opt) Set the partition be bootable

parted presents GPT partition GUIDs as flags while other tools like gdisk uses short codes (i.e. ef00). However, those codes mean nothing at all in parted.
```
(parted) set 1 boot on
or
(parted) toggle 1 esp
(parted) print free
(parted) quit
```
1. Enable either boot or esp as they imply each other (alias) on GPT disks. We can also set msftdata for Windows stick.
2. As mentioned earlier, boot or esp flag makes this partition unacessible to Windows OS file manager.
Format FAT32 filesystem
```
# mkfs.vfat -F32 /dev/sdb1
```
1. Though we sepcify fat32 argument in parted, we must explicitly format the partition.
Mount ISO file
```
# mount -t udf -o loop,ro,unhide /path/to/file.iso /mnt/iso
```
Make usre udf_fs is enabled in kernel to support DVD ISO. The default iso9660 only supports CD ISO.
Mount the USB stick
```
# mount /dev/sdb1 /mnt/usbstick
```

Copy ISO files to USB stick

# cp -rv /mnt/iso/* /mnt/usbstick/
# -or-
# rsync -rv /mnt/iso/ /mnt/usbstick/

Flush file system buffers
```
# sync
```

Umount ISO and USB stick

# umount /mnt/iso
# umount /mnt/usbstick

For Windows 10 onwards, the "install.wim" file is over 4GB and cannot be copied to a FAT32 filesystem. So we need to split the file into smaller ones before copying.
Refs
1. fedora create windows 8.1 USB stick
2. howtogeek cd/dvd ISO mount

Windows 7 Notes

By default, Windows 7 ISO does not prepare the bootx64.exe and relevant directories correctly. Use robust tool like Rufus.

For whatever reason you want to create usb stick by copying or dd, please get bootx64.exe from an existing Windows 7 OS. Alternatively, extract (i.e. by 7zip) \1\Windows\Boot\EFI\bootmgfw.efi from ISO \sources\install.wim and rename it to bootx64.exe. From within the USB stick, copy \efi\microsoft\boot directory to a upper level, namely \efi\boot into which we put bootx64.exe. Actually, Rufus just creates \efi\boot directly instead of copying.

aria2 + webui

2016-05-24T00:00:00+00:00

aria2 is a Linux command line counterpart of Windows Thunder/IDM, which supports URL, BT/PT, metalink etc. Make it simple, configure a GUI like Webui-aria2 or YAAW based on its builtin RPC (JSON or XML) support.

Installation on PC

# echo "net-misc/aria2 bittorrent metalink xmlrpc" > /etc/portage/package.use/aria2
# emerge -avt aria2

If you prefer XML RPC to default JSON RPC, enable xmlrpc scripts USEs.
We can invoke aria2 command - aria2c.

Refer to official examples.
To fully utilize functionalities, we can:
1. Enable RPC and daemon;
2. Put common arguments in configuration file;
3. Install webui-aria2 and enable directurl;
4. Set alias for different configurations.

Installation on OpenWRT

TO-DO

Configuration

If you don't bother remembering arguments like me, speicify the common ones in $HOME/.aria2/aria2.conf or $XDG_CONFIG_HOME/aria2/aria2.conf.
You can also specify a different configuration on command line by --conf-path.
Current version is 1.15.2 and limited:
1. ${HOME} or ~ cannot be parsed in configuration file. Use absolute path.
2. Doesn't support default $XDG_CONFIG_HOME/aria2/aria2.conf location. Use --conf-path.
3. More options are only available in newer versions:

# https://1024coder.com/topic/15
# https://gist.github.com/aa65535/5e956c4eb4f451ddec29
# http://aria2c.com/usage.html
# http://blog.binux.me/2012/12/aria2-examples/
#
# Version 1.15.2 does not support ${HOME} or ~ in .conf

# Log
#
log=/home/jim/.local/share/aria2/aria2.log
log-level=debug
summary-interval=120

# Disk
#
dir=/home/jim/Downloads
# Allocates file space before download begins
# This may take some time depending on file size
# and filesystem
# NTFS: falloc; Ext3/4: trunc
# Refer to `dir' option
file-allocation=none
#enable-mmap=true
# >= 1.16
#disk-cache=32M

# Concurrency
#
# Resume a download started by a web browser or another program
# which downloads files sequentially from the beginning
continue=true
# maximum downloads
max-concurrent-downloads=3
# maximum connections to one server per download
max-connection-per-server=3
# connections/threads per download 
split=5
# split if file >= 2*5M=10M
min-split-size=5M
# overall limit
max-overall-upload-limit=1K
max-overall-upload-limit=0
# Avoid terribly slow AAAA record lookup
disable-ipv6=true

# Session
#
input-file=/home/jim/.local/share/aria2/aria2.session
save-session=/home/jim/.local/share/aria2/aria2.session
# >= 1.16.1
#save-session-interval=60

# RPC
#
daemon=true
enable-rpc=true
rpc-allow-origin-all=true
# If used in OpenWRT, use 'true'
#rpc-listen-all=false
# Customize port number if the default is occupied
#rpc-listen-port=6800
rpc-user=aria2
rpc-passwd=aria2c
# better but >= 1.18.4
#rpc-secret=token

# BT/PT
#
#follow-torrent=true
#bt-max-peers=55
#listen-port=6881-6999
#dht-listen-port=6881-6999
#bt-request-peer-speed-limit=50K

# schedulled command; useful in OpenWRT
#on-download-complete=exit

PT camouflage

PT sites (mainly those on campus) forbid aria2. We should disguise it behaving like a normal uTorrent.

# PT camouflage
#
# https://www.librehat.com/aria2-camouflage-utorrent-pt-download/
# http://tieba.baidu.com/p/2894158631
# http://kzpu.com/archives/3526.html
enable-dht=false
enable-dht6=false
bt-enable-lpd=false
enable-peer-exchange=false
user-agent=uTorrent/341(109279400)(30888)
peer-id-prefix=-UT341-
seed-ratio=0
# No this option in 1.15.2
#force-save=true
bt-hash-check-seed=true
bt-seed-unverified=true
bt-save-metadata=true

Running

Version 1.15.2 does not recognize default $XDG_CONFIG_HOME/aria2/aria2.conf location. Use --conf-path.

$ aria2c --conf-path=${HOME}/.config/aria2/aria2.conf

Make sure rpc-listen-port is allowed on host by iptables (on OpenWRT or PC). If --rpc-listen-all is True, aria2 listens incoming JSON-RPC/XML-RPC requests on all network interfaces. We can strict incoming requests to a specific interface by -d host-ip.

# iptables -t filter -A INPUT -p tcp --dport 6800 -j ACCEPT
or
# iptables -t filter -A INPUT -p tcp -m tcp -d host-ip --dport 6800 -j ACCEPT
or
# iptables -t nat -A WANPREROUTING -p tcp --dport 6800 -j DNAT --to-destination host-ip:6800
# iptables -t filter -A INPUT -p tcp -m tcp -d host-ip --dport 6800 -j ACCEPT

webui-aria2

Refer to webui_aria2c. Remember that we set enable-rpc=true above to manage aria2 on RPC interface. For example, you configure aria2 at home OpenWRT. You are in office and would like to file an AV download by webui interface.

Installation

git clone

$ cd ~/workspace/
$ git clone --depth=1 https://github.com/ziahamza/webui-aria2.git
$ cd webui-aria2

Public URL

Visit Public URL and fill in Settings -> Connection Settings.

RPC connection

The easiest way: visit the Public URL.

If you don't trust the public URL like me, install webui manually.
Open index.html in browser if cloned on access PC.
Use Python simplehttpserver to host the cloned webui. RPC listens at 6800 and we can set HTTP Server listens at 6801.

This is similar to the Pbulic URL but hosted by our own. You'd better host the webui HTTP Server on PC, public IP (i.e. on VPS) or even on OpenWRT.
```
$ cd ~/workspace/webui-aria2
$ python3 -m http.server 6801 >/dev/null 2>&1&
or
$ python2 -m SimpleHTTPServer 6801 >/dev/null 2>&1&
```
Browse http://localhost:6801 or http://ip_address_of_server:6801.

Connection Settings

If you access webui by index.html or locally hosted HTTP Server, set user, pass, host etc. in configuration.js permanently.
```
$ cd ~/workspace/webui-aria2
$ nano -w configuration.js
```
These values can be changed on the webui HTML page afterwards.
If webui is hosted on public IP (VPS/OpenWRT), don't set those variables as it can be compromised. For example, if someone got your router IP and webui HTTP Server port, he could access to your aria2 daemon.

Instead, leave the configuration.js alone and fill in values on webui page (like a web page logon).

directurl

This feature allows users to download files that they download from aria2 directly from the webui dashboard.
Suppose aria2 is downloading on OpenWRT, directurl generates clickable link to downloaded files on webui. Simplest way is to use Python to setup a HTTP server like the hosting webui page. webui listens at 6801 and we can let directurl listens at 6802.

This time, the HTTP Server must be hosted on -d, --dir.

$ cd ~/Downloads/
$ python3 -m http.server 6802 >/dev/null 2>&1&
or
$ python2 -m SimpleHTTPServer 6802 >/dev/null 2>&1&

To test, just browse http://serverip:6802.

It should get a directory listing. Any browser errors and something hasn't been done properly, check IP/PORT etc.
Depend on your needs, set directURL to 'http://localhost:6802/' in configuration.js. Or set Settings -> Connection Settings -> Enter base URL on webui page.

Make sure have the / on the end
After that, all downloaded file can be dragged to local computer (PC in use) on webui page.
[to be confirmed] Make sure port 6802 is also allowed by iptables.

The simplest Directurl is file:///${HOME}/Downloads/.

Iptables

By default, everything works fine. But if aria2 does not work, check ports 6800, 6801, 6802, 6881-6999 etc. are allowed in Iptables.

rpc-listen-port: default 6800;
webui and directurl HTTP Servers ports: 6801 and 6802;
listen-port: default TCP ports 6881-6999;
dht-listen-port: default UDP ports 6881-6999.

In the OUTPUT chain, TCP/UDP 6881-6999 MUST be accepted:

-A OUTPUT -p tcp -m tcp --dport 6881:6999 -m conntrack --ctstate NEW -j ACCEPT
-A OUTPUT -p udp -m udp --dport 6881:6999 -m conntrack --ctstate NEW -j ACCEPT

For the INPUT chain, it depends.

Scheme

aria2 and webui are on the same host (OpenWRT/PC/VPS etc.).
They are on different hosts. For instance, aria2 is on OpenWRT while webui is hosted on VPS or PC.
If webui is on PC, I think there is no need to set up a separate HTTP Server for webui or directurl. We can just open the index.html from browser bookmark. And we can access our downloaded files directly on local PC.
We can add init script to load aria2 on boot (OpenWRT).

The following setting is to run both aria2 and webui on PC.

Alias on PC

We need different set of arguments for different downloads. For example, to download PT resources, we should camouflage aria2 as uTorrent. But options for PT impact BT download performance. We can set an alias for PT by --conf-path.

$ nano -w ~/.bashrc
alias aria2='aria2c --conf-path=${HOME}/.config/aria2/aria2.conf'
or
alias aria2='aria2c --conf-path=${HOME}/.config/aria2/aria2.conf; cd ${HOME}/Downloads/; python3 -m http.server 6802 >/dev/null 2>&1&'
or
alias aria2='aria2c --conf-path=${HOME}/.config/aria2/aria2.conf; cd ${HOME}/workspace/webui-aria2/; python3 -m http.server 6801 >/dev/null 2>&1& cd ${HOME}/Downloads/; python3 -m http.server 6802 >/dev/null 2>&1&'
or
alias aria2pt='aria2c --conf-path=${HOME}/.config/aria2/aria2pt.conf'
$ source ~/.bashrc

Launch on PC

Launch RPC daemon
```
$ type aria2c aria2 aria2pt
$ aria2
```
Open webui index.html file or access http://localhost:6802.

Issues

Magnet zero speed.

When downloading magnet files, there is no speed. This is due ot lack in dht entry point. Details refer to issue 504, dht bootstrap nodes, and aria2 and magnet links

Set --dht-entry-point as dht.transmissionbt.com, router.utorrent.com or router.bittorrent.com. Port is 6881.

Browser plugins

TO-DOs

Plugins for BaiduYun.
Plugins to lauch aria2.

Refs

Enforce HTTPS to your site by Cloudflare

2016-05-06T00:00:00+00:00

Cloudflare
DNS switch
1. Cache and Security
SSL settings
Cache all the things
DNSSEC
Last but trivial
GitHub CDN vs Cloudflare CDN
Refs

By default, GitHub takes HTTPS protocol to user/project pages i.e. username.github.io. However, for a custiomized domain, HTTPS is unsupported. This post tells us to enforce HTTPS to GitHub page through free Cloudflare services. Mainly, we just need to tune a few settings on Clouflare and your domain registrar.

To be honest, Cloudflare's free CDN caching may slow down instead of accelerating your sites.

Cloudflare

Cloudflare offers us free DNS and CDN services whilist concentrating on customer security. Those who would like to enforce HTTPS to his sites could take advantage of those services. For instance, Cloudflare will protect your site email address from web crawler.

The detailed free services are listed below. To achieve HTTPS, only the first three is a must.

Free automatic HTTPS for your domain - no need to buy a certificate;
Page Rules - custom settings and redirects for URL patterns;
HTTP Strict Transport Security (HSTS) - protection from MITM attacks;
DNSSEC - protection from DNS poisoning attacks;
HTTP/2 - optimised connections for browsers that support it;
CNAME flattening - so you can use a DNS CNAME at the domain apex;
"Always online" protection - Your cached website will stay up even if the host goes down;
Firewall - intelligent protection against DDOS attacks.
websocket
Require Modern TLS on Crypto tab.

We should register a Cloudflare account and then add site. Follow the add site procedure, we will finish the HTTPS enforcement. Before we go into details, we should clarify a few points:

First, write down your custom GitHub page domain/sub-domain, i.e. example.com or blog.example.com. But top domain is recommended even you are using a sub-domain.
Find your domain registrar management interface, i.e. www.freenom.com.
In the domain management interface, find the setting of DNS servers and DNS records.

We will basically do two things:

Cloudflare imports DNS records and switch DNS servers to Cloudflare's.You may delete imported DNS records from freenom, namecheap etc.
Turn on a few security settings on Cloudflare. That' all!

DNS switch

Subdomains cannot be added on their own.

After filling in your site url following add site procedure, Cloudflare will analyze and import the site's DNS records (from your previous platforms).

We can add, delete a new DNS records. Even, we can toggle Cloudflare per a record.
Then continue, Cloudflare will give us two new Cloudflare DNS servers.
Go to our domain registrar management interface, and replace all original DNS servers with the new ones.

To make use of Cloudflare service, all other platforms' DNS servers must be deleted. Only Cloudflare ones are permitted.
Up to now, Cloudflare takes over DNS servers and DNS records management.

Cache and Security

Cloudflare as a man in the middle (MITM)

By default, Cloudflare accelerate and protect (i.e. cache files) sites (an orange cloud symbol at the end of DNS record).

You may want to disable that functionality for specific sub-domains like irc.example.com. For example, local ISP blocks Cloudflare CDN servers. On the other hand you lose the free services mentioned above.

Alternatively, it hides web site's original location (i.e. IP). Refer to post V2ray for details. You can check by dig your domain and find IP changed to that of Cloudflare's CDN servers.

ATTENTION: Cloudflare decrypts all traffic from browser and negotiate new TLS with destination server. It is common that a CDN platform just uses HTTP. Hence, we cache static web pages or public multimedia. For confidential communication like password login, please avoid CDN.

SSL settings

Unfortunately GitHub doesn't yet support SSL for custom domains which would ordinarily rule out using HTTP/2. Whilst the HTTP/2 specification (RFC 7540) allows for HTTP/2 over plain-text HTTP/2, all popular browsers require HTTP/2 to run on top of Transport Layer Security; meaning HTTP/2 only being able to run over HTTPS is the de facto standard.

In the Crypto tab of your CloudFlare site you should ensure your SSL mode is set to Full but not Full (strict).
We can now add a Page Rule to enforce HTTPS, as you add other Page Rules make sure this is the primary Page Rule:
```
http://*example.com/*

Always Use HTTPS
```
Page rule happens before DNS record resolving.
We can also create a Page Rule to ensure that apex is redirected to www securely when using HTTPS:
```
example.com/*

Forwarding URL (Status Code: 301 - Permanent Redirect)

https://www.example.com/$1
```
This rule should sit before the previous one since it's more specific.
Back to the Crypto tab, enable and set HTTP Strict Transport Security (HSTS) service. HSTS (RFC 6797) is a header which allows a website to specify and enforce security policy in client web browsers.

The recommended settings are:
```
Status: On
Max-Age: 6 months (recommended)
Include subdomains: On
Preload: On
No-sniff: On
```
That's all.

Test

Though we should wait for a while (hours maybe) to acccess GitHub pages over HTTPS, we can test security settings by curl -I example.com:

HTTP/1.1 301 Moved Permanently
Date: Mon, 12 Sep 2016 11:14:57 GMT
Connection: keep-alive
Set-Cookie: __cfduid=afj4n38eahdglvmneuptoq84h; expires=Tue, 12-Sep-17 11:14:57 GMT; path=/; domain=.exmaple.com; HttpOnly
Location: https://example.com/
X-Content-Type-Options: nosniff
Server: cloudflare-nginx
CF-RAY: 3fien1q9ehpen-HKG

Disable SSL

Attention: if you decide to disable SSL for whatever causes, please disable HSTS before HTTPS.

Cache all the things

CloudFlare has a “Cache Everything” option in Page Rules. For static sites, it allows your HTML to be cached and served directly from CloudFlare's CDN. This will significantly accelerate your site access time.

Add a last rule as:

https://*example.com/*

Cache Level: Cache Everything

When deploying your site you can use the Purge Cache option in the Cache tab on CloudFlare to remove the cached version of the static pages.

DNSSEC

If DNS is the phone book of the Internet, DNSSEC is the protocol that ensures that a number in the phone book actually belongs to the contact listed. DNSSEC uses cryptographic signatures to verify that the DNS records returned for a domain are untampered.

If your domain registrar support DNSSEC, please turn on DNSSEC on Cloudflare DNS tab.

Last but trivial

As shown above, to turn on HTTPS, we just tune a few settings, leaving the GitHub page sources untouched.

But we should at least change the url variable in _config.yml to its HTTPS version.

GitHub CDN vs Cloudflare CDN

Without Cloudflare, GitHub supports CDN as well and default GitHub page acces time is trivial. However, Cloudflare offers extra HTTPS service.

Refs

TCP Obfuscation - ptproxy

2016-04-12T00:00:00+00:00

~~ptproxy~~

is almost dead. The repository is ancient whilist aisocks has dropped support for Python 3.4! Try simple obfs instead!

ABCs

ptproxy (author page) obfuscates TCP traffic with the help of Tor's PT (pluggable transports) prototol. This tool is independent of Tor and applies to any TCP traffic. In this post, I demonstrate how ptproxy obfuscate Shadowsocks traffic.

To speed up TCP connection to VPS, try KCPtun.
Before ptproxy

app -> Shadowsocks client -> Shadowsocks server -> destination host
With ptproxy

app -> Shadowsocks client -> ptproxy client -> ptproxy server -> Shadowsocks server -> destination host

In proxy chain, there are client/server node pairs. But each node (named as client or server) actually plays another pair of client/server functionalities at the same instant/simutaneously/concurrently. Each node forwards requests while listens for incoming request.

The former client/server refer to literal name of chain node, while the later pair are that node's functionalities. A server functionality address is set at your will while a client functionality address must be set to address a server functionality.
In this post, assume client resides on localhost while servers resides on VPS.
New Async version requirs >=python=3.4. The ancient shell script version is deprecated.

Python3.4 on CentOS 6.5

# cd /var/tmp
# wget https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tgz
# tar vxzf Python-3.4.4.tgz
# cd Python-3.4.4
# ./configure --prefix=/usr/local/
# make && make altinstall
# python3.4 -V
# ln -sv /usr/local/bin/python3.4 /usr/bin/python3.4
# rm Python-3.4.4.tgz

Configuration sample

{
    "role": "server",
    "state": ".",
    "local": "127.0.0.1:1080",
    "server": "0.0.0.0:23456",
    "ptexec": "obfs4proxy -logLevel ERROR -enableLogging",
    "ptname": "obfs4",
    "ptargs": "cert=AAAAAAAAAAAAAAAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA;iat-mode=0",
    "ptserveropt": "",
    "ptproxy": ""
}

It's localhost part (client) and VPS part (server).
A directory to store ptproxy running states consisting of several Json files.

We should modify obfs4_state.json to switch iat-mode.
local literally must be a loopback address (127.0.0.1:port), traffic within localhost.

For ptproxy client role, the loopback address is set (at your will) listening for local request. But for ptproxy server, it is an address to which traffic is forwarded.
Similarly, server parameter should be an address for remote connection.

For ptproxy client role, the remote address is set to ptproxy server's listening address on VPS which is, in return, equal to ptproxy server's server argument.
That's the pluggable transport executable.
ptname should be unique for each ptproxy client/server pair.
For ptproxy server, ignore this.
ptserveropt and ptproxy are usually left alone.

Server

Get ptproxy

# cd /opt/
# git clone --branch master --depth 1 https://github.com/gumblex/ptproxy.git
# git clone --branch master --depth 1 https://github.com/nibrag/aiosocks.git
# cd ptproxy/
# ln -sv /opt/aiosocks/aiosocks aiosocks

I don't like 3rd-party code snippets (Aiosocks here) mingled with system packages. Just put it together with ptproxy.

Server configuration

# cd ptproxy
# cp example.json server.json
# vim server.json

ptproxy server listens on VPS for incoming ptproxy client traffic while forwards that to Shadowsocks server.

{
    "role": "server",
    "state": "/opt/ptproxy/pt_state",
    "local": "127.0.0.1:5555",
    "server": "0.0.0.0:5554",
    "ptexec": "/usr/local/bin/obfs4proxy -logLevel ERROR -enableLogging",
    "ptname": "obfs4",
    "ptargs": "cert=AAAAAAAAAAAAAAAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA;iat-mode=0",
    "ptserveropt": "",
    "ptproxy": ""
}

The json configuration file does not allow comments. Remove them before launching.
Do NOT use bash reserving words in configuration, i.e. ${HOME} which is unkown to ptproxy.

Initial server startup
```
~ # cd /opt/ptproxy/
~ # mkdir pt_state; chown -R nobody: pt_state
~ # su nobody -s /bin/bash -c "python3.4 ptproxy.py -s server.json"
```
Must make sure nobody has write access to pt_state directory.

This is the very first launch. su nobody -s /bin/bash drop user to nobody instead of root. Useful information is printed on stdout for client side.
```
2015-10-02 18:16:49 Starting PT…
===== Server information =====
“server": “xxx.xxx.xxx.xxx:yyy",
“ptname": “obfs4″,
“ptargs": “cert=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx;iat-mode=0″,
==============================
2015-10-02 18:16:49 PT started successfully.
```
Meanwhile three state files are generated under /opt/ptproxy/pt_state. obfs4_state.json need modification for iat-mode, which means inter-arrival time controlling whether to confuscate packets' sending time by padding bits. By default, it is turned off.

In order to be more robust, change to 1 (Enabled, ScrambleSuit-style with bulk throughput optimizations) or 2 (Paranoid, Each IAT write will send a length sampled from the length distribution. This may slow down performance). Details refer to Various IAT related changes. iat-mode of Tor bridges is turned off by default either.
iat-mode

To modify obfs4_state.json, the running instance must be killed first.
```
# pkill -f "ptproxy.py"
```
Attention, on the server side, modify obfs4_state.json. Leave server.json alone.
Real launch by rc.local

Add to /etc/rc.local (symlink to /etc/rc.d/rc.local) on CentOS:
```
#!/bin/sh
# /etc/rc.local

PT_DIR="/opt/ptproxy"
/bin/su nobody -s /bin/bash -c "/usr/bin/python3.4 ${PT_DIR}/ptproxy.py -s ${PT_DIR}/server.json &"
```
To check the shell syntax, add set -x to the beginning of rc.local and sh /etc/rc.local. Errors will be printed.

If this is the only shell commands in rc.local, just execute sh /etc/rc.local. Otherwise, reboot VPS. It's not recommended to launch ptproxy server directly on SSH terminal, which may result in service killed occasionally.

ptproxy client

# cd /opt/
# git clone --branch master --depth 1 https://github.com/gumblex/ptproxy.git
# git clone --branch master --depth 1 https://github.com/nibrag/aiosocks.git
# cd ptproxy/
# ln -sv /opt/aiosocks/aiosocks aiosocks

Client configuration

Edit ss-client.json:

{
    "role": "client",
    "state": "/opt/ptproxy/pt_state",
    "local": "127.0.0.1:5553",
    "server": "ip-of-vps:5554",
    "ptexec": "/usr/local/bin/obfs4proxy -logLevel ERROR -enableLogging",
    "ptname": "obfs4",
    "ptargs": "cert=AAAAAAAAAAAAAAAAAAAAAAAAAAAAA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA;iat-mode=0",
    "ptserveropt": "",
    "ptproxy": ""
}

The ptname and ptargs should be set according to the initial server launch output. If choose to change iat-mode, remember to update local obfs4_state.json as well.

Shadowsocks client's side configuration

{
    "server":"127.0.0.1",
    "server_port":5553,
    "local_address": "127.0.0.1",
    "local_port":1080,
    "password":"gh1Tsy71nw",
    "timeout":300,
    "method":"aes-256-cfb",
    "fast_open":false
}

Change the server and server_port to ptproxy's counterparts.

Iptables

Turn on ptproxy's server listening port
```
# iptables -t nat -I OUTPUT 4 -d server-ip -p tcp -m multiport --dports 5554,5555 -j RETURN
# iptables -I OUTPUT 5 -d server-ip -p tcp -m multiport --dports 5554,5555 -j RETURN
```
5555 is Shadowsocks server port (just in case we turn off ptproxy) while 5554 is ptproxy server port.

Choose the -I rulenum based on your case.

Client startup

# cd ~/ptproxy/ptproxy
# su nobody -s /bin/bash -c "python3.4 ptproxy.py -c client.json"

Client startup script

#!/sbin/openrc-run
# To obfsucating TCP traffic, esepcially for SOCKS5 Shadowsocks

PT_DIR="/opt/ptproxy"

depend() {
  after net.wlp3s0
  after net.enp0s25
  after dhcpcd
  before shadowsocks
}

start() {
  if [ "${RC_CMD}" = "restart" ];
  then
    ebegin "Waiting"
    eend $?
  fi

  ebegin "Starting ptproxy"

  if ! pgrep -u nobody -x -f "/usr/bin/python3.4 ${PT_DIR}/ptproxy.py -c ${PT_DIR}/client.json" >/dev/null 2>&1; then
    su nobody -s /bin/bash -c "/usr/bin/python3.4 ${PT_DIR}/ptproxy.py -c ${PT_DIR}/client.json >/dev/null &"
  fi

  eend $?
}

stop() {
  ebegin "Stoping ptproxy"

  if pgrep  -u nobody -x -f "/usr/bin/python3.4 ${PT_DIR}/ptproxy.py -c ${PT_DIR}/client.json" >/dev/null 2>&1; then
    pkill -TERM -u nobody -f "/usr/bin/python3.4 ${PT_DIR}/ptproxy.py -c ${PT_DIR}/client.json"
  fi

  eend $?
}

It's recommended to merge this init script with Shadowsocks init script.

Notes

A ptproxy client instance is associated with a server instance by argument ptname.

That is to say, a client/server pair are for one application's TCP obfuscation (Shadowsocks in this post).
If want to obfuscate another application's TCP traffic, we should launch another pair of client/server instances with a new pair of client/server configuration files.

Specially, change the state location. We may use another PT binary like obfsproxy/obfs3. We also should choose new port client/server ports etc. Most important, choose new ptname.
If Tor is used with Bridges and PT. Likely an instance of obfsproxy/obfs4proxy exists on you local host. This time the Tor serves ptproxy's role.
Run as nobody:
1. su nobody -s /bin/bash -c "script here";
2. Make sure nobody have write access to pt_state directory.

Refs

Email

2016-04-01T00:00:00+00:00

The only security guarantee is setting your own Email server at home.
A/I and Riseup are relatively safer among the others.
A/I unlimited storage; up to 5 alias with many domain choices.
Riseup can change account name any time; 192MiB at most; unlimited alias with only Riseup domain.

Notice: if change account name, please add a alias for original account name. Otherwise you would not receive emails to that address.
[doubt] A/I seems to be safer than Riseup.
Main address: the email address assigned when registering account. For both sending and receving emails.
Identity: address for sending emails; the From header.

Some email providers does not support unrecognized or unregistered identity From address. Though you can use identity address to spoof From header, most of time, your main address is still exposed in the mail source.
Alias: address for receiving emails; the To header.

If the identity does not have a corresponding alias, the receivers could not reply email unless you specify the Reply-To header. You can set a (identity, alias) pair and make sure the address pair be a legal provider email address.
Notice: if your spoofed identity From address is owned by somebody else (main address or alias), the reply email will go to his mail box! DANGEROUS!

So when setting spoof From header, use one of your alias.
Main address just for account management (i.e. IMAP/SMTP authentication); does not expose main account on the Internet, especially do not register other online service with main address.
Set up alias to register other online service (receiving emails); can be exposed online for others reaching you.
If you really want to disguise identity when sending emails, use a customized identity address. The specified identity address might be constrained by your email provider (i.e. must be of their valid email domain).

This can be done easily in Gnus, just change From header. In SquirrelMail, just add identities in Options.

When forwarding Gmail to another account, by default Spam emails are left at the original account. The main idead is creating a filter to include Spam emails since Gmail filters are applied before Spam check.

The first method is dropping forwarding but create the filter like this:
```
Has the words deliveredto:your_incoming_address@gmail.com
Forward it to your_destination_address@example.com
Never send it to spam
```
Another method is applying the filter before forwarding:
```
Has the words: is:spam or label:spam or in:spam
Never send it to Spam
```
Email graph

Proxy

2016-03-08T00:00:00+00:00

Real proxy

ss (Shadowsocks), Tor, VPN, GAE, SSH tunnel etc.

Real proxy listens on a specific [IP[:port]] for incoming request.
Torify/Proxychains - direct traffic from shell

Torify (based on torsocks) forces tcp traffic through Tor for applications without built-in proxy support (including those not supporting SOCKS proxy).

Unfortunately, torify/torsocks is only for Tor. Proxychains is more powerful supporingt any SOCKS4a/5 or HTTP proxies. It works in dynamic, exact, or random modes. Also, it's fairly easy to define a chain of proxies - multiple proxies.

We can create a simple .desktop file with prefix torify/usewithtor/torsocks/proxychains.

For command line tool without GUI (i.e. wget), we just launch a sub-shell by torify bash and proxychains bash, under which all traffic goes through proxy - to proxify a shell.

A step further, add that command to .bash_profile (login shell) or .bashrc (interactive shell). Or torify/proxychains xfce4-terminal. Each new shell is proxified.

Attention, proxychains cannot not capture child process traffic. What was worse, detached GUI traffic cannot be caught either.
Assistant proxy - direct traffic from application

Privoxy, Foxyproxy, Autoproxy etc.

Assistant proxy only forwards application's traffic reqeust to real proxy. Defines which traffic uses proxy and which traffic does not. For example, Foxyproxy can add a PAC list forwarding traffic to real proxy.

Privoxy is a non-caching web proxy with advanced filtering capabilities for enhancing privacy, modifying web page data and HTTP headers, controlling access, and removing ads and other obnoxious Internet junk. Privoxy has a flexible configuration and can be customized to suit individual needs and tastes. It has application for both stand-alone systems and multi-user networks:
1. Do NOT provide proxy functionality itself but forward HTTP traffic to other proxies, i.e. Tor, Shadowsocks, GAE etc.
2. Applications should support HTTP proxy, i.e. browser, instant messager etc.
Iptables - direct traffic through kernel.
SS alone as a SOCKS proxy. Add SS to Tor as a front end proxy if no Bridges are available.

Both are SOCKS proxy. Only applications (i.e. Firefox) that support SOCKS proxy can make use of them.
Foxyproxy and Autoproxy are browser extensions while Privoxy is stand alone application. Therefore Privoxy can serve all applications (besides browser) that support HTTP proxy.

Details on how to use Privoxy, refer to the 1st reference. For example, by Privoxy, we can direct youtube.com traffic to GAE, while sex.com traffic to Tor. Another vantange of privoxy is you can filter traffic based on domain name.
Currently on my system, no extra applications (except browser) that support HTTP proxy needs proxy help. Firefox Foxyproxy has whitelist and blacklist applied to proxy. Turn on Use proxies based on their pre-defined patterns and priorities.

Of course, if Privoxy is used, just add a new HTTP proxy entry (Privoxy default 127.0.0.1:8118) to Foxyproxy.
Iptables

Each Iptables rule is strict (specific TCP/IP values) but has wide match (many applications). An exetreme example, is forceing all system traffic to Tor/SS.

One utility is Transparent proxy. At TCP/IP level, REDIRECT all system traffic to Tor, thus avoiding other supporting tools (Torify, Assistant proxy etc.). Additionally, we can place some traffic rule ahead of the Transparent proxy rule to not use Tor - achieving whitelist.

Details refer to newest Iptables rule.
One special case:

In Foxyproxy, select to use Shadowsocks for all URLs. In Iptables, enable Transparent proxy! The actual traffic path:
1. Foxyproxy modify HTTP packet to enclosed in Shadowsocks SOCKS packet;
2. Iptables Transparent proxy. To establish Tor circuit;
3. Shadowsocks front end proxy - Tor entry node - Tor middle nodes - Tor exit node;
4. Shadowsocks SOCKS packet - target URL.
So Shadowsocks is access twice in the traffic path. And target website actually sees Shadowsocks accessing its service. To avoid this, add a Iptables rule:

-A OUTPUT -d ss-ip -p tcp -m tcp --dport ss-port -j RETURN
If network connection fails, check:

Real proxy, Torify/Torsocks, Assistant proxy, Iptables etc.
Refs:
1. 如何用 Privoxy

Gentoo Networking

2016-03-01T00:00:00+00:00

Discuss networking configuration in Gentoo system, covering net.* config, dhcpcd, wpa_supplciant, mac spoofing etc.

ABCs

Wireless, PPP etc. special networking needs extra configuration before DHCP or static IP.

For example, before the wireless interface request IP from DHCP server, it must first authenticate username and password with wireless router. Otherwise, it was not even attched to a physical link. This functionality can be achieved by tools like wpa_supplicant and/or wireless-tools.
The Gentoo traditional net config is /etc/init.d/net.* and /etc/conf.d/net.
A functionality is not confined to a specific tool.

For instance, the wireless authentication functionality can be also achieved by wcid, networkmanager etc. as they all draw in wpa_supplicant package.
A tool is not constrained to a specific funciontality. Most of the time, we only need a few of them.

For example, dhcpcd can handle all networking functinalities.
If using net config or dhcpcd, we should install wpa_supplicant manually to satisfy wireless authentication.
net config thing is Gentoo's own networking configuration scripts.

Simple config

The simplest config on my system is dhcpcd + wpa_supplicant, though net config + wpa_supplicant is fine. For Ethernet, dhcpcd is enough! The extra wpa_supplicant fullfils wireless authentication.

Emerge both packages;
Add dhcpcd to default runlevel; Don't add wpa_supplicant to any runlevel.
Conigure wpa_\supplicant.conf for authentication.

Bingo~

dhcpcd

dhcpcd brings along its own hooks (/lib/dhcpcd/dhcpcd-hooks/ and /usr/share/dhcpcd/hooks/), like resolv.conf, wpa_supplicant etc. The hook tells dhcpcd to launch a relevant service. For example, wpa_supplicant hooks instruct dhcpcd to launch wpa_supplicant functionality before DHCP session.

Now, dhcpcd won't enable some hooks (i.e. wpa_\supplicant ) by default, we need to copy the hook from /usr/share/dhcpcd/hooks/ to /lib/dhcpcd/dhcpcd-hooks/. To disable a hook is trivial - either remove the hook under /lib/dhcpcd/dhcpcd-hooks/ or append nohook hook-name1, hook-name2 to /etc/dhcpcd.conf.

Here is an excerpt of dhcpcd config:

# stop resolv.conf hook to alter /etc/resolv.conf file since we will use TorDNS
# Another method is to remove /lib/dhcpcd/dhcpcd-hooks/20-reslve.conf hook
#
# Let net.wlan0 handle wpa_supplicant
nohook resolv.conf, wpa_supplicant

# Speed up DHCP by disabling ARP probing. This is useful in home network where IP collision
# rarely happens
noarp

# dhcpcd will release the lease prior to stopping the interface
# I use macchanger to randomize mac address. If without this option,
# the lease pool will burn out - NO IPs available!
release

The example above disables wpa_supplicant hook, which implies dhcpcd won't launch wpa_supplicant functionality.

MAC Spoofing

Now we do some trick on networking control. Anonymity on network is important. We can hide our IP/identity on the Internet by Tor, NAT etc. However that does not work within LAN due to MAC address!

Though MAC address won't go out of LAN, but it's plain within that scope. Enveryone in LAN can identify you by MAC address, thus motivating MAC Spoofing. Each time, we spoof a MAC address, the DHCP servers treat it as a new joint device and asign a new IP from the pool.

Notice: xx:xx:xx:xx:xx means permanent address while yy:yy:yy:yy:yy:yy is fake address.

Manually

Manually means change MAC address on command line. Pior to spoofing, we should make sure the interface id down. Basically, we resort to:

ifconfig

ifconfig wlan0 down
ifconfig wlan0 hw ether XX:XX:XX:XX:XX:XX
ifconfig wlan0 up

Some drivers might not support hw option.

iproute2

ip link set dev wlan0 down
ip link set dev wlan0 address XX:XX:XX:XX:XX:XX
ip link set dev wlan0 up

macchanger

The former two are almost the same except the syntax, while macchanger is smarter with fine control. macchanger is a package capable of complex MAC spoofing like reserve vendor type, any vendor, fully random etc.
```
emerge -avt macchanger
emerge -avt --oneshot >=netifrc-0.2.3 (opt)
```
macchanger has bumped to verion 1.7.0 which returns different string on exit. However, <netifrc-0.2.3 still examines the old exit string, error-prone. If you manage networking by OpenRC net config, then bump to >=netifrc-0.2.3.
```
ip link set dev wlan0 down
machanger -A wlan0
ip link set dev wlan0 up
```
Details refer to man page.

Automation

We prefer autmatic spoofing on system boot without user intervence, saving much trouble. Openrc init script meets the requirement. We wrap ifconfig, ip or macchanger command in init script.

We assume the networking scheme is dhcpcd + wpa_supplicant.

Notice: we should test our init script, rc-service script-name stop and rc default (or rc). Don't reboot - a waste of time!

standard init script

#!/sbin/runscript
#
# Spoof wireless interface mac address on boot
# Add other interfaces if needed
#
# `macchanger' supports complex syntax compared
# to `ip'

ETH0_PERMANET_MAC="xx:xx:xx:xx:xx:xx"
ETH0_FAKE_MAC="yy:yy:yy:yy:yy:yy"

WLAN0_PERMANENT_MAC="xx:xx:xx:xx:xx:xx"
WLAN0_FAKE_MAC="yy:yy:yy:yy:yy:yy"

depend() {
  after udev
  before dhcpcd
}

start() {
  if [ "${RC_CMD}" = "restart" ];
  then
    ebegin "Spoofing again"
    eend $?
  fi

  ebegin "Spoofing MAC Address"

  ip link set dev eth0 down
  #ip link set dev eth0 address $ETH0_FAKE_MAC
  macchanger -A eth0
  ip link set dev eth0 up

  ip link set dev wlan0 down
  #ip link set dev wlan0 address $WLAN0_FAKE_MAC
  macchanger -A wlan0
  ip link set dev wlan0 up

  eend $?
}

stop() {
  ebegin "Oops! Not a daemon and do nothing"
  eend $?
}

Since we specify before dhcpcd in dpend, the down/up wrapper can be removed unless you need manually rc-service macspoofing restart.

local init script

Similarly, MAC spoofing can be accomplished in a ordinary local script in /etc/local.d/. Actually MAC spoofing is NOT a daemon service but a few shell commands. local init script fits better.

/etc/local.d/macspoofing.start:
```
#!/bin/sh

ETH0_PERMANET_MAC="xx:xx:xx:xx:xx:xx"
ETH0_FAKE_MAC="yy:yy:yy:yy:yy:yy"

WLAN0_PERMANENT_MAC="xx:xx:xx:xx:xx:xx"
WLAN0_FAKE_MAC="yy:yy:yy:yy:yy:yy"

echo "Spoofing eth0 MAC Address"
ip link set dev eth0 down
#ip link set dev eth0 address $ETH0_FAKE_MAC
macchanger -A eth0
ip link set dev eth0 up

echo "Spoofing wlan0 MAC Address"
ip link set dev wlan0 down
#ip link set dev wlan0 address $WLAN0_FAKE_MAC
macchanger -A wlan0
ip link set dev wlan0 up
```
1. Please make sure local init is added to default runlevel: rc-update add local default.
2. Also mark the script as executable.
3. Must provide the down/up wrapper since we can not guarantee the interface is down when local script is executed.
4. We can also revoke the script by rc-service local start or rc.
udev init script

Udev allows to perform MAC address spoofing by creating the udev rule. Use address attribute to match the permanent MAC address and change it using the ifconfig/ip/macchanger command:
```
# /etc/udev/rules.d/75-mac-spoof.rules

ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="XX:XX:XX:XX:XX:XX", RUN+="/usr/bin/ip link set dev %k address YY:YY:YY:YY:YY:YY"
or
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="XX:XX:XX:XX:XX:XX", RUN+="/usr/bin/macchanger -A %k"
```
Make sure Udev is added to a init runlevel. On Gentoo, it's by default at sysinit runlevel. Udev is a low level but powerful device control module, we can do more. For example, create another rule file to rename device interface.

If you'd like, set Udev log to err (default), info or debug through:
```
# udevadm control --log-priority=info
```
Alternatively, edit /etc/udev/udev.conf manually.
net config - discusses later

Must bump to >=netifrc-0.2.3.

Release

When requiring IP by DHCP, we MUST tell DHCP client to release previous lease when it stops for IP re-use. Otherwise, eventually all available IPs are reserved by our single interface - running out of IP!

How to release? If use dhcpcd to serve DHCP, add release option to dhcpcd.conf. For traditional net config, add dhcpc_eth0="release" to /etc/conf.d/net.

net config

The above spoofing method assumes dhcpcd + wpa_supplicant networking scheme. Actually the old net config now supports MAC spoofing with macchanger. This is new networking scheme:

Now net config + DHCP + wpa_supplicant new networking scheme, where DHCP functionality is served by dhcpcd package.

Symbolic

pushd /etc/init.d/
ln -sv net.lo net.wlan0
rc-update add net.wlan0 default

/etc/conf.d/net

## global modules preference

# iproute2 over ifconfig
# both are for static IP setting
modules="iproute2"

# DHCP: over 'dhclient' / 'pump' / 'udhcpc' etc.
# For dynamic IP setting
modules="dhcpcd"

# Wireless: over 'wireless-tools' etc.
# For wireless authentication
modules="wpa_supplicant"


## ethernet - eth0

# modules preference
modules_eth0="dhcpcd"

# null - no IP
config_eth0="null dhcp"
dhcp_eth0="release"

# mac spoofing
# To randomize between the same physical type of connection (e.g. fibre,
# copper, wireless) , all vendors
mac_eth0="random-anykind"


## wireless - wlan0

# modules preference
modules_wlan0="wpa_supplicant"

# dhcp
config_wlan0="dhcp"
dhcp_wlan0="release"

# mac spoofing
# To randomize between the same physical type of connection (e.g. fibre,
# copper, wireless) , all vendors
mac_wlan0="random-anykind"

mac_wlan0="random-anykind" actually revokes macchanger -A wlan0`.

Update dhcpcd config
1. Keep dhcpcd at default runlevel though net config can launch dhcpcd process automatically!
  
  But this lanuching is not a daemon but a normal process. If revoked as a normal process, then rc-service net.wlp3s0 restart won't work due to failure stop of the dhcpcd process. Seriously, we rarely execute the service restart. Hence it's safely to remove from default runlevel.
  
  The key is we still ned dhcpcd to serve DHCP functionality.
2. Disable wpa_supplicant hook since net config will lanuch wpa_supplicant functinality automatically.
  
  net config will definitely launch wpa_supplicant but dhcpcd will launch wpa_supplicant as well. Thus wpa_supplicant is revoked twice, causing error.
3. Keep the release option in dhcpcd.conf though it's already set in /etc/conf.d/net. Just in case.
It seems that net config does NOT spoof MAC address if is no connection (i.e. no wired cable), which is a preferred way.

Booting warning:

* Starting DHCP Client Daemon ...
 [ ok ]
 * Bringing up interface wlan0
 *   Changing MAC address of wlan0 ...
 [ ok ]
 *     changed to 00:08:D3:11:9F:77
 *   Starting wpa_supplicant on wlan0 ...
Successfully initialized wpa_supplicant
 [ ok ]
 *   Starting wpa_cli on wlan0 ...
 [ ok ]
 *   Backgrounding ... ...
 * WARNING: net.wlan0 has started, but is inactive
 * WARNING: netmount will start when net.wlan0 has started
 * Starting Shadowsocks ...
INFO: loading config from /etc/shadowsocks.json
started
 [ ok ]
 * WARNING: tor will start when net.wlan0 has started

These warnings don't affect networking. The possbile cause is: net.wlan0 needs some time to warm up (started, but not finished yet) waiting for other services depending on (need net) it. We can check netmount and tor init scrpit and find need net in the depend function.

Service started but not finished might be at scheduling or inactive status. We can check this by stoping all networking init service and rc default; rc-status.

More

I can set an interface down by default. For example, to let Ethernet eth0 down default, in net config, we can add config_eth0="null". And in dhcpcd.conf, add denyinterfaces eth0.
Notes

net config is just a script manages networking at high level. To get connected, net config revokes other tools, like ifconfig, iproute2, wpa_supplicant, DHCP client etc. dhcpcd is standalone tool, with wpa_supplicant, it handles all networking connection (though dhcpcd can just serves DHCP). networkmanager is at a more higher level with GUI.

We'd better NOT mix net config and dhcpcd. Use either one of them.

MAC Spoofing Summary

The Udev init script is the easiest method.
Just a Udev rule. No other configuration.
No need of net config. Just keep original dhcpcd + wpa_supplicant settings.

Refs

wpa_supplicant

wpa_supplicant authenticates wireless router - data link layer where wireless traffic resides. Call it wireless authentication.
wpa_supplicant does NOT authenticates ISP's Internet access - namely network authentication.
Without ISP authentication, even connected to wireless router, still coult NOT connect to the Internet.
When subscribing to a ISP network service, you usually are given a account and password for network authentication. After that, you go home setting up a wireless router for home wireless authentication - a new pair of account (ssid) and password.
network authentication is usually done:
1. An automatic browser page poping up for ISP account name and password; Or use the ISP software.
2. For public WiFi, no network authentication is required. In many free WiFi situations the first time you use the service no matter where you try to go you're first intercepted and sent to a browser page where you're required to "login" or otherwise accept the terms of service. This page does not protect you at all. It has nothing to do with security, wireless or otherwise. It's nothing more than a bit of legalese to protect the internet provider.

psk VS passphrase

passphrase is what we usually called WiFI password, namely a short ( around 8) characters string. It's easy to remember.
psk is a wpa_supplicant term used to authenticate wireless router, namely a 256 characters string. It's hard to remember. To get this long psk:
```
# wpa_passphrase ssid passhrase
```
We use passphrase to generate psk. The result is shown on stdout.
We can find psk item in wpa_supplicant.conf. However, this psk is a little different from the above one. psk in wpa_supplicant.conf can be either:
1. Short passphrase with quotes;
2. Long psk without quotes.

wpa_cli

In spite of editing wpa_supplicant.conf manually, we can actually use wpa_cli to configure WiFi and optionally save to wpa_supplicant.conf.
Before running wpa_cli, wpa_supplicant must be started! Either by system networking init script or manually starting. If manually:
```
# wpa_supplicant -i wlp3s0 -D nl80211 -c /etc/wpa_supplicant/wpa_supplicant.conf -B -dd
```
1. -B puts wpa_supplicant to the background as a daemon.
2. wpa_supplicant.conf should at least include a line ctrl_interface=DIR=/var/run/wpa_supplicant. If we will save connected WiFi, then add update_config=1 which is not recommended for security.
At this point, run
```
# wpa_cli	# entering *wpa\_cli* interactive mode

> scan
OK
<3>CTRL-EVENT-SCAN-RESULTS

> scan_results
bssid / frequency / signal level / flags / ssid
00:00:00:00:00:00 2462 -49 [WPA2-PSK-CCMP][ESS] MYSSID
11:11:11:11:11:11 2437 -64 [WPA2-PSK-CCMP][ESS] ANOTHERSSID

> add_network
7
> set_network 7 ssid "ssid"
> set_network 7 psk "passphrase"
> enable_network 7
<3>CTRL-EVENT-CONNECTED - Connection to 00:00:00:00:00:00 completed (reauth) [id=7 id_str=]

>select_network 7       # optionally if the new added network is not associated with AP
   
>save_config		   # optionally if turned on 'update_config=1'
OK

>quit

# dhcpcd wlan0	# optionally if NO DHCP client client is running on the interface
```
1. You can see just scan scan_results add_network set_network enable_network is OK. The network has a number (0 above) specifying how many networks has been configured in wpa_supplicant.conf. This number increases sequencially.
2. If the SSID does not have password authentication (i.e. public WiFi), you must explicitly configure the network as keyless by replacing the command set_network 0 psk "passphrase" with set_network 0 key_mgmt NONE.
3. At the point of save_config, a wireless authentication is established. To surf the Internet, we need IP and network authentication.
  1. To get IP, usually just run a DHCP client. If there is not a DHCP client running on the interface, dhcpcd wlan0. Attention: this will only letting DHCP client running on the specified interface wlan0, neglecting others.
  2. Network authentication, discussed above.

Refs

Other functionality

Proxy
Tor

Refs

gentoo wiki

Init script

2016-03-01T00:00:00+00:00

Most daemon (service) brings along init script on installation, like LVM, Tor, Dhcpcd etc.
User defines their own daemon. In this post, I show procedures to generate a Gentoo init script of ss proxy. Make sure it's started on boot.
There are two methods to add personal init script.
1. Write a init script directly (/etc/init.d/).
  
  Init script is mainly for daemon in the background.
2. Write a simple script (/etc/local.d/) revoked by a special init script local (/etc/init.d/local). Of course, we make sure rc-update add local default.
  
  Local script focuses on simple foreground script that executes only once. For example, write a value to a file on boot.
wiki initscript and local.d.

#!/sbin/runscript
# To launch shadowsocks proxy at default runlevel
#
# https://wiki.gentoo.org/wiki/Handbook:AMD64/Working/Initscripts
#
# Tor uses Shadowsocks as frontend proxy. So launch
# it 'before tor'
#
# Put $DAEMON arguments after two succesive dashes '--'
#
# 'sslocal' create $PIDFILE by default. We just tell
# 'start-stop-daemon' the location by '--pidfile'.
#
# 'start-stop-daemon' arguments resides before '--exec'.

DAEMON="/usr/bin/sslocal"
PIDFILE="/var/run/shadowsocks.pid"

depend() {
  before tor
}

start() {
  DAEMON_OPTS="-c /etc/shadowsocks.json -q -d start --log-file /var/log/shadowsocks.log"

  if [ "${RC_CMD}" = "restart" ];
  then
    ebegin "Restarting"
    eend $?
  fi
  
  ebegin "Starting Shadowsocks"
  start-stop-daemon --start --pidfile $PIDFILE --exec $DAEMON -- $DAEMON_OPTS
  eend $?
}

stop() {
  ebegin "Stopping Shadowsocks"
  start-stop-daemon --stop --pidfile $PIDFILE --exec $DAEMON
  eend $?
}

Tor - Anonymity Online

2016-02-28T00:00:00+00:00

All ip[6]tables commands now should be converted into /var/lib/ip[6]tables/rules-save and use ip[6]tables-apply instead. Refer to post iptables.

ABCs

Tor - the onion router - mainly devotes to hide your Internet identity.
Tor prevents people from learning your location or browsing habits.
Tor is for web browsers, instant messaging clients, and more.
Tor aims at outbound traffic.

How

Tor periodically chooes different encryption routing relays. Exit relay's IP is your Tor IP.
Tor needs a centralized directory host to receve a list of relays on which Tor builds Tor circuit/route.
Tor traffic is NOT encrypted between exit relay and destination host. But that does not impact anonymoity online.
Tor is blocked:
1. Block the directory host;
2. Block the handshake process by itself fingerprints of Tor traffic between Tor client and the entry relay;
3. Block intermediate relys;
4. Malicious relays (honey pots).
To solve the issue:
1. Front end proxy;
2. Tor Bridge;
3. Tor Bridge and PT (Pluggable Transports) protocol.
Add a frontend proxy (Lantern for instance) to Tor. Frontend proxy bypasses censorship while Tor hide your Internet track. The frontend proxy must support SOCKS protocol by itself. GAE does NOT support SOCKS itself, so cannot be a frontend proxy.

In this post, I use ss (also a SOCKS5) as the frontend proxy.
On startup, Tor access the directory host to get a list of relays. But the replays there are public to everyone, prone to blocking. By specifing private relay - Bridges in configuration file, Tor build circuits based on them.
Do NOT enable Front end proxy and Bridges simutaneously, otherwise Tor would not work!

Mostly, I use Frontend proxy: mutliple proxies more secure. But for public Wi-Fi, we'd better use Bridges.
Sometimes Bridge is not enough to circumvent traffic censorship. Traffic between Tor client and the entry relay can be peeked and analyzed even the palyload is encrypted.

On such case, we need Bridges with pluggable transports support. pluggable transports obfuscate traffic between Tor client and the entry relay. Disguise the Tor traffic as normal.

It does NOT make any sense to run PT alone. It should be managed by applications that require TCP obfuscation.
Use Tor not only to surf the Internet, but for other applications that support SOCKS5.

Installation

# echo "net-misc/tor transparent-proxy
# emerge net-misc/tor
# rc-service tor start
# rc-update add tor default

Configuration - /etc/tor/torrc

Refer to /etc/tor/torrc.sample and man tor.

Basics

User tor
PIDFile /var/run/tor/tor.pid
Log notice syslog
DataDirectory /var/lib/tor/data
AvoidDiskWrites 1
DirReqStatistics 0

Frontend proxy

Tor is blocked by GFW. We add a frontend proxy to Tor bypassing GFW. This proxy server will be inserted between localhost and Tor entry relay.

This post uses ss - a SOCKS5 proxy.
```
Socks5Proxy 127.0.0.1:1080
```
Application -> Tor -> frontend proxy -> entry relay -> intermediate rely -> exit relay -> destination host.

Traffic between application and exit relay is encrypted. Localhost IP is plain between application and frontend proxy.

Though frontend proxy knows your IP but the traffic is encrypted! Exit relays and destination host know your traffic but does not know your real IP. What's more, if you visit destination host by https, then exit relay only knows encrypted https traffic.

Bingo!
Bridges with PT

Without PT protocol:

local app -> local Tor client -> Bridges Tor client/server -> destitionation site

With PT protocol:

local app -> local Tor client -> local PT protocol client -> Brdiges PT protocol server -> Bridges Tor server -> destitionation host

There are different pluggable transports, namely obfs1/2/3, scramblesuit, meek, obfs4 etc. Currently, obfs4 is best and next comes meek while the others can be blocked as well. get bridges tells how to obtain bridge address (usually 3 is enough).

To use pluggable transports, we should install the client first. The server is running at Bridges. Package net-proxy/obfsproxy (written with Python) supports managed, obfs2, dummy, obfs3, scramblesuit, b64. Unfortunately, it does not support obfs4 or meek.

We need obfs4proxy that implements the obfuscation protocols obfs2, obfs3, scrambleSuit (client only), meek (client only) and obfs4.

There is no ebuild available. I have written one in local overlay. But there are simpler way. Unlike Python script net-proxy/obfsproxy, obfs4proxy is just a binary executable. We can 1) copy the binary from official Tor browser (tor-browser_en-US/Browser/TorBrowser/Tor/PluggableTransports/); 2) build one manually.

Manually compilation is easy! obfs4proxy is written by Go language. From README, there are 6 build-time dedpencies. Only dev-lang/go should be installed while the other 5 will be cloned (by dev-vcs/git) automatically on the fly. We can compile in a normal user account.

The local compile process won't impact portage system as long as GOPATH is confined to somewhere like ~/workspace/go. Everything goes to GOPATH during the whole compiling process.

Generally, three directories will created under GOPATH, namely ${GOPATH}/src (downloaded source code), ${GOPATH}/bin (binary obfs4proxy), ${GOPATH}/pkg (build-time Go pkgs). Finally cp $GOPATH/bin/obfs4proxy /usr/local/bin.

The 6 dependencies are all built-time ones. dev-lang/go will be removed on next emerge –depclean. For the other 5, remove ${GOPATH}/src and ${GOPATH}/pkg.
```
# emerge -avt [--oneshot] dev-lang/go
$ export GOPATH=~/workspace/obfs4proxy
$ go help get/install/build
$ go get -v -x -work git.torproject.org/pluggable-transports/obfs4.git/obfs4proxy
# copy $GOPATH/bin/obfs4proxy /usr/local/bin
# chmod +x /usr/local/bin/obfs4proxy
# obfs4proxy --version
# emerge -avc dev-lang/go
$ rm -r $GOPATH
$ unset GOPATH
```
That's all. An interesting notice: the Github/Gitweb repository name is obfs4 but the finally binary is obfs4proxy. Also the compiling entry point is also obfs4.git/obfs4proxy which will depend on others like obfs4.git/transports etc. obfs4 repository implements different pluggable transports and output binary obfs4proxy.

Now update Tor config:
```
UseBridges 1
Bridge obfs3 ...
Bridge scramblesuit ...
Bridge obfs4 ...
ClientTransportPlugin obfs3,scramblesuit, obfs4 exec /usr/local/bin/obfs4proxy --enableLogging
UpdateBridgesFromAuthority 1
```
Test bridges speed on Atlas. Finally send HUP singal to Tor service killall -s HUP tor!

If Atlas fails to find the bridges, congratulations, this is a brand new bridge and unlikely censorsed.

Actually by some special configuration, obfs4proxy can be used to obfuscate any other TCP traffic. Refer to ptproxy and 你混淆了.

Ref:
Stream isolation

Isolate different applications by Tor port. By default, Tor service listens on 9050 port for all traffic. We can open more Tor ports for different applications.

For example, web browser and instant messanger should better connect to Tor ports.
```
# web browser
SocksPort 127.0.0.1:9050
# gpg client
SocksPort 127.0.0.1:9100
# instant messenger
SocksPort 127.0.0.1:9150
```
ExcludeNodes

Tor relies on routing relay. There are also malicious relays set by attacker/countries to analyze Tor traffic.

Tell Tor not to use those relays:
```
ExcludeNodes {cn},{hk},{mo}
strictnodes 1
```
Others are kp ir sy pk cu vn.
TorDNS

Though Tor + frontend proxy is secure enough to hide your surfing track, DNS server still knows where you connect. So we further hide the DNS resolving process.

ATTENTION: even you choose not to use Tor surfing the Internet, you can always use TorDNS resolver. That is why I add tor to default run level.
1. Turn on TorDNS
```
DNSPort 9053
DNSListenAddress 127.0.0.1
VirtualAddrNetworkIPv4 10.192.0.0/10
AutomapHostsOnResolve 1
AutomapHostsSuffixes .exit,.onion
```
  The 3rd line VirtualAddrNetworkIPv4 10.192.0.0/10 is very important! Otherwise, some applications cannot resolve (TorDNS) or connect to (Transparent proxy discussed below) .onion .exit address.
  
  Now TorDNS is listening at port 9053 as a DNS server. We can make use of tor-resolve to query DNS.
2. TorDNS command line tool.
```
$ tor-resolve www.baidu.com
```
  It is possible to configure your system, if so desired, to use TorDNS for all queries your system makes, regardless of whether or not you eventually use Tor to connect to your final destination.
3. [optinal] Change nameserver
```
# /etc/resolv.conf
nameserver 127.0.0.1
```
  1. If DNSListenAddress is set to 0.0.0.0, then any host IP (including 127.0.0.1) is OK.
  2. This step is optional due to the REDIRECT/DNAT rule in the next step.
4. Redirect DNS request to DNSPort
  
  By default, DNS resolving goes to port 53. DNS queries will go to 127.0.0.1:53 instead of DNSPort 9053. So need to redirect DNS queries from port 53 to 9053.
```
# iptables -t nat -A OUTPUT -p TCP --dport 53 -j REDIRECT --to-ports 9053
# iptables -t nat -A OUTPUT -p UDP --dport 53 -j DNAT --to-destination 127.0.0.1:9053
```
  A far better solution is setting DNSPort 53 and such no REDIRECT needed. Binding to low ports (53 less than 1024) need root privilege. If you send HUP signal to Tor daemone, it crashes.
  
  On system startup (OpenRC init script) or daemon restart (rc-service), Tor is ran as root and binding to low port 53 is fine. After start, Tor downgrades to tor account and drop root privileges. You can check this by ps -ef | grep -i tor. A tor user account is created after package installation. So Tor does not need to run as root.
  
  After dropping the privileges, reloading config will try to bind to low port 53 again. But this time, Tor crashes due permission. So if binding to low port, we can only restart Tor as root instead of HUP signal.
5. Disable Dhcpcd's resolve.conf hook, otherwise it will override /etc/resolv.conf file on startup.
```
# /etc/dhcpcd.conf
nohook resolv.conf
```
  Another obvious method is to remove /lib/dhcpcd/dhcpcd-hooks/20-resolv.conf file.
6. Now all DNS request on system will use TorDNS.
  
  Try ping www.google.com
torify - command line tool

torify will allow application traffic flows via the Tor network without any extra application configuration changes. To use torify, there must be a underlying SOCKS wrapper, i.e. tsocks (almost dead), torsocks etc.

Force traffic through Tor on demand in terminal, i.e. $ usewithtor emacs or $ torsocks emacs or $ torify emacs.

Application like Firefox configures their proxy settings to use Tor proxy. But it's annoying to configure each application to use Tor. What's worse, some application even does NOT allow proxy like Bitorrent, wget, apt-get, gpg, etc.

This is where torify plays a role: force applications to use proxy. Applications utilize Tor transparently.
```
# emerge -av net-proxy/torsocks

$ torify wget -qO- https://check.torproject.org/ | grep -i congratulations
$ wget -qO- https://check.torproject.org/ | grep -i congratulations
```
It calls torsocks wrapper with a tor specific configuration file (/etc/torsocks.conf). Compare the output between torify and without torify.

Refer to torify howto and torsocks.
torsocks - more

In previous step, torify make use of torsocks to force application traffic through Tor SOCKS5. torify calls torsocks with the default configuration file /etc/torsocks.conf in which the default port is 9050 - Tor's default SocksPort! There are other arguments like server address, socks type etc.

To use torsocks directly, just replace torify with usewithtor or torsocks.

$ usewithtor wget -qO- https://check.torproject.org/ | grep -i congratulations.

It seems that torsocks only supports Tor SOCKS. If I change /etc/torsocks.conf to use Shadowsocks, it does not work.

It's recommended to use proxychains instead.
Transparent proxy

configuration might be DEPCRECATED, please refer to lastest config file.

Though torify and torsocks force traffic through Tor netowrk without bothering configurating applications, they can only executed on command line. With transparent proxy setting, all non-Tor owned traffic will be redirected to Tor's TransPort with command line interfence or application configuration.

Tor alone cannot achieve transparent proxy without support from Linux kernel's netfilter functionality which is the core of Link firewall - Iptables.

SocksPort and DNSPort are Tor owned traffic. The former is Tor's TCP traffic while the later is a special case - UDP traffic. Now Iptables will redirect all non-Tor traffic to TransPort.

Once in place, applications do not need to be configured to use Tor, though Tor's SocksPort will still work. This also works for DNS via Tor's DNSPort, but realize that Tor only supports TCP, thus UDP packets other than DNS cannot be sent through Tor and therefore must be blocked entirely to prevent leaks. Using iptables to transparently torify a system affords comparatively strong leak protection.
1. Enable Tor's transparent proxy functionality.
```
TransPort 9040
TransListenAddress 127.0.0.1
```
2. Enable NETFILTER_ADVANCED and XT_MATCH_OWNER kernel option first.
```
advanced netfiler configuration netfilter_advanced
  <*>   "owner" match support xt_match_owner
```
  Without these two kernel support, iptables won't recognize -m owner match.
3. Redirect non-Tor traffic to transparent proxy
```
# iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner tor -j DNAT --to-destination 127.0.0.1:9040
# rc-service iptables save
```
  After the setting, all traffic is owned by Tor.
4. Pay attention to:
  1. -p tcp must accompany --dport since only TCP layer has a port number.
  2. -p tcp actually implied implicit arguments -m tcp which is followed by --dport.
5. Turn off Foxyproxy of Firefox, since no longer needed!
6. coarse-grained: all non-Tor traffic goes through Tor now.
  
  A fine-grained setting is turn on transparent proxy, use user-defined chain to control in detail which concrete outgoing traffic uses transparent proxy on demand. More on transparent proxy, refer to post iptables.

Disable non-Tor traffic - an alternative.

Another extreme way is to disable non-Tor traffic completely.

# iptables -P OUTPUT DROP
# iptables -A OUTPUT -i lo -j ACCEPT
# iptables -A OUTPUT -m owner --uid-owner tor -j ACCEPT
# iptables -P INPUT DROP
# iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# iptables -I INPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT

Choose one of the three shceme on security requirement. In daily usage, torify and/or torsocks suffices.
Restart Tor service

   # rc-service tor restart

Firefox setting

The main usage of Tor resides in WWW surfing. First make sure foxproxy extension is installed.

I have configured foxproxy for ss previously.
In foxproxy, Add New Proxy.
```
Proxy Name: tor-ss
  Perform remote DNS lookups on hostnames ...

Host or IP Address: 127.0.0.1 Port: 9050
SOCKS proxy SOCKS v5
```
1. Remember to place the Tor proxy at the beggining of proxy list - highest proxy priority!
2. Now Firefox has two proxies to choose from.
3. Tick Perform remote DNS lookups …
about:config
```
network.proxy.socks_remote_dns    true
network.dns.disablePrefetch       true
network.dns.disableIPv6           true
```
This way Firefox will resolve host names via Tor proxy DNS resolver (i.e. ss also support DNS resolving. Talked laster), which prevents DNS leaks.

The first item is the key to use TorDNS.

Usage

Prerequisite

As we add a frontend proxy to Tor, it should launch before Tor working.
All DNS lookups on system are through Tor. If application failed to resolve hostnames, check if Tor is running.

Applications.

Any applications that support SOCKS5 proxy can make use of Tor service.

In its proxy setting, fill socks5 server to 127.0.0.1 and port to one of the stream isolation port number.

For those do NOT support proxy or Tor, refer to transparent proxy above.

Firefox

Up to now, Firefox has two SOCKS5 proxy servers Tor and ss to choose from. You can choose either one or both simultaneously through foxproxy extension.

ss for all URLs

Use ss proxy surfing the Internet.
Tor alone

Use Tor surfing the Internet. However, Tor is blocked by GFW making this method infeasible.
Simutaneously

That's how Tor is commonly used: support from frontend proxy. Recall that we add SocksProxy in Tor's configuration to enable ss frontend proxy.

When choosing tor for all URLs in foxproxy, Firefox actually use Tor + ss concurrently.
Of course, we can choose use proxies based on their pre-defined patterns and priorities in foxproxy.

curl

The popular wget utility cannot talk to socks proxy. However, you can use the tor network to download any resource located at a given URL and save it in a FILE using curl:

$ curl --socks5-hostname 127.0.0.1:9050 -o FILE URL

DNS resolving

As discussed above, the system is configured to use TorDNS resolver preventing DNS lookup leaks. All DNS requests are redirecting to TorDNS by iptables.

Try to rc-service tor stop, then the system cannot resolve DNS anymore as TorDNS is down!

Tips

Don't use Tor to sign in Email/forum. The purpose of Tor is hiding traffic tack. Email, forum etc systems can analyze your traffic. No matter which method you use for connection, final interaction with sign in system is plain text.
关于深网

深网用户通过特殊的技术手段隐匿用户信息（如IP地址等）登陆“深网”以后，就能完全脱离世间法律或NSA（美国国家安全局）等政府管制力量的钳制和监管，在那里，你可以通过网站肆意的购买枪支弹药，可以发布悬赏杀人的告示，可以购买冰毒和可卡因，可以随意浏览儿童色情网站，也可以办理各种假证伪造身份。仅有4%的网络数据是可以通过搜索引擎找到的，其他的都深藏在“深网”中。

96%的网络数据都是无法用使用搜索引擎查找的，但这些网站的服务器实体可能分布在互联网管制薄弱的东欧国家，背后由深谙网络安全技术的黑客维护，多以.onion作为域名，采用特殊的加密机制，不能通过搜索引擎查找，即使知道域名也无法采用传统的HTTP方式直接访问，必须采用特殊手段才能接入，就好像只有游吟诗人维吉尔的引领，但丁才可以窥见Limbo里的曲径通幽处。

而今天我们这个引路人就是Tor（The Onion Router），直译过来为“洋葱路由”。既然已经通过 Tor 打开了“深网”的大门，那么让我们就将其一窥究竟，因为广大的深网网站都无法被搜索引擎收录，所以我们必须要找到一个收录各类深网网站的索引站点，“Hidden Wiki”就是其中较为出名的一个，它收录了各类深网网站地址。
Hidden wiki:
1. http://zqktlwi4fecvo6ri.onion/wiki/
2. http://torlinkbgs6aabns.onion/
3. http://wikitjerrta4qgz4.onion/
4. http://jh32yv5zgayyyts3.onion/

Refs

Snippets

2016-02-20T00:00:00+00:00

ABC
disk usage
arithmatic operation
if string comparision
apg
dig
find
Bulk rename
Number range
netstat
ln
Encoding/charset conversion
CR-LF/LF
1. Detect line ending:
2. Conversion
BOM
Remove whitespace
Empty line
newline
read line by line
darkhttpd
SSL cert
curl/wget
mtr/ping
Image crop and keep aspect ratio

ABC

cat -e input.file
sed -n l input.file

disk usage

# disk usage
du -hsc -- file1 dir1
du -hsc -- *
# disk free
df -h

arithmatic operation

echo 5.3025^2*4.871 | bc
bc <<< '100*0.13'
awk "BEGIN {print 100/3.5}"

if string comparision

if [[ "$str1" == "$str2" ]]; then
  echo "$str1 equals to $str2"
elif [[ $str1 == *"$str2"* ]]; then
  echo "$str1 contains $str2"
else
  echo "$str1 has nothing to do with $str2"
fi

Pay attention to quote.

apg

~ # apg -a1 -n1 -m16 -MSNCL -k

dig

~ # dig @dns-server -p 53 www.example.com

find

$ find -L /path/to/search -type f -iname "*filename*"
$ find -L /path/to/search -type f -iname '*filename*'
$ find -L /path/to/search -type f -iname '*.apk' -printf "%f\n" -exec cp -fv '{}' /path/to/copy \;
$ find APPs/ -maxdepth 1 -type f -iname '*.apk' -exec bash -c 'for file; do file="${file##*/}"; mkdir -p ROM/system/preset_apps/"${file%.apk}"; cp APPs/"${file}" ROM/system/preset_apps/"${file%.*}"; done' bash '{}' +
$ find -P history/ -type f -name '*.HEIC' -exec bash -c 'for file; do mv -v "$file" "${file%%.*}.heic"; done' 'bash' {} +
$ find ROM/system/media/wallpaper/ -depth -type f ! -path '*/wallpaper_15.png' -delete
$ find ROM/system/media/wallpaper/ -depth -type d -empty -delete
$ find -L . -type f -iname '*abc*' -exec bash -c 'mv "$0" "${0/abc/def}"' '{}' \;
$ export _pkg_name
$ find -H  -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) -exec bash -c 'for img; do mv "$img" pics/"${_pkg_name}"; done' bash '{}' +
$ find . -type f -name '*.jpg' -exec bash -c 'for f; do mv $f ${f//"$0"}; done' $'\302\240' '{}' +
$ find . -type f -name '*.pgn' -print0 | xargs -0 -n4 -P4 mawk '/Result/ { split($0, a, "-"); res = substr(a[1], length(a[1]), 1); if (res == 1) white++; if (res == 0) black++; if (res == 2) draw++ } END { print white+black+draw, white, black, draw }' | mawk '{games += $1; white += $2; black += $3; draw += $4; } END { print games, white, black, draw }'

Dont forget the quotes to prevent shell expansion.
In the fourth example, the random string argument (i.e. bash) preceding {} is a must. It serves as the very first argument for -exec bash command. You can either quote it or not.
Variables (i.e. _pkg_name) should be exported before calling 'find' as it spawns a new process.
-print0 prints the full file name on the standard output, followed by a null character (instead of the newline character that -print uses). This allows file names that contain newlines or other types of white space to be correctly interpreted by programs that process the find output. This option corresponds to the -0 option of xargs.

Sometimes we want to search for directories instead of files. As the very first result is the root directory itself, we may want to omit the 'bash' arbitrary string such that the root directory becomes the $0.

~ $ find -H . -type d -name '*figs*' -exec bash -c 'for d; do rm -rf $d; done' '{}' +
~ $ find -H . -type d -name '*figs*' -exec bash -c 'for d; do rm -rf $d; done' bash '{}' +

The second version will remove current directory as well. Removing the 'bash' string will leaves it untouched.

Bulk rename

a=1
for i in *.jpg; do
  new=$(printf "%04d.jpg" "$a") #04 pad to length of 4
  mv -- "$i" "$new"
  let a=a+1
done

Number range

# -w means equal width
$ seq -w 5 20

netstat

$ netstat -npeatu
$ ss -npeatu

Try l instead of a.

net-tools is deprecated in favor of iproute2. The alternative to netstat is ss.

ln

Soft/symbolic link (symlink) points to file's directory entry while hard link points to file's inode. No matter of symlink or hard link, both are real but special files.

When delete a symlink file, the orginal file is not touched. However, to delete a hard link file, the original file's hard link references count will be decreased by 1. If the final count equal to 0, then the original file is deleted as well.

Encoding/charset conversion

Charset is a set of character entities while encoding is its representation in the terms of bytes and bits. In Enca, the word encoding means the same as representation of text, i.e. the relation between sequence of character entities constituting the text and sequence of bytes (bits) constituting the file.

   $ enca/file filename, detect encoding
   $ enconv filename, convert to *locale*
   $ enconv/enca -x gb2312 filename, convert to specified encoding

enca/enconv is a tool superior to iconv.
It does NOT make backup.

CR-LF/LF

CR: Carriage Return, '\r', '\o015', '\xd'.
LF: Line Feeding, '\n', '\o12', '\xa'.
Some DOS files are ended with CR-LF, while others are LF-CR.

Detect line ending:

$ hexdump -c input.file, detect line ending
$ sed -n l input.file
$ file input.file
$ od -t c input.file

Conversion

app-text/dos2unix:

$ dos2unix/unix2dos -b input.file, '-b' makes a backup

perl:

$ perl -pi -e 's/\r//g' input.file
$ perl -pi -e 's/\n/\r\n/g' input.file

sed:

$ sed -i.bak 's/\r$//g' input.file
$ sed -i.bak 's/$/\r/g' input.file

tr:

$ tr -d '\r' < input.file > output.file, cannot do it the other way round

nano:

Type Ctrl-O and before confirming, type Alt-D (DOS) or Alt-M (Mac) to change the format.

Refer to remove CRs from CR-LF line terminators.

BOM

~ $ sed -i.bak '1s/^\xEF\xBB\xBF//' orig.txt

Remove whitespace

echo:

# `echo' without double quotes; squeeze sequencial spaces within the string to single one
str="   how   are	you   "
echo -e "$str"
echo -e $str
str=`echo $str`

sed:

$ str=" how    are    you   "
$ echo "$str" | sed 's/^[ \t]*//'
$ echo "$str" | sed 's/[ \t]*$//'
$ sed -e 's/[[:space:]]*$//' <<< "$str"
$ sed 's/^[ \t]*//;s/[ \t]*$//' <<< "$str"

awk/gawk:

# even sequencial tabs (maybe other blank characters) are squeezed to single space within string
$ awk '{$1=$1};1' file
$ awk '{$1=$1;print}' file

Empty line

~ $ sed '/^[[:space:]]*$/d' file.txt
~ $ sed '/^\s*$/d' file.txt
~ $ sed 's/^\s*$//g' file.txt   # failure as empty line replaced by a blank line

~ $ grep . file.txt

newline

Replace newline with another character:

~ $ tr '\n' ',' < file.txt

~ $ sed 's/\n/,/g' file.txt     # failure as sed read line but does not include the line terminator - newline
~ $ sed -z 's/\n/,/g' file.txt  # tell sed to use NULL as line terminator, so that newline is included when reading a line

read line by line

#!/bin/bash

input=~/workspace/bash/zh.txt
output=~/workspace/bash/zh.dict.yaml

> "$output"

while IFS=$'\t' read -r left right
do
    line=${left}$'\t'
    for word in $right; do
	if [[ $word == *"ā"* ]]; then
	    tmp=`sed 's/ā/a/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"á"* ]]; then
	    tmp=`sed 's/á/a/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǎ"* ]]; then
	    tmp=`sed 's/ǎ/a/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"à"* ]]; then
	    tmp=`sed 's/à/a/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ē"* ]]; then
	    tmp=`sed 's/ē/e/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"é"* ]]; then
	    tmp=`sed 's/é/e/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ě"* ]]; then
	    tmp=`sed 's/ě/e/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"è"* ]]; then
	    tmp=`sed 's/è/e/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ō"* ]]; then
	    tmp=`sed 's/ō/o/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"ó"* ]]; then
	    tmp=`sed 's/ó/o/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǒ"* ]]; then
	    tmp=`sed 's/ǒ/o/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ò"* ]]; then
	    tmp=`sed 's/ò/o/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ī"* ]]; then
	    tmp=`sed 's/ī/i/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"í"* ]]; then
	    tmp=`sed 's/í/i/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǐ"* ]]; then
	    tmp=`sed 's/ǐ/i/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ì"* ]]; then
	    tmp=`sed 's/ì/i/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ū"* ]]; then
	    tmp=`sed 's/ū/u/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"ú"* ]]; then
	    tmp=`sed 's/ú/u/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǔ"* ]]; then
	    tmp=`sed 's/ǔ/u/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ù"* ]]; then
	    tmp=`sed 's/ù/u/' <<< $word`
	    line=$line${tmp}4' '
	elif [[ $word == *"ǖ"* ]]; then
	    tmp=`sed 's/ǖ/ü/' <<< $word`
	    line=$line${tmp}1' '
	elif [[ $word == *"ǘ"* ]]; then
	    tmp=`sed 's/ǘ/ü/' <<< $word`
	    line=$line${tmp}2' '
	elif [[ $word == *"ǚ"* ]]; then
	    tmp=`sed 's/ǚ/ü/' <<< $word`
	    line=$line${tmp}3' '
	elif [[ $word == *"ǜ"* ]]; then
	    tmp=`sed 's/ǜ/ü/' <<< $word`
	    line=$line${tmp}4' '
	fi
    done
    sed -e 's/[[:space:]]*$//' <<< $line >/dev/null
    echo "$line" >> "$output"
done < "$input"

IFS= option before read command prevents leading/trailing whitespace from being trimmed.
The -r option passed to read command prevents backslash escapes from being interpreted.

Another example:

#!/bin/bash
file="/etc/passwd"
while IFS=: read -r f1 f2 f3 f4 f5 f6 f7
do
        # display fields using f1, f2,..,f7
        printf 'Username: %s, Shell: %s, Home Dir: %s\n' "$f1" "$f7" "$f6"
done <"$file"

Third version:

while read -p "input:" line
do
    echo "$line"
    printf "%s\n" "${line}"
done  < "${1:-/dev/stdin}"

Read from stdin or from a file parameter.

Here is an elegant version, just reading a line:

infd=0 ; [[ "${1}" ]] && : {infd}< "${1}"; read -p "input:" -r line <&"${infd}" # read -u $infd -r line

darkhttpd

$ ./darkhttpd ~/workspace/public_html --pidfile ~/workspace/httpd_traces/httpd.pid --daemon --maxconn 1 --no-listing --log ~/workspace/httpd_traces/access.log --forward example.com:8080 http://www.example.com:8080
or
$ ./darkhttpd ~/workspace/public_html --maxconn 1 --no-listing --log ~/workspace/httpd_traces/access.log --forward example.com:8080 http://www.example.com:8080 &

SSL cert

By openssl s_client:

~ $ openssl s_client -connect chat.freenode.net:6697 </dev/null
~ $ openssl s_client -connect chat.freenode.net:6697 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > freenode.pem

-showcerts show all certificates in the chain, including the CA certificate, otherwise only the server certificate was downloaded.
If the remote server is using SNI (that is, sharing multiple SSL hosts on a single IP address) you will also need to send the correct -servername in order to get the right certificate.

By gnutls-cli:

~ $ gnutls-cli --print-cert chat.freenode.net:6697 </dev/null
~ $ gnutls-cli --print-cert chat.freenode.net:6697 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > freenode.pem
~ $ gnutls-cli --print-cert --save-cert=freenode.pem chat.freenode.net:6697 </dev/null

--sni-hostname serves the same role as -servername of openssl s_client.
gnutls-cli always downloads the full certificate chain.

Examine the certificate downloaded:

~ $ cat freenode.pem; less freenode.pem
~ $ openssl x509 -noout -text -in freenode.pem
~ $ openssl x509 -noout -sha256 -fingerprint -in freenode.pem

curl/wget

Use curl/wget to test web/ftp service

~ $ wget -d http://www.example.com
~ $ wget -S http://www.example.com (Print the headers sent by HTTP servers and responses sent by FTP servers)
~ $ curl -v http://www.example.com
~ $ curl -I http://www.example.com (Fetch the headers only)
~ $ curl -i http://www.example.com (Include the HTTP response headers in the output)
~ $ curl --trace/--trace-ascii test.log http://www.example.com (Enables a full trace dump of all incoming and outgoing data, including descriptive information, to the given output file)
~ $ curl --socks5 127.0.0.1:1080 -v https://www.youtube.com (Test socks5 proxy)
~ $ curl --socks5-hostname 127.0.0.1:1080 -v https://www.youtube.com

Use curl to download a file

~ $ curl -OLv http://www.example.com/file.txt
~ $ curl -Lv -o new_file.txt http://www.example.com/file.txt

mtr/ping

~ $ mtr -rw 12.34.56.78

Image crop and keep aspect ratio

Filter the images:

#!/bin/bash

_input="kwdes.txt"

# Save as UTF-16 txt
# C-x RET r, revert-buffer-with-coding-system, UTF-8
sed -i.bak 's/\xC2\xA0//g' "${_input}" # non-breaking space
sed -i.bak 's/\r//g' "${_input}" # crlf to lf
sed -i.bak 's/[[:blank:]]*$//' "${_input}" # leading white spaces
sed -i.bak 's/^[[:blank:]]*//' "${_input}" # trailing white spaces
sed -i.bak '/^$/d' "${_input}" # empty lines
sed -i.bak -e '1d' "${_input}" # delete the 1st line
wc -l "${_input}"
#mkdir -p $(cut -d$'\t' -f1 "${_input}")
#awk -F'\t' 'FNR==linum { print $1, $4, $5 }' "${_input}"
#awk -F'\t' 'FNR==linum { printf("%s\t%s\t%s\n" $1, $4, $5) }' "${_input}"
#awk -F'\t' 'BEGIN { OFS=FS } FNR==linum { print $1, $2, $3 }' <<< $'a\tb\tc\td\n123\n'

Crop the images:

#!/bin/bash

# https://www.zedge.net/find/wallpapers/keyword, 8 * 1080x1920, com.uphone.wallpaperkeyword_description/keyword_date_num.jpg
# https://wallpaperscraft.com/catalog/keyword/1440x2560/page2
# https://wall.alphacoders.com/search.php?search=Minecraft#sorting_options
# icon_{1024*500,180*120}.jpg, ic_launcher_{512,192,148,96,72,48}.jpg
# magick -verbose input.jpg -trim -resize "1024x500^" -gravity center -extent "1024x500+0+0" +repage output.jpg # -extent -or -crop
# Parameters: line number > 1, selected filename

shopt -s extglob
#shopt -s nullglob

_workspace="${HOME}/WLshare"
cd "${_workspace}"

_input="kwdes.txt"

_linum="$1"
if [[ "${_linum}" < 1 ]]; then
    echo "line number starts from 1"
    exit
fi

_keyword=$(awk -F'\t' -v linum="${_linum}" 'FNR==linum {print $1}' "${_input}")
_keyword="${_keyword//[[:blank:]]/}"
echo "Keyword: ${_keyword}"

_pkg_name=$(awk -F'\t' -v linum="${_linum}" 'FNR==linum {print $4}' "${_input}")
echo "Package name: ${_pkg_name}"

rm -rf "${_keyword}"/*
mkdir -p "${_keyword}"/"${_pkg_name}"

_select="$2"; echo "Select: ${_select}"
_icon=(1024x500 180x120)
for i in "${_icon[@]}"; do
    magick "${_select}" -resize "${i}^" -gravity center -extent "${i}+0+0" +repage "${_keyword}"/"icon_${i%x*}.jpg"
done
_launcher=(512x512 192x192 148x148 96x96 72x72 48x48)
for i in "${_launcher[@]}"; do
    magick "${_select}" -resize "${i}^" -gravity center -extent "${i}+0+0" +repage "${_keyword}"/"ic_launcher_${i%x*}.jpg"
done

cp -f -- +(*.jpg|*.png) "${_keyword}"/"${_pkg_name}" 2>/dev/null
cd "${_keyword}"/"${_pkg_name}"
_size="1080x1920"
_counter=1
for f in *.+(jpg|png) ; do
    if [[ -f "$f" ]]; then
	magick "$f" -resize "${_size}^" -gravity center -extent "${_size}+0+0" +repage "$f"
	mv -f -- "$f" "${_keyword}_"$(date -I)"_${_counter}.jpg"
	_counter=$((_counter + 1))
    fi
done

# { cd ../../; rm -f -- *.+(jpg|png); }
echo 'done!'

GnuPG

2016-02-13T00:00:00+00:00

Anatomy
Examine Key
Create key
Edit key
Revocation
Export key
Import keys
Upload to keyservers
Key Management
1. Using trust to validate keys
2. Validity Level
Usage
1. sign
  1. verify
2. encrypt
3. sign and encrypt
expire
gpg.conf
gpg-agent
Refs

In the previous post SSH, I talked about secure connection (i.e. remote login) on the Internet. Now you will see how GnuPG (an implementation of OpenPGP) is used to encrypt/decrypt messages (i.e. email, documents). To use gpg, read official guide.

Anatomy

Public key and private key.
1. A document is encrpted by your public key and decrpted by your private key.
2. A document is signed by your privated key and verfied by your public key.
Primary key and subkey.
1. Primary key pair is your online identity, usually used for signature.
2. A subkey pair is bond to the primary key, usually used for signature or encryption. We can bind many subkeys to the primary key for different purposes.

A rule of thumb:

never ever sign and ecrypt using the same key

Examine Key

user@tux ~ $ gpg --list-keys --list-options show-usage --with-fingerprint --with-colons
user@tux ~ $ gpg --list-secret-keys
user@tux ~ $ gpg -a --export <keyid | uid> | gpg --list-packets

gpg: directory `/home/jim/.gnupg' created
gpg: new configuration file `/home/jim/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/jim/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/jim/.gnupg/pubring.gpg' created
gpg: /home/jim/.gnupg/trustdb.gpg: trustdb created

This will try to list available GPG keys on system. If it is the first time to run gpg, some directories and files essential to the correct operation and implementation will be created.

Create key

The very first step of using GnuPG is creating a key pair.

Here is what I will do:

Choose default key method: (1) RSA and RSA (default).
Keysize - the longest - 4096 bits.
Key valid for one year
Provide name and email address to identify the key.
Provide a passphrase to protect the private key. The public key don't need protection.
Get the key ID: the last 64 bits fingerprint.

# require parameters
jim@tux ~ $ gpg --full-generate-key

# use default parameters
jim@tux ~ $ gpg --gen-key (use default key parameter)

The option --full-gen-key is interactive asking for key parameters like exipration date etc.
GnuPG will take several minutes to generate the key pair. You'd better moving the mouse, browsing the web, or having streaming audio in the background to generate noise, which will help speed up the process.

By default, a primary key and a subkey are created separately for signature and encryption respectively.

Edit key

At any moment, we can update passphrase, user id, subkey, etc.

~ $ gpg --edit-key <keyid | uid>

gpg> help

Existing user ID (comment included) can not be modified after creation. But you can add or delete user ID instead. We can set a user ID to be the "primary user ID" (top in the ID list).

Revocation

Newer GnuPG version will issue revocation certificate upon key creation. However, it's always a good idea to create a new revocation certificate as long as you still access to the private key, providing a strong revocation description.

Create a revocation certificate. Doing this allow you to revoke the key in case something nasty happens (lose the key; superseded; compromised etc.).

jim@tux ~ $ gpg --output 12345678-revocation.asc --gen-revoke 12345678

If you don't give --output 12345678-revocation.asc, the revocation certificate will be printed on screen.

Please move it to a medium which you can hide away; if Mallory gets access to this certificate he can use it to make your key unusable. It is smart to print this certificate and store it away, just in case your media become unreadable. But have some caution: The print system of your machine might store the data and make it available to others!

Later on, suppose you want to revoke the key,

jim@tux ~ $ gpg --import 12345678-revocation.asc
jim@tux ~ $ gpg --keyserver pgp.mit.edu --send-keys 12345678

Issuing this command imports the revocation into your keyring, revoking your key locally.
This sends the revoked key to key server to let your friends know.

If it succeeds, you'll get the message:

gpg: success sending to `pgp.mit.edu' (status=200)

If you check your key's verbose index page on pgp.mit.edu, you'll see KEY REVOKED on the first line of the details.

New version of GnuPG creates revocation certificate automaticall unpon key creation in ~/.gnupg/openpgp-revocs.d.

Export key

Now it's time to distribute the public key.

# export public key
jim@tux ~ $ gpg --armor --export --output public.pgp email@gmx.com
# export private key
jim@tux ~ $ gpg --armor --export-secret-keys --export-options backup --output key-backup.pgp email.gmx.com

By default, the public key will be exported to STDOUT. --output name.pub exports to file. It's recommended to distribute this file with the fingerprint attached.
--armor tells gpg to generate ASCII text output instead of default binary.
We export private key (public key included) for backup or moving keys around.

A reasonable way to achieve a long term backup of OpenPGP (GnuPG, PGP, etc) keys is to print them out on paper. Paper and ink have amazingly long retention qualities - far longer than the magnetic or optical means that are generally used to back up computer data.

Import keys

Once you get a copy of the exported public key of your friend, add it to your public keyring.

Pay attention to verify fingerprint to avoid receiving a forged public key.

jim@tux ~ $ gpg --import friend.pub
jim@tux ~ $ gpg --edit-key friend-key, (enter sub-command)
   list - list the key information.
   fpr - print fingerprint of imported key (same as `gpg --fingerprint`).
   sign - sign your friend's public key by your private key.
   check - check the signature.

After importing the public key, now you can encrypt message to friend or to certify key's owner's signature.

We can also import a secret key:

jim@tux ~ $ gpg --import key-backup.pgp

Upload to keyservers

--export and --import distribute keys between friends directly. There are many public key servers where you can --recv-keys, --send-keys, and --search-keys. Since version 2.1 of GnuPG, dirmngr takes care of accessing the OpenPGP keyservers.

All these public key servers will exchange keys periodically. Most of the time, you only need to focus on one of them. You can specify keyserver in ~/.gnupg/dirmngr.conf or on command line by --keyserver.

jim@tux ~ $ gpg --search-keys "jim gray"

jim@tux ~ $ gpg --recv-key keyid
jim@tux ~ $ gpg --send-key keyid

Key Management

Key
1. Your own key pair - public/private key.
2. Friends'/organizations' public keys. Imported from key file or received from key servers.
Utility
1. Public key is distributed over the Internet, i.e. sent to key servers or handed over to friends manually.
  1. People encrypt messages before sending to you.
  2. People certify your signature.
2. Keep private key confidentially.
  1. Use it to decrypt message received.
  2. Sign documents before publication.
    
    A special case is to sign public keys from friends/key servers to validate them.
Validity
1. Before using a key, make sure the key is owned by whom you would communicate with. Our own key pair is valid by default. But how about friends' public keys? How to validate them?
2. Upon receiving a public key, fingerprint is included which should be compared with what the owner claims (i.e. handed over to you). If it's a organization's public certificate, usually fingerprint is pasted on their official page along with public key certificate download link.
3. After confirmation the public key fingerprint, you sign the public key with your private key. To manually sign a key directly is to validate/certify a key.
4. If a public key is not sign directly before usage, a gpg warning comes out like this:
  
  gpg: WARNING: This key is not certified with a trusted signature!
  
  gpg detects that the key you are using has not been validated. You can check whether a key is valid by the output of gpg --list-keys. If there is an [unknown] in front of the user's ID, the key is NOT valid in that you cannot determine if it's owned by the one claiming to own it.
Web of trust

validity and trust are different thing. If you trust somebody, you acutally accept the keys he signed as valid. But what if you have got 100 public keys at hand? Sign manually one by one? OMG! GnuPG addresses this problem with a mechanism popularly known as the web of trust. In the web of trust model, responsibility for validating public keys can be delegated to people you trust.

For example, you get public keys: k1, k2, k3, … You manually / directly signed k1 with your private key and validate it. Meanwhile, k1 signed k2 and k3 directly as well. From the viewpoint of k1, k2 and k3 are valid keys. If you trust k1 fully (level 4, discussed below), then you trust k1's signing action on k2 and k3. You view k2 and k3 as valid as well though you do not sign them directly.

What the hell does trust mean? trust means the confidence on someone's capacities at signing/validating/certifying keys.
1. Trust is subjective judgement on others' capacity. You trust on somebody according to whether he verifies public fingerprint carefully, whether he is a master of GnuPG and PKI. The purpose of trust is to relieve you from signing/validating keys one by one manually.
2. Validity is the status of a key from viewpoint of you (your gpg on system).
3. So trust serves validity. The trust relationship forms a web. gpg determines a key's validity by this web of trust.
4. Trust has levels. You can trust Bob fully while Alice partially.

Using trust to validate keys

How trust works? Recall gpg sub-command sign validates public keys. There is trust sub-command to assign trust level to a key's owner.

Trust has levels: unknown, none, marginal, ful, ultimate. Level 0 (unkown) means you don't know the key's owner's capacity at validating keys, while 5 means trust ultimately. Never trust other's key ultimately! ultimate is designated for your own key only. Details on trust levels, refer to reference 1.
Formerly, a key was considered valid only if you signed it personally. A more flexible algorithm can now be used: a key K is considered valid if it meets two conditions:
1. it is signed by enough valid keys, meaning:
  - you have signed it personally,
  - it has been signed by one fully trusted key, or
  - it has been signed by three marginally trusted keys; and
2. the path of signed keys leading from K back to your own key is five steps or shorter.
3. prerequisite: the parent key should be fully valid.
Apart from the web of trust, there is also a tree of signing on which only the first layer nodes of tree is signed by you directly! We cannot determine the validity of a node (key) on lower layers without the help of web of trust.

Validity Level

The prequisite in the algorithm: the parent key (signing key) shuld be fully valid. Actually, validity also has levels - similar to trust.

You can sign a key directly and check the result. In front of the user ID, you will find [full], means this key is fully valid. For example, you fully trust a marginal valid key k1 on the signing tree and k1 signs another key k2. k2 should NOT be regarded fully valid.

Usage

If there exist multiple keys created, use CLI option --default-key or configuration default-key to select the key.

sign

jim@tux ~ $ gpg --output message.sig --sign message.txt
or
jim@tux ~ $ gpg --clearsign message.txt
or
jim@tux ~ $ gpg --detach-sig message.txt

It uses your private key to sign.
The first output is binary format with original file compressed.

This is NOT a preferred way to sign documents.
The 2nd output will append signature to the end of original file without compression. This what we usually used to sign email.

But receiver has to recover orginal document from the signature file.
The 3rd will create a separate binary signature file without touching the original document. You can send the original file along with the signature file.

This is useful when the original document is large files.

verify

Unpon get a signed message, we can just check the signature by –verify or verify the signature and extract the original message concurrently by –decrypt. To verify a detach signature, both the oirginal document and signature file must be present.

jim@tux ~ $ gpg --verify message.sig
jim@tux ~ $ gpg --verify message.sig message
jim@tux ~ $ gpg --output message --decrypt messag.sig

encrypt

$ gpg --encrypt --recipient recipient@example.com message.txt

Should specify who is the receiver by --recipient.

sign and encrypt

$ gpg --sign --encrypt --recipient recipient@example.com message.txt

Output is binary format.
Cannot use --clearsign with --encrypt simutaneously.
There is no need for --detach-sig in this case.

When receving a secret.gpg file, you

$ gpg --output original.txt --decrypt secret.gpg

expire

Though keys can be set to never expire, you are recommended to set an expiration date since you can always extend the expiration date even after it has expired. This "expiration" is actually more of a safety valve or "dead-man switch" that will automatically trigger at some point. If you have access to the secret key material, you can untrigger it. The point is to setup something to disable your key in case you lose access to it (and have no revocation certificate).

~ $ gpg --edit-key uid

gpg> expire
...
gpg> key 1
gpg> expire
...
gpg> save

~ $ gpg -k

~ $ gpg -K

The first expire without selected key sets new exipiration date for primary key.
The second expire with selected sub-key sets new expiration date for key 1.

Ohters can notice the expiration of your public key from key servers. For private keys, only you know it unless you give them to some bad guy.

gpg.conf

# ~/.gnupg/gpg.conf

use-agent
pinentry-mode default
keyid-format 0xlong

Any long CLI options can be a configuration entry by removing the -- preifx.

gpg-agent

Like the SSH Agent, GPG agent cache the private key passphrase. A user don't need to input passphrase each time using the key.

Sometimes working with certain applications requires the use of a GPG key very frequently, which means that a passphrase must be frequently entered. In the past many applications supported a passphrase caching mechanism. This would make life easier for users because passphrases were automatically entered. However, this disallowed sharing this cache across programs (how secure would that be?) and forced applications to reinvent the wheel over and over again.

In order for apg-agent to works in Shell, we must correctly set GPG_TTY as below.

# ~/.bashrc

export GPG_TTY=$(tty)

pinentry

GnuPG includes gpg-agent. Pinentry (app-crypt/pinentry) is a helper application that gpg-agent uses to request the passphrase in a graphical window. It comes in three flavors: it can popup a window using the GTK+, QT, or curses libraries (depending on the USE flags set in /etc/portage/make.conf).

If app-crypt/pinentry was installed with more than one popup window type, it is possible to choose between the windows with the eselect pinentry command:

laptop ~ # eselect pinentry list
Available pinentry binary implementations:
  [1]   pinentry-gtk-2 *
  [2]   pinentry-curses
laptop ~ # eselect pinentry set 1

configuring gpg-agent

# ~/.gnupg/gpg-agent.conf
# called when prompt for passphrase the first time
pinentry-program /usr/bin/pinentry-gtk-2
# timeout 30 minutes
default-cache-ttl 1800

Now configure GnuPG to use an agent when appropriate. Edit ~/.gnupg/gpg.conf and add the following line:

# DEPRECATED!
use-agent

Since GPG2, use-agent is deprecated as GPG2 always require agent.

launch agent

To launch agent manually:

$ eval $(gpg-agent --sh --daemon --enable-ssh-support --write-env-file ${HOME}/.cache/gpg-agent-info)

If the arguments are already provided in configuration file, they can be omitted. Attention, command line arguments will override those in configuration file.

The command should be wrapped by eval, otherwise environment variables would not be exported.

Automation

Refer to lastest ~/.bashrc for updates

To launch GPG agent automatically on boot, add the command to startup scripts specially for servers (NO X/GUI). By default, Xfce4 starts the agent automatically on startup. Check by ps -ef | grep -i gpg-agent.

But if you launch an agent manually on consosle and then switch to start Xfce4, it will launch another agent for you. So check to launch agent in startup scripts like .bashrc and .bash_profile.

(opt) Disable Xfce4 automation

$ xfconf-query -c xfce4-session -p /startup/gpg-agent/enabled -n -t bool -s false
$ xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false

Edit .bashrc:

# Associate with existing gpg-agent
if [ -f "${HOME}/.cache/gpg-agent-info" ]; then
    . "${HOME}/.cache/gpg-agent-info"
    export GPG_AGENT_INFO
    export SSH_AUTH_SOCK
    export SSH_AGENT_PID
fi

# Start the gpg-agent if not already running
if ! pgrep -x -u "${USER}" gpg-agent >/dev/null 2>&1; then
    eval `/usr/bin/gpg-agent --sh --daemon --enable-ssh-support --write-env-file "${HOME}/.cache/gpg-agent-info"`
fi

# Refresh gpg-agent tty in case user switches into an X session
gpg-connect-agent updatestartuptty /bye >/dev/null

With this script, the previous ssh-find-agent is deprecated.

SSH mode

The OpenSSH Agent protocol is always enabled (check ${XDG_RUNTIME_DIR}/gnupg/S.gpg-agent.ssh), but gpg-agent will only set the SSH_AUTH_SOCK variable if flag --enable-ssh-support is given.

This mode does not require SSH_AGENT_PID.

GPG_AGENT_INFO

deprecated since GnuPG v2.1

Like SSH agent, GPG agent exports an environmental variable GPG_AGENT_INFO and file ~/.cache/gpg-agent-info.

Refs

IME

2016-01-01T00:00:00+00:00

Emacs的chinese-pyim输入法已经越来越复杂，脱离了一个输入法插件的要求，不建议安装。Fcitx安装时去掉了默认的输入法（通过-table USE）。额外安装fcitx-rime包。最好的用户词库工具是深蓝词库转换，支持几乎所有的主流输入法词库。

＃ Fcitx基本概念

拼音文件 —— Pinyin File is a text file with pinyin and one character per line, separated with space (built-in gbkpy.org).
词库文件 —— Phrase File is a text file with full pinyin separated with ' and the corresponding phrase (built-in pyPhrase.org).

区分拼音文件和词库文件对于理解输入法非常重要。
Fcitx的安装不难。不过要用好的话，就应该设置好拼音文件文件和词库文件。

拼音文件基本不变。目前比较好的词库是来自搜狗拼音输入法，可以在官网免费下载 *.scel 结尾的词库文件
Fcitx和chinese-pyim无法直接使用搜狗拼音输入法的*.scel词库文件，需要经过工具转换为中间格式*.org格式。经过排序，去重后：

再将 *.org 格式转换成Fcitx的二进制*.mb格式；或转换成chinese-pyim的普通文本*.pyim格式。*.org格式既可以转成chinese-pyim的*.pyim格式，也可以转换成Fcitx的*.mb格式。可以维护一份共同的*.org中间格式词库文件。
在Linux上，所有的文件必需是UTF-8编码。用enca/file test.org列出文件编码，用enca -x UTF-8 test.org把文件转换成UTF-8编码。

如果编码非 UTF-8, 可能转换出错，即使转换成功，或者转换过程丢失很多词组。
虽然chinese-pyim (参考Emacs configuration文）和Fcitx都可以用*.org词库来转换，但有所不同：
1. Fcitx词库拼音分隔符是单引号'，但是chinese-pyim则是除单引号'之外的任何分隔符。
2. chinese-pyim需要合并相同拼音前缀的行，还需要去除同一行相同的候选词。

Fcitx词库设置

Fcitx自带的拼音文件gbkpy.org可以直接使用，但自带的词库文件pyPhrase.org很差劲，根本没法用。

参考合并词库及方法步骤（2012-06-17 更新）和120余万的搜狗细胞词库.
去搜狗词库官网下载各种词库文件。譬如放到~/Downloads/scel/*.scel.

把*.scel文件转换成*.org文件：

~ $ cd Downloads
~ $ mkdir org
~ $ cd scel
~ $ find -L . -type f -iname '*.scel' -exec scel2org -o ../org/{}.org {} \;
# or
~ $ for i in *.scel; do scel2org $i -o ../org/$i.org; done
~ $ cd ..
~ $ ls org

这个命来还会打印出所有被转换的词库名称。~/Downloads/org/下存放着转换后的*.scel.org文件，此文件是普通文本文件，可以用less明令查看。

合并所有的*.scel.org文件为一个单一的org文件。
```
~ $ mkdir dict
~ $ cd dict
~ $ cat ../org/*.scel.org > 1.org
```
得到1.org文件，合并词库文件，有利於提升输入法的性能。很明显从单个文件提词比从多个文件提词快。

到目前威治我们得了一个*.org格式的大词库文件。

从Fcitx源码包解压出~/Downloads/dict/pyPhrase.org - Fcitx默认自带的词库文件，很糟糕，我们把它合并到刚刚生成的1.org中。

~ $ tar xf fcitx-4.2.4.1_dict.tar.xz
~ $ tar xf fcitx-4.2.4.1_dict.tar.xz/data/pinyin.tar.gz
~ $ tar xf data/pinyin.tar.gz
~ $ cat pyPhrase.org >> 1.org # 合并
~ $ sort 1.org > 2.org        # 排序
~ $ uniq 2.org > 3.org        # 去掉重复

转换成Fcitx使用的*.mb格式。

从Fcitx源码包解压出~/Downloads/dict/gbkpy.org - Fcitx默认自带的拼音文件，生成*.mb词库需要拼音文件的辅助。

~ $ cp fcitx-4.2.4.1/data/gbkpy.org .
~ $ man createPYMB
~ $ createPYMB gbkpy.org 3.org

如果 3.org 文件非常大的话，可能需要10分钟左右，耐心等待。最后生成 4 个新文件:

dict/
├── 3.org
├── gbkpy.org
├── pyERROR   词库中重复或有问题的词条，有兴趣可参考，可以直接忽略
├── pyPhrase.ok 可以顺利转换的词条；和pyERROR一起就是3.org
├── pyphrase.mb 最终 Fcitx 使用的二进制词库，用它覆盖Fcitx原*pyphrase.mb*。
└── pybase.mb   二进制词库的配套字码库，用于覆盖原*pybase.mb*。

只需要用新转换得到的pybase.mb和pyphrase.mb文件复盖原文件即可。

复盖Fcitx原文件。

原文件一般位於系统路经 /usr/share/fcitx/{data,pinyin} 和个人用户目录~/.config/fcitx/{data,pinyin}。如果所在目录有原文件，就覆盖，没有的话就复制过去。

在我的Gentoo下，位於{/usr/share,~/.config}/fcitx/pinyin/目录下。最后选择放到个人用户目录下。
上面描述里从*.scel到*.org再到*. mb*格式的转换过程。

如果能从网上下载到比较好的*.org或*.mb会更省时（注意文件是否是UTF-8编码）。第四个参考链接里的sougou-phrases-full.7z或fcitx-sougou-phrase-full.7z不错。

chinese-pyim词库设置

上面得到的3.org经过转换也可以给chinese-pyim使用。
用sed命令把'替换成-：
```
~ $ sed -ibak  "s/'/-/g" 3.org
```
用awk合并相同拼音前缀的行，也就是汉字的同音字/词。譬如gai-shan 改善和gai-shan 改删两行就需要合并成gai-shan 改善改删，这是chinese-pyim的要求，Fcitx并没有此要求。
```
~ $ awk '{key=$1; $1=""; a[key]=a[key] $0}; END{ for (key in a) { print key a[key];}}' 3.org > 4.org
```
重新排序。经过awk合并后，顺序打乱了：
```
~ $ sort 4.org > 5.pyim
```
在同一行去重复。譬如a-ba 阿爸阿坝阿爸就需要去除多余的阿爸，这是由于第三步合并时，可能有两行里包含相同的词。
```
 ~ $ awk '{printf("%s",$1); for (i=2;i<=NF;++i) if (!a[$1" "$i]++) printf(" %s", $i); printf("\n");}' 5.pyim > pyim-dict.pyim
```

Rime词库

设置过程中参考Rime 定製指南.

基本介绍

Rime比较好的地方是无广告、不联网、隐私高、定制性高。初期使用比较麻烦，好多概念不好理解。
添加用户自己的词库。
添加emoji输入。
同步用户定制。
Fcitx Rime配置文件~/.config/fcitx/rime。

Windows

最好在创建好多用户后，在普通帐户下安卓小狼毫。
让多用户共享同一个用户文件夹。

虽然小狼毫自带的“小狼毫安装选项”可以更改用户文件夹，但是只对管理员账户有效。

打开注册表，HKEY_CURRENT_USER\Software\Rime\Weasel里面有一个string属性的RimeUserDir参数（普通账户没有此参数要新新建一个），把它设置成新的用户文件夹地址（可能要重启），这样不同的Windows账户可以共用同一套配置。

一个更好的办法是从管理员账户注册表拷贝过来。
Windows和Linux没法共用一套配置，因为词库二进制格式不兼容。
Rime在Windows和Linux上不区分换行符，\n和\r\n都支持。

简体配置

Rime自带的5个默认输入方案，但是对与我来说，只需要使用其中的“朙月拼音·简化字”。
每一个输入方案对应的配置文件以.schema.yaml结尾。譬如“朙月拼音·简化字”对于的是luna_pinyin_simp.schema.yaml。

由一个特殊的default.yaml表示当前总配置，会随着用户其他配置的改变而改变。
用什么输入方案就修改对应的配置文件。可以直接编辑默认schema文件。但更好的方法是创建一个对应的custom配置文件，如default.custom.yaml:
```
patch:
  menu/page_size: 9
  schema_list:
    - schema: luna_pinyin_simp
```
1. patch表示对默认的配置default.yaml进行修改。
2. 候选项由5个词条改成9个。
3. 输入方案去掉其他4个不需要的，只加载简体方案。

emoji

让简体方案支持emoji表情输入。在default.custom.yaml / luna_pinyin_simp.custom.yaml加入：

engine/translators:
  - punct_translator
  - r10n_translator
  - reverse_lookup_translator
recognizer/patterns/reverse_lookup: "`[a-z]*$"
schema/dependencies:
  - emoji
abc_segmentor/extra_tags:
  - reverse_lookup
reverse_lookup:
  dictionary: emoji
  enable_completion: false
  prefix: "`"
  tips: 〔表情〕

要输入表情，按"`"，后接对应的表情字符串，具体参考rime emoji.
其中abc_segmentor/extra_tags在输入汉字时，表情包也显示在候选项中，特别是输入单个字母时（如'b')，候选项可能全是表情，可以注释掉。

词库

词库文件分两种。
1. 用户输入累计，文件名带userdb。这个词库很小，基本不足为虑，可以随便编辑，甚至直接删除。
2. 另一种就是我们要导入的大文本词库，文件名以.dict.yaml结尾。导入后，通过点击「重新部署/deploy」生成二进制格式词典以.table.bin结尾。
  
  注意，重启Fcitx没有效果，必须「重新部署/deploy」。
文本词库.dict.yaml的格式：

```

Rime dictionary

encoding: utf-8

#

部署位置：

~/.config/fcitx/rime (Linux)

~/.config/ibus/rime (Linux)

~/Library/Rime (Mac OS)

%APPDATA%\Rime (Windows)

#

于「重新部署／deploy」后生效

#

name: luna_pinyin_simp.zh-cn version: "2016.03.02" sort: by_weight use_preset_vocabulary: true import_tables: - luna_pinyin …

啊啊 a a 啊啊叫 a a jiao 啊啊声 a a sheng 啊哎 a ai 阿巴 a ba 阿爸 a ba 阿坝 a ba

   1. 前面几行'#'表示注释。
   2. 三个连续的连接符'-'开始，后面紧跟文本词库的基本配置信息。

      *name*是词库名，词库名加上*.dict.yaml*后缀就是词库文件名。上面例子中，词库名是*luna_pinyin_simp.zh-cn*，对应的词库文件名是*luna_pinyin_simp.zh-cn.dict.yaml*。
   3. *import_tables* 表示导入其他词库。

      一个文本词库文件可以不需要后面的码表，直接导入多个已经存在的基本词库，这个和C语言里面的`#include <stdio.h>`类似。

      特别注意，*import_tables* 没法递归导入。被导入词库的码表部分会加到二进制词库里，开头部分信息（包括被导入词库的 *import_tables* 部分）。譬如 A 导入 B，B 导入了 C，但是最终生成的二进制 A 词库里只有 B 的码表内容，C 的码表内容并没有导入 A。正确的做法是 A 同时导入 B 和 C。
   4. 三个连续的'.'后面紧接词库码表。
   5. 上面码表的编码（拼音）可以省略，只剩字和词。因为在生成二进制词库时，Rime 库会依据输入方案的配置自动补齐编码（拼音）部分。

      这样，拼音方案、五笔方案、双拼方案可以公用同一个文本词库，Rime 会根据方案的命令自动补齐对应的编码（全拼码，五笔码，双拼码）。
3. 简体方案词库配置

   创建*luna_pinyin_simp.custom.yaml*:

patch: translator/dictionary: luna_pinyin_simp.zh-cn

   *translator/dictionary*表示这个方案使用的词库名。默认简体方案使用自带的*luna_pinyin*词库，现在修改成用户自己的*luna_pinyin_simp.zh-cn*。注意新词库包含了默认词库。
4. 设置同步

   同步用户配置以及词库本身，方便不同的机器共享。譬如可以同步到dropbox，这样不同的机器可以及时共享配置。

   直接編輯用戶文件夾下的*installation.yaml*，添加：

sync_dir: "/home/username/workspace/rime" installation_id: "Rime"

   默认的*installation_id*是一串字母数字，改成比较好认识的。最终同步到 *sync_dir/installation_id* 下面。注意*sync_dir*必须是绝对路径，不要用tilde。

   同步分为两部分：

   1. 配置文件的同步是单向的，即有Rime目录同步到到*sync_dir*。所以在新机器上部署时，配置文件需要拷贝到Rime目录。
   2. 用户词库的同步是双向的，即在Rime目录和*sync_dir*之间双向同步。
5. 重新部署/deploy

   Fcitx的图像界面就有*deploy*字样，点击即可。如果找不到图形设置选项，可以：

   ```bash
   $ rm ~/.config/fcitx/rime/default.yaml

然后重启Fcitx。缺少default.yaml，Rime会触发重新部署deploy。

对于大的词库，可能需要等几分钟，Rime会根据文本词库生成对应的二进制词库.table.bin。

同步概要
1. 在Rime设置里点击“同步”。
2. 因为Rime默认不会同步一些依赖的配置文件。所以第一次同步，还需要手动复制一些额外配置到同步目录下：
```
~ $ cp -n ~/.config/fcitx/rime/*.yaml sync_dir/installation_id
```
OpenCC

Fcitx 和 Fcitx-rime 都会依赖 app-i18n/opencc 包。

简体繁体转换。opencc命令默认是转换成繁体，如果是繁体转换成简体，则需要参数-c，制定配置文件（/usr/share/opencc/*.ini）。
```
$ opencc -i input.file -o output.file [-c /usr/share/opencc/zhs2zht.ini]
```

重新部署

Rime把用户数据同步在sync_dir下。当需要在新机器上安装Rime时，就可以利用备份快速部署。

sync_dir:
```
# ~/opt/rime-fcitx/Rime-Fcitx
default.custom.yaml
default.yaml
emoji.schema.yaml
installation.yaml
luna_pinyin.schema.yaml
luna_pinyin.userdb.txt
luna_pinyin_simp.big.dict.yaml
luna_pinyin_simp.custom.yaml
luna_pinyin_simp.fcitx.dict.yaml
luna_pinyin_simp.schema.yaml
luna_pinyin_simp.userdb.txt
symbols.yaml
```
备份数据主要有两种格式，一种是Rime配置文件*.yaml，另一种是Rime根据用户日常输入习惯保存的小型词库*.txt。需要注意的是，这种个人习惯词库一般称为“词典”比较合适，它和上面我们提到导入的大词库在格式、加载速度上大不同。用户习惯词典是文本文件，数据量很小，加载速度慢；*.yaml词库在部署后会转成二进制*.bin文件，加载到内存，查找速度快。
导入配置文件

重点注意，虽然前面备份时，额外拷贝了诸如default.yaml之类的文件，但是导入时最好重新用官方的。因为拷贝的default.yaml可能含有多余的选项。
```
 ~ $ cp ~/opt/rime-fcitx/Rime-Fcitx/*.yaml ~/.config/fcitx/rime
```
前面说过，配置文件的同步是单向的，所以必须拷贝。
重新部署

同步词典

导入用户词典之前必须关闭Rime输入法，因为Rime运行时独享对用户习惯词典写权限。
必须在Rime目录下运行。

~ $ cd ~/.config/fcitx/rime
~ $ rime_dict_manager
~ $ rime_dict_manager -l
~ $ rime_dict_manager -i luna_pinyin_simp  ~/opt/rime-fcitx/Rime-Fcitx/luna_pinyin_simp.userdb.txt
# or
~ $ rime_dict_manager -s

启动Rime即可。

Ref

Java

2015-12-15T00:00:00+00:00

ABCs
IcedTea Open Java SE
Oracle's Java SE
Configuring
Practice
Bugs
Refs

ABCs

There are three different editions of the Java platform: Standard Edition (SE), Enterprise Edition (EE), and Micro Edition (ME).

Most of the time, we require the SE version. Currently, version 7 is most widely deployed.
There are two different Java implementations: OpenJDK and Oracle Java.

They mostly depends on the same code but OpenJDK is fully open source while Oracle Java contains some proprietary binaries. Most Java applications work fine with either implementation.
There are two different Java packages: JDK and JRE.

Java Development Kit (JDK) for Java development (programming) while Java Runtime Environment (JRE) to run Java application. JDK includes JRE. So when installing JDK, there is no need to install JRE any more.
Java Distributions.

IcedTea Open Java SE, Sun's Java 6 SE, Oracle's Java 7/8 SE, IBM Java SE, etc.
1. Oracle's Java is previously called Sun Java (<= version 6) that was acquisited by Oracle, Inc.
2. IcedTea can be consireded a distribution of OpenJDK. In most GNU/Linux systems, OpenJDK is released as IcedTea, though the release is still named as OpenJDK.
Oracle's Java 8 SE is released at the time of writing this post but some new features make it incompatible with priori versions.

However, Oracle's Java 7 SE is vulnerable in spite of its efforts to release patches. If you really like Oracle's Java, use version 8 instead!

IcedTea Open Java SE

~ # emerge -pvt virtual/jdk
# -or-
~ # emerge -pvt virtual/jre

Oracle's Java SE

Though Oracle Java SE is unfavorable, this section just shows how to install restrictive packages.

Oracle Java SE is a fetch-restricted package, thus need a few extra steps to emerge. For instance, we should manually download the distfile at first. Try:

~ # emerge -pvt oracle-jdk-bin:1.7

Slot 1.7 means Java 7 while 1.8 refers to Java 8. Likely:

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild  N F   ] dev-java/oracle-jdk-bin-1.7.0.80:1.7::gentoo  USE="X alsa fontconfig (-aqua) -derby -doc -examples -jce -nsplugin -pax_kernel (-selinux) -source" 149,933 KiB
[ebuild  N     ]  dev-java/java-config-2.2.0:2::gentoo  PYTHON_TARGETS="python2_7 python3_4 -python3_3" 51 KiB
[ebuild  N     ]   dev-java/java-config-wrapper-0.16::gentoo  8 KiB
[ebuild  N     ]   sys-apps/baselayout-java-0.1.0::gentoo  71 KiB
[ebuild  N     ]  app-eselect/eselect-java-0.1.0::gentoo  71 KiB

Total: 5 packages (5 new), Size of downloads: 150,132 KiB
Fetch Restriction: 1 package (1 unsatisfied)

Fetch instructions for dev-java/oracle-jdk-bin-1.7.0.80:
 * 
 * Oracle requires you to download the needed files manually after
 * accepting their license through a javascript capable web browser.
 * 
 * Download the following files:
 *   jdk-7u80-linux-x64.tar.gz
 * at 'http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html#jdk-7u80-oth-JPR'
 * and move them to '/usr/portage/distfiles'
 * 
 * If the above mentioned urls do not point to the correct version anymore,
 * please download the files from Oracle's java download archive:
 * 
 *    http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html#jdk-7u80-oth-JPR

Go to the specified URL of ebuild, download jdk-7u80-linux-x64.tar.gz and put it under /usr/portage/distfiles.

Accept the Oracle-BCLA-JavaSE license.

# /etc/portage/package.license/oracle-jdk-bin
# required by oracle-jdk-bin (argument)
dev-java/oracle-jdk-bin:1.7 Oracle-BCLA-JavaSE

Emerge
```
~ # emerge -av oracle-jdk-bin:1.7
```
Notes

Don't add nsplugin USE flag. Java plugin is now almost dead and vulnerable when browsing the Internet.

Configuring

A package java-config is installed alongside the JDK, which serves the same goal as Gentoo's eselect java-vm utility to choose JDK slots.
Using the java-config / eselect tool with root privileges, a system-wide default java virtual machine (VM) can be set. Of course, you can also use java-config / eselect to customize their personal VM on a user-by-user basis.
```
java-config --help
java-config -L
java-config [-S | -s] vm
# -or-
eselect java-vm help
eselect java-vm list
eselect java-vm show
eselect java-vm set [ system | user] vm
```
JAVA_HOME etc. environment variables is set automatically by /etc/profile.d/java-config-2.sh script from the ebuild.

Practice

You can install the Oracle 8 alongside IcedTea 7.
Set system-vm to IcedTea 7 while set normal user-vm to Oracle 8. IcedTea 7 is stable and widely compatible while Oracle 8 assumes new Java features.
```
~ $ eselect java-vm list
Available Java Virtual Machines:
  [1]   icedtea-bin-7  system-vm
  [2]   oracle-jdk-bin-1.8  user-vm
~ $ eselect java-vm set user 2
# eselect java-vm set system 1
```
Note: you cannot set user-vm under root account (root uses system-vm).

Bugs

dev-java/icedtea-bin:8 throws SSL error when running gradlew:

Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed

Refer to redhat bug 1417317 and gentoo bug 605430.

Currently, upgrading to dev-java/icedtea-bin-3.3.0-r1:8 fixes the issue.

Refs

IRC

2015-10-10T00:00:00+00:00

ABCs
Style
Security
server_default
Server
NickServ
Identification
Unaffilicated Cloak
proxy - Tor
Relay
Systemd user Unit
Troubleshooting

Simple Authentication and Security Layer (SASL) is a method that simplify identification to NickServ as the first step after connecting to IRC servers, before anything else happens. To connect to freenode using Tor, SASL is required. In this post, I will take Weechat and Freenode for example.

ABCs

Conncet through WebChat on browser.

Just fill in nick and channel names on freenode webchat. HTTPS (ssl) establish a secure connection IRC server compared to HTTP.

By Webchat, we cannot require other services.
Connect in the clear.

Similar to Webchat, the clear way connects directly without proxy or nick reservation (by NickServ).

Should provide IRC hostname (chat.freenode.net) and port (6697) which are implicit in Webchat (webchat.freenode.net and 80).

After connection, join a channel to talk.
Connect through Clients (Weechat Irssi etc.).
Nickserv - register/identify

Reserve and identify your nick name.
SASL authentication

Automatic identification on connection by SASL. Weechat SASL supports different authentication mechanisms:
- plain: plain text file password (default)
- ecdsa-nist256p-challenge: challenge with public/private key
- external: self-signed SSL certificate
- …
Connect with proxy (i.e. Tor).
BNC (i.e. ZNC), or Weechat relay protocol.

Weechat relay supports IRC protocol as well (to be compatibile with other IRC clients).

Style

jim@laptop ~ $ weechat

iset
```
/script search iset
Alt + i
/save
/iset
```
/save after changes. WeeChat also saves your up-to-date configuration any time you exit the program.

Script iset is decpreated and please use built-in script fset instead.
Some specific server commands (i.e. /ns) are not standard and cannot be recoginized by IRC client.
```
irc.network.send_unknown_commands on
```
This will close the buffer (or called tab) immediately when parting a channel.
```
irc.look.part_closes_buffer on
```
By default, server and core buffers are merged together. Use Ctrl+x to tear apart or:
```
irc.look.server_buffer independent
```

Security

SSL
```
irc.server_default.ssl on
irc.server_default.ssl_verify on
irc.server_default.sasl_mechanism external
```
The client verify the server certificate and also the server needs to verify the client certificate - mutual authentication!
Turn off ctcp to prevent others from querying IRC client information.
```
*ctcp*  ""
*ctcp* off
```

Message

irc.server_default.msg_part ""
irc.server_default.msg_quit ""

server_default

Some extra optional default server arguments.

irc.server_default.nicks ""
irc.server_default.username ""
irc.server_default.realname ""

Server

/server add freenode chat.freenode.net/6697 -ssl
/set irc.server.freenode.nicks "username"
/connect freenode

Check server_default section of ~/.config/weechat/irc.conf. Common default server options can be set here (i.e. irc.server_default.ssl).

NickServ

NickServ allows users to register nicks on IRC servers. Different IRC servers support different set of NickServ commands. We can check by /msg nickserv help beforehand. Make sure you are connected to IRC server with your desired nick by /nick new-nick command.

Don't execute the following commands in a channel tab, otherwise sensitive information (i.e. password) might be disposed and sniffed. All IRC commands are case-insensitive.

Take Freenode for example:

/msg NickServ REGISTER <password> youremail@example.com

In Weechat, the pasword youremail@example.com part will be displayed as asterisks. But don't worry as long as you put a space between password and youremail@example.com.

After this command, Weechat will show password and youremail@example.com as plain text. IRC server uses the current nick (username) as the account name. So username is both the nick and account name registered on freenode. This special nick is called primary nick.

In a while, you will receive an email, from which you should copy the verification command to complete the registration process:
/msg nickserv confirm <temporary-code> or /msg nickserv verify register <username> <temporary-code>

This temporary-code is only for account registration confirm.

Now, account name username has just one nick username associated on Freenode server, no one else can use nick username unless the correct password is identified (talked later on).

OFTC server does need this extra confirmation step.
/msg NickServ SET HIDE EMAIL ON

To keep your email address private, rather than displaying it publicly, mark it as hidden.

For OFTC, it's /msg nickserv set private on.
Group more nicks account.

A Freenode account can reserve multiple nicks. First switch to a new nick and then group it to our account (as long as that nick is not reserved by others). Grouping nicks in this way gives you the benefit of having all your nicks covered by the same unaffiliated cloak. The default group name is the account name, namely username.
```
# switch to nick "username_"
/nick username_
    
/msg nickserv group
```
To remove a nick from your account:
```
/nick username
/msg nickserv ungroup username_
```
/reconnect

It will prompt like username is registered… you must identify yourself…. To identify yourself as nick username:
```
/msg NickServ identify username <password>
```

Identification

Manually Identification

If you don't configure SASL, then you have two choices:

Manual identification after connection to server as above.

/reconnect freenode
   
/msg NickServ identify username <password> # freenode
/msg NickServ identify <password> username # oftc

Use freenode.command to run identify.
```
irc.server.freenode.command "/msg NickServ identify username <password>"
```
freenode.command runs a command automatically once connected to the server.

SASL plain scheme

OFTC disallow this scheme.

irc.server.freenode.sasl_mechanism plain
set irc.server.freenode.sasl_username username

The username can be any one of your grouped nicks. But usually the primary nick (namely the account name) is used.
/secure set freenode <password>

Password will be recorded in ~/.config/weechat/sec.conf.
irc.server.freenode.sasl_password "${sec.data.freenode}"

Tell weechat where to get the password.
irc.server.freenode.sasl_fail disconnect

If SASL auth fail, disconnect.

SASL external scheme

irc.server.freenode.sasl_mechanism external

As long as sasl_mechanism is changed to external, sasl_username and sasl_password setting won't take effect.
irc.server.freenode.ssl_cert "%h/ssl/freenode.pem"

We should generate a self-signed X509 certificate before /reconnect.

irc.pem

OpenSSL is a cryptography toolkit implementing the SSL/TLS protocols and related cryptography libraries. Virtually, almost any packages using SSL/TLS require OpenSSL support, OpenSSH and Firefox (HTTPS) included.

The openssl program is a command line tool from OpenSSL, providing various cryptography functions. In this post, it will used to genrate self-signed X509 certificate.

jim@laptop ~ $ openssl req -newkey rsa:4096 -nodes -keyout irc.pem -sha256 -x509 -days 365 -out irc.pem -subj "/C=US/ST=NewYork/L=NewYork/O=UST/OU=CS/CN=irc" -addext "subjectAltName = DNS:irc, DNS:weechat, DNS:127.0.0.1"
jim@laptop ~ $ mkdir -p ~/.config/weechat/ssl
jim@laptop ~ $ mv irc.pem ~/.config/weechat/ssl/

req: certificate request and certificate generating utility.
nodes: no DES encryption. The RSA private key won't be encrypted by DES.
keyout: the private key file.
sha256: the digest to sign certificate.
out: the certificate file.

The keyout and out specify the same output file name, which means private key and certificate will be merged automatically.
subj: certificate Subject field. If not provided on command line, openssl will prompt for user input.

Nick and Certificate Association

To achieve certificate authentication, we should add the certificate's fingerprint to NickServ on freenode server

Get the SHA512 fingerprint.

jim@laptop ~ $ man x509
jim@laptop ~ $ openssl x509 -noout -text -sha512 -fingerprint -in irc.pem

Add the fingerprint:

/whois nick
/msg nickserv cert list
/msg nickserv cert add/del [sha1]

/reconnect

The fingerprint value can be ommited and NickServ on the server will automatically calculate the value.

Finally reconnect to identify nick by SASL external. You can see a line:

gnutls: sending one certificate

during connection. Run /whois nick to see:

has client certificate fingerprint:

Renew certificate

Upon expiration, we need to renew the certificate. Acutally, renewing is to create a new certificate herein.

# Extract private key from certification file (PEM)
~ # openssl rsa -in irc.pem -out irc.key
# Extract public certificate from certification file (PEM)
~ # openssl x509 -in irc.pem -out irc.crt
# Converts the certificate into a certificate request (PEM)
~ # openssl x509 -x509toreq -in irc.crt -out irc.csr -signkey irc.key
# Create a new certification for one year
~ # openssl x509 -req -days 365 -in irc.csr -out irc.crt.new -signkey irc.key
# Concatenate the new certificate and original key
~ # cat irc.crt.new irc.key > irc.pem.new
# Chceck validity
~ # openssl x509 -noout -text -in irc.pem.new

Remember to add the new certificate's fingerprint to IRC server and del the old one.

Unaffilicated Cloak

Unaffiliated Cloak is to hide your IP from IRC channels.
To obtain a cloak, just join the official #freenode channel, ask for one there and reconnect.

When channel staff notices your request, you might get cloaked. Unaffiliated Cloak applies to all channels on Freenode server.
To get a cloak on OFTC is much easier:
```
/msg nickserv SET CLOAK ON
```

The above method is now deprecated! Freenode automatically turns on cloak (vHost) for every account. Just run command /msg HOSTSERV ON. Checking by /whois <username>. Before and after:

# before
(~weechat@joseon-eldv4c.jnih.mqbr.vquou2.IP): weechat
# after
(~weechat@joseon/user/username): weechat

proxy - Tor

Suppose you would like to connect to IRC server by proxy, how to set?

Add proxy
```
/proxy add tor socks5 127.0.0.1 9050
```
This will tell Weechat that a proxy named 'tor' is added.
Server copy

Suppose we find Tor proxy is troublesome someday, updates to current server must be restored. To ease configuration, issue the following command:
```
/server copy freenode freenode-tor
```
Update freenode-tor configuration instead.
Enable proxy for IRC server
```
/set irc.server.freenode-tor.proxy tor
```
However Freenode does not allow Tor connection to chat.freenode.net.

The server hostname should be changed to hidden url freenodeok2gncmy.onion while keeping the original port.
```
/set irc.server.freenode-tor.addresses freenodeok2gncmy.onion/6697
```
Onion certificate.

Cannot wait to reconnect? Yeah, me too. The server certificate cannot be verified since onion url is not certified by almost all CAs.

gnutls: the hostname in the certificate does NOT match "freenodeok2gncmy.onion"

Currently, only a few onion url is signed by digicert.

The solution is maully trust fetched certificates in which though the hostname is freenode.net instead of freenodeok2gncmy.onion.

When connected to chat.freenode.net, Weechat usually receive three certificates with their SHA-1 fingerprints listed. Collect those fingerprints and assign to:
```
/set irc.server.freenode-tor.ssl_fingerprint fp1,fp2,fp3,...
```
Alternatively we can turn off certificate verification:
```
/set irc.server.freenode-tor.ssl_verify off
```
If IRC client supports key pinning, we can choose to trust freenode's public key fingerprint instead certificate's fingerprint which benefits us in that the former keep consistent while the later varies in accord with the real freenode server you connect to. Unfortunately, Weechat does not.

If Weechat throws errors like:

gnutls: peer's certificate is NOT trusted

gnutls: peer's certificate issuer is unknown

Please check whether weechat.network.gnutls_ca_file is correctly set. On Centos, it should be /etc/pki/tls/certs/ca-bundle.crt; but on archlinux, it should be /etc/ssl/certs/ca-certificates.crt.

Recently, freenode Tor servers do not report certificate fingerprint anymore. We can manually print fingerprint by:
```
~ $ torify gnutls-cli --tofu freenodeok2gncmy.onion:6697  --print-cert | openssl x509 -noout -text -sha256 -fingerprint -in /dev/stdin
~ $ sed 's/://g' <<< "fingerprint delimited by colons"
```

Relay

To preserve or share IRC message, we can use Bounced Network Connection (BNC) like ZNC. ZNC allows connections from different end users and relay message for each of them.

To simplify the setup, we can use the standard IRC relay and its extended version - weechat relay. Weechat supports both the weechat relay and the standard IRC relay.

If you add a standard IRC relay in Weechat, you can connect to it with any IRC client like HexChat, Irssi etc. The weechat relay runs over Websocket and only allows connections from clients that support Websocket like glowing-bear or Weechat android. However, you can not connect to a weechat relay by weechat installed from Linux package manager!

This section demonstrates the following schema:

Create a weechat relay instance on a remote server, bound to loopback.
Nginx as a reverse proxy.
Glowing-bear or Weechat-android as the front-end client.

Weechat Relay

Add relay

/relay add ipv4.weechat 9000  # plaintext relay without SSL
/set relay.network.bind_address "127.0.0.1"
/set relay.network.password MY-RELAY-PASSWORD
/set relay.network.max_clients 3

Nginx Reverse Proxy

Nginx HTTPS sample

server {
    listen 80;
    ...
    # enforce https
    return 301 https://$server_name$request_uri;
}

server  {
    ...
    ssl_certificate "/etc/letsencrypt/live/irc.example.com/fullchain.pem";
    ssl_certificate_key "/etc/letsencrypt/live/irc.example.com/privkey.pem";
    ....

Add nginx proxy

# Set connection header based on upgrade header
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# Protect from brute force attack
limit_req_zone $binary_remote_addr zone=weechat:10m rate=5r/m;

server {
    listen 80;
    server_name irc.example.com;

    return 301 https://$server_name$request_uri;

}

server {
    listen 443 ssl http2;
    server_name cloud.example.com;

    ssl_certificate "/etc/letsencrypt/live/irc.example.com/fullchain.pem";
    ssl_certificate_key "/etc/letsencrypt/live/irc.example.com/privkey.pem";
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout 10m;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
	   
    location /weechat {
         proxy_pass http://127.0.0.1:12545;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection $connection_header;
         proxy_set_header Host $http_host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_read_timeout 3600;
         limit_req zone=weechat burst=1 nodelay;
    }
}

Direct Relay

Acutally, we can connect to the weechat relay directly.

Create a SSL relay and configure as below.

relay.network.max_clients    integer  5
relay.network.password       string   "password"
relay.network.ssl_cert_key   string   "%h/ssl/irc.pem"
relay.port.ssl.irc           integer  12345

Run /relay sslcertkey to load the cert and key without restart.

Connect to Relay

irc.server.ora1.addresses          string   "1.2.3.4/12345"
irc.server.ora1.password           string   "password"
irc.server.ora1.sasl_mechanism     integer  plain
irc.server.ora1.ssl                boolean  on
irc.server_default.ssl_verify      boolean  on
irc.server.ora1.ssl_fingerprint    string   "<sha512>"

We pin the relay's cert fingerprint for SSL verification.

Systemd user Unit

We can create a systemd user unit to start launch Weechat in headless mode.

Create user unit file ~/.config/systemd/user/weechat-headless.service as bellow.

[Unit]
Description=A headless WeeChat client and relay service
After=network.target

[Service]
Type=forking
ExecStart=/usr/bin/weechat-headless --daemon

[Install]
WantedBy=default.target

Enable and start user unit.

~ $ systemctl --user daemon-reload

~ $ systemctl --user enable weechat-headless.service
~ $ systemctl --user start weechat-headless.service

~ $ systemctl --user status weechat-headless.service

If something wrong, we can send signal SIGHUP to the headless process.

~ $ kill -HUP <pid>

Troubleshooting

If it reports "unable to load plugin", please install the optional dependencies.

│22:49:38 =!= | Error: unable to load plugin
│             | "/usr/lib/weechat/plugins/spell.so": libaspell.so.15:
│             | cannot open shared object file: No such file or
│             | directory
│22:49:38 =!= | If you're trying to load a script and not a C plugin,
│             | try command to load scripts (/perl, /python, ...)
│22:49:38 =!= | Error: unable to load plugin
│             | "/usr/lib/weechat/plugins/tcl.so": libtcl.so: cannot
│             | open shared object file: No such file or directory
│22:49:38 =!= | If you're trying to load a script and not a C plugin,
│             | try command to load scripts (/perl, /python, ...)
│22:49:38 =!= | Error: unable to load plugin
│             | "/usr/lib/weechat/plugins/ruby.so": libruby.so.2.7:
│             | cannot open shared object file: No such file or
│             | directory
│22:49:38 =!= | If you're trying to load a script and not a C plugin,
│             | try command to load scripts (/perl, /python, ...)

Github project page blog

2015-10-02T00:00:00+00:00

blog.example.com for Github project page, mainly hosting Chinese posts.
www.example.com and example.com for Github user page, mainly hosting technical posts.

Top domain example.com can ONLY be used as an alternative to subdomain www.

Project on Github

Create a repository projectname on Github.

Project page

Create a branch named exactly as gh-pages for repository projectname.

Default branch

In Github repository setting, set gh-pages as the default branch instead of the usual master one.

Thus every action like pull and push will take gh-pages as the default branch.

Clone

$ cd ~/workspace
$ git clone https://github.com/username/projectname.git
$ cd projectname
$ cat .git/config
$ git branch

Template

Choose a Jekyll template. Here I just use the one for www.example.com. Copy the necessary files to ~/workspace/projectname/.

Customization

CNAME file

Set the content to be:

blog.example.com

Do NOT add the prefix http://.

Freenom DNS

We should add a DNS CNAME record for blog.example.com on DNS control panel.

BLOG CNAME username.github.io

Notes

If without customization, the user page URL would be username.github.io while the project page URL would be username.github.io/projectname.
You can see that in the DNS control panel, we have two CNAME records to the same destination:

BLOG CNAME username.github.io

WWW CNAME username.github.io
Whether it is user page or project page, the DNS CNAME record points to the same destination username.github.io. You should NOT or can NOT set DNS CNAME record to username.github.io/projectname.

It Github pages server's job to determine the real (user or project) page location.
You can also find two DNS A record:

␢␢␢␢ A 192.30.252.153

␢␢␢␢ A 192.30.252.154

The four ␢ means leaving the first cell blank. You could replace ␢ with example.com, but FREENOM will ignore this field. If you replace it with other subdomain like test.example.com, then the subdomain cannot make use of Github's CDN service to speed up your site.
The two DNS A records are making example.com the alternative URL to www.example.com.

Top domain example.com can ONLY be the alternative to www subdomain.

baseurl and url

In Jekyll template, we usually need to use variable like site.url, site.baseurl, post.title etc. For a permanent URL:

http://username.github.io/projectname/2015/09/23/gentoo/

http://blog.example.com (without trailing slash) is the url variable, while /projectname (without trailing slash) is the baseurl variable.

For url in _config.yml, just set it to http://blog.example.com.
For baseurl,
1. If use custom url, then just set it to /;
2. If not, set it to /projectname (without trailing trash). When doing test locally on localhost:4000, you should run jekyll serve –baseurl '' which sets baseurl to empty.
Settings in _config.yml are key: value pairs.
1. Add least a whitespace after colon (:).
2. The value can (or not) be enclosed with quotes depending on your preference.

codepage iocharset

2015-10-02T00:00:00+00:00

1 Introduction

In Linux, X or console is usually garbled with squares and unknown characters, mojibak - the garbled text that is the result of text being decoded using an unintended character encoding. The screen is in a mess with garbage messages of either filename or file contents, especially of Windows MSDOS/FAT/NTFS filesystem. In this post, I will discuss the reason of garbled screen. And I also try to elaborate the process of storage, transfer, and display characters in computer.

There are mainly four parts to be clarified, namely Character, File, Font and Locale. Character refers to the basics of language of personal interest, including Character set and Character encoding. File involves filesystem and File Content, while Font requires the support of fontconfig and locale.

Dispalying Chinese filename and file contents of MSDOS/FAT/NTFS filesystem within Linux is troublesome in that it involves locale, kernel, userspace program, filesystem, X, Character etc.

2 Character

Computers themselves do not understand printed text as a human would. For computers, every character of text is represented by a number (more concretely binary digit). Traditionally, each set of numbers used to represent alphabets and characters (known as a coding system, encoding, or character set) was limited in size due to limitations in computer hardware.

Windows/Linux kernel or userspace application (read/write MSDOS/FAT/NTFS file content) must understand both the character set and character encoding. We usually take character encoding and character set for each other without explicitly clarification, for they always assume each other. For example, Unicode usualy implies UCS-2 or UTF-16. Keep in mind: character set is meaningless withtout character encoding.

Computer only knows 0 and 1. It does not understand graphical character as human being.
Kernel is a special superior application to those of userspace.

2.1 Character Set

字符/语言的存储表达。目前的电子计算设备只能存储和运算数字，很显然，对于非数字信息，必须有一个“转义”过程，而这个“转义”规则本身并不能完全被存储，一部分需要人工设置。我们讨论的主要对象是“字符集”，如中文字符集，英文字符集，日文字符集等。
字符集 “character set”。当我需要使用（存储、传输、显示等）字符时（如你要浏览俄文网页），首先规定一个字符范围（一般是本国语言涉及到的字符），范围内的字符的使用规则都有确切的定义，这个范围就是字符集。字符集中的每个字符将使用一个唯一的数字指代，叫“code point”。字符集其实是一张字符和数字间的逻辑映射表。

这里我要对“逻辑”二字解释一下。计算机设计中，我们通常需要用到逻辑的概念，譬如逻辑内存，逻辑地址空间，表示虚拟的、脱离实际计算机设备的概念。字符集就是逻辑概念，只对计算机设计者有意义，对计算机本身来说z这张表格没有任何意义。这张字符集表格可以写在纸张上，譬如第0项是“你”，第1024项是“好”，计算机是理解不了这个表的。

Microsoft designs its own character set standard to render different lanauge characters, which is called codepage. The cp437 is for United States and Canada English characters, while the cp936 is for Chinese characters.
字符集的定义在早期是各自为战的状态，国家、组织等某个地区的权威为各自区域/国家范围内使用到的字符定义字符集。当你的计算机系统只用到本区域/国家的字符时，并没有什么问题。

但当你需要使用其它区域/国家字符集的字符时，就会出问题，因为不同的字符集可能有“code point”冲突。日本人的日文字符集里，可能数字100是日文中的“他”；泰国的给泰文字符集里数字100则指泰文中“好”。很显然使用日文字符集的计算机上无法存储、传输、显示泰文的“好”字。

为了能让计算机同时使用各区域/国家的语言/字符，有必要设计一个包含尽可能多字符的字符集，对全世界的所有语言字符设置一个统一字符集，这个字符集表肯定非常大，就是我们耳熟能详的“Unicode”字符集。
同一语言，可以有不同的字符集。譬如中文可以有 Windows “cp936”字符集，可以有 Unicode 字符集，我们自己可以设计一个，只要实力足够说服软件和操作系统使用你设置的字符集。

2.2 Character Encoding

With just a table of character set, computer can NOT handle (understand, store, transfer, display etc.) characters. Before that, code point should be encoded into computer's language - binary digit, which is the responsibility of character encoding. The simplest way to encode is just use the index/code point of characters in chatacter set table. But that is not enough.

There are over 20,000 Chinese characters and most of their code points take at least two bytes no matter in Unicode or cp936. How does computer orgnazie the two bytes, i.e. Unicode 6E49? With big endian (6E first) or little endian (49 first)? Aside, the transfer order of the two bytes online counts when the network MTU is limited.

Does computer use fixed or varied width/bits to organize code point? If fixed width (say 16 bits), then code point 128 and 1024 are recoginized by computer as 0x0080 and 0x0400 respectively. If varied width, code point 128 and 1024 might be 0x80 and 0x400.

In varied width encoding, when encountering two bytes, how should computer treat them? Treat them as a single code point or two sequencial code points? Need pattern support.
That is where chracter encoding comes into effect. encoding takes care of representing human-being code point as computer binary digit. For a programmer, ASCII (7-bit encoding) might be the most famous character encoding method, others are ISO 8859 series, UTF-8, GBK etc. By the way, character set Unicode itself is also a character encoding (usually implies UCS-2 Or UTF-16).

code point is designed by human beings and a logical table which cannot be recognized/understood by computer. It should be encoded into binary digit, which is a process of converting human language to computer language.

Chinese "严"'s Unicode code point is 4E25, but its UTF-8 encoding is E4B8A5. So character encoding does NOT guarantee the encoded binary digit equals to original code point.
A character set might correspond to different character encodings. For example, Unicode can be encoded into UCS-2, UTF-16, UTF-8, UCS-4 etc. However, reversely a character encoding determines its character set.

For example, if a file is stored on disk as ASCII, the character set must be cp437 on Windows. GB2312 assumes cp20936, GBK assumes cp936 while GB18030 assumes cp54936. GB18030 is a varied width encoding method with some characters encoded into 4 bytes. However Windows code page only support single-byte or doulbe-byte encoding, cp54936 does NOT actually implemented on Windows. It is just a name reservation. 4.Though a language's characters can be represented by different character sets and character encodings, most often, those different sets and/or encodings are downward compatible (ASCII -> GB2312 -> GBK -> GB18030). By the way, GBK is not 国家标准.
通常看到网页上提到“内码”，这个内码就是内核处理字符的“character set”。早期 Windows 使用的是“codepage”，自从 Windows 2000 后，Windows 内核默认使用 Unicode，但依然支持那些依然依赖 codepage 编码的应用。Nowadays, both Linux and Windows kernel uses Unicode character set. So applications that takes Unicode character set can run smoothly. If application uses other character set like GBK, then we should set default character set for non-Unicode Programs in Windows control panel.

2.3 Character set/encoding evolution

The most common (or at least the most widely accepted) character set is ASCII (American Standard Code for Information Interchange) for modern English. ASCII is strictly seven-bit, meaning that it uses bit patterns representable with seven binary digits, which provides a range of 0 to 127 in decimal.

Although ASCII was enough for communication in modern English, in other European languages that include accented characters, things were not so easy. The ISO 8859 standards were developed to meet these needs. They were backwards compatible with ASCII, but instead of leaving the eighth bit blank, they used it to allow another 127 characters in each encoding.

ISO 8859's limitations soon came to light, and there are currently 15 variants of the ISO 8859 standard (8859-1 through to 8859-15). Outside of the ASCII-compatible byte range of these character sets, there is often conflict between the letters represented by each byte.

To complicate interoperability between character encodings further, Windows-1252 is used in some versions of Microsoft Windows instead for Western European languages. This is a super-set of ISO 8859-1, however it is different in several ways. These variants do all retain ASCII compatibility.

2.3.1 Unicode UTF-8

Unicode throws away the traditional single-byte limit of character sets. It uses 17 "planes" of 65,536 code points to describe a maximum of 1,114,112 characters. As the first plane, aka. "Basic Multilingual Plane" or BMP, contains almost every character a user will ever need. Many have made the wrong assumption that Unicode was a 16-bit character set (earlier Unicode encoding do indeed use 16-bit unit like UCS-2 and UTF-16).

Unicode has been mapped/encoded in many different ways, but the two most common are UTF (Unicode Transformation Format) and UCS (Universal Character Set). A number after UTF indicates the number of bits in one unit, while the number after UCS indicates the number of bytes, like UCS-2. UTF-8 has become the most widespread means for the interchange of Unicode text as a result of its eight-bit clean nature; it is therefore the subject of this document.

When Unicode first started gaining momentum in the software world, multibyte character sets were not well suited to languages like C, which is the base language of most commonly used programs. Even today, some programs are not able to handle UTF-8 properly. Fortunately the majority of programs, especially the common ones, are supported.

为每一个字符分配全球唯一的一个数字，但是并没有规定这个数字的表示方法。数字的表示方法由 UTF 规范规定。UTF-16 使用 2 个字节表示一个 UNICODE 数字，但是对于 >=216的数字使用 4 字节来表达。UTF-8 则对于 <127 的数字采取单字节表示，大于 127 的数字要根据其大小选择 2~6 个字节进行表示。UNICODE 在程序内部则简单的使用 unsigned long 即可表示一个字符。

2.4 More on Character

When storing, transferring, and displaying, computers refer to character encoding.
When doing conversion, compuers refer to code point as the middle relay: i.e. GBK -> UTF-8:
1. Use GBK encoding method to get the cp936 code point;
2. Convert cp936 code point to Unicode code point by system API call;
3. Encode Unicode code point to UTF-8.
When programming, an unsigned long will hold a Unicode code point.

3 File

3.1 filesystem

A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. The word is also used to refer to a partition or disk that is used to store the files or the type of the filesystem. It only cares about the meta data of indexing, locating, organizing files.

3.2 Windows NTFS

NTFS is superior to FAT. Most of the disucssion on FAT below applies to NTFS as well. One difference is that NTFS does not need codepage anymore. The reason of omitting codepage is not clear yet. Maybe NTFS uses Unicode for short filename.

3.3 Windows FAT

The FAT filesystem.

The traditional DOS filesystem types are MSDOS, FAT12 and FAT16. Here FAT stands for File Allocation Table: the disk is divided into clusters, the unit used by the file allocation, and each FAT entry describes which clusters are used by which files. The number of sectors per cluster is given in the boot sector byte 13.

Since MSDOS is nearly dead. I will not go into details.

3.3.1 FAT disk Layout

First the boot sector (at relative address 0), and possibly other stuff. Together these are the Reserved Sectors. Usually the boot sector is the only reserved sector.
Then the FAT entries (following the reserved sectors; the number of reserved sectors is given in the boot sector, bytes 14-15; the length of a sector is found in the boot sector, bytes 11-12). The File Allocation Table has one entry per cluster. This entry uses 12, 16 or 28 bits for FAT12, FAT16 and FAT32.
Then the Root Directory (following the FATs; the number of FATs is given in the boot sector, byte 16; each FAT entry has a number of sectors given in the boot sector, bytes 22-23).
Finally the Data Area (following the root directory; the number of root directory entries is given in the boot sector, bytes 17-18, and each directory entry takes 32 bytes; space is rounded up to entire sectors).

3.3.2 12 16 VFAT 32

DOS 1.0 and 2.0 used FAT12. The maximum possible size of a FAT12 filesystem (volume) was 8 MB (4086 clusters of at most 4 sectors each).

DOS 3.0 introduced FAT16. Everything very much like FAT12, only the FAT entries are now 16 bit. Now the maximum volume size was 32 MB, mostly since DOS 3.0 used 16-bit sector numbers. This was fixed in DOS 4.0 that uses 32-bit sector numbers. Now the maximum possible size of a FAT16 volume is 2 GB (65526 clusters of at most 64 sectors each).

In Windows 95 a variation was introduced: VFAT. VFAT (Virtual FAT) is FAT together with long filenames (LFN), that can be up to 255 bytes long. The implementation is an ugly hack. These long filenames are stored in special directory entries. VFAT is not commonly used due to its subsequent FAT32.

FAT32 was introduced in Windows 95 OSR 2 and quickly replaced VFAT on Windows system. It is not supported by Windows NT. Everything very much like FAT16, only the FAT entries are now 32 bits of which the top 4 bits are reserved. The bottom 28 bits have meanings similar to those for FAT12, FAT16. For FAT32: Cluster size used: 4096-32768 bytes. The maximum file size on a FAT32 filesystem is one byte less than 4 GiB (4294967295 bytes).

To be compatible with FAT12 FAT16, when you create a long filename (longer than 8.3) with VFAT, FAT32, or NTFS, the filesystem actually creates two different filenames. One is the actual long filename. This name is visible to Windows 95, Windows 98, and Windows NT (4.0 and later). The second filename is called an MS-DOS® alias. An MS-DOS alias is an abbreviated form of the long filename. The filesystem creates the MS-DOS alias by taking the first six characters of the long filename (not counting spaces), followed by the tilde [~] and a numeric trailer. For example, the filename Brien's Document.txt would have an alias of BRIEN'~1.txt.

Microsoft operating systems use the following rule to distinguish between FAT12, FAT16 and FAT32. First, compute the number of clusters in the data area (by taking the total number of sectors, subtracting the space for reserved sectors, FATs and root directory, and dividing, rounding down, by the number of sectors in a cluster). If the result is less that 4085 we have FAT12. Otherwise, if it is less than 65525 we have FAT16. Otherwise FAT32.

3.3.3 Short/Long filename

filename is one kind of filesystem meta data which usually causes garbled screen. FAT12 and FAT16 only support/implement short filename, while VFAT, FAT32 and NTFS implement both short and long filename. Short filename is case insensitive.

No matter what MSDOS/FAT/NTFS version is used, short filename (on VFAT and FAT32 filesystem) is stored with Windows codepage (i.e. cp936) while long filename (on FAT12 and FAT16 filesystem) is stored with Unicode (i.e. UCS-2 and UTF-16) . NTFS stores filename in Unicode despite of short or long filename.

We can even disable the shortname creation on NTFS which will improve system performance. However some old applications still depend on shortname and would not locate the correct filename.

Garbled filename with squares and weird characters is when your computer incorrectly display filesystem filename on console or X.

3.4 Linux MSDOS/FAT/NTFS support

When we say Linux supports Windows MSDOS/FAT/NTFS filesystem, it means MSDOS/FAT/NTFS part is enabled (Y or M) in Linux kernel. Pay attention to kernel keyword. If a filesystem is supported by operating system, it is indeed supported by the kernel of that operating system, including mounting the filesystem, updating filesystem meta data (creating/deleting/updating files etc.), and displaying filename on console and X which is the focus in the following discussion.

For short/long filename to be displayed properly under Linux, we need:

Enable corresponding MSDOS/FAT/NTFS in kernel.

 File systems  --->
     <*> FUSE (Filesystem in Userspace) support
     ...
     DOS/FAT/NT Filesystems  --->
     <*> MSDOS fs support
     <*> VFAT (Windows-95) fs support
     (437) Default codepage for FAT
     (iso8859-1) Default iocharset for FAT
     <M> NTFS file system support
     [ ]   NTFS debugging support
     [ ]   NTFS write support 
     ...

There is an option MSDOS fs support which is almost dead and can only be found on floppy disk. I will focus only FAT/NTFS in the following sections.

Enable codepage and iocharset in kernel.

Apart from the basic support in step 1, we should enable NLS options (mainly for displaying short/long filename) in Linux kernel configuration menu, namely - Native Language Support. NLS will build those character encoding into kernel (Y or M) for decoding/converting filename to kernel Unicode and then encoding Unicode to locale. It is important to not become confused. For the most part, the only thing that needs to be done is to build UTF-8 NLS support into the kernel, and change the Default NLS option to utf8 if we don't mount Windows MSDOS/FAT or Apple HFS and would like to maintain a clean UTF-8 system.
```
 File Systems -->
   Native Language Support -->
     (utf8) Default NLS Option
     ...
     <M>   Simplified Chinese charset (CP936, GB2312)
     ...
     ## (Also <*> or <M> other character sets that are in use in the system's FAT filesystems or Joilet CD-ROMs.)
     <*> UTF-8 NLS
```
Take MSDOS/FAT/NTFS for example, filename is encoded by codepage (short filename) or Unicode (long filename) and stored on disk. Enabled NLS options (i.e. cp936) decode binary digits (i.e. GBK or GB2312) to kernel Unicode (through kernel API) and then to system locale before displaying. The first option (set to utf8) is the default NLS used for decoding when mounting filesystem. You can also find that NLS_CODEPAGE_CP936 (decode Chinese short filename) and NLS_UTF8 (encode filename Unicode to system UTF-8 locale) are enabled.

You can notice many NLS options started with ISO 8859- of them most are not necessary for Chinese users. Russian users need to enable NLS_KOI8_R instead of NLS_CODEPAGE_CP936.

NLS is designed for not only MS-DOS/Windows filesystem, but other operating system filesystem like Apple HFS file system that uses MAC codepages. You can also enable those NLS support if needed.
cjktty patch for Chinese on console.

In order to display Chinese on console, we need cjktty patch.
mount arguments.

Having enabled MSDOS/FAT/NTFS filesystem and corresponding NLS in kernel, we then specify add those NLS to mount command through codepage, iocharset and/or utf8 arguments. Suppose a FAT32/NTFS partition with Chinese filename and file contents, they can be mounted in Linux of zh_CN.GBK locale:
```
 # mount -t vfat -o codepage=936,iocharset=cp936 /dev/sda5 /mnt/data
 # mount -t ntfs-3g -o utf8 /dev/sdb7 /mnt/misc
```
Most of the time, we use -t msdos to mount FAT12 and FAT16; use -t vfat to mound VFAT and FAT32. If you don't use special mount options (i.e. English language users), -t mouint option can be omitted since mount detects filesystem itself.

There are three options need special attention: codepage, iocharset, and utf8. Possible NLS options must be enabled in kernel. We can set their default values in kernel - Default codepage for FAT, Default iocharset for FAT, Default NLS Option which can be overriden in mount command. codepage/iocharset/utf8 arguments support Windows MSDOS/FAT/NTFS short/long filename decoding/conversion.

In short, the codepage option is for short filename, and the iocharset option is for long filename. The utf8 option alone is to replace iocharset=utf8/nls=utf8 which is not recommended.

3.4.1 codepage

In old Windows system, both filesystem meta data and file contents are stored as codepage. But in this section, I only talk short filename display under Linux.

The codepage option is needed for vfat-compatible (MSDOS/FAT) and smbfs filesystems. It should be set to the codepage number used under MS-DOS/Windows in your country.

NLS codepage option sets the codepage number for converting to shortname characters on Windows MSDOS/FAT. By default, mount will take FAT_DEFAULT_CODEPAGE in Linux kernel. Short filename is encoded by codepage and should be decoded and then converted to kernel Unicode before displaying.

When should we use codepage? MSDOS/FAT12/FAT16 only support short filename of 8.3 format by codepage (i.e. Chinese cp936) encoding (i.e. GB2312 or GBK). If you mount such partitions, NLS cp936 must be enabled in Linux kernel (M or Y) and codepage=936 argument be added to mount command such that the short filename can be displayed.

Say you have a FAT32 parition and mount it with option -t msdos -o codepage=936, filenames are displayed short. If you want to display true long filename, use -t vfat or just remove -t option instead.

Most of the time, we needn't codepage mount argument at all since MSDOS/FAT12/FAT15 and applications depending on short filename are almost dead. Currently, VFAT and FAT32 support long filename by Unicode encoding. Actually, we do omit this argument as short filename is almost deprecated.

When we use codepage=936, the kernel:

Find the relevant NLS cp936 module;
Decode short filename to GBK/GB2312 code point;
Convert the code point to Unicode code point;

Kernel operates characters with Unicode.

3.4.2 iocharset

Character set used for converting between the encoding used for user visible long filename characters (display on screen) and 16 bit Unicode characters (long filename are encoded by Unicode UTF-16). Long filenames are stored on disk in Unicode format (usually UTF-16), but Unix for the most part doesn't know how to deal with Unicode. By default, FAT_DEFAULT_IOCHARSET setting is used. Since only VFAT/FAT32/NTFS support long filename, when mounting MSDOS/FAT12/FAT16, you should not use iocharset argument.

Filesystems with MS-DOS or Windows origin (i.e.: vfat, ntfs, smbfs, cifs, iso9660, udf) need the iocharset mount option in order for non-ASCII characters in file names to be interpreted properly. The value of this option should be the same as the character set of your locale, adjusted in such a way that the kernel understands it. This works if the relevant character set definition (found under File systems -> Native Language Support) has been compiled into the kernel or built as a module.

This argument should NOT be ignored since it controls the display of true long filename unless your kernel Default NLS Option (utf8 in above kernel configuration menu) equals to your system locale. It should be set to be the corresponding character encoding of your Linux system locale. If locale is not UTF-8, like GBK or GB2312, set it to iocharset=cp936. If locale is UTF-8, use utf8, iocharset=utf8/nls=utf8.

The argument of doing UTF-8 translations with just utf8 (NOTE: "iocharset=utf8" or nls=utf8 is not recommended) is recommended if unsure. If iocharset=utf8 mount argument is used, Linux kernel VFAT module will be case sensitive which is incompatible with Windows short filename which does not differentiate filename character case. utf8 alone does not have this side effect. Attention: the counterpart of NTFS iocharset option is now deprecated and renamed as nls. Details refer to man mount.

When we use iocharset=cp936, the kernel:

Read the UTF-16 long filename directly from FAT32/NTFS;
Get the long filename Unicode code point;
Find the relevant NLS cp936 module;
Encode the Unicode code point to GBK/GB2312
Return the GBK code userspace program like X, file editor etc.

We notice that codepage option is decided basically by the language of Windows users while iocharset is set based on Linux locale. The NLS value set for iocharset and codepage should be enabled in kernel - Native Language Support.

3.4.3 Logics

There is a Windows FAT32 partition with Chinese filenames. Each file has two filenames - the short one and the long one.
Short filename is encoded by codepage cp936 / GBK.
Long filename is encoded by Unicode / UTF-16.
mount -t vfat -o codepage=936,nls=cp936 /dev/sda1 /mnt/data on Linux with zh_CH.GBK locale.
Kernel cp936 module decodes and convert short filename to Unicode

If use -t msdos option, the default true filename will be the short one.
Linux kernel decodes UTF-16 binary digits to Unicode directly (Linux kernel uses Unicode naturally).

Then Unicode code points are encoded into GBK.
GBK codes are transferred from kernel to userspace ls, thunar, emacs etc.
Userspace application decode GBK codes and transfer back to kernel input/output system for display.

3.5 File Content

After talking about filesystem and filename, let's turn to file content. filesystem does NOT either care about or know file content. The former only takes care of file meta data to organize directory, file location, indexing, etc. by system kernel, while the later are real data of user/userspace application interest.

Operating system kernel is responsible of filesystem support. For file contents, they are owned by user or userspace program (like file editor). Therefore, displaying filename (need kernel support) is different from displaying file content (need userspace program support) on screen (most of the time, it is X in Linux).

For example, we have a GBK-encoded TXT file contents on FAT32 partition and want to display the file, the corresponding program for displaying should support GBK character encoding.

If we use less or cat command to display the GBK TXT file, the X screen will be garbled, for those commands do not know GBK at all. However, if we use emacs or gedit to open the file, GBK TXT will be well rendered on X screen.

4 Font

4.1 Fonts

Fonts is the graphical skeleton of characters to display. The common western fonts are classified into sans (like Courier), serif (like Times New Man) and monospace. Mostly used Chinese fonts are "宋体", "楷体", "黑体" etc.

When filename or file content is displayed, a good font design help relieve eye and add aesthetic sense.

4.2 Fontconfig

Usually on our system, we have a set of different fonts, like "宋体" for Chinese, "Times" for English, etc. What is more, different fonts may overlap. For example, "宋体" can also be used to display English letters; Google "Nato" can display Chinese, Japanese and Korean. So when system display a character which is covered by several fonts, which font should be chosen? This is when fontconfig come into effect. Fontconfig not only chooses the font to display, but also renders the chosen font in a more aesthetic way.

Fontconfig (or fontconfig) is a free software program library designed to provide configuration, enumeration and substitution of fonts to other programs (like file editor, web browser etc.).

More on fontcnfig, refer to fontconfig.

5 Locale

Locale is the last but second step responsible for displaying characters no matter of filename or file contents. FAT's iocharset and NTFS's nls should be set to or the Default NLS Option equals to system locale for filename display. Userspace program must support GBK (through userspace library like libiconv) to decode GBK TXT file. And then your locale (at least LC_CTYPE) should be zh_CN.GBK.

Userspace X application check locale and then decode the encoded filename or file content to kernel Unicode code points; those code points are then dilivered to kernel IO system for display.

6 Console

Above discussion still does solve Chinese filename and file content display on console.

基于字符界面的控制台 console 本身并不支持点阵化字符输出，要想使之支持中文，必须将控制台切换到图形模式并加入中文解码支持。

通常Linux的控制台工作在文本模式下，要想在屏幕上正确显示，汉字必须将屏幕切换到图形模式，这可以通过调用内核FrameBuffer驱动程序来实现。此外，还要能正确识别系统输出到控制台的汉字信息，并调用汉字显示模块将其输出到屏幕。

一种方法是像DOS下的汉字系统所做的那样：利用系统时钟中断定时监视显存地址B800:0000处的显示缓冲区，动态识别缓冲区中的字符信息。这种方法要求修改内核中断和TTY驱动程序，实现起来比较困难，而且需要直接操纵硬件视频缓冲区，大大影响了系统的可移植性和稳定性。

另一种方法就是UNIX下多数中文平台采用的基于伪终端（Pseudo-Terminals）的外挂式解决方案。伪终端（Pseudo-Terminal）是一种类似于终端的特殊的进程间通信通道（channel）。通道的一端被称为主设备（master pseudo-terminal device），另一端被称为从设备。写入主设备的数据被发送到从设备，而写入从设备的数据也可从主设备读出，对用户来说就好像自己实际上连接到了真正的计算机终端之上。简而言之，伪终端是位于虚拟终端和最终的终端设备之间的一种承担着输入输出转换功能的设备。

伪终端诞生之初就得到了广泛应用，典型的例子是Telnet服务程序。Telnet是一种远程登录服务，用户使用Telnet客户端程序通过网络登录到远程主机之上进行各种操作。Telnet服务程序就是一个伪终端，一端连接到Telnet客户端程序，另一端连接到主机应用程序，客户和应用程序之间通过伪终端进行对话。

当应用程序需要从输入设备（键盘）读入数据时，它向控制台设备（/dev/console）发出系统调用read() ，接着内核 console 驱动响应该系统请求，从输入设备驱动程序（通常为键盘驱动keyboard.c）获得输入数据，并将之返回给应用程序。而当应用程序需要想控制台输出数据时，它通过write()系统调用将数据发送至console设备，再由内核 console 驱动转发到真正的输出设备（终端、打印机…）上去。

通过以上分析可以发现如果在应用程序从控制台设备（/dev/console）读入数据之前截获键盘输入信息，并提交输入法模块处理，再将处理后得到的中英文信息发送至应用程序即可完成中文输入。而在应用程序将输出数据写入/dev/console设备之前截获输出并交由汉字识别模块处理，最终由汉字显示模块输出至屏幕，即可实现中英文输出。这就是zhcon之类的中文控制台的基本工作原理。

So we need cjktty patch to to the conversion.

7 Refs

display on X VS console, different?

cjktty

2015-09-18T00:00:00+00:00

An alternative is Univt patch.

In this post, we will show how to extract cjktty.patch from a patched kernel. Afterwards, we will apply the extracted patch to our own kernel. During the process, we might look into several relevant commands.

Suppose our current system kernel is 4.0.5. We will generate a cjktty.patch suitable for this kernel.
The patched kernel is linux-cjktty from which we will extract the desired patch.

According to past experience, cjktty.patch for kernel 3.18 ~ 3.19 fits 4.0 as well. We will choose the branch 3.19-utf8.
Clone the chosen branch locally.
```
$ cd ~/workspace
$ git clone --branch 3.19-utf8 --depth 4 https://github.com/Gentoo-zh/linux-cjktty.git
```
1. --branch: we only need branch 3.19-utf8. If not specified, git will clone all the branches, spending a day maybe depending on bandwidth.
2. --depth: from the repository, we find the the top 3 commits are enough to extract our cjktty.patch. We MUST clone at least the top 3 + 1 = 4 commits.

git log

commit 9a5a7d3215307e28df3aea6ac09931a4d55e151e
Author: microcai <microcai@fedoraproject.org>
Date:   Thu Jan 13 05:18:45 2011 +0800

    [vt] add cjk font that has 65536 chars

commit 3dbe651a011ac6b6bbd976d96b5d08fb2baf709c
Author: microcai <microcai@fedoraproject.org>
Date:   Mon Feb 21 12:58:18 2011 +0800

    [vt] diable setfont if we have cjk font in kernel

commit a2553800e3e8ea1d13f3e3a4211599a3268ce9a2
Author: microcai <microcaicai@gmail.com>
Date:   Fri Jan 9 12:45:07 2015 +0800

    [vt] fix 255 glyph limit, prepare for CJK font support

commit d8cf08a565defc2f9810b9ecd33b1b3211b029e1
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Thu Mar 26 14:00:21 2015 +0100

    Linux 3.19.3

From the output, we can see only 4 commits history as expected. We will extract the patch from the top 3 commits authored by microcai.

Extraction
```
~ $ git diff d8cf08a565defc2f9810b9ecd33b1b3211b029e1 9a5a7d3215307e28df3aea6ac09931a4d55e151e -- > cjktty.patch
# or
~ $ git diff HEAD~3 -- > cjktty.patch
# or
~ $ git format-patch HEAD~3 --stdout > cjktty.patch
# or
~ $ git format-patch -3 HEAD --stdout > cjktty.patch
```
1. Pay attention to the order of commit ID (SHA1 hash). Put the earliest commit ID (4th) before the latest one (HEAD).
2. HEAD~3 means extract patch from top to 3rd commits (inclusive). The latest ID (HEAD) precedes the 3rd one.
3. format-patch is similar to git diff. -n <commit> is equivalent of <commit>~n. The patch file orgnization is a little different from that of git diff.
4. We recommend to use format-patch to include commit messages, making it more appropriate for most scenarios involving exchanging patches over the Internet. Details refer to reference 1.

Test the patch

Attention please; the -p [num] must be right, otherwise the wrong files would be patched even when no errors prompt up. Inpsect the patch file to see the pathname.

~ # cd /usr/src/linux/
   
# determine the '-p' option value
~ # less /path/to/cjktty.patch

~ # patch --verbose -p1 --dry-run < /path/to/cjktty.patch

~ # git apply --verbose --whitespace=warn -p1 --stat /path/to/cjktty.patch
~ # git apply --verbose --whitespace=warn -p1 --numstat /path/to/cjktty.patch
~ # git apply --verbose --whitespace=warn -p1 --check /path/to/cjktty.patch
~ # git apply --verbose --whitespace=warn -p1 --summary /path/to/cjktty.patch

When testing patch file, it might remind errors like:

Hunk #17 FAILED at 2708.
# or
error: patch failed: drivers/video/console/fbcon.c:2689
error: drivers/video/console/fbcon.c: patch does not apply

Error number 2708 (reported by patch) refers to line number in Linux-cjktty sources, while 2689 (reported by git apply) in Gentoo-sources. Search 2708 or 2689 in patch file and check either/both kernel sources.
Some error are caused by extra whitespaces, especially those on blank lines.

By default, all pathnames will be patched. Sometimes, we can patch only perticular pathnames by git apply --include <pattern>. patch does not support this feature.

~ # git apply --verbose --whitespace=warn -p1 --include '*myfile.lua' --stat /path/to/cjktty.patch
~ # git apply --verbose --whitespace=warn -p1 --include '/path/to/dir' --stat /path/to/cjktty.patch

Vice versa, we can --exclude <pattern>:

~ # git apply --verbose --whitespace=warn -p1 --exclude <pathname-pattern> --stat /path/to/cjktty.patch

If both --include and --exlude options are present, the first match determines if a pathname is patched or not.

Apply the patch

~ # cd /usr/src/linux/
~ # patch --verbose -p1 < /path/to/cjktty.patch
~ # git apply --verbose --whitespace=warn -p1 /path/to/cjktty.patch

We can, revert the patch by the -R option.

~ # cd /usr/src/linux/
~ # patch --verbose -p1 --dry-run -R < /path/to/cjktty.patch
~ # git apply --verbose --whitespace=warn -p1 --check -R /path/to/cjktty.patch

You can reverse a patch by adding -R argument.

If /usr/src/linux source is polluted by patch error, you can re-install the kernel source.

We cannot reverse a patch error.
```
# emerge -av --unmerge gentoo-sources-4.0.5
# rm -r /usr/src/linux-4.0.5-gentoo
# emerge -av gentoo-sources-4.0.5
```
1. Pay attention to use unmerge instead of depclean. The former will keep package dependencies.
2. Maybe we can just extract the source code to /usr/src/linux.
Reference
1. How to get patch files from multiple commits?.

127.0.0.1 VS 0.0.0.0

2015-09-14T00:00:00+00:00

Before a host is reachable, it must has an network interface device to which IP from ISP is assigned. That is to say, network interface undertakes networking functionalities.
On open public netwoks like 'Internet', usually a host is given with a IP with which a hostname is registered/associated. For example, 'google.com' binded to its server IPs.
On some small local networks, network implementation might be narrowed to reduce complexity. So usually on a LAN, you host IP is not registered with your hostname. For example, your personal PC only gets an IP.
A host can be reached by IP from ISP. Meanwhile, every host has a special reserved IP 127.0.0.1/32 (not from ISP) bound to the loopback network interface, which works only within the host itself. So a host never reach any other host (except itself) on the network by 127.0.0.1.

Actually 127.0.0.0/8 - This block is assigned for use as the Internet host loopback address. A datagram sent by a higher-level protocol to an address anywhere within this block loops back inside the host. This is ordinarily implemented using only 127.0.0.1/32 for loopback. Addresses within the entire 127.0.0.0/8 block do not legitimately appear on any network anywhere.

The same convention is defined for computer’s that support IPv6 addressing using the connotation of ::1.

The purpose of the loopback range is testing of the TCP/IP protocol implementation on a host. Since the lower layers are short-circuited, sending to a loopback address allows the higher layers (IP and above) to be effectively tested without the chance of problems at the lower layers manifesting themselves. 127.0.0.1 is the address most commonly used for testing purposes.
We can say 127.0.0.0/8 within the bound of a host forms a tiny network which only includes the host itself. So within this tiny loopback network, the host IP is associated/registered with a loopback hostname 'localhost'.
But usually in system, we are suggested to set another hostname like in Gentoo /etc/conf.d/hostname. As stated in item 3, this value is usually registered with IP from ISP, especially for personal PCs on LAN. Such hostname value is just a local label for the host. If unset, value in /detc/conf.d/hostname is default to 'localhost' such that the hostname is regesitered to the tiny loopback network. Even set, you can still bind it to loopback network by adding '127.0.0.1 hostname' into /etc/hosts.
"0.0.0.0" is a valid address syntax. So it should parse as valid wherever an IP address in traditional dotted-decimal notation is expected. The all-zero value does have a special meaning. So it is "valid", but has a meaning that may not be appropriate (and thus treated as not valid) for particular circumstances. It is basically the "no particular address" placeholder. It depends on the context of use to determine what "no particular address" really does.

In the context of a route entry, it usually means the default route.

In the context of servers, 0.0.0.0 means "all IPv4 addresses on the local machine". If a host has two ip addresses, 192.168.1.1 and 10.1.2.1, and a server running on the host listens on 0.0.0.0, it will be reachable at both of those IPs by other hosts. At the same time, server host itself can also reach the service by 127.0.0.0/8.

The IPV6 form of 0.0.0.0 is ::0 or just ::.
On my Gentoo, I use RDP to connect to local headless VB WinXP. The default headless listening address is 0.0.0.0 which means other host in the networks can reach the headless WinXP too.

So set it to 127.0.0.1 explicitly.

symbolic links to config in central location

2015-09-13T00:00:00+00:00

UPDATE: Real config files remain at where they are. Create symbolic links to them at a central place. When backup, add -L argument to cp.

There are all kinds of config files and scripts to take care of during installation or system backup. It is smart practice to put them in a central place, while creating corresponding symbolic links to them.

mkdir /gtmisc

gtmisc is the central place to store those files.
mv /etc/fstab /gtmisc
ln -sv ../gtmisc/fstab /etc/fstab

/gitmisc will store files related to root account. Repeat step 2 & 3 for other files.

For normal user files, you can operate like this in user home directory.
List of files
```
 /etc/fstab
 /etc/portage/{make.conf,repos.conf/,}
 /usr/local/bin/*
 /usr/local/sbin/*
 /usr/local/portage (local portage, pay attention to portage:portage ownership)
 /etc/wpa_supplicant/wpa_supplicant.conf (pay attention the file attributes)
 etc.
```
gtmisc can be put anywhere you desire. But pay attention to fstab which should be accessible at the early stage of boot. For example, if gtmisc{/fstab,} were put under /home/me/gtmisc, then mount points except / in fstab would not be mounted since the mount of /home depends on /home/me/gtmisc/fstab, which forms a cyclic dependency.
Relative VS absolute symbolic link
```
 ln -sv /path/to/source /path/to/target
```
1. 'target' is the symbolic link to created, pointing to 'source' (normal files or directories).
2. 'ln' command can be executed under any 'pwd' regardless of 'source' or 'target' directory.
3. relative link: 'target' will look for 'source' through relative path to its own directory.
4. absolute link: 'target' will look for 'source through absolute path.
5. If you move absolute link around, it still points to the correct target, while relative link would be dead.
6. Whether 'target' is a relative or absolute link depends on /path/to/source.
  1. If it a relative path (i.e. ln -sv ../gitmisc/fstab /etc/fstab), then target location is relative to '/etc/'. If you move '/etc/fstab' link to '/home/me/fstab', the relative path will be '/home/gitmisc/fstab' which does not exist.
  2. If it a absolute path (ie.e ln -sv /gitmisc/fstab /etc/fstab), then target location is at absolute path '/gitmisc/fstab'.
Notes
1. pvcreate, luksFormat both works on /dev/sda disk?
2. symbolic link relative absolute
3. mtab link dead
4. /home/zachary/gtmisc
5. dd boot
6. dd if=/dev/sdb1 xz > boot-image-backup.xz
7. xzcat image-file.xz dd of=/dev/sdb1
8. sdb1 fat32 on windows, mounted automatically

LVM over LUKS over LVM

2015-09-10T00:00:00+00:00

sda7 and sda8 are separated by other NTFS partitions in use.
Merge them together as if they were a single partition/disk by LVM - our first LVM container.
Encrypt the new single LVM volume by DM-crypt LUKS - our LUKS container.
Create LVM volumes in LUKS container as Gentoo / and /home - our second LVM container.
LVM -> LUKS -> LVM.
No swap partition. If possible, create a swapfile instead referring to swapfile.
BOOT and EFI partitions share and reside on a USB partition, say sdc1. This is different from traditional scheme.
1. The shared USB partition should be formated as FAT32 to satisfiy EFI filesystem requirement.
2. It should be mounted on /boot if necessary.
3. Make a directory 'EFI' at root of the shared parition.
4. The shared USB partition is not encrypted.
5. You'd better use sdc2 for the shared partition to hide it from Windows. Deatils refer to Hide in Windows next.
6. Encryption & decryption key-file is stored on USB partition as well.
Windows uses its own EFI partition on HDD, say sda2.
Why now share boot and EFI on USB stick?
1. A single USB can now help boot many PCs and notebooks, as long as their boot information is located on the USB stick.
2. Kernels now stored in USB, prevent from attack online. We can unplug the USB when booting into Gentoo.
sda7, sda8 > pvcreate, vgcreate (vg), lvcreate (crypt), vg-crypt > cryptsetup, luksOpen (cryptroot) > pvcreate, vgcreate (cryptvg), lvcreate (root, home), cryptvg-root & cryptvg-home.

USB preparation

Create shared USB partition:

# parted -a optimal /dev/sdc
# mkpart primary fat32 0% 256MiB
# set 1 boot on
# quit

EFI partition should be set boot flag.

Create FAT32 filesystem:

# mkfs.vfat -F32 /dev/sdc1
or
# busybox mkfs.vfat /dev/sdc1

USB shared partition is not encrypted. If you put the shared partition within the LVM-LUKS-LVM architecture, then you need add LUKS support to Grub2.

key-file

Preparations

# mkdir /mnt/sdc1
# mount /dev/sdc1 /mnt/sdc1
# mkdir /mnt/sdc1/luks-gnupg-key

Generate GPG-encrypted key-file:
```
$ dd if=/dev/urandom bs=8388607 count=1 | gpg --symmetric --cipher-algo AES256 --output ~/luks-gnupg-key.gpg
```
LiveDVD root acount does not support this command as gpg 2 need GUI popup to prompt password. Switch to normal user account in LiveDVD.

Without explici mark '$', all commands are executed under root account in LiveDVD.
Save key-file to USB stick:
```
$ cp ~/luks-gnupg-key.gpg /mnt/sdc1/luks-gnupg-key/
# umount /mnt/sdc1
```
The gpg-encrypted key-file is stored on the shared USB for booting. You can store it in other media though unnecessary and tedious.
Decrypt key-file for temporal usage:
```
$ gpg --decrypt ~/luks-gnupg-key.gpg > ~/luks-gnupg-key
# cp /home/gentoo/luks-gnupg-key .
```
This temporary decrypted key-file is used in root account. Don't expose decrypted key-file online or to anyone.

sda7 & sda8 preparation

On sda, find the separated free space:

# parted -a optimal /dev/sda
# unit s
# mkpart primary START END
# name 7 'Gentoo partition'
# mkpart primary START END
# name 8 'Gentoo partition'

START and END should exactly be the right sector.

Otherwise, other partitions in use would be ruined! Take /dev/sda7 for example, its previous ancestor partition /dev/sda4's last sector/unit is 100000000s. Then the START of /dev/sda7 must be 100000001s. Remember to PLUS ONE. We can verify this by examine other successive partitions in use.

If possible, resort to percentage like 0% or 100%.
We don't create filesystem ext4 on sda7 or sda8 directly since they are treated as underlying container: LVM -> LUKS -> LVM.

1st LVM container

# pvcreate /dev/sda7 /dev/sda8
# vgcreate vg /dev/sda7 /dev/sda8
# lvcreate -l 100%FREE vg -name crypt
# pvdisplay / vgdisplay / lvdisplay / lvscan

pvcreate is optional since vgcreate will automatically create PVs on sda7 and sda8 if needed.
We can find /dev/vg directory created.
Now, sda7 and sda8 is merged as a single LVM device /dev/mapper/vg-crypt or /dev/vg/crypt.

Each time you create a LVM in VG, a new mapper device will be created /dev/mapper/. The mapper device name will be 'VG name + LV name'. This mapper device is actually a simbolic link. Similarly, /dev/vg/crypt is also a symbolic linking to the same destination as /dev/mapper/vg-crypt.

However, when booting into Gentoo, these mapper device will be real device file.
Try to use pvdisplay, vgdisplay, lvdisplay, lvscan etc. anytime to get information.

LUKS container

# cryptsetup --cipher serpent-xts-plain64 --key-size 512 --hash sha512 --key-file /home/gentoo/luks-gnupg-key luksFormat /dev/mapper/vg-crypt
# cryptsetup --key-file /home/gentoo/luks-gnupg-key luksAddKey /dev/mapper/vg-crypt
# cryptsetup luksOpen /dev/mapper/vg-crypt cryptroot
# cryptsetup luksDump /dev/mapper/vg-crypt

Create LUKS container. Try cryptsetup luksDump /dev/mapper/vg-crypt.
To add a fallback LUKS passphrase, just in case we lost the key-file. According to current experience, passphrase needs less time. This passphrase is different from the one used to encrypt key-file above.

We must provide the the previous key-file or passphrase before adding a LUKS key slot. Attention, we use the decrypted key-file instead of the encrypted one since root account cannot pop up box for decryption.

Up to now, /dev/mapper/vg-crypt is protected by both LUKS key-file (gpg-encrypted) and passphrase. We have achieved full disk encrytion. Try cryptsetup luksDump /dev/mapper/vg-crypt.
Treat vg-crypt as a normal disk partition which is encrypted by DM-crypt LUKS. To make use of it, we have to decrypt it first. Here, we use the fallback passphrase to decrypt. It is faster.

When vg-crypt is decrypted, we also get a mapper device /dev/mapper/cryptroot. The device name can be set freely at your desire.

We can now treat the decrypted device as another normal disk partition, on which we will create our 2nd LVM container.

Why 2nd LVM container?

This question can be narrowed down to: why need LVM over LUKS?

A decrypted LUKS device like /dev/mapper/cryptroot should'd better be regarded as a partition instead of a disk (like /dev/sdb). So command like parted -a optimal /dev/mapper/cryptroot is NOT encouraged. It is complicated to make use of partitions created directly over LUKS device.

What if I take only part of /dev/mapper/cryptroot to install Gentoo, while leaving the rest for NTFS data or even another system Arch Linux?
What if I separate /home, /usr, /tmp etc. mount points?
What if I want to adjust Gentoo's storage?

Of course, we can prepare Gentoo installation on this decrypted LUKS partition directly if don't care above issues:

mkfs -t ext4 /dev/mapper/cryptroot
mount -t ext4 /dev/mapper/cryptroot /mnt/gentoo

Sometime, we can also re-format /dev/mapper/cryptroot for other usage like re-installing Gentoo.

With LVM over LUKS we unlock all the partitions (LVM volumes) by one key-file / passphrase, and easily resize volums without the hassle work above. For example, you want to extend / size while reducing /home size.

If you're going to simply dump everything on a single partition (which is totaly fine with some setups), there is no point of 2nd LVM container. However, if you want to have several volumes, LVM will make it more convenient and easier to use, and this is why many people go after 'LVM over LUKS'.

If you really don't like the 2nd LVM but need separate /, /home, etc. Please lvcreate LVM volumes directly over the 1st LVM container. After that, cryptsetup them repsectively with LUKS key-files or passphrases. What a nightmare!

Refer to:

2nd LVM container

# pvcreate /dev/mapper/cryptroot
# vgcreate cryptvg /dev/mapper/cryptroot
# lvcreate --size 25G --name root cryptvg
# lvcreate --extents 100%FREE --name home cryptvg
# pvdisplay / vgdisplay / lvdisplay

Like in the 1st LVM container, this step is optional.
/dev/cryptvg/ directory created.
Create LVM root device as / mount point.
Create LVM home device as /home mount point.

Wee will see cryptvg-root and cryptvg-home symbolic links under /dev/mapper. Similarly, /dev/cryptvg{/root, /home} links created as well.

Up to now, we have finished the basic LVM -> LUKS -> LVM architecture. We will install Gentoo over the newly created devices of 2nd LVM.

Format Gentoo partitions

When we say format sdxy as ext4, we actually means create ext4 filesystem on sdxy.

# mkfs.ext4 -L "root" /dev/mapper/cryptvg-root
# mkfs.ext4 -m 0 -L "home" /dev/mapper/cryptvg-home

BOOT and EFI share USB partition /dev/sdc1 (VFAT).

Mount Gentoo partitions

During mount and umount, try lsblk, blkid etc.

# mount -v -t ext4 /dev/mapper/cryptvg-root /mnt/gentoo
# mkdir -v /mnt/gentoo/{home,boot}
# mount -v -t ext4 /dev/mapper/cryptvg-home /mnt/gentoo/home
# umount -v /dev/sdc1
# mount -v -t vfat /dev/sdc1 /mnt/gentoo/boot
# mkdir /mnt/gentoo/boot/EFI

The directory name could be upper case 'EFI' or lower case 'efi'. Currently, 'EFI' works fine.

stage tarbar

# cd /mnt/gentoo
# tar xvjpf /path/to/stage3-amd64-20150319.tar.bz2

Chroot

# vgchange -a y vg, VG name can be ommited.
# cryptsetup luksOpen /dev/mapper/vg-crypt cryptroot, you could use key-file to decrypt vg-crypt as well.
# vgchange -a y vgcryptvg
# mount -v -t ext4 /dev/mapper/cryptvg-root /mnt/gentoo
# mount -v -t ext4 /dev/mapper/cryptvg-home /mnt/gentoo/home
# mount -v -t vfat /dev/sdc1 /mnt/gentoo/boot

If you return to LiveDVD and try to chroot again, please execute the above six commands first. Then follow:

# cp -v -L /etc/resolv.conf /mnt/gentoo/etc/
# mount -v -t proc proc /mnt/gentoo/proc2
# mount -v --rbind /sys /mnt/gentoo/sys
# mount -v --rbind /dev /mnt/gentoo/dev
# mount -v --make-rslave /mnt/gentoo/sys
# mount -v --make-rslave /mnt/gentoo/dev
# chroot /mnt/gentoo /bin/bash
# source /etc/profile && export PS1="(chroot) $PS1"

Installation

Refer to Gentoo Installation and gentoo over lvm luks.

Try to restore some config backup files like 'make.conf', 'fstab', 'wpa_supplicant.conf', '.bashrc' 'repos.conf/{gentoo.conf, layman.conf, local.conf} etc.
Of the most important, the fundamental difference is 'Grub2' and 'genkernel initramfs'.

Follow steps below.

Difference

initramfs

# echo "sys-kernel/genkernel cryptsetup" > /etc/portage/package.use/genkernel
# emerge -av genkernel
# genkernel --lvm --luks --gpg --busybox --install initramfs

Don't worry about –gpg problem occured in LiveDVD above. genkernell will compile GnuPG 1 instead of 2.

Grub2

# grub-install --target=x86_64-efi --efi-directory=/boot --boot-directory=/boot --bootloader-id=Grub2 --removable --modules=part\_gpt

--bootloader-id=Grub2 and /dev/sdc (USB stick) might be optional.
--efi-directory specifies the mountpoint of the ESP (i.e. the USB sdc1 is mounted at /boot). It replaces --root-directory which is deprecated.
No need to append /dev/sdc since --efi-directory= is enough. The INSTALL_DEVICE parameter of grub-install is mainly for BIOS boot.

Refer to EFI boot with GRUB2 on amd64, dual boot with Windows7 x64 and grub2 zh-CN.

Kernel and init arguments

# */etc/default/grub*:

GRUB_CMDLINE_LINUX="crypt_root=UUID='of /dev/mapper/vg-crypt' dolvm root=UUID='of /dev/mapper/cryptvg-root' rootfstype=ext4 root_keydev=PARTUUID='of boot and EFI shared partition' root_key=/relative/path/to/luks-gnupg-key-file"

crypt_root: the UUID of the partition which is encrypted by DM-crypt LUKS. In our case, this is LVM volume /dev/mapper/vg-crypt or /dev/vg/crypt.
dolvm: activate LVM volumes on bootup. This needs support from LVM support in initramfs.
root: the real / mount point for Gentoo. In our case, it is /dev/mapper/cryptvg-root or /dev/cryptvg/root.
rootfstype: Gentoo root filesystem.
root_keydev: the device where DM-crypt LUKS key-file is stored. In our case, it is boot and EFI partition on USB stick. Always prefer PARTUUID over UUID on a GPT disk.
root_key: the path to DM-crypt LUKS key-file. The value should be relative path to root_keydev mount point.
You can use device file name or use UUID instead for those arguments. It's free choince.
The 2nd reference adds a parameter 'target=cryptroot' whose usage is unclear. Don't try this if not sure.
Refer to man page of genkernel for those parameters. They can be set to device name or the device UUID/PARTUUID.

lvmetad service

After getting into new Gentoo system, grub-mkconfig might complain about lvmetad issue which does NO harm. If you really want to get rid of the warning, just run rc-service lvmetad start before grub-mkconfig and remember to stop afterwards.

# grub-mkconfig -o /boot/grub/grub.cfg

In chroot, grub-mkconfig might fail to probe (by calling os-prober) Windows on HDD. When getting into Gentoo, execute it again to make up.

Notes

In our scheme, boot and EFI shared partition is not encrypted. If it is encrypted by LUKS too, we need more configurations. Read Grub2 advanced storage.

Get out of chroot

exit
umount -lv /mnt/gentoo/home
umount -lv /mnt/gentoo/boot
umount -lv /mnt/gentoo/dev{/shm,/pts,}
umount -lv /mnt/gentoo{/proc,/sys,}
vgchange -a n cryptvg

Should give the VG name.
cryptsetup luksClose cryptroot
vgchange -a n vg
reboot

boot & EFI image backup/restore

dd

dd copies bit by bit, free space included.
Be careful on if and of argument.

# dd if=/dev/sdb2 | xz > boot-image-backup.xz
# xzcat image-file.xz | dd of=/dev/sdb2, restore from backup

tar

# tar -cvjpf /media/Misc/boot-image-backup.tar.bz2 /boot
# tar -xvjpf /media/Misc/boot-image-backup.tar.bz2 -C /boot

j can be replaced with z or Jto create .gz or .xz backup.
Avoid `/' after boot, namely /boot instead of /boot/.

Operations to USB sdc1

From the installation process, we find for sdc1, we only:

Copy the LUKS key there;
grub-install command;
grub-mkconfig;
Mount sdc1 as /boot in /etc/fstab

If data ruined in sdc1 or anything else undesirable happend, just repeat step 1-4 as long as LUKS key-file or fall-back passphrase exist.

Hide in Windows

The current USB is 1 GiB in size which only part of is needed for boot and EFI. So we could make use of the remaining around 750MiB for usual USB data storage.

We create the shared partition /dev/sdc1 formated as FAT32. However, what if this USB is inserted into Windows? Enverything in /dev/sdc1 will be exposed as normal USB stick, the luks-gnupg-key included.

Hence we would like to achieve:

Make use of remaining 750MiB capacity.
Hide the shared partition from Windows.

For 1, we just need to create a new partition on the remaining free space. For 2, please note that windows WILL NOT SHOW BOTH PARTITIONS OF USB at the same time. One partition will be hidden and another will be visible. Technically, Windows allocates a drive letter to only single partition on a USB drive. However on linux operating system both partitions will be visible without any problem.

So a better idea is to use sdc1 as the normal data storage partition, while sdc2 for the shared partition. It would be:

Number  Start   End     Size   File system  Name             Flags
 1      1049kB  752MB   751MB  ntfs         USB stick
 2      752MB   1002MB  251MB  fat32        Gentoo boot EFI  boot, esp

sdc1 now is formated as ntfs. If it was formated as mkfs.vfat (fat32), then UEFI firmware would NOT recognize sdc2 as the shared partition for system boot.

At real practice, you'd best not use sdc1 for data storage just in case for operation mistake to ruin sdc2.

Refer to how-to-create-a-separate-hidden-boot-partition-on-usb/ and hide from windows.

References

http://www.ms.unimelb.edu.au/~trice/linux/tricks/luksresize/
http://manual.aptosid.com/en/hd-install-crypt-en.htm
dm-crypt/Encrypting an entire system

Notes

deselect=y notifyd? what is the point? weechat
noatime in fstab
cjktty patch
remember to: libtool finsih
list of files need backup
os-probe will find Windows In Gentoo.

TeXLive Gentoo

2015-08-29T00:00:00+00:00

ABCs
Installation
1. 2015 to 2016
2. TeXLive packages
Fontconfig
1. Enable TeXLive fonts
Chinese

ABCs

Before anything else, read 2 Overview of TEX Live
Read LaTeX on TeX engine and format.

Installation

root@tux / # echo "app-text/texlive xetex cjk l10n_zh science publishers" > /etc/portage/package.use/texlive
root@tux / # emerge -avt app-text/texlive
root@tux / # emerge -avt font-adobe-100dpi font-adobe-75dpi

xetex is compatible with Fontconfig.

XeTeX includes xeCJK macro package which invokes XeTeX engin to compile Chinese TeX files.
(opt) cjk draws in CJK (dev-texlive/texlive-langcjk and dev-tex/cjk-latex)

The old CJK macro script requires more user involvement.
l10n_zh (dev-texlive/texlive-langchinese)

CTeX extends and depends on CJK, thus making Chinese TeX compiling much easier.
USE flags like extra depend on a world of packages many of which are unnecessary. If you are sure, just install what we need on the fly.

Specially, publishers and science USE is useful when you write conference papers like IEEE, ACM etc.
Many TeXLive packages depends on 100dpi and 75dpi bitmap fonts.
TeXLive layout /etc/texmf/web2c/texmf.cnf, /usr/share/texmf-dist/web2c/texmf.cnf and /etc/texmf/texmf.d/05searchpaths.cnf.

2015 to 2016

root@tux / # emerge -avtC --deselect=n $(qlist -IC texlive)
root@tux / # emerge -av1 app-text/texlive

If you are upgrading TeXLive, follow Tex Live Migration Guide and Upgrading TeXLive first.

TeXLive packages

app-text/texlive defines installtion configuration while app-text/texlive-core, dev-texlive/texlive-basic, dev-libs/kpathsea and web2c thereof consists of real sources. Separate TeXLive packages (i.e. dev-tex/biblatex, dev-tex/cjk-latex etc.) will go to TEXMFSITE (/usr/share/texmf-site) which is Gentoo-specific directory. This allows for more unitary upgrades or security fixes.

Use app-portage/pfl to search online repository locating specific package files.

We can also manually put TeXLive packages into TEXMFHOME or TEXMFLOCAL without portage management. By convention, we put personal macro files or packages into TEXMFHOME.

For example, for biblatex (superior to natbib) support:

root@tux / # emerge -av1 dev-tex/biblatex-3.7-r1
root@tux / # emerge -avt dev-tex/biber-2.7

Versions <dev-tex/biblatex-3.7-r1 would block dev-texlive/texlive-plaingeneric-2017. Make sure sub versions (*.5, *.7, *.10 etc.) of biblatex and biber should match.

Since verion over 2.0, biblatex package use biber (superior to bibtex) as te default backend:

root@tux / # emerge -avt dev-tex/biber

We could change it to TeXLive default bibtex in TeX source file. Read bibtex-vs-biber-and-biblatex-vs-natbib on bibliography.

Fontconfig

Though CTeX makes use of Fontconfig fonts directly, we need to make sure the fonts name are correctly resolved between Fontconfig and TeX. Check what you have:

$ fc-list :lang=zh-cn | sort

Edit /usr/share/texmf-dist/tex/latex/ctex/fontset/ctex-xecjk-winfonts.def:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
% ctex-xecjk-winfonts.def: Windows 的 xeCJK 字体设置，默认为六种中易字体
% vim:ft=tex

\setCJKmainfont[BoldFont={NotoSansHans-Bold},ItalicFont={AdobeKaitiStd-Regular}]
  {AdobeFangsongStd-Regular}
\setCJKsansfont{NotoSansHans-DemiLight}
\setCJKmonofont{AdobeSongStd-Light}

\setCJKfamilyfont{zhsong}{AdobeSongStd-Light}
\setCJKfamilyfont{zhhei}{NotoSansHans-Bold}
\setCJKfamilyfont{zhkai}{AdobeKaitiStd-Regular}
\setCJKfamilyfont{zhfs}{AdobeFangsongStd-Regular}
% \setCJKfamilyfont{zhli}{LiSu}
% \setCJKfamilyfont{zhyou}{YouYuan}

\newcommand*{\songti}{\CJKfamily{zhsong}} % 宋体
\newcommand*{\heiti}{\CJKfamily{zhhei}}   % 黑体
\newcommand*{\kaishu}{\CJKfamily{zhkai}}  % 楷书
\newcommand*{\fangsong}{\CJKfamily{zhfs}} % 仿宋
% \newcommand*{\lishu}{\CJKfamily{zhli}}    % 隶书
% \newcommand*{\youyuan}{\CJKfamily{zhyou}} % 幼圆

\endinput

Remove extra square braces. For setCJKmonofont, just use a normal 宋体 since all Chinese characters are monospace (single and same width).

XeTeX can resolve Fontconfig fonts by font file name apart from font (family) name. We should set OSFONTDIR varaible then. Details refer to TeXLive Ubuntu.

Enable TeXLive fonts

There are many free fonts shipped with TeXLive. For XeTeX, they must be enabled in Fontconfig first:

# eselect fontconfig list
# eselect fontconfig enable 09-texlive.conf

Chinese

CTeX by default requires XeTeX (xelatex binary) egnine.
Current stable TeXLive-2014 still requires ctex-xecjk-winfonts.def. Update font names there as above.
Due to a bug of xdvipdfmx, "Noto Sans S Chinese" (思源黑体) is garbled. Update to TeXLive-2015 or apply the patch.

iOS Linux

2015-08-24T00:00:00+00:00

How to mount iOS and transfer files to Iphone?

Installation

user@tux ~ $ echo "gnome-base/gvfs ios fuse" >> /etc/portage/package.use/gvfs, *fuse* is optional
user@tux ~ $ emerge -av1 gvfs
user@tux ~ $ emerge -avt usbmuxd
user@tux ~ $ usbmuxd -f -v (opt), automatically invoked when iOS device plugged in

Restart X and login as a normal user account.
Thunar will remind you to unlock passcode NO.31 in Iphone. Iphone might ask you to trust the computer. Just do it.
If you cannot find Documents on XXX's iPhone in Thunar, restart iPhone.
If fuse USE is enabled, documents can be accessed under /var/run/user/1000/gvfs/ or $HOME/.gvfs/.
Refs:
1. https://wiki.gentoo.org/wiki/Apple_iPod,_iPad,_iPhone
2. https://wiki.archlinux.org/index.php/IPod

VirtualBox

2015-08-21T00:00:00+00:00

Terminology
Installation
Schema
Windows XP 32-bit VM
Windows Embedded Standard 7 x86 VM
Android-x86 VM
Arch Linux x86_64 VM
CentOS 8.2 Amd64
References

This post indroduces installing VirtualBox in Gentoo host.

Terminology

Host OS

Operating system where VMM runs.
Guest OS

Operating system runs over VirtualBox.
VirtualBox

A virtual machine manager (VMM) owned by Orcale, also named as Base package.

Once installed, it will insert VirtualBox kernel modules into the host OS, like vboxdrv.
Oracle VM VirtualBpx Extension Packs

A proprietary bundle of packages that extends the functionality of the Base Package, like USB 3.0, VirtualBox Remote Desktop Protocol (VRDP), webcam etc.

Extension Packs is optional. However, if you choose to install, then make sure it has the same version as the Base Package.

Similar to the Base, Package, Extension Packs also presents as kernel modules on host OS, like vboxnetadp and vboxnetflt.
Oracle VM VirtualBox GuestAdditions

Package to be installed inside the guest OS that optimizes performance and usability, like mouse pointer integration, shared folders, shared clipboard, seamless windows, time synchroniation with host OS etc.

GuestAdditions also serve as kernel modules but in guest OS, like vboxsf, vboxvideo, and vboxguest.
Make sure relevant versions match.
1. VirtualBox. Versions of Base Package, Extension Packs and GuestAdditions must match.
2. Guest OS. Kernel modules are built against the current kernel. When VirtualBox is upgraded, kernel modules on host OS is re-built automatically. However, if guest OS kernel is changed, we should re-build the guest kernel modules manually (i.e. GuestAdditions).

Installation

VirtualBox depends on QT as GUI. But I would like a XFCE4 desktop without any QT packages. QT related USE flags are disabled in Gentoo make.conf.

So I will emerge VirtualBox without GUI. To manage VMs, resort to CLI.
USEs
```
root@tux / # echo "app-emulation/virtualbox headless additions extensions" > /etc/portage/package.use/virtualbox
```
1. headless builds VirtualBox without any graphic frontend.
2. attitions for Guest Addtitions.
  
  Since VirtualBox 5, this USE is removed. Emerge app-emulation/virtualbox-additions manually.
3. extension for Extension Packs.
Build
```
root@tux / # emerge -avt app-emulation/virtualbox
root@tux / # emerge -avt app-emulation/virtualbox-additions (>=5)
root@tux / # emerge -avt app-emulation/virtualbox-extpack-oracle (>=5)
```
It reminds accepting PUEL licence. Refer to VirtualBox wiki.

If fail to emerge VirtualBox, then probably you should bump to a newer version. Lastest Linux kernel usually requires lastest VirtualBox.
Group - vboxusers
```
root@tux / # gpasswd -a <username> vboxusers
```
Add current username to vboxusers group. Refer to in newgrp in Wireshark post.
Kernel modules
```
root@tux / # modinfo vboxdrv vboxnetadp vboxnetflt vboxpci
root@tux / # modprobe -v vboxdrv vboxnetadp vboxnetflt vboxpci
```
1. If you launch VirtualBox frequently, add these modules to /etc/conf.d/modules for automatic loading on boot.
2. vboxdrv is the core module for host OS and must always be present. vboxnetadp and vboxnetflt are required for any VM networking beyond the default NAT mode (i.e. host-only, bridged etc.).
When booting into a new Linux kernel (i.e kernel upgrading), you could no longer load modules like vobxdrv.
```
# modprobe vboxdrv
modprobe: FATAL: Module vboxdrv not found.
```
Read the first reference on Kernel driver not installed section. The solution is to rebuild VirtualBox external modules:
```
# emerge -avt1 app-emulation/virtualbox-modules
# -or-
# emerge -avt1 @module-rebuild
```
More details in Upgrade kernel to unstable 4.0.0.
Up to now everyting related to VirtualBox VMM is prepared on host OS. Next is to create VM through CLI.

On macOS, use brew cask to install virtualbox and virtualbox-extension-pack.

Remove quotes around sysmbol ~ and symbol $.

# /etc/udev/rules.d

# Correct /dev/{vboxdrv,vboxdrvu,vboxnetctl} permissions to 0600
KERNEL=="vboxdrv", NAME="vboxdrv", OWNER="root", GROUP="vboxusers", MODE="0660"
KERNEL=="vboxdrvu", NAME="vboxdrvu", OWNER="root", GROUP="vboxusers", MODE="0660"
KERNEL=="vboxnetctl", NAME="vboxnetctl", OWNER="root",GROUP="vboxusers", MODE="0660"

To load the new rule, execute udevadm trigger.

Schema

Avoid 64-bit guest OS. 64-bit OS occupies one third more resources (disk, memory) than the 32-bit version. What was worse, it seems VBoxGuestAddtions-amd64.exe makes no difference on the guest (double mouse, weird resolution etc.).
The less snapshots we create, the better is the performance.
Put VDI file on host's home partition for better performance.

Windows XP 32-bit VM

VM won't even start without QT GUI support. But VRDP helps! VRDP is a replacement of QT GUI!

First create VM with VirtualBox CLI;
Enable VRDP support for VM;
Connect to headless VM by VRDP client FreeRDP.

VBoxManage createvm:

user@tux ~ $ VBoxManage list ostypes
user@tux ~ $ VBoxManage createvm --name WinXP32 --ostype WindowsXP --register --basefolder ~/Documents/VirtualBox/Machines
user@tux ~ $ VboxManage list vms
user@tux ~ $ VBoxManage registervm ~/Documents/VirtualBox/Machines/WinXP32/WinXP32.vbox (opt); vboxmanage list vms
user@tux ~ $ VBoxManage modifyvm WinXP32 --memory 384 --acpi on --nic1 nat --nictype1 Am79C973 --audio alsa --audiocontroller ac97 --usb on --usbehci on --vrde on --vrdeaddress 127.0.0.1 --vrdeport 5001,5010-5012 --clipboard bidirectional
user@tux ~ $ VBoxManage createmedium --filename ~/Documents/VirtualBox/Machines/WinXP32/WinXP32.vdi --size 5000 (omitted)
user@tux ~ $ VBoxManage storagectl WinXP32 --name "IDE Controller" --add ide --controller PIIX4
user@tux ~ $ VBoxManage storageattach WinXP32 --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium ~/Documents/VirtualBox/Machines/WinXP32/WinXP32.vdi
user@tux ~ $ VBoxManage storageattach WinXP32 --storagectl "IDE Controller" --port 0 --device 1 --type hdd --medium /path/to/ISO (omitted)
user@tux ~ $ VBoxManage storageattach WinXP32 --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium /usr/share/virtualbox/VBoxGuestAdditions.iso (or --device 1 --medium additions)
user@tux ~ $ VBoxManage sharedfolder add WinXP32 --name WLshare --hostpath /media/Misc/VMs/WLshare

Here, I re-use an existing .vdi instead of guest installation ISO, thus avoiding guest OS installing process. Apparently, createmedium prodecure is omitted.

ostypes

List supported OSes. I am going to install Windows XP 32-bit. The value is WindowsXP.
createvm
- --name should be enclosed by double quotes in case of white spaces.
- --register to register the VM instantly, or run VBoxManage registervm afterwards.
- --basefoler can be used to specify this VM's files location (i.e. set to NTFS partition). If not set, it will default to ${HOME}/.VirtualBox/Machines which is different from VirtualBox confiuration file.
registervm

Bug: --register of createvm failed to register the VM though it reports negtive success. Possible solutions:
1. Specifically add --basefolder to createvm –register.
2. Pass full path of .vbox file to registervm on a separate shell command.
modifyvm
- --nictype1 Am79C973 sets virtual Ethernet hardware to AMD PCNet FAST III (Am79C973). VirtualBox 5 now use Intel PRO/1000 T Server (82543GC) as default, which requires extra drivers installed.
- --audio alsa could be pulse instead.
- --vrde on is to enable VRDP support thus I can remotely connect to the VM by RDP client.
- ` –usb on –usbehci on enables USB 1.0 and 2.0. To enable 3.0, use –usbxhci on`.
- --vrdeaddress set to loopback address. If unset, it binds to all host network interfaces accepting both IPv4 and IPv6. Refer to 127.0.0.1 vs 0.0.0.0.
- --draganddrop option is useful if you need it. However, it is vunerable to security issue.
storagectl

Set disk controller for VM. Don't use SATA related controller for WindowsXP.
storageattach VDI

WinXP32.vdi is copied from some guy in QQ group, getting rid of installing VM OS from scratch.
storageattach

VBoxGuestAddiontions.iso supports file sharing, mouse switch etc between host and guest. WE will install this toolbox after entering WindowsXP VM.
sharedfolder

Create a folder for share on host and mount it on guest.

After entering VM, open Windows Explorer and look for it under "My Networking Places", "Entire Network", "VirtualBox Shared Folders", "\\Vboxsvr".

By right-clicking on a shared folder and selecting "Map network drive" from the menu that pops up, you can assign a drive letter to that shared folder.

If you don't assign a drive letter, each time to access the shared, we have to find it under "\\Vboxsvr".

If you cannot locate the shared folder under Windows Explorer/Network, check if Network Discovery is turned on in Control Panel (default for Home networking type). You can also try CMD:
```
net use z: \\vboxsvr\WLshare
# or
net use z: \\vboxsrv\WLshare
```
Up to now, the VM is prepared! We need a RDP client.

Start VM with VRDE

VirtualBox Remote Display Protocol (VRDP) is a backwards-compatible extension to Microsoft's Remote Desktop Protocol (RDP). VirtualBox Remote Desktop Extension (VRDE) is an Oracle VRDP implementation. Apart from TCP/Ports and TCP/Address, we have:

--vrdemulticon on allows multiple simultaneous connections to the same VM, without which we can resort to --vrdereusecon on that can cut off the current connection for the new one.
--vrdeauthtype external requires username and password of the host before connection.

Here is FreeRDP setup details:

Up to now, the VM is created! But VirtualBox does not have QT GUI. We need RDP client, i.e. FreeRDP.
```
root@tux / # emerge -avt net-misc/freerdp
```
startvm
```
user@tux ~ $ VBoxManage startvm "WinXP32" --type headless
# or
user@tux ~ $ VBoxHeadless --startvm WinXP32
```
VBoxHeadless is a tool the launch VM as a server mode without GUI. If you buy a VPS, the OS mostly runs in a similar mode (maybe VNC browser). Now the VM is started, but not showing up! We need to use FreeRDP to get X.

On MacOS host, the GUI mode may report error:

You must specify a machine to start, using the command line.

Please disable Audio is guest VM setting. You may also want to read VirtualBox crashes after upgrading to macOS Catalina.
FreeRDP

The RDP TCP port is set to an optional list 5001,5010-5012 (default 3389).
```
$ xfreerdp +clipboard /w:1024 /h:576 /v:127.0.0.1:5001
```
1. The server IP is vrdeaddress, NOT IP of guest. Since I connect to VM locally, so it 127.0.0.1. xfreerdp can add many other parameters, like window size, audio, video etc.
2. 'Ctrl + Alt + Enter' to toggle full screen.
3. You might found there are two mouse pointers, one for host while another for guest. Also the screen resolution is not correctly set. Install VBoxGuestAdditions we've attached the Vm. Open file explorer, installer is located in partition (D:) VirtualBox GuestAdditions, VBoxWindowsAdditions.exe. Disable Direct3D component on a low end VM.
Shutdown
```
 $ VBoxManage controlvm WinXP32 savestate/acpipowerbutton/poweroff/pause
```
1. pause: temporarily puts a virtual machine on hold, without changing its state for good. The VM window will be painted in gray to indicate that the VM is currently paused.
2. poweroff: has the same effect on a virtual machine as pulling the power cable on a real computer. Again, the state of the VM is not saved beforehand, and data may be lost. (This is equivalent to selecting the "Close" item in the "Machine" menu of the GUI or pressing the window's close button, and then selecting "Power off the machine" in the dialog.)
3. savestate: saves the current state of the VM to disk and then stop the VM. (This is equivalent to selecting the "Close" item in the "Machine" menu of the GUI or pressing the window's close button, and then selecting "Save the machine state" in the dialog.) We can think savestate as suspend to disk (hibernate).
4. acpipowerbuttion: simulates long press the PC's power buttion to shutdown directly without saving.
Which to use, depens on:
1. If jobs in VM is not yet finished (like editing a file), use savestate;
2. If every jobs completed, use acpipoweroff (as long as ACPI is enabled for VM) or poweroff;
3. If still want to leave the VM running remotely, just close the RDP window.

Remove guest addtions ISO

 user@tux ~ $ VBoxManage storageattach WinXP32 --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium emptydrive / none

Detach ISO file after VBoxGuestAddtions is installed.

Manage VMs on Command Line

Load kernel modules.

#!/bin/bash
for m in vbox{drv,netadp,netflt,pci}; do modprobe -v $m; done
echo 'VirtualBox modules loaded!'

Startup script.

#!/usr/bin/env bash

PS3='Please enter your choice on VM WinXP32: '
options=("startvm" "rdp" "poweroff" "acpipowerbutton" "savestate" "quit")
select opt in "${options[@]}"
do
    case $opt in
    "startvm")
        echo "you choose to launch WinXP32"
        VBoxHeadless --startvm WinXP32
        break
        ;;
    "rdp")
        echo "you choose to connect WinXP32"
        xfreerdp +clipboard /sound /f /v:127.0.0.1:5001
        break
        ;;
    "acpipowerbutton")
        echo "you chose to acpipowerbuttion WinXP32"
        VBoxManage controlvm WinXP32 poweroff
        break
        ;;
    "savestate")
        echo "you chose to save WinXP32state"
        VBoxManage controlvm WinXP32 savestate
        break
        ;;
    "poweroff")
        echo "you chose to poweroff WinXP32"
        VBoxManage controlvm WinXP32 poweroff
        break
        ;;
    "quit")
        break
        ;;
    *) echo 'invalid option'
       ;;
    esac

Guest OS Configurations

After getting into the Windows XP system, we can adjust some system configurations to minimize resource consumption:

For English OS, then set non-Unicode language to Chinese and set the Regional Options values.
Disable soundman etc. with msconfig.
Set Wireless Zero Configuration, Windows Audio, Windows Themes etc. services to Disabled or Manual. Refer to Windows XP 终极优化.
If installed '迅雷', then set XLServicePlatform to Manual.
My Computer, Properties, Advanced, Performance, Settings, Adjust for best performance.

USB Supoort

Attach plugged in USB device to VM. First set the USB version to 1.0, 2.0 or 3.0 for VM.

user@tux ~ $ vboxmanage list usbhost
user@tux ~ $ vboxmanage controlvm WinXP32 usbattach USB-UUID
user@tux ~ $ vboxmanage list usbhost
user@tux ~ $ vboxmanage controlvm WinXP32 usbdetach USB-UUID

Find the USB device UUID by checking the Manufacture and Product field. Make sure the Current State field is Available.
Attach the USB to VM.
The Current State field becomes Captured (by VM).
Detach from VM.
Get UUID from vboxmanage list usbhost instead of blkid.

We can also make this attachment permanent by creating a usbfilter.

Host-only networking

Take Android-x86 for example, the very first step is to create a hostonlyif interface, as follows:

user@tux ~ $ VBoxManage hostonlyif create/remove vboxnet0
user@tux ~ $ ip link; VBoxManage list hostonlyifs
user@tux ~ $ VBoxManage hostonlyif ipconfig vboxnet0 --ip 192.168.56.1 # '--dhcp' is not supported currently
user@tux ~ $ VBoxManage modifyvm Android51 --nic2 hostonly --hostonlyadapter2 vboxnet0
user@tux ~ $ VBoxManage dhcpserver add/modify --ifname vboxnet0 (--netname HostInterfaceNetworking-vboxnet0) --ip 192.168.56.1 --netmask 255.255.255.0 --lowerip 192.168.56.100 --upperip 192.168.56.110 --enable
user@tux ~ $ VBoxManage list dhcpservers
user@tux ~ $ ip address
# boot Android51
user@tux ~ # netcfg eth1 dhcp

Check iptables and/or firewall.

user@tux ~ # iptables -I INPUT 4 -i vboxnet0 -s 192.168.56.0/24 -j ACCEPT
user@tux ~ # iptables -I OUTPUT 4 -o vboxnet0 -s 192.168.56.0/24 -j ACCEPT

Apart from NAT/bridged networking, we can use iptables redirect to let vboxnet0 traffic go outside.

Especially, guest VM can share host's proxy as long as it's listening on local network (i.e. 0.0.0.0). You may want to prohibit LAN devices connection to proxy.

user@tux ~ # iptables -I INPUT 4 -i vboxnet0 -d 192.168.56.0/24 -p tcp -m tcp --dport 1080 -j DROP
user@tux ~ # iptables -I INPUT 4 -i vboxnet0 -d 192.168.56.0/24 -p udp -m udp --dport 1080 -j DROP (opt)

Share host proxy

Alternatively, iptables redirect vboxnet0 traffic to 127.0.0.1 like:

user@tux ~ # iptables -t nat -A PREROUTING [ -s 192.168.56.101 ] -d 192.168.56.100 -i vboxnet0 -p tcp -m tcp --dports 1080,9050 -j DNAT --to-destination 127.0.0.1
user@tux ~ # sysctl -w net.ipv4.conf.vboxnet0.route_localnet=0 (runtime)
# or 
user@tux ~ # sysctl -p /etc/sysctl.d/15-route_localnet.conf
#
net.ipv4.conf.vboxnet0.route_localnet = 1 (better)
net.ipv4.conf.all.route_localnet = 1 (dangerous)

By default, kernel refuses to route from source or to destination loopback addresses (i.e. 127.0.0.1). vboxnet0 interface would not be created before VirtualBox launches. So 15-route_localnet.conf does not apply accross boot.

A more general method is enabling net.ipv4.ip_forward:

user@tux ~ # sysctl -w net.ipv4.ip_forward=1
user@tux ~ # iptables -A FORWARD -i vboxnet0 -o vboxnet0 -j ACCEPT

On MacOS, the vboxnet0 interface is not created automatically, we need to create manually like that of Android-x86 above.
IF the host OS or the guest OS is Windows system, then Host-only refuses connection.

Either turn on network discovery' or use security policy *secpol.msc to set Unidentified network as Private.

IDE Drive Port Limit

Windows XP --port 2 --device 0 refuses to attach extra ISO:

VBoxManage: error: No drive attached to device slot 0 on port 2 of controller 'IDE Controller'

Windows XP guest uses IDE PIIX4 disk drive that supports at most 2 ports. Each port supports two devices , namely (0, 0), (0, 1), (1, 0) and (1, 1). If you have a strong reason to use port 2, then create SATA disk drive like Win 7 guest.

Especially, each port of SATA disk drive only supports one device, namely (0, 0), (1, 0), (2, 0) etc.

Delete VM

user@tux ~ $ vboxmanage list vms
user@tux ~ $ vboxmanage unregistervm WinXP32 --delete

Amost everything related to the VM is deleted, especially the virtual disk image file (.vdi), snapshots, saved state files, VM xml file etc.

Windows Embedded Standard 7 x86 VM

~ $ VBoxManage list ostypes (Windows7)
~ $ VBoxManage createvm --name WES7x86 --ostype Windows7 --register --basefolder ~/Documents/VirtualBox/Machines
~ $ VBoxManage registervm ~/Documents/VirtualBox/Machines/WES7x86/WES7x86.vbox (opt)
~ $ VBoxManage modifyvm WES7x86 --memory 700 --audio alsa --audiocontroller hda --acpi on --vrde on --vrdeproperty "TCP/Ports=5001,5010-5012" --vrdeproperty "TCP/Address=127.0.0.1" --clipboard bidirectional (--draganddrop bidirectional --usb on --usbehci on, --nic1 bridged --bridgeadapter1 wlp3s0, --nic1 nat, --boot1 dvd --boot2 disk --boot3 none --boot4 none)
~ $ VBoxManage createmedium --filename ~/Documents/VirtualBox/Machines/WES7x86/WES7x86.vdi --size 7000 (createhd)
~ $ VBoxManage storagectl WES7x86 --name "SATA Controller" --add sata --controller IntelAHCI --portcount 3 (SATA and Intel AHCI)
~ $ VBoxManage storageattach WES7x86 --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium ~/Documents/VirtualBox/Machines/WES7x86/WES7x86.vdi
~ $ VBoxManage storageattach WES7x86 --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium /media/Misc/WLshare/en_windows_embedded_standard_7_runtime_x86_dvd_521803.iso
~ $ VBoxManage storageattach WES7x86 --storagectl "SATA Controller" --port 2 --device 0 --type dvddrive --medium /usr/share/virtualbox/VBoxGuestAdditions.iso
~ $ VBoxManage sharedfolder add WES7x86 --name WLshare --hostpath /media/Misc/WLshare
# Installation processing ...
~ $ VBoxManage storageattach WES7x86 --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium emptydrive
~ $ VBoxManage storageattach WES7x86 --storagectl "SATA Controller" --port 2 --device 0 --type dvddrive --medium none

For details on choosing template and components during installation, read the Windows post.

Since the ISO image is till attached to SATA Controller, booting will be directed to installation process again. Either update VM boot order or F12 at early phase.

There is no sound in WES7x86 guest. Please change --audiocontroller to Intel hda instead of default ac97.

Android-x86 VM

~ $ VBoxManage createvm --name Android51 --ostype Linux26 --register --basefolder /media/Misc/VirtualBox/Machines
~ $ VBoxManage modifyvm Android51 --memory 700 --acpi on --mouse usbtablet --usb on --usbehci on --vrde on --vrdeproperty "TCP/Ports=5001,5010-5012" --vrdeproperty "TCP/Address=127.0.0.1" --clipboard bidirectional
~ $ VBoxManage createmedium --filename /media/Misc/VirtualBox/Machines/Android51/Android51.vdi --size 4000
~ $ VBoxManage storagectl Android51  --name "IDE Controller" --add ide --controller PIIX4
~ $ VBoxManage storageattach Android51 --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium /media/Misc/VirtualBox/Machines/Android51/Android51.vdi
~ $ VBoxManage storageattach Android51 --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium ~/Downloads/cm-x86-13.0-r1.iso
~ $ VBoxManage list vms

Installation - Install Android-x86 to harddisk.
Create/Modify partitions.
Do you want to use GPT? No!
New - Primary - Bootable - Write - yes - Quit.
sda1 unkown VBOX HARDDISK.
ext3/ext4.
Format sda1 to ext4? Yes.
Do you want to install boot loader GRUB? Yes.
Do you want to install /system directory as read-write? Yes.
Run Android-x86.

If you choose GPT at 3rd step, then install EFI and GRUB2 instead of GRUB.

First boot takes several minutes to initialize system preparation.

Arch Linux x86_64 VM

~ $ VBoxManage list ostypes (ArchLinux_64)
~ $ VBoxManage createvm --name archlinux_64 --ostype ArchLinux_64 --register --basefolder /media/Misc/VirtualBox/Machines
~ $ VBoxManage modifyvm archlinux_64 --memory 512 --acpi on --vrde on --vrdeproperty "TCP/Ports=5001,5010-5012" --vrdeproperty "TCP/Address=127.0.0.1" --clipboard bidirectional
~ $ VBoxManage createmedium --filename /media/Misc/VirtualBox/Machines/archlinux_64/archlinux_64.vdi --size 4000
~ $ VBoxManage modifymedium /media/Misc/VirtualBox/Machines/archlinux_64/archlinux_64.vdi --resize 5000
~ $ VBoxManage storagectl archlinux_64 --name "SATA Controller" --add sata --controller IntelAHCI --portcount 3
~ $ VBoxManage storageattach archlinux_64 --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium /media/Misc/VirtualBox/Machines/archlinux_64/archlinux_64.vdi
~ $ VBoxManage storageattach archlinux_64 --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium ~/Downloads/archlinux-2017.11.01-x86_64.iso
~ $ VBoxManage sharedfolder add archlinux_64 --name WLshare --hostpath /media/Misc/WLshare
# starvm
~ $ VBoxManage startvm archlinux_64 --type headless

The VDI file is resized after creation.

Then follow the Arch Linux post.
VBoxGuestAdditions in Linux guest requires extra effors. Details refer to Arch Linux post.

VirtualBox GuestAdditions

VirtualBox GuestAdditions consists of device drivers and system applications that optimize the guest operating system for better performance and usability.

Some Linux guest OSes (i.e. Gentoo, Arch Linux) already come with all or part of the VirtualBox GuestAdditions.

Just install the corresponding package. For example:
```
[root@host ~]# emerge -avt app-emulation/virtualbox-guest-additions # Gentoo
[root@host ~]# pacman -S virtualbox-guest-utils/virtualbox-guest-utils-nox # archlinux
```
This method is always preferred!
Alternatively, we can mount the ISO of GuestAdditions and invoke the relevant installation script manually.

Firstly, make sure development tools like gcc, make, kernel-headers, kernel-devel etc. are present on guest OS. Also confirm the guest OS kernel version matches that of package kernel-headers and package kernel-devel.

Then, we can obtain the ISO from host by vboxmanage storageattach.
```
[root@host ~]# lsblk -f
```
We can also get ISO file from guest OS (Arch Linux) package repository:
```
[root@host ~]# pacman -S virtualbox-guest-iso
[root@host ~]# ls /usr/lib/virtualbox/additions/VBoxGuestAdditions.iso
```
Another method, is to download the ISO from VirtualBox official website. Whatever methods you choose to get the ISO, make sure it has the same version as VirtualBox VMM on host OS.

Once the ISO is obtained, we mount the ISO:
```
[root@host ~]# ll /dev/cdrom
[root@host ~]# mkdir -p /mnt/vbox
[root@host ~]# mount -o loop /dev/cdrom /mnt/vbox
[root@host ~]# ll /mnt/vbox
```
Lastly, install GuestAdditions:
```
[root@host ~]# sh /mnt/vbox/VBoxLinuxAdditions.run
```
Restart guest OS before VirtualBox guest additions take effect.
A real pactice on Arch guest
1. Gentoo host: kernel-4.12.5, VirtualBox 5.1.26.
2. Arch guest: kernel-4.13.12, VirtualBox 5.2.2.
Gentoo is relatively conservative on package rolling update compared to Arch Linux. So installing GuestAdditions directly from Arch repository is beffer.
```
[root@host ~]# pacman -S virtualbox-guest-utils (X Window)
# or
[root@host ~]# pacman -S virtualbox-guest-utils-nox (no X)
```
Enable vboxservice unit service to load vboxguest, vboxsf, and vboxvideo kernel modules.
```
[root@host ~]# systemctl enable vboxservice
```
VBoxClient (or the VBoxClient-all wrapper) is another unit service from the GuestAdditions. It manages clipboard, seamless window display, etc. The associated /etc/xdg/autostart/vboxclient.desktop launches VBoxClient-all on logon. VBoxClient-all script launches VBoxClient as:
```
[user@host ~]$ VBoxClient --clipboard --draganddrop --seamless --display --checkhostversion
```
Check Autostart section above on how to launch VBoxclient alongside with awesome.

VirtualBox sharedfolder

Make sure vboxservice is enabled and started.

Add shared folder:

user@host ~ $ VBoxManage sharedfolder add archlinux --name share_name --hostpath /path/to/host/folder [--automount]

Arch Linux guest can mount the shared folder manually (mount -t vboxsf), automatically (vboxmanage --automount), or by fstab. If --automount provided, then by default, the folder is mounted for root:vboxsf. For other accounts to read and write, do usermod -aG vboxsf <username>.

Alternatively, use fstab to flexibly control mount options:

# /etc/fstab

wlshare        /media/wlshare        vboxsf        nodev,nosuid,noexec,uid=root,gid=vboxsf,rw,iocharset=utf8,dmode=0770,fmode=0660        0 0

wlshare        /media/wlshare        vboxsf        nodev,nosuid,noexec,noauto,uid=root,gid=vboxsf,rw,iocharset=utf8,dmode=0770,fmode=0660,x-systemd.automount        0 0

The noauto option is to avoid service racing on booting. For example, GuestAdditions are not loaded yet while fstab tries to mount it.
The x-systemd.automount will create media-wlshare unit upon systemctl daemon-reload.
Only root is allowed to mount vboxsf device. So the user option is not permissible.
Read Why nodev,nosuid,noexe are important?.

Expand VDI Size

Occasionally, the guest OS warns running out of disk space when you should resize the VM disk and parition thereof.

Firstly, make sure the guest does not have snapshots. If there exist any, delete all of them.
Resizing does not work if the guest VM resides on fix-sized VDI.
Currently, only expansion is supported. You cannot decrease any VDI or partitions.

Here, I will show an example of expanding archlinux_64 VM above. On the host:

user@tux ~ $ VBoxManage list hdds
user@tux ~ $ VBoxManage modifymedium /media/Misc/VirtualBox/Machines/archlinux_64/archlinux_64.vdi --resize 10000
user@tux ~ $ VBoxManage list hdds

The VDI is increasing from 5GiB to 10GiB. Next, we should let the VM known the expansion. Basically, we attach to the VM a live booting media containing parted or gparted tool. GParted live CD and Arch Linux ISO are such examples. On the host:

user@tux ~ $ VBoxManage showvminfo archlinux_64
user@tux ~ $ VBoxManage storageattach archlinux_64 --storagectl "SATA Controller" --port 1 --device 0 --type dvddrive --medium ~/Downloads/archlinux-2018.01.01-x86_64.iso
user@tux ~ $ VBoxManage showvminfo archlinux_64

Please make sure boot the VM on the attached ISO instead of VDI. In the live system, suppose /dev/sda represents VDI:

[root@archiso / #] fdisk/blkid/lsblk/findmnt
[root@archiso / #] parted -a optimal /dev/sda
(parted) help
(parted) print free
The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Fix/Ignore/Cancel?
(parted) Fix
(parted) print free
(parted) resizepart 1 100%

parted detects the VDI expanded and reminds Fix. The key is resizepart (or resize depending on parted version) epxanding (100%) the root partition to include newly added space. That's all!

Some posts write resizepart can be done on guest OS directly as long as swap partition/file is turned off.
For Windows guest, use enclosed disk management to finish the job.

CentOS 8.2 Amd64

This section describes procedures to manage CentOS 8.2 VM:

Start the VM.

Enable the Host-only interface /etc/sysconfig/network-scripts/ifcfg-enp0s8.
Start the VM in headless mode either on command line or by GUI.

SSH into the VM.
Configure repository mirror.

dnf group install 'Development Tools'.
Insert GuestAdditions CD; mount sr0; install GuestAdditions.
1. Add shared folder.
2. Add clipboard.

References

Gentoo rootfs over LVM encrypted in LUKS container.

2015-08-15T00:00:00+00:00

System scheme

'/boot' partition resides on USB stick, say sdc1 partition.
'/boot/efi' is shared among Windows and Gentoo.
rootfs ('/, swap, /home') mountpoints were created op top of LVM group, say vg1.
vg1 LVM group lays over Dm-Crypt, LUKS encrypted sda8*.
Use keyfile to encrypt rootfs. keyfile isteself was encrypted by GnuPG and placed at USB stick, say sdc1 (same as /boot).
The installation process depends mainly on Gentoo LiveCD.

Prepare partitions

Use parted -a optimal /dev/sda to get a disk partition for rootfs. I will use rootfs for '/, swap, /home' mountpoints in this post.

Run the very first command unit s in parted to display disk partitions as sector. When creating partition with s unit, it will be well aligned.

For parted example, refer to gentoo installation and kali usb persistence.

Say '/dev/sda8' is created for rootfs.

Use parted -a optimal /dev/sdc to create '/boot' partition on USB stick '/dev/sdc1'.

'/boot' is NOT encrypted! You can use the whole USB space to hold '/boot', or just spare around 256MB parition.

Then formate the partition as ext2:

mkfs.ext2 -L 'usb boot' /dev/sdc1

cryptsetup keyfile

dm-crypt and LUKS

dm-crypt is a disk encryption system using the kernels crypto API framework and device mapper subsystem.

With dm-crypt, administrators can encrypt entire disks, logical volumes, partitions, but also single files.

The dm-crypt subsystem supports the Linux Unified Key Setup (LUKS) structure, which allows for multiple keys to access the encrypted data, as well as manipulate the keys (such as changing the keys, adding additional passphrases, etc.)

Although dm-crypt supports non-LUKS setups as well, this article will focus on the LUKS functionality mostly due to its flexibility, manageability as well as broad support in the community.

Refer to Dm-crypt and Dm-Crypt LUKS.

Keyfile

We use cryptsetup command from LiveCD to encrypt rootfs partition '/dev/sda8'. We can use passphrase or keyfile as encryption key.

Considering the length, randomness and complexity of encryption requirements a keyfile seems to be the right spot.

We first generate a ramdom keyfile for rootfs encryption. This keyfile is usually put at a secure place, i.e. USB stick.

In this scheme, keyfile and '/boot' share the same USB stick partition.

If attacker gets the USB stick, system is decrypted!! So next, we need to encrypt this keyfile by GnuPG before storing it on USB stick.

# mkdir /mnt/sdc1
# mount /dev/sdc1 /mnt/sdc1
# mkdir /dev/sdc1/key
$ dd if=/dev/urandom bs=8388607 count=1 | gpg --symmetric --cipher-algo AES256 --output ~/Desktop/luks-key.gpg
# cp /path/to/luks-key.gpg /mnt/sdc1/key/

Refer to Preparing the LUKS-LVM Filesystem and Boot USB Key.

You will be asked for passphrase to encrypt keyfile. Don't forget passphrase!

If you run gpg in LiveCD root shell, you might get the following error even though you set 'export GPG_TTY=$(tty)' as the reference mentioned:

gpg: directory `/root/.gnupg' created
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/root/.gnupg/pubring.gpg' created
No protocol specified

(pinentry:10992): Gtk-WARNING **: cannot open display: :0
gpg-agent[10991]: can't connect to the PIN entry module: End of file
gpg-agent[10991]: command get_passphrase failed: No pinentry
gpg: problem with the agent: No pinentry
gpg: error creating passphrase: Operation cancelled
gpg: symmetric encryption of `[stdin]' failed: Operation cancelled

Refer to Genkernel initramfs with LUKS+GPG key and Using gpg in mkinitcpio hook, gpg doesn't ask for a password.

This might be related to >app-crypt/gnupg-2.0 version and pinentry support. To get this issue solved:

Generate keyfile in another workable environment.
Switch to normal user account shell when run gpg command and get he decrypted temporary luks-key file for LiveCD.

Format rootfs using LUKS

$ gpg --decrypt /mnt/sdc1/key/luks-key.gpg > ~/Desktop/luks-key
# cryptsetup --cipher serpent-xts-plain64 --key-size 512 --hash sha512 --key-file /path/to/luks-key luksFormat /dev/sda8

Pay attention to --cipher serpent-xts-plain64 --key-size 512 --hash sha512 which reminds us the related kernel option must be 'Y' NOT 'M'.

Check the formating effect:

# cryptsetup luksDump /dev/sda8

Adding a Fallback Passphrase (Optional Step)

If we lost the gpg passphrase or the keyfile, we lost the entire system data. Besides the keyfile, we add a fallback passphrase.

# cryptsetup --key-file /path/to/luks-key luksAddKey /dev/sda8
# cryptsetup luksDump /dev/sda8

Creating the LVM Structure on Top of LUKS

# cryptsetup --key-file /path/to/luks-key luksOpen /dev/sda8 gentoo
# ls /dev/mapper
# pvcreate /dev/mapper/gentoo
# vgcreate vg1 /dev/mapper/gentoo
# lvcreate --size 4G --name swap vg1
# lvcreate --size 18G --name root vg1
# lvcreate --extents 100%FREE --name home vg1
# pvdisplay
# vgdisplay
# lvdisplay
# vgchange --available y
# lvscan
# ls /dev/mapper

Formatting and Mounting the LVM Logical Volumes (LVs)

# mkswap -L "swap" /dev/mapper/vg1-swap
# mkfs.ext4 -L "root" /dev/mapper/vg1-root
# mkfs.ext4 -m 0 -L "home" /dev/mapper/vg1-home
# swapon -v /dev/mapper/vg1-swap
# mount -v -t ext4 /dev/mapper/vg1-root /mnt/gentoo
# mkdir -v /mnt/gentoo/{home,boot}
# mount -v -t ext4 /dev/mapper/vg1-home /mnt/gentoo/home
# umount -v /dev/sdc1
# mount -v -t ext2 /dev/sdc1 /mnt/gentoo/boot
# mkdir /mnt/gentoo/boot/efi
# mount -v /dev/sda2 /mnt/gentoo/boot/efi

Chroot for installing

Final preparation:

# mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
# mkdir -p -v /mnt/gentoo/etc/portage/repos.conf
# cp -v /mnt/gentoo/usr/share/portage/config/repos.conf /mnt/gentoo/etc/portage/repos.conf/gentoo.conf
# nano -w /mnt/gentoo/etc/portage/repos.conf/gentoo.conf

Make gentoo.conf contents as:

[DEFAULT]
main-repo = gentoo

[gentoo]
location = /usr/portage
sync-type = rsync
sync-uri = rsync://rsync.cn.gentoo.org/gentoo-portage
auto-sync = yes

# for daily squashfs snapshots
#sync-type = squashdelta
#sync-uri = mirror://gentoo/../snapshots/squashfs

The old portage sync system use SYNC=rsync://rsync.cn.gentoo.org/gentoo-portage in make.conf. In new >portage-2.2.16, it is sync-url in gentoo.conf. To get the sync-url value:

# mirrorselect -i -r -o | sed 's/^SYNC=/sync-uri = /;s/"//g' >> /mnt/gentoo/etc/portage/repos.conf/gentoo.conf 

Finally chroot into new environment:

# cp -v -L /etc/resolv.conf /mnt/gentoo/etc/
# chroot /mnt/gentoo /bin/bash
# source /etc/profile && export PS1="(chroot) $PS1"

~~After entering *chroot* environment, the very first is to mask GnuPG 2.0. Refer to *Tips -> GnuPG* at the end of post.~~

Return to chroot environment

# cryptsetup --key-file=/path/to/luks-key luksOpen /dev/sda8 gentoo
# vgchange --available y
# lvscan
# ls /dev/mapper/
# swapon -v /dev/mapper/vg1-swap
# mount -v -t ext4 /dev/mapper/vg1-root /mnt/gentoo
# mount -v -t ext4 /dev/mapper/vg1-home /mnt/gentoo/home
# mount -v -t ext2 UUID=63053222-d5f9-43af-a7ce-76f0ca5a14ed /mnt/gentoo/boot
# mount -v /dev/sda2 /mnt/gentoo/boot/efi
# cp -v -L /etc/resolv.conf /mnt/gentoo/etc/
# mount -v -t proc proc /mnt/gentoo/proc
# mount -v --rbind /sys /mnt/gentoo/sys
# mount -v --rbind /dev /mnt/gentoo/dev
# mount -v --make-rslave /mnt/gentoo/sys
# mount -v --make-rslave /mnt/gentoo/dev
# chroot /mnt/gentoo /bin/bash
# source /etc/profile && export PS1="(chroot) $PS1"

initramfs and grub2

After getting into chroot environment, the most procedures, pleae refer to gentoo installation.

The main difference from that link, is genkernel and grub2.

Kernel option support

From the above, we use --cipher serpent-xts-plain64 and ` –hash sha512` which reminds us the related kernel option must be Y instead of 'M'.

CRYPTO_SERPENT Y

CRYPTO_SHA512 Y

Why they are should be 'Y' instead of 'M'? The boot process is basically:

Power on;
EFI firmware find the default bootloader Gentoo grub2 at '/dev/sda2/EFI/gentoo/grub64.efi';
grub2 launch initramfs into RAM;
initramfs found GnuPG keyfile at '/boot/key' and ask user for decryption; decrypted keyfile is used to decrypt rootfs.
Control pass to init scripts. After that, kernel modules loaded.

You find that if the two kernel options compiled as modules 'M', then you cannot pass the 4th step. No serpent and sha512 function for keyfile decryption. This is the error message at boot:

http://pastebin.com/fZgMwyNw

genkernel

# echo "sys-kernel/genkernel cryptsetup" > /etc/portage/package.use/genkernel
# emerge -av genkernel
# genkernel --lvm --luks --gpg --busybox --install initramfs

You see genkernel takes lvm, luks, gpg, and busybox parameter. Don't worry about the gpg version issue mentioned earlier! man genkernel shows it includes GnuPG 1.x:

--[no-]gpg
           Includes or excludes support for GnuPG 1.x, the portable standalone
           branch of GnuPG. A key can be made from gpg --symmetric -o
           /path/to/LUKS-key.gpg /path/to/LUKS-key . After that, re-point the
           root_key argument to the new .gpg file.

Some posts mention emerge -av app-crypt/gnupg lvm2 busybox cryptsetup in the chroot system before genkernel. Basically, these tools are only needed in LiveCD and 'genkernel'. In our new Gentoo system, it is not a must.

sys-kernel/genkernel cryptsetup USE flag will draw in cryptsetup, and then lvm2 in chroot, while busybox is an essential @system package which is included in stage3 tar bar. You can run equery u busybox, and find static USE flag. Try:

# emerge -pvc cryptsetup

It will reminds:

Calculating dependencies... done!
  sys-fs/cryptsetup-1.6.5 pulled in by:
    sys-kernel/genkernel-3.4.49.2 requires sys-fs/cryptsetup

>>> No packages selected for removal by depclean
Packages installed:   549
Packages in world:    38
Packages in system:   44
Required packages:    549
Number to remove:     0

There is no need to rc-update add lvm boot, for this is only useful when there is any other LVM devices like external removable LVM disks, USB stick etc.

Grub2

The core is to set the correct GRUB_CMDLINE_LINUX parameters for kernel and init scripts. Some of those parameters are indeed consumed by the kernel (a reasonable complete list of which is provided by kernel.org); however, others are realy targeted at the init script. The Linux kernel passes the init script (or program) any parameters it has not already rocessed as arguments, and the init script can also read the full command line in any event, via /proc/cmdline.

# nano -w /etc/default/grub
GRUB_CMDLINE_LINUX="crypt_root=UUID=8e105495-5a45-4297-b6ad-6e2d97abb461 dolvm root=/dev/mapper/vg1-root rootfstype=ext4 root_keydev=UUID=63053222-d5f9-43af-a7ce-76f0ca5a14ed root_key=key/luks-key.gpg"

crypt_root: this tells init (provided by genkernel) script which partition it should attempt to decrypt with cryptsetup. It is set to UUID of the LUKS formatted partition, say sda8.
dolvm: activate LVM volumes on bootup.
rootfstype: root file system type.
root: the location of the root filesystem to the kernel.
root_keydev: if necessary provide the name of the device that carries the root_key. If unset while using root_key, it will automatically look for the device in every boot. It specifies the device path of device on which the keyfile is located. In this scheme, it is the '/boot' USB stick.
root_key: In case your root is encrypted with a key, you can use a device like a usb pen to store the key. This value should be the key path relative to the mount point. In this scheme, it is in the sub-directory key/luks-key.gpg.

If it has a .gpg extension (as our case), the init script will treat it as being a gpg encrypted keyfile, and prompt for a passphrase to unlock the keyfile first (either textually at the console, or using the splash screen manager if the system sets that).
Here we use UUID instead of device file path '/dev/sda8' or '/dev/sdc1'. Since UUID is unique while device path name might change. For example, if current PC is plugged in 3 usb sticks as '/dev{/sdb,/sdc,/sdd}' and none of them is the boot stick, initramfs even cannot find the gpg keyfile location.
More about these command please read man genkernel and kernel.org.

# grub2-mkconfig -o /boot/grub/grub.cfg

In chroot environment, grub2-mkconfig and os-prober might fail to generate Windows menu. Don't worry! You can use the following template to edit '/etc/grub.d/40_custom'. Or run grub2-mkconfig again when log into the real new system.

menuentry 'Windows Boot Manager (on /dev/sda2)' --class windows --class os $menuentry_id_option 'osprober-efi-DAD1-F557' {
        insmod part_gpt
        insmod fat
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  DAD1-F557
        else
          search --no-floppy --fs-uuid --set=root DAD1-F557
        fi
        chainloader /EFI/Microsoft/Boot/bootmgfw.efi
}

fstab

# UUID of usb stick
UUID="aaa-bbb-ccc-ddd..."	/boot	ext2	noauto,noatime	1 2
/dev/sda2	/boot/efi	vfat	noauto,noatime	1 0
/dev/mapper/vg1-root	/	ext4	noatime,errors=remount-ro	0 1
/dev/mapper/vg1-swap	none	swap	sw	0 0
/dev/mapper/vg1-home	/home	ext4	defaults	0 0
/dev/sda1	/mnt/Recovery	ntfs-3g		noauto,ro	0 0
/dev/sda4	/mnt/Win81	ntfs-3g		noauto,ro	0 0
/dev/sda5	/media/Data	ntfs-3g		noauto,nls=utf8,locale=zh_CN.utf8,uid=zachary,gid=users,dmask=022,fmask=133	0 0
/dev/sda6	/media/Misc	ntfs-3g		nls=utf8,locale=zh_CN.utf8,uid=zachary,gid=users,dmask=022,fmask=133	0 0
/dev/sda7	/media/WLshare	ntfs-3g		nls=utf8,locale=zh_CN.utf8,uid=zachary,gid=users,users,dmask=022,fmask=133	0 0

Get out of chroot

# exit
# swapoff -v /dev/mapper/vg1-swap
# umount -lv /mnt/gentoo/home
# umount -lv /mnt/gentoo/boot{/eif,}
# umount -lv /mnt/gentoo/dev{/shm,/pts,}
# umount -lv /mnt/gentoo{/proc,/sys,}
# vgchange --available n
# cryptsetup luksClose gentoo
# reboot

Tips

mtab
```
# ln -vsf /proc/self/mounts /etc/mtab
```
XSESSION

When emerging xorg and xfce4, the wiki recommends setting:
```
# echo XSESSION="Xfce4" > /etc/env.d/90xsession
```
for all users on the system. This is not a good idea since root account can run X as well, which is NOT a good scheme.

So remove '/etc/env.d/90xsession' if exists.
Editor
```
 # eselect editor set "/usr/local/bin/ect"
```
Run eselect as root set the system default editor for all users. Of course, you can also use method in emacs configuration.
[outdated] GnuPG

This is outdated.

Though not necessary to emerge GnuPG in the new system. But later on, package layman will draw in git which in turn draws in GnuPG. As mentioned, >app-crypt/gnupg-2.0" will cause *pinentry issue.
```
# echo "=app-crypt/gnupg-2*" > /etc/portage/package.mask/gnupg
```
This should be the 1st thing after entering chroot.

Reference

http://ceyes.github.io/2015-06/Gentoo-Setup-LVM-over-LUKS/
https://wiki.gentoo.org/wiki/Sakaki%27s_EFI_Install_Guide/Preparing_the_LUKS-LVM_Filesystem_and_Boot_USB_Key
http://www.gentoo-wiki.info/SECURITY_System_Encryption_DM-Crypt_with_LUKS
http://blog.marek.sapota.org/article/2012/10/07/installing-gentoo-on-an-encrypted-partition.html
http://iswwwup.com/t/c1ced3aa98c4/encryption-luks-storing-keyfile-in-encrypted-usb-drive.html
http://www.gossamer-threads.com/lists/gentoo/user/300698
http://unix.stackexchange.com/questions/183666/booting-gentoo-on-lvm-inside-luks-with-gpg-encrypted-keyfile
https://bbs.archlinux.org/viewtopic.php?id=120181
https://forums.gentoo.org/viewtopic-p-7485020.html

Kali Linux Live USB Persistence

2015-07-23T00:00:00+00:00

This post introduces making a bootable Kali USB stick while making changes persistent.

Download Kali.

kali-linux-1.1.0a-amd64.iso

Don't forget to verify SHA1Sum befere proceding.
Create a bootable Kali Live USB drive. My current working system is Gentoo. Refer to Making a Kali Bootable USB Drive.
1. Make sure USB flash at least 8GB. Plug USB into PC.
2. # parted -l
  
  Find the correct USB device name. In my system it is /dev/sdb. Though dd command is magical, you would lost all the data if you provided the wrong device name to dd.
3. The blocksize parameter can be increased, and while it may speed up the operation of the dd command, it can occasionally produce unbootable USB drives, depending on your system and a lot of different factors. The recommended value, bs=512k, is conservative and reliable.
4. Magic:
```
# dd if=/path/to/kali-linux-1.1.0a-amd64.iso of=/dev/sdb bs=512k
```
  You don't need to format Live USB. dd will handle it.
5. Image the USB drive can take a good amount of time, over ten minutes or more is not unusual, as the sample output below shows. Be patient!
6. After dd, use parted command to see what happens to your USB flash /dev/sdb:
```
# parted -a optimal /dev/sdb
```
  This is my system output:
  
  zhtux ~ # parted -a optimal /dev/sdb
  GNU Parted 3.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) unit MiB
  (parted) print free
  Model: SanDisk Cruzer Edge (scsi) Disk /dev/sdb: 15267MiB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags:
  
  Number Start End Size Type File system Flags 0.02MiB 0.03MiB 0.02MiB Free Space 1 0.03MiB 2858MiB 2858MiB primary boot, hidden 2 2858MiB 2921MiB 63.0MiB primary fat16 2921MiB 15267MiB 12346MiB Free Space
7. Several points from the output.
  1. unit MiB instead unit MB. MiB shows you the exact value (at Bytes) at the exact disk position, while MB might round up and the actual position might be 500KB ahead or 500KB after the MB value. Similarly, we have GiB and TiB. Read parted manual and arch wiki rounding.
  2. You can find dd creates two primary partitions 1 (2858MiB, with boot flag) and 2 (63.0 MiB). Why dd creates two partitions? This is due to kali-linux-1.1.0a-amd64.iso itself a copy of two partitions.
    # fdisk -l /path/to/kali-linux-1.1.0a-amd64.iso or # parted /path/to/kali-linux-1.1.0a-amd64.iso print
    This is the ouput of from parted:
    
    zhtux mnt # parted /media/WLShare/kali-linux-1.1.0a-amd64/kali-linux-1.1.0a-amd64.iso print Model: (file) Disk /media/WLShare/kali-linux-1.1.0a-amd64/kali-linux-1.1.0a-amd64.iso: 3063MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags:
    
    Number Start End Size Type File system Flags 1 32.8kB 2997MB 2997MB primary boot, hidden 2 2997MB 3063MB 66.1MB primary fat16
  3. dd is a stupid command only copy bytes by bytes fromm if to of. We can find there are a lot of free space untouched after partition 2. We can make use of the reamaining free space. My Gentoo Live USB has only one partion consuming the whole flash storage.

Adding USB Persistence with LUKS Encryption, refer to Kali persistence USB.

persistence means changes to Kali system on Live USB remains accross reboots. Basically just create an extra primary partition on Live USB to store persistent files.

Create and format an additional partition on the USB drive. Continue from step 2.6, use parted tool to create an ext3 primary partition from the remaining free space.

(parted) mkpart primary 2921MiB 100%
Warning: You requested a partition from 2921MiB to 15267MiB (sectors 5982208..31266815).
The closest location we can manage is 2921MiB to 15267MiB (sectors 5983104..31266815).
Is this still acceptable to you?
Yes/No? Yes                                                               
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel?

The Kali Doc recommends Ignore. Here I want to try the disk alignment for better performance. Refer to arch wiki warnings. This alignment means the start position is not aligned. The end position 100% or -1s will align itself automatically.

Enter Ignore to go ahead anyway, print the partition table in sectors to see where it starts, and remove/recreate the partition with the start sector rounded up to increasing powers of 2 until the warning stops.

I have tried 2^8, 2^9, 2^10, 2^11. Finally, 2^11 works.

5983104s % 2048 = 896s; 2048s - 896s = 1152s; 5983104s + 1152s = 5984256s.

Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? Ignore                                                     
(parted) unit s                                                           
(parted) print free                                                       
Model: SanDisk Cruzer Edge (scsi)
Disk /dev/sdb: 31266816s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start     End        Size       Type     File system  Flags
        32s       63s        32s                 Free Space
 1      64s       5854015s   5853952s   primary               boot, hidden
 2      5854016s  5983103s   129088s    primary  fat16
 3      5983104s  31266815s  25283712s  primary               lba

(parted) rm 3                                                             
(parted) mkpart primary 5983232s 100%                   
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? Cancel                                                     
(parted) mkpart primary 5984256s 100%                                     
(parted) print free                                                       
Model: SanDisk Cruzer Edge (scsi)
Disk /dev/sdb: 31266816s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start     End        Size       Type     File system  Flags
        32s       63s        32s                 Free Space
 1      64s       5854015s   5853952s   primary               boot, hidden
 2      5854016s  5983103s   129088s    primary  fat16
        5983104s  5984255s   1152s               Free Space
 3      5984256s  31266815s  25282560s  primary               lba

(parted) q                                                                
Information: You may need to update /etc/fstab.

From the final output, we can see there is a big free space (1152 * 512B = 576 MB) between partition 2 and 3. Of course, you can just leave 3rd partition unaligned considering the limited flash storage.

Initialize the LUKS encryption on the newly-created partition. You’ll be warned that this will overwrite any data on the partion. When prompted whether you want to proceed, type YES (all upper case). Enter your selected passphrase (use kali currently) twice when asked to do so.
```
# cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb3
# cryptsetup luksOpen /dev/sdb3 my_usb
```
Format the persistence partition 3 as ext4 (the official reference use ext3).
```
# mkfs.ext4 -L persistence /dev/mapper/my_usb
# e2label /dev/mapper/my_usb persistence, this step might be optional.
```
This might take several minutes. Don't touch the keyboard!

Create a mount point, mount our new encrypted partition there, set up the persistence.conf file, and unmount the partition.

# mkdir -p /mnt/usb
# mount /dev/mapper/my_usb /mnt/usb
# echo "/ union" > /mnt/usb/persistence.conf
# umount /dev/mapper/my_usb

Close the encrypted channel to our persistence partition.
```
# cryptsetup luksClose /dev/mapper/my_usb
```

Linux Distributions

2015-07-22T00:00:00+00:00

Linux kernel is the core of system management. We still need distributions to provide users a slightly high level interface. A Linux distribution is basically a wrapper of Linux kernel, supporting GUI, package management, etc.

You never notice Windows kernel thing since Windows is a complete system.
Common Linux distributions differ in package management, update method, stable or not, security, free or not etc. Refer to 如何选择发行版
1. Debian: .deb package; stable.
  
  Ubuntu, Mint.
2. Fedora: .rpm package.
  
  RHEL (commercial use), CentOS (Community ENTerprise OS).
  
  CentOS derives from reverse engineering of RHEL, which takes away part of RHEL customers.
3. OpenSUSE: .rpm built specialy for openSUSE.
4. Arch Linux(KISS, binary package), Gentoo (source compiled locally)
  
  Rolling release instead of normal fix release. Need time and patience.
  
  Arch Linux Wiki is the best in the world!
5. BackTrack, Kali
  
  Based-on Debian now, focusing on security issue.
Package management, refer to What is the difference between dpkg and aptitude/apt-get? and Comparison of major Linux package management systems
1. Debian: .deb -> dpkg -> apt-get -> aptitude
2. Fedora: .rpm -> rpm -> yum
3. openSUSE: special .rpm -> rpm -> zypper
4. Arch: AUR binary -> pacman
5. Gentoo: portage source -> emerge (equery, eix)
They came up with a way to store the files of an application in a package so that it can be easily installed. So the .deb package was born.

They needed a tool to install these .deb files, so they came up with the dpkg tool. This tool, however, will just install the .deb file, but will not install its dependencies because it doesn't have those files and it does not have access to repositories to go pull the dependencies from.

Then, they came up with apt-get, which automates the problems in the previous point. Underneath the hood, apt-get is basically dpkg (so apt-get is a front-end for dpkg), but a clever one that will look for the dependencies and install them. It even looks at the currently installed dependencies and determines ones that are not being used by any other packages, and will inform you that you can remove them.

aptitude then came along, and it's nothing but a front-end UI for apt-get. More difference refer to what is difference between apt-get and aptitude.
Rumours

CentOS让RedHat招安了

Scientific Linux不还有吗。从来都不觉得CentOS有啥好的，package太老，连经常被嘲笑的Debian stable都看不下去了。稍微正常点用Ubuntu，文艺点用Arch Linux，CentOS没有了没什么大不了的

scientific linux你用过吗？我以前怀着希望试了一下，感觉用户群小的linux bug多啊。

CentOS更忠于RHEL原版, Scientific Linux自己改的东西多一些

这下估计 CentOS 要没有什么作为了。 CentOS 的价值在于，提供一个开源并且可以直接预装的RHEL clone. RHEL 是很多企业的选择，因为他们想要超级稳定性，然后想要技术支持，例如，有了 zero day 的bug 要赶紧补上。成熟的企业除非有特别强的Linux 背景，不是很感用随便的 distro。

RHEL 一直都在玩一个游戏就是，开源对吧。RHEL 提供src rpm。但是 RHEL 不提供可以直接预装的 binary,只对商业用户提供。这个并没有违反开源的游戏规则。而且那个 RHEL 里面含有一些程序让别人不能直接复制那个 RHEL 的安装盘放到其他地区下载。同样，升级也需要是付费用户（有免费试用）。

CentOS 就是提供这样的服务，把 RHEL 的 src rpm，从新编译最后组装出 RHEL 兼容的但是大家可以随便使用的预装光盘。大家觉得不就是编译一下 src rpm 么，简单阿。其实不是那末简单。Redhat 的 src rpm 要编译出和 RHEL一样效果的 binary 需要用到 Redhat 的不开放的编译环境，还有很怪异的编译启动参数。这些参数是从编译环境直接传过来的。也就是说，CentOS提供的服务是逆向工程了RHEL的编译环境和捣鼓出安装光盘。相当与挖了 RH 的商业壁垒，挡人财路。

被招安之后，RH 当然不会干雇人来挖自己想保护的商业壁垒的蠢事。我猜结果就是，CentOS 估计要被阉割了。肯定不会完全消失，但是要用 CentOS 不爽。但是没有不爽到要有人舍得另起炉灶的地步。

类似的例子有 ntfs 3G. NTFS 3G 搞了个用户层的文件系统，就是为了保护他们商业版本的高性能内核 NTFS 文件系统。这个方式很有效果，大多数人都比较慢的 ntfs 也行，所以搞个纯内核的NTFS 意义就小了很多。目前就没有人对着干。嵌入系统使用 NTFS，那个内核版本的可能还是比较合理的选择。

比如说哈，NSF 和 NIH 强调 open source，所以很多全国的大型计算中心在用 CentOS。这下子可就有大变动了。新的计算中心可能就要选 debian 了。

这是好事，centos的包都老掉牙了我觉得一般小地方的管理员就是为了所谓稳定，来个保守的选择centos，痛苦死了我很多东西都是自己local install的

RHEL/CentOS系只适合大企业内部养老，稍微正常点的公司都不会用。默认系统是稳定，但是很多包要么很老要么根本就没有需要自己找第三方源或者裸装，这样只要有非官方包进来，从根本上就破坏了它所谓的稳定，还不如用稍微正常点然后稳定和后续维护也还不错的Ubuntu LTS。激进和文艺一点的直接上Arch也没见出什么事

大公司用这个有，我个人观察比较大一个因素是推卸责任。人家不在乎没出什么事，那个是理所当然的。在乎的是出了事谁来背黑锅，可以推卸责任。所以比较像样的支持服务还是需要的。作为推卸责任的手段。“我们已经联系XXX公司在深入研究这个崩溃问题了，人家已经派最好的工程师在找原因…”。技术支持是推卸黑锅的很好办法，我已经用了技术好的公司了，那责任不在我这里，是那些公司不行。你要是自己用冷门的distro，万一了问题就要自己扛，不是开源没，自己动手改啊。自立更生。改不好啊，当初谁负责谁背黑锅。有技术实力的是不怕用比较新的系统的。 kernel.org 就是用fedora。人家流量也很大，一样跑得好好的。关键是里面有牛人，有问题自己能消化。

唉唉，在法律责任切分严格的行业，RHEL 还是有压倒性的优势的。比如说医疗系统。我们这里搭个 cluster，我说上 debian，IT 直接就问：出了问题算谁的？谁来承担责任并且走人？再比如说，我们这里上了一套数据库系统，因为先期投入大，造成去年预算问题，信用等级被降级，CIO 立刻走人。

嗯嗯，是啊是啊。我们这里 IT 说他们有雇员有RH的认证，所以有资质使用RHEL。买大些的东西，首先看的就是出了问题后，如何确定责任。每个人都只会支持那些让自己的职业比较安全的东西，这是底线。好不好则另说了。

企业端不用RHEL/CentOS的只怕不多。

For those that are saying Red Hat is the main project, yes it is.. But the diagram shows how the flow actually goes. Red Hat are based on stable packages of Fedora, not that Red Hat is Fedora's derivatives. CentOS however, is the stable release of Red Hat for the community. What we don't see here is that Red Hat uses both Fedora and CentOS for its experiments and stability testings to improve Red Hat, which is genius. This contributes for the corporate highly stable release of Linux, which is one of the best OS around.

Fedora is the starting point. Those packages eventually make it into RHEL once they are proven to be stable. Then the RHEL Source code is used to create CentOS without the RHEL branding.

From a corporate point of view Red Hat is the main project that Red Hat throws the amount of resources. and the one that brings the revenue. However, as others have said, on a technical level, and as the diagram depicts. Fedora flows into Red Hat and Red Hat flows into CentOS

Kali ABC

2015-07-22T00:00:00+00:00

First attempt to launch Kali system with some basic ideas. Kali originates from BackTrack which is no longer maintained.

Kali is a Debian-based distribution devoted to Penetration Testing.

"the quieter you become, the more you are about to hear"
Install Kali in VMWare.

Choose customized VMWare images offered by offensive-security instead of the official ISO. Just extract the 7z image and launch it from VMWare.

The default root password is toor.
Boot warning:
1. not starting portmapper is not running, refer to How to fix 1 and #17 floor
```
# update-rc.d -f nfs-common remove
```
2. PulseAudio configured for per-user sessions Warning, refer to not an issue
  
  This is not actually an real issue. The real issue lies in the warning message itself.

Git Configuration

2015-07-19T00:00:00+00:00

Installation
Configuration
Gitignore

Installation

Install
```
# emerge -av dev-vcs/git
```

Configuration

touch ~/.config/git/config

If you'd like a per-user config, create this file first.

In Gentoo Xfce4, $XDG_CONFIG_HOME is not set by default. Without touching this file manually, it would defaults to ~/.gitconfig instead.
git config --system user.name "<username>"

Specify a name so your commits will be properly labeled. It can be anything that is meaningful to you.
git config --system user.email "<id+username>@users.noreply.github.com"

Tell Git the email address that will be associated with your Git commits. The email you specify should be the same one found in your email settings. To keep your email address hidden, see Keeping your email address private.
git config --system core.editor emc

You can configure the default text editor that will be used when Git needs you to type in a message (without -m opton on push). If not configured, Git uses your system’s default editor, which is generally one of Vim, Emacs and Nano.
git config --system credential.helper 'cache --timeout=3600'

If you're managing GitHub repositories using HTTPS due to SSH blocked by company firewall, you can use a credential helper to tell Git to remember your GitHub username, password or API token for a while every time it talks to GitHub. By default, Git will cache your password for 15 minutes (900 seconds).
git config --system push.default simple
For EOL conversion, refer to git line ending conversion.
git config --system -l

Check current config.
For a single user system, you'd best use –system instead of –global.

Gitignore

For language-specific ignorance, refer to A collection of useful .gitignore templates.

Two consective stars followed by a slash (**/) means to match all directories. Without the trailing slash, to match anything regardless of files or directories.
```
# ignore name 'foo'
**/foo
# equivalent to:
foo

# ignore name 'bar' under directory 'foo'
**/foo/bar

# ignore anything under directory 'foo'
foo/**
```
1. All patterns are relative to the location of the .gitignore file unless a leading slash is provided.
2. Without trailing slash, a name can match a file or a directory.
Ignore a directory but a file.
```
my-dir/**
!my-dir/my-file
```
The consecutive stars ** is to improve performance. Another way to solve this issue is creating a new .gitignore under my-dir:
```
# my-dir/.gitignore
   
*
!.gitignore
   
!my-file
```
The newly created .gitignore has higher priority over those in uppper directories.
~/.config/ignore

This is the global ignore list that applys to all Git repositories for current user.

For example, on my Gentoo:
```
*~
.#*
\#*\#
```
The first line ignores Linux file auto-backup. The next two lines are for Emacs eidtor's backup.
If a file or directory is accidently pushed to remote repository, you decide to remove that later on.

Firstly, create a .gitignore file. Put patterns there to ignore unwanted names.
```
~ $ git rm -r --cached name; git commit -m "remove name"; git push
```

Cronie and Anacron

2015-07-19T00:00:00+00:00

cron automates taks sheduling at specific time. There are many different implementations of which I choose cronie with anacron USE flag enabled. It accomplishes this task by waking up every minute and checking to see if there are any cron-jobs to run in any of the user crontabs.

It is basically for servers that won't shutdown. Sheduled tasks would not be ran if the machine is powered off at the exact sheduled time. anacron however, does not assume the machine is running continuously. anacron usually relies on another cron daemon (i.e. cronie) and executes commands at intervals specified in day. It will run missed jobs upon startup.

root@tux ~ # echo 'sys-process/cronie anacron' > /etc/portage/package.use/cronie (opt)
root@tux ~ # emerge -avt virtual/cron
root@tux ~ # rc-update add cronie default

cronie has built-in anacron feature that is optionally now.

cronie schedules locates in /etc/crontab and /etc/cron.d/, while anacron in /etc/anacrontab. Like most other cron daemons, cronie depends on sys-process/cronbase whose jobs in /etc/cron.{daily,hourly,weekly,monthly}. So we have three cron components, namely cronbase, cronie, and anacron which can be combined to meet our job schedule requirements.

All those above are system-wide schedules and we can manually edit those files. We define user schedules with crontab command:

root@tux ~ # crontab [ -u USER ] [ -l | -e | -r ]

crontab command does not handle system-wide schedules. Without explicit -u argument, it defaults to root account. It edits (or creates) file /var/spool/cron/conrtabs/${USER}. File name is the same as user account name. When executing commands under su you should always use the -u option.

The built-in anacron feature does not bring in command line to edit /etc/anacrontab. Like system-wide schedules, anacron schedules should be updated manually.

Let's have a look at the default system-wide schedules. In /etc/crontab, we found:

*  * * *    root    rm -f /var/spool/cron/lastrun/cron.hourly
3  * * *     root    rm -f /var/spool/cron/lastrun/cron.daily
4  * * 6     root    rm -f /var/spool/cron/lastrun/cron.weekly
5  1 * *     root    rm -f /var/spool/cron/lastrun/cron.monthly
*/10  *  * * *  root    test -x /usr/sbin/run-crons && /usr/sbin/run-crons

cronie daemon runs run-crons script (comes with cronbase) every 10 minutes. run-crons looks into */etc/cron.[hourly daily weekly monthly]* for scripts to be executed. For example, /etc/cron.daily/logrotate.

The schedules above gurantee jobs got ran even if the computer was off when they were scheduled to run. Details can be checken from run-crons script. Therefore, there is no need to enable anacron USE.

Specially, we found /etc/cron.hourly/0anacron script. That is to say:

cronie triggers cronbase's hourly job 0anacron.

At the tail of 0anacron, anacron -s handles schedules defined /etc/anacrontab. Different from cronie that schedules jobs at fixed time , anacron schedules to run jobs every other days (day, week, month).

*/etc/cron.[daily weekly monthly]* will be ran twice by cronie (run-crons) and anacron (run-parts) at different times, leading to possible double job executions.

Either remove anacron USE or comment out any overlapping entries from /etc/crontab or /etc/anacrontab.

Before a user can access to crontab, he must be in cron group by gpasswd -a username cron. Then we update */etc/cron.{allow, deny}:

# /etc/cron.deny
all

# /etc/cron.allow
username

With this setting, all users (except root) by default cannot access to crontab. We grant access by adding user name to cron.allow.

sed awk

2015-06-12T00:00:00+00:00

These tools do a big favor when editing streams. Both programs use regular expressions for selecting and processing text.

For instance, a configuration file contains lots of comment lines starting with # and blank lines. When applying the configuration to systems, you might need to remove those lines for a clean and tidy configuration file.

sed '/^ *#/d' nginx_with_comments.conf | sed '/^$/d' > nginx.conf

Use sed(stream editor, take pipes as input) command to delete d specific pattern lines. The 1st part removes comment lines while the 2nd remove blank lines.

awk considers each file as a set of records, which by default are the lines in the file. "awk" enables you to create a condition and action pair, and for each record that matches the condition, the action will fire. Most "awk" commands follow this pattern – you have a condition, followed by an action in curly braces.

ls -al ~/ | awk '/zachary/{print $1, $9}'

This command will first file lines contain zachary and then print the 1st and 8th items/columns - to print permissions and filenames. awk is really useful when dealing with columns. You can use other conditions; for example, checking whether or not a certain field is above or below a given threshold (i.e. $5 > 2), or checking whether a record has a certain column.

ls -al | awk '$9 ~/^\./ {gsub(/zachary/, "awkSub"); print;}'

akw check wheter the 9th item (filename) starts wity dot. Then use gsub (global substitute) to substitue every occurence of zachary with awkSub.

awk VS sed

Both are tools that transform text. BUT awk can do more things besides just manipulating text. Its a programming language by itself with most of the things you learn in programming, like arrays, loops, if/else flow control etc You can "program" in sed as well, but you won't want to maintain the code written in it.

sed is a stream editor. It works with streams of characters on a per-line basis. It has a primitive programming language that includes goto-style loops and simple conditionals (in addition to pattern matching and address matching). There are essentially only two "variables": pattern space and hold space. Readability of scripts can be difficult. Mathematical operations are extraordinarily awkward at best.

awk is oriented toward delimited fields on a per-line basis. It has much more robust programming constructs including if/else, while, do/while and for (C-style and array iteration). There is complete support for variables and single-dimension associative arrays plus (IMO) kludgey multi-dimension arrays. Mathematical operations resemble those in C. It has printf and functions. The "K" in "AWK" stands for "Kernighan" as in "Kernighan and Ritchie" of the book "C Programming Language" fame (not to forget Aho and Weinberger). One could conceivably write a detector of academic plagiarism using awk.

I would tend to use sed where there are patterns in the text. For example, you could replace all the negative numbers in some text that are in the form "minus-sign followed by a sequence of digits" (e.g. "-231.45") with the "accountant's brackets" form (e.g. "(231.45)") using this (which has room for improvement):

echo "-123.456 -345.678" sed 's/-\([0-9.]\+\)/(\1)/g'

I would use awk when the text looks more like rows and columns or, as awk refers to them "records" and "fields" - matrix. Refer to the example above.

Use sed for very simple text parsing. Anything beyond that, awk is better. In fact, you can ditch sed altogether and just use awk. Since their functions overlap and awk can do more, just use awk. You will reduce your learning curve as well.

总之，sed能实现的awk也能实现；简单的模式匹配用awk；有矩阵行列特征用awk。

library soname

2015-06-07T00:00:00+00:00

In Unix operating systems, a soname is a field of data in a shared object file. The soname is a string, which is used as a "logical name" describing the functionality of the object.

The soname is often used to provide version backwards-compatibility information. For instance, if versions 1.0 through 1.9 of the shared library libx provide identical interface, they would all have the same soname, e.g. libx.so.1. If the system only includes version 1.3 of that shared object, with filename libx.so.1.3, the soname field of the shared object tells the system that it can be used to fill the dependency for a binary which was originally compiled using version 1.2.

Linux 系统，也同样面临和Window一样的问题，如何控制动态库的多个版本问题。Window之前没有处理好，为此专门有个名词来形容这个问题 “Dll hell”，其严重影响软件的升级和维护。 Dll hell 是指windows 上动态库新版本覆盖旧版本，但是却不兼容老版本。常常发生在程序升级之后，动态库更新，原有程序运行不起来；或者装新软件，但是已有的软件运行不起来。同样Linux操作系统，也有同样的问题，那么它是怎么解决的呢？

Linux 为解决这个问题，引入了一套机制，如果遵守这个机制来做，就可以避免这个问题。但是这只事一个约定，不是强制的。但是建议遵守这个约定，否则同样也会出现 Linux 版的Dll hell 问题。下面来介绍一个这个机制。这个机制是通过文件名，来控制dll （shared library）的版本。

Linux 上的Dll ，叫shared library，其有三个名字，分别又不同的目的。

Every shared library has a special name called the soname. The soname has the prefix lib, the name of the library, the phrase .so, followed by a period and a version number that is incremented whenever the interface changes (as a special exception, the lowest-level C libraries don't start with lib). A fully-qualified soname includes as a prefix the directory it's in; on a working system a fully-qualified soname is simply a symbolic link to the shared library's real name.

Every shared library also has a real name, which is the filename containing the actual library code. The real name adds to the soname a period, a minor number, another period, and the release number. The last period and release number are optional. The minor number and release number support configuration control by letting you know exactly what version(s) of the library are installed. Note that these numbers might not be the same as the numbers used to describe the library in documentation, although that does make things easier.

In addition, there's the name that the compiler uses when requesting a library, (I'll call it the linker name), which is simply the soname without any version number.

The key to managing shared libraries is the separation of these names. Programs, when they internally list the shared libraries they need, should only list the soname they need. Conversely, when you create a shared library, you only create the library with a specific filename (with more detailed version information). When you install a new version of a library, you install it in one of a few special directories and then run the program ldconfig(8). ldconfig examines the existing files and creates the sonames as symbolic links to the real names, as well as setting up the cache file /etc/ld.so.cache (described in a moment).

ldconfig doesn't set up the linker names; typically this is done during library installation, and the linker name is simply created as a symbolic link to the latest soname or the latest real name. I would recommend having the linker name be a symbolic link to the soname, since in most cases if you update the library you'd like to automatically use it when linking. I asked H. J. Lu why ldconfig doesn't automatically set up the linker names. His explanation was basically that you might want to run code using the latest version of a library, but might instead want development to link against an old (possibly incompatible) library. Therefore, ldconfig makes no assumptions about what you want programs to link to, so installers must specifically modify symbolic links to update what the linker will use for a library.

Thus, /usr/lib/libreadline.so.3 is a fully-qualified soname, which ldconfig would set to be a symbolic link to some realname like /usr/lib/libreadline.so.3.0. There should also be a linker name, /usr/lib/libreadline.so which could be a symbolic link referring to /usr/lib/libreadline.so.3.

real name: libmath.so.1.1.1234

lib + name + .so + .major number + .minor number + .release number

主板号，代表当前动态库的版本，就是共享库文件本身的文件名，里面是共享库的代码。如果动态库的接口有变化，那么这个版本号就要加1；后面的两个版本号（小版本号和 build 号）是告诉你详细的信息，比如为一个hot-fix 而生成的一个版本，其小版本号加1，build号也应有变化。
soname: libmath.so.1 动态库的soname，其是应用程序加载共享库时，帮助寻找共享库用的文件名。其格式为:

lib + name +.so + .major number

其只包含major version number，换句话说，也就是只要其接口没有变，应用程序都可以用，不管你其后minor build version or build version。soname通常存在于几个地方：作为软链接指向real name；存放在共享库本身的文件头部header；存放在应用程序的头部共享库声明部分。
linker name: libmath.so

lib + name + .so

连接名（linker name），是专门为应用程序在链接link阶段找到共享库而设置的指向soname或这real name的软链接，一般在系统安装共享库时，系统会自动创建，通常默认应该指向系统里最新的soname or real name。
在共享库编译过程中，连接（link）阶段，编译器将生成一个共享库及real name，同时将共享库的soname，写在共享库文件里的文件头里面。可以用命令 readelf -d libmath.so.1.1.1234 去查看。在应用程序编译阶段，引用共享库时，其会用到共享库的linker name，其通过linker name 找到最新的动态库库文件，并且把共享库的soname提取出来，写在自己的共享库的头里面。当应用程序加载时候就会通过soname软链接去在给定的路径下寻找该共享库。

libmath.so -> libmath.so.1 -> libmath.so.1.1.1234
linker name是link阶段找到最新共享库版本服务的

共享库小版本升级

当升级小版本时，即接口不变。共享库的soname是不变的，所以需要重新把soname的那个连接文件指定新版本就可以。调用ldconfig命令，系统会帮你做修改那个soname link文件，并把它指向新的版本呢。这时候你的应用程序就自动升级了。

共享库，主版本升级

即接口发生变化，这是共享库的soname发生了变化。如果多个版本的共享库并存，那么就多多个soname 分别链接到对应的real name，而linker name总是指向最新的那个real name。

Reference shared-libraries.

Environment Variable

2015-06-07T00:00:00+00:00

In Unix-like systems, we have to deal with environment variables such as PATH, MANPATH etc. Here are some tips for reference.

Order of directory
```
PATH=$PATH:~/opt/bin
PATH=~/opt/bin:$PATH
```
The diffeence of the two lines is the order when searching for commands. The 2nd line make directory ~/opt/bin searched before those ones in original PATH.
export or not
```
PATH=$PATH:~/opt/bin
export PATH=$PATH:~/opt/bin
```
export is a shell builtin for bash, ksh and zsh. The first line omit the export keyword which is called assignment. We don't need export if the variable is already in the environment: any change of the value of the variable is reflected in the environment for modern shells. PATH is pretty much always in the environment; all Unix-like system sets it in the very beginning (usually in the very first process, in fact).

But if the variable is not in the environment, you must add export. For example, some environment variable is for specific program newly installed. You could test by command:

```bash ~ $ declare -p VAR_NAME

Use export if not sure.
$HOME or ~
```
export PATH="$PATH:$HOME/bin"
PATH="$PATH:$HOME/bin"
PATH=$PATH:$HOME/bin
PATH=$PATH:~/bin
PATH="$PATH:~/bin"
```
The last line does not work*. The ~ character is only expanded to your home directory when it's the first character of a word and it's unquoted. For the last line, the ~ is between double quotes and therefore not expanded. Even if you wrote export "PATH=$PATH:"~/Unix/homebrew/bin, the ~ would not be expanded because it is not at the beginning of a shell word.

Use $HOME instead of ~.
Double quotes or not

If you have ~ in variable settings, omit double quotes. In all other cases, this does not matter.
~/.profile is the place to put stuff that applies to your whole session of a login shell (like zsh), such as programs that you want to start when you log in (but not graphical programs, they go into a different file, i.e. ~/.xinitrc), and environment variable definitions. For Bash shell, we have a counterpart .bash_profile.
~/.bashrc is the place to put stuff that applies only to Bash only, such as alias and function definitions, shell options, and prompt settings. (You could also put key bindings there, but for bash they normally go into ~/.inputrc.)
~/.bash_profile can be used instead of ~/.profile, but it is Bash's alternative of ~/.profile, not used by any other shell. (This is mostly a concern if you want your initialization files to work on multiple machines and your login shell isn't bash on all of them.) This is a logical place to include ~/.bashrc if the shell is interactive.

Set up your environment (PATH, LANG, EDITOR, …) in "~/.bash_profile". Set up your bash shell (PS1, functions, shopts, …) in "~/.bashrc". Then source ".bashrc" from ".bash_profile".

Check dotfiles.
You may see here and there recommendations to either put environment variable definitions in ~/.bashrc or always launch login shells in terminals. Both are bad ideas. The most common problem with either of these ideas is that your environment variables will only be set in programs launched via the terminal, not in programs started directly with an icon or desktop menu or keyboard shortcut.

But in Gentoo, the official Wiki recommends per-user setting in ~/.bashrc which is sourced by ~/.bash_profile.
~/.bash_profile is executed for login shells, while ~/.bashrc is executed for interactive non-login shells. When you login (eg: type username and password) via console, either physically sitting at the machine when booting, or remotely via ssh: .bash_profile is executed to configure things before the initial command prompt. But, if you've already logged into your machine and open a new terminal window (xterm) inside Gnome or KDE, then .bashrc is executed before the window command prompt. ~/.bashrc is also run when you start a new bash instance by typing /bin/bash in a terminal.
Execution order:

Login shell: /etc/profile -> /etc/env.d/* -> /etc/bash/bashrc -> /etc/profile.d/*.sh -> ~/.bash_profile -> ~/.bashrc

Interactive shell: /etc/bash/bashrc -> ~/.bashrc

References: 1, 2 and 3.

Shell特殊变量：Shell $0, $#, $*, $@, $?, $$和命令行参数

2015-06-05T00:00:00+00:00

Shell 中变量名只能包含数字、字母和下划线，因为某些包含其他字符的变量有特殊含义，这样的变量被称为特殊变量。

例如，$ 表示当前Shell进程的ID，即pid，看下面的代码：

$ echo $$

变量	含义
$0	当前脚本的文件名
$n	传递给脚本或函数的参数。n 是一个数字，表示第几个参数。例如，第一个参数是$1，第二个参数是$2。
$#	传递给脚本或函数的参数个数。
$*	传递给脚本或函数的所有参数
$@	传递给脚本或函数的所有参数。被双引号(" ")包含时，与 $* 稍有不同，下面将会讲到。
$?	上个命令的退出状态，或函数的返回值。
$$	当前Shell进程ID。对于 Shell 脚本，就是这些脚本所在的进程ID。

`$*` 和 `$@` 的区别

$* 和 $@ 都表示传递给函数或脚本的所有参数，不被双引号(" ")包含时，都以"$1" "$2" … "$n" 的形式输出所有参数。

但是当它们被双引号(" ")包含时，"$*" 会将所有的参数作为一个整体，以"$1 $2 … $n"的形式输出所有参数；"$@" 会将各个参数分开，以"$1", "$2", …,"$n" 的形式输出所有参数。

Filesystem Hierarchy Standard

2015-06-04T00:00:00+00:00

[The Filesystem Hierarchy Standard (FHS)][1] defines the directory structure and directory contents in Unix and Unix-like operating systems, maintained by the Linux Foundation. The current version is 3.0, released on 3 June 2015.

This command will give you a brief description on FHS:

$ man hier

We focus on some directories relevant to binaries.

</tbody> </table> 1. You will see that files and directories that are admin only are gathered in the same directory: the `s` in /sbin and /usr/sbin and /usr/local/sbin stands for system. A normal user can not even start programs that are in there. Files a normal user can start are in /bin, /usr/bin, /usr/local/bin based on where it most logically should reside. But if they are admin only they should go to the `s` version of that directory. There is a famous utility called fuser. You can kill processes with it. If a normal user could use this (s)he would be able to kill your session. 2. The same goes for /home: /home/user1 is property of user1. /home/user2 is property of user2. user2 has no business doing stuff in user1's home (and the other way around is also true: user1 has no business doing stuff in user2's home). If all the files would be in /home with no username underneath it you would have to give permissions to every file and asses if someone is allowed to write/remove those files. A nightmare if you have tens of users. 3. Addition regarding libraries. /lib/, /usr/lib/, and /usr/local/lib/ are the original locations, from before multilib systems existed and the exist to prevent breaking things. /usr/lib32, /usr/lib/64, /usr/local/lib32/, /usr/local/lib64/ are 32-/64-bit multilib inventions. 4. If I'm writing my own scripts, where should I add these? None of the above. Please use /usr/local/bin or /usr/local/sbin for system-wide (mulitple users) available scripts. The local path means it's not managed by the system packages (this is an error for Debian/Ubuntu packages). For private-owned scripts, use ~/bin in home directory. Always keep **multi-user** nature of Linux in mind. Refer to [2] and [3]. [1]:http://www.linuxfoundation.org/collaborate/workgroups/lsb/fhs-30 [2]:http://askubuntu.com/a/308048 [3]:http://askubuntu.com/a/138551

Directory	Description
/	Primary hierarchy root and root directory of the entire file system hierarchy.
/bin	This directory contains executable programs which are needed in single user mode and to bring the system up or repair it. Essential command binaries that need to be available in single user mode; for all users, e.g., cat, ls, cp. For binaries usable before the /usr partition is mounted. This is used for trivial binaries used in the very early boot stage or ones that you need to have available in booting single-user mode.
/sbin	Like /bin, this directory holds commands needed to boot the system, but which are usually not executed by normal users. Essential system binaries, e.g., init, ip, mount.
/usr	This directory is usually mounted from a separate partition. It should hold only sharable, read-only data, so that it can be mounted by various machines running Linux. Secondary hierarchy for read-only user data; contains the majority of (multi-)user utilities and applications.
/usr/bin	This is the primary directory for executable programs. Most programs executed by normal users which are not needed for booting or for repairing the system and which are not installed locally should be placed in this directory. Non-essential command binaries (not needed in single user mode); for all users.
/usr/sbin	This directory contains program binaries for system administration which are not essential for the boot process, for mounting /usr, or for system repair. Non-essential system binaries, e.g., daemons for various network-services.
/usr/local	This is where programs which are local to the site typically go. Tertiary hierarchy for local data, specific to this host. Typically has further subdirectories, e.g., bin/, lib/, share.
/usr/local/bin	Binaries for programs local to the site. For example, a simple script to run dropbox.
/usr/local/sbin	Locally installed programs for system administration. For example, a simple script to run sslocal which needs root privilege.
/opt[/bin]	This directory should contain add-on packages that contain static files. Reserved for the installation of add-on application software packages. A package to be installed in /opt must locate its static files in a separate /opt/package or /opt/provider directory tree, where package is a name that describes the software package and provider is the provider's LANANA registered name.
~/bin	This is for user-owned private binaries. Can be appended to your $PATH.

Reverse proxy and forward proxy

2015-06-03T00:00:00+00:00

一般实现代理技术的方式就是在服务器上安装代理服务软件，让其成为一个代理服务器，从而实现代理技术。常用的代理技术分为正向代理、反向代理和透明代理。本文就是针对这三种代理来讲解一些基本原理和具体的适用范围，便于大家更深入理解代理服务技术。

正向代理(Forward Proxy)

一般情况下，如果没有特别说明，代理技术默认说的是正向代理技术。关于正向代理的概念如下：

正向代理(forward)是一个位于客户端【用户A】和原始服务器(origin server)【服务器B】之间的服务器【代理服务器Z】，为了从原始服务器取得内容，用户A向代理服务器Z发送一个请求并指定目标(服务器B)，然后代理服务器Z向服务器B转交请求并将获得的内容返回给客户端。客户端必须要进行一些特别的设置才能使用正向代理。如下图所示：

从上面的概念中，我们看出，文中所谓的正向代理就是代理服务器替代访问方【用户A】去访问目标服务器【服务器B】这就是正向代理的意义所在。而为什么要用代理服务器去代替访问方【用户A】去访问服务器B呢？

A无法直接访问服务器B，譬如被拦截了，gfw。
加速。A通过代理访问B，速度更快。
cache作用。代理服务器可能已经缓存了内容。

其实翻墙的大部分内容就涉及到正向代理技术。

我们总结一下正向代理是一个位于客户端和原始服务器(origin server)之间的服务器，为了从原始服务器取得内容，客户端向代理发送一个请求并指定目标(原始服务器)，然后代理向原始服务器转交请求并将获得的内容返回给客户端。客户端必须设置正向代理服务器，当然前提是要知道正向代理服务器的IP地址，还有代理程序的端口。正向代理是从客户端的角度来分析问题的，这正是”正向“的本义。

#反向代理（reverse proxy）反向代理正好与正向代理相反，对于客户端而言代理服务器就像是原始服务器，并且客户端不需要进行任何特别的设置。客户端向反向代理的命名空间(name-space)中的内容发送普通请求，接着反向代理将判断向何处(原始服务器)转交请求，并将获得的内容返回给客户端。反向代理是从服务器端来分析问题的，这正是”反向“的本义。反向代理的作用如下：

###保护和隐藏原始资源服务器B

用户A始终认为它访问的是原始服务器B而不是代理服务器Z，但实用际上反向代理服务器接受用户A的应答，从原始资源服务器B中取得用户A的需求资源，然后发送给用户A。由于防火墙的作用，只允许代理服务器Z访问原始资源服务器B。尽管在这个虚拟的环境下，防火墙和反向代理的共同作用保护了原始资源服务器B，但用户A并不知情，A以为自己把请求发送给了B，其实请求被服务器端技术默默地发给了Z。

###负载均衡（CDN）

当反向代理服务器不止一个的时候，我们甚至可以把它们做成集群，当更多的用户访问资源服务器B的时候，让不同的代理服务器Z（x）去应答不同的用户，然后发送不同用户需要的资源。当然反向代理服务器像正向代理服务器一样拥有CACHE的作用，它可以缓存原始资源服务器B的资源，而不是每次都要向原始资源服务器B请求数据，特别是一些静态的数据，比如图片和文件，如果这些反向代理服务器能够做到和用户X来自同一个网络，那么用户X访问反向代理服务器X，就会得到很高质量的速度。这正是CDN技术的核心，我们并不是讲解CDN，所以去掉了CDN最关键的核心技术智能DNS，只是展示CDN技术实际上利用的正是反向代理原理这块。如下图:

反向代理结论与正向代理正好相反，对于客户端而言它就像是原始服务器，并且客户端不需要进行任何特别的设置。客户端向反向代理的命名空间(name-space)中的内容发送普通请求，接着反向代理将判断向何处(原始服务器)转交请求，并将获得的内容返回给客户端，就像这些内容原本就是它自己的一样。基本上，网上做正反向代理的程序很多，能做正向代理的软件大部分也可以做反向代理。开源软件中最流行的就是squid，既可以做正向代理，也有很多人用来做反向代理的前端服务器。另外MS ISA也可以用来在WINDOWS平台下做正向代理。反向代理中最主要的实践就是WEB服务，近些年来最火的就是Nginx了。网上有人说NGINX不能做正向代理，其实是不对的。NGINX也可以做正向代理，不过用的人比较少了。

透明代理

如果把正向代理、反向代理和透明代理按照人类血缘关系来划分的话。那么正向代理和透明代理是很明显堂亲关系，而正向代理和反向代理就是表亲关系了。透明代理的意思是客户端根本不需要知道有代理服务器的存在，它改编你的request fields（报文），并会传送真实IP。注意，加密的透明代理则是属于匿名代理，意思是不用设置使用代理了。透明代理实践的例子就是时下很多公司使用的行为管理软件。如下图：

用户A和用户B并不知道行为管理设备充当透明代理行为，当用户A或用户B向服务器A或服务器B提交请求的时候，透明代理设备根据自身策略拦截并修改用户A或B的报文，并作为实际的请求方，向服务器A或B发送请求，当接收信息回传，透明代理再根据自身的设置把允许的报文发回至用户A或B，如上图，如果透明代理设置不允许访问服务器B，那么用户A或者用户B就不会得到服务器B的数据。

#代理与NAT 代理与NAT可以实现类似的功能，即隐藏地址。但它们是有本质区别的。代理是应用层的，基于的是请求转发；而NAT是IP层的，基于的是IP包转发效率要高一些。NAT的本意也想对所有的协议进行透明转换，但是NAT无法对某些数据包中的IP地址转换；部分通过代理（应用网关）能解决，但部分应用根本无法通过NAT。NAT使每一台机器对应一个相应的“假”地址，但代理服务器把所有内部网络的机器全部映射成自己的地址。而且代理服务器打破了客户机与外部服务器之间点对点连接的这种模式。这也是代理服务器与NAT最大的不同。同时也是代理服务器发展的最大障碍。因为点对点连接是Internet上提供某些服务的基础，打破了这种连接，也就无法实现这些服务，透明代理也有一样的问题。

stardict

2015-05-31T00:00:00+00:00

A Linux dictionary which I think is the best. Easy configuration, and support customized dictionary.

Refer to stardict-3 and www.huzheng.org/stardict, and get an overview of the package.
Installation
```
# emerge -av app-text/stardict
```
When finished installtion, it might remind:
```
* Note: festival text to speech (TTS) plugin is not built. To use festival
* TTS plugin, please, emerge festival and enable "Use TTS program." at:
* "Preferences -> Dictionary -> Sound" and fill in "Commandline" with:
* "echo %s | festival --tts"
* 
* You will now need to install stardict dictionary files. If
* you have not, execute the below to get a list of dictionaries:
* 
*   emerge -s stardict-
```
Overlook this information. Since we will install real people sound. Don't need the TTL simulation one.

stardict depends on gnome-doc-utils package and might fail with error:

xsltApplyStylesheet: saving to C/db-xref.xsldoc may not be possible

This error might be solved by:
1. MAKEOPTS="-j1" emerge -avt app-text/stardict;
2. Run python-updater before emerge.
By default, the installer does not include any dictionaries for copyright issue. So we need to install dictionaries and real people sound.
Dicts

For each dictionary:

$ for filename in /path/to/stardic*.zip; do unzip $filename -d /media/Misc/Dicts/; done
$ for filename in /path/to/stardic*.bz2; do tar vxjpf $filename -C /media/Misc/Dicts/; done
# ln -sv /media/Misc/Dicts /usr/share/stardict/dic/myDicts

The dictionaries and real sound destination could also be ~/.local/share. More flexibly, extract the dcits and sound to Windows partition and create symlinks.

Real people sound:

You are recommended to use 142000个单词语音库.

Stardict only uses WAV sound. For MP3, we should convert to MP3 first.

If pronounce USE is enabled, WyabdcRealPeopleTTS will be installed automatically. The OtdRealPeopleTTS (MP3) sound must be converted to WAV by script included, which finally occupies around 3GiB space.
```
# tar xjvf WyabdcRealPeopleTTS.tar.bz2 -C /media/Misc/
```
If use 142000 sound, configure StartDict to find it. Specially, Stardict find WAV files in 2nd directory layer.
~~Set the fonts to *tahoma* otherwise the sound symbols would not display correctly or naturally~~
.
Set scan:
sdcv is the console version of stardict, sharing the same set of dictionaries in /usr/share/stardict/dic. It's really useful when you cannot find an X environment!

You can even add sdcv to Emacs.

Capital in 21st Century Pickups

2015-05-25T00:00:00+00:00

In fact, nothing can be further from truth.

It's indispensable to make use of complementary sources as well.

It's enlightening to …

by contrast,

There is very reason to believe that …

fare: progress; get on 进展; 过日子: How did you fare (ie What were your experiences) while you were abroad? 你在国外时好吗（感受如何）?

beneficiary: n, person who receives sth, esp one who receives money, property, etc when sb dies 受益者; 受惠者; （尤指）承受遗产者.

nominal: existing, etc in name only; not real or actual 名义上的; 不实际的; 不真实的: the nominal ruler of the country 名义上的国家统治者; (of a sum of money, etc) very small, but paid because some payment is necessary （指一笔钱等）很少的, 象徵性的

foregoing: listed, mentioned, or occurring before ; preceding; just mentioned 在前的; 刚提到的; 上述的: the foregoing analysis, description, discussion, etc 以上分析﹑ 描述﹑ 论述等

the foregoing: n [sing or pl v] (fml 文) what has just been mentioned 刚提及的事物: The foregoing have all been included in the proposals. 以上各点均包括在建议中.

replete with …

in the aftermath of …; its immediate aftermath

touch on/upon sth mention or deal with (a subject) briefly 提及或涉及（某问题）: The matter was hardly touched on. 那件事几乎没涉及到.

seethe: (of liquids) bubble and froth as if boiling （指液体）起泡, 冒泡（似沸腾）: They fell into the seething waters of the rapids. 他们跌进了翻腾的急流中.

be seething with <人> [因生气等而] 大发雷霆[with]; [因不平、不满等而] 举国骚动,群情哗然[with]

It is tempting to see/note … 迷人的,引人的,诱人的; 引人心动 [食欲] 的

stun: [Tn] daze or shock (sb), eg with sth unexpected 使（某人）目瞪口呆或感到震惊: I was stunned by the news of his death. 我得知他的死讯十分震惊

stunning: surprising or shocking 令人惊奇的; 令人震惊的: a stunning revelation 惊人的新发现

far from negligible

hike: n, rise in prices, costs, etc （价格﹑ 价值等的）提高, 增加: The union demands a 7% wage hike. 工会要求提高工资7%

have not failed to attract attention

phenomenon -> pl, phenomena

from the turn of twentieth century until now

dreadful: causing great fear or suffering; shocking 产生极大恐惧或痛苦的; 使人震惊的: a dreadful accident, disease, nightmare 可怕的事故﹑ 疾病﹑ 恶梦

rentier: 靠利息、养老金、地租、股息等生活的人

heady: having a quick effect on the senses; very exciting 迅速作用於感官的; 兴奋的: a heady perfume 气味扑鼻的香水；(of a person) excited and acting rashly （指人）激动得忘乎所以的: be heady with success 因成功而得意忘形

tax heaven 避税天堂

euphoria:[U] intense feeling of happiness and pleasant excitement 愉快和兴奋的感觉: She was still in a state of euphoria hours after her victory. 她获胜後几小时仍喜气洋洋

stock market euphoria

reach unprecedented height/level/degree

incessant: adj not stopping; continual 不停的; 连续的; 不断的: a week of almost incessant rain 雨差不多没有停过的一个星期 * an incessant stream of visitors 络绎不绝的参观人流

remunerate: [Tn, Tn.pr] ~ sb (for sth) (fml 文) pay or reward sb for work or services 酬报某人（为其工作或服务）

remuneration:  [U] (fml or rhet 文或修辞) payment; reward 酬金; 报酬

oscillate: ~ (between sth and sth) (fml fig 文, 比喻) keep moving backwards and forwards between extremes of feeling, behaviour, opinion, etc; waver （感情﹑ 行为﹑ 想法等）波动, 动摇, 犹豫: He oscillates between political extremes. 他的政治观点在两个极端之间摇摆不定；[I, Tn] (cause sth to) move repeatedly and regularly from one position to another and back again （使某物）摆动: A pendulum oscillates. 摆锤能摆动

vacillate: [I, Ipr] ~ (between sth and sth) (fml usu derog 文, 通常作贬义) keep changing one's mind; move backwards and forwards between two emotions （思想）动摇不定; 犹豫; （在两种情绪之间）变化不定: She vacillated between hope and fear.她时而抱有希望, 时而心存恐惧

plummet：[I, Ipr, Ip] fall steeply or rapidly 大坡度或快速落下: House prices have plummeted in this area. 此地房价大跌. * Pieces of rock plummeted down the mountainside to the ground below. 岩石一块块顺着山的陡坡滚落到地面.

scrupulous: ~ (in sth/doing sth) careful not to do wrong; absolutely honest 审慎的; 极诚实的: scrupulous in all her business dealings （她）对所有交易都十分老实

unsrupulous: without moral principles 无道德原则的; 不讲道德的: unscrupulousmethods, behaviour 不道德的方法﹑ 行为

incontestable: adj that cannot be disputed or disagreed with 无可争辩的; 不能不同意的: an incontestable fact 无可争辩的事实

trade deficit; trade surplus

dispel: [Tn] drive (sth) away; cause to vanish 驱走（某事物）; 使消失: dispel sb's doubts/fears/worries 消除某人的疑虑[恐惧/烦恼] * The company is trying to dispel rumours about a take-over. 公司力图澄清有关控制权转移的流言

temporal: of worldly affairs, ie not spiritual; secular 世俗的（即非宗教的）; 现世的；物质世界的或与之有关的；世俗的 the temporal possessions of the Church教堂拥有的世俗的财产；Of, relating to, or limited by time时间的：时间的、与时间有关的或受时间限制的

tempral evolution

supplant: (fml 文) take the place of (sb/sth); replace 取代（某人[某事物]）; 代替: Oil has supplanted coffee as our main export. 我们的主要出口货已由原来的咖啡改为石油了. * The party leader has been supplanted by his rival. 那位党的领导人已被其对手取而代之. * She has been supplanted by another in his affections, ie He now loves sb else. 他爱上了另一个女人, 不再爱她了.

advent: the ~ of sth/sb the approach or arrival of (an important person, event, etc) （重要人物﹑ 事件等的）来临, 到来: With the advent of the new chairman, the company began to prosper. 随着新主席的到来, 公司也开始有了起色

blunt: (of a person, remark, etc) frank and straightforward; not trying to be polite or tactful （指人﹑ 言语等）坦诚的, 直率的, 不客气的, 欠圆通的: a blunt refusal 不客气的拒绝 * Let me be quite blunt (with you) your work is appalling. （对你）直说吧–你的工作太差劲.

rest on: ~ on sb/sth depend or rely on sb/sth 依靠或依赖某人[某事物]; (of a look, etc) be directed steadily at sb/sth （指目光等）停留在某人[某物]上

relegate: [esp passive 尤用於被动语态: Tn, Tn.pr] ~ sb/sth (to sth) dismiss sb/sth to a lower or less important rank, task or state 使某人[某事物]降级﹑ 降职或降低地位: I have been relegated to the role of a mere assistant.

meritocracy: (a) [U] system of government by people of high achievement 英才管理（制度）. (b) [CGp] such people in a society （社会的）英才, 精英, 贤能.

meritocratic: adj

aristocracy: [CGp] highest social class; the nobility 贵族阶级; 贵族: members of the aristocracy 贵族成员；[C] most able or gifted members of any class （任何阶级或阶层的）最优秀的人物: an aristocracy of talent 人杰

aristocratic: adj 贵族的；有贵族气派的; 贵族化的; 装贵族派头的

leave aside the fact that …; leave aside, too, the fact that …

instrumental: [pred 作表语] ~ in doing sth being the means of bringing sth about 作为促成某事物之手段; 有帮助; 起作用: Our artistic director was instrumental in persuading the orchestra to come and play for us. 我们的艺术指导大费唇舌请来管弦乐队为我们演出.

peculiar: odd or strange 奇怪的; 奇异的; 罕有的: a peculiar taste, smell, noise, etc 怪异的口味﹑ 气味﹑ 噪音等；used or practised only by sb/sth 专用的; 特有的: customs peculiar to the 18th century 18世纪特有的风俗习惯

envisage: v [Tn, Tf, Tw, Tg, Tsg] picture (an event, action, etc) in the mind as a future possibility; imagine 展望; 想像: Nobody can envisage the consequences of total nuclear war. 没有人能想像出全面核子战争的後果. * I can't envisage the plan('s) working. 我无法设想计划能否行得通.

buy this argument

Anglo Saxon model 又称“新美国模式”

obscure the fact that …

continental Europe 欧洲大陆，或称大陆的欧洲部分，主要排除英国

stupendous: adj amazingly large, impressive, good, etc 极大的; 极感人的; 极好的: a stupendous mistake, achievement 极大的错误﹑ 成就 * The opera was quite stupendous! 这部歌剧很精彩!

Denmark 丹麦

Nordic countries: 北欧国家

tumult: n, disturbance or confusion, esp of a large mass of people 混乱; （尤指大群人的）骚乱: The demonstration broke up in tumult. 示威集会在纷乱中解散了；disturbed or agitated state of mind; turmoil （思想上的）波动, 烦乱: Her mind was/Her thoughts were in a tumult. 她心烦意乱

tumultuous/chaos/chaotic

at/on sb`s heels; on the heels of sth following closely after sb/sth 紧跟在某人[某事物]後面: The thief ran off with an angry crowd at his heels. 那小偷在前面跑, 一群愤怒的人在後面紧追

plunge: ~ (sth) into sth; ~ (sth) in:

(cause sth to) fall into sth suddenly and with force （使某物）突然而猛力投入﹑ 穿入﹑ 进入等: plunge (one's hand) into cold water （把手）一下子伸进冷水中 
(cause sth to) enter a specified state or condition （使某事物）进入或陷入某状态: The country (was) plunged into civil war after the death of the President. 总统死後全国陷入了内战. 

armament: [C often pl 常作复数] weapons, esp the guns on a tank, an aircraft, etc 武器; （尤指坦克﹑ 飞机等配备的）大炮；武装力量; 军事力量

inpassing: incidentally; in the course of doing something else; by the way;顺便提及

lacuna: (pl. lacunae) section missing from a book, an argument, etc; gap （书籍﹑ 论据等中的）脱漏, 阙文, 缺漏, 空白: a lacuna in the manuscript 原稿中的脱漏

amiss: adj, wrong(ly); inappropriate(ly) 错误; 不恰当: Something seems to be amiss can I help? 好像有点儿不对头–要我帮忙吗?

nadir: (fig 比喻) lowest point; time of greatest depression, despair, etc 最低点; 最压抑﹑ 消沉等的时刻: This failure was the nadir of her career. 这次失败是她事业上的低谷.

come by: obtain；acquire获得；得到 How did you come by that beautiful landscape painting？你是怎样弄到那张漂亮的风景画的？

meager: Deficient in quantity, fullness, or extent; scanty 不足的，缺乏的：在数量、程度或范围上缺乏的；不足的

deteriorate: ~ (into sth) become worse in quality or condition 变坏; 变质; 恶化: Leather can deteriorate in damp conditions. 皮革受潮可变质. * The discussion deteriorated into a bitter quarrel. 这场讨论演变成了激烈的争吵

disaffection, alienation, estrangement 疏远,疏离,不和

all the more to an even greater degree更；格外；更加；越加

all the more so 更加如此
all the more reason to do/for (doing) sth 更有理由。。。
He loved his wife and children，but he loved his country all the more．他爱妻子儿女，但是他更爱祖国
I know you find the subject difficult， but that is all the more reason why yod should work hard at it．我知道你感到这门学科困难，但这就是你更加应该努力钻研它的理由

rudiment: [C] imperfect beginning of sth that is not yet fully developed 初级形态: working on the rudiments of a new idea 研究一种新思想的萌芽；part or organ that is incompletely developed 未充分发展的部分; 发育不成熟的器官

>rudimentary adj.

complacent: adj ~ (about sb/sth) (usu derog 通常作贬义) calmly satisfied with oneself, one's work, etc 自满的; 自鸣得意的: a complacent smile, manner, tone of voice 自满的微笑﹑ 姿态﹑ 声调 * We must not be complacent about our achievements; there is still a lot to be done. 我们绝不能满足於自己的成绩, 还有很多事情要做.

top decile; top centile/percentile; top thousandth;

illiterate: 不会读或不会写的; 不识字的；缺乏教育的; 教育程度很低的

illiteracy: [U] state of being illiterate 文盲; 缺乏教育; 无知

apartheid: 种族隔离(南非共和国之白人对黑人所采取之种族歧视政策)

extravagant: (of ideas, speech or behaviour) going beyond what is reasonable, usual or necessary （指想法﹑ 言行）放肆的, 越轨的, 过度的: extravagant praise, behaviour, claims 过分的赞扬﹑ 放肆的行为﹑ 过高的要求 * pay extravagant compliments 过分夸奖；(in the habit of) using or spending too much; (of actions) showing this tendency （惯於）奢侈的, 挥霍的; （指行为）放纵的: an extravagant man 挥霍无度的人

compensation: (美)报酬,薪水,工资,俸给 (salary)；补偿,赔偿,弥补,偿还,抵偿

till: (银行、商店等的) 放现款用的抽屉; 放贵重物品用的抽屉 have one's fingers/hands in the till 从自己的工作部门偷钱，监守自盗

apt: suitable; appropriate 适当的; 恰当的: an apt quotation/metaphor 恰当的引语/比喻

[pred 作表语] ~ to do sth likely or having a tendency to do sth 易於做某事物; 有做某事物的倾向: apt to be forgetful, careless, quick-tempered, etc 健忘﹑ 总是粗心大意﹑ 动不动就发脾气 * My pen is rather apt to leak. 我的钢笔爱漏墨水.

[not X] any more than [Y]：Y和X一样不 …

per se: by or of itself; intrinsically 本身; 本质上: The drug is not harmful per se, but is dangerous when taken with alcohol. 该药本身并无害处, 但与酒类同服则有危险.

disparate: so different in kind or degree that they cannot be compared 迥然不同的; 无法比较的: The five experiments gave quite disparate results. 这五次试验所获得的结果迥然不同

mete: mete sth out (to sb) (fml 文) give or administer (punishment, rewards, etc) 给予, 加以（惩罚﹑ 奖励等）,分给[人],给与: The judge meted out severe penalties. 法官对犯人予以严惩. * Justice was meted out to the offenders. 犯人均已绳之以法

propensity: n ~ (for/to/towards sth); ~ (for doing/to do sth) (fml 文) inclination or tendency 倾向; 习性: a propensity to exaggerate/towards exaggeration 浮夸的倾向 * a propensity for getting into debt 借债的习性.

mischief: [U] behaviour (esp of children) that is annoying or does slight damage, but is not malicious (used esp as in the expressions shown) 恶作剧, 捣蛋, 顽皮, 淘气（尤用於以下示例）: act out of mischief 调皮的举动

confiscate: [Tn] take possession of (sb's property) by authority, without payment or compensation 没收（某人的财产）; 充公: The headmaster confiscated Tommy's pea-shooter. 校长没收了汤米的射豆枪. * If you are caught smuggling goods into the country, they will probably be confiscated. 假若查出你向该国走私货物, 你的货物准得没收.

levy: [Tn, Tn.pr] ~ sth (on sb) collect (a payment, etc) by authority or force; impose sth 徵收, 徵集（款额等）; 强加某事物: a departure tax levied on all travellers 向所有旅客徵收的离境税

windfall: an unexpected, unearned, or sudden gain or advantage 意外的收获

think tank: 智库，智囊团 A group or an institution organized for intensive research and solving of problems, especially in the areas of technology, social or political strategy, or armament. 智囊：尤指为集中研究并解决在技术、社会或政策及军事领域中的问题而组织起来的群体或机构

worrisome: causing worry; troublesome 令人忧虑的; 令人烦恼的

The top decile of the wealth hierarchy own a clear majority of what there is to own.

candid: adj not hiding one's thoughts; frank and honest 率直的; 坦白而诚实的: a candid opinion, statement, person 直言﹑ 直说﹑ 直性人 * Let me be quite candid with you: your work is not good enough. 不瞒你说, 你的工作不怎麽好

to be candid,
candidly,

probate: [U] official process of proving that a will is correct 遗嘱检验; 遗嘱认证: apply for/take out probate 申请检验遗嘱 * grant probate 通过对遗嘱的认证；[C] copy of a will with an official certificate that it is correct 经认证的遗嘱

[attrib 作定语] a probate court 遗嘱检验法庭

observatory: A structure overlooking an extensive view 观察台，了望台，望楼：俯瞰广阔景象的建筑；A building, a place, or an institution designed and equipped for making observations of astronomical, meteorological, or other natural phenomena 观测所，气象台，天文台：一座建筑或一个机构，用以观测天文的、气象学的或其它自然现象

coffer: 1 (C) (放置金钱等贵重物品的) 箱柜,银柜；2 [~s] (银行等的) 金库; 财源 (funds)；the ~s of the State 国库；3 (C)‘建筑’ (格子、天花板、拱门等的) 饰板,镶板,藻井

regime: (a) method or system of government 统治方式或制度; 政体; 政权: a socialist, fascist, etc regime 社会主义﹑ 法西斯等制度. (b) prevailing method or system of administration (eg in a business) 盛行的管理方式或制度（如商业中的）: changes made under the present regime 现行管理方法带来的变化 * the old regime versus the new 新管理制度对旧管理制度

regiment: 军队中的团； ~ of sth/sb (fig 比喻) large number of things or people 大量的物或人: a whole regiment of volunteers 大批志愿者

colonel: 上校（美国陆军、空军或海军陆战队中校之上准将之下的军衔）

bequest: [n]act of bequeathing 遗赠: the bequest of one's paintings to a gallery 把绘画遗赠给美术馆；thing bequeathed; legacy 遗产; 遗物: leave a bequest of 2000 each to one's grandchildren 留给孙儿每人一笔2000英镑的遗产

bequeath: (fml 文) [Tn, Dn.n, Dn.pr] ~ sth (to sb) 1 arrange, by making a will, to give (property, money etc) (to sb) when one dies 将（财物等）遗赠（给某人）: He bequeathed 1000 (to charity). 他遗赠（给慈善事业）1000英镑. * She has bequeathed me her jewellery. 她把珠宝遗赠给我了

cadastre:an official register of the quantity, value, and ownership of real estate used in apportioning taxes

conveyance: [U] (fml 文) conveying 运送; 传送; 传达; 转让: the conveyance of goods by rail 铁路货运

(law 律) (a) [U] conveying property 转让财产: an expert in conveyance 承办产权转让事务的专家. (b) [C] document that conveys property 产权转让证书: draw up a conveyance 拟就产权转让证书.

draw reasonable conclusions

kin: [pl v] (dated or fml 旧或文) one's family and relatives 家人和亲戚: All his kin were at the wedding. 他的家人和亲戚都参加了婚礼. * He's my kin, ie related to me. 他是我的亲戚. * We are near kin, ie closely related. 我们是近亲；

synonym：kindred
no kin to sb: not related to sb 与某人无亲属关系
kith and kin: 亲友（亲戚和朋友）
kinship:亲戚血缘关系

metamorphose: [I, Ipr, Tn, Tn.pr] ~ (sb/sth) (into sth) (fml 文) (cause sb/sth to) change in form or nature （使某人[某物]）变形, 变质: A larva metamorphoses into a chrysalis and then into a butterfly. 幼虫变为蛹, 然後再变成蝴蝶. * The magician metamorphosed the frog into a prince. 魔术师把青蛙变成了王子

metamorphosis/metamorphoses: change of form or nature, eg by natural growth or development 变形; 变态; 变质 the social metamorphosis that has occurred in China 中国发生的社会变化

rhetoric: 修辞; 修辞艺术; 修辞学: impassioned rhetoric 富有表现力的修辞

(derog 贬) elaborate language which is intended to impress but is often insincere, meaningless or exaggerated 华丽的词藻（常含华而不实之意）; 虚夸的言辞: the empty rhetoric of politicians 政客们的花言巧语

reassure: ~ sb (about sth) remove sb's fears or doubts; make sb confident again 消除某人的恐惧或疑虑; 恢复某人的信心; 使某人放心；

reassuring: adj that reassures 使人消除恐惧或疑虑的; 使人放心的: a reassuring glance, word, pat on the back 使人恢复信心的目光﹑ 话﹑ 轻拍一下背部

reassuring rhetoric

cohort；number of people banded together 一群人，一帮人； each of the ten units forming a legion 步兵大队（军团的十分之一）

suffrage：n [U] right to vote in political elections （政治性选举的）选举权, 投票权: universal suffrage, ie the right of all adults to vote 普选权（成年人均有的投票权） * Women had to fight for their suffrage. 妇女须为享有选举权而斗争

inexorable：adj continuing unstoppably; relentless 不可阻挡的; 坚持不懈的; 无情的: inexorable demands, pressures, etc 无可变更的要求﹑ 无情的压力 * the inexorable march of progress 势不可挡的进展 * the inexorable decline of wealth share

procure: [Tn, Dn.n, Dn.pr] ~ sth (for sb) (fml 文) obtain sth, esp with care or effort; acquire 取得某事物（尤指费心或费力）; 获得: The book is out of print and difficult to procure. 那书已绝版, 很难弄到手； [I, Tn, Dn.pr] ~ sb (for sb) (derog 贬) find (prostitutes) for clients 为宿娼者介绍（娼妓）; 拉皮条: He was accused of procuring women for his business associates. 他被指控为其生意合伙人招妓

interim：n. (idm 习语) in the interim, the time between one event, process, or period and another 在其间; 在其时；adj. serving during an intermediate interval of time 暂时的; 临时的 interim arrangements, measures, proposals, etc 临时的安排﹑ 措施﹑ 建议等

heir: 继承人

renounce：[Tn] agree to give up ownership or possession of (sth), esp formally 同意放弃（某事物）的所有权或占有权（尤指正式地）: renounce a claim, title, right, privilege 宣布放弃要求﹑ 头衔﹑ 权利﹑ 特权；[Tn] refuse to associate with or acknowledge (esp sth/sb with a claim to one's care, affection, etc) 拒绝与（某事物[某人]）发生联系; 与（某事物[某人]）断绝关系: renounce a friendship 绝交 * He renounced his son (as an unworthy heir). 他与儿子断绝了父子关系（认为他不配作继承人）.

renounce inheritance

abject: (of conditions) wretched; hopeless （指境况）凄惨的, 绝望的: living in abject poverty/misery 过着极穷困[悲惨]的生活

in abject poverty

dun: [Tn] persistently demand payment of a debt from (sb) （不断地向某人）讨债; 催讨债款

budge；(cause sth to) move slightly （使某物）稍微移动, 动一动: My car's stuck in the mud, and it won't budge/I can't budge it. 我的汽车陷入泥中, 一动也不动[我无法使它移动].

agrarian: adj [usu attrib 通常作定语] (of the cultivation or ownership) of land （指耕种或所有权）土地的: agrarian laws, problems, reforms 土地法﹑ 问题﹑ 改革

the Middle Ages; antiquity(usually before the Middle Ages)

state of affairs: If you refer to a particular state of affairs, you mean the general situation and circumstances connected with someone or something. e.g. This state of affairs cannot continue for too long, if parliament is to recover.

in the wake of: as a result of；right after；following由于；紧随

There were many troubles in the wake of the war．战后许多麻烦接踵而来。
There were heavy losses of property in the wake of the flood．由于洪水使得财产受到重大损失。
Traders came in the wake of the conquering armies．商人随着征服的军队而来。
The students followed in the wake of the teacher．学生们跟在老师的后面。

for the sake of: for the purpose of; because of 为了；因为 *We must be patient for the sake of peace．为了和平，我们必须有耐心。

The overall trajectory is fairly similar

minuscule: adj very small; tiny 极小的; 微小的

caveat: (fml 文) warning; proviso 警告; 限制性条款: I recommend the deal, but with certain caveats. 我介绍这笔交易, 但有几项要提请注意.

supplement VS complement: Supplement would refer to extra/additional information in this context对被补充对象的一种加强，两者属于同一个context. Complement refers to an item that completes or goes well with another item，两者输入不同的context，或者说是两个不同性质的东西，但是两者可以在一起相互作用，产生的更好的结果.

be accustomed to the fact that …

dynamism: (in a person) quality of being dynamic （人的）精力, 活力, 干劲

decry: (pt, pp decried) [Tn, Cn.n/a] ~ sb/sth (as sth) speak critically of sb/sth to make him/it seem less valuable, useful, etc; disparage sb/sth 诋毁某人[某事物]以贬低其价值: He decried her efforts (as a waste of time). 他贬低她所作的努力（认为是浪费时间）.

sanctuary: 圣域,避难所

ineluctable: not to be avoided, changed, or resisted : INEVITABLE 不可避免的; 难免的

ground: [I, Ipr, Tn, Tn.pr] ~ (sth) (in/on sth) (of a ship) touch the sea bottom; cause (a ship) to do this （指船）触海底, 搁浅; 使（船）触海底﹑ 搁浅: Our ship grounded in shallow water/on a sandbank. 我们的船在浅水中[在沙滩上]搁浅

ground: [C esp pl 尤作复数] ~ (for sth/doing sth/to do sth) reason(s) or justification for saying, doing or believing sth 说﹑ 做或相信某事的原因或理由: You have no grounds for complaint/for complaining. 你没有抱怨的理由. * If you continue to behave like this you will give them/provide them with grounds for dismissing you. 你照这样下去就让他们找到辞退你的理由了. * Desertion is a ground (ie legally sufficient reason) for divorce. 被配偶遗弃是离婚的充足理由. * They had no grounds to arrest him. 他们没有理由逮捕他.

in a sense: in a way；in one respect在某种意义上

nostalgia: [U] sentimental longing for things that are past 对往事的怀恋; 怀旧

heyday: [sing] time of greatest success, prosperity, power, etc 最成功﹑ 最繁荣﹑ 最强盛等的时期: She was a great singer in her heyday. 她在自己的黄金时代是个了不起的歌唱家. * Steam railways had their heyday in the 19th century. 19世纪是蒸汽机车的全盛时期.

one's heyday, heyday of

curb: [Tn] prevent (sth) from getting out of control; restrain 防止（某事物）失控; 约束: curb one's anger, feelings, etc 抑制怒火﹑ 感情等 * curb spending, waste, etc 限制开支﹑ 浪费等. [n] strap or chain passing under a horse's jaw, used to restrain the horse 马勒, 马嚼子（横放在马嘴里的皮带或铁链, 用以驾御马匹）.

supremacy: 至尊,至上,无上; 最高位

controvert: [Tn] (fml 文) deny the truth of (sth); argue about 否定（某事物）; 争论; 反驳: a fact that cannot be controverted 无可置辩的事实

controversy: a discussion marked especially by the expression of opposing views (有关意见分歧大的问题等长期性的,尤其以文字发表的) 争论,论战,争吵
incontrovertible: adj so obvious and certain that it cannot be disputed or denied 无可辩驳的; 不容否认的: incontrovertible evidence 确凿无疑的证据
It is an incontrovertible history reality that ...

subsist: [I, Ipr] ~ (on sth) (fml 文) (continue to) stay alive, esp with little food or money; exist （继续）存活, 活下去; （尤指藉微少的食物或金钱）维持生活; 生存: How do they manage to subsist (on such a low wage)? 他们（区区这点工资）怎样餬口? * He subsisted mainly on vegetables and fruit. 他主要依靠蔬菜和水果维持生命.

subsistence: [U] (means of) subsisting 存活; 生存; 生计; 维生之道 * [attrib 作定语] subsistence farming, ie farming that produces only enough crops for the farmer and his family to live on, leaving no surplus which could besold 自给农业（收成仅供自家食用, 无剩余出售者） * a subsistence wage, ie one that is only just enough to enable a worker to live 勉强餬口的工资

projection: [C] estimate of future situations or trends, etc based on a study of present ones 对未来形势的估计; 预测; 推断: sales projections for the next financial year 对下一财政年度销售情况的预测.

foreordain: [usu passive 通常用於被动语态: Tn, Tf] (fml 文) (of God or fate) arrange or determine (sth) before it actually happens （指上帝或命运）注定: It was foreordained that the company would suffer a spectacular collapse. 这个公司注定要彻底垮台.

ordain: [Tn, Cn.n] make (sb) a priest or minister 任命（某人）为司铎﹑ 神父或牧师: He was ordained priest last year. 去年授予他司铎一职； [Tn, Tf] (fml 文) (of God, law, authority, fate, etc) order or command; decide in advance （指上帝﹑ 法律﹑ 权威﹑ 命运等）命令, 规定, 注定: Fate had ordained that he should die in poverty. 命中注定他得死於贫困

NOTE ON USAGE 用法: When talking about giving orders, decree and dictate can be used of individuals in positions of authority. 谈到发命令, decree和dictate均可指下达者为掌权的个人. Decree usually suggests the public announcement of a decision made by a ruler or government without consulting others *decree通常指统治者或政府不与别人商议而公开宣布的决定: The dictator decreed that his birthday would be a public holiday. 该独裁者发布命令, 将其生日定为公众假期. Dictate indicates people using their power over others *dictate用以指人用权力支配他人: Her skills were in such demand that she could dictate her own salary. 她因其技术奇货可居, 故可以主宰自己的薪金待遇. Ordain and prescribe suggest a more impersonal authority such as the law. *ordain和prescribe多用以指如法律等非个人的权力. Ordain is formal and can be used of God *ordain含庄重色彩, 可用以指上帝: Is it ordained in heaven that women should work in the home? 女人就该在家里干活儿, 这是上天的旨意吗? Prescribe is used of the law *prescribe用於法律方面: Regulations prescribe certain standards for building materials. 有规章规定建筑材料的某些标准.

net: adj, ~ (of sth) remaining when nothing more is to be taken away 净的; 纯的: a net price, ie one from which a discount has been deducted 实价 * net profit, ie one that remains when working expenses have been deducted 纯利 * net weight, ie that of the contents only, excluding the weight of the wrappings, the container, etc 净重 * What do you earn, net of tax (ie after tax has been paid)? 你完税後净得多少?

[attrib 作定语] (of an effect, etc) final, after all the major factors have been considered （指结果等）最後的, 最终的: The net result of the long police investigation is that the identity of the killer is still a complete mystery. 警方经长时间调查, 结果凶手的身分仍全然不知

concatenation: ~ (of sth) (fml 文) series of things or events linked together 一系列互相关联的事物: an unfortunate concatenation of mishaps 接连的不幸.

dubious: [esp pred 尤作表语] ~ (about sth/doing sth) not certain and slightly suspicious about sth; doubtful 半信半疑; 可疑: I remain dubious about her motives. 我对她的动机仍存疑念.

recapitulate: (also infml 口语作 recap) v [I, Tn, Tw] state again or summarize the main points of (a discussion, etc) 重述或概括（讨论等）的要点: Let me just recapitulate (on) what we've agreed so far. 让我仅扼要重述一下到目前为止我们已取得的一致意见.

口语中一般用recap

proposition: [n] ~ (that…)statement that expresses a judgement or an opinion; assertion 观点; 见解; 主张: The proposition is so clear that it needs no explanation. 观点十分明确, 无须解释；[Tn] propose sexual intercourse to (a woman), esp in a direct and offensive way 向（女方）提出性交要求（尤指直言而放肆）; 求欢: She was propositioned several times in the course of the evening. 整个晚上有人几次向她提出非分要求

propose

tautology: (a) [U] saying the same thing more than once in different ways withoutmaking one's meaning clearer or more forceful; needlessrepetition 无谓的重复; 赘述. (b) [C] instance of this 重复的话; 赘言

tautological adj 不必要的反复使用,同字 [同义字] 之重复,重复的,赘述的,赘言的

jargon: [U] (often derog 常作贬义) technical or specialized words used by a particular group of people and difficult for others to understand 术语; 行话; 切口: scientific jargon 科学术语 * She uses so much jargon I can never understand her explanations. 她用的术语太多, 我怎麽也听不懂她的讲解

placate: [Tn] make (sb) less angry; soothe or pacify 安慰, 抚慰（某人）; 使（某人）平静﹑ 息怒

placatory: <言行等>安抚的,怀柔的,安慰的,抚慰的,和解的: placatory remarks 安抚的话
implacable: adj, incapable of being placated, that cannot be changed or satisfied 不能变动的; 无法平息或安抚的: implacable hatred, fury, opposition 无法消解的仇恨﹑ 愤怒﹑ 对立 * an implacable enemy, rival, etc 势不两立的敌人﹑ 竞争对手等

amass: [Tn] gather together or collect (sth), esp in large quantities （尤指大量地）积累, 积聚, 收集（某事物）: amass a fortune 积累财富 * They amassed enough evidence to convict him on six charges. 他们搜集了足够的证据, 宣判他有六条罪状

confluence: (fml 文) coming together, esp of large numbers of people 汇合, 汇集（尤指大群的人）.

confluent: adj,

distinct: adj easily heard, seen, felt or understood; definite 清楚的; 清晰的; 明显的; 明白的; 明确的: The footprints are quite distinct; they must be fresh. 足迹清晰易辨, 一定是不久前留下来的.

distinctive: adj ~ (of sth) that distinguishes sth by making it different from others 特别的; 有特色的: a distinctive appearance, style, smell 特殊的外表﹑ 风格﹑ 气味 *

这个单词突出有特点，有别于其它的优点，peculiar，characteristic。
与distinct，是完全不同的意思，apparent obvious evident manifest definite。

vertigo：[U] feeling of losing one's balance, caused esp by looking down from a great height; dizziness 眩晕; （尤指）高处俯视性眩晕; 头晕: suffer from (an attack of) vertigo 感到（一阵）眩晕

vertiginous: adj
dizziness

primogeniture: [U] fact of being a first-born child 长嗣身分；(also right of primogeniture) (law 律) system of inheritance by which an eldest son receives his parents' property 长子继承权

diminish: I Tn become smaller or less; decrease 变小; 变少; 使（某事物）变小; 使（某事物）变少; 缩小; 减少; 降低: His strength has diminished over the years. 经过这许多年月, 他的体力不如从前了.

entail: [Tn] make (sth) necessary; involve 使（某事物）必要; 牵涉；必然伴有,需要，使必需, 使蒙受, 使承担: This job entails a lot of hard work. 这项工作需要十分努力. * That will entail an early start tomorrow morning. 那就需要明晨很早动身；[esp passive 尤用於被动语态: Tn, Tn.pr] ~ sth (on sb) (law 律) leave (land) to a line of heirs in such a way that none of them can give it away or sell it 限定（地产）继承人: The house and estate are entailed on the eldest daughter. 这所房子和地产限定由长女继承. * He would have sold the property long ago had it not been entailed. 这些财产若非限定继承的话, 他早就卖掉了

entail:[n] [U] practice of entailing land （地产的）限定继承 [C] entailed property 限定继承的财产

misfortune: [U] bad luck 不幸; 厄运: suffer great misfortune 遭到极大不幸； [C] instance of this; unfortunate condition, accident or event 不幸事故; 灾难; 灾祸

spinster: (a) (law or fml 律或文) unmarried woman 未婚女子. (b) (often derog 常作贬义) woman who remains single after the usual age for marrying 老处女

bachelor: 单身汉

whim: [n] sudden desire or idea, esp an unusual or unreasonable one; caprice 一时的兴致; 突然的念头; （尤指）突发奇想, 异想天开; 心血来潮: It's only a passing whim, ie one that will soon be forgotten. 这只不过是一闪之念. * They seem ready to indulge (ie satisfy) his every whim. 他们简直惯着他随心所欲.

detriment: n (idm 习语) to the detriment of sb/sth; without detriment to sb/sth harming/not harming sb/sth 对某人[某事物]有害 [无害]; 有损於[无损於]某人[某事物]: He works long hours, to the detriment of his health. 他长时间地工作, 有损健康. * This tax cannot be introduced without detriment to the economy. 这一税制一旦施行, 必然会危害国民经济.

detrimental: adj

Civil Code 民法

equal –> equality: 等同性, 同等, 平等, 相等, 等式 –> equilibrium: A condition in which all acting influences are canceled by others, resulting in a stable, balanced, or unchanging system.See Synonyms at balance 平衡，均衡：一种所有动作的影响都互相抵消，整个系统处于平稳的、均势的、不变的状态

egalitarian: [n, adj] n, adj (person) showing or holding a belief in equal rights, benefits and opportunities for everybody 平等主义的; 平等主义者: an egalitarian attitude to voting 对投票持平等主义的态度.

inegalitarain: adj Marked by or accepting of social, economic, or political inequality. 不平等的：具有或接受社会的、经济的或政治不平等性的

splendor: Magnificent appearance or display; grandeur 华丽，壮丽：辉煌华丽的表面或陈设；壮观

zeal: [U] (fml 文) ~ (for sth) (usu intense) energy or enthusiasm; keenness （通常为高度的）热忱, 热情; 热心: show zeal for a cause 表现出对一事业的热忱 * work with great zeal 热情洋溢地工作 * revolutionary,religious zeal 革命的﹑ 宗教的热诚

butler: [n] chief male servant of a house, usu in charge of the wine-cellar 男管家（通常负责管理酒窖）,大管家,家务中的仆役长

pussy: 小猫咪; 女性的阴部;

kitten: young cat 小猫.

compatriot：[n] person who was born in, or is a citizen of, the same country as another; fellow-countryman 同胞; 同国人

fascist: [U] extreme right-wing dictatorial political system or views, esp (Fascism) as originally seen in Italy between 1922 and 1943 法西斯主义

fscist: [C][常 F~]法西斯主义者,法西斯党员

capitalize: (phr v) capitalize on sth, use sth to one's own advantage; profit from sth 利用某事物; 从某事物中获利: capitalize on the mistakes made by a rival firm 从对方公司的错误中获益

hail: [Cn.n/a] ~ sb/sth as sth enthusiastically acknowledge sb/sth as sth 热情地承认某人[某事物]为…: crowds hailing him as king, as a hero 拥他为王﹑ 赞他为英雄的群众 * (fig 比喻) The book was hailed as a masterpiece/as masterly. 这本书被誉为杰作.

thereabouts: (also US thereabout) adv ((usu after or 通常用於or之後) near that number, quantity, time, etc 近於（某数目﹑ 数量﹑ 时间等）; 大约; 左右: I'll be home at 8 o'clock or thereabouts. 我8点钟回家, 8点左右吧；somewhere near there 在附近的某地: The factory is in Leeds or somewhere thereabouts. 那工厂在利兹, 也许在利兹附近

wary: adj (-ier, -iest) ~ (of sb/sth) looking out for possible danger or difficulty; cautious （对可能发生的危险或困难）留意的, 小心的, 警惕的: keep a wary eye on sb 密切注意某人 * She was wary of strangers. 她很警惕陌生人. * be wary of giving offence 唯恐冒犯他人

poles apart/asunder: completely different相差甚远 *They were in agreement on most world issues，but they were poles apart about the war．他们对于多数世界大事意见是一致的，但对于战争的看法却截然相反。 *Their opinions are poles asunder．他们的意见截然相反。 *The two brothers are poles apart in character．这兄弟二人性格迥异。

In fact, nothing can be further from truth.

outset: at/from the outset (of sth) at/from the beginning (of sth) 开端; 开始: At the outset of her career she was full of optimism but not now. 她事业伊始十分乐观, 但现在已今非昔比了. * From the outset it was clear that he was guilty. 一开始就很清楚: 他有罪

verge: [n] (idm 习语) on/to the verge of sth, at or close to the point where sth new begins or takes place 在某事即将发生之际; 接近出现某事物之点: on the verge of war, success, bankruptcy 在接近战争爆发﹑ 成功﹑ 破产之际 * Her misery brought her to the verge of tears. 她难过得快要哭了

on/to the verge of sth

default: [U] (esp law 尤用於法律) failure to do sth, esp to pay a debt or appear in court 不做某事; （尤指）不还债, 不出庭，不履行债务，‘法律’不出庭,不到案

go into default 变成不履行债务
(idm 习语) by default：because the other party, team, etc does not appear 因对方﹑ 他队等未出场: win a case/a game by default 因对方未出席[未出场]而赢得诉讼[比赛]. 
in default of sth/sb (fml 文): because or in case sth/sb is absent 因某物[某人]未在场; 在无某物[某人]时: He was acquitted in default of strong evidence of his guilt. 因无确凿证据而判他无罪. * The committee will not meet in default of a chairman. 没有主席委员会就不开会.

on the verge of default 赖账的边缘

prodigious: adj very great in size, amount or degree, so as to cause amazement or admiration; enormous （在体积﹑ 数量或程度上）大得令人惊叹; 巨大的: a prodigious achievement 巨大的成就 * It cost a prodigious amount (of money). 这用了一笔巨款

prodigiously: adv

emblem: [n] object that represents sth; symbol 象徵; 标志; 标记: The dove is an emblem of peace. 鸽子是和平的象徵. * The ring was important to her as an emblem of their love. 这枚戒指是他们爱情的象徵, 对她至为重要. * The thistle is the emblem of Scotland. 蓟是苏格兰的标志.

emblematic: adj [usu pred 通常作表语] ~ (of sth) (fml 文) serving as an emblem; symbolic 用作象徵; 作为标志

solidarity: [U] unity and agreement resulting from shared interests, feelings, actions, sympathies, etc 团结; 一致: national solidarity in the face of danger 危难当头举国团结一致 * `We must show solidarity with the strikers,' declared the student leaders. 学生领袖声称: ‘我们要申明与罢工工人团结一致.’

solidarity tax: 团结税

concentration of wealth

spearhead: 及物动词 to be the leader of (a movement, for example): 率先：做…的先锋（如运动）：

discern: [Tn] see (sth) clearly (with the senses or the mind), esp with an effort 看出, 识别, 辨认（某事物）（尤指需费力）: In the gloom I could only just discern the outline of a building. 在黑暗中我只能依稀分辨出一座建筑物的轮廓来. * One can faintly discern the flavour of lemon. 可以隐约觉得有一点柠檬味. * discern sb's true intentions 弄清某人的真实意图

upshot: [sing] the ~ (of sth) the final result or outcome 最後结果; 结局；结果,结局,结论: The upshot of it all was that he resigned. 结果他辞职了

rejoice: [I, Ipr, It] ~ (at/over sth) (fml 文) feel or show great joy 极欢喜; 极高兴; rejoice over a victory 为胜利而欣喜 * rejoice at sb's success 为某人的成功而高兴 * I rejoice to hear that you are well again. 得知你身体康复, 甚为欣慰. * We rejoiced that the war was over. 我们为战争结束而欢欣鼓舞.

rejoice in：(joc 谑) have or glory in (a title, etc) 有（某种称号等）; 以（有某种称号等）为荣: She rejoices in the name of Cassandra Postlethwaite. 她为有卡桑德拉?波斯尔思韦特这样的名字而引以为荣.

Wine (staging) QQ

2015-05-15T00:00:00+00:00

You cannot get rid of QQ even under Linux system. First install wine; then QQ. But a lot of extra configuration needs tunning.

Install lastest Wine

# echo "~app-emulation/wine-1.7.46 ~amd64" > /etc/portage/package.accept_keywords/wine

Wine has many bugs and lacks features, and as such we'd better enable the staging USE flag. As Gentoo Wiki suggests, staging flag Installs the external patchset from Wine-Staging. Includes many bugfixes and features, including better performance in gaming.

# echo "app-emulation/wine staging" » /etc/portage/package.use/wine

# emerge -av app-emulation/wine

Actually from Wine-Staging, we can install multiple Wine instances in parallel, for example from overlay wine-overlay which is binary package. So it won't take that long for emerge. For this case, I only choose the portage method instead. I don't want two Wine instances.

Wines needs abi_x86_32 support. It will reminds you update package.use for wine.

This will take around nearly 6 hours since it needs nearly 85 dependencies, we'd better emerge it at night.
Wine QQ @ phpcj.org or Wine QQ @ allog.ml.

From above two links, get wineQQ7.3L.tar.xz. Don't forget -C option.

$ tar xvf /path/to/wineQQ7.3L.tar.xz -C ~/

This command will extract three directories:
```
.wine
.local
.fonts
```
The .wine directory is actually where Wine stores user configuration and program files. This extracted .wine is fine-tuned for the our QQ 7.3.

Though .local fonts exists in current home directory, extract command wont delete any existing files. Instead, it only add the new files to there corresponding sub-directories. Don't worry!

We don't need the .fonts directory since fonts on my system is already configured pretty good.

$ rm -rf ~/.fonts
Attention:

We just installed Wine, but did NOT configure it yet. Instead, we use the fine-tuned configuration .wine from wineQQ7.3L.tar.xz.

Up to now, QQ was installed. You can check it from the Application menu -> Internet -> Wine QQ. Don't launch it at this moment.
Disabling the Menubuilder before installing or lanunching any Wine applications.

Wine will ruin your user file association. For example, on my Xfce4 and Thunar desktop, Wine sets notepad (installed along by default) as the default plain text file editor. What is more, Wine adds an right-click emnu Open with Notepad. Actually, the notepad of Wine looks ugly and does not support Chinese as well. So this step should be the very 1st thing after emerging to prevent Wine from adding menu entries and desktop links.
1. From Gentoo Wiki for per-user bash environment.
```
# File: ~/.bashrc
# Prevent Wine from adding menu entries and desktop links. The 'd' argument means 'disable'.
export WINEDLLOVERRIDES='winemenubuilder.exe=d'
```
2. Froom Wine FAQ, we can achieve the same goal as in step 1. We can check the result by searching winemenubuilder.exe in ~/.wine/user.reg:
  
  "winemenubuilder.exe"=""
  
  Personally, I prefer the Wine FAQ way, though I have adopted both.
3. We can actually apply this setting to system-wide (i.e. for root, for multiple accounts) according to the 2nd step of How to prevent Wine from adding file associations?. This will update ~/.wine/system.reg file as well.
  
  Attention, we would lost changes to /usr/share/wine/wine.inf if Wine were updated, while the relevant update in ~/.wine/systemreg will remain untouched.
4. If you have already installed Wine and run App without Disabling the Menubuilder bofore, just rm -rf ~/.wine and any relevant configurations under home directory as illustrated in those references.
Run QQ: Applications Menu -> Internet -> Wine QQ

If this is the first time to launch Wine related application, then Wine reminds you waiting for updating ~/.wine directory.

Why? Though we use the fine-tuned configuration of Wine QQ. But Wine was just installed and does not know anything about it. It carry on its own initialization. But don't worry, this initilization process will keep the previously fine-tuned config items.

Continuous search for less/man command

2015-05-14T00:00:00+00:00

When reading man or less command information, you need to search contents with /pattern. If continuous search needed:

n: continuous forward search

N: continous backward search

tar

2015-05-13T00:00:00+00:00

"tar" stands for tape archive. It is an archiving file format.

List archive

~ $ tar -tvf hongkan.tar.xz

Create archive

~ $ tar -cJpvf hongkan.tar.xz hongkan/

c: create tar file;
J: compress the tar file with xz;

Alternatives are: z for gzip, j for bzip2.
p: extract information about file permissions;
v: verbosely list files processed;
f: archive file name.

Encrypt/sign archive

~ $ gpg --recipient you@email.com --sign --encrypt hongkan.tar.xz
~ $ gpg --output hongkan.tar.xz --decrypt hongkan.tar.xz.gpg

Extract archive

~ $ tar -tvf hongkan.tar.xz
~ $ tar -xJpvf hongkan.tar.xz [hongkan/test.md]

Update archive

# edit hongkan/test.md
~ $ unxz hongkan.tar.xz
~ $ tar --delete -vf hongkan.tar hongkan/test.md
~ $ tar -rpvf hongkan.tar hongkan/test.md
~ $ xz hongkan.tar

r: append files to the end of an archive.

There is no real way to replace an existing file with the same name. Even the -u --update parameter does not help.

We must first --delete and then -r --append the file to archive.
Cannot update compressed archives. Decompress tar first.

Firewall

2015-05-12T00:00:00+00:00

Concepts

The paramount is to get your own VPS (virtual private server) - a server located outside of the firewall. VPS can be bought from commercial supplier. If you possess a PC outside of the firewall, that can be treated as your own VPS. Most users don't own a physical PC outside the firewall, which is an awkward situation urging them buying VPS.
Then you can deploy VPN or Shadowsocks (ss) servers on your own VPS.
Up to now, a local VPN or ss needs installed.
After that, you can penetrate the firewall through VPN or ss.
VPN and ss are parallel services. You only need one of them unless you would like to switch between these two services for better QoS.
1. ss支持区分国内外流量，传统VPN在翻出墙外后访问国内站点会变慢.
2. ss use small number of memory.
Choose ss instead of VPN.

VPS

There are many VPS suppliers of which I chose bandwagonhost.com or the so-called "搬瓦工". bandwagonhost is easy to manage by web portal including installing VPN or ss server, offering different kinds of pricing package specifying RAM, DISK, BANDWIDTH etc.

My choice is $9.99 USD annually. Of course, it will reminds you to register your web account before paying through paypal.

Pay attention to the pricing link which is an inviting link. If you buy VPS through bandwagonhost.com, you might not locate the $9.99 USD annually pricing package.

You need to wait for a few minutes for VPS system initialization. The default VPS system is CentOS6 x86. You can also reinstall or choose a different operating system.

bandwagonhost: the web portal login. The most important page is Services -> My Services.
1. You can also click on KiviVM Control Panel to get to the 2nd step.
KiviVM Control Panel: VPS management page. Briefly go through the management panel.
1. The first tool I avail of is two-factor authentication (iOS Google Authenticator) thus another temporary code is required for each login into KiviVM.
2. Since CentOS6 x86 root password is not send through email any more, generate root password through root password modification on the lest panel. You can now SSH into your VPS Centos with clients like Putty, MobaXterm and even SSH command line. Attention: the default SSH port is different from normal 22. You can get the port from Main controls.
  
  Don't use the root password often (create a new user account for daily operation, see below). If you need root privilege, just generate a new one!
3. Under KiviVM password modification, set password for KiviVM Control Panel.
You can SSH into VPS often to do work. First SSH as root, then create a daily use user account.
```
 adduser username
 passwd username
```
Then SSH as username for daily operation.
```
 ssh -p xxx username@host
```
A better way of SSH, please refer to OpenSSH.

ss server Python version

At the bottom of KiviVM Control Panel lies KiviVM Extras from which you find Shadowsocks Server. What a relief! You no longer are bothered installing ss server manually. The default is Shadowsocks Python version. After installing finished, click Go back. Instructions on setting ss client for Windows system is illustrated. ss server will run automatically after the automatic installation.
1. Command ps -ef | grep ssserver will print the ss server command:
```
root       415     1  0 01:21 ?        00:00:00 /usr/bin/python /usr/bin/ssserver -p 443 -k PASSWORD -m aes-256-cfb --user nobody --workers 2 -d start
nobody     417   415  0 01:21 ?        00:00:02 /usr/bin/python /usr/bin/ssserver -p 443 -k PASSWORD -m aes-256-cfb --user nobody --workers 2 -d start
nobody     418   415  0 01:21 ?        00:00:02 /usr/bin/python /usr/bin/ssserver -p 443 -k PASSWORD -m aes-256-cfb --user nobody --workers 2 -d start
root       822   708  0 07:03 pts/0    00:00:00 grep ssserver	
```
2. ssserver -h will show help information
3. -d means run as a daemon in the background.
4. --workers 2 will generate two processes belong to user nobody
5. This automatic method does not run ssserver with a configuration file, but with bare command line arguments.
6. ss server was set to run at boot. Let's check:
  
  [root@localhost ~]# ls /etc/rc.local -al
  
  [root@localhost ~]# cat /etc/rc.d/rc.local
  
  /usr/bin/ssserver -p cat /root/.kiwivm-shadowsocks-port -k cat /root/.kiwivm-shadowsocks-password -m cat /root/.kiwivm-shadowsocks-encryption –user nobody –workers 2 -d start
  
  Pay attention to the command line arguments are stored in /root/.kiwivm-shadowsocks-* files.
7. Add option --forbidden-ip 127.0.0.1,::1 to the /etc/rc.d/rc.local file ssserver command for security reason.

Of course, you can also install ss server manually. Among the others, there mainly four versions of ss server: Python version, C libev version, Go version, and C++ with Qt version. Take the Python version as an example:

yum install python-setuptools && easy_install pip
pip install shadowsocks

/usr/bin/python /usr/bin/ssserver -p 443 -k PASSWORD -m aes-256-cfb --user nobody --workers 2 -d start as step 1.
ssserver can also run with -c parameter to specify a configuration file rather than command parameters.
ssserver -c /path/to/shadowsocks.json -d start for example.

A simple configuration file /etc/shadowsocks.json:

{
“server”:”0.0.0.0″,
“local_address”: “127.0.0.1”,
“local_port”:1080,
“server_port”: 8388,
“password”: “password”,
“timeout”:60,
“method”:”aes-256-cfb”,
“fast_open”: false,
“workers”: 2
}

multiple users:

{
“server”:”0.0.0.0″,
“local_address”: “127.0.0.1”,
“local_port”:1080,
“port_password”:
{
 “8388”: “password8388″,
 “8398”: “password8398″,
 “8418”: “password8418″
  },
“timeout”:60,
“method”:”aes-256-cfb”,
“fast_open”: false,
“workers”: 2
}

Shadowsocks server ssserver并没有加入到开机启动，如果需要则要创建一个启动脚本，使其开机启动。
Refer to shadowsocks 2.6.8; VPS之自建shadowsocks服务器（Centos及Ubuntu方法）
特别要注意：CentOS 6的/etc/rc.local里的命令必须要提供全路径，仅仅提供命令名重启时不运行。如要用/usr/bin/ssserver，而不是ssserver.

ss client

###Windows

There are many clients available, my current windows 8.1 client is shadowsocks-csharp. Fill in the encryption method, server port, password, and proxy port for client. The default proxy mode is PAC (Proxy auto-config) not Global.

###Linux

Basically, different shadowsocks on Linux system share serve and client - YES, the same package. For example, my banwagonhost uses Python shadowsocks, while Gentoo uses the same package. After installation, the package will install both server side and client side. Usually server side command is ssserver or ss-server, while client side command is sslocal or ss-local.

# emerge -av shadowsocks

Run sslocal -h to show the detailed help message.

# sslocal -s xx.xx.xx.xx -p yyyy -b 127.0.0.1 -l zzzz -k PASSWORD -m aes-256-cfb -d start

If each time to input this command, then it is tedious, so need to write a script shadowsocks-sslocal.sh. If you'd like, add it to boot:

#!/bin/bash

/usr/bin/python /usr/bin/sslocal -s xx.xx.xx.xx -p yyyy -b 127.0.0.1 -l zzzz -k PASSWORD -m aes-256-cfb --log-file /var/log/shadowsocks-ssloal.log -d start

Move this script to /usr/local/sbin/. Details refer to bin sbin difference. Change access mode to 755 and added to root:root.

So each time, if need get out of GFW:

# /usr/local/sbin/shawdowsocks-sslocal.sh

Up to now, connected to my VPS server! But one step further - foxy proxy for Firefox. After installing, Add New Proxy and Add New Pattern Subscriptioin -> Go.

https://autoproxy-gfwlist.googlecode.com/svn/trunk/gfwlist.txt

Refer to this link for usage: shadowsocks-go. Though this is shadowsocks-go version, but the principle is the similar.

PAC VS Global on WIndows

pac是只对被墙的使用ss，全局就是无论什么网站都用ss。

pac可以自己修改，添加任意网站。当然也可以用网上网友维护的文件，最有名的是就是GFWlist列表。windows下右键点击ss client，出现一个菜单"Update PAC from GFWlist" 。

说的简单点，就是一个被Q网址收集汇总，只要配对上就会走代理，没有配对上就不走代理，这样子就节省了一些流量，包括上网速度等问题。

参考ShadowSocks教程:SS软件中的pac自动代理模式是什么？

VPS credentials

Official web portal client area
KiviVM Control Panel two-factor authentication:
1. KiviVM password
2. Temporary code from Google Authenticator.
CentOS6 x86 root password: ssh into CentOS for OS-specific management.
ss server password for ss client connectoin on Shadowsocks Server of KiviVM control panel.

References

Notes

SwichSharp is no longer needed.
If VPS CentOS restarted, then make sure to run ssserver again as a daemon if you don't set it run on boot.
DO NOT use OpenVPN on VPS. GFW can easily siniff OpenVPN traffic and block VPS IP.
ss优化

Emerge

2015-05-12T00:00:00+00:00

# emerge –resume

Think of a scenario when you have to reboot or shutdown while stilling emerging. You can just Ctrl+C to interrupt the emerging process. Later on when entering the system, run this command before any other emerge-related commands. This will save you a lot of time especially you have a long emerge list. For example, you are in the last one of 100 packages. This command will only continue to emerge the last package.

Verbose mode (-v)

--verbose [ y | n ] (-v short option)
Tell  emerge to run in verbose mode.  Currently this flag causes
emerge to print out GNU info errors, if any, and to show the USE
flags  that  will  be used for each package when pretending. The
following symbols are affixed to USE flags in order to  indicate
their status:


Symbol   Location    Meaning
──────────────────────────────────────────────────────────────

-        prefix      not enabled (either disabled or removed)
*        suffix      transition to or from the enabled state
%        suffix      newly added or removed
()       circumfix   forced, masked, or removed
{}       circumfix   state is bound to FEATURES settings

Manual download.

Everytime emerging a package, the first thing emerge to do is downloading the package source code *.tar.gz, and puting it in /usr/portage/distfiles/.

If the Internet is slow or command line http/https is blocked (Dropbox due to GFW), you can follow the download links from emerge message in terminal to get a copy of *.tar.gz to /usr/portage/distfiles/, and then emerge again.
World set
1. Install a pkg
  
  --select=y -w: This is default action. Emerge a pkg and add to world file.
  
  --oneshot -1: Emerge as normal, but do not add the packages to the world file for later updating.
2. Already installed
--deselect=y: Remove atoms and/or sets from the world file. No installation is involved.

--noreplace: Skips the packages specified on the command-line that have already been installed. One usage is add an installed pkg to world file.
1. --noreplace has other usage like emerge -avtuDN --with-bdeps=y --noreplace firefox @world. This tells not to update firefox.

OpenSSH

2015-05-11T00:00:00+00:00

Overview
Goal
Generate Public-key pair
Distribute the public key
ssh
Passwordless
ssh-find-agent
SSH config
SSH tunnel
Make it simple
Backup
Change passphrase
Reference

Overview

Secure Shell, or SSH, is a cryptographic (encrypted) network protocol to allow remote login and other network services to operate securely over an unsecured network.

In the old ages, telnet protocol traffic is plain text subject to interception. Now services on SSH can be encrypted by different ciphers (i.e. RSA, DSA etc.). In addition to remote login, other services can reside on SSH protocol like content transfer.

protocol (i.e. SSH) is different from implementation (OpenSSH) or command (i.e. ssh, scp etc.). OpenSSH gives tool set like ssh, scp, sftp, ssh-keygen etc.

Goal

In this post, I will setup remote login with Public-key infrastructure to achieve passwordless connection.

local host
remote host
ssh from local host to remote host with key pair.

Generate Public-key pair

The core of Public-key infrastructure is creating personal public and private key pair. The public key literally is distributed publicly over the Internet, while the private key is privately kept by yourself.

On local host that we connect from, run ssh-keygen:

~ $ ssh-keygen -l [ -f ~/.ssh/id_rsa_github.pub ]

~ $ ssh-keygen -t ecdsa -b 384 -o -C "zachary.hu@konghq.com (Kong Dev)" -f ~/.ssh/id_rsa_kh

~ $ ssh-keygen -t rsa -b 4096 -o -C "email@example.com (openssh-rsa-4096)" -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:abcdedajfajqiefafanfaqrnafafaf8rqhfafdafjajfafeaqn rsa-keys
The key's randomart image is:
+---[RSA 2048]----+
|        .+o+o.+==|
|       oE.o..q..=|
|      . o=o......|
|     + o.=o..s. .|
|    . + S . .... |
|     . + . ..q.  |
|    . . +    .   |
|   o ..+     w   |
|  .o==+ +        |
+----[SHA256]-----+

Do NOT use "ssh-dss" (DSA) public key algorithm. Refer to openssh legacy. The default algorithm is rsa such that -t rsa may be omitted..
~/.ssh/id_rsa is the private key (also termed as identity in SSH); ~/.ssh/id_rsa.pub is the public key to be shared over the Internet.
It prompts for passphrase to encrypt private key locally. If someone steal the private key file, he cannot impersonate you to communicate on the Internet. You are encouraged to set a passphrase.

You can just press ENTER key to not use passphrase - empty passphrase to achieve passwordless. This does make a difference when connecting to remote host. Details refer below.

It can be changed afterwards by -p -f argument.
The default key size is 2048 which is subject to modernized brute force break. Use 4096!
The CLI option -o is deprecated. Read What does ssh-keygen "-o" do?.

Distribute the public key

Now share the public key file ~/.ssh/id_rsa.pub over the Internet to remote host. If we lost hte public key, we can restore it from the private key:

~ $ ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

append the public key to ~/.ssh/authorized_keys on remote host.

authorized_keys is a file that stores SSH public keys. Each time a new publick-key pair is generated, append the new public key to it. To accomplish this, run ssh-copy-id:

ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 uname@192.168.0.10

Alternatively, we can use:

cat ~/.ssh/id_rsa.pub | ssh -p 22 uname@192.168.0.10 "mkdir -p ~/.ssh && cat - >>  ~/.ssh/authorized_keys"

192.168.0.10 can also be the domain name of remote host.
Both ways achieve the same purpose: append the public key to authorized_keys file. ssh-copy-id automatically creates ~/.ssh/authorized_keys if it does not exist.
Pay attention to -p argument of mkdir: make parent directories as needed, no error if existing.

No matter which command you chose, you should see something like:

The authenticity of host '[192.168.0.10]:22 ([192.168.0.10]:22)' can't be established.
RSA key fingerprint is SHA256:afjaijfjfajeiqrityqpqvnbvlaeafjadfj.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
uname@192.168.0.10's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -p '22' 'uname@192.168.0.10'"
and check to make sure that only the key(s) you wanted were added.

SSH with Public-key has not been configured yet. So you are required to provide the remote host uname password.
SSH into remote host to make sure authorized_keys permission is 600.

ssh-copy-id can handle permissions automatically while the alternative method might not.
It is distribute/copy, NOT cut from local host. That means the public key should be kept locally.

ssh

By default, sshd on remote server accepts both key-based or password-based connection. The key method is superior to password method.

~$ ssh -p 22 uname@192.168.0.10

Enter passphrase for key '/home/username/.ssh/id_rsa': 
Last login: Thu Dec  3 07:59:14 2015 from 122.205.7.55

From the output, ssh prompts you to input passphrase of public-key files. Recall that we have set a passphrase when generating public-key pair to encrypt public-key files on local host.

If you leave the passphrase empty at setup, ssh will login immediately without any interaction - passwordless login. However, such login lose security of public-key files themselves.

On macOS, the Terminal app would set locale environment variables on startup when SSH to a remote server. This would disrupt remote locale. Therefore, either disable that option in Terminal settings or customize remote server's locale setting. For example CentOS, check /etc/locale.conf.

Passwordless

A better way on local host is to:

Use passphrase to encrypt key pair.
Use ssh-agent to unlock and cache private key.

The little program ssh-agent does you a favor by managing your keys for you. You enter the passphrase once, and after that, ssh-agent keeps your key cached in its memory and pulls it up whenever it is asked for it during current X/shell session.

To use the agent first create one. Just enter eval $(ssh-agent -s) and that all. This will put you in a bash shell with two important environment variables SSH_AGENT_PID (agent process ID) and SSH_AUTH_SOCK (agent socket location on system) exported. ssh-agent -s alone does not export the two variables automatically, which applies to ssh-agent -k etc. as well.

Append the contents below to ~/.bashrc.

# Start the ssh-agent if not already running
if ! pgrep -x -u "${USER}" ssh-agent >/dev/null 2>&1; then
    eval $(/usr/bin/gpg-agent -s)
fi

After that you should add your key (file) by ssh-add. Without arguments, it adds the standard identity ~/.ssh/id_rsa, ~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/identity (SSHv1). To add an identity deviant from standard location or name, run ssh-add /location/of/key. ssh-add will ask for passphrase. After that the key is loaded into the key manager ssh-agent for the whole X/shell session.

Check if any agents exist on system: ps -ef | grep -i ssh-agent.

ssh-agent is usually started at the beginning of the X session, or from a shell startup script like "~/.bash_profile". It works by creating a unix-socket, and registering the appropriate environment variables so that all subsequent applications can take advantage of its services by connecting to that socket. Clearly, it only makes sense to start it in the parent process of an X session to use the set of decrypted private keys in all subsequent X applications.

On my Gentoo system, "startx" will help launch ssh-agent automatically.
printenv | grep -i ssh, check whether SSH_AGENT_PID (agent process ID) and SSH_AUTH_SOCK (agent socket location on system) are exported to current X/shell environment. All subsequent applications depend on these two variables to make use of ssh agent on system.
Current agent

Agents found on system does NOT guarantee those two variables exported. For example, a sever shared among multiple users has many agents running which might be created by some of the users. Now you login for an X/shell session. That very session (applications thereof) does NOT know anything about existing agents on system since SSH_AUTH_SOCK and SSH_AGENT_PID are not set to one of the agents.

The current agent means the agent to which SSH_AGENT_PID and SSH_AUTH_SOCK are pointed. To switch to another agent, set the two variables to the new agent's process ID and socket path.
Example of adding key to agent.
```
ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/username/.ssh/id_rsa: 
Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa)
```
Type your passhrase. Now, you should NOT be prompted for a password whenever you use ssh, scp, or sftp etc. commands.
Limit: ssh-add does NOT persist across reboot/shutdown/logout due to loss of ssh agents or the two variables. So usuallly the passwordless login only lasts for the current agent.

Each time of login and SSH connection, run ssh-add once to achieve passwordless login. For a server, this is fine. But on personal laptop/PC that require shutdown frequently, it is tedious/annoying to run ssh-add.

This is a trade-off between security and utility. If you want passwordless forever, resort to keychain.
Reuse existing agents.

For example, on my Gentoo, ssh-agent started with startx XFCE4 DE is attached to that specific X session (Xfce4-terminal emulator included). If I don't launch startx or switch to another virtual terminal (Ctrl+Alt+F2) and login, this login shell does not either start ssh-agent or know the agent already launched!

We can launch ssh-agent in shell's configuration script (.bashrc, .bash_profile, .xinitrc etc.). Add eval `ssh-agent -s` > /dev/null and ssh-add to .xinitrc (mainly for X), bash_profile (mainly for login shell and subsequent interactive shell), and .bashrc (mainly for interactive shell).

At the end of .bash_profile, .bashrc is sourced. If you want to execute a shell script only once for the whole login, add to .bash_profile instead of .bashrc (would be executed each time a new terminal emulator launched).

However, adding ssh-agent and ssh-add to shell's configuration script will always launch an new agent on every login. We only need SSH on demand! So the basic idea is to:
1. let new X/shell reuse eixsting agents.
2. If there is not any agents, create one.

ssh-find-agent

ssh-find-agent is a tool:

locating existing ssh compatible agent sockets (e.g., ssh-agent, gpg-agent, gnome-keyring, osx-keychain).
prompt to create one if no agents found.
optionally (invoked with -a or -c), sets SSH_AUTH_SOCK, SSH_AGENT_PID environment variables accordingly.
1. and set temporal ssh alias

For lastest update, check GitHub.

SSH config

man ssh_config. The client side config file /etc/ssh/ssh_config or ~/.ssh/config. Local host - the host that connect from.

ssh command supports a variety of different parameters. For example, in order to restrict malicious attempt, you are recommended to change default port number. Other parameters are remote host IP etc. difficult and tedious to remember and input.

There's a much more elegant and flexible solution - put those arguments in ~/.ssh/config. The configuration files contain sections separated by “Host” specifications, and that section is only applied for hosts that match one of the patterns given in the specification.

Each host section defines a host alias. When connecting, just ssh host-alias. ssh will look up alias from config file and choose the first matched to connect to.

The first obtained value for each parameter is used, more host-specific declarations should be given near the beginning of the file, and general defaults at the end. Values at the beginning are superior to those afterwards. The order of privilege:
```
1. command-line options
2. user's configuration file (~/.ssh/config)
3. system-wide configuration file (/etc/ssh/ssh_config)
```
Here is an example:
```
# ~/.ssh/config
Host vps
     HostName 192.168.0.10
     Port 1025
     User bob
     IdentityFile ~/.ssh/vps_rsa_key
Host github-project1
    User git
    HostName github.com
    IdentityFile ~/.ssh/github.project1.key
Host github-org
    User git
    HostName github.com
    IdentityFile ~/.ssh/github.org.key
Host github.com
    User git
    IdentityFile ~/.ssh/github.key
Host tunnel
    HostName database.example.com
    User coolio
    LocalForward 9906 127.0.0.1:3306
    IdentityFile ~/.ssh/coolio.example.key
```
From the example, different keys are set for different github.com projects.

The "LocalForward" is interesting! 9906 (or 127.0.0.1:9906) is local port and xx.xx.xx.xx:3306 is a remote host port. SSH will establish a tunnel in between 127.0.0.1:9906 and xx.xx.xx.xx:3306 through the intermediate HostName. For example, the target host xx.xx.xx.xx only accepts connection to port 3306 locally. This example is special case for database operations. The target host is the same as HostName. Actually, it can even be "www.youtube.com:443"

To create the tunnel, just run ssh -qfCNT tunnel:
1. -q causes most warning and diagnostic messages to be suppressed.
2. -f force ssh to background just before command execution but after password authentication.
3. -C compresses all data (including stdin, stdout, stderr, and data for forwarded X11, TCP and UNIX-domain connections). Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks.
4. -N does not execute a remote command.
5. -T disable pseudo-terminal allocation. As database does not require pseduo-terminal.
The options are useful for forwarding ports and dynamic application-layer tunnel (need extra -D parameter). We cans specify these options on a per-host basis in configuration file (refer to -o of man ssh).

In addition to LocalForward, there is also RemoteRorward. No matter of local or remote forward, VPS should have a public IP while the LocalForward or RemoteForward host (NOT the vps - uname@remote-host) usually don't have public IP or are blocked from each other.

For instance, two host A and C are separated by their NAT networks. Host B are on the Internet with public IP which is accessible by A and B but not the reverse! How to achieve A and C's communication by SSH? Use remoteforward! A establishes remote forward to B. After that C connection to B will be forwarded to A.

Aside, there is another port forwarding - dynamic port forwarding (details below).

Refer to SSH 隧道与端口转发（v2ex 首发）.

scp can also make use of the config file.

~ $ scp bandwagon:~/Document/file.txt .
~ $ scp ~/Document/video.mp4 bandwagon:/Downloads

man sshd_config. The server side config file /etc/ssh/sshd_config. Remote host - the host that connect to.

You can disable account password login by PasswordAuthentication no. Then remote host can only be accessed by public-key pair. Be careful, if you lost the key pair, you probably could no longer login.

You can also put ~/.ssh/authorized_keys somewhere, and set AuthorsizedKeysFile /path/to/authorized_keys.

SSH tunnel

We can use SSH tunnel to bypass GFW.

$ ssh -qfNT -D 127.0.0.1:12345 user@vps.ip

On slow network connections, try -C to compress data. However, -C will downgrade connection on fast network.

Refer to SSH隧道翻墙的原理和实现 and ssh隧道翻墙.

Make it simple

By ssh-find-agent:
1. ssh-find-agent
2. ssh
What we have:
1. eval ssh-agent -s.
2. ssh-add
3. ssh

Backup

Different from Gnupg key's binary format, OpenSSH key (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) is stored in armor format. To backup the key, just copying the private key.

Public key can be easily generated from private key by ssh-keygen -f ~/.ssh/id_rsa -y > ~/.ssh/id_rsa.pub.

Change passphrase

~ $ ssh-keygen -p -f ~/.ssh/id_rsa

Reference

local overlay for wpa_supplicant

2015-05-11T00:00:00+00:00

Once for a while after an update, you system might bump into problem due to a new package version or to an unstable package. Thus the trouble-making packages should be downgraded to its old working version.

Recently, wpa_supplicant has upgraded from 2.3 to 2.4, resulting in unable to connecting to HUST_WIRELESS_AUTO. After days of surfing on the Internet, I found the solution: downgrade wpa_supplicant to 2.3. But before that, let's see how to position the problem. That is to say how do I get to know it is due to wpa_supplicant update causing no Wifi connection.

At the very beginning, I am in a mess on the Internet without any useful information since this problem is brand new. Even Google cannot search any clues. Af very first, I thought it was due to patches to kernel. So I reinstall the system with 4.0.0 instead of 4.0.0-r3, which turned out be in vain.

I searched on the Internet again for a day with nothing gain. I also read through the Gentoo wpa_supplicant wiki since this package is the key of no connection to Wifi. Then I used dmesg | tail -n 35 or the recommended debug mode wpa_supplicant -Dnl80211 -iwlp3s0 -C/var/run/wpa_supplicant/ -c/etc/wpa_supplicant/wpa_supplicant.conf -dd. Found an error message reason 15: 4 way handshake timeout, which is not useful at all when searching online.

Finally, on some Internet page, I was directed to an important link: wpa_supplicant Linux documentation page. Great! One step closer to solution.

From that link, you can find many useful information: development branch ChangeLog, stable branch ChangeLog, and New mailing list archives where the newest package related issues usually lies. Mailing list reports immediate package discussion even before Bug report system.

Following the mailing archives, I found Unable to connect to WPA2-Enterprise since 2.4-r1: WPA_ALG_PMK bug? which unfold the mystery on Wifi problem. Everything is there! Again, I searched the subject get another link Unable to connect to WPA2-Enterprise since 2.4-r1: WPA_ALG_PMK bug? which is fairly concise: wpa_supplicant >= 2.4 encountering problems with WPA2-Enterprise networks.

Through the discussion there, I found several solutions:

Downgrade wpa_supplicant to 2.3 (<2.4) with the help of local overlay.
Ask the university to update its FreeRadius server from to 2.2.6 to 2.2.7.
Disable TLS v1.2 with phase1="tls_disable_tlsv1_2=1" in /etc/wpa_supplicant/wpa_supplicant.conf for HUST_WIRELESS_AUTO. That said, this does result in older TLS version being used and that is not really a good long term solution.
Set key_mgmt_offload=0 to the wpa_supplicant configuration file (at global level, i.e., not within a network block).

Obviously, the 3rd method simple and effective. In reality, I prefer the 3rd method. But I would like to try the 1st instead since I want to try the process of creating a local overlay. I did not test the 4th option.

Search wpa_supplicant version information:

emerge --search wpa_supplicant

But that returns only trouble versions >=2.4. That is to say, the old working version <2.4 is no longer in the official portage tree.
Mask versions that does not work:

echo ">=net-wireless/wpa\_supplicant-2.4" > /etc/portage/package.mask/wpa_supplicant
From above two steps, we know there are no working versions in official portage tree. All the trouble versions in official portage are masked. How to handle this?

Local overlay! The basic idea is to create local overlay and then download the old version .ebuild and corresponding supporting files.
Create local overlay, refer to Overlay/Local overlay:
1. # mkdir -p /usr/local/portage/{metadata,profiles}
2. # echo 'zhtux' > /usr/local/portage/profiles/repo_name
3. # echo 'masters = gentoo' > /usr/local/portage/metadata/layout.conf
4. # chown -R root:portage /usr/local/portage
5. # chmod g+s /usr/local/portage
6. # nano -w /etc/portage/repos.conf/local.conf:
```
[zhtux]
location = /usr/local/portage
masters = gentoo
auto-sync = no
```
Adding an ebuild to the local overlay. Now we need to locate wpa_supplicant 2.3, 2.3-r1, or 2.3-r2 ebuild file and download it
1. Go through /net-wireless/wpa_supplicant
2. You cannot find wpa_supplicant-2.3*.ebuild since version 2.3 is out of portage tree: dead.
3. Click on Show 78 dead files from which you will find old ebuild files: wpa_supplicant-2.3.ebuild, wpa_supplicant-2.3-r1.ebuild, and wpa_supplicant-2.3-r2.ebuild. Here I choose wpa_supplicant-2.3-r2.ebuild. Download this file from Links to HEAD: (view) (download) (annotate). Then click on Hide 78 dead files.
4. # mkdir -p /usr/local/portage/net-wireless/wpa_supplicant
5. # cd /usr/local/portage/net-wireless/wpa_supplicant
6. # cp /path/to/downloaded/wpa_supplicant-2.3-r2.ebuild /usr/local/portage/net-wireless/wpa_supplicant/
7. # cp /usr/portage/net-wireless/wpa_supplicant/metadata.xml .
  
  metadata.xml is a XML file denoting package information. Usually just a few lines.
8. # repoman manifest (pay attention to: the current directory) or ebuild wpa_supplicant-2.3-r2.ebuild digest
  
  Calculate the package source (download from GENTOO_MIRROR or SRC_URI) checksum and put into Manifest.
9. # repoman scan -d
10. # repoman full -d -x
  
  Check if any errors. If successfull, get a line:
  
  RepoMan sez: "If everyone were like you, I'd be out of business!"
11. # chown -R portage:portage /usr/local/portage
12. # emerge -av –oneshot wpa_supplicant
  1. This will not install wpa_supplicant 2.3-r2 since it complains lacking files. If you go back to step 1, you will find a folder /net-wireless/wpa_supplicant/files. Items under this directory is supporting files like patches. Supporting files are downloaded when you sync the portage. For local overly, we need to put the files manually.
  2. On the other hand, emerge only downloads the source tarbar for wpa_supplicant-2.3-r2. It is specified in the ebuild file:
  # grep "SRC_URI=" /usr/local/portage/net-wireless/wpa_supplicant/wpa_supplicant-2.3-r2.ebuild
  
  The downloaded source file is kept under /usr/portage/distfiles/.
  1. The 3rd emerge optiono is number one, not character L. This means for one time installation.
13. # mkdir files
  1. Now based the emerge error message, we go to step 1, and download the missing files.
  2. We can also read through the ebuild files to locate what supporting files needed.
  3. The top of the page in step 1 reminds that we can use cvs command to locate the files.
14. Each time, change the ebuild, re-run repoman.
GitHub

It's useful to push local portage to Github for backup.

Reference:

https://wireless.wiki.kernel.org/en/users/documentation/wpa_supplicant
https://www.marc.info/?t=143013943600001&r=1&w=4
http://lists.shmoo.com/pipermail/hostap/2015-April/032685.html
https://www.marc.info/?l=hostap&m=143077239911058&w=4

phase1="tls_disable_tlsv1_2=1"
http://blog.csdn.net/zhuyingqingfen/article/details/7830624
https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-wireless/wpa_supplicant/
https://wiki.gentoo.org/wiki/Overlay/Local_overlay

as such

2015-05-11T00:00:00+00:00

as such 这一短语中的 such 是代词，指代前面提到的某一名词或名词短语，关键是，“such"作为代词。

一、“像这样的人或事物”、“以这种名义、身份或资格”、“像这样的情况”，也可以当”因此“来翻译，therefore, so, as a result等可以替代它。Such is a pronoun that must have an identifiable antecedent. If it doesn't have one, its use is incorrect. 这里as such主要用于分句中，指代前面分句或语句中的名词、代词。

1）He is a brilliant scholar in biological engineering and he is everywhere recognized as such.     
   他是一位生物工程杰出的学者，这是世界各地都一致公认的。
2）She' is a competent leader and has always been regarded as such by hercolleagues.   
   她是个很有能力的领导人, 她的同事一向都是这样认为的。
3）Man is a biological organism and as such has needs in common with all animals for food and shelter.   
   人是生物体,作为生物体人与动物一样需食物和住所。
4）So far as I am aware, she is the author of the article, but she does not desire to be known as such.    
   据我所知，她是这篇文章的作者，但她不想让人知道她是这篇文章的作者。
5）She is a kind woman and as such is known to everybody.      
   她是位好心肠的女人，这是大家知道的。
6）The film was a romance, and as such it had the usual happy ending.     
   那部影片很浪漫,作为浪漫影片,它通常都有一个愉快的结局。
7）You are new to this job, and as such, I will have to train you.  
   这份工作你是新手,我将不得不训练你。
8）I cannot accept your manuscript as such. It needs revisions.   
   你这样的手稿我不能接受，它需要修改。
9）As such, it is not completely convincing. More proofs are needed.   
   仅此还不足以令人信服, 还需要有更多的证据。
10）As such, the style of study is a question of extraordinary, indeed of primary, importance.   
   既然是这样，学风问题就是一个非常突出的问题，第一个重要的问题。

Refer to Don't misuse as such; 英语‘as such’的用法

二、置于名词后面，表示in the exact sense of the word，意为“就本身而论”、“… 本身”、“如词本身意思所指”。as the word is usually understood; in the strict sense of the word 按照某词通用的词义; 根据某词严格的词义。

1）Money, as such, does not bring happiness.      
   钱本身并不给人带来快乐。
2）I am not against taxes as such, but do object when taxation is justified on
   spurious or dishonest grounds.   
   我不是反对税收本身,但我确实反对莫名的或不正当的理由的税收。
3）The shop doesn't sell books as such, but it does sell magazines and
   newspapers.   
   商店不卖书,但它确实卖杂志和报纸。
4）I don’t oppose the plan as such, but I don’t thing it is proper at the
   present time.   
   我不是反对这个计划本身，我只是认为现在实施这个计划不合时宜。
5）There are no bad materials as such, but only bad articles.     
   没有不好的素材，只有不好的文章。
6）The gift as such wmay be of little worth，but it embodies her goodness.     
   这礼物本身也许没有价值, 但它包含着她的一片好意。
7）The officer of the law, as such, is entitled to respect.  
	一个执法官本身应受到人们的尊敬。

三、有时用于否定句中表示按某词词义“算不上…”、“并不真是…”、“象样的”，同样至于名词后面，和第二条用法一致。这里只是多了一个否定not，用于强调否定句子。

1）It’s not an agreement as such, but it will have virtually the same effect as one.      
  就协议的词义而言，它算不上是协定，但实际上起到了协定的作用。
2）I can’t call my book a best seller as such, but it’s very popular.     
  我不敢说我的书是畅销书，但可以说很受欢迎。
3）The new job is not a promotion as such，but it has good prospects.    
  这份新工作算不上是晋升,但它有较好的前景。
4）The position, as such, does not appeal to him, but the salary is a lure.   
  这个位置称算不上对他有什么吸引力,但薪水倒是蛮有诱惑的。
5）He isn't American as such, but he's spent most of his life there.   
  他算不上是美国人,但他一生中大部分时间都是在美国生活的。

wgetpaste default to gists

2015-05-06T00:00:00+00:00

wgetpaste is tool to conveniently paste your text and code snippet online around. However, it defaults to bpaste which relatively weak compared to gist. In this post, I will configure wgetpaste to use gist instead.
$ wgetpaste -h

This command will show the basic command options.

$ wgetpaste -S

This will list free service supported by wgetpaste.

 Services supported: (case sensitive):
    Name:        | Url:
    =============|=================
    *bpaste      | https://bpaste.net/
     ca          | http://pastebin.ca/
     codepad     | http://codepad.org/
     dpaste      | http://dpaste.com/
     lugons      | https://paste.lugons.org/
     poundpython | http://paste.pound-python.org/
     gists       | https://api.github.com/gists

The default service is bpaste with a * ahead.

Now we need to change it to gists.
```
 # ect /etc/wgetpaste.d/gists.conf
 DEFAULT_SERVICE="gists"
```
wgetpaste supports many other options, DEFAULT_{NICK,LANGUAGE,EXPIRATION}[_${SERVICE}] included.

Run wgetpaste -S again to verify configuration.
$ wgetpaste ~/Documents/test-file

Up to now, wgetpaste will paste text and code snippet as anonymous gist.github.com.

anonymous gist cannot be searched later on.
wgetpaste to GitHub account.
1. Go to GitHub settings and find Personal access tokens.
2. Choose Generate new token and set the token description to wgetpaste.
3. Change token right to gist only.
4. Edit /etc/wgetpaste.d/gists.conf:
```
 # /etc/wgetpaste.d/gists.conf
 HEADER_gists="Authorization: token 1234abc56789..."
```
Test again wgetpaste ~/Documents/test-file.
Now each wgetpaste execution will paste contents to GitHub acount, which enables search, edit, comment etc. later on.

Actually since gist supports markdown, we can use gist as a pure text blog.
If once for a while, you don't want to expose your GitHub account on the Internet, you could add -g option to generate bpaste service. You can also use -g -s gists to generate anonymous gist. -g option tells wgetpaste to ignore configurations.

robocopy

2015-04-30T00:00:00+00:00

Robust File Copy for Windows command robocopy can mirror or incrementally backup files/directories, with versitle parameters.

The basic usage of robocopy is like:

robocopy <source-dir> <dest-dir> [file [file] ...] [options]

# robocopy d:\work e:\back *.txt *.doc *.bmp *.tif /s

Attention please: Windows does not differentiate letter case. Hence, /s and /S have the same effect.

By default, file/directory to copy can be wildcards like * (default to *.*; omitted usually). Read Robocopy 指令範例參考 (MIR問題很多，小心使用) or check robocopy /?.

robocopy will copy any name where the Timestamp or file size differs:

Timestamp refers to any of Creation Date, Last Modified Date or Last Access Date.
It does not care about whether the source or the dest is newer.
It does not count on any hash functions.
In order to avoid unnecessary copy, we should tune arguments accordingly.

The following is an example that mirrors a disk parittion:

# /MIR = /E /PURGE
F:\>robocopy D:\ E:\ /MIR /XO

F:\>robocopy D:\ E:\ /MIR /XO /COPY:DAT /DCOPY:DAT [/MINAGE:1] /MAXLAD:7 /XA:SH /XD "System Volume Information" "$RECYCLE.BIN" /R:3 /W:10 /MT:16 /UNILOG+:F:\robocopy.log /TEE /Z

/MIR mirrors a directory tree, equivalent to /E /PURGE.
/XO skips older files from the source.
/COPY:DATS copies Data, Attributes, Timestamp and NTFS Security ACLs fo files.

By default, Security info of NTFS is not copied. However, the system may deny this for security considerations.
/DCOPY:DAT copies Data, Atrributes and Timestamp of directories.

By default, Timestamp is not copied.
/MINAGE:n and /MAXAGE:n exclude files according to their Last Modified Date.

MINAGE:1 guarantees that files being updated today won't copied. On the contrary, MAXAGE skip files old enough.

n can be the number of days or date in YYYYMMDD format. It should adapt to the bakcup cron job.
/MINLAD:n and /MAXLAD:n exclude files according to their Last Access Date.
/XA:SH excludes files with Hidden or System attributes.

In return, we have /IA: to copy only files with the specified attributes.
/XD <dir> excludes specific directories.

In return, we have /XF to exclude specific files.
/R:3 retries 3 times on failed copies. By default, it is 1 million.

/W:10 wait for 10 seconds between retries.
/MT:16 creats 16 concurrent threads to copy files.
/UNILOG+:F:\robocopy.log redirects log to file with UNICODE encoding.
/Z runs in restartable mode such that we can resume the transfer in case it is interrupted.

The Linux alternative of robocopy is rsync. Please check fmt.pdf.

Font

2015-04-13T00:00:00+00:00

ABC
Fontconfig
1. Per-user installation
2. System-wide installation
Xorg FontPath
1. Per-user configuration
2. Rehash without logout
Sphere - infinality
1. infinality symlinks
Practice
Reference:

ABC

Applications use fonts by font systems. There are two font systems, namely XLFD and XFT.
X Logical Font Description (XLFD) is an old font system, originally designed for bitmap fonts. Later on, XLFD adds in support for scalable fonts like PostScript Type1, TrueType (.ttf) and even newer OpenType (.otf). XLFD fonts is usually located under /usr/share/fonts like 75dpi, 100dpi etc. An XLFD font name is a long string, as below:
```
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso8859-1
```
The name contains fourteen hyphen-separated fields, with each field representing a property of the font. Not all fields are required to be present. In Emacs, we can use C-x = over a character so show the XLFD name.

XLFD is almost depcreted but still required by ancient programs like xterm, urxvt and GTK+1. To set XLFD for X, we use xfontsel, and xlsfonts.
1. For X, we can use xset.
XFT is a newer font system, supporting much more font types. depending on Fontconfig's matching feature and Freetype's font rendering feature. Freetype makes fonts (e.g. anti‑aliasing and sub‑pixel rasterization) look smooth and beautiful.

We configure Fontconfig to use XFT /etc/fonts/fonts.conf.
Application decides to use which font system. Almost all applications use XFT now, Xorg included. Emacs supports both XLFD and XFT (try M-x: describe-font).

However, we may tune FontPath in xorg.conf covering bitmap fonts for ancient programs. Most of the time, we do not include XFT fonts in xorg.conf. To check fonts knowns to X:
```
$ grep /fonts /var/log/Xorg.0.log
or
$ xset q
```

Fontconfig

Commands like fc-list, fc-list, fc-query, fc-match etc. can help check fonts' information.

This section demonstrates how to manully install personal fonts.

Per-user installation

~ $ mkdir -p ~/.local/share/fonts/winfonts
~ $ cd ~/.local/share/fonts/winfonts
~ $ cp /path/to/{abc.ttf,123.otf} .

~ $ fc-cache -fv ./*
~ $ mkfontscale ./ && mkfontdir ./ && xset fp rehash

For manually copied fonts, we have to create the font cache with fc-cache. Font cache is to speed up font loading. Package manager (i.e. emerge) takes care of it automatically.
mkfontscale and mkfontdir will create file fonts.scale and fonts.dir for XLFD fonts. xset fp rehash makes new fonts immediately available to current X sessions.

System-wide installation

To manually install fonts for the whole system is similar except that fonts are put under /usr/share/fonts/ or /opt/fonts/.

/ # mkdir /opt/fonts/winfonts
/ # chmod 755 /opt/fonts/winfonts
/ # cd /opt/fonts/winfonts
/ # cp /path/to/{abc.ttf,123.otf} ./
/ # chmod -R 644 .
/ 
/ # fc-cache -fv ./*
/ # mkfontscale ./ && mkfontdir ./

Xorg FontPath

Most of time, Fontconfig works out of box without any further configuration. To support ancient applications that use XLFD, we might need to tune "FontPath".

For Xorg to load fonts directly (as opposed to the use of a font server - XFT), the directory for your newly added font must be added with a "FontPath" entry. This entry is located in the "Files" section of Xorg configuration file. An official example is /usr/share/doc/xorg-server-1.18.4/xorg.conf.example.bz2.

Create /etc/X11/xorg.conf.d/10-fontpath.conf:

Section "Files"
# Multiple FontPath entries are allowed (which are concatenated together),
# as well as specifying multiple comma-separated entries in one FontPath
# command (or a combination of both methods).
# The default path is shown here.
#    FontPath	/usr/share/fonts/misc/,/usr/share/fonts/TTF/,/usr/share/fonts/OTF/,/usr/share/fonts/Type1/,/usr/share/fonts/100dpi/,/usr/share/fonts/75dpi/

    FontPath "/usr/share/fonts/misc"
    FontPath "/usr/share/fonts/100dpi,/usr/share/fonts/75dpi"
    FontPath "/usr/share/fonts/util"
    FontPath "/usr/share/fonts/encodings"
    ...
    FontPath "/usr/share/fonts/winfonts"
EndSection

"FontPath" entry can be a single font path or multiple comma-separated font paths (or a combination of both). Font paths must be quoted.
The default pathes (i.e. 100dpi, 75dpi etc.) are always scanned.
Pathes lack fonts.scale (mkfontscale) and fonts.dir (mkfontdir) are neglected.

Per-user configuration

Settings Font Path under /etc/X11/xorg.conf.d/ are system-wide effective. For per-user setting, add

xset +fp /usr/share/fonts/dejavu/           # Prepend a custom font path to Xorg's list of known font paths

to ~/.xinitrc. Alternatively, execute it on command line.

To remove a Font Path, use -fp instead.

Rehash without logout

After the setting, we want immediate effect:

# xsetq fp rehash

Sphere - infinality

infinality USE can be enabled globally in /etc/portage/make.conf.

# echo "media-libs/freetype adobe-cff infinality" > /etc/portage/package.use/freetype
# emerge -av media-libs/freetype
# eselect fontconfig list
# eselect fontconfig enable 52-infinality.conf

fontconfig-infinality will draw in its own settings (/etc/fonts/infinality) which will interfere with the other fontconfig configuration. It's recommended to disable most of Fontconfig options while keeping 52-infinality.conf. Specially, configurations related to specific fonts can be kept like 62-croscore-*.conf* and 57-dejavu-*.conf.

You will find fontconfig (/etc/fonts/) and infinality (/etc/fonts/infinality/) has the nearly the same directory architecture. The conf.d sub-directory stores the selected symlinks, while conf.avail (or conf.src) stores all existing configurations. We use eselect which chooses and creates symbolic links under conf.d sub-directory.

Optionally, refer to reference number 3, set embeddedbitmap in /etc/fonts/infinality/infinality.conf to true.

infinality symlinks

# eselect lcdfilter list
# eselect lcdfilter set 14, set to *windows-7*
# eselect infinality list
# eselect infinality set to *win7*

Make sure lcdfilter and infinality choose the same category.

Practice

Practice from Jin Buguo Linux字体美化实战(Fontconfig配置) is outdated. However, I want to use his recommended fonts, except the Korean ones. Additionally, we need to install noto-fonts-cjk as Firefox may fallback to those fonts.

Ancient applications using XLFD refer to X11 Font Path directly. Make sure fonts.scale (by mkfontscale) and *fonts.dir (by mkfontdir) exist under each directory of font directory. Paths under /opt/fonts/ can be appended to Font Path if you'd like.

Reference:

Gentoo Installation

2015-03-25T00:00:00+00:00

Gentoo installation along with Windows 8.1 and Ubuntu 14.04 with UEFI booting.

Tips

Follow Handbook.
To emerge packages, you'd better read ArchWiki.
Ref:

LiveDVD USB stick

Download the LiveDVD like livedvd-amd64-multilib-20140826.iso instead of Minimal installation CD like install-amd64-minimal-20150319.iso.
1. Minimal CD cannot generates UEFI bootable USB stick.
2. LiveDVD gives desktop environment to ease network connection and WWW surfing.
Create UEFI USB stick. Similar tools are:
1. Rufus;
2. UNetbootin;
3. Universal USB Installer;
4. diskpart terminal command;
5. Just copy the ISO contents into a FAT32 format USB stick;
6. Use dd command;
7. Ubuntu disk creater.
Boot into the KDE desktop environment with USB stick made before. By default, there is no password required for account gentoo.
1. The very first thing is to connect to Wi-Fi or Ethernet through networkmanager.
2. Use shortcut F12 to Open/Retract Yakuake/Guake terminal in KDE destop.
3. Refer to Gentoo Ten LiveDVD Frequently Asked Questions.
sudo su - to root.

You can use passwd USERNAME to change the password of gentoo. As root, you can change any account passworld by passwd username. All the password thing within the LiveDVD environment is volatile.

The command prompt is livecd ~ # different from the handbook one - root #.

Disk preparation

parted -a optimal /dev/sda or fdisk /dev/sda. Erase /dev/sda10 (NTFS partition) to install Gentoo on.

NOTE: parted takes effect immediately without final confirmation like fdisk. Read Kali Linux Live USB Persistence first.

# parted -a optimal /dev/sda
(parted) help
(parted) p
(parted) unit MiB
(parted) rm 10
(parted) p
(parted) mkpart primary 309921MiB 310049MiB, create a boot partition sda10 for Gentoo
(parted) p
(parted) name 10 'Gentoo boot partition'
(parted) mkpart primary 310049MiB -1, create root partition sda12 for Gentoo
(parted) p
(parted) name 12 'Gentoo root partition'
(parted) p
(parted) q

The partition Type is Basic data partition when checking with fdisk /dev/sda. You change that through Disk GUI application under Ubuntu.

Newly created /dev/sda10 will be the boot partition while /dev/sda12 the root partition.

We don't create Swap (sda7) or EFI (sda2) partition. Just share them with Ubuntu and Windows. You can also create a separate home partition, say sda13.
Format the new partition. You are suggested to format boot partition as ext2.
```
# mkfs.ext2 /dev/sda10
# mkfs.ext4 /dev/sda12
# mkfs.ext4 /dev/sda13
```
swap partition needs activated
```
# mkswap /dev/sda7 (opt)
# swapon /dev/sda7
```
If you prefer an independent swap partition, mkswap to format it firstly.

Mount partitions.

# mount /dev/sda12 /mnt/gentoo
# mkdir -p /mnt/gentoo/boot/efi && mount /dev/sda10 /mnt/gentoo/boot && mount /dev/sda2 /mnt/gentoo/boot/efi
# mkdir /mnt/gentoo/home && mount /dev/sda13 /mnt/gentoo/home

Install stage3

Check date and time using date command. Set by date MMDDhhmm if it's incorrect.
Downloading and verify stage3 tarball. Usually, it's downloaded beforehand along with LiveDVD ISO.
Now unpack the downloaded stage onto the system.
```
# cd /mnt/gentoo
# tar xvjpf /mnt/cdrom/stage3-amd64-20150319.tar.bz2 --xattrs
```
1. x stands for Extract;
2. v for Verbose to see what happens during the extraction process (optional);
3. j to Decompress with bzip2;
4. p for Preserve permissions;
5. f denotes that we want to extract a File, not standard input;
6. --xattrs includes the extended attributes stored in the archive as well.

make.conf

CFLAGS == CXXFLAGS

Check your CPU architecture to set -march= parameter. Refer to Intel. grep -m1 -A3 "vendor_id" /proc/cpuinfo offers CPU architecure. That link also teach you how to precisely detect -march= parameter by touching, compiling and comparing two .gcc files.
1. CFALGS="-march=sandybridge -O2 -pipe"
2. CXXFLAGS="${CFLAGS}"
The MAKEOPTS variable defines how many parallel compilations should occur when emerging packages.

The recommended value is the number of logical processors in the CPU plus 1. But this rule is outdated. I have 4GB memory and swap/swapfile enabled, emerge will make use of swap havily.

swap naturally slows down performance but supports more parallel tasks. So turn down to 2 or even 1 to reduce usage of swap file.
1. Add a line MAKEOPTS="-j3".
2. The boot screen shows several penguins, that is the number of logical hardware cores. If kernel X86_SYSFB and FB_SIMPLE were turned off, you could not see boot penguins.
3. This value does not apply to kernel compiling. We explicitly specify make -j3.
4. Refer to MAKEOPTS.
CPU_FLAGS_X86: The 'USE' flags corresponding to the CPU instruction sets and other features specific to the x86/amd64 architecture are being moved into a separate USE flag group called CPU_FLAGS_X86.
1. Refer to cpu_flags_x86 instruction.
2. Emerge app-portage/cpuinfo2cpuflags and run in LiveDVD. Edit 'make.conf' to set CPU_FLAGS_X86.
3. Up to now, some packages in portage and overlays are not yet migrating those flags from 'USE' to 'CPU_FLAGS_X86'. So:
4. Remove the old CPU specific flags from 'USE'. Then add ${CPU_FLAGS_X86} to 'USE'.

Chrooting

cp -L /etc/resolv.conf /mnt/gentoo/etc/, to ensure that networking after chrooting.

Mounting the necessary filesystems.

# mount -t proc proc /mnt/gentoo/proc
# mount --rbind /sys /mnt/gentoo/sys
# mount --make-rslave /mnt/gentoo/sys
# mount --rbind /dev /mnt/gentoo/dev
# mount --make-rslave /mnt/gentoo/dev
# chroot /mnt/gentoo /bin/bash
# source /etc/profile && export PS1="(chroot) $PS1"

Mirrors and Sync

Selecting mirrors
```
# mirrorselect -s3 -b10 -o -D >> /mnt/gentoo/etc/portage/make.conf (opt)
```
Choose the 3 fastest mirrors for package sources downloading. This will takes around 10 minutes. So usually, we just manually choose 3 physically close servers.
[deprecated] ~~Portage sync~~
```
# mirrorselect -i -r -o >> /mnt/gentoo/etc/portage/make.conf
```
Selects the rsync server to use when synchronizing portage tree. It is recommended to choose a rotation link, such as rsync.us.gentoo.org, rather than choosing a single mirror (i.e. ftp). This helps spread out the load and provides a fail-safe in case a specific mirror is offline.
The new plug-in sync system (>=sys-apps/portage-2.2.16)

The new plug-in sync system currently supports rsync, git, svn, websync, webrsync, cvs, laymansync synchronizing types. And configurations reside now under /etc/portage/repos.conf/.

After installing stage3:
```
# mkdir /mnt/gentoo/etc/portage/repos.conf
# cp /mnt/gentoo/usr/share/portage/config/repos.conf /mnt/gentoo/etc/portage/repos.conf/gentoo.conf
```

Portage tree

Portage snapshot
```
# emerge-webrsync
```
webrsync type is used to create a complete Portage tree. It might complain about a missing /usr/portage/ location. This is to be expected and nothing to worry about - it will be created on demand.
Profile
```
# eselect profile list
# eselect profile set 3
```
Choose the desktop profile instead of desktop/gnome or desktop/kde. We will install XFCE later on.
USE flags

Try emerge --info | grep ^USE to check USEs set by profile and make.conf. Refer to Xfce/Guide.
```
USE="${CPU_FLAGS_X86} -bindist -qt4 -libav vaapi"
```
By default, bindist is enabled by stage3's default make.conf. If we don't plan to distribute packages, remove it. bindist may cause conflicts, i.e. between openssh and openssl.

Localization

Time
1. Prime Meridian was (arbitrarily) chosen to pass through the Royal Observatory in Greenwich.
2. UTC (previously called Greenwich Mean Time GMT): time at zero degrees longitude (the Prime Meridian) is called Coordinated Universal Time.
3. Unix time: Measured as the number of seconds since epoch (the beginning of 1970 in UTC). Unix time is not affected by time zones or daylight saving.
4. Local Time (UTC + Timezone) is what we use daily.
  
  Setting operating Software Time equal correctly to Local Time is our goal.
5. Hardware Time resides on BIOS, which can be set manually and updated by operating system. Both Windows and Linux sets Software Time based on Hardware Time.
  
  It's regarded as permanent placeholder to set Software Time.
6. Windows treats Hardware Time as Local Time and sets Software Time identically to Hardware Time without any translation on boot. Timezone is to synchronize time with time server. Upon shutdown, Windows writes back Software Time to Hardware Time.
7. Linux treats Hardware Time as UTC and adds Timezone (i.e. +8) to Hardware Time on boot. Similarly, Software Time is written back to Hardware Time.
8. Suppose Windows Software Time and Hardware Time are correctly synched to Local Time, Linux (dual boot installation afterwards) Software Time will be Hardware Time + Timezone = Windows Software Time + Timezone = Local Time + 2xTimezone. So usually Linux Software Time is ahead of Localtion for Timezone.
  
  If we shutdown Linux and boot Windows, Windows Software Time will be ahead of Local Time too. Each time we switch back and forth, Software Time will increase linearly by Timezone step.
9. We must tell Linux that Hardware Time is Local Time instead of UTC before Software Timezone is configured.
hwclock

Set clock=local in /etc/conf.d/hwclock. We can find other options concerning reading/updating Hardware Time.

Afterwards, set timezone information.
Timezone

This step should be after /etc/conf.d/hwclock update. Otherwise the Linux Software Time is usually ahead of local time by 8 hours, thus resulting in portage tree time stamp issues.
```
# ls /usr/share/zoneinfo
# echo "Asia/Shanghai" > /etc/timezone
# emerge --config sys-libs/timezone-data
# date
```
locale

The locale format is like xx_YY.ZZ, where xx and YY denote lanugage code and country code respectively. ZZ stands for charset (encoding/decoding). xx and YY mainly affects GUI (DE, app menus etc.), while ZZ takes care of encoding/decoding.

eselect locale set defines all locale settings at once by LANG varaible while allowing further individual customization via the LC_* sub-options (i.e. LC_CTYPE).
```
# cat /usr/share/i18n/SUPPORTED \| grep zh_CN >> /etc/locale.gen
# uncomment en_US.UTF-8 UTF-8
# locale-gen
# locale -a
# eselect locale list
```
In the 3rd step, if reminded to run ". /etc/profile" to reload the variable, just remember export PS1="(chroot) $PS1".

Use xx_YY.UTF-8 (some applications does NOT recognize xx_YY.utf8) instead of the illegal format xx_YY.UTF8. How to achieve this? Use the free form of Gentoo eselect.
```
# eselect locale set en_US.UTF-8
# locale
```
LC_* sub-options can be further set in /etc/env.d/02locale:
```
LANG="en_US.UTF-8"
LC_COLLATE="C"
LC_CTYPE="zh_CN.UTF-8"
```
If you don't have privileged access, export them in shell RC file like .bashrc.

Remember to
```
# env-update && source /etc/profile && export PS1="(chroot) $PS1"
# locale
```
Chinese

Why English display is not a concern? You may say that English is the mostly used accross the world and hence mostly used by programmers and the applications thereof. Yep, that's right but superficial. The underlying core is that ASCII table (English characters) is covered by nearly all encoding schemes present, GB2312/GBK/GB18030 included! No matter which encoding scheme is chosen, English is always correctly displayed.

To be specific, language display is divided into two aspects:
1. GUI (i.e. menu, popup box, botton, log etc.).
2. File content which is what we refer to without explicit explanation (i.e. a HTML page).
Let's talk about GUI first. You may be confused. Most applications GUI use ASCII characters (i.e. popup error dialog), thus correctly displayed no matter what system locale is set. But applications written for Chinese like QQ use Chinese GUI by default. What if the author could not be bothered to offer English GUI? Then we have to tune system locale to Chinese.

What if you would like more Chinese on your system GUI, which looks confortable? That's where nls and l10n_* (linguas_* will be deprecated) USEs play a role. Many applications supports either nls or l10n, which is the part users can control. The localized/translated GUI messages is stored in */usr/share/locale//LC_MESSAGES/.mo* files. *nls* installs all possible locale messages, while *l10n_\** only installs the locale message specified by USE. The system locale must be *LANG=zh_CN.ZZ* to let installed locale messages displayed, where ZZ charset must covers zh_CN characters (GB2312 GBK GB18030). If you only set sub-option *LC_CTYPE=zh_CN.ZZ*, then GUI sticks to English.

When it comes to file content display/updating, it depends on the context. Client browser decodes HTM page by the charset tag. Emacs is smtart at detecting file encoding. Mousepad is stupid and always decode by system locale. And will ask for user confirmation upon failure. However, for Unicode-based encoding, it's easy to guess the encoding through BOM. Modern applications is smart at detection. If the application fails to detect file encoding, it may default to ZZ part of locale, to which many comand line tools belongs.

Let's go on details of ZZ part. For Chinese, it should be one of GB2312, GBK, GB18030 and UTF-8. Apart from the fallback decoding role, it mainly determine encoding of user generated contents (i.e. default encoding of new file).

Remember that:
1. Chinese characters are encoded and stored. Upon display, the encoding scheme should be found (by GTK/QT, text editor, browser, even by user involvement).
2. The advantage of setting ZZ to UTF-8 is that it's widely used and reduces mojibak.
3. UTF-8 requires more disk space (50% higher) for CJK characters.
It's not over yet. Up to now, only character encoding is detected. In order to display the character, relevant font should be matched (i.e. by Fontconfig). Translation between the character encoding and glyph is done through font's intermediate charmap/cmap/bmp files. cmap is a table mapping character encoding to glyph's internal index. Each font may include multiple cmap files corresponding to different character encodings (i.e. GBK, UTF-8). Then font engine (i.e. FreeType2) renders the returned glyph.

Attention, cmap itself is encoded as well, mostly by Unicode (namely UTF-16/UCS-2, NOT UTF-8).

Details on XFT, refer to fontconfig.

Kernel building

Kernel sources tree

# emerge -avt sys-kernel/gentoo-sources
# eselect kernel list/set
# ls -l /usr/src && cd /usr/src/linux;
# git apply [--whitespace=warn] [--numstat] [--check] < /path/to/cjktty.patch
or
# patch [--dry-run] -p1 < /path/to/cjktty.patch (opt)

For the first time we install kernel sources, /etc/src/linux symlink is created automatically.

If possible, apply kernel patches like cjktty.patch.

Tips
1. You'd best have a .config backup to start with.
2. If possible, try sys-apps/pciutils, sys-apps/usbutils, and sys-apps/{lshw,dmidecode} in LiveDVD. Don't emerge those packages in chroot, it's not necessary.
3. During kernel config, search the kernel options on page Linux-3.10-x86_64 内核配置选项简介 and refer to LiveDVD's configuration.
4. Try lspci -n and paste it's output to device driver check page; that site gives kernel options that must be enabled.
5. We can refer to the LiveDVD's kernel config directly.
6. Search with /. When search "snd-hda-intel", replace the hypen with dash.
  
  Usually there are several search outputs numerated (1, 2, 3 …). Press the number to enter the kernel option.
7. exit or two successive ESCs to get back.
8. When confronted with issues related to kernel options, we can choose M instead of Y which might be a solution.
.config backup

Suppose we have an old .config backup:
```
# cd /usr/src/linux
# cp /path/to/.config-backup .config && chmod -x .config
# make help
# make silentoldconfig
```
1. If the .config backup belongs to the same kernel version, we omit make silentoldconfig.
2. oldconfig asks for both new and old options.
3. silentoldconfig only asks for NEW options while preserving old ones. Press ENTER to choose default setting.
4. olddefconfig is similar but sets NEW options to default without confirmation.
We might tune kernel options by graphical interface make menuconfig below.
Menuconfig
```
# cd /usr/src/linux
# make menuconfig
```
1. i915 e100e snd-hda-intel iTCO-wdt ahci i2c-i801 iwlwifi sdhci-pci.
  
  Basic dirvers that needs activated.
2. IKCONFIG should be Y instead of M. This allows you to inspect (by ./scripts/extract-ikconfig /path/to/vmlinuz)the configuration of the kernel while it is running, without having to worry whether you've changed or cleaned the source directory after it was built. IKCONFIG_PROC stores kernel config at /proc/config.gz.
3. NR_CPUS set to 4 since laptop has 2 cores and 4 threading. The smaller this value is, the smaller the kernel image is and the less memory is consumed by kernel.
4. MICROCDE and MICROCDE_INTEL are Y by default. Refer to microcode-ctl below.
5. Graphics. i915 = Intel 8xx/9xx/G3x/G4x/HD Graphics, DRM_I915 uses the default Y.
6. FB FRAMEBUFFER_CONSOLE X86_SYSFB FB_SIMPLE all set to Y.
  
  Or FB FRAMEBUFFER_CONSOLE FB_EFI to Y.
7. Ethernet. e1000e = Intel (R) PRO/1000 PCI-Express Gigabit Ethernet support set to M.
8. Audio. snd_hda_intel = Intel HD Audio, CONFIG_SND_HDA_INTEL. The default value is Y, now MUST be M. Choose the audio codec:
  
  Refer to no sound for how to decide the audio cdoec support.
  
  Execute cat /proc/asound/card0/codec#* \| grep -i codec in LiveDVD:
```
Codec Conexant CX20590
Codec Intel CougarPoint HDMI
```
  1. Then select those two codecs as M. Build HDMI/DisplayPort HD-audio codec support = SND_HDA_CODEC_HDMI. Conexant HD-audio support = SND_HDA_CODEC_CONEXANT.
  2. Enable generic HD-audio codec parser = SND_HDA_GENERIC must be selected (default).
  3. Set Default time-out for HD-audio power-save mode = CONFIG_SND_HDA_POWER_SAVE_DEFAULT to 10.
  4. Set Pre-allocated buffer size for HD-audio driver to 4096.
  5. You notice these options are ALL set to M! You can also set them all to Y. But never set some to M while set others to Y, othewise you would get no sound at all.
9. Webcamera
  1. lsusb
    Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 006: ID 04f2:b217 Chicony Electronics Co., Ltd Lenovo Integrated Camera (0.3MP) Bus 001 Device 005: ID 0a5c:217f Broadcom Corp. BCM2045B (BDC-2.1) Bus 001 Device 004: ID 147e:2016 Upek Biometric Touchchip/Touchstrip Fingerprint Sensor Bus 001 Device 003: ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    The 3rd item is the integrated webcamera which is a multimedia USB device.
  2. Enable MEDIA_SUPPORT = Multimedia support, MEDIA_CAMERA_SUPPORT = Cameras/video grabbers support, MEDIA_USB_SUPPORT = Media USB Adapters, and USB_VIDEO_CLASS = USB Video Class (UVC). The 2nd and 4th items are the key to make webcamera to work.
10. Processor type and features
  1. lscpu | grep -i 'CPU family'. If the ouput is 6, select the 3rd item as Core 2/newer Xeon = CONFIG_MCORE2, otherwise the output would be 15, please select the 5th item.
  2. Turn off NUMA = Numa Memory Allocation and Scheduler Support. Refer to What Is NUMA?.
  3. Remove several AMD items under Processor type and features by searching AMD. They are AGP_AMD64 X86_MCE_AMD MICROCODE_AMD AMD_NUMA AMD_IOMMU.
  4. Enable EFI runtime service support = EFI if UEFI is used to boot the system. Turn off EFI stub support = EFI_STUB, EFI mixed-mode support = EFI_MIXED, Built-in kernel command line = CMDLINE_BOOL, Built-in kernel command string = CONFIG_CMDLINE and Initramfs source file(s) = INITRAMFS_SOURCE. These options are closely related mainly for compiling kernel boot arguments and initramfs into kernel image. Then the system can boot and load kernel directly from EFI firmware instead of bootloader. Details refer to efi stub kernel.
11. Set INPUT_PCSPKR as M or Y for the standard PC Speaker to be used for bells and whistles. Don't enable SND_PCSP (read the kernel help message)!
12. Set X86_X32 = x32 ABI for 64-bit mode to Y. This option is useful for 32 bit binaries to take advantage of the system 64-bit registers.
13. Set IOMMU_SUPPORT, INTEL_IOMMU, and INTEL_IOMMU_DEFAULT_ON set to N since laptop lacks VT-d (but has VT-x). VT-d is what Intel calls their IOMMU implementation. Refer to i5 2410M. Without IOMMU enabled, HD3000 graphics will tear and corrupt.
14. Watchdog. Intel TCO Timer/Watchdog = ITCO_WDT set to M.
15. SATA. ahci = AHCI SATA support, SATA_AHCI for SATA disks selected 'Y' default. Disable ATA SFF support (for legacy IDE and PATA) = ATA_SFF set to N.
16. Intel 82801 (ICH/PCH) = I2C_I801 uses default 'Y'.
17. Wireless. Intel Wireless WiFi Next Gen AGN - Wireless-N/Advanced-N/Ultimate-N (iwlwifi), CONFIG_IWLWIFI set to M. By the way, wpa_supplicant needs nl80211 wifi driver. Actually relates to cfg80211 - wireless configuration API = CFG80211 which is set to Y default already.
18. (opt) Bluetooth. BT = Bluetooth subsystem support to M. Enter and find BT_RFCOMM = RFCOMM protocol support. I think this should be 'M', otherwise package like obexfs or obexftp did not work. Choose USB driver BT_HCIBTUSB = HCI USB driver as M. The sub-option BT_HCIBTUSB_BCM turned on automatically. BT_HIDP = HIDP protocol support is for human interface device like bluetooth mouse, bluetooth headset, bluetooth keyboard etc. Since I dont' use them, so leave it as N. My bluetooth Logitech mouse works perfectly even when turnning off all the related bluetooth kernel options. That is due to the extra LOGITECH related kernel drivers. Read more from bluetooth - bluez obexfs.
19. MMC. sdhci_pci = SDHCI support on PCI bus = MMC_SDHCI_PCI, but you cannot positioninig the item since its parent Secure Digital Host Controller Interface Support is turned off by default. So turn this on first. By the way, set Ricoh MMC Controller Disabler = MMC_RICOH_MMC` as Y.
20. Refer to Xorg configruation to enable Xorg kernel support. However, according to this reference, nothing needs updated.
21. NTFS. NTFS_FS and FUSE_FS to M. Refer to NTFS wiki. You need to install ntfs-3g package later on. Since ntfs-3g embeds NTFS write, setNTFS_RW to N.
22. Turn on PACKET (default Y) for wireless tool wpa_supplicant which will be installed later on.
23. Turn off NET_VENDOR_NVIDIA since no NVIDIA card in laptop.
24. Set FAT_DEFAULT_CODEPAGE to 936 (without prefix cp), FAT_DEFAULT_IOCHARSET uses default iso8859-1 (this value should include the prefix cp such as cp936) but turn on it's sub-option FAT_DEFAULT_UTF8.
  
  Enable NLS_CODEPAGE_936 to Y or M. Current initramfs reads LUKS key from VFAT USB partition on boot, MUST set to be Y, otherwise it would fail to find (mount) the key device.
  
  Turn off NLS_CODEPAGE_437 and NLS_ASCII since we either use UTF-8 or GBK system. Keep NLS_ISO8859_1 since it's the default value of FAT_DEFAULT_IOCHARSET.
25. Dm-crypt. Device mapper support = BLK_DEV_DM is set Y by default. Crypt target support = DM_CRYPT must be M or Y. XTS support = CRYPTO_XTS and AES cipher algorithms (x86_64) = CRYPTO_AES_X86_64 optionally set to M (recommended). Refer to Dm-crypt. Refer to Cryptsetup step below. CRYPTO_SERPENT and CRYPTO_SHA512 must be Y instead of M if relevant LUKS algorithms are used. Refer to gentoo over lvm luks.
26. Iptables. NETFILTER_ADVANCED & XT_MATCH_OWNER (-m owner). IP_NF_TARGET_REDIRECT (-j REDIRECT) which will enable NF_NAT_REDIRECT (in return, NETFILTER_XT_TARGET_REDIRECT be enabled as well). We only need NETFILTER_XT_TARGET_REDIRECT for iptables REDIRECT, so to simplify kernel, just enable it alone. If necessary, turn on IP6_NF_NAT (NF_NAT_IPV6 as dependency) to enable ip6tables nat table. Turn on NETFILTER_XT_MATCH_MULTIPORT for -m multiport. Turn on CONFIG_NETFILTER_XT_TARGET_TPROXY for -j TPROXY target support.
27. System log: SECURITY_DMESG_RESTRICT to Y. Refer to Restrict unprivileged access to kernel syslog.
  
  More refer to syslog-ng below.
28. Filesystem. Set UDF_FS as M to mount ISO file.
29. SquashFS. CONFIG_SQUASHFS as M; CONFIG_SQUASHFS_XATTR and CONFIG_SQUASHFS_XZ as Y. (sys-fs/squashfs-tools).
30. IO scheduler - BFQ: IOSCHED_BFQ, BLK_CGROUP and BFQ_GROUP_IOSCHED as Y.
31. Magic SysRq key CONFIG_MAGIC_SYSRQ as Y. CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE configure default value of /proc/sys/kernel/sysrq.
32. WireGuard: NET_FOU (NET_UDP_TUNNEL) and PADATA.
33. IP set: IP_SET, IP_SET_HASH_IP, IP_SET_HASH_NET, IP_SET_LIST_SET; NETFILTER_XT_SET, NETFILTER_XT_MARK.
34. (opt) Thinkpad-related. ThinkPad ACPI Laptop Extras = THINKPAD_ACPI set to M.
35. [e-sources / cjktty patch specific options].FONTS and FONT_8x16 set to Y. And console 16x16 CJK font ( cover BMP ) = FONT_16x16_CJK which is cjktty patch. These options are for Chinese characters display in TTY (Ctrl + Alt + Fn).
36. Reference links: Linux-3.10-x86_64 内核配置选项简介; Linux Kernel in a Nutshell; kernel-seeds; device driver check page; How do you get hardware info and select drivers to be kept in a kernel compiled from source; and Working with Kernel Seeds.
Compiling
```
# cd /usr/src/linux
# mount /dev/sda10 /boot; mount /dev/sda2 /boot/efi
# make clean/mrproper/distclean; echo 3 > /proc/sys/vm/drop_caches
# make modules_prepare
# make -j3 && make modules_install && make install
# genkernel --install initramfs
# grub-mkconfig -o /boot/grub/grub.cfg
# emerge -avt @module-rebuild
# reboot
```
1. Make sure we are in kernel sources directory.
2. This is usually done before chrooting.
3. Clean previous building leftovers.
  1. clean removes most generated files but keep the .config and enough build support to build external modules (such that we overlook make modules_prepare).
  2. mrproper removes all generated files + .config + various backup files.
  3. distclean = mrproper + remove editor backup and patch files.
  4. drop_caches is set to 3.
  They are chosen if we re-compile kernel sources in case of compiling error. What's worse, a successful re-compiling generates unworkable kernel binaries.
  
  Please be noted that mrproper and disclean will remove backup files, especially the .config.
4. make modules_prepare is somewhat complicated.
  
  External kernel modules (i.e. self-written codes; VirtualBox binary kernel modules) are built against kernel tree (/usr/src/linux/). That's because external modules building need support from kernel sources support (i.e. kernel sources' head file).
  
  So the kernel tree should be prepared to support external kernel modules building, which can be achieve by several ways:
  1. If the kernel tree is brand new (i.e. just unpacked) and we will build external kernel modules before kernel building, we should prepare by make modules_prepare under /usr/src/linux/.
  2. If the kernel have already been built (i.e. make -j3), it's already prepared. Need to prepare.
5. Compiling.
  1. Compiles the kernel. Keep -j3 since MAKEOPTS in make.conf does not apply to kernel compiling.
  2. Install kernel modules into /lib/modules/`uname -r`. If we are re-compiling the kernel, old modules will be overriden. Ether backup the old modules or install modules to a new location (by INSTALL_MOD_PATH) when we want the old kernel be bootable (i.e. we remove a module from kernel).
  3. Install kernel binaries System.map, config, initramfs, vmlinuz into /boot and rename old identical version kernel binaries on demand. We can copy and even rename those binaries manually as long as filenames are kept consistent.
6. For LVM/LUKS containers, emerge genkernel with cryptsetup USE.
  
  Extra arguments --lvm --luks --gpg --busybox should be supplied upon generates initramfs. Refer to Gentoo rootfs over LVM encrypted in LUKS container and lvm luks lvm.
  
  Like modules, genkernel does not backup initramfs automatically when re-compiling the kernel. We are responsible for bakcuping manually, especially when the genkernel arguments are different.
7. Append current kernel to Grub menu.
8. Although we are on the old kernel, @module-rebuild knows how to re-install external kernel modules for the new kernel as long as /usr/src/linux symlink pointing the new kernel source tree (see eselect kernel list).
  
  Optionally, this step could be omitted if we are re-compiling the kernel, or we have done make modules_prepare.
Linux firmware

Some drivers require additional firmware to be installed on the system before they work. This is often the case for network interfaces, especially wireless network interfaces.
```
# emerge -avt --fechonly sys-kernel/linux-firmware
```
Firstly just fetch the package sources. If network failed connect after booting into new kernel, emerge it then.

System configuration

fstab

Creating the /etc/fstab file. Use backup at best. The default /etc/fstab file provided by Gentoo is not a valid fstab file but instead more of a template.

/dev/sda10   /boot		ext2    defaults,noatime	1 2
/dev/sda12   /   		ext4    noatime	       0 1
/dev/sda7    none	  	swap	sw	       0 0
proc         /proc          proc    nosuid,nodev,noexec,hidepid=2,gid=wheel 0 0
none         /tmp           tmpfs   nodev,nosuid,noexec     0 0

Double-check the /etc/fstab file, save and quit to continue. Fault fstab results in boot failure.

hostname and domain
1. nano -w /etc/conf.d/hostname
```
hostname="myhost"
```
2. On login, usually get message like This is myhost.unkown_domain … which is not a error, but annoying. There are two ways to get rid of it.
  
  The first way is to set a fake domain for localhost, edit /etc/hosts. Add fake hostname.domain to localhost.
```
# <ip address>	<fully qualified hostname>	<aliases>
127.0.0.1       myhost.jiantu.boxes      myhost   localhost
::1             myhost.jiantu.boxes      myhost   localhost
```
  Now try ping myhost.jiantu.boxes/myhost/localhost to test.
  
  Another method is edit /etc/issue, remove .\O.
Enable OpenRC log

Edit /etc/rc.conf and set rc_logger="YES".
Set root password
```
# passwd
```

Networking

dhcpcd

We don't follow Handbook's net config. net-misc/netifrc requires support from DHCP, while net-misc/dhcpcd can handle network configuration alone.
```
# emerge -avt net-misc/dhcpcd
# rc-update add dhcpcd default
# emerge -avt net-wireless/wpa_supplicant
```
1. Personal DNS servers can be put into /etc/resolv.conf.head and/or /etc/resolv.conf.tail.
2. In case of the network interface card should be configured with a static IP address, entries can also be manually added to /etc/dhcpcd.conf.
3. If need GUI tool, use NetworkManager instead of wicd since the later one don't support nl80211 driver. Also Networkmanager depends on wpa_supplicant and dhcpcd or dhcpclient.
4. Reference: Network management using DHCPCD; wpa_supplicant; Handbook:AMD64/Networking/Wireless; configuration example; wpa_supplicant.conf for sMobileNet in HKUST; wpa_supplicant.conf.

wpa_supplicant

dhcpcd manages Ethernet connection on boot while revoking wpa_supplicant hook script to initiate wireless networking.
Remove the recommended options GROUP=wheel and update_config=1 from Wiki for security reason.

/etc/wpa_supplicant/wpa_supplicant.conf.

ctrl_interface=DIR=/var/run/wpa_supplicant
ctrl_interface_group=0
fast_reauth=1
eapol_version=1
ap_scan=1

network={
        disabled=0
        ssid="public-wifi"
        key_mgmt=NONE
        priority=-999
}

network={
        disabled=1
        ssid="Network1"
        proto=WPA RSN
        key_mgmt=WPA-EAP
        pairwise=CCMP TKIP
        group=CCMP TKIP 
        eap=PEAP
        identity="XXXXXXXX"
        password="YYYYYYYY"
        ca_cert="/etc/ssl/certs/Thawte_Premium_Server_CA.pem"
        phase1="peaplabel=0"
        phase2="auth=MSCHAPV2"
        priority=10
}

network={
        disabled=1
        ssid="Network2"
        proto=WPA RSN
        key_mgmt=WPA-EAP
        pairwise=CCMP TKIP
        group=CCMP TKIP 
        eap=PEAP
        identity="XXXXXXXX"
        password="YYYYYYYY"
# Don't need certificate
#       ca_cert="/etc/ssl/certs/Thawte_Premium_Server_CA.pem"
# If use the latest wpa_supplicant, try 'tls_disable_tls_v1_2=1'
#       phase1="tls_disable_tlsv1_2=1"
        phase1="peaplabel=0"
        phase2="auth=MSCHAPV2"
        priority=20
}

network={
        disabled=0
        ssid="Network3"
        #psk="12345678"
        psk=aaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbccccccccccccccccccc
        priority=30
}

For home wireless connection, usuaully you just need to specify ssid and psk arguments. psk can be a normal wifi password (call passphrase) or a 256-character string. When use normal password, need quotes, while 256-character string does not. How to generate the long string?

# wpa_passphrase wifi-ssid wifi-password >> /etc/wpa_supplicant/wpa_supplicant.conf

We'd better change the permissions to ensure that WiFi passwords can not be viewed in psk or passphrase by normal user account.

# chmod 600 /etc/wpa_supplicant/wpa_supplicant.conf`.

dhcpcd invokes wpa_supplicant through /lib/dhcpcd/dhcpcd-hooks/10-wpa_supplicant hook. From dhcpcd-6.10.0 onward, 10-wpa_supplicant hook is no longer there by default. We should copy /usr/share/dhcpcd/hooks/10-wpa_supplicant to /lib/dhcpcd/dhcpcd-hooks/10-wpa_supplicant.

lspci -k shows wireless driver in use is iwlwifi. However if specify -D iwlwifi on command line, wpa_supplicant will fail:

Unsupported driver 'iwlwifi'

The actual driver in use was nl80211.

If you have installed net-misc/netifrc (by default from stage3) and created /etc/ini.d/net.* and /etc/conf.d/net files, refer to Migration from Gentoo net.* scripts.
wpa_supplicant-2.4 might cause authentication problem for PEAP Wifi. Refer to Downgrade Package && wpa_supplicant && local overlay

System tools

Portage tools

# emerge -avt eix; eix-update
# emerge -avt gentoolkit

System logger.
```
# emerge -avt app-admin/syslog-ng app-admin/logrotate
# rc-update add syslog-ng default
```
By default, there is a line
```
log { source(src); destination(console_all); };
```
in /etc/syslog-ng/syslog-ng.conf, which outputs system logs to /dev/tty12. That's to say, any users can read system logs by switching to the 12th virtual terminal. For security reason, comment out that line if there are multiple users.

Details on logrotate for cron jobs refer to Cronie and Anacron.
Cron daemon

A cron daemon executes scheduled commands. It is very handy if some command needs to be executed regularly (for instance daily, weekly or monthly).
```
# echo "sys-process/cronie anacron" > /etc/portage/package.use/cronie (opt)
# emerge -avt sys-process/cronie
# rc-update add cronie default
```
1. anaron USE is not must if no rigid shechuled tasks.
2. cronie must be in a runlevel to shedule logrotate.
3. Details on running scheduled tasks based on input from the command crontab, refer to Cronie and Anacron.

# emerge -avt sys-apps/mlocate, file indexing
# emerge -avt sys-fs/ntfs3g, mounting NTFS filesystem
# rc-update add sshd default, incoming SSH connection (opt)

Bootloader - Grub2

Installation

# echo 'GRUB_PLATFORMS="efi-64"' >> /etc/portage/make.conf
# emerge -avt grub:2 os-prober
# mount /dev/sda10 /boot; mount /dev/sda2 /boot/efi
# grub-install --target=x86_64-efi [--bootloader-id=Gentoo_Grub] [--efi-directory=esp] [--removable] [/dev/sda]

This must be the 1st step. Otherwise it would show error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist.
Default to slot 2. If there are other operating system (i.e. dual boot), os-prober is suggested.
Make sure boot and EFI partition are mounted. Because Gentoo, Ubuntu, Windows share the EFI partition, we mount the shared EFI partion here.

A bit of explanation

By default, EFI partition is mounted at /boot/efi. Actually, we can mount it anywhere as long as --efi-directory is correctly set.

Install the GRUB UEFI application grubx64.efi to esp/EFI/Gentoo_Grub/ and install Grub modules to /boot/grub/x86_64-efi/.

Chceck --removable argument, without which the booting entry would not show up in UEFI NVRAM list within BIOS setting. You will find esp/EFI/BOOT/BOOTX64.EFI is just a copy of esp/EFI/Gentoo_Grub/grub64.efi.

You'd better set INSTALL_DEVICE when there exists multiple disks and/or multiple EFI partitions in case of ambiguity.

As long as the /boot and /boot/efi partitions are mounted, grub-mkconfig automatically detects other operating systems and writes to /etc/grub.d/30_os_prober through sys-boot/os-prober.

Chainloading Windows system manully in /etc/grub.d/40_custom. The traditional chainloader +1 does not boot Windows based on UEFI.

menuentry "Microsoft Windows 8.1 x86_64" {
    insmod part_gpt
    insmod fat
    insmod search_fs_uuid
    insmod chain
    search --fs-uuid --no-floppy --set=root $hints_string $fs_uuid
    chainloader /efi/Microsoft/Boot/bootmgfw.efi
}

The next is to replace $hints_string and $fs_uuid. This is where os-prober comes into playing a role.

# grub-probe --target=hints\_string /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi
# grub-probe --target=fs\_uuid /boot/efi/EFI/Microsoft/Boot/bootmgfw.efi

Print $hints_string.
Print fs_uuid.
Now replace the relevant values.
Refer to Windows installed in UEFI-GPT Mode menu entry and Can GRUB2 share the EFI system partition with Windows?.

# grub-mkconfig -o /boot/grub/grub.cfg

Get out of Chroot - Reboot

Exit the chrooted environment: exit and unmount all mounted partitions:
```
# exit
# umount -lv /mnt/gentoo/home
# umount -lv /mnt/gentoo/boot{/efi,}
# umount -lv /mnt/gentoo/dev{/shm,/pts,}
# umount -lv /mnt/gentoo{/proc,/sys,}
# reboot
```
When rebooting, if the LiveDVD usb stick is still plugged onto the computer, the chainload to Ubuntu grub does not work. You find error: disk hd0,gpt2 not found. This is because Grub treats the USB stick as hd0 while the hard disk as hd1. You can unplug the USB, and CTRL+ALT+DEL. Another way is to edit the Grub menu from hd0 to hd1, then press F10 to boot.

Finalizing

User

The very first thing after getis to create a regular user account:

# check all accounts
[root@host ~]# passwd -Sa

# create a new one
[root@host ~]# useradd -G wheel,video -ms /bin/bash <username>
[root@host ~]# passwd <username

Compared to useradd, adduser has a nice interactive mode.

Check wireless network.

If fail to connect,
```
# emerge -avt linux-firmware
```
Remember we have fetched the package sources.

System upate

# eix-sync
# emerge -avtuDN --with-bdeps=y @world
# dispatch-conf (opt)
# revdep-rebuild -pv (opt)
# emerge -avt @preserved-rebuild
# emerge -av --depclean

For >=portageq-2.2.16, emaint sync replaces original emerge –sync to sync Portage tree.
–with-bdeps=y will calculate build time dependencies for updates. Append this argument occasionally.
If configuration needs updated, there will be a corresponding ._cfg* file.
As a tool of gentoolkit, revdep-rebuild is Gentoo's Reverse Dependency rebuilder. It will scan the installed ebuilds to find broken packages as a result of an upgrade of their dependencies. Those packages will be re-merged. However, it can also happen that a given package does not work with the currently upgraded dependencies, in which case you should upgrade the broken package to a more recent version. revdep-rebuild will pass flags to emerge which lets you use the –pretend flag to see what is going to be emerged again before going any further.
emerge --info \| grep FEATURES prints preserve-libs FEATURE is enabledby default. It tells Portage to preserve libraries when soname's change during upgrade or downgrade, only as necessary to satisfy shared library dependencies of installed consumers/packages. Preserved libraries are automatically removed when there are no remaining consumers, which occurs when consumer packages are rebuilt or uninstalled. Ideally, rebuilds are triggered automatically during updates, in order to satisfy slot-operator dependencies.

However, before emerge exits during installing updates, if there are remaining preserved libraries because slot-operator dependencies have not been used to trigger automatic rebuilds, then emerge will display a message:
```
!!! existing preserved libs:
>>> package: sys-libs/libfoo-1
 * - /lib/libfoo.so.1
 *      used by /usr/bin/bar (app-foo/bar-1)
Use emerge @preserved-rebuild to rebuild packages using these libraries
```
Clean packages out of dependency tree.

Cleans the system by removing packages that are not associated with explicitly merged packages. Depclean works by creating the full dependency tree from the @world set, then comparing it to installed packages. Packages installed, but not part of the dependency tree, will be uninstalled by depclean.

depclean is the last stage of system update.

Chroot - a complete routine

Boot with LiveDVD, then

# swapon /dev/sda7
# mount /dev/sda12 /mnt/gentoo
# mount /dev/sda10 /mnt/gentoo/boot
# mount /dev/sda2 /mnt/gentoo/boot/efi
# cp -L /etc/resolv.conf /mnt/gentoo/etc
# mount -t proc proc /mnt/gentoo/proc
# mount --rbind /sys /mnt/gentoo/sys
# mount --rbind /dev /mnt/gentoo/dev 
# mount --make-rslave /mnt/gentoo/sys 
# mount --make-rslave /mnt/gentoo/dev
# chroot /mnt/gentoo /bin/bash
# source /etc/profile && export PS1="(chroot) $PS1"

System notes

Use ffmpeg instead of libav including the USEs.
Prefer openssl/ssl USE (dev-libs/openssl) to gnutls USE (net-libs/gnutls) for SSL/TLS secure connection. Please Google openssl vs gnutls. Gnutls seems to suffer library bug.
1. dev-libs/openssl: full-strength general purpose cryptography library (including SSL and TLS).
2. net-libs/gnutls: a TLS 1.2 and SSL 3.0 implementation for the GNU project.
Vaapi

ffmpeg, mesa, mpv, and firefox enable vaapi USE.

Globally USEs:

USE="${CPU_FLAGS_X86} -bindist -qt4 -qt5 -libav vaapi"

Try to to –with-bdeps=y occasionally on system update.

Updates fundamental dependencies like git and ffmpeg.
Try –oneshot -1 when re-merging dependencies.

This option should only be used for packages that are reachable from the @world package set (those that would NOT be removed by –depclean).
Use UUID to identify a partition instead of /dev/sdaxy.
Python

Since python-exec-2.3 / eselect-python-20160207, the preferred method of altering Python configuration is to use the edit mode that opens python-exec.conf in an editor. The previous set mode is deprecated.
```
~ # eselect python list/edit`.
```
For example, remove Python 3.4 after upgrading to 3.5.
Perl update

The official way of upgrading Perl, e.g. from Perl 5.22 to Perl 5.24, is upgrading your entire world set, and upgrading Perl with it.
```
# emerge -avtuDN --with-bdeps=y --backtrack 30 @world
# perl-cleaner --all
```
1. If 30 does not help, try 100 instead.
2. Attention: both --with-bdeps=y and --backtrack 30 are required.

upgrading Gcc

After upgrading from 4.9 to 5.4, I encountered librime-1.2.9 bug.

~ # gcc-config --list-profiles
~ # gcc-config 2
~ # env-update && source /etc/profile
~ # emerge -av1 sys-devel/libtool
~ # gcc --version
~ # emerge -avc =sys-devel/gcc-4.9.4
~ # revdep-rebuild --library 'libstdc++.so.6' -- --exclude gcc (opt)

The last comman triggers over 60 packages rebuild.

SysRq
1. Alt - SysRq/PrintScreen - <argument> simutaneously; ThinkPad Fn is not required.
2. Or press the key one by one: Alt, [Fn] - SysRq, and argument.
3. The detailed arguments can be found at kernel sysrq. I generally unraw(r), sync(s), umount(u), and/or reboot(b) when my system is dead.

X Window System

X Window System, X11, and X (shortname) are used interchangebly.

X originated at the Massachusetts Institute of Technology (MIT) in 1984. The protocol has been version 11 (hence "X11") since September 1987.
Xorg Server.
1. Mainly x11-base/xorg-server (depending on x11-base/xorg-drivers and x11-libs/mesa).
2. vaapi USE is enabled to utilize hardware acceleration.
3. The i965 and i915 drivers split at system application (media-libs/mesa) level. The kernel always enable i915 instead.
Do NOT do:
```
# echo XSESSION="Xfce4" > /etc/env.d/90xsession
```
since root does not launch X. If really need, set in ~/.bash_profile (detailed below). It is referenced by /etc/X11/chooser.sh to determine command in /etc/X11/xinit/xinitrc.
Display Manager: GDM, LightDM, SDDM etc. DM presents we a graphical login window, equivalen of startx.

Most display managers source /etc/xprofile, ~/.xprofile and /etc/X11/xinit/xinitrc.d/.
xinit

We will use startx (wrapper of xinit binary) to read ~/.xinitrc (the default is /etc/X11/xinit/xinitrc or /etc/xdg/xfce4/xinitrc). One of the main functions of ~/.xinitrc is to dictate which client (i.e. Xfce4) for the X Window System is invoked with startx or xinit programs on a per-user basis.
Window Manager: Awesome, OpenBox, Xfwm4 etc.
Desktop: Xfce and Xfce/Guide, KDE, Gnome etc.
1. Refer to XFCE_PLUGINS.
2. Similar to xfce-extra/xfce4-notifyd, x11-themes/gnome-icon-theme can be explicitly emerged along with xfce-base/xfce4-meta. Details refer to Missing icons in Xfce4 configuration below.
3. Add -qt4 -qt5 to make.conf.
It's essential to outline the diagram of X system:
1. login shell -> /etc/profile -> .bash_profile -> .bashrc -> startx/xinit
2. startx is a front wrapper to xinit program. startx/xinit launch X server (Xorg/X) and X client (Xfce4).
3. Command: startx/xinit --
4. Without command line options, startx try to parse .xinitrc and .xserver respectively.
5. Put environment variables for GUI applications into ~/.xinitrc while for terminal into ~/.bashrc or ~/.bash_profile.

X Configuration

Per-user Theming locations:
1. *.desktop file should be placed in ~/.local/share/applications/.
2. Themes are placed in ~/.local/share/themes/.
3. Icons are placed in ~/.local/share/icons/.

consolekit is emerged as dependency during X installation. But Shutdown, Suspend etc. are greyed after getting into Xfce4.

# echo "sys-auth/consolekit pm-utils" >> /etc/portage/package.use/consolekit
# emerge -avt consolekit
# rc-update add consolekit default

Xinit (Dispaly Manager)

The default OpenRC Xinit conguration is located under /etc/X11/xinit. Customization:

First get a copy of the default. The reason of doing this (instead of creating one from scratch) is to preserve some desired default behaviour in the original file, such as sourcing shell scripts from /etc/X11/xinit/xinitrc.d.
```
# cp /etc/X11/xinit/xinitrc ~/.xinitrc
```
The default Xinitrc examines Xresources and Xmodmap and prepares Consolekit and Dbus. The last section either tries twm and xterm or lanuches XSESSION. We can change exec $command to echo $command and test the script within virtual terminal (console). It looks like:

ck-launch-session dbus-launch –sh-syntax –exit-with-session /etc/X11/Sessions/Xfce4

The default is enough to launch Xfce4 desktop. However we might require extra stuff like:
```
# Refer to /etc/X11/xinit/xinitrc
# or replace 'startxfce4' with 'xfce4-session':

#exec ck-launch-session dbus-launch --sh-syntax --exit-with-session xfce4-session
exec $command
```
startx
```
# startx -- vt7 -nolisten tcp
```
Arguments after two bashes are passed to Xorg server.
1. The xserverrc file is a shell script responsible for starting up the Xorg server. Both startx and xinit execute ~/.xserverrc if it exists, startx will use default /etc/X11/xinit/xserverrc otherwise.
2. Attention, -nolisten tcp is to disallow TCP connection to X server.
3. vt7 must be appended, otherwise switches between X and virtual terminal would freeze the whole X seesion to death. Refer to Intel HD3000 Tearing/Corruption/Glitch.
The default /etc/X11/xinit/xserverrc fails to set the correct virtual terminal to start X server. It depends on $XDG_VTNR which is not available on OpenRC but Systemd.

An alternative is:
```
# cp /etc/X11/xinit/xserverrc ~/.xserverrc
```
Modify a little bit:
```
#!/bin/sh
if [ -z "$XDG_VTNR" ]; then
  exec /usr/bin/X -nolisten tcp "$@" vt7
else
  exec /usr/bin/X -nolisten tcp "$@" vt$XDG_VTNR
fi
```
If a Display Manager is used, those arguments would be handled correctly without user concern.

Automatic Xfce4 on login

Edit ~/.bash_profile:

# /etc/skel/.bash_profile

# This file is sourced by bash for login shells.  The following line
# runs your .bashrc and is recommended by the bash info pages.
if [[ -f ~/.bashrc ]] ; then
        . ~/.bashrc
fi

# If ~/.xinitrc does not exist, startx takes it as a fallback
XSESSION="Xfce4"

if shopt -q login_shell; then
        [[ -t 0 && $(tty) == /dev/tty1 && "$USER" == "username" && ! $DISPLAY ]] && exec startx -- vt7 -nolisten tcp
fi

If ~/.xserverrc exists, vt7 -nolisten tcp part can be removed.
If you would like to remain logged in when the X session ends, remove exec.

Clear Xfce configuration

# rm -r ~/.cache/sessions
# rm -r ~/.config/xfce*
# rm -r ~/.config/Thunar

Hide unmounted partitions from user interface (opt)

When you get into the Xfce4 desktop, you may found many unnecessary disk icons on the desktop or thunar sidebar. It's annoying. Use udev/udisks utility.

Edit /etc/udev/rules.d/80-hide-disks.rules:
```
ENV{ID_FS_UUID}=="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", ENV{UDISKS_IGNORE}="1"
or
KERNEL=="sdaXY", ENV{UDISKS_IGNORE}="1"
```
To load the new rules without reboot:
```
# udevadm control --reload
# udevadm trigger
```
1. XY is the disk partition number you would like to hide. As noted in the reference below, UDISKS_PRESENTATION_HIDE is deprecated and replaced by UDISKS_IGNORE.
2. Refer to /lib64/udev/rules.d/80-udisks2.rules, read section:
  
  Devices which should not be display in the user interface
3. Refer to udev 99-hide-disks.rules is no longer working and udisks on archwiki.

Touchpad

If Touchpad does not support tap click. As long as x11-drivers/xf86-input-synaptics is installed, a default configuration is located at /usr/share/X11/xorg.conf.d/50-synaptics.conf. Copy it to /etc/X11/xorg.conf.d/ and edit:

Section "InputClass"
    Identifier "touchpad"
    Driver "synaptics"
    MatchIsTouchpad "on"
        Option "TapButton1" "1"
        Option "TapButton2" "2"
        Option "TapButton3" "3"
        Option "VertEdgeScroll" "on"
        Option "VertTwoFingerScroll" "on"
        Option "HorizEdgeScroll" "on"
        Option "HorizTwoFingerScroll" "on"
        Option "CircularScrolling" "on"
        Option "CircScrollTrigger" "2"
        Option "EmulateTwoFingerMinZ" "40"
        Option "EmulateTwoFingerMinW" "8"
        Option "CoastingSpeed" "0"
        Option "FingerLow" "35"
        Option "FingerHigh" "40"
        Option "MaxTapTime" "125"
EndSection

You can also set a temporary config at command line. Check command line synclient.

New plug-in sync system - Layman/Overlay

Layman adds, removes, syncs etc. Overlays on system.
With the help of Portage new plug-in sync system, Overlays can be easily synced by Portage emaint. Therefore, Layman is purely used to add/remove Overlays.
From 2.3.0 onwards, Layman supports the new plug-in sync system naturally. But current stable version is 2.0.0. Ether accept 2.3.0 or manually write new plug-in sync configurations for newly added Overlays.
```
# emerge -avt app-portage/layman
```
By default, the subversion USE is not enabled, which support subversion-based Overlays.

After emerging <layman-2.3.0, don't do any configuration (i.e. PORTDIR, PORTDIR_OVERLAY) as told on Wiki since we will manually add new plug-in sync system configuration.

Add/remove an Overlay:

# layman -a gentoo-zh
# layman -d gentoo-zh

Manual Overlay configuration (<layman-2.3.0)

Add /etc/portage/repos.conf/gentoo-zh.conf:
```
[gentoo-zh]
location = /var/lib/layman/gentoo-zh
sync-type = git
sync-uri = git://github.com/microcai/gentoo-zh.git
auto-sync = yes
```
We choose git instead of traditional rsync. If needed, official Portage tree gentoo.conf can be changed to git type as well.

Portage tree switch to git sync

# mv /usr/portage/distfiles ~/
# mv /usr/portage/packages ~/
# rm -rf /usr/portage/*
# emaint sync -r gentoo
# mv ~/distfiles /usr/portage
# mv ~/packages /usr/portage

We delete the Portage tree and emaint sync -r will git clone locally. Refer to Portage git sync.

(opt) Sync
```
# eix-sync, or
# emaint sync
```

Applications

Xfce4 goodies

xfce4-power-manager; xfce-extra/xfce4-pulseaudio-plugin and media-sound/pavucontrol; xfce4-screenshooter.
1. Go to Applications > Settings > Keyboard, Application Shortcuts. Add the xfce4-screenshooter -r command to use the PrtSc key.
Recommendations

apg, ~~guake~~, ~~st term~~, rxvt-unicode, wgetpaste, weechat, wps-office, evince, TeXLive, remmina/freerdp.
1. wps math formula fonts and fontconfig. Those fonts are essential to display formulas. However, WPS-linux does not have built-in formula creation function due to copyright.
Sound - ALSA/PulseAudio
1. PulseAudio depends on (both kernel and package parts) of ALSA - they coexist instead of excluding each other. It serves as a proxy to sound applications using existing kernel sound components like ALSA or OSS.
2. ALSA includes both kernel sound card drivers and userspace package media-libs/alsa-lib, while PulseAudio builds on top of ALSA kernel part while offering userspace package media-sound/pulseaudio.
3. PulseAudio is configured to automatically detect all sound cards and manage them. It takes control of all detected ALSA devices and redirect all audio streams to itself, making the PulseAudio daemon the central configuration point.
Check if alsa-lib and alsa-utils are installed or not. By default, the alsa USE flag is enabled in profile, so these packages should be emerged already.

By default system sound is muted. Use alsamixer to unmoute:
```
~ $ alsamixer
```
To test sound:
```
# speaker-test -t wav -c 2
```
Enable global pulseaudio USE:
```
# /etc/portage/make.conf
~ # euse -E pulseaudio
# remove all users from *audio* group
~ # gpasswd -d username audio
~ # emerge -avtuDN --with-bdeps=y @world
```
1. Enabling pulseaudio USE flag will pull in media-sound/pulseaudio automatically.
2. If you plan running pulseaudio in the system-wide mode, then the special user pulse should still be in the audio group in order to have access to the sound card.
GStreamer support:
```
~ $ gconftool-2 -t string --set /system/gstreamer/0.10/default/audiosink pulsesink
~ $ gconftool-2 -t string --set /system/gstreamer/0.10/default/audiosrc pulsesrc
```
From what we've done, you know enabling pulseaudio is quite easy as no kernel options required like ALSA.

xfce4-pulseaudio-plugin reports "pavucontrol binary not found":
```
root@tux / # emerge -avt media-sound/pavucontrol
```
Firefox
1. To make use of vaapi, make sure ffmpeg and hwaccel USEs are enabled. BTW, ffmpeg should enable vaapi USE too.
2. For Media Source Extensions, turn on media.fragmented-mp4.exposed, media.fragmented-mp4.ffmpeg.enabled, media.mediasource.enabled, media.mediasource.mp4.enabled and media.mediasource.webm.enabled in about:config, while disabling media.fragmented-mp4.use-blank-decoder.
3. Add FoxyProxy Standard, uBlock Origin, NoScript (and/or RefControl), HTTPS Everywhere, DISCONNECT, Open With etc. add-ons. Remove unecessary default whitelist of NoScript plugin.
4. privacy.trackingprotection.enabled, Network.proxy.socks_remote_dns to True.
5. network.IDN_show_punycode to be True.
6. browser.cache.check_doc_frequency set to 0/1/2/3; Ctrl + Shift + I, Gear, Advanced Settings, Disable Cache (when toolbox is open).
7. Harden Firefox security and disable useragent. Like general.useragent.vendor/override.
8. fingerprint test.
Fcitx
```
# echo "app-i18n/fcitx gtk gtk3 -table" >> /etc/portage/package.use/fcitx
# emerge -av fcitx fcitx-rime fcitx-configtool
```
1. table USE will draw several built-in input methods (i.e. Wubi) which fail to meet my requirement. fcitx-rime will be installed instead.
2. I have disabled the QT globally. So no qt4 or qt5 USE.
3. fcitx-configtool is a GTK configuration tool. To install QT configuration tool for KDE, install kcm-fcitx.
Put the following lines before exec of ~/.xinitrc:
```
export GTK_IM_MODULE=fcitx
export QT_IM_MODULE=xim
export XMODIFIERS="@im=fcitx"
```
They should be ahead of exec startxfce4 or xfce4-session. Commands after exec won't be executed! Refer to xfce4安装fcitx不能激活！很简单的一个原因！.
FFmpeg
```
# echo "media-video/ffmpeg vaapi librtmp openssl vpx" >> /etc/portage/package.use/ffmpeg
# emerge -avt ffmpeg
```
1. ffmpeg is emerged by some other packages, one of which might be Firefox or Mpv.
2. Add -libav to make.conf in favor of system-wide ffmpeg.
3. v4l and libv4lUSE for the webcamera.
4. vaapi USE for hardware decoding.
5. librtmp USE to replace the native RTMP implementation (bad performance).
6. network (default) and openssl to support HTTP/HTTPS streaming.
Mpv
```
# echo "media-video/mpv vaapi" >> /etc/portage/package.use/mpv
# echo "=media-video/mpv-9999 **" >> /etc/portage/package.keywords/mpv
# emerge -avt mpv
```
1. You are recommended to accpet mpv live build.
2. Rely on ffmpeg and youtube-dl.
3. Enable vaapi USE for mpv and *ffmpeg to utilize hardware acceleration.
4. Re-merge mpv if ffmpeg is upgraded
5. The default lua USE flag of mpv brings along a youtube-dl hook script. mpv url revokes youtube-dl to stream video playback.
  
  And make sure ffmpeg is installed with network USE flag. To stream HTTPS url, enable either openssl or gnutls USE of ffmpeg. Ether openssl is fine and you don't need both.
6. youtube-dl is suggested to install under Python3 virtual environment by pyvenv-3.4 thus receving immediate updates. If youtube-dl does not stream Chinese url, try mpv $(youtube-dl –get-url http://v.youku.com/v_show/id_XMTU2Mjc4MTc4NA==.html).
  
  After installing youtube-dl in Python3 virtual environment:
```
$ ln -sf /absolute/path/to/py3venv/youtube-dl ~/.config/mpv/youtue-dl
```
  If youtube-dl sucks, could try you-get (for Chinese urls) and livestreamer (for live broadcast). If the url is blocked, we should first enable proxychains or torify.
7. When streaming, mpv might throtle/stuck a lot waiting for cache. This is due to ISP throtling; video provider streaming limit; network bandwidth etc.
  1. Set bigger –cache value; press SPACE to pause for fechting more cache.
  2. Run youtube-dl url and mpv partial-downloaded-file separately (say in two terminals). If does not play the partial file (HASH video and audio are usually downloaded separately), try -f best argument which tell to download audio and video into a single file. If downloading audio and video separately, there would be no audio or video when playing.
8. We can use Firefox addons Open With to revoke mpv.
```
/usr/bin/proxychains /usr/bin/mpv  --
or
~/workspace/virtualenv3/bin/you-get -p /usr/bin/mpv --
```
  The last two hypens means arguments afterwards are input files/urls. Since using the Open With addon, save more CPU and memory than Firfox.
9. Details on youtube-dl and you-get, refer to their Github pages.
10. Some videos with aac audio track won't show video (only audio). -v --no-config gives:
  
  [lavf] Edit lists are not correctly supported (FFmpeg issue).
  
  Before ffmpeg fixes the edit list parsing, add --demuxer-lavf-o=ignore_editlist=1 option.
Emacs
1. Use athena Xaw3d -gtk -gtk3 -motif USEs to replace GTK toolkit if multiple monitors are used. Refer to daemon mode bug reddit and wiki.
2. xft for Fontconfig. libxml2 support shr enables eww HTML viewer. gnutls/ssl supports Gnus IMAP connection.
3. Chinese input with Fcitx. Change the Fcitx input method trigger to WIN+I instead of CTRL+SPACE.
4. If X frame does not input Chinese, emerge two fonts: font-adobe-100dpi and font-adobe-75dpi. You can read post-emerge message:
```
if use X; then
    elog "You need to install some fonts for Emacs."
    elog "Installing media-fonts/font-adobe-{75,100}dpi on the X server's"
    elog "machine would satisfy basic Emacs requirements under X11."
    elog "See also http://www.gentoo.org/proj/en/lisp/emacs/xft.xml"
    elog "for how to enable anti-aliased fonts."
    elog
fi
```
5. emacs-daemon
```
# emerge -avt emacs-daemon
# cd /etc/init.d/
# ln -sv emacs emacs.usernmae
# rc-update add emacs.username default, (do NOT add */etc/init.d/emacs* to default runlevel directly)
# rc-service emacs.username start
```
6. XIM
  
  Since emacs-daemon is launched before (OpenRC init) setting variable (XMODIFIERS="@im=fcitx" by startx in ~/.xinitrc), emacsclient -c cannot find external Fcitx's XIM server. Possible solutions:
```
#/etc/init.d/emacs.username
export SHELL EMACS EMACS_TIMEOUT EMACS_DEBUG
export XMODIFIERS=@im=fcitx
```
  We set the variable in init start() function. Alternatively, set XMODIFIERS through --env argument of start-stop-daemon.
```
start-stop-daemon --start \
  --user "${USER}" --pidfile "${PIDFILE}" --chdir "${home}" \
  --env XMODIFIERS="@im=fcitx" --exec "${EMACS_START}" \
 -- ${EMACS_OPTS}
```
  However, direct modification init script is discouraged (suppose you are on a multiple user accounts system). emacs-daemon has a wrraper script (/usr/libexec/emacs/emacs-wrapper.sh) to start the daemon with a login shell. We can set XMODIFIERS in bash startup script (~/.bashrc or ~/.bash_profile).
  
  A more elegant way is to create a new copy of /etc/conf.d/emacs and do update there:
```
# pushd /etc/conf.d
# cp emacs emacs.username
# nano -w emacs.username
export XMODIFIERS="@im=fcitx"
```
  Don't worry! openrc-run looks for both /etc/conf.d/emacs and /etc/conf.d/emacs.username.
  
  Emacs uses XIM to support external input method instead of GTK/QT toolkit. Details on XMODIFIERS/XIM, refer to Fcitx's official page. Read more at fcitx not work on emacsclient –create-frame and app-emacs/emacs-daemon should allow for setting up a proper environment.
Nano

After installing Emacs, nano will be removed on next emerge -avc (check package virtual/editor). To keep nano:
```
 # emerge -avt --noreplace nano
```
Turn on a few Nano options in /etc/nanorc: set autoindent, set backup, set tabsize 4 etc. If need to totally disable an option, use *unset
Archive
```
# emerge -av file-roller
# emerge -av thunar-archive-plugin
```
1. file-roller is a front-end GTK interface - Archive Manager, which needs background Archiver support - zip/unzip, bzip2, tar, 7zip, unrar etc.
2. thunar-archive-plugin is a Thunar plugin (right-click menu). If thunar-archive-plugin cannot find a suitable archive manager, check thunar archive plugin cannot integrate with file-roller and doesn't work anymore with recent file-roller.
3. By default, unzip cannot extract arhives containing no-ASCII file names (i.e. archives created on Chinese Windows). Enable natspec USE that brings in -O command line option. Check -O by -h as it's not present in man page. Attention, natpsec requires LC_CTYPE be zh_CN.
4. (opt) p7zip has a higher data compression ratio.
  
  p7zip brings 3 archive binaries, namely 7z, 7za and 7zr. Only 7zr can extract non-ascii zip files. However file-roller will take 7z and 7za primarily. Rename /usr/bin/7z and /usr/bin/7za to something else.

NetEase-MusicBox

# emerge -avt media-sound/mpg123
$ source ~/opt/pyvenv3/bin/activate
$ pip(3) install NetEase-MusicBox
$ pip(3) install lxml

If encountered error during installing NetEase-MusicBox,

running build_ext
running build_configure
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in `/tmp/pip-build-y5q1g0ib/pycrypto':
configure: error: cannot run C compiled programs.

The reason is that /tmp is mounted as noexec. To temporarily solve it,

~ $ TMPDIR=~/tmp pip(3) install NetEase-MusicBox

Sphere

Blacklist/install modules

Prevent rarely used modules from loading on boot. Always refer to command line tool lsmod, modinfo if unsure. Create /etc/modprobe.d/blacklist.conf and put unwanted modules there.
```
# /etc/modprobe.d/blacklist.conf
blacklist btusb
blacklist btbcm
blacklist btintel
blacklist bluetooth
```
We can replace blacklist with install … true/false which makes a difference. A blacklisted module will be loaded if another non-blacklisted module depends on it, or if it is loaded manually. For example, bluetooth is blacklisted. If we manullay modprobe btusb. bluetooth module will load as a dependency. Hence, it's necessary to remove related init service (if exist) from runlevels. Otherwise blacklisted modules were loaded.

To ensure the modules are never inserted, even if they are needed by other modules or manually modprobe, use install. To install a module as true or false does not make much difference. false just files an extra error message while true silence it. Refer to Modprobe is better disabled by using /bin/true (not /bin/false.

Refer to arch wiki blacklist; changes to module blacklisting.
Intel Microcde (opt)
```
# emerge -av microcode-ctl
# rc-update add microcode_ctl boot
```
Refer to intel microcode. Updates to CPU microcode have to be re-applied each time the computer is booted, because the memory updated is volatile (despite the term firmware also being used for microcode).

According to the reference, you should run dmesg | grep -i microcode to check whether CPU microcode is updated or not. If not, I think there is not need to add microcode-ctl to boot level until microcode-ctl package is updated.

It's recommended to update CPU microcode through BIOS update. We can do this under Windows.
(opt) swapfile
```
# dd if=/dev/zero of=/mnt/1GB-swapfile bs=1M count=1024
# chmod 600 /mnt/1GB-swapfile
# mkswap /mnt/1GB-swapfile
# swapon /mnt/1GB-swapfile
# swapon -s
# ect /etc/fstab
/mnt/1GB-swapfile none swap defaults 0 0
```
1. Since OpenRC-0.22, please enable rc_need="localmount" in /etc/conf.d/swap.
2. Use dd to create a file occupying continuing disk space instead of a sparse file. Refer to swap partition vs file for performance?.
3. Refer to swap file creation.
It is recommended to turn down vm.swappiness by sysctl. By default, it is 60, and we can change it to 10 or so.
```
# /etc/sysctl.d 25-swappiness.conf

vm.swappiness=10
```