ZNHOO Whatever you are, be a good one!

CentOS RHRL

  1. Fedora RHEL CentOS
  2. Module
  3. rpm yum dnf
    1. rpm
    2. yum
    3. dnf
  4. Repositories
    1. EPEL not from RHEL
    2. Inline with Upstream Stable (IUS)
    3. Webtatic (DEPRECATED)
    4. CentOS-5.8
    5. Upgrade System
  5. rpmbuild
    1. Extracting RPM
  6. locale
  7. systemd
    1. logging
    2. systemctl mask
  8. SSH
  9. Create User Account
  10. Pip/Virtualenv
  11. SS
    1. Server Json
    2. TCP Fast Open
      1. A quick test
    3. Systemd Unit
      1. simple Type
      2. forking Type
    4. kcptun

Fedora RHEL CentOS

  1. Fedora is a Linux distribution released under GPL.
  2. Red Hat, Inc. copy the source code from Fedora, make modifications, release it as Red Hat Enterprise Linux (RHEL) distribution.

    Due to GPL, Red Hat, Inc. is obligated to disclose RHEL source code. But Red Hat can make money from its professional after-sale services.

  3. CentOS (Community ENTerprise OS) is a Linux distribution built from the source of RHEL directly without any modification, with the intention of making CentOS binary compatible with RHEL such that library versions the same and binaries work on RHEL also work on CentOS.
    1. The main purpose of CentOS is to make RHEL available to pulbic users, benefitting from Red Hat's professional Linux service.
    2. Remove Red Hat, Inc. logo.
  4. Fedora - RHEL - CentOS.

Module

Module is a new packaging feature brought in from RHEL 8.

  1. Package

    Traditional .rpm file.

  2. Group and Environment

    A large set of packages as a whole for specific system categories like X window, desktop, security, development, etc.

  3. Module is a collection of a few packages, as a logical unit for a specific project, like Php and Nginx.

    1. Stream is the upstream version.
    2. Version is the build version.
    3. Profile is the sub-functionality of a module, similar to Gentoo's tag.

rpm yum dnf

The relation between rpm (Redhat Package Manager) command and yum (Yellow dog Updater, Modified) command is analogous to dpkg and apt-get. dnf is the new version of yum with performance improvement and module feature. From RHEL 8 onwards, dnf replaces yum as the default manager.

Basically, yum is more intelligent than rpm. rpm wants to know the exact location of target .rpm file while yum only needs the package name. yum also takes care of dependencies. However, rpm is somehow a lightweight tool with concise output.

Check 'SPECIFYING PACKAGE NAMES' in the man page of yum to check how to pass globs arguments. The full name of a .rpm package is as follows:

# <name>-<version>-<release>.<os>.<arch>.rpm

vim-enhanced-7.4.160-4.el7.x86_64
# name:     vim-enhanced
# version:  7.4.160
# release:  4
# os:       el7
# arch:     x86_64
# suffix:   rpm

centos-release-7-4.1708.el7.centos.x86_64
# name:     centos-release
# version:  7
# release:  4.1708
# os:       el7.centos
# arch:     x86_64
  1. Package version is also called major version, indicating the upstream source version from developers. Package release is also called minor version, indicating the final .rpm file by RHEL compliling/building/patching. Especially, RHEL may add customized patches or bug fixes to the major version, building a new .rpm package to the repository.
  2. os may be el6 (rhel6), centos6, el5, suse11 etc.

By design, it is not possible to install difference versions of the same package alongside. The workaround is to make the new version a different package. For example, python2 and python3 are actually different package names.

rpm

rpm manages package file directly, maintaining a database of package information located under /var/lib/rpm/. It does not care too much the package version but the RPM file you provided.

Query installed packages:

~ # rpm -qa name='globbing'
~ # rpm -qa | grep 'regex'

~ # rpm -qi pkg; rpm -qip pkg.rpm                     # query pkg information
~ # rpm -ql pkg; rpm -qlp pkg.rpm                     # list pkg files
~ # rpm -qR pkg; rpm -qRp pkg.rpm                     # list package it requires - dependencies

~ # rpm -qf /path/to/file                             # file owner
~ # rpm -qdf /path/to/file                            # file owner's man pages

Verify:

~ # rpm -V pkg; rpm -Vp pkg.rpm                       # verify

Verifying a package compares information about files installed locally with information about files taken from the package metadata stored in the RPM database. Any discrepancies are displayed.

Install:

~ # rpm -ivvh --test /path/to/pkg.rpm      # dry run
~ # rpm -ivh /path/to/pkg.rpm              # install
~ # rpm --reinstall -vh /path/to/same.rom  # reinstall

~ # rpm -Uvh /path/to/pkg.rpm              # upgrade
~ # rpm -Fvh /path/to/pkg.rpm              # refresh
~ # rpm -Fvh --force /path/to/old.rpm      # downgrade
  1. The path can be an ftp/http URL.
  2. -i, --install is the general form of installing a package.
  3. -U, --upgrade is the combination of -i, --install and -e. It firstly install a package and then erase earlier versions if present.

    -U does not require an earlier version is installed, so it can be a replacement of -i.

  4. -F, --freshen is similar to -U but mandates an earlier version is already installed.
  5. --force, --replacepkgs, --replacefiles and --oldpackage do the same thing.

Uninstall/Erase:

~ # rpm -evv pkg                     # uninstall a pkg

Kering:

~ # rpm --import /path/to/key.pub
~ # rpmkeys --import http://www.example.com/key.pub

Once imported, all public keys in the keyring can be managed as a package. Commands applied to a package also apply to a key like query, erase, information etc.

~ # ls /etc/pki/rpm-gpg/
~ # rpm -qa gpg-pubkey*
~ # rpm -q gpg-pubkey --queryformat "%{summary} ->%{version}-%{release}\n"
~ # rpm -qi gpg-pubkey-db42a60e

yum

The main configuration of YUM is /etc/yum.conf and specific repository configuration files are placed in /etc/yum.repos.d/. Within the main configuration file, cachedir defines the location of cached .rpm packages (defaults to /var/cache/yum/$basearch/$releasever).

By default, of the built-in repositories, only 'CentOS-Base.repo' is enabled.

  1. $releasever indicates the main version. For example, the main version of CentOS 5.8 is 5.
  2. $arch indicates the architecture like x8_64, i386/i586/i686.
  3. $basearch is similar to $arch, but is base architecture: either x86_64 or i386.

Cache:

~ # yum clean [ all | packages | metadata | expire-cache | rpmdb | plugins ]
~ # yum makecache

After any edits of repo files, in order to clear any cached information, and make sure the changes are immediately recoginized, as root run yum clean all.

Query:

~ # yum history

~ # yum list [--all | --installed | --available | --upgrades] [pkg]
~ # yum info [--all | --installed | --available | --upgrades] [pkg] # more details
~ # yum search pkg

~ # yum repolist [--all | --enabled | --disabled]
~ # yum check-update
~ # yum upgrade [pkg]

~ # yum provides /path/to/file             # rpm -qf

~ # repoquery --whatprovides '*bin/grep'
~ # repoquery --list pkg
  1. update is synonym to upgrade.
  2. list –updates is almost the same as check-update. As the command form implies, check-update is useful in Shell script while list –updates is for humans on the command line.

    Please pay attention, their exit status code difference.

  3. 'yum provides' only search which package provides the pathname. 'rpm -qf' requires that the package is installed or existence of the .rpm file.

    Tp speed up the search, use 'repoquery' from 'yum-utils' package. Especially, it list all files provided by a package even it is not installed.

Install:

~ # yum [-y] install pkg1 pkg2
~ # yum reinstall pkg1 pkg2

~ # yum upgrade pkg1 pkg2

~ # yum remove pkg1 pkg2

To install packages for group or environment set:

~ # yum group summary
~ # yum group [ list | info |install | upgrade | remove | mark ] [grp]

dnf

dnf is compatible with but more powerful than yum. One of the newest feature provided is module.

~ # dnf module -h
~ # dnf module list nginx
~ # dnf module info nginx

# NAME:STREAM:VERSION:CONTEXT:ARCH/PROFILE
~ # dnf module info nginx:1.16:8010020191122190044:cdc1202b:x86_64/common
~ # dnf module install nginx:1.14/common

The purpose of module is to manage package versions and profiles in an integrated way. The following two methods is equivalent:

~ # dnf module install nginx
~ # dnf install nginx

The only difference is that dnf module can provide better control when selecting pakage stream, version, arch etc.

Repositories

Of the 3rd party repositories, IUS and Remi (both depend on EPEL) is recommended over Webtatic.

EPEL not from RHEL

EPEL (Extra Packages for Enterprise Linux) is open source and free community based repository project from Fedora team which provides 100% high quality add-on software packages.

~ # yum info epel-release
~ # yum install epel-release
~ # yum repolist enabled

To specify repository when installing package:

~ # yum --enablerepo=epel install nginx

The --enablerepo option overides the permanent option setting in the /etc/yum/*.repo files for only the current command. --disablerepo does the opposite for enabled repos.

Inline with Upstream Stable (IUS)

~ # rpm -Uvh https://centos7.iuscommunity.org/ius-release.rpm

Read IUS Usage for more information.

Webtatic (DEPRECATED)

The Webtatic Yum repository is a CentOS/RHEL repository containing updated web-related packages like git.

~ # rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm

CentOS-5.8

CentOS-5 reached the end of its lifecycle as of March 31, 2017. The official baseurl is deprecated and moved to a backup location Vault. Packages there is no longer maintained or upgraded, and may suffer from security risks.

However, we sometimes have to maintain ancient releases for whatever reasons. To install packages through yum, point repositories to the new location: either manually edit /etc/yum.repos.d/CentOS-Base.repo or install centos-release-5-8.el5.centos.x86_64.rpm file.

  1. For each section (i.e. Base) Comment out both the mirrorlist= and baseurl= lines.
  2. Add a new baseurl line:

    baseurl=http://vault.centos.org/5.8/os/$basearch/
    
    1. Replace /os/ part approriately for other sections like /updates/.
    2. It is highly recommended to replace 5.8 with 5.11, upgrading the system to CentOS 5.11

    Alternatively, install the corresponding repository package:

    root@tux ~ # wget http://vault.centos.org/5.11/os/x86_64/CentOS/centos-release-5-11.el5.centos.x86_64.rpm
    root@tux ~ # rpm -Uvh centos-release-5-11.el5.centos.x86_64.rpm
    

    For other repositories like EPEL, Webtatic etc., find their archive URL and repeat the same procedures.

  3. Update

    root@tux ~ # rpm -q centos-release; cat /etc/redhat-release; cat /etc/centos-release
    root@tux ~ # yum clean all; yum makecache
    root@tux ~ # yum list updates; yum update
    root@tux ~ # reboot
    root@tux ~ # yum install epel-release; yum install git
    

Here is an example of manually edited Centos-Base.repo:

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=http://vault.centos.org/5.11/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#released updates 
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
baseurl=http://vault.centos.org/5.11/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
baseurl=http://vault.centos.org/5.11/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
baseurl=http://vault.centos.org/5.11/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
baseurl=http://vault.centos.org/5.11/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

Upgrade System

~ # dnf check-update
~ # dnf upgrade kernel
~ # dnf upgrade
~ # reboot

rpmbuild

For how to build RPM package, refer to "logging/filebeat-zookeeper-kafka".

Extracting RPM

Upon receving a .rpm file, we can extract the contents with rpm2cpio and cipo:

~ $ mkdir dst; cd dst
~ $ rpm2cpio /path/to/pkg.rpm | cpio -idmv
~ $ find .
  1. -i: extract;
  2. -d: make directories;
  3. -m: preserve modification time;
  4. -v: verbose.

locale

Redhad-based distributions places locale customization in /etc/locale.conf. Check man locale.conf.

systemd

~ # systemctl show/cat unit.service
~ # systemctl status/enable/start/stop/restart/disable unit@instance.service
~ # systemctl list-unit-files/list-units/--failed/reset-failed
~ # systemctl edit unit.service (overide system unit)
~ # systemctl daemon-reload/reload
~ # systemctl enable/disable unit@sub.service

logging

~ # journalctl -xef
~ # journalctl -u nginx
~ # systemctl status unit.service

systemctl mask

~ # systemctl stop iptables-services
~ # sysremctl disable iptables-services
~ # systemctl mask iptables-services

SSH

~ # ssh-add
~ # ssh-copy-id -p 12345 root@12.23.56.78
# add an entry to ~/.ssh/config

Create User Account

Check useradd.

~ # ssh-copy-id -p 12345 username@12.34.56.78
# add an entry to ~/.ssh/config

Pip/Virtualenv

~ # yum install python-pip python-virtualenv
~ # pip install -U pip

SS

~ # mkdir -p ~/opt/pyvenv2.6
~ # virtualenv --system-site-packages ~/opt/pyvenv2.6
~ # pip install git+https://github.com/shadowsocks/shadowsocks.git@master
# or
~ # wget https://github.com/shadowsocks/shadowsocks/archive/master.zip
~ # pip install master.zip

Server Json

~ # mkdir -p /etc/shadowsocks
~ # vi /etc/shadowsocks/server.json

Fill in the fileds without any comments:

{
    "server":"::0",
    "server_port":3900,
    "local_address":"127.0.0.1",
    "local_port":1080,
    "password":"mypassword",
    "timeout":300,
    "method":"aes-256-cfb",
    "fast_open":true,
    "workers":2
}

TCP Fast Open

~ # vi /etc/sysctl.d/10-tcp-fast-open.conf
# net.ipv4.tcp_fastopen = 3
~ # sysctl -p /etc/sysctl.d/10-tcp-fast-open.conf
  1. Unfortunately CentOS 7 does not satisfy lower kernel bound - 3.7.1.
  2. It should be enabled on cleint and server simutaneously.

A quick test

~ # ssserver -c /etc/shadowsocks/server.json --user nobody -d start -vv

Systemd Unit

Please reinstall Shadowsocks into system-wide location. The above virtualenv version was just a test.

# vi /etc/systemd/system/shadowsocks.service

simple Type

[Unit]
Description=Shadowsocks Server
After=network.target

[Service]
Type=simple
PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /run/shadowsocks
ExecStartPre=/usr/bin/chown nobody:nobody /run/shadowsocks
ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody
ExecStart=/usr/bin/ssserver -c /etc/shadowsocks/server.json
Restart=on-abort
User=nobody
Group=nobody
UMask=0027

[Install]
WantedBy=multi-user.target

Please be noted that:

  1. simple Type is used so that Shadowsocks does not forking itself so that daemon mode is handed over to systemd.
    1. Do not add -d start/stop/restart arguments to ExecStart
    2. No PIDFile and --pid-file required. Read more on doesn't make sense to set PIDFile= by simple services.
  2. You may find a special line with kcptun. Neglect it now for it will be discussed later on.
  3. ref1 and ref2.

forking Type

[Unit]
Description=Shadowsocks Server
After=network.target

[Service]
Type=forking
PermissionsStartOnly=true
PIDFile=/run/shadowsocks/ss.pid
ExecStartPre=/usr/bin/mkdir -p /run/shadowsocks
ExecStartPre=/usr/bin/chown nobody:nobody /run/shadowsocks
ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody
ExecStart=/usr/bin/ssserver -d start -c /etc/shadowsocks/ss.json --pid-file /run/shadowsocks/ss.pid --log-file /run/shadowsocks/ss.log
Restart=on-abort
User=nobody
Group=nobody
UMask=0027

[Install]
WantedBy=multi-user.target

kcptun

kcptun estabilishs a KCP tunnel to speed up TCP connection by encapsulating TC packets within UDP flooding. It may introduce twice or even triple traffic depending on arguments choosen. You are advised to use it only in WI-FI environment.

To add kcptun support, just insert a line into Systemd service file:

ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody

Attention to trailing & of -c argument.