ZNHOO Whatever you are, be a good one!

CentOS RHEL

  1. Fedora RHEL CentOS
  2. Module
  3. rpm yum dnf
    1. rpm
    2. yum
    3. dnf
  4. Repositories
    1. EPEL not from RHEL
    2. Inline with Upstream Stable (IUS)
    3. Webtatic (DEPRECATED)
    4. CentOS-5.8
    5. Upgrade System
  5. rpmbuild
    1. Extracting RPM
  6. locale
  7. SSH
  8. Create User Account
  9. Pip/Virtualenv
  10. SS
    1. Server Json
    2. TCP Fast Open
      1. A quick test
    3. Systemd Unit
      1. simple Type
      2. forking Type
    4. kcptun

Fedora RHEL CentOS

  1. Fedora is a Linux distribution released under GPL.
  2. Red Hat, Inc. copy the source code from Fedora, make modifications, release it as Red Hat Enterprise Linux (RHEL) distribution.

    Due to GPL, Red Hat, Inc. is obligated to disclose RHEL source code. But Red Hat can make money from its professional after-sale services.

  3. CentOS (Community ENTerprise OS) is a Linux distribution built from the source of RHEL directly without any modification, with the intention of making CentOS binary compatible with RHEL such that library versions the same and binaries work on RHEL also work on CentOS.
    1. The main purpose of CentOS is to make RHEL available to pulbic users, benefitting from Red Hat's professional Linux service.
    2. Remove Red Hat, Inc. logo.
  4. Fedora - RHEL - CentOS.

Module

Module is a new packaging feature brought in from RHEL 8.

  1. Package

    Traditional .rpm file.

  2. Group and Environment

    A large set of packages as a whole for specific system categories like X window, desktop, security, development, etc.

  3. Module is a collection of a few packages, as a logical unit for a specific project, like Php and Nginx.

    1. Stream is the upstream version.
    2. Version is the build version.
    3. Profile is the sub-functionality of a module, similar to Gentoo's tag.

rpm yum dnf

The relation between rpm (Redhat Package Manager) command and yum (Yellow dog Updater, Modified) command is analogous to dpkg and apt-get. dnf is the new version of yum with performance improvement and module feature. From RHEL 8 onwards, dnf replaces yum as the default manager.

Basically, yum is more intelligent than rpm. rpm wants to know the exact location of target .rpm file while yum only needs the package name. yum also takes care of dependencies. However, rpm is somehow a lightweight tool with concise output.

Check 'SPECIFYING PACKAGE NAMES' in the man page of yum to check how to pass globs arguments. The full name of a .rpm package is as follows:

# <name>-<version>-<release>.<os>.<arch>.rpm

vim-enhanced-7.4.160-4.el7.x86_64
# name:     vim-enhanced
# version:  7.4.160
# release:  4
# os:       el7
# arch:     x86_64
# suffix:   rpm

centos-release-7-4.1708.el7.centos.x86_64
# name:     centos-release
# version:  7
# release:  4.1708
# os:       el7.centos
# arch:     x86_64
  1. Package version is also called major version, indicating the upstream source version from developers. Package release is also called minor version, indicating the final .rpm file by RHEL compliling/building/patching. Especially, RHEL may add customized patches or bug fixes to the major version, building a new .rpm package to the repository.
  2. os may be el6 (rhel6), centos6, el5, suse11 etc.

By design, it is not possible to install difference versions of the same package alongside. The workaround is to make the new version a different package. For example, python2 and python3 are actually different package names.

rpm

rpm manages package file directly, maintaining a database of package information located under /var/lib/rpm/. It does not care too much the package version but the RPM file you provided.

Query installed packages:

~ # rpm -qa 'globbing'
~ # rpm -qa | grep 'regex'

~ # rpm -qi pkg; rpm -qip pkg.rpm                     # query pkg information
~ # rpm -qR pkg; rpm -qRp pkg.rpm                     # dependencies

~ # rpm -ql pkg; rpm -qlp pkg.rpm                     # files in a package
~ # rpm -qf /path/to/file                             # package that owns the file

Verify. List discrepancies between installed files and files recorded in metadata from local RPM database.

~ # rpm -V pkg; rpm -Vp pkg.rpm

Kering:

~ # rpm --import /path/to/key.pub
~ # rpmkeys --import http://www.example.com/key.pub

Once imported, all public keys in the keyring can be managed as a package. Commands applied to a package also apply to a key like query, erase, information etc.

~ # ls /etc/pki/rpm-gpg/

~ # rpm -qa gpg-pubkey*

~ # rpm -qi gpg-pubkey-db42a60e

Signature. To verify a RPM file is not corrupted (e.g. due to network transmission) and correctly signed (by developer).

# verify MD5
# ok
rpm -Kv --nosignature kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm:
    Header SHA256 digest: OK
    Payload SHA256 digest: OK

# verify both MD5 and signature
# MD5: ok
# signature: key not found
~ $ rpm -Kv kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm:
    Header V4 RSA/SHA256 Signature, key ID 1a62ff7c: NOKEY
    Header SHA256 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA256 Signature, key ID 1a62ff7c: NOKEY

# get signing key ID from RPM file
~ $ rpm -qip kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm | grep Signature
warning: kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm: Header V4 RSA/SHA256 Signature, key ID 1a62ff7c: NOKEY
Signature   : RSA/SHA256, Sat 01 Jul 2023 04:11:14 AM UTC, Key ID 998dff461a62ff7c

~ $ rpm -qp --queryformat '%{NAME}-%{VERSION}-%{RELEASE} %{SIGPGP:pgpsig}\n' kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0-1 RSA/SHA256, Sat 01 Jul 2023 04:11:15 AM UTC, Key ID 998dff461a62ff7c

# import the signing key
~ $ sudo rpm --import https://docs.konghq.com/assets/keys/2023-rpm.asc

# list GPG key in RPM database
~ $ rpm -qa gpg-pubkey*
gpg-pubkey-be1229cf-5631588c
gpg-pubkey-fd431d51-4ae0493b
gpg-pubkey-1a62ff7c-639852e2
gpg-pubkey-d4082792-5b32db75

# examine the GPG key
~ $ rpm -qi gpg-pubkey-1a62ff7c-639852e2
Name        : gpg-pubkey
Version     : 1a62ff7c
Release     : 639852e2
Architecture: (none)
Install Date: Mon 03 Jul 2023 05:45:24 AM UTC
Group       : Public Keys
Size        : 0
License     : pubkey
Signature   : (none)
Source RPM  : (none)
Build Date  : Tue 13 Dec 2022 10:24:34 AM UTC
Build Host  : localhost
Relocations : (not relocatable)
Packager    : Bazel Testing
Summary     : gpg(Bazel Testing)
Description :
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: rpm-4.14.3 (NSS-3)
...

# verify again
~ $ rpm -Kv kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm
kong-enterprise-edition-3.3.1.0.rhel8.amd64.rpm:
    Header V4 RSA/SHA256 Signature, key ID 1a62ff7c: OK
    Header SHA256 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA256 Signature, key ID 1a62ff7c: OK

Install:

~ # rpm -ivvh --test /path/to/pkg.rpm      # dry run
~ # rpm -ivh /path/to/pkg.rpm              # install
~ # rpm --reinstall -vh /path/to/same.rom  # reinstall

~ # rpm -Uvh /path/to/pkg.rpm              # upgrade
~ # rpm -Fvh /path/to/pkg.rpm              # refresh
~ # rpm -Fvh --force /path/to/old.rpm      # downgrade
  1. The path can be an ftp/http URL.
  2. -i, --install is the general form of installing a package.
  3. -U, --upgrade is the combination of -i, --install and -e. It firstly install a package and then erase earlier versions if present.

    -U does not require an earlier version is installed, so it can be a replacement of -i.

  4. -F, --freshen is similar to -U but mandates an earlier version is already installed.
  5. --force, --replacepkgs, --replacefiles and --oldpackage do the same thing.

Uninstall/Erase:

~ # rpm -evv pkg                     # uninstall a pkg

yum

The main configuration of YUM is /etc/yum.conf and specific repository configuration files are placed in /etc/yum.repos.d/. Within the main configuration file, cachedir defines the location of cached .rpm packages (defaults to /var/cache/yum/$basearch/$releasever).

By default, of the built-in repositories, only 'CentOS-Base.repo' is enabled.

  1. $releasever indicates the main version. For example, the main version of CentOS 5.8 is 5.
  2. $arch indicates the architecture like x8_64, i386/i586/i686.
  3. $basearch is similar to $arch, but is base architecture: either x86_64 or i386.

Cache:

~ $ yum help [sub-command]

~ # yum clean [ all | packages | metadata | expire-cache | rpmdb | plugins ]
~ # yum makecache

After any edits of repo files, in order to clear any cached information, and make sure the changes are immediately recoginized, as root run yum clean all.

Query:

~ # yum history

~ # yum list [all | installed | available | updates ] [pkg]
~ # yum info [all | installed | available | updates ] [pkg] # more details

~ # yum search pkg
~ # yum --showduplicates search pkg    # show all versions

~ # yum repolist [--all | --enabled | --disabled]
~ # yum check-update
~ # yum upgrade [pkg]

~ # yum provides /path/to/file             # rpm -qf
~ # repoquery -f /path/to/file
~ # repoqeury --provides /path/to/file
~ # repoqeury --whatprovides /path/to/file

~ # repoquery -l pkg

~ # repoquery --deplist pkg
~ # repoquery --whatdepends pkg
  1. update is synonym to upgrade.
  2. list –updates is almost the same as check-update. As the command form implies, check-update is useful in Shell script while list –updates is for humans on the command line.

    Please pay attention, their exit status code difference.

  3. Make sure yum-utils is installed to use repoquery.

Install:

~ # yum [-y] install pkg1 pkg2

# will install dependency automatically
~ # yum install /path/to/pkg.rpm

~ # yum reinstall pkg1 pkg2

~ # yum upgrade pkg1 pkg2

~ # yum remove pkg1 pkg2

To install packages for group or environment set:

~ # yum group summary
~ # yum group [ list | info |install | upgrade | remove | mark ] [grp]

dnf

dnf is compatible with but more powerful than yum. One of the newest feature provided is module.

~ # dnf module -h
~ # dnf module list nginx
~ # dnf module info nginx

# NAME:STREAM:VERSION:CONTEXT:ARCH/PROFILE
~ # dnf module info nginx:1.16:8010020191122190044:cdc1202b:x86_64/common
~ # dnf module install nginx:1.14/common

The purpose of module is to manage package versions and profiles in an integrated way. The following two methods is equivalent:

~ # dnf module install nginx
~ # dnf install nginx

The only difference is that dnf module can provide better control when selecting pakage stream, version, arch etc.

The following example shows how to control repo with dnf (applying to yum as well):

~ $ dnf help repolist 
~ $ dnf repolist [--all | --enabled | --disabled]

~ $ sudo dnf config-manager --add-repo /etc/yum.repos.d/oracle-epel-ol8.repo

~ $ grep enabled /etc/yum.repos.d/oracle-epel-ol8.repo
~ $ sudo dnf config-manager --set-enabled ol8_developer_EPEL
~ $ duso dnf config-manager --set-disabled ol8_developer_EPEL

~ $ dnf --enablerepo=repo1,repo2 -disablerepo=repo3,repo4 install pkg

Repositories

Of the 3rd party repositories, IUS and Remi (both depend on EPEL) is recommended over Webtatic.

~ # yum repolist [all|enabled|disabled]

~ # yum-config-manager --add-repo <repo-url>
~ # yum update -y

~ # yum repolist [all|enabled|disabled]
~ # yum-config-manager --disable <repo-id>

~ # yum-config-manager --enable <repo-id>
~ # yum update -y

To remove a repo, just remove the file within /etc/yum.repos.d.

EPEL not from RHEL

EPEL (Extra Packages for Enterprise Linux) is open source and free community based repository project from Fedora team which provides 100% high quality add-on software packages.

~ # yum info epel-release
~ # yum install epel-release

To specify repository when installing package:

~ # yum --enablerepo=epel install nginx

The --enablerepo option overides the permanent option setting in the /etc/yum/*.repo files for only the current command. --disablerepo does the opposite for enabled repos.

Inline with Upstream Stable (IUS)

~ # rpm -Uvh https://centos7.iuscommunity.org/ius-release.rpm

Read IUS Usage for more information.

Webtatic (DEPRECATED)

The Webtatic Yum repository is a CentOS/RHEL repository containing updated web-related packages like git.

~ # rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm

CentOS-5.8

CentOS-5 reached the end of its lifecycle as of March 31, 2017. The official baseurl is deprecated and moved to a backup location Vault. Packages there is no longer maintained or upgraded, and may suffer from security risks.

However, we sometimes have to maintain ancient releases for whatever reasons. To install packages through yum, point repositories to the new location: either manually edit /etc/yum.repos.d/CentOS-Base.repo or install centos-release-5-8.el5.centos.x86_64.rpm file.

  1. For each section (i.e. Base) Comment out both the mirrorlist= and baseurl= lines.
  2. Add a new baseurl line:

    baseurl=http://vault.centos.org/5.8/os/$basearch/
    
    1. Replace /os/ part approriately for other sections like /updates/.
    2. It is highly recommended to replace 5.8 with 5.11, upgrading the system to CentOS 5.11

    Alternatively, install the corresponding repository package:

    root@tux ~ # wget http://vault.centos.org/5.11/os/x86_64/CentOS/centos-release-5-11.el5.centos.x86_64.rpm
    root@tux ~ # rpm -Uvh centos-release-5-11.el5.centos.x86_64.rpm
    

    For other repositories like EPEL, Webtatic etc., find their archive URL and repeat the same procedures.

  3. Update

    root@tux ~ # rpm -q centos-release; cat /etc/redhat-release; cat /etc/centos-release
    root@tux ~ # yum clean all; yum makecache
    root@tux ~ # yum list updates; yum update
    root@tux ~ # reboot
    root@tux ~ # yum install epel-release; yum install git
    

Here is an example of manually edited Centos-Base.repo:

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=http://vault.centos.org/5.11/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#released updates 
[updates]
name=CentOS-$releasever - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
baseurl=http://vault.centos.org/5.11/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
baseurl=http://vault.centos.org/5.11/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
baseurl=http://vault.centos.org/5.11/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
baseurl=http://vault.centos.org/5.11/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

Upgrade System

~ # dnf check-update
~ # dnf upgrade kernel
~ # dnf upgrade
~ # reboot

rpmbuild

For how to build RPM package, refer to "logging/filebeat-zookeeper-kafka".

Refer to How to create a Linux RPM package.

Extracting RPM

Upon receving a .rpm file, we can extract the contents with rpm2cpio and cipo:

~ $ mkdir dst; cd dst
~ $ rpm2cpio /path/to/pkg.rpm | cpio -idmv
~ $ find .
  1. -i: extract;
  2. -d: make directories;
  3. -m: preserve modification time;
  4. -v: verbose.

locale

Redhad-based distributions places locale customization in /etc/locale.conf. Check man locale.conf.

SSH

~ # ssh-add
~ # ssh-copy-id -p 12345 root@12.23.56.78
# add an entry to ~/.ssh/config

Create User Account

Check useradd.

~ # ssh-copy-id -p 12345 username@12.34.56.78
# add an entry to ~/.ssh/config

Pip/Virtualenv

~ # yum install python-pip python-virtualenv
~ # pip install -U pip

SS

~ # mkdir -p ~/opt/pyvenv2.6
~ # virtualenv --system-site-packages ~/opt/pyvenv2.6
~ # pip install git+https://github.com/shadowsocks/shadowsocks.git@master
# or
~ # wget https://github.com/shadowsocks/shadowsocks/archive/master.zip
~ # pip install master.zip

Server Json

~ # mkdir -p /etc/shadowsocks
~ # vi /etc/shadowsocks/server.json

Fill in the fileds without any comments:

{
    "server":"::0",
    "server_port":3900,
    "local_address":"127.0.0.1",
    "local_port":1080,
    "password":"mypassword",
    "timeout":300,
    "method":"aes-256-cfb",
    "fast_open":true,
    "workers":2
}

TCP Fast Open

~ # vi /etc/sysctl.d/10-tcp-fast-open.conf
# net.ipv4.tcp_fastopen = 3
~ # sysctl -p /etc/sysctl.d/10-tcp-fast-open.conf
  1. Unfortunately CentOS 7 does not satisfy lower kernel bound - 3.7.1.
  2. It should be enabled on cleint and server simutaneously.

A quick test

~ # ssserver -c /etc/shadowsocks/server.json --user nobody -d start -vv

Systemd Unit

Please reinstall Shadowsocks into system-wide location. The above virtualenv version was just a test.

# vi /etc/systemd/system/shadowsocks.service

simple Type

[Unit]
Description=Shadowsocks Server
After=network.target

[Service]
Type=simple
PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /run/shadowsocks
ExecStartPre=/usr/bin/chown nobody:nobody /run/shadowsocks
ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody
ExecStart=/usr/bin/ssserver -c /etc/shadowsocks/server.json
Restart=on-abort
User=nobody
Group=nobody
UMask=0027

[Install]
WantedBy=multi-user.target

Please be noted that:

  1. simple Type is used so that Shadowsocks does not forking itself so that daemon mode is handed over to systemd.
    1. Do not add -d start/stop/restart arguments to ExecStart
    2. No PIDFile and --pid-file required. Read more on doesn't make sense to set PIDFile= by simple services.
  2. You may find a special line with kcptun. Neglect it now for it will be discussed later on.
  3. ref1 and ref2.

forking Type

[Unit]
Description=Shadowsocks Server
After=network.target

[Service]
Type=forking
PermissionsStartOnly=true
PIDFile=/run/shadowsocks/ss.pid
ExecStartPre=/usr/bin/mkdir -p /run/shadowsocks
ExecStartPre=/usr/bin/chown nobody:nobody /run/shadowsocks
ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody
ExecStart=/usr/bin/ssserver -d start -c /etc/shadowsocks/ss.json --pid-file /run/shadowsocks/ss.pid --log-file /run/shadowsocks/ss.log
Restart=on-abort
User=nobody
Group=nobody
UMask=0027

[Install]
WantedBy=multi-user.target

kcptun

kcptun estabilishs a KCP tunnel to speed up TCP connection by encapsulating TC packets within UDP flooding. It may introduce twice or even triple traffic depending on arguments choosen. You are advised to use it only in WI-FI environment.

To add kcptun support, just insert a line into Systemd service file:

ExecStartPre=/usr/bin/su -s /bin/bash -c "/usr/local/bin/kcptun-server -c /etc/shadowsocks/kcptun.json &" nobody

Attention to trailing & of -c argument.