cli tools

sysctl: Linux System Tweaking

2019-10-302020-10-17VintaDevOps, Web Development

sysctl is a command-lin tool to modify kernel parameters at runtime in Linux.

ref:
http://man7.org/linux/man-pages/man8/sysctl.8.html

Usage

List All Parameters

$ sudo sysctl -a
$ sudo sysctl -a | grep tcp

The parameters available are those listed under /proc/sys/.

$ cat /proc/sys/net/core/somaxconn
1024

Show the Entry of a Specified Parameter

$ sudo sysctl net.core.somaxconn
net.core.somaxconn = 1024

### Show the Value of a Specified Parameter

```console
$ sysctl -n net.core.somaxconn
1024

Change a Specified Parameter

# Elasticsearch
$ sysctl -w vm.max_map_count = 262143

# Redis
$ sysctl -w vm.overcommit_memory = 1

ref:
https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
https://redis.io/topics/admin

Persistence

`sysctl -w` only modify parameters at runtime, and they would be set to default values after the system is restarted. You must write those settings in `/etc/sysctl.conf` to persistent them.

# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

# Prevents SYN DOS attacks. Applies to ipv6 as well, despite name.
net.ipv4.tcp_syncookies = 1

# Prevents ip spoofing.
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1

# Only groups within this id range can use ping.
net.ipv4.ping_group_range=999 59999

# Redirects can potentially be used to maliciously alter hosts routing tables.
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv6.conf.all.accept_redirects = 0

# The source routing feature includes some known vulnerabilities.
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0

# See RFC 1337
net.ipv4.tcp_rfc1337 = 1

# Enable IPv6 Privacy Extensions (see RFC4941 and RFC3041)
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.all.use_tempaddr = 2

# Restarts computer after 120 seconds after kernel panic
kernel.panic = 120

# Users should not be able to create soft or hard links to files which they do not own. This mitigates several privilege escalation vulnerabilities.
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

ref:
https://blog.runcloud.io/how-to-secure-your-linux-server/
https://www.percona.com/blog/2019/02/25/mysql-challenge-100k-connections/
https://www.nginx.com/blog/tuning-nginx/

Activate parameters from the configuration file.

$ sudo sysctl -p

Troubleshooting

OS error code 24: Too many open files

$ sudo vim /etc/sysctl.conf
fs.file-max = 601017

$ sudo sysctl -p

$ sudo vim /etc/security/limits.d/nofile.conf
* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535

$ ulimit -n 65535

OS error code 99: Cannot assign requested address

For MySQL. Because there's no available local network ports left. You might need to set `net.ipv4.tcp_tw_reuse = 1` instead of `net.ipv4.tcp_tw_recycle = 1`.

$ sudo vim /etc/sysctl.conf
net.ipv4.tcp_tw_reuse = 1

$ sudo sysctl -p

ref:
https://www.percona.com/blog/2014/12/08/what-happens-when-your-application-cannot-open-yet-another-connection-to-mysql/
https://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both

Parameters are missing from `sysctl -a` or `/proc/sys`

Sometimes you might find some parameters are not in `sysctl -a` or `/proc/sys`.

You can find them in `/sys`:

$ echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
$ echo "never" > /sys/kernel/mm/transparent_hugepage/defrag

$ cat /sys/kernel/mm/transparent_hugepage/enabled

To persistent them:

$ vim /etc/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
fi

$ systemctl enable rc-local

If /etc/rc.local doesn't exist, create one and run chmod 644 /etc/rc.local.

ref:
https://redis.io/topics/admin
https://unix.stackexchange.com/questions/99154/disable-transparent-hugepages

mitmproxy: proxy any network traffic through your local machine

2018-10-212019-10-30VintaDevOps, Python, Web Development

mitmproxy is your swiss-army knife for interactive HTTP/HTTPS proxy. In fact, it can be used to intercept, inspect, modify and replay web traffic such as HTTP/1, HTTP/2, WebSockets, or any other SSL/TLS-protected protocols.

Moreover, mitproxy has a powerful Python API offers full control over any intercepted request and response.

ref:
https://mitmproxy.org/
https://docs.mitmproxy.org/stable/

Concept

ref:
https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/

Installation

$ brew install mitmproxy

$ mitmproxy --version
Mitmproxy: 4.0.4
Python:    3.7.0
OpenSSL:   OpenSSL 1.0.2p  14 Aug 2018
Platform:  Darwin-18.0.0-x86_64-i386-64bit

ref:
https://docs.mitmproxy.org/stable/overview-installation/

Configuration

Make your computer become the man of man-in-the-middle attack.

macOS

$ ipconfig getifaddr en0
192.168.0.128

$ mitmproxy -p 8888
# or
$ mitmweb -p 8888
$ open http://127.0.0.1:8081/

Flow List keys:

?: Show help
q: Exit the current view
f: Set view filter
r: Replay this flow
i: Set intercept filter
hjkl or arrow: Move left/down/up/right
enter: Select

Flow Details keys:

tab: Select next
m: Set flow view mode
e: Edit this flow (request or response)
a: Accept this intercepted flow

ref:
https://docs.mitmproxy.org/stable/tools-mitmproxy/
https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/tools/console/defaultkeys.py

iOS

Go to Settings > Wi-Fi > Your Wi-Fi > Configure Proxy
- Select Manual, enter the following values:
  - Server: 192.168.0.128
  - Port: 8888
  - Authentication: unchecked
Open http://mitm.it/ on Safari
- Install the corresponding certificate for your device
Go to Settings > General > About > Certificate Trust Settings
- Turn on the mitmproxy certificate
Open any app you want to watch

ref:
https://docs.mitmproxy.org/stable/concepts-certificates/

Usage

The most exciting feature is you could alter any request and response using a Python script, mitmdump -s!

ref:
https://docs.mitmproxy.org/stable/tools-mitmdump/
https://github.com/mitmproxy/mitmproxy/tree/master/examples

Deal With Certificate Pinning

You can use your own certificate by passing the --certs example.com=/path/to/example.com.pem option to mitmproxy. Mitmproxy then uses the provided certificate for interception of the specified domain.

The certificate file is expected to be in the PEM format which would roughly looks like this:

$ mitmproxy -p 8888 --certs example.com=example.com.pem

ref:
https://docs.mitmproxy.org/stable/concepts-certificates/#using-a-custom-server-certificate

Redirect Requests To Your Local Development Server

# redirect_to_localhost.py
from mitmproxy import ctx
from mitmproxy import http

REMOTE_HOST = 'api.example.com'
DEV_HOST = '192.168.0.128'
DEV_PORT = 8000

def request(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_host in [REMOTE_HOST, DEV_HOST]:
        ctx.log.info('=== request')
        ctx.log.info(str(flow.request.headers))
        ctx.log.info(f'content: {str(flow.request.content)}')

        flow.request.scheme = 'http'
        flow.request.host = DEV_HOST
        flow.request.port = DEV_PORT

def response(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_host == DEV_HOST:
        ctx.log.info('=== response')
        ctx.log.info(str(flow.response.headers))
        if flow.response.headers.get('Content-Type', '').startswith('image/'):
            return
        ctx.log.info(f'body: {str(flow.response.get_content())}')

ref:
https://discourse.mitmproxy.org/t/reverse-mode-change-request-host-according-to-the-sni-https/466

You could use negative regex with --ignore-hosts to only watch specific domains. Of course, you are still able to blacklist any domain you don't want: --ignore-hosts 'apple.com|icloud.com|itunes.com|facebook.com|googleapis.com|crashlytics.com'.

Currently, changing the Host server for HTTP/2 connections is not allowed, but you could just disable HTTP/2 proxy to solve the issue if you don't need HTTP/2 for local development.

$ mitmdump -p 8888 \
--certs example.com=example.com.pem \
-v --flow-detail 3 \
--ignore-hosts '^(?!.*example\.com)' \
--no-http2 \
-s redirect_to_localhost.py

ref:
https://stackoverflow.com/questions/29414158/regex-negative-lookahead-with-wildcard

Import MySQL database but ignore specific tables

2018-03-132019-10-22VintaDatabase, DevOps

You could just exclude tables you do not need.

$ brew install gnu-sed --with-default-names
$ sed -r '/INSERT INTO `(table_to_skip_1|table_to_skip_2|table_to_skip_3)`/d' db.20180101.sql > db.reduced.sql

ref:
https://stackoverflow.com/questions/16735344/mysql-import-database-but-ignore-specific-table

ngrok: Share your localhost services with friends

2018-03-062019-10-22VintaDevOps

Generate a https://xxx.ngrok.com URL for letting other people access your localhost services.

ref:
https://github.com/inconshreveable/ngrok
https://github.com/localtunnel/localtunnel

Install

Download ngrok from https://ngrok.com/download.

$ unzip ngrok-stable-darwin-amd64.zip && \
sudo mv ngrok /usr/local/bin && \
sudo chown vinta:admin /usr/local/bin/ngrok

$ ngrok --version
ngrok version 2.3.35

Usage

Get your auth token in https://dashboard.ngrok.com/auth.

$ ngrok authtoken YOUR_TOKEN

# open a session to local port 8000
# you can also specify a custom subdomain for the tunnel
$ ngrok http 8000
$ ngrok http -subdomain=vinta-test-server -region=ap 8000
$ open https://vinta-test-server.ap.ngrok.io/

# view ngrok sessions
$ open http://localhost:4040/

ref:
https://ngrok.com/docs

Use Makefile as a task runner for arbitrary projects

2017-08-312019-10-22VintaDevOps

Use the GNU make, Luke!

Some notes:

The first target will be executed by default when we call make without any subcommand.
The order of the targets does not matter.
Add an @ to suppress output of the command that is executed.

ref:
https://www.gnu.org/software/make/manual/make.html

.PHONY

Let's assume you have install target, which is a very common in Makefiles. If you do not add .PHONY, and a file or a directory named install exists in the same directory as the Makefile, then make install will do nothing.

ref:
https://stackoverflow.com/questions/2145590/what-is-the-purpose-of-phony-in-a-makefile

Automatic Variables

ref:
https://www.gnu.org/software/make/manual/make.html#toc-How-to-Use-Variables
https://www.gnu.org/software/make/manual/make.html#Automatic-Variables

Set Environment Variables From .env

export $(grep -v '^#' .env | xargs -0)
set -o allexport; source .env; set +o allexport

include .env
export $(shell sed 's/=.*//' .env)

run_web:
    poetry run python -m flask run -h 0.0.0.0 -p 8000 --reload

ref:
https://unix.stackexchange.com/questions/235223/makefile-include-env-file
https://stackoverflow.com/questions/19331497/set-environment-variables-from-file-of-key-pair-values

Also see: https://github.com/Tarrasch/zsh-autoenv

Run Another Target in the Same Makefile

Say that coverage depends on clean.

.PHONY: clean coverage

clean:
     find . -regex "\(.*__pycache__.*\|*.py[co]\)" -delete

coverage: clean
     docker exec -i -t streetvoice_django_1 python -m coverage run manage.py test --failfast
     docker exec -i -t streetvoice_django_1 python -m coverage html -i

ref:
https://stackoverflow.com/questions/13337727/how-do-i-make-a-target-in-a-makefile-invoke-another-target-in-the-makefile

Pass Arguments to make command

.PHONY: something

something:
ifeq ($(var),foo)
    @echo $(var) "bar"
else
    @echo "others"
endif

$ make something var=foo
foo bar

$ make something
others

ref:
https://stackoverflow.com/questions/2214575/passing-arguments-to-make-run

Detect OS

.PHONY: update

update:
ifeq ($(shell uname),Darwin)
    brew update
else
    apt-get update
endif

Check Whether a File or Directory Exists

.PHONY: up install

ifneq ($(wildcard /usr/local/HAL-9000/bin/hal),)
    UP_COMMAND = /usr/local/HAL-9000/bin/hal up
else
    UP_COMMAND = docker-compose up
endif

up:
    $(UP_COMMAND)

install:
    pip install -r requirements_dev.txt

ref:
https://stackoverflow.com/questions/20763629/test-whether-a-directory-exists-inside-a-makefile

Run Targets Parallelly

You could add MAKEFLAGS += --jobs=4 in your Makefile.

MAKEFLAGS += --jobs

.PHONY: task1 task2 task3 task4

task1:
    echo "hello $@"

task2:
    echo "hello $@"

task3:
    echo "hello $@"

task4:
    echo "hello $@"

tasks: task1 task2 task3 task4

$ make tasks

Or you could call make with -j 4 explicitly.

# allow 4 jobs at once
$ make -j 4 tasks

# allow infinite jobs with no arg
$ make -j tasks

ref:
https://stackoverflow.com/questions/10567890/parallel-make-set-j8-as-the-default-option

Simple Examples

Example 1:

include .env
export $(shell sed 's/=.*//' .env)

.PHONY: clean run_web run_worker

clean:
    find . \( -name \*.pyc -o -name \*.pyo -o -name __pycache__ \) -prune -exec rm -rf {} +

run_web:
    poetry run python -m flask run -h 0.0.0.0 -p 8000 --reload

run_worker:
    poetry run watchmedo auto-restart -d . -p '*.py' -R -- celery -A app:celery worker --pid= --without-gossip --prefetch-multiplier 1 -Ofair -l debug --purge -P gevent

Example 2:

MAKEFLAGS += --jobs=4

INDICES = music_group music_album music_recording music_composition

.PHONY: prepare up stop get_indices $(INDICES)

prepare:
    mkdir -p ../Muzeum-Node-Data

up: prepare
    docker-compose up

stop:
    docker-compose stop

ipfs:
    docker-compose exec ipfs sh

$(INDICES):
    mkdir -p ../Muzeum-Node-Data/ipfs/export/soundscape/$@
    docker-compose exec ipfs ipfs get /ipns/ipfs.soundscape.net/$@/index.json -o soundscape/$@/index.json

get_indices: $(INDICES)