cli tools

1Password CLI: How NOT to Store Plaintext AWS Credentials or .env on Localhost

2025-11-112025-11-21VintaDevOps

No More `~/.aws/credetials`

According to AWS security best practices, human users should access AWS services using short-term credentials provided by IAM Identity Center. Long-term credentials ("Access Key ID" and "Secret Access Key") created by IAM users should be avoided, especially since they are often stored in plaintext on disk: ~/.aws/credetials.

However, if you somehow have to use AWS access keys but want an extra layer of protection, 1Password CLI can help.

ref:
https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html
https://developer.1password.com/docs/cli/get-started

First, delete your local plaintext AWS credentials. Don't worry, you could generate new one any time on AWS Management Console.

rm -rf ~/.aws/credetials

Re-create aws-cli configuration file, but DO NOT provide any credentials.

aws configure

AWS Access Key ID [None]: JUST PRESS ENTER, DO NOT TYPE ANYTHING
AWS Secret Access Key [None]: JUST PRESS ENTER, DO NOT TYPE ANYTHING
Default region name [None]: ap-northeast-1
Default output format [None]: json

Edit ~/.aws/credentials:

[your-profile-name]
credential_process = sh -c "op item get \"AWS Access Key\" --account=my.1password.com --vault=Private --format=json --fields label=AccessKeyId,label=SecretAccessKey | jq 'map({key: .label, value: .value}) | from_entries + {Version: 1}'"

The magic is credential_process which sourcing AWS credentials from an external process: 1Password CLI's op item get command.

The one-liner script assumes you have an item named AWS Access Key in a vault named Private in 1Password, and the item has following fields:

AccessKeyId
SecretAccessKey

ref:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html
https://developer.1password.com/docs/cli/reference/management-commands/item#item-get

That's it.

When you run aws-cli commands or access AWS services from your code via aws-sdk, your terminal will prompt you to unlock 1Password with biometrics to source AWS credentials (once per terminal session). No more plaintext AWS access keys on localhost!

# aws-cli
aws s3 ls --profile=perp
aws logs tail --profile=perp --region=ap-northeast-1 /aws/containerinsights/perp-staging/application --follow

# aws-sdk
AWS_PROFILE=perp OTHER_ENV=123 ts-node src/index.ts

# serverless v4 supports credential_process by default
# serverless v3 requires installing a plugin: serverless-better-credentials
# https://github.com/thomasmichaelwallace/serverless-better-credentials
sls deploy --stage=staging --aws-profile=perp

# if you're using serverless-offline, you might need to add the following configs to serverless.yml
custom:
  serverless-offline:
    useInProcess: true

It's worth noting that if you prefer not to use 1Password, there is also a tool called aws-vault which can achieve a similar goal.

ref:
https://github.com/99designs/aws-vault

No More `.env`

If you would like to store .env file entirely in 1Password, try 1Password Environments.

ref:
https://developer.1password.com/docs/environments
https://developer.1password.com/docs/environments/local-env-file

sysctl: Linux System Tweaking

2019-10-302020-10-17VintaDevOps, Web Development

sysctl is a command-lin tool to modify kernel parameters at runtime in Linux.

ref:
http://man7.org/linux/man-pages/man8/sysctl.8.html

Usage

List All Parameters

$ sudo sysctl -a
$ sudo sysctl -a | grep tcp

The parameters available are those listed under /proc/sys/.

$ cat /proc/sys/net/core/somaxconn
1024

Show the Entry of a Specified Parameter

$ sudo sysctl net.core.somaxconn
net.core.somaxconn = 1024

### Show the Value of a Specified Parameter

```console
$ sysctl -n net.core.somaxconn
1024

Change a Specified Parameter

# Elasticsearch
$ sysctl -w vm.max_map_count = 262143

# Redis
$ sysctl -w vm.overcommit_memory = 1

ref:
https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
https://redis.io/topics/admin

Persistence

`sysctl -w` only modify parameters at runtime, and they would be set to default values after the system is restarted. You must write those settings in `/etc/sysctl.conf` to persistent them.

# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

# Prevents SYN DOS attacks. Applies to ipv6 as well, despite name.
net.ipv4.tcp_syncookies = 1

# Prevents ip spoofing.
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1

# Only groups within this id range can use ping.
net.ipv4.ping_group_range=999 59999

# Redirects can potentially be used to maliciously alter hosts routing tables.
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv6.conf.all.accept_redirects = 0

# The source routing feature includes some known vulnerabilities.
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0

# See RFC 1337
net.ipv4.tcp_rfc1337 = 1

# Enable IPv6 Privacy Extensions (see RFC4941 and RFC3041)
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.all.use_tempaddr = 2

# Restarts computer after 120 seconds after kernel panic
kernel.panic = 120

# Users should not be able to create soft or hard links to files which they do not own. This mitigates several privilege escalation vulnerabilities.
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

ref:
https://blog.runcloud.io/how-to-secure-your-linux-server/
https://www.percona.com/blog/2019/02/25/mysql-challenge-100k-connections/
https://www.nginx.com/blog/tuning-nginx/

Activate parameters from the configuration file.

$ sudo sysctl -p

Troubleshooting

OS error code 24: Too many open files

$ sudo vim /etc/sysctl.conf
fs.file-max = 601017

$ sudo sysctl -p

$ sudo vim /etc/security/limits.d/nofile.conf
* soft nofile 65535
* hard nofile 65535
root soft nofile 65535
root hard nofile 65535

$ ulimit -n 65535

OS error code 99: Cannot assign requested address

For MySQL. Because there's no available local network ports left. You might need to set `net.ipv4.tcp_tw_reuse = 1` instead of `net.ipv4.tcp_tw_recycle = 1`.

$ sudo vim /etc/sysctl.conf
net.ipv4.tcp_tw_reuse = 1

$ sudo sysctl -p

ref:
https://www.percona.com/blog/2014/12/08/what-happens-when-your-application-cannot-open-yet-another-connection-to-mysql/
https://stackoverflow.com/questions/6426253/tcp-tw-reuse-vs-tcp-tw-recycle-which-to-use-or-both

Parameters are missing from `sysctl -a` or `/proc/sys`

Sometimes you might find some parameters are not in `sysctl -a` or `/proc/sys`.

You can find them in `/sys`:

$ echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
$ echo "never" > /sys/kernel/mm/transparent_hugepage/defrag

$ cat /sys/kernel/mm/transparent_hugepage/enabled

To persistent them:

$ vim /etc/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
fi

$ systemctl enable rc-local

If /etc/rc.local doesn't exist, create one and run chmod 644 /etc/rc.local.

ref:
https://redis.io/topics/admin
https://unix.stackexchange.com/questions/99154/disable-transparent-hugepages

mitmproxy: proxy any network traffic through your local machine

2018-10-212019-10-30VintaDevOps, Python, Web Development

mitmproxy is your swiss-army knife for interactive HTTP/HTTPS proxy. In fact, it can be used to intercept, inspect, modify and replay web traffic such as HTTP/1, HTTP/2, WebSockets, or any other SSL/TLS-protected protocols.

Moreover, mitproxy has a powerful Python API offers full control over any intercepted request and response.

ref:
https://mitmproxy.org/
https://docs.mitmproxy.org/stable/

Concept

ref:
https://docs.mitmproxy.org/stable/concepts-howmitmproxyworks/

Installation

$ brew install mitmproxy

$ mitmproxy --version
Mitmproxy: 4.0.4
Python:    3.7.0
OpenSSL:   OpenSSL 1.0.2p  14 Aug 2018
Platform:  Darwin-18.0.0-x86_64-i386-64bit

ref:
https://docs.mitmproxy.org/stable/overview-installation/

Configuration

Make your computer become the man of man-in-the-middle attack.

macOS

$ ipconfig getifaddr en0
192.168.0.128

$ mitmproxy -p 8888
# or
$ mitmweb -p 8888
$ open http://127.0.0.1:8081/

Flow List keys:

?: Show help
q: Exit the current view
f: Set view filter
r: Replay this flow
i: Set intercept filter
hjkl or arrow: Move left/down/up/right
enter: Select

Flow Details keys:

tab: Select next
m: Set flow view mode
e: Edit this flow (request or response)
a: Accept this intercepted flow

ref:
https://docs.mitmproxy.org/stable/tools-mitmproxy/
https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/tools/console/defaultkeys.py

iOS

Go to Settings > Wi-Fi > Your Wi-Fi > Configure Proxy
- Select Manual, enter the following values:
  - Server: 192.168.0.128
  - Port: 8888
  - Authentication: unchecked
Open http://mitm.it/ on Safari
- Install the corresponding certificate for your device
Go to Settings > General > About > Certificate Trust Settings
- Turn on the mitmproxy certificate
Open any app you want to watch

ref:
https://docs.mitmproxy.org/stable/concepts-certificates/

Usage

The most exciting feature is you could alter any request and response using a Python script, mitmdump -s!

ref:
https://docs.mitmproxy.org/stable/tools-mitmdump/
https://github.com/mitmproxy/mitmproxy/tree/master/examples

Deal With Certificate Pinning

You can use your own certificate by passing the --certs example.com=/path/to/example.com.pem option to mitmproxy. Mitmproxy then uses the provided certificate for interception of the specified domain.

The certificate file is expected to be in the PEM format which would roughly looks like this:

$ mitmproxy -p 8888 --certs example.com=example.com.pem

ref:
https://docs.mitmproxy.org/stable/concepts-certificates/#using-a-custom-server-certificate

Redirect Requests To Your Local Development Server

# redirect_to_localhost.py
from mitmproxy import ctx
from mitmproxy import http

REMOTE_HOST = 'api.example.com'
DEV_HOST = '192.168.0.128'
DEV_PORT = 8000

def request(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_host in [REMOTE_HOST, DEV_HOST]:
        ctx.log.info('=== request')
        ctx.log.info(str(flow.request.headers))
        ctx.log.info(f'content: {str(flow.request.content)}')

        flow.request.scheme = 'http'
        flow.request.host = DEV_HOST
        flow.request.port = DEV_PORT

def response(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_host == DEV_HOST:
        ctx.log.info('=== response')
        ctx.log.info(str(flow.response.headers))
        if flow.response.headers.get('Content-Type', '').startswith('image/'):
            return
        ctx.log.info(f'body: {str(flow.response.get_content())}')

ref:
https://discourse.mitmproxy.org/t/reverse-mode-change-request-host-according-to-the-sni-https/466

You could use negative regex with --ignore-hosts to only watch specific domains. Of course, you are still able to blacklist any domain you don't want: --ignore-hosts 'apple.com|icloud.com|itunes.com|facebook.com|googleapis.com|crashlytics.com'.

Currently, changing the Host server for HTTP/2 connections is not allowed, but you could just disable HTTP/2 proxy to solve the issue if you don't need HTTP/2 for local development.

$ mitmdump -p 8888 \
--certs example.com=example.com.pem \
-v --flow-detail 3 \
--ignore-hosts '^(?!.*example\.com)' \
--no-http2 \
-s redirect_to_localhost.py

ref:
https://stackoverflow.com/questions/29414158/regex-negative-lookahead-with-wildcard

Import MySQL database but ignore specific tables

2018-03-132019-10-22VintaDatabase, DevOps

You could just exclude tables you do not need.

$ brew install gnu-sed --with-default-names
$ sed -r '/INSERT INTO `(table_to_skip_1|table_to_skip_2|table_to_skip_3)`/d' db.20180101.sql > db.reduced.sql

ref:
https://stackoverflow.com/questions/16735344/mysql-import-database-but-ignore-specific-table

ngrok: Share your localhost services with friends

2018-03-062019-10-22VintaDevOps

Generate a https://xxx.ngrok.com URL for letting other people access your localhost services.

ref:
https://github.com/inconshreveable/ngrok
https://github.com/localtunnel/localtunnel

Install

Download ngrok from https://ngrok.com/download.

$ unzip ngrok-stable-darwin-amd64.zip && \
sudo mv ngrok /usr/local/bin && \
sudo chown vinta:admin /usr/local/bin/ngrok

$ ngrok --version
ngrok version 2.3.35

Usage

Get your auth token in https://dashboard.ngrok.com/auth.

$ ngrok authtoken YOUR_TOKEN

# open a session to local port 8000
# you can also specify a custom subdomain for the tunnel
$ ngrok http 8000
$ ngrok http -subdomain=vinta-test-server -region=ap 8000
$ open https://vinta-test-server.ap.ngrok.io/

# view ngrok sessions
$ open http://localhost:4040/

ref:
https://ngrok.com/docs

No More ~/.aws/credetials

No More .env

Usage

List All Parameters

Show the Entry of a Specified Parameter

Change a Specified Parameter

Persistence

Troubleshooting

OS error code 24: Too many open files

OS error code 99: Cannot assign requested address

Parameters are missing from `sysctl -a` or `/proc/sys`

Concept

Installation

Configuration

macOS

iOS

Usage

Deal With Certificate Pinning

Redirect Requests To Your Local Development Server

Install

Usage

No More `~/.aws/credetials`

No More `.env`