I Failed the Turing Test

Use Makefile as a task runner for arbitrary projects

Posted on Author VintaPosted in DevOps

Use the GNU make, Luke!

Some notes:

  • The first target will be executed by default when we call make without any subcommand.
  • The order of the targets does not matter.
  • Add an @ to suppress output of the command that is executed.

ref:
https://www.gnu.org/software/make/manual/make.html

.PHONY

Let's assume you have install target, which is a very common in Makefiles. If you do not add .PHONY, and a file or a directory named install exists in the same directory as the Makefile, then make install will do nothing.

ref:
https://stackoverflow.com/questions/2145590/what-is-the-purpose-of-phony-in-a-makefile

Automatic Variables

ref:
https://www.gnu.org/software/make/manual/make.html#toc-How-to-Use-Variables
https://www.gnu.org/software/make/manual/make.html#Automatic-Variables

Set Environment Variables From .env

  • export $(grep -v '^#' .env | xargs -0)
  • set -o allexport; source .env; set +o allexport
include .env
export $(shell sed 's/=.*//' .env)

run_web:
    poetry run python -m flask run -h 0.0.0.0 -p 8000 --reload

ref:
https://unix.stackexchange.com/questions/235223/makefile-include-env-file
https://stackoverflow.com/questions/19331497/set-environment-variables-from-file-of-key-pair-values

Also see: https://github.com/Tarrasch/zsh-autoenv

Run Another Target in the Same Makefile

Say that coverage depends on clean.

.PHONY: clean coverage

clean:
     find . -regex "\(.*__pycache__.*\|*.py[co]\)" -delete

coverage: clean
     docker exec -i -t streetvoice_django_1 python -m coverage run manage.py test --failfast
     docker exec -i -t streetvoice_django_1 python -m coverage html -i

ref:
https://stackoverflow.com/questions/13337727/how-do-i-make-a-target-in-a-makefile-invoke-another-target-in-the-makefile

Pass Arguments to make command

.PHONY: something

something:
ifeq ($(var),foo)
    @echo $(var) "bar"
else
    @echo "others"
endif
$ make something var=foo
foo bar

$ make something
others

ref:
https://stackoverflow.com/questions/2214575/passing-arguments-to-make-run

Detect OS

.PHONY: update

update:
ifeq ($(shell uname),Darwin)
    brew update
else
    apt-get update
endif

Check Whether a File or Directory Exists

.PHONY: up install

ifneq ($(wildcard /usr/local/HAL-9000/bin/hal),)
    UP_COMMAND = /usr/local/HAL-9000/bin/hal up
else
    UP_COMMAND = docker-compose up
endif

up:
    $(UP_COMMAND)

install:
    pip install -r requirements_dev.txt

ref:
https://stackoverflow.com/questions/20763629/test-whether-a-directory-exists-inside-a-makefile

Run Targets Parallelly

You could add MAKEFLAGS += --jobs=4 in your Makefile.

MAKEFLAGS += --jobs

.PHONY: task1 task2 task3 task4

task1:
    echo "hello $@"

task2:
    echo "hello $@"

task3:
    echo "hello $@"

task4:
    echo "hello $@"

tasks: task1 task2 task3 task4
$ make tasks

Or you could call make with -j 4 explicitly.

# allow 4 jobs at once
$ make -j 4 tasks

# allow infinite jobs with no arg
$ make -j tasks

ref:
https://stackoverflow.com/questions/10567890/parallel-make-set-j8-as-the-default-option

Simple Examples

Example 1:

include .env
export $(shell sed 's/=.*//' .env)

.PHONY: clean run_web run_worker

clean:
    find . \( -name \*.pyc -o -name \*.pyo -o -name __pycache__ \) -prune -exec rm -rf {} +

run_web:
    poetry run python -m flask run -h 0.0.0.0 -p 8000 --reload

run_worker:
    poetry run watchmedo auto-restart -d . -p '*.py' -R -- celery -A app:celery worker --pid= --without-gossip --prefetch-multiplier 1 -Ofair -l debug --purge -P gevent

Example 2:

MAKEFLAGS += --jobs=4

INDICES = music_group music_album music_recording music_composition

.PHONY: prepare up stop get_indices $(INDICES)

prepare:
    mkdir -p ../Muzeum-Node-Data

up: prepare
    docker-compose up

stop:
    docker-compose stop

ipfs:
    docker-compose exec ipfs sh

$(INDICES):
    mkdir -p ../Muzeum-Node-Data/ipfs/export/soundscape/$@
    docker-compose exec ipfs ipfs get /ipns/ipfs.soundscape.net/$@/index.json -o soundscape/$@/index.json

get_indices: $(INDICES)