I Failed the Turing Test

All posts tagged ci

Speed up Python and Node.js builds on Travis CI

Speed up Python and Node.js builds on Travis CI

Posted on Author VintaPosted in DevOps, JavaScript, Python

Travis CI's caching archives all directories listed in the configuration and uploads them to Amazon S3. Cached contents are available to any build on the repository, including Pull Requests. For Python and Node.js projects, you could cache both site-packages and node_modules directories in every Travis CI build.

Here is an example of .travis.yml:

sudo: false

language: python

python:
  - "2.7"

node_js: 4

cache:
  directories:
    - $HOME/.cache/pip
    - $HOME/virtualenv/python2.7.9/lib/python2.7/site-packages
    - node_modules

before_install:
  - pip install -U pip

install:
  - pip install -r requirements.txt
  - pip install coverage --ignore-installed
  - npm install

script:
  - coverage run manage.py test

In the case of mine, after applying these changes, the installation time of pip and npm reduces from 180 seconds to 5 seconds.

One thing should be mentioned here: Since we didn't specify any bin folder in the configuration (and I don't think that's necessary), any execution file that being installed by pip such as coverage or django-admin.py will not exist in subsequent builds. If you need those commands, you could just force install them by adding pip install some_package --ignore-installed.

ref:
https://docs.travis-ci.com/user/caching/
https://stackoverflow.com/questions/19422229/how-to-cache-requirements-for-a-django-project-on-travis-ci
https://tzangms.com/how-to-speed-up-python-unit-test-on-travis-ci/

Upload files to Amazon S3 when Travis CI builds pass

Upload files to Amazon S3 when Travis CI builds pass

Posted on Author VintaPosted in DevOps, Python, Web Development

Assume that you want to upload a xxx.whl file generated by pip wheel to Amazon S3 so that you will be able to run pip install https://url/to/s3/bucket/xxx.whl.

CAUTION! By default, only master branch's builds could trigger deployments in Travis CI.

Configuration

before_install:
  - pip install -U pip
  - pip install wheel

script:
  - python setup.py test

before_deploy:
  - pip wheel --wheel-dir=wheelhouse .

deploy:
  provider: s3
  access_key_id: "YOUR_KEY"
  secret_access_key: "YOUR_SECRET"
  bucket: YOUR_BUCKET
  acl: public_read
  local_dir: wheelhouse
  upload_dir: wheels
  skip_cleanup: true
# install from an URL directly
$ pip install https://url/to/s3/bucket/wheels/xxx.whl

ref:
https://docs.travis-ci.com/user/deployment/s3

MkDocs: Deploy your Markdown documents on GitHub Pages

MkDocs: Deploy your Markdown documents on GitHub Pages

Posted on Author VintaPosted in Misc

MkDocs is a static site generator that builds modern webpages based on your Markdown documents and a simple YAML file.

ref:
https://www.mkdocs.org/

Here is the website which is generated by MkDocs in this post:
https://awesome-python.com/
https://github.com/vinta/awesome-python

Installation

$ pip install mkdocs

Configuration

in mkdocs.yml

site_name: Awesome Python
site_url: https://awesome-python.com
site_description: A curated list of awesome Python frameworks, libraries and software
site_author: Vinta Chen
repo_name: vinta/awesome-python
repo_url: https://github.com/vinta/awesome-python
theme:
  name: material
  palette:
    primary: red
    accent: pink
extra:
  social:
    - type: github
      link: https://github.com/vinta
    - type: twitter
      link: https://twitter.com/vinta
    - type: linkedin
      link: https://www.linkedin.com/in/vinta
google_analytics:
  - UA-510626-7
  - auto
extra_css:
    - css/extra.css
nav:
  - "Life is short, you need Python.": "index.md"

There are more themes:

in Makefile

site_install:
    pip install -r requirements.txt

site_link:
    ln -sf $(CURDIR)/README.md $(CURDIR)/docs/index.md

site_preview: site_link
    mkdocs serve

site_build: site_link
    mkdocs build

site_deploy: site_link
    mkdocs gh-deploy --clean

Custom Domain for GitHub Pages

in docs/CNAME

awesome-python.com

After deploying your GitHub Page, just pointing your domain to following IPs with DNS A records:

  • 185.199.108.153
  • 185.199.109.153
  • 185.199.110.153
  • 185.199.111.153

ref:
https://help.github.com/articles/setting-up-an-apex-domain/#configuring-a-records-with-your-dns-provider
https://help.github.com/articles/troubleshooting-custom-domains/#https-errors

Automatic Deployment Via Travis CI

You need to 

language: python

python:
  - "3.6"

script:
  - cp README.md docs/index.md
  - mkdocs build

deploy:
  provider: pages
  local-dir: site
  skip-cleanup: true
  keep-history: true
  github-token: $GITHUB_TOKEN
  on:
    branch: master

ref:
https://docs.travis-ci.com/user/deployment/pages/

Travis-CI for Python: a .travis.yml example

Travis-CI for Python: a .travis.yml example

Posted on Author VintaPosted in Python, Web Development

Configuration

in .travis.yml

language: python

# 多個 env 要寫在同一行
env:
  - DPS_ENV=test OTHER_ENV=whatever
  - DPS_ENV=development OTHER_ENV=whatever

python:
  - "2.7"
  - "3.3"

before_install:
  - sudo apt-get update -qq
  - sudo apt-get install -qq libxml2-dev libxslt1-dev

install:
  - pip install -r requirements_test.txt --use-mirrors

script:
  - coverage run --source=haul setup.py test

notifications:
  email:
    - [email protected]

after_success:
  - coveralls

要注意的是多個 environment variables 要寫在同一行
否則會被當成多個環境
每個環境都會跑一次 CI

ref:
http://docs.travis-ci.com/user/ci-environment/
http://docs.travis-ci.com/user/build-configuration/
http://docs.travis-ci.com/user/languages/python/
http://docs.travis-ci.com/user/database-setup/

in .coveralls.yml

如果是 public repo 的話，不需要特別設定什麼

ref:
https://coveralls.io/docs/python
https://github.com/coagulant/coveralls-python