github

Build Docker Images using Google Cloud Build from Your GitHub Repository

2023-08-26VintaDevOps

Triggering a Docker image build on Google Cloud Build when pushing commits to a GitHub repository.

Create Google Cloud Build Configuration File

Create a cloudbuild.yaml in the root folder:

steps:
- id: my-blog-cache-image
  name: gcr.io/cloud-builders/docker
  entrypoint: "/bin/bash"
  args:
   - "-c"
   - |
     docker pull gcr.io/$PROJECT_ID/my-blog:$BRANCH_NAME || exit 0
  waitFor: ["-"]
- id: my-blog-build-image
  name: gcr.io/cloud-builders/docker
  args: [
    "build",
    "-t", "gcr.io/$PROJECT_ID/my-blog:$BRANCH_NAME",
    "-t", "gcr.io/$PROJECT_ID/my-blog:$SHORT_SHA",
    "docker/my-blog/",
  ]
  waitFor: ["my-blog-cache-image"]
images:
- gcr.io/$PROJECT_ID/my-blog:$SHORT_SHA

If you want to push images to another region, simply rename gcr.io to something like asia.gcr.io.

ref:
https://cloud.google.com/build/docs/configuring-builds/create-basic-configuration
https://cloud.google.com/cloud-build/docs/configuring-builds/store-images-artifacts

Configure Google Cloud Build with Your GitHub Repository

Go to Google Cloud Dashboard > Google Cloud Build > Triggers > Create trigger

Region: Global
Event: Push to a branch
Source: 1st Gen
Repository:
- Connect new repository
- Source code management provider: GitHub (Cloud Build GitHub App)
- Install Google Cloud Build GitHub app
- Only select repositories
Branch: ^main$
Type: Autodetected
Location: Repository

That's it, now you could push some commits to your GitHub repository.

ref:
https://console.cloud.google.com/cloud-build/triggers
https://console.cloud.google.com/cloud-build/builds

Play with GitHub Archive Dataset on BigQuery

2017-09-012019-10-29VintaBig Data

Google BigQuery is a web service that lets you do interactive analysis of very massive datasets - analyzing billions of rows in seconds.

ref:
https://www.githubarchive.org/#bigquery
https://bigquery.cloud.google.com/table/githubarchive:month.201612

Show repository informations (1)

WITH repo_info AS (
  SELECT repo.id AS id, repo.name AS name, JSON_EXTRACT_SCALAR(payload, '$.pull_request.base.repo.description') AS description
  FROM `githubarchive.month.2017*`
  -- FROM `githubarchive.year.2016`
  -- FROM `githubarchive.year.*`
  WHERE type = "PullRequestEvent"
)

SELECT repo_info.name, ANY_VALUE(repo_info.description) AS description
FROM repo_info
WHERE
  repo_info.description IS NOT NULL AND
  repo_info.description != ""
GROUP BY repo_info.name
ORDER BY repo_info.name

ref:
https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#json-functions
https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#any_value

Show repository informations (2)

WITH repo_info AS (
  SELECT repo.id AS id, repo.name AS name, JSON_EXTRACT_SCALAR(payload, '$.description') AS description
  FROM `githubarchive.month.201708`
  WHERE type = "CreateEvent"
)

SELECT repo_info.name, ANY_VALUE(repo_info.description) AS description
FROM repo_info
WHERE
  repo_info.description IS NOT NULL AND
  repo_info.description != ""
GROUP BY repo_info.name
ORDER BY repo_info.name

Show repository informations (3)

SELECT name, description
FROM `ghtorrent-bq.ght_2017_04_01.projects`
WHERE
  forked_from IS NULL AND
  description IS NOT NULL AND
  description != ""

Show starred repositories by a specific user

You must use WatchEvent for starring a repo:
https://developer.github.com/v3/activity/events/types/#watchevent

SELECT repo.name, created_at
FROM TABLE_QUERY([githubarchive:month], 'LEFT(table_ID,4) IN ("2017","2016","2015")') 
WHERE type = "WatchEvent" AND actor.login = 'vinta'
GROUP BY repo.name, created_at
ORDER BY created_at DESC

Show starred repositories per user who has 10+ starred repositories

WITH stars AS (
     SELECT DISTINCT actor.login AS user, repo.name AS repo
     FROM `githubarchive.month.2017*`
     WHERE type="WatchEvent"
),
repositories_stars AS (
     SELECT repo, COUNT(*) as c FROM stars GROUP BY repo
     ORDER BY c DESC
     LIMIT 1000
),
users_stars AS (
    SELECT user, COUNT(*) as c FROM  stars
    WHERE repo IN (SELECT repo FROM repositories_stars)
    GROUP BY user
    HAVING c >= 10
    LIMIT 10000
)
SELECT user, repo FROM stars
WHERE repo IN (SELECT repo FROM repositories_stars)
AND user IN (SELECT user FROM users_stars)

ref:
https://gist.github.com/jbochi/2e8ddcc5939e70e5368326aa034a144e

MkDocs: Deploy your Markdown documents on GitHub Pages

2015-06-072019-10-22VintaMisc

MkDocs is a static site generator that builds modern webpages based on your Markdown documents and a simple YAML file.

ref:
https://www.mkdocs.org/

Here is the website which is generated by MkDocs in this post:
https://awesome-python.com/
https://github.com/vinta/awesome-python

Installation

$ pip install mkdocs

Configuration

in mkdocs.yml

site_name: Awesome Python
site_url: https://awesome-python.com
site_description: A curated list of awesome Python frameworks, libraries and software
site_author: Vinta Chen
repo_name: vinta/awesome-python
repo_url: https://github.com/vinta/awesome-python
theme:
  name: material
  palette:
    primary: red
    accent: pink
extra:
  social:
    - type: github
      link: https://github.com/vinta
    - type: twitter
      link: https://twitter.com/vinta
    - type: linkedin
      link: https://www.linkedin.com/in/vinta
google_analytics:
  - UA-510626-7
  - auto
extra_css:
    - css/extra.css
nav:
  - "Life is short, you need Python.": "index.md"

There are more themes:

in Makefile

site_install:
    pip install -r requirements.txt

site_link:
    ln -sf $(CURDIR)/README.md $(CURDIR)/docs/index.md

site_preview: site_link
    mkdocs serve

site_build: site_link
    mkdocs build

site_deploy: site_link
    mkdocs gh-deploy --clean

Custom Domain for GitHub Pages

in docs/CNAME

awesome-python.com

After deploying your GitHub Page, just pointing your domain to following IPs with DNS A records:

185.199.108.153
185.199.109.153
185.199.110.153
185.199.111.153

ref:
https://help.github.com/articles/setting-up-an-apex-domain/#configuring-a-records-with-your-dns-provider
https://help.github.com/articles/troubleshooting-custom-domains/#https-errors

Automatic Deployment Via Travis CI

You need to

language: python

python:
  - "3.6"

script:
  - cp README.md docs/index.md
  - mkdocs build

deploy:
  provider: pages
  local-dir: site
  skip-cleanup: true
  keep-history: true
  github-token: $GITHUB_TOKEN
  on:
    branch: master

ref:
https://docs.travis-ci.com/user/deployment/pages/