IPFS: The (Very Slow) Distributed Permanent Web

IPFS: The (Very Slow) Distributed Permanent Web

IPFS stands for InterPlanetary File System, but you could simply consider it as a distributed, permanent, but ridiculously slow, not properly functioning version of web. You could upload any static file and static website to IPFS. And the whole swarm would probably distribute your files to the moon, that might be why IPFS is so fucking slow.

ref:
https://ipfs.io/

Installation

Install on macOS.

$ brew install ipfs

Start your IPFS node.

$ ipfs init
initializing IPFS node at /Users/vinta/.ipfs
generating 2048-bit RSA keypair... done
peer identity: QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6

$ ipfs daemon

ref:
https://ipfs.io/docs/commands/#ipfs-init
https://ipfs.io/docs/commands/#ipfs-daemon

Furthermore, you might want to run your IPFS node in a Docker container.

# docker-compose.yml
version: "3"
services:
    ipfs:
        image: ipfs/go-ipfs:v0.4.15
        working_dir: /export
        ports:
            - "4001:4001" # Swarm
            - "5001:5001" # web UI
            - "8080:8080" # HTTP proxy
        volumes:
            - "~/.ipfs:/data/ipfs"
            - "~/.ipfs/export:/export"

ref:
https://hub.docker.com/r/ipfs/go-ipfs/

Usage

Show Node Info

$ ipfs id
{
    "ID": "QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6",
    "PublicKey": "A_LONG_LONG_LONG_KEY,
    "Addresses": [
        "/ip4/127.0.0.1/tcp/4001/ipfs/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6",
        "/ip4/172.19.0.2/tcp/4001/ipfs/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6"
    ],
    "AgentVersion": "go-ipfs/0.4.14/5db3846",
    "ProtocolVersion": "ipfs/0.1.0"
}

ref:
https://ipfs.io/docs/getting-started/

Add Other Nodes to Your Bootstrap List

This one is from Muzeum, https://muzeum.pro/.

$ ipfs bootstrap add /ip4/52.221.121.238/tcp/4001/ipfs/QmTKYdZDkqHiY24kPynSmKbmRdk7cJxWsvvfvvvZArQ1N9

# you could also connect to a node directly
$ ipfs swarm connect /ip4/52.221.121.238/tcp/4001/ipfs/QmTKYdZDkqHiY24kPynSmKbmRdk7cJxWsvvfvvvZArQ1N9

ref:
https://ipfs.io/docs/commands/#ipfs-bootstrap
https://ipfs.io/docs/commands/#ipfs-swarm

Add Files to IPFS

Every IPFS node's default storage is 10GB, and a single node could only store data it needs, which also means each node only stores a small amount of whole data on IPFS. If there is not enough nodes, your data might be distributed to no one except your own node.

Your content is automatically pinned when you ipfs add it.

$ ipfs add -r mysite
added QmRticJ3P5fnb9GGnUj3U9XMkYvGEnv9AQfk6YmgRhivYA mysite/index.html
added QmY9cxiHqTFoWamkQVkpmmqzBrY3hCBEL2XNu3NtX74Fuu mysite/readme.md
added QmTLhFgeWLacpbiGNYmhchHGQAhfNyDZcLt5akJFFLV89V mysite

If files/folders under the folder change, the hash of the folder changes too.

$ vim mysite/index.html
$ ipfs add -r mysite
added QmQTTe3deLfeULKjPHnQTcyFuCmY5JZiwSTiPT4nSt1KVK mysite/index.html # changed
added QmS85tb3aKQNurFm51FaxtK6NyNei4ej3gDR21baDZXRoU mysite            # changed

ref:
https://ipfs.io/docs/commands/#ipfs-add

Pin Files from IPFS

Pinning means storing IPFS files on local node, and prevent them from getting garbage collected. Also, you could access them much quickly. You only need to do ipfs pin add to pin contents someone else uploaded.

$ ipfs pin add -r --progress /ipns/ipfs.soundscape.net/

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_group/index.json
pinned QmZwTEhdjT4MyvEnWndVEJzBjp8zGGZH1cEBpshBQs75rY recursively

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_album/index.json
pinned QmSAuGU5xt5SdR2ca2EDgeHFATSrAQhTfTYpYs9K9qmqED recursively

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_recording/index.json
pinned QmcTiadA9jRMXx77tydPa6492QJAtjXkKkA4gERaFksy94 recursively

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_composition/index.json
pinned QmTfqVaGVRnaPRQgYypGYXUvTK1UcDfK5VWYvU4rwK3m26 recursively

P.S. Sometimes when I ipfs pin add a file which is not on my node, the command just hangs there. I'm not sure why that once I access the file first (through curl or any browser), then ipfs pin add works fine. But it does not make sense: if I already get/access/download the file, I could just ipfs add the file and it would be automatically pinned.

ref:
https://ipfs.io/docs/commands/#ipfs-pin

Get Files

You have several ways to get files or folders from IPFS:

  • ipfs get dir-hash -o readable-dir-name
    • ipfs get QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk -o mysite
  • ipfs get file-hash -o readable-file-name.ext
    • ipfs get Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u -o cat.jpg
  • ipfs get /ipfs/dir-hash/path/to/file.txt
    • ipfs get /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/readme
  • ipfs get /ipns/example.com/path/to/file.txt
    • ipfs get /ipns/ipfs.soundscape.net/music_group/index.json

You could also access IPFS files through any public gateway:

  • curl https://ipfs.io/ipns/peer-id/path/to/file.txt
    • curl https://ipfs.io/ipns/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6/index.html
  • curl https://ipfs.io/ipns/example.com/path/to/file.txt
    • curl https://ipfs.io/ipns/ipfs.soundscape.net/music_group/index.json
  • curl http://127.0.0.1:8080/ipns/example.com/path/to/file.txt
    • curl http://127.0.0.1:8080/ipns/ipfs.soundscape.net/music_group/index.json

Download IPFS objects with ipfs get.

$ ipfs ls QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ
Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u 443362 cat.jpg

# you could get a folder
$ ipfs get QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ
$ ls QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ
cat.jpg

# as well as a file
$ ipfs get Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u -o cat.jpg

# get files and rename them
$ mkdir -p soundscape/music_group/ soundscape/music_album/ soundscape/music_recording/ soundscape/music_composition/ && \
  ipfs get /ipns/ipfs.soundscape.net/music_group/index.json -o soundscape/music_group/index.json; \
  ipfs get /ipns/ipfs.soundscape.net/music_album/index.json -o soundscape/music_album/index.json; \
  ipfs get /ipns/ipfs.soundscape.net/music_recording/index.json -o soundscape/music_recording/index.json; \
  ipfs get /ipns/ipfs.soundscape.net/music_composition/index.json -o soundscape/music_composition/index.json

# get whole folders
$ ipfs get /ipns/ipfs.soundscape.net/music_group; \
  ipfs get /ipns/ipfs.soundscape.net/music_album; \
  ipfs get /ipns/ipfs.soundscape.net/music_recording; \
  ipfs get /ipns/ipfs.soundscape.net/music_composition

ref:
https://ipfs.io/docs/commands/#ipfs-get
https://discuss.ipfs.io/t/trying-to-better-understand-the-pinning-concept/754

Display IPFS object data with ipfs cat.

$ ipfs cat Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u > cat.jpg
$ ipfs cat QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

Publish a Website to IPNS

IPNS stands for InterPlanetary Naming System.

Everytime you change files under a folder, the hash of the folder also changes. So you need a static reference which always points to the latest hash of your folder. You could publish your static website (a folder) to IPNS with the static reference, which is your peer ID as well as the hash of your public key.

By default, every IPFS node has only one pair of private and public key. Therefore, you could only publish one folder with your peer ID. But you could add new keypairs through ipfs key gen and publish multiple folders.

$ ipfs add -r mysite
added QmeqHWZgvgx5C7T6DakX75CJDRgAUoSDZayLYrcnAP8Fma mysite/index.html
added QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5 mysite

$ ipfs name publish QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5
published to QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6: /ipfs/QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5

$ ipfs name resolve QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6
/ipfs/QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5

Click following links to see contents.

After you change something, publish it again with new hash.

$ vim mysite/index.html
$ ipfs add -r mysite
added QmNjbhdks8RUgDt6QiNFe5QGe2HrbCsq5FKda9D9hLVkkU mysite/index.html # changed
added QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk mysite            # changed

$ ipfs name publish QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk
published to QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6: /ipfs/QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk

ref:
https://ipfs.io/docs/commands/#ipfs-name

Create a Domain Name Alias for Your Peer ID

The hash is not very friendly for humans. Fortunately, you could and probably should associate a domain name with your peer ID.

First, you need to add a TXT record whose value is dnslink=/ipns/YOUR_PEER_ID to your domain name. In the following article, we assume the domain name you choose is ipfs.kittenphile.com.

$ dig +short TXT ipfs.kittenphile.com
"dnslink=/ipns/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6"

$ ipfs name resolve -r ipfs.kittenphile.com
/ipfs/QmaE2DcNxGjPGPfzfTQuTBTW9D57abVSv319WqC89Av1y1

Click following links to see contents.

ref:
https://ipfs.io/docs/examples/example-viewer/example#../websites/README.md
https://hackernoon.com/ten-terrible-attempts-to-make-the-inter-planetary-file-system-human-friendly-e4e95df0c6fa

Public Gateway

If you have a public gateway and people retrieve files through it. Your public gateway fetches and stores the data, but it doesn't pin them. Files get removed with the next garbage collection run.

ref:
https://discuss.ipfs.io/t/public-facing-gateway-and-pinning/449

你都去哪裡看技術文章?

你都去哪裡看技術文章?

因為前陣子跟朋友們一起弄了一個技術週刊:CodeTengu Weekly 碼天狗週刊,每個禮拜在考慮要放哪些內容的時候,突然覺得:「你都去哪裡看技術文章?」或許也會是個有價值而且實用的主題,所以乾脆就來跟大家分享一下,我覺得不錯的每日資訊來源。

你可以訂閱的週報

五花八門

程式語言

資料庫

DevOps

Machine Learning

你可以瀏覽的網站

如果要推薦值得一看的網站或網誌,說八年都說不完,而且現在大家也都不用 RSS reader 了(真的很可惜,明明就很方便),這裡就只提幾個「內容聚合網站」(news aggregator)。你可以在這些網站上 follow 特定的主題,例如 Python、Golang、Apache Cassandra、Docker 之類的,他們就會自動把相關的文章推送給你,比較特別的是,網站還會根據你的個人喜好和你在 Twitter 上關注的對象來調整推送給你的內容。

我最早用過這一類的服務是 Zite,但是直到它老是推薦「印度一條六公尺的巨蟒(Python)吞食了一個人類小孩」的新聞給我之後,我就把它刪掉了。雖然說 Zite 已經被收購,整合進 Flipboard 裡,但是我已經對它沒信心啦。

2015.09.06 更新:

你可以關注的人

以下列出的是許多喜歡在 Twitter 上分享技術文章而且推文頻率又比較高的開發者:

出沒於 Twitter

出沒於 Facebook

Computer Names for Sci-Fi Maniac Developers

Computer Names for Sci-Fi Maniac Developers

The list is collected from books I read, movies I watched, and video games I played.

  • ADA (from Zone of the Enders: The 2nd Runner)
  • Albedo (from Hyperion) --> used for one of my code projects
  • Asurada (from Future GPX Cyber Formula)
  • BrainPal (from Old Man's War)
  • ctOS (from Watch Dogs)
  • Deep Thought (from The Hitchhiker's Guide to the Galaxy)
  • Eddie (from The Hitchhiker's Guide to the Galaxy)
  • EDI (from Mass Effect) --> used for one of my code projects
  • EVA-00 (from Neon Genesis Evangelion)
  • EVA-01 (from Neon Genesis Evangelion)
  • EVA-02 (from Neon Genesis Evangelion)
  • Fine Till You Came Along (from Culture: Look to Windward)
  • GERTY 3000 (from Moon)
  • GLaDOS (from Portal)
  • HAL 9000 (from 2001: A Space Odyssey) --> used for one of my code projects
  • Jane (from Ender's Game: Speaker for the Dead) --> used for My AirPods
  • JARVIS (from Iron Man)
  • KOS-MOS (from Xenosaga)
  • Limiting Factor (from The Culture: The Player of Games)
  • MAGI (from Neon Genesis Evangelion) --> used for my iPad
  • Melchizedek (from Gunnm Last Order)
  • Mike (from The Moon Is a Harsh Mistress)
  • Of Course I Still Love You (from The Culture: The Player of Games)
  • Pip-Boy 3000 (from Fallout) --> used for my iPhone
  • Project 2501 (from Ghost in the Shell)
  • Samantha (from Her)
  • Skynet (from The Terminator)
  • SAL-9000 (2010: The Year We Make Contact)
  • Sol-9000 (from Xenogears)
  • Sophon (from The Three-Body Problem)
  • Tachikoma (from Ghost in the Shell)
  • TARS (from Interstellar)
  • TechnoCore (from Hyperion) --> used for my Macbook Pro
  • Ummon (from Hyperion) --> used for one of my code projects
  • VIKI (from I, Robot)
  • Wheatley (from Portal)

ref:
https://en.wikipedia.org/wiki/List_of_fictional_computers
https://en.wikipedia.org/wiki/Artificial_intelligence_in_fiction

MkDocs: Deploy your Markdown documents on GitHub Pages

MkDocs: Deploy your Markdown documents on GitHub Pages

MkDocs is a static site generator that builds modern webpages based on your Markdown documents and a simple YAML file.

ref:
https://www.mkdocs.org/

Here is the website which is generated by MkDocs in this post:
https://awesome-python.com/
https://github.com/vinta/awesome-python

Installation

$ pip install mkdocs

Configuration

in mkdocs.yml

site_name: Awesome Python
site_url: https://awesome-python.com
site_description: A curated list of awesome Python frameworks, libraries and software
site_author: Vinta Chen
repo_name: vinta/awesome-python
repo_url: https://github.com/vinta/awesome-python
theme:
  name: material
  palette:
    primary: red
    accent: pink
extra:
  social:
    - type: github
      link: https://github.com/vinta
    - type: twitter
      link: https://twitter.com/vinta
    - type: linkedin
      link: https://www.linkedin.com/in/vinta
google_analytics:
  - UA-510626-7
  - auto
extra_css:
    - css/extra.css
nav:
  - "Life is short, you need Python.": "index.md"

There are more themes:

in Makefile

site_install:
    pip install -r requirements.txt

site_link:
    ln -sf $(CURDIR)/README.md $(CURDIR)/docs/index.md

site_preview: site_link
    mkdocs serve

site_build: site_link
    mkdocs build

site_deploy: site_link
    mkdocs gh-deploy --clean

Custom Domain for GitHub Pages

in docs/CNAME

awesome-python.com

After deploying your GitHub Page, just pointing your domain to following IPs with DNS A records:

  • 185.199.108.153
  • 185.199.109.153
  • 185.199.110.153
  • 185.199.111.153

ref:
https://help.github.com/articles/setting-up-an-apex-domain/#configuring-a-records-with-your-dns-provider
https://help.github.com/articles/troubleshooting-custom-domains/#https-errors

Automatic Deployment Via Travis CI

You need to

language: python

python:
  - "3.6"

script:
  - cp README.md docs/index.md
  - mkdocs build

deploy:
  provider: pages
  local-dir: site
  skip-cleanup: true
  keep-history: true
  github-token: $GITHUB_TOKEN
  on:
    branch: master

ref:
https://docs.travis-ci.com/user/deployment/pages/

Upload your Java Artifacts to Maven Central Repository

Upload your Java Artifacts to Maven Central Repository

你需要:

  1. 一個使用 Maven 管理的 Java project(廢話)
  2. 一個 GPG key(deploy 的時候會用來 sign 要提交的 .jar)
  3. 一個 Sonatype JIRA 的帳號
  4. 開一張 JIRA 的 ticket 告訴 Sonatype 的人你要發佈 library,告知他們你的 groupId
  5. 按照 Requirements 的指示完善你的 pom.xml
  6. deploy 到 snapshot repository
  7. deploy 到 staging repository
  8. 在 OSSRH 的 Staging Repositories 把你剛剛 deploy 的 library 給 close 掉,這樣才算是 release
  9. 回到那張 ticket,通知 Sonatype 讓他們把你的 library 同步到 Maven Central Repositir

最後一個步驟只有第一次 release 的時候才需要
之後 release 就會自動同步了

Requirements

http://maven.apache.org/guides/mini/guide-central-repository-upload.html
http://central.sonatype.org/pages/requirements.html
http://central.sonatype.org/pages/ossrh-guide.html
http://central.sonatype.org/pages/apache-maven.html
http://central.sonatype.org/pages/releasing-the-deployment.html

參考 Pangu.java 的 pom.xml
https://github.com/vinta/pangu.java/blob/master/pom.xml

Deployment

You need following plugins:

  • maven-source-plugin
  • maven-javadoc-plugin
  • maven-gpg-plugin
  • nexus-staging-maven-plugin
  • maven-release-plugin

deploy 之前
必須確定你的 local 的程式碼跟 scm 的程式碼是同步的
如果你要發布 1.0.0 版本的話
你的 pom.xml 裡要寫 1.0.0-SNAPSHOT
然後執行:

# deploy to snapshot repository
$ mvn clean deploy

你可以在 https://oss.sonatype.org/ 搜尋到
SNAPSHOT 版本測試都沒問題之後(當然你要先設定讓 Maven 能夠下載 SNAPSHOT 版本的 libraries)
就可以正式 release 了:

# cleanup for the release
$ mvn release:clean

# 要回答一些關於版本號的問題
# 它會自動幫你新增一個 tag 並且把 pom.xml 裡的 `<version>` 改成下個版本
$ mvn release:prepare

# deploy to staging repository
# 然後 Maven 會把上一步新增的 git tag 和 pom.xml 的變更直接 push 到 GitHub
$ mvn release:perform

Maven 會自動在 library 進到 staging repository 的時候把 -SNAPSHOT 字串拿掉

(第一次 release 才需要以下的動作)

然後你就可以在 https://oss.sonatype.org/#stagingRepositories
找到你剛剛 deploy 的 library
通常長得像是 wsvinta-1000(前面是 groupId)
要把它 close
然後再 release

除了第一次 release 要去 ticket 留言之外
之後 release 就會自動同步到 Maven Central Repository
不過通常會需要等一陣子才會在 Maven 上看到

ref:
http://dev.solita.fi/2014/10/22/publishing-to-maven-central-repository.html
http://lkrnac.net/blog/2014/03/deploy-to-maven-central/
http://kirang89.github.io/blog/2013/01/20/uploading-your-jar-to-maven-central/
http://superwang.me/2014/03/22/publish-your-library-to-maven-central-repository-part-1/
http://www.kongch.com/2013/05/deploy-to-central-repo/

如果你在 release 的過程中出了錯
要重新 release 的話
你得 revert 你的 git commit 到執行 mvn release:prepare 之前
然後再重新跑一次