Read and write files in Go

Reading file line by line

如果要一行一行地讀
建議用 bufio.Scanner
但是 Scanner 有個缺點
就是一行太長(超過 64K)的時候會出現 bufio.Scanner: token too long 的錯誤
這時候還是得用 bufio.Reader

fin, err := os.Open(path)
if err != nil {
    fmt.Println(err)
}
defer fin.Close()

scanner := bufio.NewScanner(fin)
for scanner.Scan() {
    line := scanner.Text()
    fmt.Fprintln(os.Stdin, line)
}

if err := scanner.Err(); err != nil {
    fmt.Fprintln(os.Stderr, err)
}

If you know the maximum length of the tokens you will be reading, copy the bufio.Scanner code into your project and change the const MaxScanTokenSize value.

ref:
http://stackoverflow.com/questions/6141604/go-readline-string
http://stackoverflow.com/questions/1821811/how-to-read-write-from-to-file
http://stackoverflow.com/questions/8757389/reading-file-line-by-line-in-go
http://stackoverflow.com/questions/5884154/golang-read-text-file-into-string-array-and-write
https://github.com/polaris1119/The-Golang-Standard-Library-by-Example/blob/master/chapter01/01.4.md

bufio
https://golang.org/pkg/bufio/
https://golang.org/pkg/bufio/#Scanner

Reading and writing file line by line

fmt.Fprintln(writer, line)bw.WriteString(line) 還要快

func FileSpacing(filename string, w io.Writer) (err error) {
    fr, err := os.Open(filename)
    if err != nil {
        return err
    }
    defer fr.Close()

    br := bufio.NewReader(fr)
    bw := bufio.NewWriter(w)

    for {
        line, err := br.ReadString('\n')
        if err == nil {
            fmt.Fprint(bw, TextSpacing(line))
        } else {
            if err == io.EOF {
                fmt.Fprint(bw, TextSpacing(line))
                break
            }
            return err
        }
    }
    defer bw.Flush()

    return nil
}

Copy a file

fin, _ := os.Open("source.txt")
fout, _ := os.Create("destination.txt")

io.Copy(fout, fin)

defer fout.Close()
defer fin.Close()

ref:
http://stackoverflow.com/questions/23272663/transfer-a-big-file-in-golang
http://golang.org/pkg/io/#Copy

Compute MD5 of a file

func md5Of(filename string) string {
    var result []byte

    file, err := os.Open(filename)
    checkError(err)
    defer file.Close()

    hash := md5.New()
    _, err = io.Copy(hash, file)
    checkError(err)

    checksum := hex.EncodeToString(hash.Sum(result))

    return checksum
}

ref:
http://stackoverflow.com/questions/29505089/how-can-i-compare-two-files-in-golang

goroutine, channel

channel

如果是沒有 buffer 的 channel
讀取 channel(value <- ch)會 block 當前的 goroutine,直到別的 goroutine 寫入數據(ch <- 1)
寫入 channel(ch <- 1)也會 block 當前的 goroutine,直到別的 goroutine 接收數據(value <- ch)
main() 其實也是個 goroutine

runtime.GOMAXPROCS(1) 的情況下(也是默認的情況)
同一時間只會有一個 goroutine 在跑
當 goroutine 遇到阻塞(例如 IO, time.Sleep())時
就會讓出 CPU 給別的 goroutine(相當於執行了 runtime.Gosched
也就是說如果某個 goroutine 沒有遇到阻塞
它就不會讓出執行權給其他的 goroutine
而是會一直執行到 return

func say(s string) {
    for i := 0; i < 5; i++ {
        fmt.Println(s)
    }
}

# 默認只會用到一個 CPU(只在一個 thread 裡跑)
# main() 這個 goroutine 被這個無限迴圈的 for loop 佔滿了(無限迴圈不算是阻塞)
# 沒有機會把執行權交給其他的 goroutine
func main() {
    go say("something")
    for {
    }
}

ref:
http://eleme.io/blog/2014/goroutine-1/
http://eleme.io/blog/2014/goroutine-2/
http://eleme.io/blog/2014/goroutine-3/

buffered channel

# 默認是沒有 buffer 的 channel
# 只要一有數據存入,在數據沒被取出之前,channel 都會阻塞
ch0 := make(chan int)

# 容量為 2 的 channel,可以想成是一個 queue
# 在 queue 滿之前,沒有 goroutine 會被阻塞
ch2 := make(chan int, 2)

runtime.GOMAXPROCS

Go 默認只會使用一個 CPU 來執行 goroutine
不會依據你的 CPU 數來切換
但是你可以用環境變數 GOMAXPROCS 指定
或是在程式裡指定

import "runtime"

runtime.GOMAXPROCS(runtime.NumCPU())

sync.WaitGroup

WaitGroups are more useful for doing different tasks in parallel.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "sync"
)

func main() {
    var wg sync.WaitGroup

    var aww string
    wg.Add(1)
    go func() {
        defer wg.Done()
        aww = fetch("http://www.reddit.com/r/aww.json")
    }()

    var funny string
    wg.Add(1)
    go func() {
        defer wg.Done()
        funny = fetch("http://www.reddit.com/r/funny.json")
    }()

    wg.Wait()

    fmt.Println("aww:", aww)
    fmt.Println("funny:", funny)
}

func fetch(url string) string {
    res, err := http.Get(url)
    if err != nil {
        log.Fatal(err)
    }
    body, err := ioutil.ReadAll(res.Body)
    res.Body.Close()
    if err != nil {
        log.Fatal(err)
    }
    return string(body)
}

ref:
http://nathanleclaire.com/blog/2014/02/21/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing-part-two-fixing-my-ooops/

fatal error: all goroutines are asleep - deadlock!

func main() {
    ch := make(chan int)
    <- ch
}

因為沒有任何人可以向 ch 寫入數據
所以 main() 的 goroutine 會一直阻塞
等待果陀的數據

godoc

Preview Go documents on localhost

$ go get golang.org/x/tools/cmd/godoc
$ godoc -http=:6060
$ open http://localhost:6060/pkg/

ref:
http://godoc.org/golang.org/x/tools/cmd/godoc

example
https://golang.org/pkg/bufio/
https://github.com/golang/go/blob/master/src%2Fbufio%2Fexample_test.go

View your project documentation on GoDoc.org

直接在 http://godoc.org/ 的搜尋框輸入你的 project path
例如 github.com/vinta/pangu
GoDoc 就會去 GitHub 把你的 project 抓進來

pangu
http://godoc.org/github.com/vinta/pangu