Category: Big Data
-
Updated
Spark SQL cookbook (Python)
Category: AI, Big Data, Python
Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed.
-
Updated
Spark RDD methods (Python /
Scala) Category: AI, Big Data, Python
以下的 methods 有些是所有 RDD 都能用，有些則是 PairRDD 才有的功能。然後因為在不同的 projects 我先後用了 Spark 的 Python API 和 Scala API，所以以下的內容可能會混雜著兩者的範例。Scala API 要特別注意每個 method 接受和回傳的 type 的差異；Python API 就沒有這種限制了，畢竟是動態語言。
-
Updated
Setup Spark on macOS
Category: AI, Big Data, Python
Install Spark on macOS with Homebrew or the pre-built tarball. You need Java 8 JDK first.