Category: AI
-
Updated
Spark SQL cookbook (Python)
Category: AI, Big Data, Python
Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed.
-
Updated
Spark RDD methods (Python /
Scala) Category: AI, Big Data, Python
以下的 methods 有些是所有 RDD 都能用，有些則是 PairRDD 才有的功能。然後因為在不同的 projects 我先後用了 Spark 的 Python API 和 Scala API，所以以下的內容可能會混雜著兩者的範例。Scala API 要特別注意每個 method 接受和回傳的 type 的差異；Python API 就沒有這種限制了，畢竟是動態語言。
-
Updated
Setup Spark on macOS
Category: AI, Big Data, Python
Install Spark on macOS with Homebrew or the pre-built tarball. You need Java 8 JDK first.
-
Updated
Recommender System: Matrix Factorization
Category: AI
只要用戶對物品有過行為（Rui > 0），我們就假設用戶對該物品有偏好（Pui = 1），用戶對該物品的行為越多（Rui 越大），Cui 就會越大，表示置信度越高，假設越可信；如果用戶對該物品沒有行為（Rui = 0），就假設用戶對該物品沒有偏好（Pui = 0），但是因為 Cui = 1 + alpha x 0，所以這個假設相對來說置信度較低，比較不可信。
-
Updated
Calculate the similarity of two vectors
You should only calculate Pearson Correlations when the number of items in common between two users is > 1, preferably greater than 5/10. Only calculate the Pearson Correlation for two users where they have commonly rated items.
-
Updated
Recommender System: Collaborative Filtering 協同過濾推薦演算法
Category: AI
dataset 會是 m 個用戶對 n 個物品的評分 utility matrix，因為通常只有部分用戶和部份物品會有評分資料，所以是一個 sparse matrix（稀疏矩陣）。目標是利用這些稀疏的資料去預測出用戶對他還沒評分過的物品的評分。
-
Updated
Machine Learning glossary 常見名詞解釋
Category: AI
Common machine learning terms, each explained in a few lines, with the Chinese names alongside.
-
Updated
Setup Jupyter and other Machine Learning tools on macOS
Jupyter Notebook is an interactive environment for running code in the browser. It allows you to create interactive documents that contain live code, rich text elements and visualizations. It's also a widely used tool for Data Scientists to make prototypes or demonstrations.