Category: Big Data
-
Updated
Build a recommender system with Spark: Logistic Regression
在本篇文章中，我們將以 Ranking 階段常用的方法之一：Logistic Regression 邏輯迴歸為例，利用 Apache Spark 的 Logistic Regression 模型建立一個 GitHub repositories 的推薦系統，以用戶對 repo 的打星紀錄和用戶與 repo 的各項屬性做為特徵，預測出用戶會不會打星某個 repo（分類問題）。最後訓練出來的模型就可以做為我們的推薦系統的 Ranking 模組。不過因為 LR 是線性模型，所以通常需要大量的 Feature Engineering 來習得非線性關係。所以這篇文章的重點是 Spark ML 的 Pipeline 機制和特徵工程，不會在演算法的部分著墨太多。
-
Updated
Spark best practices
Category: Big Data
A Spark application is a set of processes running on a cluster, all these processes are coordinated by the driver program. The driver program is 1) the process where the main() method of your program runs, 2) the process running the code that creates a SparkSession, RDDs, DataFrames, and stages up or sends off transformations and actions.
-
Updated
Build a recommender system with Spark: Content-based and Elasticsearch
在本篇文章中，我們以 Candidate Generation 階段常用的方法之一：Content-based recommendation 基於內容的推薦為例，利用 Elasticsearch 的 More Like This query 建立一個 GitHub repositories 的推薦系統，以用戶最近打星過的 repo 作為輸入數據，比對出相似的其他 repo 作為候選物品集。
-
Updated
Spark ML cookbook (Scala)
Scala is the first class citizen language for interacting with Apache Spark, but it's difficult to learn. This article is mostly about Spark ML - the new Spark Machine Learning library which was rewritten in DataFrame-based API.
-
Updated
Spark SQL cookbook (Scala)
Scala is the first class citizen language for interacting with Apache Spark, but it's difficult to learn. This article is mostly about operating DataFrame or Dataset in Spark SQL.
-
Updated
Play with GitHub Archive Dataset on BigQuery
Category: Big Data
Google BigQuery is a web service that lets you do interactive analysis of very massive datasets - analyzing billions of rows in seconds.
-
Updated
Setup Spark, Scala and Maven with Intellij IDEA
Category: Big Data
IntelliJ IDEA supports Scala and Apache Spark perfectly. You're able to browse a complete Spark project built with IntelliJ IDEA on GitHub: https://github.com/vinta/albedo
-
Updated
-
Updated
Build a recommender system with Spark: Implicit ALS
Category: AI, Big Data, Python
在本篇文章中，我們以 Candidate Generation 階段常用的方法之一：Collaborative Filtering 協同過濾演算法為例，利用 Apache Spark 的 ALS (Alternating Least Squares) 模型建立一個 GitHub repositories 的推薦系統，以用戶對 repo 的打星紀錄作為訓練數據，推薦出用戶可能會感興趣的其他 repo 作為候選物品集。
-
Updated
Spark ML cookbook (Python)
Category: AI, Big Data, Python
As of Spark 2.0, the RDD-based APIs in the spark.mllib package have entered maintenance mode. The primary Machine Learning API for Spark is now the DataFrame-based API in the spark.ml package.