\u5728\u9019\u500b\u7cfb\u5217\u7684\u6587\u7ae0\u88e1\uff0c\u6211\u5011\u5c07\u4f7f\u7528 Apache Spark\u3001XGBoost\u3001Elasticsearch \u548c MySQL \u7b49\u5de5\u5177\u4f86\u642d\u5efa\u4e00\u500b\u63a8\u85a6\u7cfb\u7d71\u7684 Machine Learning Pipeline\u3002\u63a8\u85a6\u7cfb\u7d71\u7684\u7d44\u6210\u53ef\u4ee5\u7c97\u7565\u5730\u5206\u6210 Candidate Generation \u548c Ranking \u5169\u500b\u90e8\u5206\uff0c\u524d\u8005\u662f\u91dd\u5c0d\u7528\u6236\u7522\u751f\u5019\u9078\u7269\u54c1\u96c6\uff0c\u5e38\u7528\u7684\u65b9\u6cd5\u6709 Collaborative Filtering\u3001Content-based\u3001\u6a19\u7c64\u914d\u5c0d\u3001\u71b1\u9580\u6392\u884c\u6216\u4eba\u5de5\u7cbe\u9078\u7b49\uff1b\u5f8c\u8005\u5247\u662f\u5c0d\u9019\u4e9b\u5019\u9078\u7269\u54c1\u6392\u5e8f\uff0c\u4ee5 Top N \u7684\u65b9\u5f0f\u5448\u73fe\u6700\u7d42\u7684\u63a8\u85a6\u7d50\u679c\uff0c\u5e38\u7528\u7684\u65b9\u6cd5\u6709 Logistic Regression\u3002<\/p>\n
\u5728\u672c\u7bc7\u6587\u7ae0\u4e2d\uff0c\u6211\u5011\u5c07\u4ee5 Candidate Generation \u968e\u6bb5\u5e38\u7528\u7684\u65b9\u6cd5\u4e4b\u4e00\uff1aContent-based recommendation \u57fa\u65bc\u5167\u5bb9\u7684\u63a8\u85a6\u70ba\u4f8b\uff0c\u5229\u7528 Elasticsearch \u7684 More Like This query \u5efa\u7acb\u4e00\u500b GitHub repositories \u7684\u63a8\u85a6\u7cfb\u7d71\uff0c\u4ee5\u7528\u6236\u6700\u8fd1\u6253\u661f\u904e\u7684 repo \u4f5c\u70ba\u8f38\u5165\u6578\u64da\uff0c\u6bd4\u5c0d\u51fa\u76f8\u4f3c\u7684\u5176\u4ed6 repo \u4f5c\u70ba\u5019\u9078\u7269\u54c1\u96c6\u3002<\/p>\n
\u984c\u5916\u8a71\uff0c\u6211\u539f\u672c\u662f\u6253\u7b97\u7528 Spark \u628a repo \u7684\u6587\u672c\u8cc7\u6599\u8f49\u6210 Word2Vec \u5411\u91cf\uff0c\u7136\u5f8c\u4e8b\u5148\u8a08\u7b97\u597d\u5404\u500b repo \u4e4b\u9593\u7684\u76f8\u4f3c\u5ea6\uff08\u6240\u8b02\u7684 Similarity Join\uff09\uff0c\u4f46\u662f\u8981\u8a08\u7b97\u9019\u9ebc\u591a repo \u4e4b\u9593\u7684\u76f8\u4f3c\u5ea6\u5be6\u5728\u592a\u8017\u6642\u9593\u548c\u6a5f\u5668\u4e86\uff0c\u5c31\u7b97\u7528\u4e86 DIMSUM \u548c Locality Sensitive Hashing (LSH) \u7684 Approximate Nearest Neighbor Search \u7684\u6548\u679c\u4e5f\u4e0d\u662f\u5f88\u597d\u3002\u5f8c\u4f86\u4e00\u60f3\uff0c\u5c0b\u627e\u76f8\u4f3c\u6216\u76f8\u95dc\u7269\u54c1\u9019\u4ef6\u4e8b\u4e0d\u5c31\u662f\u641c\u5c0b\u5f15\u64ce\u5728\u505a\u7684\u55ce\uff0c\u6240\u4ee5\u76f4\u63a5\u628a repo \u7684\u5404\u7a2e\u8cc7\u6599\u4e1f\u9032 Elasticsearch\uff0c\u7528 document id \u7576\u4f5c\u641c\u5c0b\u689d\u4ef6\uff0c\u4e00\u500b More Like This query \u5c31\u89e3\u6c7a\u4e86\uff0c\u723d\u5feb\u3002\u7562\u7adf\u4e0d\u9700\u8981\u6240\u6709\u7684\u4e8b\u60c5\u90fd\u5728 Spark \u88e1\u89e3\u6c7a\u561b\u3002<\/p>\n
\u5b8c\u6574\u7684\u7a0b\u5f0f\u78bc\u53ef\u4ee5\u5728 https:\/\/github.com\/vinta\/albedo<\/a> \u627e\u5230\u3002<\/p>\n \u7cfb\u5217\u6587\u7ae0\uff1a<\/p>\n \u70ba\u4e86\u8b93\u4e8b\u60c5\u7c21\u55ae\u4e00\u9ede\uff0c\u6211\u5011\u76f4\u63a5\u7528\u5b98\u65b9\u5305\u88dd\u597d\u7684 Docker image\u3002\u53e6\u5916\u8981\u6ce8\u610f\u7684\u662f\uff0cElasticsearch 5.x\/6.x \u8ddf\u4e4b\u524d\u7684\u7248\u672c\u6bd4\u8d77\u4f86\u6709\u4e0d\u5c0f\u7684\u6539\u52d5\uff0c\u4f8b\u5982 X-Pack\u3001high-level REST client \u548c\u4ee5\u5f8c\u6bcf\u500b index \u53ea\u80fd\u6709\u4e00\u500b mapping type \u7b49\u7b49\uff0c\u5efa\u8b70\u5927\u5bb6\u6709\u7a7a\u53ef\u4ee5\u7ffb\u4e00\u4e0b\u6587\u4ef6\u3002<\/p>\n \u7136\u5f8c\u5c31\u53ef\u4ee5\u5728 http:\/\/127.0.0.1:9200\/<\/a> \u5b58\u53d6\u4f60\u7684 Elasticsearch cluster \u4e86\u3002<\/p>\n ref: \u9019\u88e1\u7528 Elasticsearch: More than a Search Engine ref: \u4f60\u53ef\u4ee5\u900f\u904e\u5f88\u591a\u7a2e\u624b\u6bb5\u628a\u5b58\u5728 MySQL \u88e1\u7684\u8cc7\u6599\u5012\u9032 Elasticsearch\uff0c\u4f8b\u5982 cronjob\u3001Celery \u6216 MySQL binglog replication\uff0c\u4e0d\u904e\u56e0\u70ba\u6211\u5011\u4e3b\u8981\u7684 data models \u662f\u7528 Django ORM \u5beb\u7684\uff0c\u9019\u88e1\u5c31\u7c21\u55ae\u5730\u5beb\u500b Django command \u628a\u8cc7\u6599\u5012\u9032\u53bb\u5c31\u597d\u3002<\/p>\n noplay\/python-mysql-replication \u56e0\u70ba\u4e4b\u5f8c\u6703\u5728 Spark \u88e1\u4f5c\u70ba\u63a8\u85a6\u7cfb\u7d71\u7684\u5019\u9078\u7269\u54c1\u96c6\u7684\u4f86\u6e90\u4e4b\u4e00\uff0c\u6211\u5011\u6703\u628a Elasticsearch \u7684 More Like This API \u5c01\u88dd\u6210\u4e00\u500b Spark \u7684 Transformer\uff0c\u6240\u4ee5\u4ee5\u4e0b\u7684\u90e8\u5206\u662f\u7528 Scala \u5beb\u7684\u3002<\/p>\n Elasticsearch 5.x \u4e4b\u5f8c\u5b98\u65b9\u5efa\u8b70\u4f7f\u7528 High-level REST Client\uff0c\u7528\u6cd5\u8ddf\u4ee5\u524d Java \u7684 ref: \u6211\u5011\u6703\u8f38\u5165\u4e00\u500b \u4f60\u53ef\u4ee5\u5728 GitHub \u627e\u5230\u5b8c\u6574\u7684\u7a0b\u5f0f\u78bc ref: \u5728\u672c\u7bc7\u6587\u7ae0\u4e2d\uff0c\u6211\u5011\u4ee5 Candidate Generation \u968e\u6bb5\u5e38\u7528\u7684\u65b9\u6cd5\u4e4b\u4e00\uff1aContent-based recommendation \u57fa\u65bc\u5167\u5bb9\u7684\u63a8\u85a6\u70ba\u4f8b\uff0c\u5229\u7528 Elasticsearch \u7684 More Like This query \u5efa\u7acb\u4e00\u500b GitHub repositories \u7684\u63a8\u85a6\u7cfb\u7d71\uff0c\u4ee5\u7528\u6236\u6700\u8fd1\u6253\u661f\u904e\u7684 repo \u4f5c\u70ba\u8f38\u5165\u6578\u64da\uff0c\u6bd4\u5c0d\u51fa\u76f8\u4f3c\u7684\u5176\u4ed6 repo \u4f5c\u70ba\u5019\u9078\u7269\u54c1\u96c6\u3002<\/p>\n","protected":false},"author":1,"featured_media":447,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[97,112],"tags":[108,70,98,2,104,109,71],"class_list":["post-446","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-about-ai","category-about-big-data","tag-apache-spark","tag-elasticsearch","tag-machine-learning","tag-python","tag-recommender-system","tag-scala","tag-search"],"_links":{"self":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts\/446","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/comments?post=446"}],"version-history":[{"count":0,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts\/446\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/media\/447"}],"wp:attachment":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/media?parent=446"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/categories?post=446"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/tags?post=446"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
Setup Elasticsearch<\/h2>\n
# in elasticsearch.yml\nbootstrap.memory_lock: true\ncluster.name: albedo\ndiscovery.type: single-node\nhttp.host: 0.0.0.0\nnode.name: ${HOSTNAME}\nxpack.security.enabled: false<\/code><\/pre>\n# in docker-compose.yml\nversion: \"3\"\nservices:\n django:\n build: .\n hostname: django\n working_dir: \/app\n env_file: .docker-assets\/django.env\n command: .docker-assets\/django_start.sh\n ports:\n - 8000:8000\n volumes:\n - \".:\/app\"\n - \"..\/albedo-vendors\/bin:\/usr\/local\/bin\"\n - \"..\/albedo-vendors\/dist-packages:\/usr\/local\/lib\/python3.5\/dist-packages\"\n links:\n - mysql\n - elasticsearch\n mysql:\n image: vinta\/mysql:5.7\n hostname: mysql\n env_file: .docker-assets\/mysql.env\n command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci\n ports:\n - 3306:3306\n elasticsearch:\n image: docker.elastic.co\/elasticsearch\/elasticsearch:5.6.2\n ports:\n - 9200:9200\n - 9300:9300\n volumes:\n - \".\/.docker-assets\/elasticsearch.yml:\/usr\/share\/elasticsearch\/config\/elasticsearch.yml\"\n environment:\n - \"ES_JAVA_OPTS=-Xms512m -Xmx512m\"<\/code><\/pre>\n$ docker-compose up<\/code><\/pre>\n
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/5.6\/docker.html<\/a>
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/5.6\/security-settings.html<\/a><\/p>\nDefine the Mapping (Data Schema)<\/h2>\n
elasticsearch-dsl-py<\/code> \u5b9a\u7fa9\u4e86\u4e00\u500b index \u548c mapping type\u3002<\/p>\nfrom elasticsearch.helpers import bulk\nfrom elasticsearch_dsl import analyzer\nfrom elasticsearch_dsl import Date, Integer, Keyword, Text, Boolean\nfrom elasticsearch_dsl import Index, DocType\nfrom elasticsearch_dsl.connections import connections\n\nclient = connections.create_connection(hosts=['elasticsearch'])\n\nrepo_index = Index('repo')\nrepo_index.settings(\n number_of_shards=1,\n number_of_replicas=0\n)\n\ntext_analyzer = analyzer(\n 'text_analyzer',\n char_filter=[\"html_strip\"],\n tokenizer=\"standard\",\n filter=[\"asciifolding\", \"lowercase\", \"snowball\", \"stop\"]\n)\nrepo_index.analyzer(text_analyzer)\n\n@repo_index.doc_type\nclass RepoInfoDoc(DocType):\n owner_id = Keyword()\n owner_username = Keyword()\n owner_type = Keyword()\n name = Text(text_analyzer, fields={'raw': Keyword()})\n full_name = Text(text_analyzer, fields={'raw': Keyword()})\n description = Text(text_analyzer)\n language = Keyword()\n created_at = Date()\n updated_at = Date()\n pushed_at = Date()\n homepage = Keyword()\n size = Integer()\n stargazers_count = Integer()\n forks_count = Integer()\n subscribers_count = Integer()\n fork = Boolean()\n has_issues = Boolean()\n has_projects = Boolean()\n has_downloads = Boolean()\n has_wiki = Boolean()\n has_pages = Boolean()\n open_issues_count = Integer()\n topics = Keyword(multi=True)\n\n class Meta:\n index = repo_index._name\n\n @classmethod\n def bulk_save(cls, documents):\n dicts = (d.to_dict(include_meta=True) for d in documents)\n return bulk(client, dicts)\n\n def save(self, **kwargs):\n return super(RepoInfoDoc, self).save(**kwargs)\n\nRepoInfoDoc.init()<\/code><\/pre>\n
\nhttps:\/\/vinta.ws\/code\/elasticsearch-more-than-a-search-engine.html<\/a><\/p>\n
\nhttps:\/\/github.com\/elastic\/elasticsearch-dsl-py<\/a><\/p>\nImport Data into Elasticsearch<\/h2>\n
from django.core.management.base import BaseCommand\n\nfrom app.mappings import RepoInfoDoc\nfrom app.models import RepoInfo\n\nclass Command(BaseCommand):\n def handle(self, *args, **options):\n def batch_qs(qs, batch_size=500):\n total = qs.count()\n for start in range(0, total, batch_size):\n end = min(start + batch_size, total)\n yield (start, end, total, qs[start:end])\n\n large_qs = RepoInfo.objects.filter(stargazers_count__gte=10, stargazers_count__lte=290000, fork=False)\n for start, end, total, qs_chunk in batch_qs(large_qs):\n documents = []\n for repo_info in qs_chunk:\n repo_info_doc = RepoInfoDoc()\n repo_info_doc.meta.id = repo_info.id\n repo_info_doc.owner_id = repo_info.owner_id\n repo_info_doc.owner_username = repo_info.owner_username\n repo_info_doc.owner_type = repo_info.owner_type\n repo_info_doc.name = repo_info.name\n repo_info_doc.full_name = repo_info.full_name\n repo_info_doc.description = repo_info.description\n repo_info_doc.language = repo_info.language\n repo_info_doc.created_at = repo_info.created_at\n repo_info_doc.updated_at = repo_info.updated_at\n repo_info_doc.pushed_at = repo_info.pushed_at\n repo_info_doc.homepage = repo_info.homepage\n repo_info_doc.size = repo_info.size\n repo_info_doc.stargazers_count = repo_info.stargazers_count\n repo_info_doc.forks_count = repo_info.forks_count\n repo_info_doc.subscribers_count = repo_info.subscribers_count\n repo_info_doc.fork = repo_info.fork\n repo_info_doc.has_issues = repo_info.has_issues\n repo_info_doc.has_projects = repo_info.has_projects\n repo_info_doc.has_downloads = repo_info.has_downloads\n repo_info_doc.has_wiki = repo_info.has_wiki\n repo_info_doc.has_pages = repo_info.has_pages\n repo_info_doc.open_issues_count = repo_info.open_issues_count\n repo_info_doc.topics = repo_info.topics\n\n documents.append(repo_info_doc)\n\n RepoInfoDoc.bulk_save(documents)<\/code><\/pre>\n
\nhttps:\/\/github.com\/noplay\/python-mysql-replication<\/a><\/p>\nFind Similar Items<\/h2>\n
Initialize High-level REST Client<\/h3>\n
TransportClient<\/code> \u7a0d\u5fae\u6709\u9ede\u4e0d\u540c\u3002<\/p>\nimport org.apache.http.HttpHost\nimport org.elasticsearch.client.{RestClient, RestHighLevelClient}\n\nval lowClient = RestClient.builder(new HttpHost(\"127.0.0.1\", 9200, \"http\")).build()\nval highClient = new RestHighLevelClient(lowClient)<\/code><\/pre>\n
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/client\/java-rest\/current\/java-rest-low-usage-initialization.html<\/a>
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/client\/java-rest\/current\/java-rest-high-getting-started-initialization.html<\/a><\/p>\nPerform the More Like This Query<\/h3>\n
userDF<\/code>\uff0c\u662f\u4e00\u500b\u8981\u7522\u751f\u5019\u9078\u7269\u54c1\u96c6\u7684\u7528\u6236\u7684 DataFrame\uff0c\u7136\u5f8c\u6703\u5148\u62ff\u5230\u6bcf\u500b\u7528\u6236\u6700\u8fd1\u6253\u661f\u904e\u7684 repo \u7684\u5217\u8868\uff0crepo id \u5c31\u662f Elasticsearch \u7684 document id\uff0c\u4ee5\u6b64\u70ba\u689d\u4ef6\u7528 More Like This query \u627e\u51fa\u76f8\u4f3c\u7684\u5176\u4ed6 repo\u3002<\/p>\nval userRecommendedItemDF = userDF\n .flatMap {\n case (userId: Int) => {\n val itemIds = selectUserStarredRepos(userId)\n\n val lowClient = RestClient.builder(new HttpHost(\"127.0.0.1\", 9200, \"http\")).build()\n val highClient = new RestHighLevelClient(lowClient)\n\n val fields = Array(\"description\", \"full_name\", \"language\", \"topics\")\n val texts = Array(\"\")\n val items = itemIds.map((itemId: Int) => new Item(\"repo\", \"repo_info_doc\", itemId.toString))\n val queryBuilder = moreLikeThisQuery(fields, texts, items)\n .minTermFreq(1)\n .maxQueryTerms(20)\n\n val searchSourceBuilder = new SearchSourceBuilder()\n searchSourceBuilder.query(queryBuilder)\n searchSourceBuilder.from(0)\n searchSourceBuilder.size($(topK))\n\n val searchRequest = new SearchRequest()\n searchRequest.indices(\"repo\")\n searchRequest.types(\"repo_info_doc\")\n searchRequest.source(searchSourceBuilder)\n\n val searchResponse = highClient.search(searchRequest)\n val hits = searchResponse.getHits\n val searchHits = hits.getHits\n\n val userItemScoreTuples = searchHits.map((searchHit: SearchHit) => {\n val itemId = searchHit.getId.toInt\n val score = searchHit.getScore\n (userId, itemId, score)\n })\n\n lowClient.close()\n\n userItemScoreTuples\n }\n }\n .toDF($(userCol), $(itemCol), $(scoreCol))\n .withColumn($(sourceCol), lit(source))\n\nuserRecommendedItemDF.show()\n\/\/ +-------+--------+---------+-------+\n\/\/ |user_id|repo_id |score |source |\n\/\/ +-------+--------+---------+-------+\n\/\/ |652070 |26152923|44.360096|content|\n\/\/ |652070 |28451314|38.752697|content|\n\/\/ |652070 |16175350|35.676353|content|\n\/\/ |652070 |10885469|30.280012|content|\n\/\/ |652070 |24037308|28.488512|content|\n\/\/ +-------+--------+---------+-------+<\/code><\/pre>\n
\nhttps:\/\/github.com\/vinta\/albedo\/blob\/master\/src\/main\/scala\/ws\/vinta\/albedo\/ContentRecommenderBuilder.scala<\/a><\/p>\n
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/query-dsl-mlt-query.html<\/a>
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/client\/java-api\/current\/java-specialized-queries.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"