建站
咨詢
售后
建站咨詢
新聞中心
Apache Pig是一個分析大型數(shù)據(jù)集的平臺，它由表達(dá)數(shù)據(jù)分析程序的高級語言和評估這些程序的基礎(chǔ)設(shè)施組成。Pig程序的突出特性是其結(jié)構(gòu)可以進(jìn)行大量的并行化，進(jìn)而使其能夠處理非常大的數(shù)據(jù)集。
創(chuàng)新互聯(lián)建站成都網(wǎng)站建設(shè)按需策劃，是成都網(wǎng)站設(shè)計(jì)公司,為成都展覽展示提供網(wǎng)站建設(shè)服務(wù),有成熟的網(wǎng)站定制合作流程，提供網(wǎng)站定制設(shè)計(jì)服務(wù)：原型圖制作、網(wǎng)站創(chuàng)意設(shè)計(jì)、前端HTML5制作、后臺程序開發(fā)等。成都網(wǎng)站制作熱線：18982081108
1. 問題描述
收集日志avro數(shù)據(jù)中有兩個Map字段appInstall、appUse分別表示已安裝的app、正在使用的app，且key值為app的名稱，value值為app使用信息?，F(xiàn)在要得到一份匹配上購物類app支付寶|京東|淘寶|天貓的用戶名單；MapReduce 解決辦法如下：
public static class M extends Mapper {
Text text = new Text();
@SuppressWarnings("unchecked")
@Override
protected void map(String key, Pair value, Context context) throws IOException, InterruptedException {
Map data = value.fields.data;
String dvc = data.get("dvc").toString();
Map appInstall = (Map) data.get("appInstall");
Map appUse = (Map) data.get("appUse");
for(String app: appInstall.keySet()) {
if(app.matches("支付寶|京東|淘寶|天貓")) {
text.set(appInstall.keySet().toString());
context.write(dvc, text);
return;
}
}
for(String app: appUse.keySet()) {
if(app.matches("支付寶|京東|淘寶|天貓")) {
text.set(appUse.keySet().toString());
context.write(dvc, text);
return;
}
}
}
}
但是，如果要匹配游戲類的app、金融類的app類呢？如果匹配關(guān)鍵詞發(fā)生了變化呢？顯然，我們應(yīng)該將匹配關(guān)鍵詞開放成API，可以自由地匹配正則表達(dá)式。這時(shí)，pig派上了用場。
2. Bag正則匹配
A = load '//' using org.apache.pig.piggybank.storage.avro.AvroStorage();
-- A: {key: chararray,value: (fields: (data: map[]))}
B = foreach A generate value.fields.data#'dvc' as dvc, value.fields.data#'appInstall' as ins:map[], value.fields.data#'appUse' as use:map[];
-- B: {dvc: bytearray,ins: map[],use: map[]}
C = foreach B generate dvc, KEYSET(ins) as insk, KEYSET(use) as usek;
-- C: {dvc: bytearray,insk: {(chararray)},usek: {(chararray)}}
在上述代碼中，load 數(shù)據(jù)轉(zhuǎn)換得到bag類型的app-set（insk與usek）；但是，應(yīng)如何遍歷bag中的tuple與正則表達(dá)式做匹配呢？答案是UDF。
Apache DataFu Pig 提供了豐富的UDF，其中關(guān)于bags的UDF可以參看這里。TupleFromBag 提供根據(jù)index從bag提取tuple，支持三個輸入?yún)?shù)。依葫蘆畫瓢，遍歷bag匹配正則表達(dá)式的UDF如下：
package com.pig.udf.bag;
/**
* This UDF will return true if one tuple from a bag matches regex.
*
* There are two input parameter:
* 1. DataBag
* 2. Regex String
*/
public class BagMatchRegex extends FilterFunc {
@Override
public Boolean exec(Tuple tinput) throws IOException {
try{
DataBag samples = (DataBag) tinput.get(0);
String regex = (String) tinput.get(1);
for (Tuple tuple : samples) {
if(((String) tuple.get(0)).matches(regex)){
return true;
}
}
}
catch (Exception e) {
return false;
}
return false;
}
}
其中，F(xiàn)ilterFunc為過濾UDF的基類，繼承于EvalFunc，即exec(Tuple tinput)的返回值必為Boolean類型。bag正則匹配的pig腳本如下：
REGISTER ../piglib/udf-0.0.1-SNAPSHOT-jar-with-dependencies.jar
define BagMatchRegex com.pig.udf.bag.BagMatchRegex();
A = load '/user/../current/*.avro' using org.apache.pig.piggybank.storage.avro.AvroStorage();
B = foreach A generate value.fields.data#'dvc' as dvc, value.fields.data#'appInstall' as ins:map[], value.fields.data#'appUse' as use:map[];
C = foreach B generate dvc, KEYSET(ins) as insk, KEYSET(use) as usek;
D = filter C by BagMatchRegex(insk, '支付寶|京東|淘寶|天貓') or BagMatchRegex(usek, '支付寶|京東|淘寶|天貓');
3. 優(yōu)化
還有沒有可以做優(yōu)化的地方呢？我們先來看看pig中的KEYSET實(shí)現(xiàn)：
package org.apache.pig.builtin;
public class KEYSET extends EvalFunc {
private static final TupleFactory TUPLE_FACTORY = TupleFactory.getInstance();
@SuppressWarnings("unchecked")
@Override
public DataBag exec(Tuple input) throws IOException {
if(input == null || input.size() == 0) {
return null;
}
Map m = null;
// Input must be of type Map. This is verified at compile time
m = (Map)(input.get(0));
if(m == null) {
return null;
}
DataBag bag = new NonSpillableDataBag(m.size());
for (String s : m.keySet()) {
Tuple t = TUPLE_FACTORY.newTuple(s);
bag.add(t);
}
return bag;
}
...
}
需要指出的一點(diǎn)——pig的map數(shù)據(jù)類型是由Java類Map實(shí)現(xiàn)的。從KEYSET源碼中可以看出在調(diào)用時(shí)已經(jīng)將map遍歷了一次，然后在調(diào)用BagMatchRegex時(shí)又需要將key-set的bag再遍歷一次。其實(shí)，完全可以只用一次遍歷做map-key值的正則匹配：
package com.pig.udf.map;
/**
* This UDF will return true if map's key matches regex. * * There are two input parameter: * 1. Map * 2. Regex String */ public class KeyMatchRegex extends FilterFunc { @SuppressWarnings("unchecked") @Override public Boolean exec(Tuple input) throws IOException { try{ Map m = null; // Input must be of type Map. This is verified at compile time m = (Map)(input.get(0)); String regex = (String) input.get(1); for (String key : m.keySet()) { if(key.matches(regex)){ return true; } } } catch (Exception e) { return false; } return false; } } 本文題目：通過Pig實(shí)現(xiàn)關(guān)鍵詞匹配
文章網(wǎng)址：http://m.fisionsoft.com.cn/article/dhdhcgj.html
