和Elasticsearch脚本评分、重评分机制

2020-12-19 10:06:37 编辑 0 阅读量:1977

## 1. 简单的脚本
- [1][基于Elasticsearch的自定义评分算法扩展 (转)](http://www.cnblogs.com/lwj--/p/6601663.html)
- [2][Elasticsearch使用rescore重打分机制](http://www.jianshu.com/p/98876ed6f27c)  ,主要是rescore相关
- [3][Appendix A. Painless API Reference](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/painless-api-reference.html)
- [4][groovy的Elasticsearch语法说明](https://www.packtpub.com/books/content/extending-elasticsearch-scripting)
- [5][高级文本评分脚本](http://cwiki.apachecn.org/pages/viewpage.action?pageId=9405381)

```
GET /lawplatform_tmp/criminal_law_tmp/_search
{
  "query": {
    "function_score": {
        "query": {"match": {"all": "刑事案件"} },
        "script_score": {
            "script": {
                "lang": "groovy",
                "inline": "score=0;termInfo=_index['title'].get('某某',_POSITIONS);println 322222;println termInfo.tf();for (pos in termInfo) {println pos.position;score = score + pos.position};return score;"
            }
        }
    }
  },
  "_source": "title"
}
println _score;
"script": {
                "lang": "groovy",
                "inline": "score=0;termInfo=_index['title'].get(terms,_POSITIONS);println 322222;println termInfo.tf();for (pos in termInfo) {println pos.position;score = score + pos.position};return score;",
                "params": {
                    "term": "某某"
				}
            }
```
## 2. 复杂的脚本
结合一大堆资料后，终于写出一个评分脚本
- [1][Advanced text scoring in scripts](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-advanced-scripting.html)
- [2][Token count datatype](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/token-count.html)
- [3][Accessing document fields and special variables](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-scripting-fields.html)
- [4][Groovy Scripting Language](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-scripting-groovy.html)
- [5][How to use scripts](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-scripting-using.html)
- [6][Function Score Query](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/query-dsl-function-score-query.html)
- [7][Painless Scripting in Elasticsearch](https://qbox.io/blog/how-to-painless-scripting-in-elasticsearch)
- [8][精通 Groovy](https://www.ibm.com/developerworks/cn/education/java/j-groovy/j-groovy.html)
- [9][groovy w3c教程](https://www.w3cschool.cn/groovy/groovy_overview.html)

```
GET /index_name/doc_type_name/_search
{
  "query": {
    "function_score": {
        "script_score": {
            "script": {
                "lang": "groovy",
                "inline": """
score=0
b = 0.35 
k1 = 1.2
k3 = 8.0
rate = 0.045
sigma = 20
occurqijd=0
min_score = -10000
double_min=4.9e-324
tfqijd=0
score_crter=0
score_bm25=0

field = "all" // 要查询的域，需要从外面传入
docs_num = _index[field].docCount()
field_length = doc['all.length' ].value  // 建索引时需要指定token_count，指该文档的该域的词长
avg_field_length = _index[field].sumttf() / docs_num  //一些基本信息
K = k1 * ((1 - b) + b * field_length/ avg_field_length)
qtf = []
for (term in terms) {
    within = 0;
    for (term_next in terms) {
        if (term == term_next) {
            within = within + 1;
        }   
    }   
    qtf.add(within)
}
println qtf 
terms_size = terms.size()
for(qi=0;qi<terms_size; qi++){
    term=terms[qi]
    term_info=_index[field].get(term, _FREQUENCIES | _POSITIONS | _CACHE)
    tfqid = term_info.tf()
    if (tfqid == 0){
        continue
    }
    term_info_list = []
    for (pos in term_info){
      term_info_list.add(pos.position)
    }
    println term;
    println term_info_list;
    //crossterm 算法，基于词的位置信息
    for (qj=qi+1; qj<terms_size; qj++){
        term_next = terms[qj]
        term_next_info = _index[field].get(term_next, _FREQUENCIES | _POSITIONS | _CACHE)
        ndqij = 0 
        tfqjd = term_next_info.tf()
        if (tfqjd == 0)
            continue
        term_next_info_list = []
        for (pos in term_next_info){
          term_next_info_list.add(pos.position)
        }
        withinqij = min(qtf[qi], qtf[qj])
        qtf_kernel = my_kernel(intersect(1), sigma) * withinqij
        for (pos_term in term_info_list) {
            position_k = pos_term
            for (pos_term_next in term_next_info_list) {
                position_k2 = pos_term_next
                kernel_termp = my_kernel(intersect(1) * abs(position_k - position_k2), sigma)
                if (kernel_termp >= double_min) {
                    tfqijd += kernel_termp
                    occurqijd += 1
                }
            }
        if (occurqijd != 0){
          ndqij += tfqijd / occurqijd
        }
        TFqijd = (k1 + 1) * tfqijd / (K + tfqijd)
        IDFqijd = log2((docs_num - ndqij + 0.5)/ (ndqij + 0.5))
        QTF = (k3 + 1) * qtf_kernel / (k3 + qtf_kernel)
        score_crter += TFqijd * IDFqijd * QTF
        }
    }
    // for BM25                                            
    df = _index[field][term].df()         
    TF = (k1 + 1) * tfqid / (K + tfqid)           
    IDF = log2((docs_num - df + 0.5) / (df + 0.5))  
    QTF = (k3 + 1) * qtf[qi] / (k3 + qtf[qi])    
    score_bm25 += TF * IDF * QTF
};
// score_bm25和score_crter均需要归一化，采用zmin-zmax归一化，但是由于这里获取不到最大值，所以计划max和min用经验值代替，就是用很多词先跑一次，得出所有中的最大值和最小值，然后作为这里的归一化参数
score = score_bm25 * (1 -rate) + score_crter * rate
return score;
def my_kernel(distfromcenter, sigma){
    if (distfromcenter > sigma){
        return 0.0
    }
    else{
        return 1.0 - distfromcenter / sigma
    }
}

def intersect(distbetween){
    return distbetween / 2.0
}

def log2(num){
    return log(num) / log(2)
}
""",
                "params": {
                  "terms": ["强奸","判刑"]
                }
            }
        }
    }
  },
  "_source": ["title", "all"]
}
```
## 3. 如何使用
### 3.1 将脚本写成文件，加入到集群
groovy运行有风险，所以一般禁止groovy在inline模式运行，防止用户在inline中使用恶意代码，所以groovy脚本写完之后需要放入到集群的scripts中。放入需要以下几个步骤：
- 脚本名写成`your_name.groovy`, 其中your_name可以自定义，这个其实就是script_id, 后缀为脚本使用的语言
- 脚本放入**集群每一个节点**的$ELASTIC_HOME/config/scripts中，节点会自动扫描编译脚本
>The scripts directory will be rescanned every 60s (configurable with the resource.reload.interval setting) and new, changed, or removed scripts will be compiled, updated, or deleted from the script cache.
Script reloading can be completely disabled by setting script.auto_reload_enabled to false.

### 3.2 完全使用脚本评分
每个节点扫描完后，使用方式：
```
GET lawplatform_test/_search
{
  "query": {
    "script": {
      "script": {
        "lang": "groovy",
        "file": "your_name",
        "params": {
          "your_params": "balabala"
        }
      }
    }
  },
  "_source": "title"
}
```
### 3.3 使用脚本作为二次评分（如果用做二次评分，则不能再使用sort）
需要注意的是，如果纯粹用脚本评分来查询，有可能造成两个问题：
- 效率过低，groovy比painless效率低4,5倍，painless又比java低一些
- 查询获得的文档数会有误差，因为在评分脚本里面不好控制哪些文档不需要返回
所以，评分脚本一般用户使用es查询之后的重评分
```
GET lawplatform_test/_search
{
  "query": {
    "match": {
      "all": "盗窃 抢劫"
    }
  },
  "rescore": {
    "window_size": 100,
    "query":{
      "query_weight": 0,  # 默认重评分和原始查询的分数相加（可以修改为去平均值，相乘等操作），得到文档的真正评分，这里weight设置为0表示不需要原始查询的得分，还有一个参数"rescore_query_weight"表示脚本得分的权重
      "rescore_query":{
        "function_score":{
          "script_score": {
            "script": {
              "lang": "groovy",
              "file": "crter_demo",
              "params": {
                "terms": ["盗窃","抢劫"],
                "boosts": [1, 1],
                "fields": ["all"],
                "qtf": [1, 1]
               }
            }
          }
        }
      }
    }
  }, 
  "_source": "title"
}
```

有的时候，我们会需要显示查询结果，并使页面（page）上靠前文档的顺序能受一些额外的规则控制。但遗憾的是，我们并不能通过二次评分来实现。也许读者会第一时间想到window_size参数，然而实际上这个参数与返回列表中靠前文档并无关联，它只是指定了每个分片应该返回的文档数。除此之外，window_size不能小于页面大小（如果小于页面大小，ElasticSearch会不予警告地将window_size设置为页面大小）。另外，有个非常重要的事实：二次评分功能并**不能与排序一起使用**，这是因为排序发生在二次评分之前，所以排序没有考虑后续新计算出来的文档得分。
前面提到的各种二次评分功能的限制和缺陷（例如，对返回文档的前3个文档及后续的5个文档分别采用不同的二次评分定义）可能限制了该功能的使用，因而用户在使用之前应三思。[《深入理解ElasticSearch》——2.3　二次评分](https://yq.aliyun.com/articles/107938)

### 3.3 inline查询
```
GET /lawplatform_tmp/criminal_law_tmp/_search
{
  "query": {
    "function_score": {
        "script_score": {
            "script": {
                "lang": "groovy",
                "inline": """
score=0
b = 0.35 
k1 = 1.2
k3 = 8.0
rate = 0.045
sigma = 20
occurqijd=0
min_score = -10000
double_min=4.9e-324
tfqijd=0
score_crter=0
score_bm25=0

field_size = fields.size()
for (field_id=0; field_id<field_size; field_id++){
  field = fields[field_id]
  terms = terms_list[field_id]
  boosts = boosts_list[field_id]
  qtf = qtf_list[field_id]
  terms_size = terms.size()
  docs_num = _index[field].docCount()
  field_length = doc[field + '.length' ].value
  avg_field_length = _index[field].sumttf() / docs_num
  K = k1 * ((1 - b) + b * field_length/ avg_field_length)
  for(qi=0;qi<terms_size; qi++){
      term=terms[qi]
      term_info=_index[field].get(term, _FREQUENCIES | _POSITIONS | _CACHE)
      tfqid = term_info.tf()
      if (tfqid == 0){
          continue
      }
      term_info_list = []
      for (pos in term_info){
        term_info_list.add(pos.position)
      }
      for (qj=qi+1; qj<terms_size; qj++){
          term_next = terms[qj]
          term_next_info = _index[field].get(term_next, _FREQUENCIES | _POSITIONS | _CACHE)
          ndqij = 0 
          tfqjd = term_next_info.tf()
          if (tfqjd == 0)
              continue
          term_next_info_list = []
          for (pos in term_next_info){
            term_next_info_list.add(pos.position)
          }
          withinqij = min(qtf[qi], qtf[qj])
          qtf_kernel = my_kernel(intersect(1), sigma) * withinqij
          for (pos_term in term_info_list) {
              position_k = pos_term
              for (pos_term_next in term_next_info_list) {
                  position_k2 = pos_term_next
                  kernel_termp = my_kernel(intersect(1) * abs(position_k - position_k2), sigma)
                  if (kernel_termp >= double_min) {
                      tfqijd += kernel_termp
                      occurqijd += 1
                  }
              }
          if (occurqijd != 0){
            ndqij += tfqijd / occurqijd
          }
          TFqijd = (k1 + 1) * tfqijd / (K + tfqijd)
          IDFqijd = log2((docs_num - ndqij + 0.5)/ (ndqij + 0.5))
          QTF = (k3 + 1) * qtf_kernel / (k3 + qtf_kernel)
          boost = min(boosts[qi], boosts[qj])
          score_crter += TFqijd * IDFqijd * QTF * boost
          }
      }
      // for BM25                                            
      df = _index[field][term].df()         
      TF = (k1 + 1) * tfqid / (K + tfqid)           
      IDF = log2((docs_num - df + 0.5) / (df + 0.5))  
      QTF = (k3 + 1) * qtf[qi] / (k3 + qtf[qi])    
      score_bm25 += TF * IDF * QTF * boosts[qi]
  };
}
score = score_bm25 * (1 -rate) + score_crter * rate
return score;

def my_kernel(distfromcenter, sigma){
    if (distfromcenter > sigma){
        return 0.0
    }
    else{
        return 1.0 - distfromcenter / sigma
    }
}

def intersect(distbetween){
    return distbetween / 2.0
}

def log2(num){
    return log(num) / log(2)
}
""",
                "params": {
                  "terms_list": [["盗窃","抢劫"], ["盗窃"]],
                  "boosts_list": [[1, 1],[1]],
                  "fields": ["all", "title"],
                  "qtf_list": [[1, 1],[1]]
                }
            }
        }
    }
  },
  "_source": ["title"]
}
```

## 4. 对比查询效果
这里使用3中效果对比，纯脚本评分，原始查询，原始查询后进行二次评分
```
# 纯脚本评分
GET /lawplatform2/_search
{
  "query": {
    "function_score": {
      "functions": [
        {"script_score": {
          "script": {
            "lang": "groovy",
            "file": "crter_demo",
            "params": {
              "terms": ["盗窃","抢劫"],
              "boosts": [1, 1],
              "fields": ["all"],
              "qtf": [1, 1]
          }
        }}
        }
      ]
    }
  },
  "_source": "title"
}
# 原始查询，也就是ES的bm25
GET /lawplatform2/_search
{
  "query": {
    "match": {
      "all": "盗窃 抢劫"
    }
  },
  "_source": ["title", "trial_request"]
}
# 原始查询后进行二次评分
GET lawplatform2/_search
{
  "query": {
    "match": {
      "all": "盗窃 抢劫"
    }
  },
  "rescore": {
    "window_size": 100,
    "query":{
      "query_weight": 0,    
      "rescore_query":{
        "function_score":{
          "script_score": {
            "script": {
              "lang": "groovy",
              "file": "crter_demo",
              "params": {
                "terms": ["强奸","抢劫"],
                "boosts": [1, 1],
                "fields": ["all"],
                "qtf": [1, 1]
               }
            }
          }
        }
      }
    }
  }, 
  "_source": "title"
}
```

---
参考：
- [0][ How to use scripts](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html)
- [1][Advanced text scoring in scripts](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-advanced-scripting.html)
- [2][Token count datatype](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/token-count.html)
- [3][Accessing document fields and special variables](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-scripting-fields.html)
- [4][Groovy Scripting Language](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-scripting-groovy.html)
- [5][How to use scripts](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-scripting-using.html)
- [6][Function Score Query](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/query-dsl-function-score-query.html)
- [7][Painless Scripting in Elasticsearch](https://qbox.io/blog/how-to-painless-scripting-in-elasticsearch)
- [8][精通 Groovy](https://www.ibm.com/developerworks/cn/education/java/j-groovy/j-groovy.html)
- [9][groovy w3c教程](https://www.w3cschool.cn/groovy/groovy_overview.html)

- [1][基于Elasticsearch的自定义评分算法扩展 (转)](http://www.cnblogs.com/lwj--/p/6601663.html)
- [2][Elasticsearch使用rescore重打分机制](http://www.jianshu.com/p/98876ed6f27c)  ,主要是rescore相关
- [3][Appendix A. Painless API Reference](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/painless-api-reference.html)
- [4][groovy的Elasticsearch语法说明](https://www.packtpub.com/books/content/extending-elasticsearch-scripting)
- [5][高级文本评分脚本](http://cwiki.apachecn.org/pages/viewpage.action?pageId=9405381)

和Elasticsearch脚本评分、重评分机制

附件: