8wDlpd.png
8wDFp9.png
8wDEOx.png
8wDMfH.png
8wDKte.png

催化剂规则返回错误的逻辑计划

kluszon 2月前

12 0

def apply(计划:LogicalPlan):LogicalPlan = {计划转换{case unresolvedRelation:UnresolvedRelation => val tblSchemaName:Array [String] = unresolvedRelation.tableName.sp ...

  def apply(plan: LogicalPlan): LogicalPlan = {
    plan transform {
      case unresolvedRelation: UnresolvedRelation =>
        val tblSchemaName: Array[String] = unresolvedRelation.tableName.split("\\.")
        if (tblSchemaName.length == 1) return plan

        val schema = tblSchemaName.apply(0)
        val tblName = tblSchemaName.apply(1)
        for (ref <- this.refs) {
          if (tblName == ref.nqName) {
            return unresolvedRelation.copy(multipartIdentifier = Seq(schema.toUpperCase, tblName.toUpperCase), unresolvedRelation.options, unresolvedRelation.isStreaming)
          }
        }
        unresolvedRelation

      case unresolvedWith: UnresolvedWith =>
        val newCteRelations = unresolvedWith.cteRelations.map {
          case (aliasName, subqueryAlias) =>
            val newSubqueryAlias = apply(subqueryAlias).asInstanceOf[SubqueryAlias]
            (aliasName, newSubqueryAlias)
        }

        val modifiedCTEUnresolvedWith = unresolvedWith.copy(cteRelations = newCteRelations)
        modifiedCTEUnresolvedWith
        
      case otherPlan: LogicalPlan =>
        val newChildren = otherPlan.children.map(child => apply(child))
        val modified_plan = otherPlan.withNewChildren(newChildren)

        modified_plan
    }
  }
}

rules += UpperCaseTableRule(refs)

val optimizer = new RuleExecutor[LogicalPlan] {
    val batches = Seq(
      Batch("Rewrite", Once, rules.toList: _*)
    )
}

val logicalPlanRewrite = optimizer.execute(logicalPlan)

通过上面的代码,我想要实现的是如果表在列表中则将其名称大写。

代码可以使用select语句,但是当涉及到CTE时,unresolvedWith可以返回正确的CTE,但是优化器执行后的logicalPlanRewrite仍然是小写的。

例如

SELECT * FROM schema.t1 a JOIN schema.t2 b ON a.id = b.id

我可以将 SCHEMA.T1 和 SCHEMA.T2 大写

但对于 CTE

WITH x AS (SELECT * FROM schema.t1),y AS (SELECT * FROM schema.t2) " SELECT x.id, y.date FROM x JOIN y ON x.id = y.id

我可以看到代码进入了unresolvedWith,然后变成了SCHEMA.T1和SCHEMA.T2

但是当 optimizer.execute 返回时,它会恢复为小写。

简而言之,它只在 select 语句中有效,而不适用于 CTE。

有人能帮助我理解为什么吗?

帖子版权声明 1、本帖标题:催化剂规则返回错误的逻辑计划
    本站网址:http://xjnalaquan.com/
2、本网站的资源部分来源于网络,如有侵权,请联系站长进行删除处理。
3、会员发帖仅代表会员个人观点,并不代表本站赞同其观点和对其真实性负责。
4、本站一律禁止以任何方式发布或转载任何违法的相关信息,访客发现请向站长举报
5、站长邮箱:yeweds@126.com 除非注明,本帖由kluszon在本站《apache-spark》版块原创发布, 转载请注明出处!
最新回复 (0)
返回
作者最近主题: