spark-sql-LogicalPlan-LogicalRDD.adoc

LogicalRDD — Logical Scan Over RDD

LogicalRDD is a leaf logical operator with MultiInstanceRelation support for a logical representation of a scan over RDD of internal binary rows.

LogicalRDD is created when:

• Dataset is requested to checkpoint

• SparkSession is requested to create a DataFrame from an RDD of internal binary rows

Note	LogicalRDD is resolved to RDDScanExec when BasicOperators execution planning strategy is executed.

newInstance Method

newInstance(): LogicalRDD.this.type

Note	newInstance is part of MultiInstanceRelation Contract to…​FIXME.

newInstance…​FIXME

Computing Statistics — computeStats Method

computeStats(): Statistics

Note	computeStats is part of LeafNode Contract to compute statistics for cost-based optimizer.

computeStats…​FIXME

Creating LogicalRDD Instance

LogicalRDD takes the following when created:

• Output schema attributes

• RDD of internal binary rows

• Partitioning

• Output ordering (SortOrder)

• isStreaming flag

• SparkSession