spark-sql-SparkStrategy-InMemoryScans.adoc

InMemoryScans Execution Planning Strategy

InMemoryScans is an execution planning strategy that plans InMemoryRelation logical operators to InMemoryTableScanExec physical operators.

val spark: SparkSession = ...
// query uses InMemoryRelation logical operator
val q = spark.range(5).cache
val plan = q.queryExecution.optimizedPlan
scala> println(plan.numberedTreeString)
00 InMemoryRelation [id#208L], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
01    +- *Range (0, 5, step=1, splits=8)

// InMemoryScans is an internal class of SparkStrategies
import spark.sessionState.planner.InMemoryScans
val physicalPlan = InMemoryScans.apply(plan).head
scala> println(physicalPlan.numberedTreeString)
00 InMemoryTableScan [id#208L]
01    +- InMemoryRelation [id#208L], true, 10000, StorageLevel(disk, memory, deserialized, 1 replicas)
02          +- *Range (0, 5, step=1, splits=8)

InMemoryScans is part of the standard execution planning strategies of SparkPlanner.

Applying InMemoryScans Strategy to Logical Plan (Executing InMemoryScans) — apply Method

apply(plan: LogicalPlan): Seq[SparkPlan]

Note	apply is part of GenericStrategy Contract to generate a collection of SparkPlans for a given logical plan.

apply requests PhysicalOperation extractor to destructure the input logical plan to a InMemoryRelation logical operator.

In the end, apply pruneFilterProject with a new InMemoryTableScanExec physical operator.