Skip to content

Latest commit

 

History

History
22 lines (14 loc) · 1.37 KB

spark-sql-vectorized-parquet-reader.adoc

File metadata and controls

22 lines (14 loc) · 1.37 KB

Vectorized Parquet Reader

Vectorized Parquet Reader (aka Vectorized Parquet Decoding) allows for reading datasets in parquet format in batches, i.e. rows are decoded in batches. That aims at improving memory locality and cache utilization.

The parquet encodings are largely designed to decode faster in batches, column by column. This can speed up the decoding considerably.

VectorizedParquetRecordReader is responsible for vectorized decoding and is used only when spark.sql.parquet.enableVectorizedReader configuration property is enabled and the result schema uses AtomicType data types only.

spark.sql.parquet.enableVectorizedReader Configuration Property

spark.sql.parquet.enableVectorizedReader configuration property is on by default.

scala> spark.version
res0: String = 2.3.0

val isParquetVectorizedReaderEnabled = spark.conf.get("spark.sql.parquet.enableVectorizedReader").toBoolean
assert(isParquetVectorizedReaderEnabled, "spark.sql.parquet.enableVectorizedReader should be enabled by default")