Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update VirtualTable to use an expression for defining table records #786

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions proto/substrait/algebra.proto
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,29 @@ message ReadRel {
}
}

// Wrapper for repeated literal struct records
message LiteralRecords {
repeated Expression.Literal.Struct values = 1;
}

// Wrapper for repeated nested struct expressions
message NestedRecords {
repeated Expression.Nested.Struct expressions = 1;
}

// A table composed of expressions.
message VirtualTable {
repeated Expression.Literal.Struct values = 1 [deprecated = true];
repeated Expression.Nested.Struct expressions = 2;
oneof records_type {
LiteralRecords values = 1 [deprecated = true];
// Represents table data as a list of nested struct expressions.
// Each struct corresponds to a record, with expressions defining column values.
NestedRecords expressions = 2;
// Represents table data as an expression that should evaluate to a list of structs,
// i.e., its type is LIST<STRUCT<T1,...,Tn>>.
// This approach provides flexibility, allowing the use of a dynamic parameter
// to define the virtual table's content.
Expression record_list_expression = 3;
}
}

// A stub type that can be used to extend/introduce new table types outside
Expand Down
19 changes: 12 additions & 7 deletions site/docs/relations/logical_relations.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,15 +50,20 @@ Read definition types (like the rest of the features in Substrait) are built by

#### Virtual Table

A virtual table is a table whose contents are embedded in the plan itself. The table data
is encoded as records consisting of literal values or expressions that can be resolved without referencing any input data.
For example, a literal, a function call involving literals, or any other expression that does
not require input.
A virtual table is a table whose contents are embedded within the plan itself.
The table data can be represented in one of two ways:

| Property | Description | Required |
| -------- | ----------- | -------- |
| Data | Required | Required |
* As explicit records - a list of struct-based expressions defining each record directly.
* As a single expression - an expression that evaluates into a list of records, with the type `LIST&lt;STRUCT&lt;T1,...,Tn&gt;&gt;`.

The records should consist of literal values or expressions that can be resolved without referencing any input data.
For example, a literal, a function call involving literals, or any other expression that does not require input.
Encoding the data as an expression using nested types provides flexibility, allowing the use of a dynamic parameter
expression as a single expression for the virtual table.

| Property | Description | Required |
| ---------| ----------------- | -------- |
| Data | A list of records | Required |

#### Named Table

Expand Down
Loading