Skip to content

Commit

Permalink
Merge pull request #6 from caerbannogwhite/preludio-sort-and-group
Browse files Browse the repository at this point in the history 
Preludio sort and group
  • Loading branch information
@caerbannogwhite
caerbannogwhite authored Aug 20, 2023
2 parents a102498 + 55a4df0 commit c4600e0
Show file tree
Hide file tree
Showing 40 changed files with 2,836 additions and 2,253 deletions.
35 changes: 24 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@

![](media/logo_med.png)

# Preludio

### A PRQL based data transformation language

Preludio is a data transformation language based on PRQL. It is a language that allows you to transform and
manipulate data in a simple and intuitive way, batteries included.

No libraries or external dependencies are required to run the language.

### Example

This is a simple example of what you can already do with Preludio.
It reads a CSV file, derives two new columns, selects some columns and writes the result to a new CSV file.

Expand All @@ -18,6 +19,7 @@ let clean = (
readCSV "test_files\\Cars.csv" delimiter: ";" header:true
strReplace [MPG, Displacement, Horsepower, Acceleration] old:"," new:"."
asFloat [MPG, Displacement, Horsepower, Acceleration]
orderBy [-Origin, Cylinders, -MPG]
)
let europe5Cylinders = (
Expand All @@ -41,25 +43,29 @@ let europe5Cylinders = (
![](media/repl_example.gif)

### Features
- [x] Arithmetic and logical operators
- [x] Read and write CSV files
- [x] Derive new columns
- [x] Select columns
- [x] Filter rows
- [ ] Sort rows
- [ ] Group by and aggregate
- [ ] Join tables

- [x] Arithmetic and logical operators
- [x] Read and write CSV files
- [x] Derive new columns
- [x] Select columns
- [x] Filter rows
- [x] Sort rows
- [ ] Group by and aggregate
- [ ] Join tables

### Installation

To run it, you need to have [Go](https://golang.org/doc/install) installed.
Once you have Go, you can clone this repository.

To run the program, you can use the following command:

```bash
go run .
```

### Future Features

- [x] Move to [Gandalff](https://github.com/caerbannogwhite/preludio/tree/main/core/gandalff) library
- [ ] Add statistical functions
- [ ] Add support for Excel files
Expand All @@ -80,16 +86,23 @@ In case the language becomes quite successful, I will consider adding:
- [ ] Integration with OpenAI (https://openai.com/blog/openai-api/), ie. image to table

### Contributing

If you want to contribute to this project, you can do so by forking the repository and submitting a pull request.

### Developers

If the grammar is changed, the parser must be regenerated. To do this, run the following command:

(on Windows)

```
make.ps1
```

### Log
- **2 / 08 / 2023** Preludio is now using the Gandalff library for managing data.
- **21 / 03 / 2023** First publishing of the repository. Many things are still not working.

- **20 / 08 / 2023** After exactly one year from the first commit, Preludio is fairly stable and usable. The language is still missing a few core features (like `join` and aggregators, already supported by Gandalff), but it is already possible to perform many operations with it.
- **02 / 08 / 2023** Preludio is now using the Gandalff library for managing data.
- **21 / 03 / 2023** First publishing of the repository. Many things are still not working.
- **18 / 03 / 2023** Gandalff library: fist commit.
- **20 / 08 / 2022** Preludio: fist commit.
1 change: 0 additions & 1 deletion core/bytefeeder/bytefeeder.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,6 @@ func (bf *ByteFeeder) ExitExprUnary(ctx *ExprUnaryContext) {

case "not":
bf.AppendInstruction(typesys.OP_UNARY_NOT, 0, 0)

}
}
}
Expand Down
124 changes: 93 additions & 31 deletions core/full_language_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package preludiocore
import (
"bytefeeder"
"gandalff"
"math"
"os"
"testing"
)
Expand All @@ -17,144 +18,205 @@ func Test_Expressions(t *testing.T) {

bytecode, _, _ = bytefeeder.CompileSource(`true`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, true); err != nil {
if err = checkCurrentResult(be, true); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`false`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, false); err != nil {
if err = checkCurrentResult(be, false); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true * false`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, int64(0)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true / false`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, math.Inf(1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true % false`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, math.NaN()); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true ** false`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, int64(1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true + false`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(1)); err != nil {
if err = checkCurrentResult(be, int64(1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true - false`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, int64(1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true and false`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, false); err != nil {
if err = checkCurrentResult(be, false); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true or false`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, true); err != nil {
if err = checkCurrentResult(be, true); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`not true`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, false); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true or (false and true)`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, true); err != nil {
if err = checkCurrentResult(be, true); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`true or not false and true or not true`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, true); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1 * 5`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(5)); err != nil {
if err = checkCurrentResult(be, int64(5)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1 / 3`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(0.3333333333333333)); err != nil {
if err = checkCurrentResult(be, float64(0.3333333333333333)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`4682 % 427`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(412)); err != nil {
if err = checkCurrentResult(be, float64(412)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`3 ** 4`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(81)); err != nil {
if err = checkCurrentResult(be, int64(81)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`2 ** (2 + 1 * 2)`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(16)); err != nil {
if err = checkCurrentResult(be, int64(16)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1 - 2`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(-1)); err != nil {
if err = checkCurrentResult(be, int64(-1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1 + 2`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(3)); err != nil {
if err = checkCurrentResult(be, int64(3)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`+1 + 2`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, int64(3)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`-1`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, int64(-1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`-1 + 2`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, int64(1)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`-1.0 - 2`)
be.RunBytecode(bytecode)
if err = checkCurrentResult(be, float64(-3)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1.325235e-3 * 5`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(0.006626175)); err != nil {
if err = checkCurrentResult(be, float64(0.006626175)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1.325235e-3 / 3`)
be.RunBytecode(bytecode)

if be.__currentResult == nil {
t.Error("Expected result, got nil")
} else if be.__currentResult.isFloat64Scalar() == false {
t.Error("Expected float scalar, got", be.__currentResult)
} else if f, err := be.__currentResult.getFloat64Scalar(); err != nil || f != 0.00044174499999999995 {
t.Error("Expected 0.00044174499999999995, got", f, err)
if err = checkCurrentResult(be, float64(0.00044174499999999995)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`"hello" + "world"`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, "helloworld"); err != nil {
if err = checkCurrentResult(be, "helloworld"); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1 + 2 * 3 - 4 + 5 * 6`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(33)); err != nil {
if err = checkCurrentResult(be, int64(33)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1 + 2 * 3 - 4 + 5 * 6 % 7 + "hello"`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, "5hello"); err != nil {
if err = checkCurrentResult(be, "5hello"); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`3.4 + 2.3 * 3.2 - 4.1 + 5.0 * 6.9`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(41.16)); err != nil {
if err = checkCurrentResult(be, float64(41.16)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`(1 + 2) * (3 - 4) + 5 * 6`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(27)); err != nil {
if err = checkCurrentResult(be, int64(27)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`(1 + (2 * 3)) - (4 + (5 * (6 % 7 + 8))) / ((9) + (10 * 11 - 12 % 13))`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(6.308411214953271)); err != nil {
if err = checkCurrentResult(be, float64(6.308411214953271)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`(1 + (2 * (3 - (4 + (5 * (6 % (7 + (8 - (9 + (10 * (11 - (12 + (13 * (14 % (15 + (16 - (17 + (18 * (19 - (20 + (21 * (22 % (23 + (24 - (25 + (26 * (27 - (28 + (29 * (30 - (31 + (32 * (33 - (34 + (35 * (36 % (37 + (38 - (39 + (40 * (41 - (42 + (43 * (44 % (45 + (46 - (47 + (48 * (49 - (50 + (51 * (52 % (53 + (54 - (55 + (56 * (57 - (58 + (59 * (60 % (61 + (62 - (63 + (64 * (65 - (66 + (67 * (68 % (69 + (70 - (71 + (72 * (73 - (74 + (75 * (76 % (77 + (78 - (79 + (80 * (81 - (82 + (83 * (84 % (85 + (86 - (87 + (88 * (89 - (90 + (91 * (92 % (93 + (94 - (95 + (96 * (97 - (98 + (99 * (100))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(-61.0)); err != nil {
if err = checkCurrentResult(be, float64(-61.0)); err != nil {
t.Error(err)
}

bytecode, _, _ = bytefeeder.CompileSource(`1e30 / 1.000001 / 1.000002 / 1.000003 / 1.000004 / 1.000005 / 1.000006 / 1.000007 / 1.000008 / 1.000009 / 1.000010 / 1.000011 / 1.000012 / 1.000013 / 1.000014 / 1.000015 / 1.000016 / 1.000017 / 1.000018 / 1.000019 / 1.000020 / 1.000021 / 1.000022 / 1.000023 / 1.000024 / 1.000025 / 1.000026 / 1.000027 / 1.000028 / 1.000029 / 1.000030 / 1.000031 / 1.000032 / 1.000033 / 1.000034 / 1.000035 / 1.000036 / 1.000037 / 1.000038 / 1.000039 / 1.000040 / 1.000041 / 1.000042 / 1.000043 / 1.000044 / 1.000045 / 1.000046 / 1.000047 / 1.000048 / 1.000049 / 1.000050 / 1.000051 / 1.000052 / 1.000053 / 1.000054 / 1.000055 / 1.000056 / 1.000057 / 1.000058 / 1.000059 / 1.000060 / 1.000061 / 1.000062 / 1.000063 / 1.000064 / 1.000065 / 1.000066 / 1.000067 / 1.000068 / 1.000069 / 1.000070 / 1.000071 / 1.000072 / 1.000073 / 1.000074 / 1.000075 / 1.000076 / 1.000077 / 1.000078 / 1.000079 / 1.000080 / 1.000081 / 1.000082 / 1.000083 / 1.000084 / 1.000085 / 1.000086 / 1.000087 / 1.000088 / 1.000089 / 1.000090 / 1.000091 / 1.000092 / 1.000093 / 1.000094 / 1.000095 / 1.000096 / 1.000097 / 1.000098 / 1.000099 / 1.000100`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, float64(9.949628981268441e+29)); err != nil {
if err = checkCurrentResult(be, float64(9.949628981268441e+29)); err != nil {
t.Error(err)
}
}
Expand All @@ -172,7 +234,7 @@ func Test_Assignements(t *testing.T) {

bytecode, _, _ = bytefeeder.CompileSource(source)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, int64(33)); err != nil {
if err = checkCurrentResult(be, int64(33)); err != nil {
t.Error(err)
}

Expand All @@ -185,7 +247,7 @@ func Test_Assignements(t *testing.T) {

bytecode, _, _ = bytefeeder.CompileSource(`d * (e * (f * (g * (d * (e * (f * (g)))))))`)
be.RunBytecode(bytecode)
if err = currentResultChecker(be, 1.000000040200004); err != nil {
if err = checkCurrentResult(be, 1.000000040200004); err != nil {
t.Error(err)
}

Expand Down
Loading

0 comments on commit c4600e0

Please sign in to comment.