Skip to content

Commit

Permalink
🔧 Test(FileDataSourceReader): Add tests for FileDataSourceReader class
Browse files Browse the repository at this point in the history
  • Loading branch information
@camemre49
camemre49 committed Sep 10, 2024
1 parent 30828c3 commit 77c5431
Show file tree
Hide file tree
Showing 22 changed files with 573 additions and 0 deletions.
5 changes: 5 additions & 0 deletions ...-engine/src/test/resources/file-data-source-reader-test-data/folder-test/csv/patient1.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
! This line is a comment to test spark options
pid,gender,birthDate
p1,male,2000-05-10
p2,male,1985-05-08
p3,male,1997-02
5 changes: 5 additions & 0 deletions ...-engine/src/test/resources/file-data-source-reader-test-data/folder-test/csv/patient2.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
! This line is a comment to test spark options
pid,gender,birthDate
p4,male,1999-06-05
p5,male,1965-10-01
p6,female,1991-03
5 changes: 5 additions & 0 deletions ...-engine/src/test/resources/file-data-source-reader-test-data/folder-test/csv/patient3.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
! This line is a comment to test spark options
pid,gender,birthDate
p7,female,1972-10-25
p8,female,2010-01-10
p9,female,1999-05-12
3 changes: 3 additions & 0 deletions ...ngine/src/test/resources/file-data-source-reader-test-data/folder-test/json/patient1.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"pid": "p1", "gender": "male", "birthDate": "2000-05-10"},
{"pid": "p2", "gender": "male", "birthDate": "1985-05-08"},
{"pid": "p3", "gender": "male", "birthDate": "1997-02"}
3 changes: 3 additions & 0 deletions ...ngine/src/test/resources/file-data-source-reader-test-data/folder-test/json/patient2.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"pid": "p4", "gender": "male", "birthDate": "1999-06-05"},
{"pid": "p5", "gender": "male", "birthDate": "1965-10-01"},
{"pid": "p6", "gender": "female", "birthDate": "1991-03"}
3 changes: 3 additions & 0 deletions ...ngine/src/test/resources/file-data-source-reader-test-data/folder-test/json/patient3.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"pid": "p7", "gender": "female", "birthDate": "1972-10-25"},
{"pid": "p8", "gender": "female", "birthDate": "2010-01-10"},
{"pid": "p9", "gender": "female", "birthDate": "1999-05-12"}
Binary file added ...rc/test/resources/file-data-source-reader-test-data/folder-test/parquet/patients1.parquet
View file
Binary file not shown.
Binary file added ...rc/test/resources/file-data-source-reader-test-data/folder-test/parquet/patients2.parquet
View file
Binary file not shown.
Binary file added ...rc/test/resources/file-data-source-reader-test-data/folder-test/parquet/patients3.parquet
View file
Binary file not shown.
4 changes: 4 additions & 0 deletions ...ine/src/test/resources/file-data-source-reader-test-data/folder-test/txt-csv/patient1.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
pid,gender,birthDate
p1,male,2000-05-10
p2,male,1985-05-08
p3,male,1997-02
4 changes: 4 additions & 0 deletions ...ine/src/test/resources/file-data-source-reader-test-data/folder-test/txt-csv/patient2.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
pid,gender,birthDate
p4,male,1999-06-05
p5,male,1965-10-01
p6,female,1991-03
4 changes: 4 additions & 0 deletions ...ine/src/test/resources/file-data-source-reader-test-data/folder-test/txt-csv/patient3.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
pid,gender,birthDate
p7,female,1972-10-25
p8,female,2010-01-10
p9,female,1999-05-12
3 changes: 3 additions & 0 deletions .../src/test/resources/file-data-source-reader-test-data/folder-test/txt-ndjson/patient1.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"pid": "p1", "gender": "male", "birthDate": "2000-05-10"}
{"pid": "p2", "gender": "male", "birthDate": "1985-05-08"}
{"pid": "p3", "gender": "male", "birthDate": "1997-02"}
3 changes: 3 additions & 0 deletions .../src/test/resources/file-data-source-reader-test-data/folder-test/txt-ndjson/patient2.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"pid": "p4", "gender": "male", "birthDate": "1999-06-05"}
{"pid": "p5", "gender": "male", "birthDate": "1965-10-01"}
{"pid": "p6", "gender": "female", "birthDate": "1991-03"}
3 changes: 3 additions & 0 deletions .../src/test/resources/file-data-source-reader-test-data/folder-test/txt-ndjson/patient3.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"pid": "p7", "gender": "female", "birthDate": "1972-10-25"}
{"pid": "p8", "gender": "female", "birthDate": "2010-01-10"}
{"pid": "p9", "gender": "female", "birthDate": "1999-05-12"}
11 changes: 11 additions & 0 deletions ...src/test/resources/file-data-source-reader-test-data/single-file-test/patients-ndjson.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{ "pid": "p1", "gender": "male", "birthDate": "2000-05-10", "deceasedDateTime": null, "homePostalCode": null }
{ "pid": "p2", "gender": "male", "birthDate": "1985-05-08", "deceasedDateTime": "2017-03-10", "homePostalCode": "G02547" }
{ "pid": "p3", "gender": "male", "birthDate": "1997-02", "deceasedDateTime": null, "homePostalCode": null }
{ "pid": "p4", "gender": "male", "birthDate": "1999-06-05", "deceasedDateTime": null, "homePostalCode": "H10564" }
{ "pid": "p5", "gender": "male", "birthDate": "1965-10-01", "deceasedDateTime": "2019-04-21", "homePostalCode": "G02547" }
{ "pid": "p6", "gender": "female", "birthDate": "1991-03", "deceasedDateTime": null, "homePostalCode": null }
/* This line is a comment to test spark options */
{ "pid": "p7", "gender": "female", "birthDate": "1972-10-25", "deceasedDateTime": null, "homePostalCode": "V13135" }
{ "pid": "p8", "gender": "female", "birthDate": "2010-01-10", "deceasedDateTime": null, "homePostalCode": "Z54564" }
{ "pid": "p9", "gender": "female", "birthDate": "1999-05-12", "deceasedDateTime": null, "homePostalCode": null }
{ "pid": "p10", "gender": "female", "birthDate": "2003-11", "deceasedDateTime": null, "homePostalCode": null }
12 changes: 12 additions & 0 deletions ...engine/src/test/resources/file-data-source-reader-test-data/single-file-test/patients.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
! This file has trailing spaces for testing purposes
pid,gender,birthDate,deceasedDateTime ,homePostalCode
p1 ,male,2000-05-10,,
p2,male,1985-05-08,2017-03-10,G02547
p3,male,1997-02,,
p4,male,1999-06-05,,H10564
p5,male,1965-10-01,2019-04-21,G02547
p6,female ,1991-03,,
p7 ,female,1972-10-25,,V13135
p8,female,2010-01-10,,Z54564
p9,female,1999-05-12,,
p10,female,2003-11 ,,
11 changes: 11 additions & 0 deletions ...ngine/src/test/resources/file-data-source-reader-test-data/single-file-test/patients.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{ "pid": "p1", "gender": "male", "birthDate": "2000-05-10", "deceasedDateTime": null, "homePostalCode": null },
{ "pid": "p2", "gender": "male", "birthDate": "1985-05-08", "deceasedDateTime": "2017-03-10", "homePostalCode": "G02547" },
{ "pid": "p3", "gender": "male", "birthDate": "1997-02", "deceasedDateTime": null, "homePostalCode": null },
{ "pid": "p4", "gender": "male", "birthDate": "1999-06-05", "deceasedDateTime": null, "homePostalCode": "H10564" },
{ "pid": "p5", "gender": "male", "birthDate": "1965-10-01", "deceasedDateTime": "2019-04-21", "homePostalCode": "G02547" },
{ "pid": "p6", "gender": "female", "birthDate": "1991-03", "deceasedDateTime": null, "homePostalCode": null },
/* This line is a comment to test spark options */
{ "pid": "p7", "gender": "female", "birthDate": "1972-10-25", "deceasedDateTime": null, "homePostalCode": "V13135" },
{ "pid": "p8", "gender": "female", "birthDate": "2010-01-10", "deceasedDateTime": null, "homePostalCode": "Z54564" },
{ "pid": "p9", "gender": "female", "birthDate": "1999-05-12", "deceasedDateTime": null, "homePostalCode": null },
{ "pid": "p10", "gender": "female", "birthDate": "2003-11", "deceasedDateTime": null, "homePostalCode": null }
Binary file added ...ne/src/test/resources/file-data-source-reader-test-data/single-file-test/patients.parquet
View file
Binary file not shown.
11 changes: 11 additions & 0 deletions ...engine/src/test/resources/file-data-source-reader-test-data/single-file-test/patients.tsv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
pid gender birthDate deceasedDateTime homePostalCode
p1 male 2000-05-10
p2 male 1985-05-08 2017-03-10 G02547
p3 male 1997-02
p4 male 1999-06-05 H10564
p5 male 1965-10-01 2019-04-21 G02547
p6 female 1991-03
p7 female 1972-10-25 V13135
p8 female 2010-01-10 Z54564
p9 female 1999-05-12
p10 female 2003-11
12 changes: 12 additions & 0 deletions ...engine/src/test/resources/file-data-source-reader-test-data/single-file-test/patients.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
! This file has trailing spaces for testing purposes
pid,gender,birthDate,deceasedDateTime ,homePostalCode
p1 ,male,2000-05-10,,
p2,male,1985-05-08,2017-03-10,G02547
p3,male,1997-02,,
p4,male,1999-06-05,,H10564
p5,male,1965-10-01,2019-04-21,G02547
p6,female ,1991-03,,
p7 ,female,1972-10-25,,V13135
p8,female,2010-01-10,,Z54564
p9,female,1999-05-12,,
p10,female,2003-11 ,,
Loading

0 comments on commit 77c5431

Please sign in to comment.