New feature: Export Oracle SQL script to DBML script

[MartinRucka]

Summary

• Add Oracle exporter for transform Oracle SQL code to DBML code
• Support for CLI
• Exporter Tests

Issue

#440

Lasting Changes (Technical)

(please list down: code changes/things that have wide-effect; new libraries/functions added that can be used by others; examples below)

• Added oracleParser

Checklist

Please check directly on the box once each of these are done

• Documentation (if necessary)
• Tests (integration test/unit test)
• Integration Tests Passed
• Code Review

Summary by CodeRabbit

• New Features
  • Added support for Oracle databases in SQL to DBML conversion, enhancing the tool's versatility.
  • Introduced parsing functionality for Oracle-specific SQL syntax and commands.
• Tests
  • Implemented comprehensive tests for Oracle database importers to ensure reliability and correctness in conversions.
• Documentation
  • Updated documentation to include details and examples of Oracle database handling.

[coderabbitai]

Walkthrough

This update introduces extensive support for Oracle SQL in the DBML project, including parsing, importing, and testing capabilities. New functionalities are added to handle Oracle-specific syntax and schema operations, enhancing the tool's versatility and compatibility with Oracle databases.

Changes

File Path	Change Summary
packages/dbml-cli/src/cli/index.js	Added --oracle option support in sql2dbml function.
packages/dbml-core/__tests__/importer/...	Added tests for Oracle importer and various Oracle SQL schema definitions and operations.
packages/dbml-core/src/parse/...	Expanded parsing capabilities to include Oracle SQL, with support for different commands and syntax.

Possibly related issues

• Add support for imports and exports #468: The enhancements in Oracle SQL support could indirectly contribute to better management of complex schemas by broadening the scope of databases DBML can handle, thus aligning with the issue's objectives of enhancing modularity and schema management.

Poem

🐇 CodeRabbit hops along,
Bringing Oracle where it belongs.
Parsing, testing, all so sleek,
DBML's future, no longer bleak.
Cheers to the devs, hip hooray!
In databases, we now have a say. 🎉

---

Recent Review Details

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between e75d2f7 and b75e5fc.

Files selected for processing (28)

• packages/dbml-cli/src/cli/index.js (1 hunks)
• packages/dbml-core/tests/importer/importer.spec.js (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/input/general_schema.in.sql (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/input/ignored.in.sql (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/input/increment_type.in.sql (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/input/product_order_db.in.sql (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/input/shorthand_syntax.in.sql (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/output/db_dump.out.dbml (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/output/general_schema.out.dbml (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/output/ignored.out.dbml (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/output/increment_type.out.dbml (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/output/product_order_db.out.dbml (1 hunks)
• packages/dbml-core/tests/importer/oracle_importer/output/shorthand_syntax.out.dbml (1 hunks)
• packages/dbml-core/src/parse/Parser.js (3 hunks)
• packages/dbml-core/src/parse/buildParser.js (1 hunks)
• packages/dbml-core/src/parse/oracle/Base_rules.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Alter_table/Alter_table.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Commands.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Comment.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Create_index.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Create_table/Create_table.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Create_table/Create_table_normal.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Commands/Ignore_syntax.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Expression.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/InitializerUtils.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/Keywords.pegjs (1 hunks)
• packages/dbml-core/src/parse/oracle/get_parser.js (1 hunks)
• packages/dbml-core/src/parse/oracle/parser.pegjs (1 hunks)

Files not summarized due to errors (1)

• packages/dbml-core/tests/importer/oracle_importer/output/db_dump.out.dbml: Error: Message exceeds token limit

Additional comments not posted (44)

packages/dbml-core/__tests__/importer/oracle_importer/output/ignored.out.dbml (1)

1-3: The DBML representation of the table with an auto-increment primary key is correctly formatted.

packages/dbml-core/src/parse/oracle/Commands/Create_table/Create_table.pegjs (2)

1-6: The grammar rule for parsing CREATE TABLE statements is correctly defined and structured.

---

8-8: Ensure the import path for 'Create_table_normal.pegjs' is correct and the file exists at the specified location.

Verification successful

---

The verification process confirms that the file Create_table_normal.pegjs exists at the specified location within the packages/dbml-core/src/parse/oracle/Commands/Create_table directory. This aligns with the import statement provided in the code snippet, ensuring that the path is correctly specified and the file is accessible.

Conclusion: The import path for 'Create_table_normal.pegjs' is correct and the file does exist at the specified location.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Check if 'Create_table_normal.pegjs' exists in the expected directory
fd 'Create_table_normal.pegjs' 'packages/dbml-core/src/parse/oracle/Commands/Create_table'

Length of output: 172

packages/dbml-core/src/parse/oracle/get_parser.js (1)

10-10: Consider removing the console log statement if it was used for debugging purposes and is no longer necessary.

packages/dbml-core/__tests__/importer/oracle_importer/output/increment_type.out.dbml (1)

1-19: The DBML representations for the tables are correctly formatted and accurately reflect the Oracle SQL definitions.

packages/dbml-core/src/parse/oracle/Expression.pegjs (1)

1-14: The grammar rules for parsing expressions are correctly defined and structured, using lodash effectively for data manipulation.

packages/dbml-core/src/parse/oracle/Commands/Commands.pegjs (2)

1-5: The grammar rules for dispatching to various SQL command parsers are correctly defined.

---

7-11: Ensure all import paths for the command parsers are correct and the files exist at the specified locations.

Verification successful

---

The review comment requested verification of the existence and correctness of the import paths for various command parser files in the packages/dbml-core/src/parse/oracle/Commands directory and its subdirectories. The shell script results confirm that all specified files exist at the expected locations:

1. Create_table.pegjs is located at packages/dbml-core/src/parse/oracle/Commands/Create_table/Create_table.pegjs.
2. Alter_table.pegjs is located at packages/dbml-core/src/parse/oracle/Commands/Alter_table/Alter_table.pegjs.
3. Create_index.pegjs is located at packages/dbml-core/src/parse/oracle/Commands/Create_index.pegjs.
4. Comment.pegjs is located at packages/dbml-core/src/parse/oracle/Commands/Comment.pegjs.
5. Ignore_syntax.pegjs is located at packages/dbml-core/src/parse/oracle/Commands/Ignore_syntax.pegjs.

All import paths are correct, and the files exist as specified in the review comment. Therefore, the verification is successful.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Check if the imported parser files exist in the expected directories
fd 'Create_table.pegjs' 'packages/dbml-core/src/parse/oracle/Commands'
fd 'Alter_table.pegjs' 'packages/dbml-core/src/parse/oracle/Commands'
fd 'Create_index.pegjs' 'packages/dbml-core/src/parse/oracle'
fd 'Comment.pegjs' 'packages/dbml-core/src/parse/oracle'
fd 'Ignore_syntax.pegjs' 'packages/dbml-core/src/parse/oracle'

Length of output: 652

packages/dbml-core/__tests__/importer/oracle_importer/input/increment_type.in.sql (1)

1-19: The SQL CREATE TABLE statements are correctly formatted and accurately reflect the use of auto-incrementing primary keys in Oracle SQL.

packages/dbml-core/__tests__/importer/oracle_importer/input/ignored.in.sql (1)

1-40: The SQL statements are correctly formatted and serve as appropriate test cases for verifying the parser's ability to ignore specified syntax while focusing on relevant statements for DBML conversion.

packages/dbml-core/__tests__/importer/oracle_importer/output/shorthand_syntax.out.dbml (6)

1-9: The DBML definition for the table "booking_reference" is correctly formatted and includes primary key constraints on the composite of reference_id and cust_id.

---

11-19: The DBML definition for the table "br_flight" is correctly formatted and includes primary key constraints on the composite of reference_id and flight_id.

---

21-30: The DBML definition for the table "users" correctly includes various data types and constraints. The country_code field is linked to the "countries" table, which is a good practice for maintaining referential integrity.

---

32-36: The DBML definition for the table "countries" is concise and correctly formatted, including primary key and standard fields.

---

38-40: The foreign key reference from "users" to "countries" is correctly defined, ensuring referential integrity between these tables.

---

40-40: The foreign key reference for a composite key between "booking_reference" and "br_flight" is correctly defined. This ensures that the relationship between these tables is accurately represented in the DBML output.

packages/dbml-core/src/parse/oracle/Commands/Ignore_syntax.pegjs (1)

1-26: The grammar rules for ignoring specific SQL syntax are well-defined. Each rule captures a different SQL command and returns a structured object with the command name and details. This setup helps in cleanly skipping unsupported or irrelevant SQL commands during the parsing process.

packages/dbml-core/__tests__/importer/oracle_importer/input/shorthand_syntax.in.sql (1)

1-35: The SQL commands for creating tables ("booking_reference", "br_flight", "users", "countries") and defining primary and foreign key constraints are correctly formatted. These commands are essential for setting up the database schema and ensuring data integrity through relationships.

packages/dbml-core/src/parse/oracle/InitializerUtils.pegjs (1)

1-28: The utility functions removeReduntdantSpNewline and findTable are well-implemented. The removeReduntdantSpNewline function efficiently cleans up strings by removing redundant spaces and newlines, which is crucial for consistent parsing. The findTable function correctly handles optional schema names and searches for tables, which is essential for resolving table references in SQL scripts.

packages/dbml-cli/src/cli/index.js (1)

33-33: The addition of the --oracle option to the sql2dbml function in the CLI is correctly implemented. This change allows users to specify Oracle SQL scripts for conversion, enhancing the tool's flexibility and utility.

packages/dbml-core/src/parse/oracle/Commands/Comment.pegjs (1)

1-45: The grammar rules for parsing SQL comments and associating them with database objects are correctly implemented. The rules handle different path lengths to accurately determine the database, schema, and table names, which is crucial for correctly applying comments in the DBML output.

packages/dbml-core/__tests__/importer/oracle_importer/output/product_order_db.out.dbml (1)

1-49: The DBML definitions for the tables "Customer", "Product", "Store", "Order", and "OrderItem" are correctly formatted. The relationships between these tables are also accurately represented, ensuring that the DBML output correctly reflects the database schema.

packages/dbml-core/src/parse/buildParser.js (1)

27-34: The addition of the Oracle parser to the parser building process is correctly implemented. The use of pegjs-require-import to load the parser definition and the integration with bluebird promises ensure that the parser is built efficiently and correctly.

packages/dbml-core/__tests__/importer/importer.spec.js (1)

34-36: LGTM! The new test block for the Oracle importer follows the established pattern and correctly integrates with the existing testing framework.

packages/dbml-core/__tests__/importer/oracle_importer/input/product_order_db.in.sql (1)

7-51: LGTM! The Oracle SQL script is well-formed and covers a comprehensive range of features for testing the Oracle importer.

packages/dbml-core/src/parse/oracle/Commands/Create_index.pegjs (1)

1-47: LGTM! The grammar for parsing CREATE INDEX statements is correctly defined and handles various Oracle SQL specifics effectively.

packages/dbml-core/__tests__/importer/oracle_importer/output/general_schema.out.dbml (1)

1-82: LGTM! The DBML output is correctly formatted and accurately reflects the structure and constraints from the Oracle SQL input.

packages/dbml-core/__tests__/importer/oracle_importer/input/general_schema.in.sql (1)

1-81: LGTM! The Oracle SQL script is well-formed and includes a variety of table definitions and alterations, making it suitable for comprehensive testing of the Oracle importer.

packages/dbml-core/src/parse/oracle/Commands/Alter_table/Alter_table.pegjs (1)

1-95: LGTM! The grammar for parsing ALTER TABLE statements is correctly defined and handles various actions effectively, ensuring accurate parsing and conversion.

packages/dbml-core/src/parse/oracle/Keywords.pegjs (1)

1-128: LGTM! The list of keywords is comprehensive and correctly defined, covering a wide range of Oracle SQL syntax necessary for accurate parsing.

packages/dbml-core/src/parse/Parser.js (2)

50-52: LGTM! The method parseOracleToJSON is correctly implemented and integrates well with the existing parsing framework.

---

113-115: Ensure that the new oracle case in the switch statement is correctly handling Oracle SQL parsing as expected.

packages/dbml-core/src/parse/oracle/Base_rules.pegjs (3)

5-16: Consider adding error handling for unexpected input formats in path_name.

The current implementation assumes a certain structure and length for the names array. It would be beneficial to add error handling or validation to ensure that the input matches expected formats, preventing runtime errors or incorrect parsing results.

---

18-20: Ensure consistency in naming conventions.

The function table_name returns an object with properties dbName and schemaName derived from path_name, and name from identifier. It's important to maintain consistent naming conventions across the codebase. Consider renaming dbName to databaseName to match the verbosity of schemaName.

---

102-114: Validate type_name against a list of known types to prevent incorrect type assignments.

The function type_name constructs a type name based on the input characters and optional arguments. To prevent the assignment of incorrect or unsupported types, consider validating the constructed type_name against a list of known and supported data types.

packages/dbml-core/src/parse/oracle/parser.pegjs (3)

16-23: Consider optimizing the regular expression in removeReduntdantSpNewline.

The current regular expression used in the map function seems overly complex and could be simplified. Additionally, the function name contains a typo ("Reduntdant" should be "Redundant"). Consider renaming the function to removeRedundantSpacesAndNewlines for clarity.

---

39-163: Ensure comprehensive handling of SQL commands in the parser.

The parser handles a variety of SQL commands, but it's crucial to ensure that all intended Oracle SQL features are supported. Consider adding more test cases or reviewing the Oracle SQL documentation to verify that the parser covers all necessary commands and syntax variations.

---

714-728: Check for potential typos in storage parameter names.

Some storage parameter names like "parallel_worlers" might be typos ("workers" was likely intended). Verify these parameter names against the Oracle documentation to ensure accuracy.

packages/dbml-core/__tests__/importer/oracle_importer/output/db_dump.out.dbml (6)

1-7: Ensure consistent use of quotes for default values in DBML.

In the DBML for the department table, the default value for modifieddate is specified without quotes (default: SYSDATE). It's a best practice in DBML to consistently use quotes for default values to avoid parsing errors or inconsistencies, especially when the values are expressions or functions.

---

9-26: Validate data type consistency for salariedflag, vacationhours, sickleavehours, and currentflag.

In the employee table, fields like salariedflag, vacationhours, sickleavehours, and currentflag use a specific data type format NUMBER(1, 0). It's important to ensure that similar fields across different tables use consistent data types for maintainability and to avoid errors during data operations.

---

28-36: Check for potential data integrity issues with nullable enddate.

In the employeedepartmenthistory table, the enddate field is nullable, which is intended to represent current departments by setting it to NULL. Ensure that application logic or database triggers handle this appropriately to maintain data integrity, especially in reports or queries that involve date ranges.

---

38-45: Review the use of SYSTIMESTAMP for default values in timestamps.

For the employeepayhistory table, the default value for modifieddate is SYSTIMESTAMP. If the intention is to capture the exact time including timezone differences, this is appropriate. However, if the application is meant to operate in a single timezone, using CURRENT_TIMESTAMP might be more suitable to avoid potential confusion or errors in time calculations.

---

47-53: Ensure proper handling of special characters in notes.

In the jobcandidate table, the note for the resume field contains special characters (Résumé). Ensure that these characters are correctly encoded or represented in the DBML and the database to prevent issues with display or processing.

---

55-62: Confirm the handling of nullable starttime and endtime in the shift table.

The shift table allows starttime and endtime to be nullable. This design choice should be confirmed to ensure it aligns with business rules, especially in scenarios where shifts might not have fixed start or end times.

Verification successful

---

The verification of the starttime and endtime fields in the shift table has been successfully completed. The fields are indeed nullable as indicated by their definitions in the DBML file:

• "starttime" timestamp [note: 'Shift start time.']
• "endtime" timestamp [note: 'Shift end time.']

There are no additional constraints or notes indicating that these fields are non-nullable, which aligns with the initial review comment's concern about confirming their nullable status.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Check for any constraints or triggers related to nullable 'starttime' and 'endtime' fields.
rg --type dbml 'starttime' packages/dbml-core/__tests__/importer/oracle_importer/output/db_dump.out.dbml
rg --type dbml 'endtime' packages/dbml-core/__tests__/importer/oracle_importer/output/db_dump.out.dbml

Length of output: 265

---

---

Script:

#!/bin/bash
# Check for any constraints or triggers related to nullable 'starttime' and 'endtime' fields in the DBML file.
rg 'starttime' packages/dbml-core/__tests__/importer/oracle_importer/output/db_dump.out.dbml
rg 'endtime' packages/dbml-core/__tests__/importer/oracle_importer/output/db_dump.out.dbml

Length of output: 283

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[NQPhuc]

Hi @MartinRucka,

Thank you for contributing to dbml.

I can that you initially use ANTLR grammar to implement Oracle SQL parsers, but then switch to define your own syntax in PegJS.

We're in the progress of migrating to our SQL parsers to ANTLR grammar to reduce the maintain effort on grammars for each SQLs dialects, as well as avoiding most errors for un-supported statements (for example, DML like SELECT, INSERT,...).

Is there any reason you can't implement this in ANTLR?

Thanks