[Proposal] Improve Parquet Writer for Better Performance and Maintainability

[hash-data]

Current Implementation:
Olake currently uses the Fraugster Parquet library, which has the following limitations:

• Inefficient memory usage: The library consumes more memory than optimal.
• Concurrency issues: It is not well-suited for concurrent writes, limiting performance in multi-threaded environments.
• Maintenance concerns: The library is not actively maintained, which could lead to potential issues with updates or compatibility.

Proposed Improvement:
Switch to a more optimized Parquet writer that addresses these concerns, focusing on:

• Improved memory efficiency: Lower memory consumption during operations.
• Enhanced concurrency support: Better handling of concurrent writes for improved performance.
• Active maintenance: Ensuring long-term reliability and compatibility with updates.
  One such library is parquet-go, which is actively maintained and optimized for both memory usage and concurrency.

Benefits of the Proposal:

• Reduced resource consumption
• Improved performance in multi-threaded use cases
• Better maintainability and future-proofing of the project

[sebzz2k2]

Hey @hash-data
I would like to work on this, if no one is currently working on it

[hash-data]

Hi @sebzz2k2 sure you can start working on it, let me know if any help required.

[hash-data]

@sebzz2k2 quick check-in, any update on the same?
any help required ?