Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating relese with shaded dependencies #54

Open
kristoffSC opened this issue Feb 14, 2023 · 3 comments
Open

Creating relese with shaded dependencies #54

kristoffSC opened this issue Feb 14, 2023 · 3 comments

Comments

@kristoffSC
Copy link

Hi,
great library :)

Would it be possible to release a new version with all dependencies shaded?
Especially those:

  • org.apache.calcite:calcite-core
  • org.apache.calcite:calcite-druid
  • org.apache.calcite:calcite-linq4j
  • org.apache.calcite:calcite-avatica
  • org.apache.parquet", "parquet-column"
  • org.apache.parquet", "parquet-hadoop-bundle"
  • org.apache.hive", "hive-exec"
  • org.apache.hadoop", "hadoop-common"

I would like to use this library for creating tests for one of the Apache Flink connectors but I have very nasty conflicts on those libraries, since Flink also have those but with different versions that are not binary compatible with eachother.

Having this library with shaded dependencies would solve the problem

@patduin
Copy link
Contributor

patduin commented Feb 15, 2023

Hi and thanks,

Probably easier to exclude those from your pom.
I think for this library we should make a new release where we make all those deps set to provided.
What do you think @abhimanyugupta07 ?

@kristoffSC
Copy link
Author

kristoffSC commented Feb 15, 2023

Hi @patduin
thanks for replay.

we make all those deps set to provided

This will not help in my case at all. There is a binary incompatibility between dependencies versions that you are using and that Flink [1] does.

Probably easier to exclude those from your pom.

Actually that was very hard, what I ended up with is using pure hive-exec and hive-metastore and reusing high level classes from your library. In fact i had to exclude a bunch of stuff from those two libraries, at the end I have:

"org.apache.hive" % "hive-metastore" % "3.1.2" % "test" excludeAll(
        ExclusionRule("org.apache.avro", "avro"),
        ExclusionRule("org.slf4j", "slf4j-log4j12"),
        ExclusionRule("org.pentaho"),
        ExclusionRule("org.apache.hbase"),
        ExclusionRule("org.apache.hbase"),
        ExclusionRule("co.cask.tephra"),
        ExclusionRule("com.google.code.findbugs", "jsr305"),
        ExclusionRule("org.eclipse.jetty.aggregate", "module: 'jetty-all"),
        ExclusionRule("org.eclipse.jetty.orbit", "javax.servlet"),
        ExclusionRule("org.apache.parquet", "parquet-hadoop-bundle"),
        ExclusionRule("com.tdunning", "json"),
        ExclusionRule("javax.transaction", "transaction-api"),
        ExclusionRule("'com.zaxxer", "HikariCP"),
      ),
      // Exclusions due to conflicts with Flink's libraries from table planer, hive, calcite etc.
      "org.apache.hive" % "hive-exec" % "3.1.2" % "test" classifier "core" excludeAll(
        ExclusionRule("'org.apache.avro", "avro"),
        ExclusionRule("org.slf4j", "slf4j-log4j12"),
        ExclusionRule("org.pentaho"),
        ExclusionRule("com.google.code.findbugs", "jsr305"),
        ExclusionRule("org.apache.calcite.avatica"),
        ExclusionRule("org.apache.calcite"),
        ExclusionRule("org.apache.hive", "hive-llap-tez"),
        ExclusionRule("org.apache.logging.log4j"),
        ExclusionRule("com.google.protobuf", "protobuf-java"),
      ),

When I tried same exclusions on your library I still had issues with binary incompatibility.
From my perspective would be the best if library would be self contained and have its all dependencies shared but I can understand if you would not want to do it.

Maybe having two artifacts? Shaded and not?

[1] https://nightlies.apache.org/flink/flink-docs-release-1.16/

@patduin
Copy link
Contributor

patduin commented Feb 21, 2023

Thanks for the detailed reply! I see what you mean. We'll have to look into this a bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants