Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]NoSuchMethodError breeze.linalg.SliceVector #2188

Open
3 of 19 tasks
544cl opened this issue Apr 2, 2024 · 3 comments · Fixed by #2284
Open
3 of 19 tasks

[BUG]NoSuchMethodError breeze.linalg.SliceVector #2188

544cl opened this issue Apr 2, 2024 · 3 comments · Fixed by #2284
Assignees
@memoryz
Labels
bug fixed in latest triage

Comments

@544cl
Copy link

544cl commented Apr 2, 2024 •
edited
Loading

SynapseML version

0.10.1-22-95f451ab-SNAPSHOT

System information

  • Language version: python 3.8, scala 2.12
  • Spark Version : 3.2
  • Spark Platform : Synapse

Describe the problem

Getting the following error when using TabularSHAP in synapse ml. This was working few weeks ago suddenly stopped working. No changes were done in synapse spark pool.

Code to reproduce issue

from pyspark.ml.regression import LinearRegression
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler
from synapse.ml.explainers import TabularSHAP

df1 = spark.createDataFrame([
    {'a':1.0, 'b':2, 'c': 1.0},
    {'a':2.7, 'b':3, 'c': 2.0},
    {'a':3.8, 'b':4, 'c': 3.0},
    {'a':5.7, 'b':9, 'c': 4.0},
    {'a':4.2, 'b':10, 'c': 1.0},
    {'a':6.1, 'b':15, 'c': 10.0},
])

model = LinearRegression(
    featuresCol="features",
    labelCol='b'
)
assembler = VectorAssembler(inputCols=['a','b'], outputCol="features")
pipeline = Pipeline(stages=[assembler, model])

lrModel = pipeline.fit(df1)

shap = TabularSHAP(
    inputCols=['a', 'b'],
    outputCol='outCol',
    model=lrModel,
    backgroundData=df1,
    targetCol='prediction',
)


df_shap = shap.transform(df1)

display(df_shap)

Other info / logs

Py4JJavaError                             Traceback (most recent call last)
/tmp/ipykernel_6705/3699412776.py in <module>
     26 df_shap = shap.transform(df1)
     27 
---> 28 display(df_shap)

~/cluster-env/clonedenv/lib/python3.8/site-packages/notebookutils/visualization/display.py in display(data, summary)
    238         log4jLogger \
    239             .error(f"display failed with error, language: python, error: {err}, correlationId={correlation_id}")
--> 240         raise err
    241     finally:
    242         duration_ms = ceil((time.time() - start_time) * 1000)

~/cluster-env/clonedenv/lib/python3.8/site-packages/notebookutils/visualization/display.py in display(data, summary)
    216                     from IPython.display import publish_display_data
    217                     publish_display_data({
--> 218                         "application/vnd.synapse.display-widget+json": sc._jvm.display.getDisplayResultForIPython(
    219                             df._jdf, summary, correlation_id)
    220                     })

~/cluster-env/clonedenv/lib/python3.8/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1319 
   1320         answer = self.gateway_client.send_command(command)
-> 1321         return_value = get_return_value(
   1322             answer, self.gateway_client, self.target_id, self.name)
   1323 

/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw)
    109     def deco(*a, **kw):
    110         try:
--> 111             return f(*a, **kw)
    112         except py4j.protocol.Py4JJavaError as e:
    113             converted = convert_exception(e.java_exception)

~/cluster-env/clonedenv/lib/python3.8/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling z:com.microsoft.spark.notebook.visualization.display.getDisplayResultForIPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 45 in stage 510.0 failed 4 times, most recent failure: Lost task 45.3 in stage 510.0 (TID 12952) (vm-68767911 executor 1): org.apache.spark.SparkException: Failed to execute user defined function (functions$$$Lambda$8434/530344905: (struct<a:bigint>, struct<a:bigint>) => array<struct<sample:struct<a:bigint>,coalition:struct<type:tinyint,size:int,indices:array<int>,values:array<double>>,weight:double>>)
	at org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:136)
	at org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.generate_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:763)
	at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:300)
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$10(ShuffleExchangeExec.scala:375)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:905)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:905)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:338)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:338)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:208)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateSampleSizes(KernelSHAPSampler.scala:47)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateSampleSizes$(KernelSHAPSampler.scala:44)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.generateSampleSizes(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateCoalitions(KernelSHAPSampler.scala:137)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateCoalitions$(KernelSHAPSampler.scala:129)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.generateCoalitions(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator(KernelSHAPSampler.scala:34)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator$(KernelSHAPSampler.scala:33)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator$lzycompute(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.nextState(KernelSHAPSampler.scala:31)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.nextState$(KernelSHAPSampler.scala:31)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.nextState(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSampler.sample(KernelSHAPSampler.scala:16)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSampler.sample$(KernelSHAPSampler.scala:15)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.sample(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.TabularSHAP.$anonfun$createSamples$7(TabularSHAP.scala:62)
	at com.microsoft.azure.synapse.ml.explainers.TabularSHAP.$anonfun$createSamples$7$adapted(TabularSHAP.scala:60)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.immutable.Range.foreach(Range.scala:158)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at com.microsoft.azure.synapse.ml.explainers.TabularSHAP.$anonfun$createSamples$6(TabularSHAP.scala:59)
	at org.apache.spark.injections.UDFUtils$$anon$2.call(UDFUtils.scala:29)
	at org.apache.spark.sql.functions$.$anonfun$udf$93(functions.scala:5188)
	... 24 more

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2464)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2413)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2412)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2412)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1168)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1168)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1168)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2652)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2594)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2583)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function (functions$$$Lambda$8434/530344905: (struct<a:bigint>, struct<a:bigint>) => array<struct<sample:struct<a:bigint>,coalition:struct<type:tinyint,size:int,indices:array<int>,values:array<double>>,weight:double>>)
	at org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:136)
	at org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.generate_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:763)
	at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:300)
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$10(ShuffleExchangeExec.scala:375)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:905)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:905)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:338)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:338)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:208)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateSampleSizes(KernelSHAPSampler.scala:47)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateSampleSizes$(KernelSHAPSampler.scala:44)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.generateSampleSizes(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateCoalitions(KernelSHAPSampler.scala:137)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.generateCoalitions$(KernelSHAPSampler.scala:129)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.generateCoalitions(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator(KernelSHAPSampler.scala:34)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator$(KernelSHAPSampler.scala:33)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator$lzycompute(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.com$microsoft$azure$synapse$ml$explainers$KernelSHAPSamplerSupport$$randomStateGenerator(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.nextState(KernelSHAPSampler.scala:31)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSamplerSupport.nextState$(KernelSHAPSampler.scala:31)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.nextState(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSampler.sample(KernelSHAPSampler.scala:16)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPSampler.sample$(KernelSHAPSampler.scala:15)
	at com.microsoft.azure.synapse.ml.explainers.KernelSHAPTabularSampler.sample(Sampler.scala:202)
	at com.microsoft.azure.synapse.ml.explainers.TabularSHAP.$anonfun$createSamples$7(TabularSHAP.scala:62)
	at com.microsoft.azure.synapse.ml.explainers.TabularSHAP.$anonfun$createSamples$7$adapted(TabularSHAP.scala:60)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.immutable.Range.foreach(Range.scala:158)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at com.microsoft.azure.synapse.ml.explainers.TabularSHAP.$anonfun$createSamples$6(TabularSHAP.scala:59)
	at org.apache.spark.injections.UDFUtils$$anon$2.call(UDFUtils.scala:29)
	at org.apache.spark.sql.functions$.$anonfun$udf$93(functions.scala:5188)
	... 24 more

What component(s) does this bug affect?

  • area/cognitive: Cognitive project
  • area/core: Core project
  • area/deep-learning: DeepLearning project
  • area/lightgbm: Lightgbm project
  • area/opencv: Opencv project
  • area/vw: VW project
  • area/website: Website
  • area/build: Project build system
  • area/notebooks: Samples under notebooks folder
  • area/docker: Docker usage
  • area/models: models related issue

What language(s) does this bug affect?

  • language/scala: Scala source code
  • language/python: Pyspark APIs
  • language/r: R APIs
  • language/csharp: .NET APIs
  • language/new: Proposals for new client languages

What integration(s) does this bug affect?

  • integrations/synapse: Azure Synapse integrations
  • integrations/azureml: Azure ML integrations
  • integrations/databricks: Databricks integrations
@544cl 544cl added the bug label Apr 2, 2024
@github-actions github-actions bot added the triage label Apr 2, 2024
@MalekBhz
Copy link

Hello, I have the same issue as you do. Have you found any solution for this ? Thank you!

@544cl
Copy link
Author

544cl commented Apr 19, 2024 •
edited
Loading

@MalekBhz
Unfortunately I didn't find any solution for spark version 3.2, i had to upgrade to spark version to 3.3.

@memoryz memoryz self-assigned this Sep 7, 2024
@memoryz
Copy link
Contributor

memoryz commented Sep 7, 2024 •
edited
Loading

Hi, Spark 3.2 is no longer supported by Synapse, and I can no longer create a 3.2 pool to reproduce the issue. However, I did test your code in Spark 3.3 and 3.4 pools with the following configurations, and the code works in both cases.

Spark 3.3 pool:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.11.4-spark3.3",
      "spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

Spark 3.4 pool:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.0.5",
      "spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

@memoryz memoryz mentioned this issue Sep 10, 2024
Merged
4 tasks
@memoryz memoryz linked a pull request Sep 10, 2024 that will close this issue
fix: NoSuchMethodError from breeze in interpretibility notebook when running in Fabric 3.3 pool #2284
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@memoryz memoryz

Labels
bug fixed in latest triage
Projects
None yet
Milestone
No milestone
3 participants
@memoryz @544cl @MalekBhz