Updated Dotnet and Arcade

[grazy27]

Changes:

• Updated .Net6 -> .Net8
• Updated .Net Framework 4.6.1 -> 4.8
• Updated Nuget dependencies
• Updated Arcade 1.0 -> 8. Used release 8.0 branch
• Added a few git ignores and fixed a warning in the build pipeline

---

Affected Tickets:

• [FEATURE REQUEST]: .Net 8.0 support #1170
• [FEATURE REQUEST]: Update Arcade version to allow updating .NET #1198

[wudanzy]

Can we split this into another PR? Since we are not familiar with this, we may need to involve some people from legal department to review. Splitting them would be clearer.

[grazy27]

That's part of Arcade validation pipeline, the same license is used all dotnet MIT projects

Arcade PR for license validation

[wudanzy]

I see, let me check how to handle this.

[wudanzy]

Why removing it?

[grazy27]

For pr pipeline - we don't define it

[grazy27]

Removed vscode config as a separate commit

Btw, there's .NET9 already available, but it's not LTS, so I still chose .NET8. Does my logic make sense to you?

[wudanzy]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[grazy27]

@wudanzy build timed out, could you please share the logs where exactly did it stall?

[wudanzy]

Maybe flaky, retried.

[wudanzy]

Hi @grazy27, you have more experience than me on .NET. Do you know if the upgraded .NET 8 (or net48) runtime can run spark jobs compiled by .NET 6 (or net47)? Or do we need an integration test?

[wudanzy]

LGTM from my sides, let @SparkSnail to have another look

[grazy27]

Hi @grazy27, you have more experience than me on .NET. Do you know if the upgraded .NET 8 (or net48) runtime can run spark jobs compiled by .NET 6 (or net47)? Or do we need an integration test?

In order to update Dotnet.Spark users will have to pull a new nuget, rebuild their project, and test that it's upgraded successfully. And if spark.worker is used - copypaste dlls to accessible location for it

Overall CLRs of 6 & 8, and 4.6 & 4.8 are compatible,
For framework, users might have to tinker with binding redirects in app.config.
For .NET8 - no changes needed, the only caveat is that there might be APIs that were deprecated or removed in .net8, so if .net6 lib references these APIs - there will be an error.

This situation can theoretically occur with spark.worker and UDF dlls, if users decide not to upgrade their project to 8:

[SparkSnail]

3-5

[grazy27]

Yup, I'll fix in the next PR

[GeorgeS2019]

@SparkSnail @grazy27 @wudanzy @radical @shanselman what are next to do list?

[wudanzy]

There is some discussion in #1162, and I personally want to remove 2.X support.