Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require all packages to solve / compile and include all valid compilers in their metadata #669

Open
wants to merge 56 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
0b6dd4e
Add 'compilers' field to metadata
thomashoneyman Nov 8, 2023
e15e4a8
Add utilities for building with many compilers
thomashoneyman Nov 11, 2023
d8e7e41
Remove PackageSource and require all packages to solve/compile
thomashoneyman Nov 11, 2023
8e069b6
Determine all compilers for package in publish pipeline
thomashoneyman Nov 11, 2023
5348ee2
Initial cut at discovering compiler in legacy import
thomashoneyman Nov 12, 2023
630c0bf
Always look up metadata / manifests in each publishing step
thomashoneyman Nov 12, 2023
77d6e68
Testing the pipeline...
thomashoneyman Nov 13, 2023
8749bea
Better reporting of failures
thomashoneyman Nov 13, 2023
be93d18
Update union of package set / spago / bower deps, consider ranges in …
thomashoneyman Nov 14, 2023
5a15433
Include spago.yaml files in legacy import
thomashoneyman Nov 15, 2023
559275c
Retain compilation in cache
thomashoneyman Nov 15, 2023
09d515a
Consider compilers when solving
thomashoneyman Nov 16, 2023
98ef892
Rely on solver per-compiler instead of looking at metadata for compat…
thomashoneyman Nov 16, 2023
ae621da
Adjust unused dependency pruning to replace used transitive deps
thomashoneyman Nov 17, 2023
5c54103
Remove unused functions
thomashoneyman Nov 17, 2023
441b960
wip
thomashoneyman Nov 17, 2023
3495edb
Use cache when finding first suitable compiler
thomashoneyman Nov 19, 2023
7ceab4c
WIP: Include missing direct imports
thomashoneyman Nov 19, 2023
3b85cd5
No longer try to insert missing dependencies
thomashoneyman Nov 19, 2023
3fa90b5
Address internal comments
thomashoneyman Nov 20, 2023
628fdf0
Merge branch 'master' into trh/compilers-in-metadata
thomashoneyman Nov 20, 2023
0d3cef9
Re-enable comment
thomashoneyman Nov 20, 2023
4e8cb87
Remove unnecessary
thomashoneyman Nov 20, 2023
d7c4180
Merge branch 'master' into trh/compilers-in-metadata
thomashoneyman Dec 1, 2023
81c85a4
Fix 'removed packages' stats
thomashoneyman Dec 1, 2023
10bccee
Feedback
thomashoneyman Dec 1, 2023
26c5aa0
Always print publish stats
thomashoneyman Dec 1, 2023
b11917e
tweaks
thomashoneyman Dec 4, 2023
3ddde82
Better publish stats formatting and write removals
thomashoneyman Dec 4, 2023
ec388d1
Merge branch 'master' into trh/compilers-in-metadata
thomashoneyman Dec 4, 2023
5b17cb3
Update flake
thomashoneyman Dec 5, 2023
f924b31
Integrate inserting missing dependencies
thomashoneyman Dec 7, 2023
3cdb9b9
Tweaks for efficiency
thomashoneyman Dec 7, 2023
d0181e5
(hopefully) final run of the importer
thomashoneyman Dec 8, 2023
6f9f0cd
Update spec to note transitive dependencies requirement.
thomashoneyman Dec 8, 2023
2721c6a
attempt to discover publish compiler with both legacy and current ind…
thomashoneyman Dec 8, 2023
f8d0f80
Tweaks
thomashoneyman Dec 10, 2023
e2d6e87
Patch some legacy manifests
thomashoneyman Dec 10, 2023
b8a21a8
Range tweaks for bolson/deku/rito
thomashoneyman Dec 11, 2023
3d7ab49
Update to fix darwin support for spago builds
thomashoneyman Dec 18, 2023
6bc8d09
Clean up publish stats
thomashoneyman Dec 18, 2023
9acbc94
Enforce an explicit 0.13 date cutoff / core org cutoff
thomashoneyman Dec 19, 2023
d2c3b9a
Merge branch 'master' into trh/compilers-in-metadata
thomashoneyman Jan 5, 2024
bea2013
Move location check above manifest parse
thomashoneyman Jan 17, 2024
c942722
Merge branch 'master'
thomashoneyman Jul 29, 2024
637a757
format
thomashoneyman Jul 29, 2024
ab184f2
Fix octokit codec merge error
thomashoneyman Jul 29, 2024
9cc56e7
Revert "Fix octokit codec merge error"
thomashoneyman Jul 29, 2024
c05fcb9
Set compiler explicitly to 0.15.5
thomashoneyman Jul 29, 2024
637488d
Tweaks
thomashoneyman Jul 29, 2024
662dd00
Set all purs test compilers to 0.15.4 range
thomashoneyman Jul 29, 2024
8156aa2
Update retry logic to fix integration test
thomashoneyman Jul 30, 2024
ed7913c
Complete run of legacy importer
thomashoneyman Aug 26, 2024
ec8e3ff
Format
thomashoneyman Aug 26, 2024
d7d5e49
Merge branch 'master' into trh/compilers-in-metadata
thomashoneyman Aug 29, 2024
7d74da3
Merge branch 'master' into trh/compilers-in-metadata
thomashoneyman Oct 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
No longer try to insert missing dependencies
  • Loading branch information
thomashoneyman committed Nov 19, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 3b85cd573ac9966099c7e51419277f1d6720d146
86 changes: 28 additions & 58 deletions app/src/App/API.purs
Original file line number Diff line number Diff line change
@@ -1288,39 +1288,27 @@ fixManifestDependencies { source, compiler, index, manifest: Manifest manifest,
let directPackages = Set.mapMaybe (flip Map.lookup associated) directImports
Log.debug $ "Found packages directly imported by project source code: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable directPackages))

-- Unused packages are those which are listed in the manifest dependencies
-- but which are not imported by the package source code.
let unusedInManifest = Set.filter (not <<< flip Set.member directPackages) (Map.keys manifest.dependencies)
when (Set.size unusedInManifest > 0) do
Log.warn $ "Manifest includes unused packages: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable unusedInManifest))

let missingInManifest = Set.filter (not <<< flip Map.member manifest.dependencies) directPackages
when (Set.size missingInManifest > 0) do
Log.warn $ "Manifest does not include imported packages: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable missingInManifest))

if Set.isEmpty unusedInManifest && Set.isEmpty missingInManifest then
if Set.isEmpty unusedInManifest then
-- If there are no unused dependencies then we don't need to fix anything.
pure $ Tuple (Manifest manifest) verified
else do
Log.debug $ "Found unused dependencies: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable unusedInManifest))

let
registry :: Solver.TransitivizedRegistry
registry = Solver.initializeRegistry $ un CompilerIndex index

solveSteps :: Map PackageName Range -> Map PackageName Range
solveSteps init =
Map.mapMaybe (\intersect -> Range.mk (Solver.lowerBound intersect) (Solver.upperBound intersect))
$ Safe.Coerce.coerce
$ _.required
$ Solver.solveSteps
$ Solver.solveSeed { registry, required: Solver.initializeRequired init }

expandedManifest :: Map PackageName Range
expandedManifest = solveSteps manifest.dependencies

pruneUnused :: Map PackageName Range -> Map PackageName Range
pruneUnused deps = do
prune :: Map PackageName Range -> Map PackageName Range
prune deps = do
let
partition = partitionEithers $ map (\entry -> entry # if Set.member (fst entry) directPackages then Right else Left) $ Map.toUnfoldable deps
remainingUnused = Map.fromFoldable partition.fail
unusedDeps = Map.fromFoldable partition.fail

if Map.isEmpty remainingUnused then
if Map.isEmpty unusedDeps then
deps
else do
let
@@ -1329,54 +1317,36 @@ fixManifestDependencies { source, compiler, index, manifest: Manifest manifest,

unusedTransitive :: Map PackageName Range
unusedTransitive =
Map.mapMaybeWithKey (\key intersect -> if Map.member key remainingUnused then Nothing else Range.mk (Solver.lowerBound intersect) (Solver.upperBound intersect))
Map.mapMaybeWithKey (\key intersect -> if Map.member key unusedDeps then Nothing else Range.mk (Solver.lowerBound intersect) (Solver.upperBound intersect))
$ Safe.Coerce.coerce
$ _.required
$ Solver.solveSteps (Solver.solveSeed { registry, required: Solver.initializeRequired remainingUnused })

pruneUnused $ Map.unionWith (\used unused -> fromMaybe used (Range.intersect used unused)) usedDeps unusedTransitive

fixedDependencies = pruneUnused expandedManifest

-- Missing packages are those which are imported by the package source
-- but which are not listed in the manifest dependencies.
let missing = Set.filter (not <<< flip Set.member (Map.keys fixedDependencies)) directPackages
case Set.size missing of
0 -> pure unit
n -> do
Log.warn $ show n <> " packages still missing!"
unsafeCrashWith $ String.joinWith "\n\n"
[ "ORIGINAL DEPS:\n" <> printJson (Internal.Codec.packageMap Range.codec) manifest.dependencies
, "EXPANDED DEPS:\n" <> printJson (Internal.Codec.packageMap Range.codec) expandedManifest
, "PRUNED DEPS:\n" <> printJson (Internal.Codec.packageMap Range.codec) fixedDependencies
, "DIRECT IMPORTS: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable directPackages))
, "MISSING : " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable missing))
, "RESOLUTIONS : " <> printJson (Internal.Codec.packageMap Version.codec) verified
]
$ Solver.solveSteps (Solver.solveSeed { registry, required: Solver.initializeRequired unusedDeps })

prune $ Map.unionWith (\used unused -> fromMaybe used (Range.intersect used unused)) usedDeps unusedTransitive

prunedDependencies = prune manifest.dependencies

case Solver.solveFull { registry, required: Solver.initializeRequired fixedDependencies } of
case Solver.solveFull { registry, required: Solver.initializeRequired prunedDependencies } of
Left failure ->
unsafeCrashWith $ "Failed to solve for dependencies while fixing manifest: " <> Foldable1.foldMap1 (append "\n" <<< Solver.printSolverError) failure
Except.throw $ "Failed to solve for dependencies while fixing manifest: " <> Foldable1.foldMap1 (append "\n" <<< Solver.printSolverError) failure
Right new' -> do
let purs = unsafeFromRight (PackageName.parse "purs")
let newResolutions = Map.delete purs new'
let removed = Map.keys $ Map.difference manifest.dependencies fixedDependencies
let added = Map.difference fixedDependencies manifest.dependencies
let removed = Map.keys $ Map.difference manifest.dependencies prunedDependencies
let added = Map.difference prunedDependencies manifest.dependencies
Comment.comment $ Array.fold
[ "Your package is using a legacy manifest format, so we have adjusted your dependencies to remove unused ones and add directly-imported ones. Your dependency list was:\n"
, "```json\n"
[ "Your package is using a legacy manifest format, so we have adjusted your dependencies to remove unused ones. Your dependency list was:"
, "\n```json\n"
, printJson (Internal.Codec.packageMap Range.codec) manifest.dependencies
, "\n```\n"
, Monoid.guard (not (Set.isEmpty removed)) do
" - We have removed the following packages: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable removed)) <> "\n"
, Monoid.guard (not (Map.isEmpty added)) do
" - We have added the following packages: " <> String.joinWith ", " (map (\(Tuple name range) -> PackageName.print name <> "(" <> Range.print range <> ")") (Map.toUnfoldable added)) <> "\n"
, "Your new dependency list is:\n"
, "```json\n"
, printJson (Internal.Codec.packageMap Range.codec) fixedDependencies
, Monoid.guard (not (Set.isEmpty removed)) $ " - We have removed the following packages: " <> String.joinWith ", " (map PackageName.print (Set.toUnfoldable removed)) <> "\n"
, Monoid.guard (not (Map.isEmpty added)) $ " - We have added the following packages: " <> String.joinWith ", " (map (\(Tuple name range) -> PackageName.print name <> "(" <> Range.print range <> ")") (Map.toUnfoldable added)) <> "\n"
, "Your new dependency list is:"
, "\n```json\n"
, printJson (Internal.Codec.packageMap Range.codec) prunedDependencies
, "\n```\n"
]
pure $ Tuple (Manifest (manifest { dependencies = fixedDependencies })) newResolutions
pure $ Tuple (Manifest (manifest { dependencies = prunedDependencies })) newResolutions

type COMPILER_CACHE r = (compilerCache :: Cache CompilerCache | r)

2 changes: 1 addition & 1 deletion scripts/src/LegacyImporter.purs
Original file line number Diff line number Diff line change
@@ -386,7 +386,7 @@ runLegacyImport logs = do
, "----------"
]

void $ for (Array.take 1000 manifests) publishLegacyPackage
void $ for manifests publishLegacyPackage

Log.info "Finished publishing! Collecting all publish failures and writing to disk."
let