Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compiler dependency does not give access to compiler #1166

Open
arnetheduck opened this issue Dec 4, 2023 · 8 comments
Open

compiler dependency does not give access to compiler #1166

arnetheduck opened this issue Dec 4, 2023 · 8 comments

Comments

@arnetheduck
Copy link

arnetheduck commented Dec 4, 2023

.nimble:

requires "nim >= 1.6.16",
         "compiler"
nimble setup -l
   Warning: Using project local deps mode
      Info: using /home/arnetheduck/src/nimph/nimbledeps/pkgs2/nim-2.0.0-35b9d926d314ce8c3326902c138f5c2247cfd390/bin/nim for compilation
  Verifying dependencies for [email protected]
     Info:  "nimble.paths" is updated.
     Info:  "config.nims" is already set up.

nimble.paths:

arnetheduck@praeceps:~/src/nimph$ cat nimble.paths
--noNimblePath

notably, compiler path is missing (compiler/ is not part of nim standard paths)

@daylinmorgan
Copy link
Contributor

Related to #1163

@jmgomez
Copy link
Collaborator

jmgomez commented Feb 13, 2024

Should we treat it as a some sort of "special" package? As far as I can tell is defined as:

  {
    "name": "compiler",
    "alias": "nim"
  },

Without knowing much about nimble, all I can say is either we expand the definition or we treat compiler differently

@daylinmorgan
Copy link
Contributor

Nimble is translating that to the nim package nimpkgs link instead of packages.json.

Which when using either nimble install nim or nimble install compiler is using a before install task defined in the nim.nimble in the main repo to run the build script. But ignores it as requires "compiler" in a .nimble.

Being able to access the "compiler" as a source itself should really only be necessary if you need a version separate from the compiler you are using. Since imports can reference the compiler with import "$nim" / compiler.

We could either package the compiler source as a standalone repo that would be importable with import compiler and it's own compiler.nimble that doesn't build the nim binary or tools. Or modify nimble to do this behind the scenes and treat the compiler dependency special rather than as an alias for nim.

I would say you we could do something with https://github.com/nim-lang/nim/subdir?=compiler and put the compiler.nimble there Oh wait this already exists!

But the current compiler.nimble isn't usable with nimble, since nimble only copies the subdir to the nimbleDir.

But if we change it to include "$lib" instead of include ../lib. Then the following seems to work for using the compiler as a package.
dummy.nimble

version       = "0.1.0"
author        = "Daylin Morgan"
description   = "A new awesome nimble package"
license       = "MIT"
srcDir        = "src"
bin           = @["dummy"]

requires "nim >= 2.0.2"
requires "https://github.com/daylinmorgan/nim?subdir=compiler#compiler-nimble-fix"

src/dummy.nim

import compiler/ast

when isMainModule:
  echo("Hello, World!")

So if we make a PR to nim-lang/nim to change the compiler.nimble and update the packages.json to treat the compiler as a package pointing to the subdir then it would work. Though the changes to packages.json should probably coincide with the first nim version release containing the updated compiler.nimble

@arnetheduck
Copy link
Author

arnetheduck commented Feb 14, 2024

Should we treat it as a some sort of "special" package? As far as I can tell is defined as:

preferably no - the majority of the code therein can be reasoned about as if it were a package like any other.

Being able to access the "compiler" as a source itself should really only be necessary if you need a version separate from the compiler you are using.

there's no particular reason to treat the majority of code in the compiler in any special way.

iff the compiler needs to have a version constraint on the standard library for whatever reason, it should express that constraint in its declaration of requirements, and those parts should ideally be separated into their own package (nim-lang-core or whatever).

It is not a given that I want the same version of compiler as the source code is being compiled with - ie if I want to access the source code parser of the compiler as it worked in nim 1.6, I should (ideally) be able to access that version regardless of which nim version I'm using to compile my application. The $nim/... is a hack at best because it overconstrains the versioning requirements and leaves the natural flexibility that would come from not special-casing all of the compiler on the table.

This follows from a much broader point: the code in the compiler is / can be treated as a "library" and follow the rules of any other library (and just like some libraries come with "supporting binaries", so can the nim compiler be treated as a supporting binary (ie think c2nim, parser generators and other similar transformation tooling).

The same applies to the standard library: it is just one more nim library and large parts of it need no special version of the language (broadly this is also true because most of the standard library is unmaintained / obsolete / stable and doesn't change from nim version to version).

Where this leads, in terms of long-term planning:

  • there exists a (small) subset of the standard library that is tightly integrated with the version of the language and the code generation - "broadly", this is system/ and its magics
  • there exists a (small) subset of the compiler that depends on the above (small) subset of the standard library to which it is tightly coupled
  • everthing else is fair game to be extracted into separately accessible components (ie std lib can be split up into constituent parts as can the compiler.
  • in some distant future, it would be nice to have modern PM capabilities which treat each version of a package as a separate package, effectively allowing multiple versions of a component to be accessed from the same application - simple use case: a tool like nph wants to access multiple parser versions (1.6, 2.0 etc) to provide compatibility for source code for multiple nim versions from the same application and adapt to it

With the above long-term picture in mind, where this leads, in terms of this bug report in particular:

  • it would be nice if I could depend on compiler and not have to perform any "special" tricks like $nim/... even if the experience is not as flexible as it could be (ie fully free version selection to access a particular version of the compiler source code)
  • for the time being, because of the monolithic nature of both compiler and standard library, the versions could be locked in step

That limited feature set is compatible with the more modern approach from above and already brings benefits.

@jmgomez
Copy link
Collaborator

jmgomez commented Feb 14, 2024

it would be nice if I could depend on compiler and not have to perform any "special" tricks like $nim/...

What about adding a path option to the package definition where the user can add a list of search paths that will be expanded in the nimble.path file?

@arnetheduck
Copy link
Author

What about adding a path option to the package definition

that's what I would expect nimble to do when I say requires "compiler" (so that it propagates across transitive deps etc)

@jmgomez
Copy link
Collaborator

jmgomez commented Feb 14, 2024

I mean, right now compiler is defined as:

 {
    "name": "compiler",
    "alias": "nim"
  },

Im referring to expand the nimble functionality so when defined like so:

 {
  "name": "compiler",
  "alias": "nim"
  "paths": ["./compiler"]
}

it will expand it in the nimble.path (paths would work in both the real and the alias)

@arnetheduck
Copy link
Author

Im referring to expand the nimble functionality so when defined like so:

oh - didn't know about that, no idea - intuitively feels like a hack that might bridge the gap until a proper solution is found (that treats the compiler as a separate package on its own - notably, this might require more .. surgery)

jmgomez added a commit to jmgomez/nimble that referenced this issue Feb 17, 2024
Araq pushed a commit that referenced this issue Apr 4, 2024
* Adds the `paths` feature option

##1166

* re-enable tests

* do not allow `paths` escape the package root dir

* improves `isSubdirOf`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants