Add a README generator

[doonv]

Closes #4.

This PR adds a script that generates a README with a table containing all the scripts in the repo.

Basically what is does is use a template to create the README and insert a table of all the scripts, It also excludes files from the before_6.0 directory and the custom-completions\auto-generate\completions (aka the fish dump) directory.

I also made the README better in general too with a few badges. Heres a preview: readme_generator_preview.md

[fdncred]

it might be good to have a blurb in here about what happens if you script has a def main [] { } i.e. you can do nu script_name.nu --param1 --param2. last I checked we could do that with scripts that don't have main.

[fdncred]

thanks for addressing #4, I think this looks good.

[doonv]

I would like to implement a system that could read the scripts for some metadata (like a Description) and then put it inside the table. Any ideas on how metadata could be inserted into the scripts?

[fdncred]

I would like to implement a system that could read the scripts for some metadata (like a Description) and then put it inside the table. Any ideas on how metadata could be inserted into the scripts?

if custom commands have a # description text above the def then that should be shown in the usage in $nu.scope.commands | where is_custom == true and help commands | where is_custom == true. Maybe you can use that.

[skelly37]

I would like to implement a system that could read the scripts for some metadata (like a Description) and then put it inside the table. Any ideas on how metadata could be inserted into the scripts?

You might also be interested in parameter descriptions but I agree on that # descriptions above the def are the most important ones.

[doonv]

I would like to implement a system that could read the scripts for some metadata (like a Description) and then put it inside the table. Any ideas on how metadata could be inserted into the scripts?

if custom commands have a # description text above the def then that should be shown in the usage in $nu.scope.commands | where is_custom == true and help commands | where is_custom == true. Maybe you can use that.

But what if scripts have multiple commands?

[skelly37]

I would like to implement a system that could read the scripts for some metadata (like a Description) and then put it inside the table. Any ideas on how metadata could be inserted into the scripts?

if custom commands have a # description text above the def then that should be shown in the usage in $nu.scope.commands | where is_custom == true and help commands | where is_custom == true. Maybe you can use that.

But what if scripts have multiple commands?

We can develop a manifest template and encourage people to use it. Actually this is a must-have for developing a package manager, so it'd be a milestone

[skelly37]

For now you can do sth like:
1 def: include the function description/no description message
more defs: manifest/no manifest message

[doonv]

Okay so this is what I came up with:

#?name: ~/.fehbg.nu
#?version: 1.0
#
#?short-desc: Selects a random image from a directory and sets it as a wallpaper
#
#? This script selects a random image from a directory and sets it as a
#? wallpaper. An additional feature is that it draws the path of the image in
#? the lower left corner which makes it easier to find the image later.

Which would be parsed as:

{
	"name": "~/.fehbg.nu",
	"version": "1.0",
	"short-desc": "Selects a random image from a directory and sets it as a wallpaper",
	"description": "This script selects a random image from a directory and sets it as a wallpaper. An additional feature is that it draws the path of the image in the lower left corner which makes it easier to find the image later."
}

Basically all commands starting with #? are for documentation. Comments with the #?{KEY}:{VALUE} syntax will be converted to json and added to the parsed output. Other comments will be appended to the description. I'ma work on a implementation for this first.

[skelly37]

Okay so this is what I came up with:

#?name: ~/.fehbg.nu
#?version: 1.0
#
#?short-desc: Selects a random image from a directory and sets it as a wallpaper
#
#? This script selects a random image from a directory and sets it as a
#? wallpaper. An additional feature is that it draws the path of the image in
#? the lower left corner which makes it easier to find the image later.

Which would be parsed as:

{
	"name": "~/.fehbg.nu",
	"version": "1.0",
	"short-desc": "Selects a random image from a directory and sets it as a wallpaper",
	"description": "This script selects a random image from a directory and sets it as a wallpaper. An additional feature is that it draws the path of the image in the lower left corner which makes it easier to find the image later."
}

Basically all commands starting with #? are for documentation. Comments with the #?{KEY}:{VALUE} syntax will be converted to json and added to the parsed output. Other comments will be appended to the description. I'ma work on a implementation for this first.

Generally OK, but I suggest a few changes:

1. Don't store paths in name, make it just fehbg (or fehbg.nu) in this case
2. Add #?source: https://github.com/nushell/nu_scripts/blob/main/fehbg.nu
3. Add #?update-url: https://raw.githubusercontent.com/nushell/nu_scripts/main/fehbg.nu
4. Maybe add #?author, too? I'm not saying it's neccessary, but it'd be nice and it might turn out to be useful in the future.

[doonv]

Generally OK, but I suggest a few changes:

1. Don't store paths in name, make it just `fehbg` (or `fehbg.nu`) in this case

2. Add `#?source: https://github.com/nushell/nu_scripts/blob/main/fehbg.nu`

3. Add `#?update-url: https://raw.githubusercontent.com/nushell/nu_scripts/main/fehbg.nu`

4. Maybe add `#?author`, too? I'm not saying it's neccessary, but it'd be nice and it might turn out to be useful in the future.

I agree with your first point. but all of the rest can be autogenerated.

• #?source: is just https://github.com/nushell/nu_scripts/blob/main/{PATH TO SCRIPT}
• #?update-url: is just https://raw.githubusercontent.com/nushell/nu_scripts/main/{PATH TO SCRIPT}
• #?author can be fetched from GitHub.

[skelly37]

Generally OK, but I suggest a few changes:

1. Don't store paths in name, make it just `fehbg` (or `fehbg.nu`) in this case

2. Add `#?source: https://github.com/nushell/nu_scripts/blob/main/fehbg.nu`

3. Add `#?update-url: https://raw.githubusercontent.com/nushell/nu_scripts/main/fehbg.nu`

4. Maybe add `#?author`, too? I'm not saying it's neccessary, but it'd be nice and it might turn out to be useful in the future.

I agree with your first point. but all of the rest can be autogenerated.

* `#?source:` is just `https://github.com/nushell/nu_scripts/blob/main/{PATH TO SCRIPT}`

* `#?update-url:` is just `https://raw.githubusercontent.com/nushell/nu_scripts/main/{PATH TO SCRIPT}`

* `#?author` can be fetched from GitHub.

Okay, agreed with author and autogeneration.

Though I think that we should store the script's location in the repo ($PATH_TO_SCRIPT here), so the package management system can autogenerate mentioned links when needed

[fdncred]

I like where this is going. Also, I really always wanted to have script attribution somewhere, so it's nice to see that "author" can be added easily. I'm just wondering how this will work on all the existing scripts. I mean, is the plan to write some nushell script code to gather most of this information and populate it in the script files auto-magically?

[skelly37]

I mean, is the plan to write some nushell script code to gather most of this information and populate it in the script files auto-magically?

I think that everything (except for descriptions, of course) could be done "auto-magically". With versions, we can default to 1.0, while generating manifests

Adding descriptions will take some time (like porting to engine-q) but we can force new PRs to comply with a template that we'll agree here on.

[skelly37]

When the template will be agreed on, we can provide a sample pre-commit hook that uses the script to generate manifests (will need 2 arguments: short and long descriptions), so the change will be as little painful as it can be

[doonv]

Alright so I got a test working of how getting the script metadata/manifests would work.

I think its pretty good. Also the get-script-data function is very well documented.

I also did some refinement to the generator itself. What do you think?

[fdncred]

very nice!

[skelly37]

Excellent work!

Though I still think that #?name should contain only the script's name and that file's location in nu_scripts repo should be included. Besides that, everything seems to be okay.

[doonv]

Excellent work!

Though I still think that #?name should contain only the script's name and that file's location in nu_scripts repo should be included. Besides that, everything seems to be okay.

Yeah yeah, i'll do that. Currently working on the auto generation for names, authors, etc.

[skelly37]

Yeah yeah, i'll do that

In that case I have nothing to add and just can't wait to see the results :)

[doonv]

The auto-generation is working!

In this image the following properties are auto-generated:

• name: The filename.
• script-url: https://github.com/nushell/nu_scripts/blob/main/{PATH TO SCRIPT}
• authors: Fetched from the GitHub API.
• path: The to the script relative to the nu-scripts folder (current directory, by default)

[skelly37]

awesome!

[doonv]

Should we move this generator into it's own directory?

[skelly37]

Should we move this generator into it's own directory?

Imo yes, automation directory where we could also push the package manager when it‘s ready

[doonv]

When the template will be agreed on, we can...

It's about time we start discussing the template and how it will look like, I'll start and we will make changes to the following template:

#?type: 
#     script (A script that is run with `nu script.nu`) 
#     lib, libary (A libary meant to be `source`d by other files)
#     config, cfg (A configuration file meant to be `source`d by a user, can also include commands)
#?name: (Autogenerated) The name of the file
#?display-name: A fancy name that can have spaces and such
#?version: The version of the script (Must be semantic aka {Major}.{Minor}.{Patch}) (Will be required for a future script package manager's update system)
#?short-desc: A short description with less than 75 characters.
#
#? Having no key makes it a part of the long description.
#? A description should have less than 500 characters.

What do you think?

[skelly37]

#?name: (Autogenerated) The name of the file

Means the path to the file (for logical purposes)?

#?display-name: A fancy name that can have spaces and such

While this is the name for user-interface purposes, right?

[skelly37]

Besides that, I'd add #?author and it's done for me.

[doonv]

#?name: (Autogenerated) The name of the file

Means the path to the file (for logical purposes)?

No. That's a separate property which i forgot to mention called path.

#?display-name: A fancy name that can have spaces and such

While this is the name for user-interface purposes, right?

Yes this is only for user-interface purposes.

Besides that, I'd add #?author and it's done for me.

Again, i forgor to mention that property 💀(It's autogenerated)

[doonv]

Here's the new template then:

#?type: 
#     script (A script that is run with `nu script.nu`) 
#     lib, libary (A libary meant to be `source`d by other files)
#     config, cfg (A configuration file meant to be `source`d by a user, can also include commands)
#?name: (Autogenerated) The name of the file
#?path: (Autogenerated) The path to the file
#?script-url: (Autogenerated) https://github.com/nushell/nu_scripts/blob/main/{PATH TO SCRIPT}
#?script-raw-url: (Autogenerated) https://raw.githubusercontent.com/nushell/nu_scripts/main/{PATH TO SCRIPT}
#?authors: (Autogenerated)  Fetched from Github API
#?display-name: A fancy name that can have spaces and such (For user interface purposes)
#?version: The version of the script (Must be semantic aka {Major}.{Minor}.{Patch}) (Will be required for a future script package manager's update system)
#?short-desc: A short description with less than 75 characters.
#
#? Having no key makes it a part of the long description.
#? A description should have less than 500 characters.

[skelly37]

Excellent, it can do imo

[skelly37]

I'm not sure if we should create both #?name and #?display-name as separate entities. We have #?path for the package manager logic, we have #?display-name for the users, what's the purpose of the 3rd one?

[skelly37]

Also, @Dan-Gamin, if you're interested in the package manager, check out my thoughts about it. We can talk about it there or write something and create a draft PR here. Of course, I'm not forcing you to code anything but I'd be happy if you tell me what you think about the project :)

[doonv]

what's the purpose of the 3rd one?

What 3rd one? The script-url?

[doonv]

@skelly37 Yeah I want to do that in the (hopefully near) future. I'll get started on that once this get's merged.

[skelly37]

What 3rd one? The script-url?

#?name. Because, as I said. The package manager would use #?path, while the user would see #?display-name.

[skelly37]

@skelly37 Yeah I want to do that in the (hopefully near) future. I'll get started on that once this get's merged.

Don't rush with that. A friend of mine has already written some code and promised to publish it this week, so it'd be useless to do the same job. I'll let you know when that happens

[skelly37]

After consulting my pillow:

1. We already store url and raw-url, so #?path became useless.
2. #?keywords parameter with comma-separated values could be very useful for package management purposes
3. We could also drop #?raw-url, since the in-file manifest should be user-readable. Raw URLs can be handled by the repo file generator and included only there.
4. For the same reasons as in 3., I'd drop #?display-name and move its use to #?name. Filename can be easily extracted from URL.

[Yethal]

@Dan-Gamin you can make metadata parsing 10x simpler by storing it in an inline yaml file inside the script

#?name: example
#?version: 1.0.0
#?short-desc: short description
#?long-desc: >
#?  long ass description here
#?  more description
#?  even more
export def example [

] {
    echo hello
}

and then you can parse it with:

open example.nu
|lines -s
|where $it starts-with '#?'
|str replace -a -s '#?' ''
|str collect (char nl)
|from yaml

There is no need to use any custom parsing for the long description, yaml natively supports multiline strings with > operator

As for the readme generator itself it shouldn't use input since it shouldn't be ran by a human in the first place. Instead it should be ran by a Github Action on every push to main branch.

[doonv]

@Yethal thanks for the ferdback. I'll make changes soon. Hopefully.

[fdncred]

Are we ready to merge this or did we all get busy doing other things?

[skelly37]

@fdncred from my (nupac) point of view:

We need required attributes supported, ref: https://github.com/skelly37/nupac/blob/main/docs/PACKAGE_MANIFEST.md

Also, see the current way we handle repo cache generation: https://github.com/skelly37/nupac/blob/refactor-generate-repo-cachenu/generate-repo-cache.nu

---

Needed attributes:

• "author" — autogenerated yaml list from gh or user-provided
• "name" — autogenerated from filename (.nu stripped) or user-provided
• "os": in case no supported OS was provided, we assume that all of them are when generating repo cache
• "short-desc" — user provided, must be something here before accepting a script.
• "raw-url" — autogenerated
• "url" — autogenerated
• "version" — autogenerated with some default value, then must be updated by user

[skelly37]

Sorry for all the mess with the attributes but everything gets more clear during the development process

[fdncred]

i totally understand. I'm good with landing what we have, at just about any point, and continuing to work on it with subsequent PRs. i.e. if this mostly works but still needs the things we want. We can land this and you can submit a different PR with those points you mentioned above.

[fdncred]

Sorry to see this go. Thanks for your help @doonv and @skelly37. nu_scripts is kind of a mess.