Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EngAut] Automate Sync/Release the polakdot-sdk templates #6

Closed
Tracked by #3155
mordamax opened this issue Mar 6, 2024 · 53 comments
Closed
Tracked by #3155

[EngAut] Automate Sync/Release the polakdot-sdk templates #6

mordamax opened this issue Mar 6, 2024 · 53 comments
Assignees

Comments

@mordamax
Copy link
Collaborator

mordamax commented Mar 6, 2024

Origin: paritytech/polkadot-sdk#3155 (comment)

Who has requested: @kianenigma

OKR Q2 2024:
Objective 1: Improving the Polkadot developer experience
KR: New dev templates are shipped as independent repositories automatically upon each stable release with source of truth in polkadot-sdk

Problem:
paritytech/polkadot-sdk#3155
The existence of substrate-developer-hub in itself is questionable in my opinion. It is not well maintained, and I much rather everything to be in /paritytech, and ideally in polkadot-sdk repo.

Details
From @kianenigma
What we suggest doing is the following:

  • Bring all templates of polkadot-sdk under one folder. /templates/solo-chain, /templates/parachain and /templates/minimal are the main 3. These are the source of truth and always kept up to date.
  • Create repositories corresponding for each, called polkadot-sdk-solo-chain-template, polkadot-sdk-solo-parachain and polkadot-sdk-minimal-tempalte.
  • Write a script that receives a commit hash somewhere in the stable branch as input, and mirrors the templates into their corresponding repo. This script needs to replace all Cargo.toml files to use the released crates.
  • We run the above script upon each update of stable branch, i.e. every 3 month.
@rzadp
Copy link

rzadp commented Mar 12, 2024

Requested Github bot creation: https://github.com/paritytech/security/issues/45

@rzadp
Copy link

rzadp commented Mar 14, 2024

I have a first draft in the paritytech-stg organization.

The draft PR: paritytech-stg/polkadot-sdk#24
The current result (minimal template): https://github.com/paritytech-stg/polkadot-sdk-minimal-template


I had to remove a dependency on frame because it is not available as a published crate.
Used frame-* and sp-* dependencies instead, and re-declared some types that were only available in frame/src.

@kianenigma Could you take a look and let me know if the direction I took makes sense?

@kianenigma
Copy link

kianenigma commented Mar 14, 2024

(cc-ing @gupnik as he might be faster to reply while I am attending to some urgent debugging)

I think now's the time to publish frame crate as well. Would be good to ask @EgorPopelyaev or @Morganamilo if it had any cargo-related issues? probably no reason, just because we didn't mark it.

@gupnik
Copy link

gupnik commented Mar 15, 2024

Thanks @rzadp for looking into this. It seems that I don't have access to https://github.com/paritytech-stg/polkadot-sdk-minimal-template. Could you add me please?

I had to remove a dependency on frame because it is not available as a published crate.

Yeah, I agree with @kianenigma to publish the frame crate instead of removing it altogether. Let me know if I can help in any way.

@rzadp
Copy link

rzadp commented Mar 15, 2024

It seems that I don't have access to https://github.com/paritytech-stg/polkadot-sdk-minimal-template. Could you add me please?

Ah right, made it public now @gupnik

@rzadp
Copy link

rzadp commented Mar 15, 2024

I have a problem building the current result template, I'm missing a ExtrinsicInclusionMode enum because it does not exists (yet) in the released sp_runtime crate.

@gupnik Are you able to help with that?

The real problem is that with the automation, I'm trying to use released crates from release-crates-io-v1.8.0 branch, using the template code from master. I cannot run the automation from the release branch yet, because the templates didn't exist in the monorepo back then.


Shall we wait until:

  • The next crates release, which will include the templates code that works properly with the released code.
  • frame crate starts being released.
  • @kianenigma I see you also have some ongoing work changing the templates?

Or shall I keep working on it with what we have presently?

@gupnik
Copy link

gupnik commented Mar 15, 2024

@rzadp I think we need to get the frame crate released. Can we do that please?

@rzadp
Copy link

rzadp commented Mar 15, 2024

@EgorPopelyaev @Morganamilo Could we have the frame crate released please?

@kianenigma
Copy link

@kianenigma I see you also have some paritytech/polkadot-sdk#3597 changing the templates?

This is an experimental branch (as indicated by "DNM"), if finalized, it will be a new template, so feel free to work on the 3 that is already there.

@gupnik
Copy link

gupnik commented Mar 21, 2024

@rzadp This has been published: https://crates.io/crates/polkadot-sdk-frame. Are we good to proceed?

@rzadp
Copy link

rzadp commented Mar 21, 2024

@rzadp This has been published: https://crates.io/crates/polkadot-sdk-frame. Are we good to proceed?

Great, I will try it out.

@rzadp
Copy link

rzadp commented Mar 21, 2024

The crate is not a part of the most recent release branch release-polkadot-v1.9.0, so I will still have to do some additional dance around it.

@rzadp
Copy link

rzadp commented Mar 21, 2024

Sent a PR to psvm to include v1.9.0: paritytech/psvm#2

@rzadp
Copy link

rzadp commented Mar 21, 2024

I got stuck trying to use this newly released crate. I'm probably doing something stupid, @gupnik could you give me a hand please?

To reproduce my issue:

git clone https://github.com/paritytech-stg/polkadot-sdk-minimal-template.git
cd polkadot-sdk-minimal-template
cargo +nightly build

Issue:

error: Could not find `frame-system` in `dependencies` or `dev-dependencies` in `<snip>/pallets/template/Cargo.toml`!
  --> pallets/template/src/lib.rs:10:1
   |
10 | #[polkadot_sdk_frame::pallet]

I don't think I should be changing the template and adding individual frame-* to dependencies?


It compiles fine when building inside the workspace.
Inside the workspace, I'm using a modified import and changed references from frame to polkadot_sdk_frame:

- frame = { path = "../../../../substrate/frame", default-features = false, features = ["experimental","runtime",] }
+ polkadot-sdk-frame = { package = "frame", path = "../../../../substrate/frame", default-features = false, features = ["experimental", "runtime"], version = "0.1.0" }

It is because the package is still called frame in the workspace.

The diff of my changes in the workspace is here.

When yanked out of the workspace and using released crates, the aforementioned issue happens.

@gupnik
Copy link

gupnik commented Mar 22, 2024

@rzadp I will take a look. Thanks.

@gupnik
Copy link

gupnik commented Mar 22, 2024

Seems like this is caused by https://github.com/paritytech/polkadot-sdk/blob/ea5f4e9a4d78234eb68bbb9130a24b0598c7893a/substrate/frame/support/procedural/tools/src/lib.rs#L115 where we assume the crate's name to be frame. Will raise a PR with the fix soon.

CC: @kianenigma in case you have something to add.

github-merge-queue bot pushed a commit to paritytech/polkadot-sdk that referenced this issue Apr 4, 2024
Step in #3155

Needed for paritytech/eng-automation#6

This PR renames `frame` crate to `polkadot-sdk-frame` as `frame` is not
available on crates.io

---------

Co-authored-by: Kian Paimani <[email protected]>
@gupnik
Copy link

gupnik commented Apr 4, 2024

@rzadp paritytech/polkadot-sdk#3813 is merged now. Could we proceed here please?

@rzadp
Copy link

rzadp commented Apr 4, 2024

@rzadp paritytech/polkadot-sdk#3813 is merged now. Could we proceed here please?

Sure, I'll continue.

@rzadp
Copy link

rzadp commented Apr 4, 2024

I've checked and all seems good for me, however to really proceed I'm gonna have to wait until a new crate release happens, that includes this PR.

@gupnik
Copy link

gupnik commented Apr 4, 2024

I've checked and all seems good for me, however to really proceed I'm gonna have to wait until a new crate release happens, that includes this PR.

@rzadp A new version has been published. Please let me know if you need anything else.

@rzadp
Copy link

rzadp commented Apr 4, 2024

Hey @gupnik, I'm still running into exactly same issue as described before.

The reproduction is also the same - try to build the master of https://github.com/paritytech-stg/polkadot-sdk-minimal-template.

It has been updated to use the newly released crates, including [email protected]

The result is the same, no matter if I import it like this:

frame = { package = "polkadot-sdk-frame", version = "0.1.1", (...)

use frame;

Or like this:

polkadot-sdk-frame = { version = "0.1.1", (...)

use polkadot_sdk_frame;

@gupnik
Copy link

gupnik commented Apr 5, 2024

@rzadp Seems that one of the dependencies has not been updated yet: https://crates.io/crates/frame-support-procedural-tools. Have asked to get this fixed.

@gupnik
Copy link

gupnik commented Apr 5, 2024

@rzadp This is fixed now.

@rzadp
Copy link

rzadp commented Apr 5, 2024

@rzadp This is fixed now.

Looks good, my minimal template now successfully builds outside of the monorepo, and can run a local dev chain 👍

Next I'll check the parachain and solochain templates.

Ank4n pushed a commit to paritytech/polkadot-sdk that referenced this issue Apr 9, 2024
Step in #3155

Needed for paritytech/eng-automation#6

This PR renames `frame` crate to `polkadot-sdk-frame` as `frame` is not
available on crates.io

---------

Co-authored-by: Kian Paimani <[email protected]>
@rzadp
Copy link

rzadp commented Apr 13, 2024

The PR has been merged and tested out in polkadot-sdk.

Results:


Questions for next steps:

@kianenigma @gupnik Is there anything I can do further in order to improve those templates?

@EgorPopelyaev @Morganamilo What could be a trigger to turn it into a fully automatic job?
A push to a release-crates-io-v* branch? Note that the crates need to be already published when the job starts. Is there a tag that gets created when crate release finishes perhaps?

@PierreBesson
Copy link

In addition to updating the standalone repos consistently. It would be good that each of those repos have proper CI pipelines set up for Continuous building and releasing. ping @lovelaced

We (Devops Team) are very interested in having those templates continuously released as binary and docker image so we can directly deploy and try to onboard them on a testnet for example. This is the closest we can get of e2e testing of a community parachain. This is important for keeping good support of those templates in our deployment tooling (ansible, helm-chart) and ensure a good devops experience for parachain teams.

Even better would be if the release team (ping @EgorPopelyaev) can build those templates in advance of them being synced to their own repos so we avoid syncing broken templates to the master branch of paritytech/polkadot-sdk-*-template where parachain teams will use it as a base of their parachain node.

@rzadp
Copy link

rzadp commented Apr 15, 2024

build those templates in advance of them being synced to their own repos so we avoid syncing broken templates

@PierreBesson In case you missed it - The templates are being build before syncing, as well as after syncing.

@PierreBesson
Copy link

PierreBesson commented Apr 16, 2024

@rzadp cargo check && cargo test is nice to do but it's not a release. We are not building the final binary and eg. testing it with Zombienet to validate that the parachain onboards properly.

@rzadp
Copy link

rzadp commented Apr 22, 2024

I created this PR which we'll need to synchronize templates with recent version 1.10.0 and following ones.

@EgorPopelyaev Do you have a standard way of testing those templates inside the monorepo before a release, that could be re-used before syncing the templates? Could we collaborate on it?

@EgorPopelyaev
Copy link

@rzadp Unfortunately, no, we don't have such tests in place for now.

@rzadp
Copy link

rzadp commented Apr 23, 2024

Note: We agreed to kick off the synchronization job as part of the manual process upon the crate releases, for now.

@rzadp
Copy link

rzadp commented Apr 24, 2024

The last version of psvm has introduced two problems for this workflow - with aliased import of crates, and apparently with build dependencies.

The problem can be seen in this automated PR - crates frame/polkadot-sdk-frame and substrate-build-script-utils. I'll be looking into it next week.

@rzadp
Copy link

rzadp commented Apr 29, 2024

Note: the problem with frame/polkadot-sdk-frame seems to be caused by the fact that the crate is named frame on the 1.10.0 branch - but the (desired) name polkadot-sdk-frame is there on the 1.9.0 branch, and on master. I'm trying to figure out what happened.

@rzadp
Copy link

rzadp commented Apr 29, 2024

Another note:

The last version of psvm has introduced two problems for this workflow - with aliased import of crates, and apparently with build dependencies.

Correction - the problem is not only about build dependencies, but also regular dependencies. The newest version of psvm seems to not convert packages that previously worked - work example sc-telemetry. I'm trying to find a pattern which packages are not working, and why.

@rzadp
Copy link

rzadp commented Apr 29, 2024

I have pinpointed the issues I'm having to psvm preferring to use Plan.toml for determining versions instead of Cargo.lock (as before).

The functionality to read Cargo.lock is still there (as a fallback) and it continues to work for my case.

I have asked here.

@rzadp
Copy link

rzadp commented May 2, 2024

The issue with psvm has been overcome. (So my 4 previous comments above are resolved now).

And now we ran into another issue: paritytech/polkadot-sdk#4347

@gupnik
Copy link

gupnik commented May 4, 2024

The issue with psvm has been overcome. (So my 4 previous comments above are resolved now).

And now we ran into another issue: paritytech/polkadot-sdk#4347

@rzadp Now that this is resolved, can we push an update please?

@rzadp
Copy link

rzadp commented May 6, 2024

@gupnik Looks like we're almost there.

minimal and solochain templates have been just updated with version 1.11.0 and all seems well with those two.

However I've run into an issue with the parachain one (wonder when I'll stop running into issues with this task 😅)

It appears that parachain-template-runtime and pallet-parachain-template are published as crates, which conflicts with the usage of psvm tool which changes local dependencies to crate dependencies, but should not do it in this case.

I'm not sure why they're published - intentionally or some leftover?

I could add some workarounds to it - but first I'll try to figure out if those are needed on crates.io - if not, yanking them will solve my issue.

@gupnik
Copy link

gupnik commented May 6, 2024

minimal and solochain templates have been just updated with version 1.11.0 and all seems well with those two.

Thanks @rzadp. However, there's now a doc-only crate that's not available in the minimal template. Similarly, it's not using the README.md from the repo. It seems that this file is also not updated.

It appears that parachain-template-runtime and pallet-parachain-template are published as crates, which conflicts with the usage of psvm tool which changes local dependencies to crate dependencies, but should not do it in this case.

But I do see that it has publish=false. So, it should not have been published in the first place?

@rzadp
Copy link

rzadp commented May 6, 2024

But I do see that it has publish=false. So, it should not have been published in the first place?

I would think so.
There is nothing wrong with this crate existing in crates.io on itself, but in our case it messes up the template when using psvm.
On the other hand there are downloads of this crate so I don't know. @EgorPopelyaev @Morganamilo any idea?


Thanks @rzadp. However, there's now a doc-only crate that's not available in the minimal template. Similarly, it's not using the README.md from the repo. It seems that this file is also not updated.

Please look at release-crates-io-v1.11.0 branch, not master - that's the source of the update.

@rzadp
Copy link

rzadp commented May 6, 2024

However, there's now a doc-only crate

Ah, it'll probably cause issues when syncing the next release 🥲

@gupnik
Copy link

gupnik commented May 6, 2024

Please look at release-crates-io-v1.11.0 branch, not master - that's the source of the update.

I actually see only master.

Screenshot 2024-05-06 at 4 57 41 PM

@gupnik
Copy link

gupnik commented May 6, 2024

However, there's now a doc-only crate

Ah, it'll probably cause issues when syncing the next release 🥲

Can we push that as well please? We are planning to use these templates for PBA and would be good to have the doc-only crate in there as well.

@rzadp
Copy link

rzadp commented May 6, 2024

@gupnik I meant the release-crates-io-v1.11.0 branch in polkadot-sdk - it's been branched out of master so does not contain all the recent changes from master. For example readme is still empty at that branch.

@rzadp
Copy link

rzadp commented May 6, 2024

Can we push that as well please? We are planning to use these templates for PBA and would be good to have the doc-only crate in there as well.

I'm not sure. In the other thread there is explicit discussion to remove it: paritytech/polkadot-sdk#3155 (comment)
Also relevant piece: https://github.com/paritytech/polkadot-sdk/blob/73c89d308fefcedfc3619f0273e13b6623766b81/docs/sdk/Cargo.toml#L9-L10

@gupnik
Copy link

gupnik commented May 6, 2024

@gupnik I meant the release-crates-io-v1.11.0 branch in polkadot-sdk

Oh, got it. Thanks!

Can we push that as well please? We are planning to use these templates for PBA and would be good to have the doc-only crate in there as well.

I'm not sure. In the other thread there is explicit discussion to remove it: paritytech/polkadot-sdk#3155 (comment) Also relevant piece: https://github.com/paritytech/polkadot-sdk/blob/73c89d308fefcedfc3619f0273e13b6623766b81/docs/sdk/Cargo.toml#L9-L10

@kianenigma What are your thoughts here?

@rzadp
Copy link

rzadp commented May 7, 2024

minimal and solochain templates have been just updated with version 1.11.0 and all seems well with those two.

Note: The issue with updating the parachain template should be resolved with a 1.12.0 release.

@rzadp
Copy link

rzadp commented May 21, 2024

Note: The 1.12.0 has not appeared yet so I'm waiting for that.

@kianenigma
Copy link

It appears that parachain-template-runtime and pallet-parachain-template are published as crates, which conflicts with the usage of psvm tool which changes local dependencies to crate dependencies, but should not do it in this case.

I'm not sure why they're published - intentionally or some leftover?

I don't see any reason to publish them, please make a PR to the base repo and mark them as unpunlish, thank you :)

@rzadp
Copy link

rzadp commented May 23, 2024

I don't see any reason to publish them, please make a PR to the base repo and mark them as unpunlish, thank you :)

@Morganamilo has yanked the crates, so this problem is already solved :) And in the repo they're already marked as unpublish.

@rzadp
Copy link

rzadp commented May 27, 2024

After a small fix in psvm, the synchronization has worked with no issues.

All templates got automatically updated to 1.12.0.

@rzadp
Copy link

rzadp commented May 31, 2024

Closing as completed - I'll still try to help with the source templates, but the automation of synchronization is done.

@rzadp rzadp closed this as completed May 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants