[LFX'24 term-3 Proposed idea] introduce kcl.mod.sum

[AkashKumar7902]

When the release of the minimum version selection algorithm is complete, we can slowly deprecate kcl.mod.lock.

kcl.mod.lock also records hashes of the module's direct and indirect dependencies, we can store this hashes into a sum file (like go.sum).

[AkashKumar7902]

cc: @zong-zhe

[Peefy]

Sorry, I think kcl.mod.lock and kcl.mod.sum are the same thing?

In addition, MVS has not been enabled in KCL v0.9.0 yet, and there are some issues that we still need to rigorously test and improve.

cc @AkashKumar7902 @zong-zhe

[AkashKumar7902]

Can you please create or add labels to those issues related to mvs? It could be given in pretasks and worked on during 1st half of the term.

Also, do we have a checksum database that kcl programs consult to check if the calculated checksum of the module is correct to ensure it is complete and not tampered ?

[AkashKumar7902]

Also I believe the lock and sum are not the same thing: https://stackoverflow.com/questions/53837919/should-go-sum-file-be-checked-in-to-the-git-repository#:~:text=sum%20is%20not%20a%20lock,enough%20information%20for%20reproducible%20builds).

[zong-zhe]

Hi @AkashKumar7902 😊

Your thoughts are meaningful, and I appreciate your enthusiasm for the KCL project. This idea is excellent 👍, and we can consider some features around kcl.mod.lock.

Here are some of my options:

1. Personally, the main functionality of go.sum is to provide a file for storing the checksum information of third-party dependencies to support the verification. As for the name of this file, whether it is called kcl.mod.lock or go.sum, it does not matter. Currently, kcl.mod.lock contains metadata of KCL third-party dependencies and has a sum field that can be used to store checksums. Given that many users and third-party dependencies already use kcl.mod.lock, changing the name and content of this file is a high-risk work with relatively low returns. 😳 Users are likely to question why kcl.sum is being used when kcl.mod.lock already has a "sum" field. The contents in https://stackoverflow.com/questions/53837919/should-go-sum-file-be-checked-in-to-the-git-repository#:~:text=sum%20is%20not%20a%20lock,enough%20information%20for%20reproducible%20builds do not alter my thought.

2. The idea of a checksum database is good 👍 and is a mature solution in the industry. We are currently working on the design for checksum verification. However, the task of "establishing a database" may not be the best solution, as we currently do not have the necessary resources to support it. My current thoughts are:

   1. Data storage does not necessarily have to be handled by a database; it could be stored in a git repository, which is a viable option, as seen in the example of Rust, where some metadata is stored in a git repository https://github.com/rust-lang/crates.io-index

   2. Alternatively, the OCI Registry itself can store some metadata, and regarding the checksums of certain packages, we could store the package's OCI manifest metadata. There is already some groundwork in place.

   related source code :

   kpm/pkg/client/client.go

   Line 220 in 09233a5

   if value, ok := manifest.Annotations[constants.DEFAULT_KCL_OCI_MANIFEST_SUM]; ok {

   the oci manifest of KCL looks like: https://github.com/kcl-lang/kpm/blob/09233a5a2ceb4251ec04776036ecf85070f6a8fc/test/e2e/test_suites/test_data/expected_oci_manifest.json

3. Regarding some issues about MVS, I will add a few issues later to provides more details about certain problems.

And I think that ‘The verification of checksum for KCL dependencies’ may be seen as a significant project for LFX 3, the KCL package management tool has done some work on this part, but it doesn't work very well.

• This part of the work may include adjusting checksum in some of the dependencies that exist, as some of them currently provided by KCL do not contain checksum fields. This is a high risk job.

The above are just some of my current ideas, not the final conclusion. We can continue to discuss this to determine the final topic. We still have time until Jul 3-Jul 23 5:00PM PDT 😄 😄 😄

[AkashKumar7902]

yes, after thinking on this for a while, I agree that making such a major change without any UX improvement is not going to be good, and existing users would be greatly against this.