v0.7.0-rc.1

This release comprises a number of bug fixes and small improvements, as well as more ground work for Modules, WebAssembly, and the core evaluator's performance refactors.

Note that v0.7 was originally planned to center around the core evaluator's performance improvements. Since those refactors are not ready, and we have other fixes and improvements we want to release, we have slightly altered the release plan accordingly. We will share more details on our next community call.

As a reminder: users can register their projects with unity, our regression and performance testing setup. unity is used to ensure that a project's CUE evaluations do not unexpectedly stop working, or regress in terms of performance. unity continues to catch multiple issues with each release. Adding your project to unity not only guarantees that we will not break your tests (if we do, we will work with you to fix your CUE code), but it also helps to improve the quality of each CUE release. Follow this link to learn more about Unity, install it, or get in touch with any questions.

Thank you to @SteVwonder, @bozaro, @cedricgc, @howardjohn, @mpvl, @mvdan, @myitcv, @nickfiggins, @rogpeppe, @rudifa, and @uhthomas for contributing to this release!

And a special thanks to all who joined the recent contributor office hours calls on our community calendar, as well as our #contributing channel on Slack! Thanks to their involvement, more issues can be investigated and fixed each release.

Go API

Note that this version of CUE requires Go 1.20 or later, per our policy to support the latest two stable Go releases just like upstream.

CL 1172105 fixes a regression introduced in v0.6.0 where calling Iterator.Selector.Index while iterating over a list would incorrectly panic.

CL 1167597 fixes cue/load so it now errors on package import cycles, following the spec.

CL 1167647 adjusts cue.Value.Decode to decode empty CUE lists into a Go interface{} as a non-nil empty slice.

CL 547369 teaches cue.Value.Decode how to decode values which aren't entirely concrete into a Go type by using cue.Value as part of the destination type.

Language

There are no changes to the language in this version.

Spec

CL 1171216 fixes two errors in a dynamic fields example.

Core Evaluator

CL 1172014 disallows the direct use of unary operators with basic types, since they would result in confusing bounds which seemed incorrect.

CL 1172013 fixes a closedness bug where close did not properly apply when used inside definitions.

CL 1172874 fixes a panic in cue export introduced by v0.6.0.

Encoders

CL 1172314 teaches encoding/protobuf to follow the field_behavior annotation marking a field as either optional or required when decoding.

Builtins

CL 1172991 adds a mustSucceed boolean parameter to tool/exec.Run, which can be set to false to allow a command to fail and set its own field success to false.

CL 557322 fixes the values of math's Log2E and Log10E constants, which were being incorrectly truncated.

cmd/cue

CLs 1170966 and 1171302 fix a number of issues in cue fmt (and by extension the cue/format package), resulting in better and more consistent formatting of CUE files.

CLs 1171292, 1171015, 1170115, and 1171971 implement a variety of improvements and bug fixes for cue get go.

CL 1172017 fixes a number of issues with the line and column positions reported by our YAML decoder, which could result in weird CUE formatting when using cue import or misleading positions being shown to the user.

CL 1169709 increases the robustness of cue export -o, which in some situations could ignore file errors or incorrectly replace an existing file without the -f flag.

CL 1173072 fixes a panic when using cue import --list with empty YAML input.

CL 1168436 updates the cue export documentation to add the missing cue and binary supported export formats.

Modules

A number of changes are included to support an experimental implementation of the proposed modules and package management support. These aren't enabled by default, and will be announced soon.

Full list of changes since v0.6.0

• cmd/cue: fix panic when importing empty files as a list by @uhthomas in b36b477
• internal/core/adt: change semantics of unary_op(basictype) by @mpvl in e1ceed4
• internal/core/adt: fix closedness bug by @mpvl in 886eefd
• internal/core/adt: add test to prepare for fix by @mpvl in 145764d
• CONTRIBUTING: mention the Slack channel and office hours calls by @mvdan in 16a0cda
• pkg/tool/exec: add a Run.mustSucceed parameter by @nickfiggins in 10c4796
• cue: explicitly document values are not safe for concurrent use by @mvdan in d805c22
• internal/ci: bump Go and GoReleaser for v0.7.0-rc.1 by @mvdan in 8f88dcb
• internal/slices: move from internal/mod/mvs/internal/slices by @rogpeppe in 0e51dbb
• internal/mod/internal/par: move from internal/mod/mvs/internal/par by @rogpeppe in fe0e04a
• internal/registrytest: do not check that dependencies exist by @rogpeppe in 99bdbad
• internal/mod/modimports: new package by @rogpeppe in 79033c2
• internal/core/export: fix a recent nil pointer regression by @rudifa in 141925a
• internal/core/adt: use state directly by @mpvl in bba2263
• internal/core/adt: expand on permutations for test by @mpvl in 596fdc9
• internal/cuetxtar: include diff between test and fallback test by @mpvl in 6774d0d
• internal/txtarfs: factor out from registrytest by @rogpeppe in 265a7a0
• internal/cueimports: export API and fix for CUE syntax by @rogpeppe in 47d6dce
• internal/cueimports: copy code from cue/load by @rogpeppe in 9e55783
• cue/load: remove unused functionality by @rogpeppe in b727189
• all: replace html/template by text/template by @mvdan in 361ae4f
• encoding/protobuf: support field_behavior "required" option by @howardjohn in 80b0eee
• cmd/cue: fix incorrect position in yaml import by @uhthomas in d97b0c3
• all: adopt .gitignore for new delve binary naming by @mpvl in 68e9d5d
• internal/core/adt: make ListMarker Src an expression by @mpvl in f59894c
• internal/core/adt: changes needed for new evaluator by @mpvl in fb88beb
• cue: don't panic on Iterator.Selector.Index with lists by @mvdan in 9dd0f82
• cue: allow top level JSON marshalers in EncodeType by @rogpeppe in 8f796bf
• cue: support decoding into cue.Value fields by @rogpeppe in b25fc05
• internal/mod/mvs: discard copied par tests by @mvdan in 4e30101
• CONTRIBUTING: fix repo reference rendering by @cedricgc in 7a4ea86
• internal/e2e: make unique module names more robust by @mvdan in a404c89
• cmd/cue: support optional comments in get go by @uhthomas in 67ea9cf
• cmd/cue: ignore complex types in get go by @uhthomas in 112b0b2
• pkg/internal/builtintest: don't print superfluous newlines by @uhthomas in 8b32c49
• internal/tdtest: document how the name field is used by @rogpeppe in deab307
• internal/e2e: add a test for a private github repo by @mvdan in 4d2ba88
• internal/ci: add gcloud auth to the e2e tests by @mvdan in 95a88a5
• CONTRIBUTING: clarify two sections by @cedricgc in d947a69
• internal/e2e: add a "cue mod upload" end-to-end test with gcloud by @mvdan in a060f85
• internal/e2e: rename existing testscript and add a summary by @mvdan in 4dba8dd
• internal/e2e: tweak how we use secret env vars by @mvdan in d3efde7
• internal/mod/modfile: add copyright message by @rogpeppe in f9c8d98
• cmd/cue: cue mod upload...

v0.6.0

The main focus of this release is the introduction of required fields, as well as fixing a number of issues and regressions introduced in the v0.5.0 release.

As a reminder: users can register their projects with unity, our regression and performance testing setup. unity is used to ensure that a project's CUE evaluations do not unexpectedly stop working, or regress in terms of performance. unity continues to catch multiple issues with each release. Adding your project to unity not only guarantees that we will not break your tests (if we do, we will work with you to fix your CUE code), but it also helps to improve the quality of each CUE release. Follow this link to learn more about Unity, install it, or get in touch with any questions.

Thank you to @4ad, @Abirdcfly, @alexandear, @chee-zaram, @eraserhd, @ghostwheel42, @joanlopez, @jpluscplusm, @kcburge, @mpvl, @mvdan, @myitcv, @rogpeppe, @toshi0607, and @zeithaste for contributing to this release!

API

CL 543335 adds arch to set of injectable system variables understood by cue/load. The text at cue help injection explains how this in more detail.

CL 552142 adds support for zero values in cue.Value.Float64, which has the effect of fixing the error when attempting to use strconv.FormatFloat with a zero value.

CL 548783 fixes a long-standing bug to make HTML escaping in JSON an opt-in. This means that cue export now respects the --escape flag when set, and encoding/json only escapes HTML when HTMLEscape is used.

Language

Required fields

The main focus of the v0.6.0 release is the introduction of required fields.

CUE already supports the “optional field constraint”, denoted foo?: value.

Required fields add a “required field constraint”, denoted foo!: value, which is like foo?: value, but requires a regular field be provided for foo (a field foo: without !: or ?:).

We refer to optional field constraints and required field constraints collectively as “field constraints”.

As a general rule, all data and data templating should be defined with regular fields, whereas schemas would be defined in terms of field constraints. Of course, CUE being CUE, mixing these two fields is allowed: this rule is not a restriction but suggested as a matter of style and proper usage.

Here are some examples of how exclamation marks can be used to express things that were previously not possible.

#Def: {
    kind!: "def"
    intList!: [...int]
}

Using required fields can also result in better error messages. Consider this schema:

#Person: {
    name: string
    age?: int
}

Note that this is non-idiomatic, because our new guidelines suggest schemas should only be defined in terms of field constraints, but we will use this for illustration purposes.

Now consider this usage of #Person:

jack: #Person & {age: 3}

In data mode, the error message here is currently jack.name: incomplete value string, which does not provide much actionable information to the user to help them fix the problem.

Now consider how #Person looks with required fields, idiomatically only using field constraints:

#Person: {
    name!: string
    age?:  int
}

jack: #Person & {age: 3}

Now the error message reads:

jack.name: field is required but not present

which more closely reflects the underlying problem..

This error could be resolved by adding jack: name: "Jack".

For more details and background on the change, please see the original required fields proposal.

Other changes

Whilst it should not be a breaking change from a CUE perspective, we have upgraded to use github.com/cockroachdb/apd/v3. We have also increased apd.Context precision from 24 to 34.

CL 551207 adds support for making dynamic fields optional or required. For example, the following is now possible:

x:	"y"
(x):  "foo"
(x)?: "foo"
(x)!: "foo"

and yields:

x: "y"
y: "foo"

CL 546886 removes support for old-style :: definitions. This also includes deprecation support. In a similar vein, CL 547011 removes the last vestiges of <foo>: T. This was once the notation for pattern constraints.

Spec

Various bug fixes, with special thanks to @nicuveo for raising many of these.

Builtins

The following four functions have been added to the net package:

• PathEscape
• PathUnescape
• QueryEscape
• QueryUnescape

Thanks to @eraserhd for this change.

CL 549087 reimplements pkg/list.Sort. The resulting reduction in the number of allocations and other work gives rise to a ~80% reduction in running time against CUE benchmarks.

cmd/cue

CL 547212 improves the documentation for the -l flag passed to cue import. This addresses a frequent point of confusion in questions to GitHub Discussions and on Slack.

CL 550616 fixes cue get go to respect the --exclude flag for constants. This makes it possible to (for example) exclude all unexported identifiers from a cue get go run.

CL 555576 fixed an important bug where cmd/cue vet was not properly consuming all input data.

CL 556526 fixed a bug where CUE files beginning with an underscore were not being loaded when explicitly given as filename arguments.

WebAssembly (Wasm)

We have added preliminary support for Wasm. Users can compile code from any language and toolchain that supports Wasm into plugins that are dynamically loaded by CUE. Users can then call and use functions from these Wasm modules, just like they can use standard library functions.

See the documentation at cuelang.org/go/cue/interpreter/wasm to learn more about Wasm and its current limitations.

Changelog

Expand for the full changelog

• internal/core/dep: always introduce Environment for Comprehension Value to fix panic by @mpvl in 99e8578
• internal/core/dep: adapt Recurse to mimic dep.Visit by @mvdan in 97d7109
• internal/ci: use cue-lang/cuelang.org workflow_dispatch for tip by @myitcv in 33b7393
• cmd/cue: fix regression when injecting vars with cue cmd by @mvdan in da3ee7d
• internal/core/adt: lets of comprehensions are multi lets by @mpvl in 43b4674
• internal/core/adt: always mark let in comprehension as "complete" by @mpvl in 5123737
• cue/ast: clarify what is a valid identifier by @mvdan in c4ced5e
• internal/core/adt: find late-stage conflict errors by @mpvl in 0c48837
• internal/core/adt: add tests to fix 2481 by @mpvl in d3444e3
• internal/core/adt: fix stack overflow with comprehensions by @mpvl in 33071c0
• deps: upgrade to go1.20.6 for release builds by @myitcv in 538eb9c
• internal/core/compile: reject "_" as an alias name by @mvdan in bcbe967
• all: remove a few more unused bits of code by @mvdan in c8e51a5
• internal/core/dep: prevent passing "empty" vertices to tools/flow by @mvdan in c3eda1d
• internal/core/adt: do not delay processing of fields by @mpvl in e7cfb50
• internal/core/adt: add tests for upcoming issues by @mpvl in cabc1db
• CONTRIBUTING: fix references to the old Unity repo by @mvdan in 23e265d
• cmd/cue: print returned error by @mpvl in 450ff90
• deduplicate content between README and CONTRIBUTING by @mvdan in 4c48cee
• move doc/contribute.md to CONTRIBUTING.md by @mvdan in db28ec6
• readme: warn new contributors not to use SSH auth by @jpluscplusm in f4c5639
• internal/core/adt: pass state to addList by @mpvl in b38aece
• internal/core/adt: set scalar more proactively by @mpvl in ab15bb8
• internal/core/adt: better error message for cycles by @mpvl in 5e84a6d
• internal/ci: count number of Signed-off-by lines by @myitcv in 4ba1b15
• internal: remove a few unused APIs by @mvdan in 1c1a7b3
• cue/load: fix Instances race by @rogpeppe in c10d7cb
• cue/load: allow _*.cue files when explicitly listed by @myitcv in b16f6ce
• internal/core/dep: fix recursive let processing by @mpvl in ca1f812
• internal/core/dep: add Dependency.Recurse by @mpvl in f6e49bd
• internal/core/dep: prepare for public APi by @mpvl in 0554d4e
• internal/core/adt: mark comprehension key as dynamic by @mpvl in d4fd104...

v0.6.0-rc.1

Thank you to @mpvl, and @mvdan for contributing to this release!

Changelog

• internal/core/adt: lets of comprehensions are multi lets by @mpvl in 43b4674
• internal/core/adt: always mark let in comprehension as "complete" by @mpvl in 5123737
• cue/ast: clarify what is a valid identifier by @mvdan in c4ced5e
• internal/core/adt: find late-stage conflict errors by @mpvl in 0c48837
• internal/core/adt: add tests to fix 2481 by @mpvl in d3444e3

v0.6.0-beta.1

Thank you to @jpluscplusm, @mpvl, @mvdan, @myitcv, and @rogpeppe for contributing to this release!

Changelog

• internal/core/adt: fix stack overflow with comprehensions by @mpvl in 33071c0
• deps: upgrade to go1.20.6 for release builds by @myitcv in 538eb9c
• internal/core/compile: reject "_" as an alias name by @mvdan in bcbe967
• all: remove a few more unused bits of code by @mvdan in c8e51a5
• internal/core/dep: prevent passing "empty" vertices to tools/flow by @mvdan in c3eda1d
• internal/core/adt: do not delay processing of fields by @mpvl in e7cfb50
• internal/core/adt: add tests for upcoming issues by @mpvl in cabc1db
• CONTRIBUTING: fix references to the old Unity repo by @mvdan in 23e265d
• cmd/cue: print returned error by @mpvl in 450ff90
• deduplicate content between README and CONTRIBUTING by @mvdan in 4c48cee
• move doc/contribute.md to CONTRIBUTING.md by @mvdan in db28ec6
• readme: warn new contributors not to use SSH auth by @jpluscplusm in f4c5639
• internal/core/adt: pass state to addList by @mpvl in b38aece
• internal/core/adt: set scalar more proactively by @mpvl in ab15bb8
• internal/core/adt: better error message for cycles by @mpvl in 5e84a6d
• internal/ci: count number of Signed-off-by lines by @myitcv in 4ba1b15
• internal: remove a few unused APIs by @mvdan in 1c1a7b3
• cue/load: fix Instances race by @rogpeppe in c10d7cb
• cue/load: allow _*.cue files when explicitly listed by @myitcv in b16f6ce
• internal/core/dep: fix recursive let processing by @mpvl in ca1f812
• internal/core/dep: add Dependency.Recurse by @mpvl in f6e49bd
• internal/core/dep: prepare for public APi by @mpvl in 0554d4e
• internal/core/adt: mark comprehension key as dynamic by @mpvl in d4fd104
• internal/core/dep: further improve dependency analysis by @mpvl in f513654
• internal/core/dep: add more tests ahead of changes by @mpvl in 202eb3e
• cue: fix some low hanging fruit from staticcheck by @mvdan in 41b38fe
• all: replace godebug/diff with go-cmp/cmp by @mvdan in fa6e229
• encoding/gocode/gocodec: use different generic type signature by @rogpeppe in 1a98ee1
• internal/core/dep: add references and line in test output by @mpvl in 518bafa
• internal: use Scanner API to parse attributes by @rogpeppe in 3400e74
• internal/core/dep: require importPath argument by @mpvl in 4fcf29c
• internal/core/dep: handle inline composite literals by @mpvl in 07ae1be
• cue: report useful error for uninitalized Context by @mpvl in b8e2bf0
• cue: add extra TestAttributeString test by @rogpeppe in 876af7a
• internal/core/export: avoid _#def if unnecessary by @mpvl in d112f64
• cmd/cue/cmd: add test for issue 2251 by @mpvl in 19af1e9
• cue/scanner: add Offset method and export DontInsertCommas by @rogpeppe in de9e760
• cue: add more attribute tests by @rogpeppe in d8a0e26
• internal: change attribute tests to be less implementation-specific by @rogpeppe in 7741d21
• encoding/gocode/gocodec: allow cue.Context by @rogpeppe in 1e76095
• internal/core/dep: add and modify some tests for issue 2247 by @mpvl in 24e8d56
• internal/core/export: fix panic converting int label to identifier by @mpvl in 24f3fb3
• encoding/json: check that required fields are present by @mpvl in adbe80e
• pkg/encoding/json: add tests that needs to be fixed by @mpvl in 8df1034
• pkg/encoding/json: add more structure to tests by @mpvl in 6922e14
• cue: document UnifyAccept by @mpvl in c054274
• internal/core/export: preserve original field aliases by @mpvl in 7be6224
• internal/core/compile: fix offset error in pattern alias by @mpvl in 914d64d
• internal/ci: consistently trigger Unity as porcuepine by @mvdan in a0510ac

v0.6.0-alpha.2

As announced on the 2023-06-20 Community Call, this release includes preliminary support for Wasm. It also includes a number of bug fixes, and a couple of builtin API additions.

Thank you to @4ad, @chee-zaram, @eraserhd, @ghostwheel42, @mpvl, @mvdan, @myitcv, @rogpeppe, and @toshi0607 for contributing to this release!

Language

No changes to note. There are various bug fixes related to required fields, which were first introduced and explained in the v0.6.0-alpha.1 release.

API

No changes to note.

Project

No changes to note.

Core Evaluator

Whilst it should not be a breaking change from a CUE perspective, we have upgraded to use github.com/cockroachdb/apd/v3. We have also increased apd.Context precision from 24 to 34.

Encoders

No changes to note.

Builtins

The following four functions have been added to the net package:

• PathEscape
• PathUnescape
• QueryEscape
• QueryUnescape

Thanks to @eraserhd for this change.

cmd/cue

https://cuelang.org/cl/555576 fixed an important bug where cmd/cue vet was not properly consuming all input data.

Spec

Various bug fixes, with special thanks to @nicuveo for raising many of these.

WebAssembly (Wasm)

We have added preliminary support for Wasm. Users can compile code from any language and toolchain that supports Wasm into plugins that are dynamically loaded by CUE. Users can then call and use functions from these Wasm modules, just like they can use standard library functions.

See the documentation at cuelang.org/go/cue/interpreter/wasm to learn more about Wasm and its current limitations.

Changelog

• internal/ci: bump Go and GoReleaser prior to alpha.2 by @mvdan in b6a32da
• correct misleading phrases in documentation by @chee-zaram in 0755dc8
• internal/core/export: avoid panics on computed comprehensions by @mvdan in 6db2253
• internal/core/adt: return from Accept earlier for hidden fields by @mpvl in c724eef
• cue/ast: fix staticcheck warnings by @mvdan in cc6b544
• internal/core/adt: add missed error by @mpvl in 19749aa
• internal/core/adt: fail on missing required field in comprehension by @mpvl in a991bd6
• cue: allow iterating hidden fields and definitions of embedded scalars by @mpvl in de44cab
• internal/core/adt: add position for required field error by @mpvl in 452e143
• internal/core/validate: allow required fields in definitions by @mpvl in 317f88d
• cue/ast: support specifying required fields in NewStruct by @mpvl in 1432683
• tools/flow: check cycle marker by @mpvl in 9a566ed
• spec: add definition for dynamic fields by @mpvl in 3a635fc
• spec: various fixes by @mpvl in e46324e
• cmd/cue: properly consume all data input in vet by @mvdan in b22ec6a
• cmd/cue: add AllocBytes and AllocObjects to stats by @mvdan in 03d85b8
• increase apd.Context precision from 24 to 34 by @mvdan in 7bcbcd6
• update to cockroachdb/apd v3 by @mvdan in 4728223
• cue: add a "decimal" benchmark by @mvdan in f3be10a
• deps: switch to use quicktest instead of testify by @myitcv in 12908b3
• cue/testdata: add regression test for #2244 by @mvdan in 442c023
• internal/cuetest: remove support for CUE_LONG by @mvdan in cb461fd
• all: sed-replace all trivial uses of ioutil by @mvdan in f56ccf0
• pkg/path: remove unused code by @rogpeppe in ab48c0f
• encoding/protobuf/jsonpb: remove unused code by @rogpeppe in b2476bd
• encoding/protobuf: remove unused code by @rogpeppe in b41f26f
• tools/fix: remove unused code by @rogpeppe in 1f888b5
• all: use strings.Builder in a few more places by @mvdan in 97cba0b
• internal/encoding: remove unused code by @rogpeppe in 251790e
• encoding/openapi: remove unused code by @rogpeppe in 36645d1
• all: go fix to remove old build tags by @mvdan in 8868ae6
• internal/ci: trigger Unity on master as porcuepine by @mvdan in 2148aed
• internal/ci: remove cue-unity/unity from codereview.cfg by @mvdan in c341427
• internal/core/conver: make use of reflect.Value.IsZero by @mvdan in 5318af1
• internal/ci: further tidy up by @myitcv in 82f96c5
• cue/interpreter/wasm: add user-level docs by @4ad in 40fb30d
• cue/interpreter/wasm: add Wasm support for abi=c by @4ad in 0520a3f
• internal/pkg: move pkg/internal to internal/pkg by @4ad in 88922d1
• codereview.cfg: add cue-unity-new by @mvdan in a670427
• doc/tutorial: fix typo by @toshi0607 in 2304cb5
• cue/load: replace go-internal/semver with x/mod by @mvdan in 6c92698
• internal/cuetxtar: print error details by @rogpeppe in 80d6ea0
• tools/flow: support tasks inside slices by @rogpeppe in 6e0366f
• internal/core/convert: split the cue tag commas after the last quote by @ghostwheel42 in 49a64f4
• pkg/net: add Path{E,Une}sape and Query{E,Une}scape by @eraserhd in 9082446
• cmd/cue: refactor our use of cobra by @mvdan in 59080b6
• cmd/cue: do not defer closing files in fmt's loop by @mvdan in 7f79469
• cmd/cue: check all file I/O errors in fmt by @mvdan in 8c6d9ba
• docs/tutorial: revert to using cue eval -i by @myitcv in 5e66c83
• all: move pkg/gen and internal/cmd/qgo to build-ignored files by @mvdan in bad36b2

v0.6.0-alpha.1

The main focus of this release is the introduction of required fields.

As a reminder: users can register their projects with unity, our regression and performance testing setup. unity is used to ensure that a project's CUE evaluations do not unexpectedly stop working, or regress in terms of performance. unity continues to catch multiple issues with each release. Adding your project to unity not only guarantees that we will not break your tests (if we do, we will work with you to fix your CUE code), but it also helps to improve the quality of each CUE release. We are currently in the process of releasing Unity as a GitHub App on the public Marketplace. Sign up to receive updates; we will be in touch as soon as the GitHub App is ready.

Thank you to @4ad, @Abirdcfly, @alexandear, @joanlopez, @jpluscplusm, @kcburge, @mpvl, @mvdan, @myitcv, @rogpeppe, and @zeithaste for contributing to this release!

API

CL 543335 adds arch to set of injectable system variables understood by cue/load. The text at cue help injection explains how this in more detail.

CL 552142 adds support for zero values in cue.Value.Float64, which has the effect of fixing the error when attempting to use strconv.FormatFloat with a zero value.

CL 548783 fixes a long-standing bug to make HTML escaping in JSON an opt-in. This means that cue export now respects the --escape flag when set, and encoding/json only escapes HTML when HTMLEscape is used.

Language

Required fields

The main focus of the v0.6.0 release is the introduction of required fields.

CUE already supports the “optional field constraint”, denoted foo?: value.

Required fields add a “required field constraint”, denoted foo!: value, which is like foo?: value, but requires a regular field be provided for foo (a field foo: without !: or ?:).

We refer to optional field constraints and required field constraints collectively as “field constraints”.

As a general rule, all data and data templating should be defined with regular fields, whereas schemas would be defined in terms of field constraints. Of course, CUE being CUE, mixing these two fields is allowed: this rule is not a restriction but suggested as a matter of style and proper usage.

Here are some examples of how exclamation marks can be used to express things that were previously not possible.

#Def: {
	kind!: "def"
	intList!: [...int]
}

Using required fields can also result in better error messages. Consider this schema:

#Person: {
	name: string
	age?: int
}

Note that this is non-idiomatic, because our new guidelines suggest schemas should only be defined in terms of field constraints, but we will use this for illustration purposes.

Now consider this usage of #Person:

jack: #Person & {age: 3}

In data mode, the error message here is currently jack.name: incomplete value string, which does not provide much actionable information to the user to help them fix the problem.

Now consider how #Person looks with required fields, idiomatically only using field constraints:

#Person: {
	name!: string
	age?:  int
}

jack: #Person & {age: 3}

Now the error message reads:

jack.name: field is required but not present

which more closely reflects the underlying problem..

This error could be resolved by adding jack: name: "Jack".

For more details and background on the change, please see the original required fields proposal.

Other changes

CL 551207 adds support for making dynamic fields optional or required. For example, the following is now possible:

x:    "y"
(x):  "foo"
(x)?: "foo"
(x)!: "foo"

and yields:

x: "y"
y: "foo"

CL 546886 removes support for old-style :: definitions. This also includes deprecation support. In a similar vein, CL 547011 removes the last vestiges of <foo>: T. This was once the notation for pattern constraints.

Builtins

CL 549087 reimplements pkg/list.Sort. The resulting reduction in the number of allocations and other work gives rise to a ~80% reduction in running time against CUE benchmarks.

cmd/cue

CL 547212 improves the documentation for the -l flag passed to cue import. This addresses a frequent point of confusion in questions to GitHub Discussions and on Slack.

CL 550616 fixes cue get go to respect the --exclude flag for constants. This makes it possible to (for example) exclude all unexported identifiers from a cue get go run.

Changelog

• internal/ci: use go1.20.3 for releases by @myitcv in 9db1b95
• internal/tdtest: use field name in default error message by @mpvl in 401cab2
• internal/core/adt: fix Definition closedness bug in API by @mpvl in c3c8eb7
• update txtpbfmt to drop indirect dependency on glog by @mvdan in 410eea1
• internal/core/adt: fix NotEqualOp for lists with differing lengths by @mvdan in 6e185bb
• cmd/cue: remove skipped test scripts for fixing old definitions by @mvdan in 9505e90
• all: make HTML escaping in JSON an opt-in by @mvdan in 48207fb
• all: add tests to cover HTML escaping via encoding/json by @mvdan in 1aefd09
• cue/build: remove context.Context field by @rogpeppe in 1932295
• cue/ast: fix IsValidIdent for _0 by @rogpeppe in 9422314
• internal/diff: remove unused code by @rogpeppe in 9e22283
• cmd/cue/cmd: remove unused code by @rogpeppe in aa8c66f
• internal/core/adt: fix let closedness issue with API use by @mpvl in 731ddd9
• internal/core/export: avoid introducing shadowing by @mpvl in 1c9a3b2
• internal/tdtest: new table-driven test package by @mpvl in 102150d
• cue: more support for required fields for selectors by @mpvl in 0e16084
• internal/core: fail on missing required only when concrete by @mpvl in f85172a
• cue: fail on required field in various cases by @mpvl in e7d7c6a
• internal/core/adt: add wrapper for inline Vertex by @mpvl in 071c4ab
• internal/core/export: always export required fields by @mpvl in d5dd9ec
• cue: remove field type by @mpvl in d05d9c1
• cmd/cue: avoid loading the current package in cue help by @mvdan in 143b102
• all: remove duplicate words in comments by @Abirdcfly in d8c71fa
• internal/ci: fix cache eviction by @myitcv in 53d3bc7
• internal/ci: address review feedback from CL 551352 by @myitcv in 06397b5
• cue: add support for zero values in Value.Float64 by @kcburge in 8d31dad
• cue/load/tags: add arch to set of injectable system variables by @jpluscplusm in 752b8e4
• internal/ci: allow trybot workflows to be workflow_dispatch-ed by @myitcv in 09221a3
• internal/core/adt: required fields checked for closedness by @mpvl in 42580d5
• doc/ref/spec.md: fix uint16 and uint32 values range by @joanlopez in c67fe60
• all: correct typos and duplicate words in comments by @alexandear in 2e28b01
• cue/errors: remove redundant Sort call in list.sanitize by @zeithaste in f106acf
• internal/ci: refactor CI workflows by @myitcv in d71a76c
• internal/ci: remove hard-codings in base package by @myitcv in f881884
• internal/core/adt: change addList signature by @mpvl in 0c840f6
• internal/core/adt: introduce ArcNotPresent by @mpvl in e3a7fcf
• internal/core/adt: use value array for Disjunction by @mpvl in 166ac88
• internal/core/adt: specialize injectComprehension by @mpvl in fa4f2d3
• internal/core/adt: hoist comprehension loop body by @mpvl in e0f5681
• internal/ci: move evict_caches to base by @myitcv in cef63a3
• internal/ci: remove mention of matrix and runner from base by @myitcv in 3ed9f92
• internal/ci: remove trybot from push branches by @myitcv in 5cd427a
• internal/ci: refactor base caching pattern by @myitcv in 92b28c4
• internal/ci: drop pullThroughProxy step by @myitcv in 79aceb7
• cue/errors: better vetting of error formats by @rogpeppe in c630554
• internal/ci: improve push_tip_to_trybot defaults by @myitcv in be0601b
• internal/c...

v0.5.0

This release focuses on changes to the implementation of comprehensions, the algorithm for detecting cycles in CUE, and the reimplementation of let comprehensions as fields.

Also included are a number of bug fixes and changes to help the KubeVela project upgrade off CUE v0.2.2 to the latest release. v0.5.0 is the first release since v0.4.3 back in April 2022. It has been a long time in the making, and is comprised of 277 commits.

It is worth noting that we do not plan nor wish for such long gaps between releases. For more details on upcoming releases and steps we are taking to increase release cadence as well as our confidence in each release, please see this discussion.

As a reminder: users can register their projects with unity, our regression and performance testing setup. unity is used to ensure that a project's CUE evaluations do not unexpectedly stop working, or regress in terms of performance. unity continues to catch multiple issues with each release. Adding your project to unity not only guarantees that we will not break your tests (if we do, we will work with you to fix your CUE code), but it also helps to improve the quality of each CUE release. We are currently in the process of releasing Unity as a GitHub App on the public Marketplace. Sign up to receive updates; we will be in touch as soon as the GitHub App is ready.

Thank you to @djosephsen, @joaopapereira, @mpvl, @mvdan, @myitcv, @qequ, @rogpeppe, @rytswd, @satotake, @spachava753, and @tmm1 for contributing to this release!

Project-level changes

Various CI-related changes have been made to simplify and streamline the GerritHub-based CI setup across the various CUE repositories. A common base package is vendored into each repository and used as a template for GitHub workflows and other configuration. For those interested in studying the setup, please see the files beneath internal/ci.

This release drops support for go1.16.x , go1.17.x and go1.18.x, is tested against go1.19.x and go1.20.x, and is built using go1.20.2.

Our dependency on github.com/rogpeppe/go-internal is upgraded to v1.9.0 and various testscript-related improvements have been made to CUE's tests.

GoReleaser has been upgrade to v1.16.2 as part of the release workflow. In a related change, CL 546920 moves us to a CUE-based source configuration for GoReleaser. This allows us to define the concept of 'latest', state which indicates whether we should perform the homebrew release and the Docker manifest latest steps. This fixes a long-standing bug whereby homebrew taps and Docker images were created for the latest release, regardless of whether that was a pre-release, or indeed a patch version of a non-current minor version.

As of CL 547507, CUE build artefacts published on GitHub are entirely reproducible, thanks to Go's support for reproducible, verifiable and verified builds.

Encoders

When cue import-ing YAML, the -R recursive flag now only recursively interprets text as YAML if it contains a newline CL 536902.

API

Various APIs have been transitioned to drop use of the deprecated cue.Runtime type, as well as migrating away from cue.Instance via cue.InstanceOrValue. Some use of these deprecated types remains: #1035, #1735, #1806. These will be tidied up in later releases as the API surface matures towards v1.0.0.

The cue package now supports InlineImports for Value.Syntax CL 541561. This leverages new support within the internal exporter to creating self-contained CUE. This change also deprecates Value.ResolveReferences().

Package cue adds Selector.Type to make it easier to pull out selector-specific information from a selector (for example, it makes it possible and cheap to retrieve the unquoted string label from a regular field selector) CL 542738.

Package cue adds Value.BuildInstance(). If a cue.Value exactly represents a package, BuildInstance() returns the build instance corresponding to the value CL 542769.

cue/stats is introduced as a new package for getting CUE operation statistics. The functionality is hoisted from internal/core/adt, but the package should for all practical purposes be seen as a new implementation. The stats have been aggregated as experimental functionality to the tools/flow package.

CL 546763 disallowed deprecated language features in the Go API. These features had long been deprecated in cmd/cue. This means that API operations will now fail on deprecated features whereas before they were handled in compatibility mode. The parser.FromVersion feature can be used to reenable these deprecated features.

Language

This release does not introduce any language changes. CL 537264 made some minor adjustments to the language specification to improve readability.

CL 541457 removes support for re-writing of old-style let declarations. This had already been deprecated for a long time and this change was made to enable support for aliases of embeddings.

Core evaluator

Per-field comprehensions

One of the major changes introduced in this release is how comprehensions are handled. CL 529524 move the evaluator to a model of per-field comprehensions.

Comprehensions are expanded for, if, and let clauses that yield zero or more structs to be embedded in the enclosing list or struct.

CUE does not allow cyclic dependencies between comprehensions if they result in an ambiguous set of fields. It does allow cascading insertions as in:

a?: int
b?: int
if a != _|_ {
    b: 2
}
if b != _|_ {
    c: 3
    d: 4
}

The way this works is that for fields with a fixed prefix path in a comprehension value, the comprehension is assigned to these respective fields.

More concretely, the above example is rewritten to:

a?: int
b: if a != _|_ { 2 }
c: if b != _|_ { 3 }
d: if b != _|_ { 4 }

where the fields with if clause are only inserted if their condition resolves to true. Note that this is not valid CUE; it may be in the future.

With this rewrite, dependencies in comprehension expressions are handled analogously to other expressions.

Note that a single comprehension may be distributed across multiple fields. The evaluator will ensure, however, that a comprehension is only evaluated once.

Analogous to reference cycles, as of CL 544129 we allow comprehensions to use the struct in which they are defined as source, as long as they do not introduce new fields, which would alter the source on which they iterate.

Reimplementation of cycle detection

The new algorithm is inspired by the cycle detection used by Tomabechi's [^Tomabechi] and Van Lohuizen's [^VanLohuizen] graph unification algorithms.

Unlike with traditional graph unification, however, CUE uses references, which, unlike node equivalence, are unidirectional. This means that the technique to track equivalence through dereference, as common in graph unification algorithms like Tomabechi's, does not work unaltered.

The unidirectional nature of references imply that each reference equates an exact copy of the value it points to. This renders the original approach of node-pointer equivalence useless.

The new algorithm, a variant of Tomabechi's and Van Lohuizen's algorithm, looks at references equality on conjuncts, instead of node equality. This simplifies the accounting and allows for far simpler and precise accounting when using references instead of the traditional node equivalence.

More background info on cycles

We have to define when a cycle is detected. CUE implementations MUST report an error upon a structural cycle, and SHOULD report cycles at the shortest possible paths at which they occur, but MAY report these at deeper paths. For instance, the following CUE has a structural cycle:

f: g: f

The shortest path at which the cycle can be reported is f.g, but as all failed configurations are logically equal, it is fine for implementations to report them at f.g.g, for instance.

It is not, however, correct to assume that a reference to a parent is always a cycle. Consider this case:

a: [string]: b: a

Even though reference a refers to a parent node, the cycle needs to be fed by a concrete field in struct a to persist, meaning it cannot result in a cycle as defined in the spec as it is defined here. Note however, that a specialization of this configuration can result in a cycle. Consider

a: [string]: b: a
a: c: _

Here reference a is guaranteed to result in a structural cycle, as field c will match the pattern constraint unconditionally.

In other words, it is not possible to exclude tracking references across pattern constraints from cycle checking.

It is tempting to try to find a complete set of these edge cases with the aim to statically determine cases in which this occurs. But as Carpenter [^Carpenter] demonstrates, it is possible for cycles to be created as a result of unifying two graphs that are themselves acyclic. The following example is a translation of Carpenters example to CUE:

y: {
	f: h: g
	g: _
}
x: {
	f: _
	g: f
}

Even though the above contains no cycles, the resu...

v0.5.0-rc.1

Full release notes to follow in the final v0.5.0 release.

• internal/ci: move evict_caches to base by @myitcv in 584b21e
• internal/ci: remove mention of matrix and runner from base by @myitcv in 7c07680
• internal/ci: remove trybot from push branches by @myitcv in 9f3af8f
• internal/ci: refactor base caching pattern by @myitcv in 390bca5
• internal/ci: drop pullThroughProxy step by @myitcv in a14fbf4
• [release-branch.v0.5] internal/ci: bring inline with be0601b by @myitcv in f4d04a7
• internal/ci: bump pinned version of Go for releases by @mvdan in 374a2cc
• internal/core/adt: revert fix for 2209 by @mpvl in c0421b4
• internal/core/adt: fix possible infinte loop for API usage by @mpvl in 1ce9120
• internal/core/adt: fix regression in comparing to bottom by @mpvl in 87c79ef
• internal/core/adt: remove unnecessary check by @mpvl in 28321f6
• internal/core/adt: detect closed lists as fatal errors by @mpvl in 5d42c6c
• internal/core/adt: sharpen condition for scalar setting by @mpvl in 3ba76ba

v0.5.0-beta.5

Full release notes to follow in the final v0.5.0 release.

• internal/ci: fix goreleaser in non-snapshot mode again by @mvdan in 576d0e4
• internal/core/adt: fix 2235 by @mpvl in ca411de
• internal/core/adt: add tests for 2235 by @mpvl in 89693e1
• cmd/cue: include Go in the version command by @mvdan in 9b1f248
• internal/ci: fix and test the release workflow on branches by @mvdan in 63749fe
• cmd/cue: add evaluator stats support by @mpvl in d573e0c
• cmd/cue: add tests for issue 345 by @mpvl in 352a736
• internal/core/adt: filter more disjunctions by @mpvl in 5a6bd19
• internal/core/adt: add tests for 2209 by @mpvl in 6b45d7e
• internal/core/adt: improve mechanism for conjunct insertion by @mpvl in 887f564
• internal/core/adt: add tests for conjunct insertion by @mpvl in 81e4cba
• internal/core/adt: only set scalar if not already defined by @mpvl in f23e3d5
• github: bring the bug report template up to date by @mvdan in b75fdc8
• internal/ci: make github CLI shell invocations safer by @mvdan in 94765d6
• internal/core: clone let upon use of for var references by @mpvl in 3a79e96
• internal/core/debug: change notation for "multi" lets by @mpvl in fd05908
• internal/core/adt: don't add errors to "void" arcs by @mpvl in 0b41dbb
• cue/testdata/benchmarks: add benchmark for let key by @mpvl in a8d7c59
• internal/core/adt: evaluate expressions directly by @mpvl in 8fb13d4
• internal/core/eval: node should not finalize itself mid flight by @mpvl in 9d27e0d
• cmd/cue: don't panic in vet if a schema has incomplete errors by @mvdan in f6841d1
• internal/ci: deploy tip.cuelang.org via repository_dispatch by @myitcv in 632cc62
• internal/ci: enable push_to_tip_trybot by @myitcv in fe692c5
• internal/ci: apply various CI-related changes from cuelang.org repo by @myitcv in 4eaa946
• internal/ci/goreleaser: actually make builds reproducible by @myitcv in 45aa5c0
• internal/ci/goreleaser: make builds reproducible by @myitcv in 16edb00
• cue/export/extract: fix comments with comprehensions by @mpvl in 959dc46
• internal/core/adt: don't evaluate inlined structs prematurely by @mpvl in 45ff282
• internal/core/adt: add tests for issue #2163 by @mpvl in a9f4fc3
• internal/ci: review follow-up changes to goreleaser_tool.cue by @mvdan in 7ea44a3
• internal/ci: switch to CUE based definition for goreleaser by @myitcv in 79a476b
• internal/ci: import for current goreleaser configuration by @myitcv in e69c517
• internal/ci: upgrade to later GitHub workflow schema by @myitcv in f036b34
• internal/ci: upgrade to latest GoReleaser version by @myitcv in 248f2b3
• goreleaser: formatting changes in Yaml file by @myitcv in e93b5ed
• internal/ci: use newer versions of the docker github actions by @mvdan in f56600d

v0.5.0-beta.2

Full release notes to follow in the final v0.5.0 release.

• internal/core/adt: base error mode on requested state by @mpvl in 214e112
• internal/core/adt: only set validation result when safe by @mpvl in 88857e0
• internal/core/adt: pass desired state to more functions by @mpvl in 55cdb74
• internal/core/adt: pass more state around by @mpvl in 3dfeb41
• internal/core/adt: pass state to validation by @mpvl in 0954c6f
• internal/core/adt: pass state to evaluate by @mpvl in 7d50228
• internal/core/adt: avoid duplicate insertion of conjuncts by @mpvl in d341814
• internal/core/adt: tighten uniqueness of conjuncts by @mpvl in a68c6ce
• cue/parser: disallow deprecated language features in API by @mpvl in 67ace83
• internal/core/adt: remove nested comprehension special casing by @mpvl in 3c0b2ec
• README: fix the stale GitHub Actions badge by @mvdan in 4fee020
• internal/core/adt: disambiguate let cache by @mpvl in 43fc681
• internal/core/adt: update stats to fix build breakage by @mpvl in a800e6f
• internal/core/adt: merge Conjunct and Partial processing by @mpvl in 121894c
• internal/core/adt: remove assertion by @mpvl in 129c343
• internal/core/adt: add test cases for issue 2119 by @mpvl in 2499719
• internal/core/adt: add tests related to Issue #2113 by @mpvl in 525648d
• internal/core/adt: evaluate let fields earlier in comprehensions by @mpvl in e5fb5af
• internal/core/adt: detect cycles in patterns by @mpvl in c048135
• cue: update Value.Source doc comment by @rogpeppe in 45ce7a3
• tools/flow: add debug support via CUE_DEBUG_TOOLS_FLOW by @myitcv in d85f950
• internal/core/adt: fix comprehension cycle regression by @mpvl in 32bfffd
• internal/core/adt: add tests to address regressions by @mpvl in 82bf036
• internal/core/adt: split some tests into files by @mpvl in 69ee994
• infra: upgrade goreleaser by @myitcv in 7bdeb14