v0.4.1-beta.6

Bug fixes

After an extended summer break following CUE's migration to cue-lang/cue, this release mainly focuses on bug fixes.

Thank you to @mvdan, @ameowlia, @antong, @slewiskelly, @hrvolapeter, @caarlos0, @yujunz, and @chai2010 for contributing to this release!

Project-level changes

A number of release-related issues that were missed in the migration from cuelang/cue to cue-lang/cue have been fixed. GoReleaser now correctly marks prereleases as such.

The CUE project now uses v1 of GoReleaser and as such we produce both amd64 and arm64 release assets for linux, windows and darwin, with accompanying amd64 and arm64 Docker images and manifests. The homebrew-tap recipe has also been updated to avoid an error message reported via https://github.com/cue-lang/homebrew-tap/issues/1.

As we near the halfway point to go1.18, go1.15 has been removed from the build matrix. The project's policy for previous versions of Go will be formalized in the near future, but is likely to be that each major Go release is supported until there are two newer major releases, with a 3 month grace period following the release of a new major version (3 months being half a 6-month release cycle) before support for the oldest version is dropped. Correspondingly, go install instructions in the README have been updated to drop support for the old-style GO111MODULE form, instead preferring the go install cuelang.org/go@$version form.

Ahead of a live stream focussing on contributing to the CUE project back in November, the contribution guide was updated to provide more detail around the PR workflow. Further improvements will be made to reflect feedback following the livestream.

Encoders

An upgrade to the version of gopkg.in/yaml.v3 used by the CUE project may result in some formatting changes for Yaml output. This new version now also supports better string escaping for YAML v1.1 backwards compatibility support. However, it does not seem to apply these escapes consistently, so for now we added our own implementation (see also #1368 and #1076)

For instance, some unquoted strings might, when consumed by some old YAML parsers, be interpreted as values other than strings (float 60, bool, date-time...). Now:

service: "2222:22" 

is consistently exported to Yaml as:

service: "2222:22" // before RHS was 2222:22

If this more aggressive quoting still proves insufficient, we can consider quoting all string values by default.

cmd/cue

Thanks to @chai2010, a working directory can be set for an exec.Run command in the tooling layer:

import "tool/exec"

command: ls: exec.Run & {
	cmd: "ls"
	dir: "/tmp"
}

cue eval exporting previously resulted in some strange results that involved let expressions (#1116). These bugs have been fixed and corresponding updates made to test expectations in unity. Similarly, a cue def bug where multiple list conjuncts resulted in an output containing a {} conjunct is now fixed (#1284).

API

cue.ResolveReferences has been fixed to work for the KubeVela project, although it is anticipated a better long-term solution is required.

@ameowlia helped fix an issue with a test example for cue.Context.

Various other bug fixes.

Evaluator

91476ab fixes a number of issues related to the use of aliases in lists (#1308), by introducing a new scope with list values. A side effect of this change is that value aliases now work with list values:

y: X = [1, 2, X[0]]   // y: [1, 2, 1]

A long-standing issue with defaults and string and number lists (#1025, #1404) has been fixed such that the following example no longer results in an incomplete-error:

exec cue export

-- x.cue --
package x

import "strings"

#something: {
	image:    string | *"some default"
	toString: strings.Join([image], "")
}

x: #something

A bug where comprehensions inside a definition resulted in an error (#1404) has now been fixed, meaning the following is correctly allowed:

#D: {
	a: foo: 123
	b: {for k, v in a {(k): v}}
}

An important bug related to the handling of default values that result in error has been fixed (#1304 and #1257).

Various unexpected panics have been fixed, including a panic in the case of circular let declarations (#1042).

Language

This release makes no changes to the language.

The CUE spec now correctly defines $ as an identifier (#1264).

Changelog

• 547e5cc ci: configure GoReleaser to mark prereleases as such
• 6e68ca4 ci: only trigger new version events after successful new version
• 795719e ci: fix GoReleaser arm64 template details
• b68f4c3 ci: fix target for homebrew tap
• 02556bc ci: fix up GoReleaser image template names
• e7da62f ci: use GoReleaser v1.0.0
• f68e000 cue: support ResolveReferences
• d413bb8 internal/core/export: keep open lists open
• 9a040dc internal/core/export: fix let exporting
• 910ff4d pkg: fix default issue for string and numbers lists
• 6c30678 internal/core/adt: fix regression of dynamic closed fields in definitions
• 9e7d4d6 internal/core/export: treat empty data vertices as structs
• 4d505e0 cmd/cue: confirm issues 1152 and 1153 are fixed
• eb53c4e cmd/cue: get go: ignore anonymous enums
• b99ce0a internal/core/adt: close open lists for Default
• 061e661 cue: auto-unwrap single embedded value in Expr
• 7922f9b pkg/internal: compute combined error severity
• c490d4c internal/core/export: fix bug with API-generated top-level close
• e62a95a ci: remove go1.15 from the build matrix
• f880467 cue: fix output in ExampleContext
• 9982526 internal/core/adt: don't cache incomplete errors for let
• 4f3988f cue/testdata: hoist let cycle errors
• 91476ab internal/core: use scope for lists
• 718c9a0 ci: correct location of unity
• f65dd0f internal/core/compile: remove unnecessary code
• b89ad85 cue: fix subsumption bug for disjuntion filtering
• a8b721a internal/encoding/yaml: upgrade go-yaml.v3
• 15adb6c encoding/openapi: fix crash in reference
• 4e76bb4 cmd/cue/cmd: improve shell arg parsing in tests
• 0e56b82 internal/core/export: don't add empty temp struct
• 75c3d6b doc/ref/spec.md: include $ in identifier
• c1c2cda internal/encoding/yaml: canonicalize non-string map keys
• a83c6c6 encoding/yaml: quote strings more aggressively
• b9ac1ea doc/tutorial/kubernetes: update types
• 59bdbb5 internal/core/adt: fix bound simplification for floats
• 62e876d internal/core/adt: fix disjunction bug
• 9bb874c cue: Expr handles empty lists to or/and builtins
• 72936e4 cue: define all Op constant values explicitly with that type
• 29a686d simplify install instructions assuming Go 1.16+
• dbf5992 doc: revamp contribution guide ahead of contributing live-stream
• 9217c4d ci: update to use go1.17.3 and go1.16.10
• 437920e doc/ref/spec: fix typos and heading anchors
• fca61f5 cue: define all Kind constant values explicitly with that type
• 8031fe7 spec: Update example with regards to pattern constraints
• ddd96c6 ci: latest Go versions in build matrix
• 5cf1aa1 feat: update go build matrix
• 35c2f3c doc: fix repeated word typos in spec
• e8550b8 doc: remove useless whitespace in main README
• 0f53054 doc/contrib: update git codereview change to use -s flag
• ecb17c9 cmd/cue: revert version string change
• 370fac9 ci: improve goreleaser workflow
• 38c8f7d doc/tutorial: fix errant use of list operators
• a4a38ed ci: tidy up encoding of JSON args to curl
• 7d25ee2 ci: re-enable full CI workflows
• 59079b1 ci: tidy up quoting of JSON args in workflow commands
• 5f28bb4 readme: fix top-level CONTRIBUTING guide
• 5ab75b8 readme: link to CoC from main readme and contributing guide
• 35c4960 general: use cuelang.org/issues as the general issue link
• a68df18 github: fix up empty issue template
• 723a7fe github: tidy up issue templates
• 3d9cd71 general: update unity location
• 6da50d9 ci: fix Docker workflow
• 06484a3 all: change references from cuelang/cue to cue-lang/cue
• 20ed9ab internal/core/compile: fix let cycle error
• 0070bb1 ci: fix up new version trigger
• aa61ee7 doc/tutorial/kubernetes: fix invalid user commands
• 67bf363 doc/tutorial/kubernetes: update k8s version
• ec427f9 cmd/cue: fix up get go docs around "enum" types
• 8e053d4 Allow setting working directory for exec.Run
• e8de4b1 interanal/core/runtime: remove commented out fields

Docker images

• docker pull docker.io/cuelang/cue:0.4.1-beta.6

v0.4.0

In line with our aim to release smaller changes more frequently on our way to language stability, the v0.4.0 release comes just under two months after v0.3.0. It includes one language addition, a fairly significant reshaping of the API and tooling improvements, as well as the usual host of bug fixes, and performance and error message enhancements.

This release includes a more comprehensive implementation of protobuf, supporting textproto and JSON protobuf mappings, and paves the way for binary protobuf support. See the v0.4.0-alpha.1 release notes for more details.

This release also sets the direction for the CUE API, and brings the overall API much closer to where we would like it to go in the run up to v1.0.0. It also prepares the API for the query extension. See the v0.4.0-alpha.2 release notes for more details.

The v0.4.0 minor release follows the pre-v1.0.0 release policy described in the v0.3.0 release notes: it contains backwards incompatible changes discussed below.

As a reminder: users can register their projects with unity. unity is used to ensure that a project's CUE evaluations do not unexpectedly stop working, or regress in terms of performance. We are in the process of adding support for API-based tests, as well as private instances.

Language additions

• Value aliases: v0.4.0-beta.1

Tooling

• Tag variables to inject contextual system values: v0.4.0-beta.1
• Binary file type: v0.4.0-beta.1

API additions

• Context.NewList: v0.4.0-beta.1
• Value.FillPath: v0.4.0-beta.1
• Context.Encode: v0.4.0-beta.1
• The cue.Path model: v0.4.0-alpha.2
• Phasing out Instance: v0.4.0-alpha.2
• Context to replace Runtime: v0.4.0-alpha.2
• Value.Allows: v0.4.0-alpha.2

CUE package additions

• crypto/hmac: v0.4.0-beta.1
• uuid: v0.4.0-beta.1

Performance and error handling

• Error message tuning: v0.4.0-alpha.2
• Fix for pathological performance hit when using Fill or Unify: v0.4.0-alpha.2

Backwards incompatible changes

• Dropped support for old-style aliases: v0.4.0-beta.1
• Dropped support for old-style comprehensions: v0.4.0-beta.1
• Removed ast.TemplateLabel type from API: v0.4.0-beta.1
• Value.Expr: v0.4.0-beta.1
• Value.Decode: v0.4.0-beta.1
• Value.Format implementation: v0.4.0-alpha.2
• cue/encoding: v0.4.0-alpha.2
• @protobuf tag: v0.4.0-alpha.1
• JSON mappings: v0.4.0-alpha.1
• Enum mappings: v0.4.0-alpha.1

Changelog

b39a2d0 cmd/cue/cmd: avoid roundtrip when printing non-CUE in eval
3b0a537 cmd/cue/cmd: fix bug in resolving builtin package shorthands
cd94426 cue: don't extract embedded values
37bf801 cue: keep sane references for embedded disjunctions in Expr
a4e0f52 encoding/openapi: detect cycles when expanding references

v0.4.0-rc.1

This release includes various bug fixes.

Changelog

393ec28 cue/cmd/cue: fix regression for -H flag
13a4d3c encoding/openapi: finalise when constructing openapi AST
287d43c internal/core/export: don't wrap embedded scalars in _#def
1e14710 internal/third_party/yaml: fix faulty decoding of hidden fields

Various bug fixes

This release includes various bug fixes and error message improvements.

Changelog

d5041a1 cmd/cue/cmd: better checks for incorrect flags
858efa9 cue/ast/astutil: fix resolution bugs
5b8ab47 cue: elide comma after ...
3fdad16 cue: move field name to end of error message
dd188a6 cue: move more labels and values to end of error message
48ddb2b doc/ref/spec.md: fix typos
30abd66 internal/core/adt: fix typo
cd286a8 internal/core/export: fix export of disjunction with eliminated defaults
7180c7d pkg/tool/http: fill in status and statusCode
073c6aa pkg/tool/http: make request.body optional

Bug fixes and various new features

This release fixes some bugs, improves some errors and adds API. This version also drops support for some old constructs that have not been supported in the CUE tool for some time now. The main reason to start being more “aggressive” about this is to avoid surprises as we’re inching towards a full backwards compatibility guarantee. Note that “aggressive” is probably a strong word, these constructs have not been supported for over a year.

Language additions

Value aliases

CUE now allows value aliases, as suggested in various proposals and remarks. This allows, for instance, users to rewrite:

foo: {
    a: b + c
    b: int
    c: int
}
bar: foo & {b: 1, c: 2}

as

foo: X={
    a: X.b + X.c
}
bar: foo & {b: 1, c: 2}

In this case the benefit may be small, but there are various cases where not having this ability results in significant inconvenience. Also the query proposal, the UX design around the usage of the proposed must builtin, as well as various other patterns, rely on this ability.

Note the subtle but important difference between this and using field aliases. Using a field alias here, say X=foo: a: X.b + X.c would be equivalent to foo: a: foo.b + foo.c. As a result, in bar: foo, X.b would still be bound to the original location and not resolve to any b specified in bar.

Which one to use depends on the application. As a general rule, field aliases should be used when defining “types”, whereas value aliases should be used when defining “values”. An example of where one wants to use field aliases:

List="my-list-type": {
    value: _
    next:  List
}

Here, using a value alias would result in a list where every value must be the same. This is probably the most subtle aspect of CUE. The guideline of using values aliases when defining values, and field aliases when defining types, however, should give users a good steer on the matter.

Tooling

Tag variables to inject contextual system values

The injection mechanism now allows injecting specific variables, such as current time, username, or hostname, or a random number, into fields marked with a @tag attribute. For instance:

import "path"

_os: string @tag(os,var=os)

path.Base(dir, _os)

allows for an OS-specific Base operation. Except for cue cmd, the var will only be injected when the -T is explicitly specified on the command line.

This approach 1) allows packages to remain hermetic, 2) allows using such functionality without having to resort to cue cmd, and 3) avoids introducing another injection point.

See cue help injection for more details.

Binary file type

The cue tool now supports binary files. These are very much like the supported “text” files, but translate the file contents to a CUE binary literal, rather than a text literal.

There are no default file extensions to recognize a binary file. The binary: file qualifier can be used to read a file as binary. See cue help filetypes for more info.

To make loading binary files in bulk more useful when importing, the import command now supports the --ext flag, allowing certain file types to be interpreted as a certain format. This also works for other types. See cue help import for more info.

--force/-f

This flag is now allowed for any command that outputs files.

API additions

Context.NewList

Uses can now construct a new list cue.Value with Context.NewList. This is defined on Context to allow the creation of empty lists, which still need to be associated with a Context.

Value.FillPath

FillPath now accepts list indices. The Go-to-CUE are now documented in more detail.

Fillpath now also accepts the selectors AnyString and AnyIndex, which allows setting pattern constraints. For instance, v.FillPath(cue.MakePath(cue.Str("a"), cue.AnyString, x), adds a: [string]: x to v.

It also allows setting optional fields. For instance, v.FillPath(cue.MakePath(cue.Str("b").Optional()), 1) will add b?: 1 to v, using the usual rules for unifying such fields.

Context.Encode

The documentation of this method has been significantly improved. This also fixes some bugs that were uncovered in the process.

These are now also two additional options: InferBuiltins and ImportPath. These were exposed to support the cue command in using the new API, but thus also allows other tool writers to use the same kind of functionality.

CUE package additions

crypto/hmac

The new package hmac implements the Keyed-Hash Message Authentication Code (HMAC) as defined in U.S. Federal Information Processing Standards Publication 198.

uuid

The new package uuid defines functionality for creating UUIDs as defined in RFC 4122.

Currently only Version 5 (SHA1) and Version 3 (MD5) are supported. The new tag variable injection mechanism that was introduced in this release, however, also opens up the possibility to include other versions, if needed.

Backwards incompatible changes

Dropped support for old-style aliases

Aliases of the style X = 4 (now let X = 4) have been deprecated since v0.1.4. However, by setting the version or in API use, these aliases would still work. Support for them has been fully removed. Down the line, this syntax can be reused as value aliases as well, relaxing the places where value aliases can be used.

For the time being, cue fix can still be used to rewrite old-style aliases to their let equivalent. There was a bug that missed the detection of some forms of the deprecated alias type. This has now been fixed.

Dropped support for old-style comprehensions

Support for the old-style comprehensions (e.g. [ x+1 for x in foo]) has now been fully removed. Use an older version of CUE to rewrite such old-style comprehensions using cue fmt or cue fix.

Removed ast.TemplateLabel type from API

This has not been parsed, supported, or used for ages. The type has now been removed from the API.

Value.Expr

Value.Expr now translates the or and and builtins as if they were native | or & constructs. This fixes a bug in encoding/openapi and is likely to prevent many issues. If one was relying on detecting the use of these builtins, however, this now will no longer work.

Value.Decode

This method was reimplemented to not use Value.MarshalJSON. This fixes many bugs, but may cause issues if users were relying on old behavior. The new implementation still strictly adheres to the JSON encoding spec when this makes sense, but deviates in some important aspects. For instance, numbers like 1 are now integers in CUE, rather than float. Also binary values are now handled correctly.

Bug fixes

• hidden references used on the command line now resolve to hidden fields in the package under evaluation
• equality of null value

Changelog

5b14995 ci: debug tip trigger failures
24ce27e ci: fix broken tip and new version triggers
a8369ae ci: fix tip and new version curl and env problems
c143a3a ci: fix tip triggers with proper quoting
746e02e ci: more debugging of tip triggers
6c6b4e7 ci: trigger unity and cuelang.org builds/equivalent on tip/new versions
97cac92 cmd/cue/cmd: do not silently ignore unknown filetypes
89abc18 cmd/cue/cmd: fix resolution of hidden values across tools
4f7caea cmd/cue/cmd: move tag flags handling to central location
b9e7d90 cmd/cue/cmd: support binary file type
b449c0f cmd/cue: support --force for all commands that allow output flags
aa70399 cue/ast: fully remove support for old-style comprehensions
b73ab0b cue/ast: remove support for TemplateLabel
1f1c3f6 cue/load: provide reason for exclusion of files
975ba50 cue/load: remove unused code
bcdf277 cue/load: support injection of system variables
c2a68a9 cue/parser: allow pattern constraints unconditionally
3e10918 cue/parser: better error messages for use of deprecated constructs
3ef90b3 cue/testdata, doc, pkg: fix some it's/its typos
f60c88a cue: Value.Expr: process or and and builtins semantically.
d9af603 cue: add InferBuiltins EncodeOption
b49cbbd cue: add NewList
1f618f0 cue: align import with top-level package for -e
66efc67 cue: allow setting list elements in FillPath
67c6b6f cue: better document Context.Encode and friends
9d044f0 cue: hoist attribute-related functionality to separate file
ed5cdf0 cue: more path cleanup
3347302 cue: move Decode to separate file
c365513 cue: prepare API implementation for structure sharing
0457356 cue: properly decode "any" labels
819cf95 cue: reimplement Decode to not use MarshalJSON
5481b41 cue: remove use of isDef
2e1ac0f cue: support optional fields for FillPath
ac7e992 cue: undeprecate Iterator.Label
3316905 doc/ref/spec.md: introduction of value aliases
a31dd01 doc/ref/spec.md: remove "dead code"
2198ac3 docs: add link to Code of Conduct to CONTRIBUTING guide
f5213be github: add bug report template link to repro advice
1ad66ab internal/core/adt: fix null values equality
3cdf845 internal/core/adt: use Closed for checking recursive closedness
5cd76bb internal/core/comple: drop support for old-style aliases
8823e2a internal/core: support field value aliases
6d67b48 pkg/crypto/hmac: add crypto/hmac support
2118921 pkg/crypto/hmac: fix package documentation
a7d3987 pkg/time: fix documentation bug in Unix
fd05bf4 pkg/uuid: implementation of hermetic UUID functions

v0.4.0-alpha.2

This release focuses on setting direction for the CUE API. This brings the overall API much closer to where we would like it to go. It also prepares the API for the query extension.

Note that this is deliberately marked as an alpha release to allow people to give feedback in the case we need to make some tweaks to the API.

A central piece in that are the cue.Path and cue.Selector types. Another key part is getting rid of the Instance type. This type proves to be unnecessary and not using it results in a considerably nicer UX.

The old functionality is marked as deprecated and will stay around until we provide a way to automatically rewrite them. We will just no longer support them and we use a trick mentioned in this tweet to hide most of the deprecated methods from the Go docs in the meantime (pending a "proper" fix in pkg.go.dev).

Other improvements include:

• another big round of tuning error messages, most notably adding more line information.
• a performance bug fix that caused significant slowdown when doing repeated calls of Fill or Unify on the same values.

API additions

The cue.Path model

The old API was designed before the existence of definitions. The fact that definitions live in a different namespace, broke the API at various levels, and patching this up resulted in a brittle and complicated API.

The cue.Path-centred API addresses these issues, resulting in a much smaller API surface while at the same time being less prone to error.

This new API also paves the way for the query extension.

Some deprecated types and methods:

The following types and methods all get replaced by LookupPath:

• FieldInfo
• Instance.Lookup
• Instance.LookupDef
• Instance.LookupField
• Value.Elem
• Value.FieldByName
• Value.Lookup
• Value.LookupDef
• Value.LookupField
• Value.Struct
• Value.Template
• Struct

Similarly, FillPath and ReferencePath replace (Value|Instance).Fill and Reference. The latter also is instrumental in getting rid of Instance. In the long run we will want to repurpose Value.Lookup and Value.Fill, but that will be a long way out.

The new Iterator.Selector method replaces:

• Iterator.Label
• Iterator.IsLabel
• Iterator.IsOptional
• Iterator.IsDefinition

Using Selector also makes it more explicit what kind of labels one has and is thus less error prone. The selectors can be used to construct paths for lookup methods without loss of information, making the API more precise.

Phasing out Instance

The Instance type was initially intended to allow enforcing certain constraints. The data model of the new evaluator allows doing so without the need for Instance.

Getting rid of Instance has some big benefits. Most notably, the need to link Value to Instance makes it very hard to avoid memory retention, even when a user believes a Value is no longer needed. This causes long-running CUE services to grow in memory usage over time.

Another issue is usability: aside for duplication of methods, it may be hard for a user to know when to use a Value or Instance.

Context to replace Runtime

There are a few reasons for this change. As part of removing Instance, we needed a new way to build Values. The Context type now defines a new set of Value constructors.

Another reason was to have a cleaner way to break the dependency cycle that existed for linking in the core builtins. Previously, the user had to add a import _ "cuelang.org/go/pkg" somewhere in the code, or use a package that already did so, to make the builtins available. Now the user is expected to create a Context using cuecontext.New(), which takes care of this.

Finally, the name Runtime was confusing to some. The Context maintains indices and tables that are shared between creations of values, but there is no inherent “running” state.

The Context now also makes it easier to resolve identifiers in another Value’s scope or to directly encode Go values and types.

Value.Allows

This method now enables querying whether a Value would support a field of a specific type if it were to be added. This also uses the Selector type to specify the kind of field.

This replaces IsClosed.

Backwards incompatible changes

The new APIs are just additions. In many cases, the old API has been implemented in terms of the new API, but should still function as usual. This did result in some bug fixes, however, so one may observe changes.

Value.Format

The one change that may cause backwards incompatibility is the standard fmt.Formatter implementation of Value, which now has a more principled implementation. The standard %v formatter now prints it as a value, but allows incomplete values. The %+v formatter, as before, prints the fully evaluated value and is similar to cue export. The %#v formatter, which previously printed an esoteric debug format, now prints an equivalent of cue def.

Many of the standard Go formatting verbs will now be interpreted as such if the Value is of a compatible Go type. See the documentation of Value.Format for more details.

There have been various bug fixes in the exporter code as part of this change.

cue/encoding

This package has been removed. It really didn’t do anything except from being a distraction. In the off chance that anybody was using this package, just deleting that code would probably solve it.

Changelog

9e34a41 cue/ast/astutil: export ImportPathName
50c137a cue/encoding: removed unused package.
dcb2a1f cue/errors: add Wrap
f044ad1 cue/format: expose indent option
6822433 cue: add Iterator.Selector
f14c9a4 cue: add Selector.PkgPath
c5c9125 cue: add test for filling empty path
17d4e16 cue: clean up Format
618c10c cue: deprecate Instance.(Doc|Fill)
790bed3 cue: get rid of NewRuntime
4937cb9 cue: hide deprecated methods from docs
421ead3 cue: introduce Context to replace Runtime
d76e2cc cue: remove MakeValue
908614e internal/core/adt: dedup errors
b8ce660 internal/core/adt: improve performance for repeated calls to Unify/Fill
276ce26 internal/core/adt: record more error positions
14ec6e2 internal/core/export: add real Final option
b5b0429 internal/core/export: bug fixes for exporting API-generated values
c62b750 internal/core/export: bug fixes in definitions
64ede63 internal/core/export: extract docs from root
b937727 internal/core/export: fix definition wrapping
c290772 internal/value: implement interface type that is both value and instance
1f78a8d internal: replace untyped functions with typed ones
d5ff672 pkg: clean up builtin errors

Prototext and JSON PB support

This release introduces a more comprehensive implementation of protobuf, supporting textproto and JSON protobuf mappings, and paving the way for binary protobuf support as well.

This release required some significant changes to the cue command logic.

A hallmark property of protobuf data formats is that they cannot be parsed without a schema (unlike JSON, for instance). This is true to various degrees for the different formats, but it holds true in some shape or form for all of these. This required some changes in the CUE tooling to make this possible. It also required some backwards incompatible changes.

Protobuf changes

Filetype *.textproto

CUE now supports interpreting and writing text proto files. Text proto files cannot be interpreted, or even parsed, without a schema. This means that .textproto files can only be read with an explicit schema using the --schema/-d flag. Moreover, the schema must have @protobuf for certain types, such as maps, to be interpreted properly.

Note that the available .textproto parsing libraries are incredibly buggy. So there will be some rough edges that are kind out of CUE’s hands.

JSON conversion: package jsonpb and json+pb

The Protobuf documentation has a recommendation on how Proto messages should map to JSON. Package jsonpb now supports this mapping in both directions. It implements this by rewriting a CUE AST based on a given schema, a cue.Value. This allows this mapping to also be combined in conjunction with Yaml, for instance.

On the command line this feature can be used through the pb “interpretation”, for instance as json+pb or yaml+pb. Both input and output are supported.

Note that interpretations to CUE require a schema for the interpretation. This can be an explicitly specified schema using the --schema/-d flag or an implicit only by unifying it with a schema value or playing it within a schema using the placement flags (see cue help flags).

Backwards incompatible changes

@protobuf tag

The protobuf previously had only one required argument: the numeric value of the enum. The type was optional and only included if different in name from the CUE type. As it turned out, though, the CUE type was not always sufficient information to be able to represent proto values. Most notably, integer values are encoded differently in JSON depending on the type of integer in the proto specification (this is not a typo!!).

The new format includes the type unconditionally as the second argument. CUE does its best to recognize old formats for backwards compatibility purposes, but this may cause issues.

@protobuf(<tag num>,<type>, ...options)

Protobuf options are still represented as string literals, allowing CUE options, such as alternative names (name=x) to be represented as the usual <key>=<value> format.

Another change is the representation for map types. Previously, the Protobuf map type was included verbatim (map<T,U>). This was somewhat inconvenient for parsing attributes, though: angular brackets are, unlike (), [], and {} not matched. So the comma in the map type commanded some escaping. To avoid this, maps are now represented as map[T]U. We contemplated using [T]:U, but opted for the map prefix for clarity and future extendibility.

JSON mappings

An initial design decision for the Proto to CUE mapping was to have CUE follow the Proto to JSON mapping. By hindsight, though, this did not make much sense. The inconsistency of having integers represented as strings does not make sense for a language that supports expressions on such integers (unless we want to give up typing, perhaps).

The stance now is to take the representation that makes sense, and have a protobuf-specific CUE to/from JSON converters. This is akin to the JSON schema and OpenAPI converters, which map CUE to some data format, using JSON or YAML, for instance, as a transport layer.

Luckily, the Protobuf to CUE mapping already deviated from the recommended mapping, always defaulting to int for integer types. So there are no changes there, other than that there is now support for following the recommended mappings for import and export.

Enum mappings

The most noteworthy backwards incompatible change is how enum types are mapped. Previously, CUE followed the recommended JSON mapping of using strings. This turned out to be a bad idea for various reasons. Firstly, the source of truth are integer values, really, not the names. There may be multiple names per number, making comparing names somewhat meaningless. Finally, most other languages used integers as representations, making the CUE representation not interoperate well with such languages.

Although the old mapping is still supported, the integer-based mapping is now recommended.

The default enum representation when using cue import proto is now to represent enums as integers in CUE. The --proto_enum=json flag can be used to invoke the old conversion behavior.

The API will keep converting using the JSON format, but now has a Config.EnumMode option for selecting the alternative behavior.

The Protobuf to JSON interpretation (filetype json+pb or package jsonpb) supports converting back and forth in either format.

Changelog

aaf6e84 cmd/cue/cmd: compute schema before values
0b5084a cmd/cue/cmd: hook up protobuf encodings types
8e5eeab cmd/cue/cmd: parseArgs: split values and schemas early
f228236 cmd/cue/cmd: preparse command line path expressions
75d0180 cmd/cue/cmd: simplify parseArgs in preparation of proto support
dea3c5d cue/build: organized Instance fields
660b090 cue/build: remove support for file lists
38ad7c3 cue: add ReferencePath
af509c6 cue: eliminate context type
52db572 cue: get rid of internal index type
c0fe9ce cue: refactor index
362e3a5 encoding/protobuf/jsonpb: add encoder
4a288d5 encoding/protobuf/textproto: add decoder implementation
a0035de encoding/protobuf/textproto: add encoder
3bdfa5d encoding/protobuf: always include type as second argument
dcfff00 encoding/protobuf: support integer enums
8ba98ee internal/encoding: pass schema to config
80a0a6e internal: move DebugStr to new astinternal package
22abdad internal: replace internal.CoreValue with more type-safe variant

A couple of API bug fixes

Not a big release, just a couple of API bug fixes.

Changelog

c8b7fe0 internal/core/export: export floats as float literals
f1d3d81 internal/core/export: fix decl export printing and extraction

Bug fixes and API work

This release contains several bug fixes, API additions, some API deprecations, and a language addition. One of the bug fixes may lead to unexpected behavior (see below).

Backwards Incompatible Bug Fix

One common use case of cmd/cue involves combining CUE and non-CUE files. Consider the following example:

// check.cue
package check

map: {for n in nodes {"\(n.name)": n}}

# data.yaml
nodes:
- name: bar
  parent: foo

- name: baz
  parent: foo

With v0.3.0 and earlier, cue eval would happily combine these two files:

$ cue eval data.yaml check.cue
map: {
    bar: {
        name:   "bar"
        parent: "foo"
    }
    baz: {
        name:   "baz"
        parent: "foo"
    }
}
nodes: [{
    name:   "bar"
    parent: "foo"
}, {
    name:   "baz"
    parent: "foo"
}]

But cue vet would complain:

$ cue vet data.yaml check.cue
map: reference "nodes" not found:
    ./check.cue:3:16

cue vet is actually correct here. Identifier resolution should only happen across files that belong to the same package: non-CUE files are equivalent to a CUE file without a package clause or an anonymous package clause (package _), hence cue eval should really fail in this instance.

v0.3.1 makes cue eval and other commands consistent with cue vet. This was deemed a sufficiently breaking change to warrant its own release.

The fix in this case is to define nodes in the scope of package check:

// check.cue

package check

nodes: _
map: {for n in nodes {"\(n.name)": n}}

at which point cue eval succeeds as before.

An upcoming change will also ensure this condition of identifier resolution is satisfied for all non-package CUE file arguments.

API changes

The cue.Selector model has been extended to allow querying optional fields and arbitrary elements.

Deprecations

The following methods of Value have now been deprecated:
FieldByName
Elem
Template

Really, you should only need LookupPath for your lookup needs. For the most part, these old methods have been reimplemented using the new API.

Also, updated the doc that one really, really should not use Merge.

If you are a user of the API we strongly recommend running staticcheck: it will raise errors where you are using deprecated parts of the API.

Language Additions

CUE now allows parenthesized expressions as labels. This is not (yet) published in the spec, but part of the query proposal. So "\(x)": foo, where x is a string, may now be written as (x): foo.

Changelog

a8ae7d1 cmd/cue/cmd: allow "exec" in commands for TestX
460357b cue/ast: allow parentheses as labels
e70a1db cue/build: remove unused Scope field
f0adb4e cue: deprecate Value.Elem and Value.Template
f063a61 cue: deprecate Value.FieldByName
6a1ae9c cue: move LookupPath to query
e440183 cue: remove error type aliases
4459434 cue: resolve identifiers in an ast.Expr for FillPath
c505c19 cue: separate closedness for lists and structs
957003c cue: support optional field lookup in LookupPath
ad4d1a1 cue: update Merge comment to REALLY NOT USE IT
5c2b281 internal/core: don't resolve values into anonymous packages
72e8fb4 internal/diff: fix printing and top-level diffs

First large step towards backwards compatibility guarantee

After a long road, we have now an official v0.3.0 release. Aside from implementing some important changes to the language (see release notes), it prepares CUE for exciting new features such as the query and policy extensions and other language extensions as well as much better performance.

The focus will now be to get to language stability and a v1.0.0 release.

With this release will also come a different approach to versioning leading up to the v1.0.0 release. The goal is to get to the stability guarantee for the language first, then the API and tooling. Up to v1.0.0, version increments will have the following meanings:

• minor (_.x._)
  • backwards incompatible changes
  • bug fixes with notable backwards incompatible changes
  • large new features
• patch (_._.x)
  • bug and spec conformance fixes
  • new features

We plan to get the few small remaining backwards incompatible languages chances out as soon as possible. To make the transition as simple as possible, we aim to put possible impactful bug fixes in small releases, so that users get a chance to get used to them. And, of course, we plan to make any necessary transitions as smooth as possible using cue fix aids when possible.

Note that to smooth out any transition, users can register their repos at https://github.com/cue-sh/unity. This is used to ensure that their CUE evaluations will not unexpectedly stop working.

API

encoding/protobuf/jsonpb

Protobuf defines its own, somewhat peculiar, mapping to JSON. This package provides a way to interpret JSON for such mappings, given a CUE schema. This is a first step towards broader Protobuf value support.

cue.Selector

Several new features have been added to Selector, including the possibility to query whether a selector is a regular string field and the ability to create a Selector form an ast.Label.

Value.Attributes

Value has been extended with a more extensive API to retrieve attributes.

Value.Subsume

Now allows raw (CUE-native) subsumption, instead of just the "Schema" or "Data" interpretations.

Value.FillPath

For completeness, note that beta.8 already introduced FillPath.

Changelog

4476060 cmd/cue/cmd: load import dependencies of tool files
276e164 cue/literal: expose some internal data
c24a281 cue/load: drop out-of-date comment about type of value in Overlay map
792da39 cue: add function for Value.Attributes()
1ea47e0 cue: allow raw subsumption, taking defaults into account
20a4878 cue: expose some path utilities
48f2a22 doc/cmd: refer people to cue help for now
aa99414 doc: fix various typos
2115955 encoding/openapi: correctly extract type in the presence of null
a1903ca encoding/openapi: dedup conjuncts and disjuncts
3701bef encoding/protobuf/jsonpb: add Rewrite* for interpreting JSON in PB terms
96e84eb tools/flow/testdata: add test with package dependency