From 44abf0200c89df64e64dd49059c69651932857cd Mon Sep 17 00:00:00 2001 From: Lars Vogel Date: Wed, 24 Apr 2024 09:58:25 +0200 Subject: [PATCH] Migrate multiple p2 wiki pages to mark-down docs/Concepts.md docs/Customizing_Metadata.md docs/Engine.md docs/Installable_Units.md docs/Query_Language_for_p2.md For https://github.com/eclipse-equinox/p2/issues/497 --- docs/Concepts.md | 58 +++++ docs/Customizing_Metadata.md | 320 +++++++++++++++++++++++ docs/Engine.md | 114 +++++++++ docs/Installable_Units.md | 165 ++++++++++++ docs/Query_Language_for_p2.md | 468 ++++++++++++++++++++++++++++++++++ 5 files changed, 1125 insertions(+) create mode 100644 docs/Concepts.md create mode 100644 docs/Customizing_Metadata.md create mode 100644 docs/Engine.md create mode 100644 docs/Installable_Units.md create mode 100644 docs/Query_Language_for_p2.md diff --git a/docs/Concepts.md b/docs/Concepts.md new file mode 100644 index 000000000..4817b2635 --- /dev/null +++ b/docs/Concepts.md @@ -0,0 +1,58 @@ +Equinox/p2/Concepts +=================== + + +The following is a set of terms and concepts that are prevalent in the Equinox Provisioning work. + +Agent + +The provisioning infrastructure on client machines is generally referred to as the agent. Agents can manage themselves as well as other profiles. An agent may run separate from any other Eclipse system being managed or may be embedded inside of another Eclipse system. Agents can manage many profiles (see below) and indeed, a given system may have many agents running on it. There is no such thing as "the p2 agent" or a bundle that goes by that name. This is because p2 is modular and what you need to run on an embedded device is not the same than what you need on a desktop or an autonomous server. + +Artifact + +Artifacts are the actual content being installed or managed. Bundle JARs and executable files are examples of artifacts. + +Artifact Repository + +Artifact repositories hold artifacts + +Director + +The director is a high level API that combines the work of the planner and the engine. That is, the director invokes the planner to compute the provisioning operations to perform, and then invokes the engine with the planner's output to achieve the desired profile changes. + +[Engine](/Equinox/p2/Engine "Equinox/p2/Engine") + +The engine is responsible for carrying out the desired provisioning operations as determined by a director. Whereas the subject of the director's work is metadata, the subject of the engine's work is the artifacts and configuration information contained in the IUs selected by the director. Engines cooperate with repositories and transport mechanisms to ensure that the required artifacts are available in the desired locations. The engine runs by invoking a set of engine Phases and working with the various Touchpoints to effect the desired result. + +Garbage Collection + +Element of repositories (metadata and artifact) can be garbage collected by tracing reachability from a set of known _roots_. For example, the set of all profiles managed by an agent transitively identifies all IUs that are currently of direct interest to the provisioning agent. Similarly, the IUs identify the artifacts required to run the profiles. Any IUs or artifacts that are not in the transitive list are _garbage_ and can be collected. + +[Installable Units](/Installable_Units "Installable Units")(IU) + +Installable Units are **metadata** that describe things that can be installed. They are **not** the things themselves. So an IU for a bundle is not the bundle but a description of the bundle: its name, version, capabilities, requirements, etc.. The bundle JAR is an _artifact_. + +Metadata Repository + +A metadata repository holds installable units. + +Mirroring + +The basic operation of distribution is mirroring. The key here is that metadata and artifacts are not _downloaded_, they are mirrored. The subtle distinction is that local mirrors are a) simple caches of something that is remote and b) potential sources of further mirroring. This means that locally held information can be deleted and replaced as needed by re-mirroring. Similarly, having local copies act as mirrors opens the path to natural peer-to-peer distribution. Note that metadata and artifacts are quite separate and having an IU mirrored in one repo does not imply that the associated artifacts are in/near/beside/... that repo. + +Phase + +Provisioning operations generally happen by walking through a set of steps or phases. At each phase a particular kind of activity takes place. For example, during the Fetch phase, the various artifacts required for the operation are Mirrored while during the Configure phase IUs are woven into the underlying runtime system by their associated Touchpoints. + +Planner + +The planner is responsible for determining what should be done to a given profile to reshape it as requested. That is, given the current state of a profile, a description of the desired end state of that profile and metadata describing the available IUs, a planner produces a list of provisioning operations (e.g., install, update or uninstall) to perform on the related IUs. + +Profile + +Profiles are the target of install/management operations. They are a list of IUs that go together to make up a system. They are roughly equivalent to the traditional Eclipse _configurations_. When an IU is _installed_ it is added to a profile. That profile can then be run and the artifacts associated with the installed IUs executed (or whatever). Later the IU can be uninstalled or updated in that profile. The exact same IU can be installed simultaneously in many profiles. + +Touchpoint + +A part of the engine that is responsible for integrating the provisioning system to a particular runtime or management system. For example, the Eclipse Touchpoint understands how Equinox stores and manages bundles. Different platforms have different _Native_ Touchpoints that integrate with the Windows desktop, RPMs, various registries etc. See also [Touchpoint Use Cases](/Touchpoint_Use_Cases "Touchpoint Use Cases"). + diff --git a/docs/Customizing_Metadata.md b/docs/Customizing_Metadata.md new file mode 100644 index 000000000..b9be6cb8e --- /dev/null +++ b/docs/Customizing_Metadata.md @@ -0,0 +1,320 @@ +Equinox/p2/Customizing Metadata +=============================== + +Contents +-------- + +* [1 Customizing Installable Unit Metadata](#Customizing-Installable-Unit-Metadata) + * [1.1 Advice file format](#Advice-file-format) + * [1.2 Capability Advice:](#Capability-Advice:) + * [1.3 Update descriptor advice:](#Update-descriptor-advice:) + * [1.4 Property Advice:](#Property-Advice:) + * [1.5 Touchpoint Instruction Advice:](#Touchpoint-Instruction-Advice:) + * [1.5.1 Variable Substitutions](#Variable-Substitutions) + * [1.6 Additional Installable Unit Advice:](#Additional-Installable-Unit-Advice:) +* [2 Category Generation Using p2.inf](#Category-Generation-Using-p2.inf) + * [2.1 Categorizing plug-ins](#Categorizing-plug-ins) + * [2.2 Nested categories](#Nested-categories) + +Customizing Installable Unit Metadata +===================================== + +Advice file format +------------------ + +An Installable Unit can be augmented at generation time by writing a p2 advice file (p2.inf). The format of this file is java properties file containing key=value pairs. In Eclipse 3.5, touchpoint advice files can be placed: + +* In bundles (META-INF/p2.inf): The instructions are added to the installable unit for the bundle +* In features (a p2.inf file co-located with the feature.xml): The instructions are added to the installable unit for the feature group +* In products (a p2.inf file co-located with the .product file): The instructions are added to the root installable unit for that product. + +Two special value parameters are supported: + +* $version$ - returns the string form of the containing IU's version +* $qualifier$ - returns just the string form of the qualifier of the containing IU's version + +For the 3.5 release the p2.inf file can be used to augment Installable Units: capabilities, properties, and instructions. In addition support is provided for defining additional installable units that are in some way related to the container IU. + + +Properties in this file often contain <#> index segments. These index segments serve to separate groups of similar properties from each other. The actual value of these indexes are not important, only that a given set of properties which all refer to the same logical item all use the same index. The following example is talking about two seperate properties, each uses a different index to distinguish it from the other. + + properties.0.name = testName1 + properties.0.value = testValue1 + properties.1.name = testName2 + properties.1.value = testValue2 + +Capability Advice: +------------------ + +There are three different type of capability advice: + +* "provides" - these are capabilities that an IU will offer to satisfy the needs of other IUs +* "requires" - these are the capabilities that an IU requires from other IUs in order to resolve correctly +* "metaRequirements" - these are capabilities that the IU puts on the profile that must already be installed before this IU can be installed. + +Capability advice will "replace" an existing capability of the same type on the IU if the name/namespace match. + + provides.<#>.namespace = + provides.<#>.name = + provides.<#>.version = _(optional / default: 1.0.0)_ + + requires.<#>.namespace = + requires.<#>.name = + requires.<#>.range = _(optional / default: 0.0.0)\]_ + requires.<#>.matchExp = (note that in this case the namespace, name and range attributes are not + requires.<#>.greedy = _(optional / default: true)_ + requires.<#>.optional = _(optional / default: false)_ + requires.<#>.multiple = _(optional / default: false)_ + requires.<#>.filter = _(optional)_ + +Negative requirements can be published by setting min and max values on the requirement to 0. For example + + requires.<#>.namespace = org.eclipse.equinox.p2.iu + requires.<#>.name = some.feature.feature.group + requires.<#>.min = 0 + requires.<#>.max = 0 + + metaRequirements.<#>.namespace = + metaRequirements.<#>.name = + metaRequirements.<#>.range = _(optional / default: 0.0.0)_ + metaRequirements.<#>.matchExp = {p2QL expression} (note that in this case the namespace, name and range attributes are not used) + metaRequirements.<#>.greedy = _(optional / default: true)_ + metaRequirements.<#>.optional = _(optional / default: false)_ + metaRequirements.<#>.multiple = _(optional / default: false)_ + +Where <#> is an index for the property, , and are the associated named strings, and are version and version range strings respectively. + +For example: + + provides.0.namespace = testNamespace1 + provides.0.name = testName1 + provides.0.version = 1.2.3.$qualifier$ + provides.1.namespace = testNamespace2 + provides.1.name = testName2 + provides.1.version = $version$ + + requires.0.namespace = testNamespace1 + requires.0.name = testName1 + requires.0.range = \[1.2.3.$qualifier$, 2) + requires.0.greedy = true + requires.0.optional = true + requires.0.multiple = true + requires.1.namespace = testNamespace2 + requires.1.name = testName2 + requires.1.range = \[$version$, $version$\] + requires.1.greedy = false + + metaRequirements.0.namespace = testNamespace1 + metaRequirements.0.name = testName1 + metaRequirements.0.range = \[1.2.3, 2) + metaRequirements.0.greedy = true + metaRequirements.0.optional = true + metaRequirements.0.multiple = true + metaRequirements.1.namespace = testNamespace2 + metaRequirements.1.name = testName2 + metaRequirements.1.range = $version$ + metaRequirements.1.greedy = false + +Update descriptor advice: +------------------------- + +The update descriptor advice allows to override the default update descriptor generated by p2. Typically this is useful if an IU has been renamed and automatic update detection is still desired. + + update.id = + update.range = + update.matchExp = {a match expression identifying the IU being updated}. (When this is specified the values of id and range are ignored) + update.severity = <0|1> + +To allow updating a renamed artifact, you must provide a matchExp that matches both the old and new artifact names. For example, when renaming \*old\* feature to \*new\*: + + update.matchExp = providedCapabilities.exists(pc | \ + pc.namespace == 'org.eclipse.equinox.p2.iu' && \ + (pc.name == 'old.feature.group' || \ + (pc.name == 'new.feature.group' && pc.version ~= range('\[0.0.0,$version$)')))) + +These match expressions are written in the [p2 Query Language p2ql](/Equinox/p2/Query_Language_for_p2 "Equinox/p2/Query Language for p2"). + +Property Advice: +---------------- + + properties.<#>.name = + properties.<#>.value = + +Where <#> is an index for the property, , and hold the name and value strings for the property. + +For example: + + properties.0.name = testName1 + properties.0.value = testValue1 + properties.1.name = testName2 + properties.1.value = testValue2 + +Touchpoint Instruction Advice: +------------------------------ + + instructions. = + instructions..import = \[,\]* _(optional)_ + +Where is a p2 phases (collect, configure, install, uninstall, unconfigure, etc). + +[Note:](/index.php?title=Note:&action=edit&redlink=1 "Note: (page does not exist)") + +* The will be "appended" to the end of any instructions already being generated. +* The qualified action names for the IU's touchpoint type are implicitly imported. All other actions need to be imported. + +For example: + + instructions.install = \ + ln(targetDir:@artifact,linkTarget:foo/lib.1.so,linkName:lib.so);\ + chmod(targetDir:${artifact.location},targetFile:lib/lib.so,permissions:755); + instructions.install.import= \ + org.eclipse.equinox.p2.touchpoint.natives.ln,\ + org.eclipse.equinox.p2.touchpoint.natives.chmod + +### Variable Substitutions + +The parameters passed to the touchpoints may contain variables whose values are substituted. The following variables are supported. This is the complete list of supported variables as of Oxygen. + +* ${installFolder} the root folder to which the bundle is being installed. This is the folder that contains the 'plugins' and 'features' folders. +* ${forced} usually "false". This may be "true" for the unconfigure and uninstall phases to ensure exceptions don't prevent the action and is probably of little value as a variable substitution. +* ${phaseId} eg. "install" +* ${artifact.location} the full file name of the target location to which the jar will be installed. +* ${lastResult} the result of the execution of the action for the previous phase. The result is substituted only if it + +is a String. + +Additional Installable Unit Advice: +----------------------------------- + +In addition to customizing attributes of the containing IU one can also author addtional installable units that work with the container IU. Typically this mechanism is used to author an IUFragment that customizes the containing IU or one of it's dependencies. + + iu.<#>.id= + iu.<#>.version= _(optional)_ + +Where <#> is an index for the Installable Unit, so multiple Installable Units can be declared. A fairly full range of IU customizations are supported including: + + id + version + singleton + copyright + licenses + filter + touchpoint + update + artifacts + properties + provides + requires + metaRequirements + hostRequirements + instructions + +To illustrate all the various settings for these customizations here's a more complete example of: (unit.0) a minimal IU and (unit.1) a full featured IU: + + units.0.id = testid0 + units.0.version = 1.2.3 + + units.1.id = testid1 + units.1.version = 1.2.4 + units.1.singleton = true + units.1.copyright = testCopyright + units.1.copyright.location = [http://localhost/test](http://localhost/test) + units.1.filter = test=testFilter + units.1.touchpoint.id = testTouchpointId + units.1.touchpoint.version = 1.2.5 + units.1.update.match = p2QL expression describing the update. When this is specified update.id and update.range are ignored. + units.1.update.id = testid1 + units.1.update.range = (1,2) + units.1.update.severity = 2 + units.1.update.description = some description + units.1.artifacts.0.id = testArtifact1 + units.1.artifacts.0.version = 1.2.6 + units.1.artifacts.0.classifier = testClassifier1 + units.1.artifacts.1.id = testArtifact2 + units.1.artifacts.1.version = 1.2.7 + units.1.artifacts.1.classifier = testClassifier2 + units.1.licenses.0 = testLicense + units.1.licenses.0.location = [http://localhost/license](http://localhost/license) + units.1.properties.0.name = testName1 + units.1.properties.0.value = testValue1 + units.1.properties.1.name = testName2 + units.1.properties.1.value = testValue2 + units.1.requires.0.namespace = testNamespace1 + units.1.requires.0.name = testName1 + units.1.requires.0.range = \[1.2.3.$qualifier$, 2) + units.1.requires.0.greedy = true + units.1.requires.0.optional = true + units.1.requires.0.multiple = true + units.1.requires.1.namespace = testNamespace2 + units.1.requires.1.name = testName2 + units.1.requires.1.range = $version$ + units.1.requires.1.greedy = false + units.1.requires.1.optional = false + units.1.metaRequirements.0.namespace = testNamespace1 + units.1.metaRequirements.0.name = testName1 + units.1.metaRequirements.0.range = \[1.2.3.$qualifier$, 2) + units.1.metaRequirements.0.greedy = true + units.1.metaRequirements.0.optional = true + units.1.metaRequirements.0.multiple = true + units.1.metaRequirements.1.namespace = testNamespace2 + units.1.metaRequirements.1.name = testName2 + units.1.metaRequirements.1.range = $version$ + units.1.metaRequirements.1.greedy = false + units.1.metaRequirements.1.optional = false + units.1.provides.0.namespace = testNamespace1 + units.1.provides.0.name = testName1 + units.1.provides.0.version = 1.2.3.$qualifier$ + units.1.provides.1.namespace = testNamespace2 + units.1.provides.1.name = testName2 + units.1.provides.1.version = $version$ + units.1.instructions.configure = addProgramArg(programArg:-startup); addProgramArg(programArg:@artifact); + units.1.instructions.unconfigure = removeProgramArg(programArg:-startup); removeProgramArg(programArg:@artifact);) + units.1.instructions.unconfigure.import = some.removeProgramArg + units.1.hostRequirements.0.namespace = testNamespace1 + units.1.hostRequirements.0.name = testName1 + units.1.hostRequirements.0.range = \[1.2.3.$qualifier$, 2) + units.1.hostRequirements.0.greedy = true + units.1.hostRequirements.0.optional = true + units.1.hostRequirements.0.multiple = true + +Category Generation Using p2.inf +================================ + +The p2 UI allows for hierarchical organization of Installable Units based on the concept of "categories" where the children of categories are what's installable. On occasion we might want to take finer grained control of the contents of a category and what it contains. For example we might want to support further categorization of a features contents to allow individual plugins to be installed instead of the more typical features. + +To support this we can tag a feature as a category as follows: + + properties.1.name=org.eclipse.equinox.p2.type.category + properties.1.value=true + +Another possibility is to use "additional IU advice" to create a specialized category IU like this: + + units.1.id=my.product.category + units.1.version=1.0.0 + units.1.provides.1.namespace=org.eclipse.equinox.p2.iu + units.1.provides.1.name=my.product.category + units.1.provides.1.version=1.0.0 + units.1.properties.1.name=org.eclipse.equinox.p2.type.category + units.1.properties.1.value=true + units.1.properties.2.name=org.eclipse.equinox.p2.name + units.1.properties.2.value=My Category Name + requires.1.namespace=org.eclipse.equinox.p2.iu + requires.1.name=my.product + requires.1.range=\[1.0.0,1.0.0\] + requires.1.greedy=true + +Categorizing plug-ins +--------------------- + +The following file contains two projects showing how to create a category to show plug-ins [File:CategorizedPlugins.zip](/File:CategorizedPlugins.zip "File:CategorizedPlugins.zip"). + +Nested categories +----------------- + +**Work in Progress** + +* p2 parser and publisher for category.xml to support nested categories: [https://bugs.eclipse.org/bugs/show_bug.cgi?id=406902](https://bugs.eclipse.org/bugs/show_bug.cgi?id=406902) +* PDE category editor to support nested categories: [https://bugs.eclipse.org/bugs/show_bug.cgi?id=296392](https://bugs.eclipse.org/bugs/show_bug.cgi?id=296392) + +**Current workaround** The following file containss a number of projects showing how to create nested categories [File:NestedCategories.zip](/File:NestedCategories.zip "File:NestedCategories.zip"). + diff --git a/docs/Engine.md b/docs/Engine.md new file mode 100644 index 000000000..c722f2d69 --- /dev/null +++ b/docs/Engine.md @@ -0,0 +1,114 @@ +Equinox/p2/Engine +================= + +Contents +-------- + +* [1 Overview](#Overview) +* [2 Engine API](#Engine-API) +* [3 Usage](#Usage) +* [4 Touchpoint Instructions, Actions and Variables](#Touchpoint-Instructions.2C-Actions-and-Variables) +* [5 Extra Notes](#Extra-Notes) + +Overview +-------- + +* The p2 Engine is the component that supports modification to the contents of an installation profile. +* Key Concepts: + * Profile - Concrete model of an installation (e.g. install directory and the specific IUs installed) + * PhaseSet - The collection of in order phases that will be carried out during a call to the Engine. + * Phase - Phases are typically named to collect similar operations together. For example the "install" phase. + * Operand - A pair of IUs used to represent (before, after) state. The phases use operands to determine what actions to carry out. + * Touchpoint - Each IU is associated with a touchpoint which identify for what type of product it's for. For example the bundles for Eclipse are associated with the Eclipse touchpoint. + * Action - Actions are the unit of work done by the Engine and are used to modify the contents of a profile + +Engine API +---------- + +The primary means of interaction is through the perform method: + + **public MultiStatus perform(Profile profile, PhaseSet phaseSet, Operand\[\] operands, IProgressMonitor monitor)** + + +* _profile_ is the Profile you want to modify +* _phaseSet_ provides the set of Phase(s) that you want the engine to run through +* _operands_ is the set of IU pairs you want to operate over +* _monitor_ is used to provide incremental status feedback. + +Each call to the engine is atomic - either all changes happen or in the event of failure no changes occur. To the caller this is indicated by the MultiStatus object returned - OK signals success, ERROR signals a failure. + +Usage +----- + +Using a custom PhaseSet is supported but in most cases an instance of the "DefaultPhaseSet" should be used for installation operations. The DefaultPhaseSet is made up of the following in-order Phases: + +* _Collect_ is used to mirror any artifacts identified in the IUs so that they are available for later steps in the installation. +* _Unconfigure_ is used to remove the configuration information for an IU from the profile or touchpoint specific locations. +* _Uninstall_ removes the IU from the profile and performs any actions required to remove it from the touchpoint. +* _Install_ adds the IU to the profile and performs any of the actions required by the touchpoint to perform and installation +* _Configure_ is used to perform any configuration steps required for the IU. + +Each Phase is responsible for determining the appropriate Actions to execute for a particular operand. The collect phase has an implicit behaviour to download the associated artifacts. The unconfigure, uninstall, install, and configure Phases will all look for "instructions" in the touchpoint data section with the same name. For example during the "install" phase the IU's touchpoint data section is queried for "install" instructions. + +The format for the instruction section is as follows: + + **actionName( paramName1 : paramValue1, paramName2 : paramValue2, paramName3 : paramValue3);** + +An instruction may contain multiple action statements. The Phase will lookup the "actionName" using it's own phase specific actions and also those made available by the associated touchpoint. For example the set of "install" instructions for a bundle might consist of the following: + + **installBundle(bundle:${artifact});** + +* _installBundle_ is the action name +* _bundle_ is the parameter name +* _${artifact}_ is the parameter value. The value ${artifact} signifies the use of a pre-defined variable named "artifact". + +Touchpoint Instructions, Actions and Variables +---------------------------------------------- + +See [here](/Equinox/p2/Engine/Touchpoint_Instructions "Equinox/p2/Engine/Touchpoint Instructions") for information on touchpoint instructions. + +Extra Notes +----------- + +(Be aware that much of this discussion is out of date and was part of the discussion during the design of the p2 engine) + +* Is the engine only invoked for provisioning related operations? For example could it be used to obtain the log file of a particular touchpoint. + * For now the answer is no. The engine is only used for provisioning related operations that cause modification of the set of IUs installed in a profile. + +* Some additonal operations that we \*might\* need to invoke on the engine. Current thinking is that introspection operations should occur outside of the scope of an Engine session. We have not dug too deeply here so this might change. + * Reboot the targeted touchpoint. + * Compute the size of things being downloaded. It would invoke the touchpoint to actually know whether or not the IU should be obtained. + * Validation of checks contained in IUs + * Qualification, for example discovery of an eclipse install + +* Engine API: + * The OSGi DMT Admin service has an interesting API that we should mimic. Specifically DMTAdmin.getSession + * A mapping of some of the DMT Admin concepts would require us to look at the API as acting on a single profile instead of a sub-tree. + * The DMT Admin service has the concept of a DMTSession for holding the current state for deployment activity. This might be useful internally to track transactional state. It's not clear yet if we want to expose the transactionactional state in the public API however we should leave that option open. + * The parameters passed to the engine do not give us flexibility for other operations than Install and Uninstall. {Done} + * We still want to keep this grammar as small as possible: install / update / uninstall until we have concrete requirements for more operations. + * We're currently passing a "single" operation and set of operands. We should move to a model where we pass a set of operations with internal operands. This allows us to "perform" a mixture of the various operations in a single call to the Engine. + * The scope of an Engine "perform" command is a single profile and only one Engine provisioning session should be active on a profile at one time. This has some consequences for transactions that we will need to think about. + +* Engine processing model and phases: + * The engine is built on a model where there is a "fixed" set of phases + * When running a partiuclar set of operations each operation is in turn asked if it will participate in a particular phase. + * Currently the processing is done breadth first, in that first all the IUs are being fetched, then they are all installed and finally configured (where fetch, install, and configure are phases). Is that too strict? Should we allow for some phases to specify how they should be run? + * One benefit of breadth first is that ordering of operations is no longer a problem however the phase ordering is static. + * Depth first more or less has the opposite characteristics where we have greater control of the ordering of phases that make up an operation however this is at the expense of greater complexity in terms of ordering a set of provisioning operations. + * A first try at an in-order set of phases is as follows: + * collect + * validate + * uninstall / unconfigure + * update / migrate + * install / initconfig + * configure + * verify + +* Operations / Phases / Actions: + * Phase-data is represented in an IUs touchpoint data. An IU's phase-data will eventually consist of a series of atomic "actions". The actual definition of actions should probable be declarative. Currently this is done in JavaScript. + * Our mapping of operations to phases is as follows. + * Install - Collect, Validate, Install / InitConfig, Configure, Verify. + * Update - Collect, Validate, Update / Migrate, Configure, Verify. + * Uninstall - Validate, Uninstall / UnConfigure, Verify. + diff --git a/docs/Installable_Units.md b/docs/Installable_Units.md new file mode 100644 index 000000000..f56ccb8e1 --- /dev/null +++ b/docs/Installable_Units.md @@ -0,0 +1,165 @@ +Installable Units +================= + +Contents +-------- + +* [1 Installable Unit](#Installable-Unit) + * [1.1 IU Identity](#IU-Identity) + * [1.2 Enablement filter](#Enablement-filter) + * [1.3 IU dependencies and capabilities](#IU-dependencies-and-capabilities) + * [1.3.1 Capability](#Capability) + * [1.3.2 Requirement expression](#Requirement-expression) + * [1.3.3 Requirement](#Requirement) + * [1.3.4 Example](#Example) + * [1.4 Content aspect](#Content-aspect) + * [1.5 Touchpoint, touchpoint data](#Touchpoint.2C-touchpoint-data) + * [1.6 Update information](#Update-information) + * [1.7 Fixes](#Fixes) + * [1.8 Properties](#Properties) +* [2 Grouping](#Grouping) +* [3 Installable Unit fragments](#Installable-Unit-fragments) +* [4 Installable Unit best practices](#Installable-Unit-best-practices) + +Installable Unit +---------------- + +As the name implies, Installable Units (IUs for short) describe things that can be installed, updated or uninstalled. They do not contain the actual artifacts but rather essential information about such artifacts (e.g., names, ids, version numbers, dependencies, etc) and are not aware about what they deliver. They describe things. They are NOT the things. So for example an IU for a bundle is NOT the bundle. The bundle is an "artifact". The metadata allows dependencies to be structured as graphs without forcing containment relationships between nodes. Here is detailed presentation of what an installable unit is made of. + +### IU Identity + +An IU is uniquely identified by an ID and a version. + +### Enablement filter + +The enablement filter is of the form of an LDAP filter \[1\]. It indicates in which contexts an installable unit can be installed. The evaluation of this filter is done against a set of valued variables called an “evaluation context”. + +### IU dependencies and capabilities + +In the same way bundles have import and export packages, IUs have dependencies to talk about their prerequisites and provide capabilities to tell others what they offer. + +#### Capability + +A capability has the three following attributes: + +* A namespace +* A name +* A version + +We often say that an IU provides capabilities. + + +Dependencies are expressed against those capabilities to express all the requirements of an IU. This approach offers great flexibility to express dependencies. + +#### Requirement expression + +A requirement expression is composed of two parts: + +* An enablement filter of the form of an LDAP filter \[1\]. The absence of a filter is equivalent to a filter evaluating to true. When a filter evaluates to false, the requirement is ignored. +* A Conjunctive Normal Form of Requirements. + +#### Requirement + +A requirement has the following attributes: + +* A namespace +* A name +* A version range +* A greediness flag, indicates whether or not a new IU should be added to the solution to define satisfy this requirement +* A multiplicity flag, indicates whether or not multiple IUs should be added to the solution to satisfy this requirement +* An optionality flag + +Requirements are satisfied by capabilities. + + +Note that the id of the IU is also exposed as a capability in the org.eclipse.equinox.p2.iu. + +These dependencies information are used by the agent to decide what needs to be installed. For example if you are installing the org.eclipse.jdt.ui IU, the dependencies expressed will cause the transitive closure of IUs reachable to be installed. + +#### Example + +The syntax used here is not normative. In fact p2 will not define a serialization format for IUs to allow for greater flexibility in storage and manipulation. + + IU org.eclipse.swt v 3.2.0 + Capabilities: + {namespace=package, name=a, version=1.0.0} + {namespace=foo, name=b, version=1.3.0} + {namespace=package, name=c, version=4.1.0} + Requirement expressions + (true) -> + {namespace=package, name=r1, range=\[1.0.0, 2.0.0)} and + {namespace=foo, name=r1, range=\[3.2.0, 4.0.0)} + (& (os=linux) (ws=gtk)) -> + {namespace=package, name=r2, range=\[1.0.0, 2.0.0)} or + {namespace=foo, name=bar, range=\[3.2.0, 4.0.0)} + IU org.eclipse.jface v 3.3.0 + Capabilities: + {namespace=package, name=a, version=1.0.0} + {namespace=package, name=jface, version=3.1.0} + Requirement expressions: + (true) -> + {namespace=package, name=a, range=\[1.0.0, 2.0.0)} and + {namespace=foo, name=b, range=\[1.0.0, 4.0.0)} + (& (os=linux) (ws=gtk)) -> + {namespace=package, name=a, range=\[1.0.0, 1.1.0)} or + {namespace=foo, name=bar, range=\[3.2.0, 4.0.0)} + +### Content aspect + +The IU does not deliver any content. Instead it refers to artifacts. The artifacts are mirrored from an artifact server into a local artifact server on the request of touchpoints. + +### Touchpoint, touchpoint data + +IUs can be stamped with a type. Using this type, the engine identifies the touchpoint responsible for marrying the IU with the related system. The touchpoint data contains information that will be used to apply the software lifecycle (install, uninstall, update, configure, etc). + +### Update information + +The lineage information of an IU is explicit. Each IU can express the IU(s) it is an update of. This information is stored in the form of one requirement, thus allowing for an IU to be an update of multiple of its predecessors. + +We are contemplating supporting multiple requirements to allow for cases where an IU has been split into multiple IUs or where multiple IUs have merged into one. Another thing that we are contemplating the addition of "staged update" concept. This would allow for cases where an update must be applied even though an higher version exists. + +### Fixes + +Support for fixes will be added, however the format has not been decided yet. + +### Properties + +An IU can carry arbitrary properties. These properties are usually only considered by the user interface and the director. The properties targeted at the user are the one containing user readable name information (UI name, license, description, etc.) and the one allowing for better filtering of what is being shown to the user (see grouping section). The properties targeted at the director are usually used as hints/advices to the resolution process. + +For properties influencing the director, they should be such that even if the director to which these properties are targeted at is not used, the Installable Unit should still be successfully resolvable. + +Grouping +-------- + +There are various circumstances where grouping is necessary. To address this, p2 does not call out for a specific construct. Instead in p2 groups are just IUs expressing requirements on other IUs. For example, here is an excerpt of the group representing the RCP functionality of eclipse + + IU org.eclipse.rcp v 3.2.0 + Requirement expressions + (true) -> + {namespace=iu, name=org.eclipse.osgi, range=\[3.2.0, 3.3.0)} and + {namespace=iu, name=org.eclipse.jface, range=\[3.2.0, 3.3.0)} + (& (os=linux) (ws=gtk)) -> + {namespace=iu, name=org.eclipse.swt.linux.gtk, range=\[3.3.0, 3.4.0)} + (& (os=win32) (ws=win32) (arch=x86)) -> + {namespace=iu, name=org.eclipse.swt.win32.win32.x86, range=\[3.3.0, 3.4.0)} + +For filtering in the user interface a property flagging group as such is set on the IU. + +Installable Unit fragments +-------------------------- + +Installable unit fragments are installable units that complement an existing installable unit. When a fragment applies to an installable unit, it is being attached to this installable unit. A fragment can apply to multiple installable unit. Once the fragment has been attached its content is seamlessly accessible from the installable unit. + +An installable unit fragment can not modify the dependencies of the installable unit to which it is attached. + +Installable unit fragments are used to deliver touchpoint data common to multiple installable units. It could also be used to deliver metadata translation. + +Installable Unit best practices +------------------------------- + +The information contained in an installable unit must be kept generic to allow for reuse. For example, the org.eclipse.equinox.common bundle needs to be started. However the start level at which it needs to be started differs based on the scenario where the bundle is being used (e.g. in a RCP context, it needs to be started at level 2, whereas it needs to be started at level 3 in the server side context). + +Therefore for the IU to stay generic the touchpoint data for org.eclipse.equinox.common can not specify a start level. The IU should only contain information related to the dependencies and capabilities that the IU has. + +In order to deliver the start level information, installable unit fragments will be created. In our example we would have one installable unit fragment to be used in an RCP Context and another one for the server side context. Each of these IU fragments will have its own unique ID and will not collide with each others. + diff --git a/docs/Query_Language_for_p2.md b/docs/Query_Language_for_p2.md new file mode 100644 index 000000000..a8fda206d --- /dev/null +++ b/docs/Query_Language_for_p2.md @@ -0,0 +1,468 @@ +Equinox/p2/Query Language for p2 +================================ + +Contents +-------- + +* [1 Background](#Background) + * [1.1 The problem](#The-problem) + * [1.2 Bugzilla](#Bugzilla) +* [2 Language Design](#Language-Design) + * [2.1 There are two major ingredients in p2ql:](#There-are-two-major-ingredients-in-p2ql:) + * [2.1.1 The language itself](#The-language-itself) + * [2.1.2 How the language integrates with Java objects](#How-the-language-integrates-with-Java-objects) + * [2.2 Implementation](#Implementation) + * [2.3 Java bindings](#Java-bindings) + * [2.3.1 Types of queries](#Types-of-queries) + * [2.3.2 Variable binding](#Variable-binding) + * [2.4 Boolean Queries](#Boolean-Queries) + * [2.5 Collection Queries](#Collection-Queries) + * [2.6 Special operators](#Special-operators) + * [2.7 Functions](#Functions) + * [2.8 Collection functions](#Collection-functions) + * [2.9 Query parameters](#Query-parameters) +* [3 Basic IQuery examples](#Basic-IQuery-examples) +* [4 Localization](#Localization) +* [5 Currying](#Currying) +* [6 Performance comparison: p2QL traversal versus the p2 Slicer](#Performance-comparison:-p2QL-traversal-versus-the-p2-Slicer) +* [7 Java API](#Java-API) +* [8 The BNF](#The-BNF) + +### Background + +As p2 gets more widely used, we predict that repositories will grow larger and larger over time. The Galileo repository alone contains 3866 IU's (in one single service release). Starting with Helios, we plan to keep everything over time which will put the IU count close to 10.000. And that's just one year of Eclipse IP approved material. OSGi is gaining in popularity and p2 is getting increasingly popular. Some companies plan to map the central maven repository as p2. We can soon expect to see p2 repositories with over a 100.000 IU's in them. + +#### The problem + +p2 has a query mechanism today that makes it hard to create a repository implementation that is based on a database. It is also diffiult to create an efficient client/server solution. The reason for this is that most of the queries are written as implementations of the IQuery interface, either as ContextQuery derivates that need access to all rows in the database, or as MatchQuery derivates with a java method to which all rows need to be passed. + +There is no uniform way to translate these queries into an expression that can be understood by a database or by something running in another process. + +Some attempts have been made to rectify this and a repository implementation could potentially make some intelligent decisions based on information derived from looking at the query (what class it is, and in some cases, what properties that are exposed). While this is an improvement, the general problem remains; A large amount of the queries are still more or less completely black-boxed and it's enough with one single such query to force a full selection of everything from the database/remote server. + +The p2QL expression language discussed here is an attempt to rectify this problem. It can be used in two ways: + +* Coexisting with current IQuery / IQueryable + + Using the ExpressionContextQuery (implements IQuery), or the ExpressionQuery (implements IMatchQuery), it can coexist with todays IQuery/IQueryable. The queries are created based on an expression and an array of parameters. See section [#IQuery examples](#IQuery-examples). + +* With more explicit separation of concerns + + It can also be used in an alternative solution where an expression returns an Iterator. This usage scenario is particularly useful when implementing remote repositories or implementations on top of a database. It is also a very efficient way of querying for things that you don't want to collect in memory. + + +#### Bugzilla + +We discussed the use of a query language at length on the p2 meeting on November 9 and November 16. The issue is captured in bugzilla [Create a QueryLanguage for p2](https://bugs.eclipse.org/bugs/show_bug.cgi?id=294691). + +### Language Design + +TODO: Add some text here to explain where the ideas stem from (xtend, xtext, Scala, Java) and other motivation behind the choice of syntax. + +#### There are two major ingredients in p2ql: + +* The language itself +* How the language integrates with and interacts with Java objects + +##### The language itself + +A query language at its heart is simply an expression evaluation language for two major kinds of expressions: + +* Iterating over a collection and subsetting it in some way +* Supplying a boolean expression that is used as a callback when some other object iterates over a collection and computes a subset. + +That, then is what p2ql is: a simple language for describing subsets of collections. + +##### How the language integrates with Java objects + +Since p2ql must execute inside a Java runtime, it must interact with Java objects. In the context of P2, there are two kinds of Java objects that a query language must interact with: + +* A P2 IQueryable +* Arbitrary Java objects that are passed in as parameters to the query + +#### Implementation + +p2ql itself is a small dynamically-typed language in the Smalltalk tradition. It has receivers (objects) and messages (methods), and whether an object can receive a message is determined dynamically at runtime. Even so, because it automatically takes advantage of various opportunities for concurrency, it is very fast. It is also a pure functional language, which means that all variables are final. As was noted before, it is an expression language--which means that while you can use predefined objects and functions, you cannot define your own objects nor your own named functions. + +However, you can define anonymous functions (lambdas, closures, anonymous runnables) in the same way that a SQL "where" clause is really an anonymous function or a nested SQL query is also an anonymous function. + +#### Java bindings + +When you define a p2ql query, you are really dynamically defining and running a new method on a P2 IQueryable object, usually a metadata repository. + +IQuery q = QueryUtil.createMatchQuery("this.id == $0", id); +metadataRepository.query(q); + +##### Types of queries + +There are two main kinds of queries. + +* Queries that define a boolean expression to evaluate against every element in the IQueryable. This is similar to just defining a "where" clause in SQL. + +IQuery query = QueryUtil.createMatchQuery("this.id == $0", id); + +* Queries that subset the IQueryable's "collection contents" and return a new collection + +IQuery query = QueryUtil.createQuery("everything.select(x | x.id == $0)", id); + +##### Variable binding + +Within these types of queries, variables are bound in the following ways: + +* In a match query, "this" refers to the "current row object" in the IQueryable. + +* In a generic query, "everything" refers to the IQueryable collection itself. + +* Dot notation follows the following rules: + * If the receiver of the message is an Iterable or an IQueryable, the message must be a function that iterates across the collection's elements. (These predefined functions are described below.) + * If the receiver of the message is a POJO, then the property name denoted by the message is converted into a "getRegurlarProperty" or "isBooleanProperty" Java call and the property value is returned. + +So, in the examples above, this.id is the same as writing "rowObject.getId()" in Java. + +If you prefer, "this" or "everything" can be left out. In other words, unqualified property names or function calls always refer to the "this" or the "everything" object. So, the following queries are equivalent to the first examples. + +IQuery query = QueryUtil.createMatchQuery("id == $0", id); + +IQuery query = QueryUtil.createQuery("select(x | x.id == $0)", id); + +The $0, $1, ..., $n variables simply refer to the Java parameter objects that were passed into the query. The same dot operator rules apply to these variables as apply to the "everything" and "this" variables: If the variable is an Iterable, you have to call a function on it that processes its contents. Otherwise, you can treat it like a Java Bean and the dot operator accesses its properties. + +With that introduction, let's take a look at this material again, but expressed more formally. + +#### Boolean Queries + +Queries can be written as simple predicates such as "id == $0". When executed, the predicate will be evaluated once for each row in the queried material. This form is used for the IMatchQuery implementation. The current value is available in a predefined variable named _this_. Like in Java, _this_ is normally implicit so that: + +id == $0 + +is actually the same as: + +this.id == $0 + +#### Collection Queries + +A query can also be declared as something that acts on "everything". This is particularly useful when things like search for latest or doing requirements traversal. These queries make use of a predefined variable named _everything_. When accessed, that variable returns an iterator over all queried material. As with _this_ in a predicate query, _everything_ is something that you don't normally need to specify, i.e.: + +select(x | x.id == $0) + +is the same as: + +everything.select(x | x.id == $0) + +A boolean query can always be written as a collection query. These two queries are fully equivalent: + +Boolean: id == $0 +Collection: select(x | x.id == $0) + +#### Special operators + +* **Matches operator '~='** + + A large amount of queries involve versions, version ranges, and capability matching. So managing that is important. Another thing that is important is to be able to support filtering based on ID's. All of this is now also built in to the language through the matches operator **~=**. It can be though of as the 'satisfies' method on an installable unit or provided capability, as 'is included in' when used with a version and version range, as a plain matches when used with a string and a pattern, or as 'instanceof' when comparing with a class. + + The following things can be matched using the operator + + | LHS | RHS | Implemented as | + | --- | --- | --- | + | IInstallableUnit | IRequirement | lhs.satisfies(rhs) | + | IInstallableUnit | Filter | rhs.matches(lhs.properties) | + | IInstallableUnit | IUpdateDescriptor | rhs.isUpdateOf(lhs) | + | Version | VersionRange | rhs.isIncluded(lhs) | + | Map | Filter | rhs.match(lhs) | + | Dictionary | Filter | rhs.match(lhs) | + | String | SimplePattern | rhs.isMatch(lhs) | + | String | VersionRange | rhs.isIncluded(version(lhs)) | + | | Class | rhs.isInstance(lhs) | + | Class | Class | rhs.isAssignableFrom(lhs) | + +* **And operator '&&'** + This operator checks if the first operand evaluates to a boolean. If it is, the it is assumed that all other operands also are booleans and the full evaluation is _true_ if all its operands evaluate to _true_. If the first result was not a boolean, then it is assumed that it is a collection and that all other operands also evaluates collections. The operator will then function as a **intersect** operator and the result consists those elements that could be found in all collections. +* **Or operator '||'** + This operator checks if the first operand evaluates to a boolean. If it is, the it is assumed that all other operands also are booleans and the full evaluation is _false_ if none of its operands evaluate to _true_. If the first result was not a boolean, then it is assumed that it is a collection and that all other operands also evaluates collections. The operator will then function as a **union** operator and the result is the unique sum of all elements from all collections. + +#### Functions + +A small number of functions are available to assist with some common problems. Example: An IRequiredCapability.getFilter() returns a String. I only want the capabilities that has a filter that match certain properties. Consequently, I need an instance of an OSGi Filter. Not a string. I can do this using the _filter_ function. Like this: + +requirements.exists(rc | rc.filter == null || $1 ~= filter(rc.filter)) + +The currently available constructors are: + +| name | arguments | creates | +| --- | --- | --- | +| filter | string | an instance of org.osgi.framework.Filter | +| version | string | a p2 Version | +| range | string | a p2 VersionRange | +| class | string | a Class | +| boolean | string | a boolean | +| set | comma separated list of expressions | an instance of java.util.HashSet | +| iquery | IQuery \[, variable \[, collector\]\] | The result of evaluating the query | + +A note about the **iquery** constructor. +The constructor operates in one of two modes depending on its IQuery argument. + +If the argument implements IMatchQuery, the the constructor will return the boolean result of invoking the isMatch(value) method on that query. The value is picked from the variable and the default variable is 'item'. It is considered an error to pass a third argument in combination with an IMatchQuery. + +When the query does not implement the IMatchQuery interface, it is considered a context query and its method perform(iterator, collector) will be called. The iterator is picked from the variable and the default variable is 'everything'. The collector can be passed in as the third argument. If no third argument is present, a new collector will be created. + +#### Collection functions + +A number of functions was added to enable various manipulations of collections. A common way of writing collection functions is: + +elements.someFunction(element | ) + +Here are the functions that are available in the p2QL language: + +* **Select** + The **select** function will evaluate its expression, once for each element, and return a new collection containing all elements for which the evaluation result was _true_. Example: + + select(x | x.id == 'org.eclipse.equinox.p2.ql') + +* **Collect** + The **collect** collects the result of evaluating each element in a new collection. Example: + + collect(x | x.id) + + returns a collection consisting of the value of the id property of each element in the original collection. + +* **Exists** + The **exists** function will evaluate an expression for each element in the collection until an evaluation yields _true_. If that happens, the result of the whole evaluation yields _true_. If no such element is found, the whole evaluation yields _false_. Example: + + providedCapabilities.exists(p | p.namespace == 'osgi.bundle') + +* **All** + The **all** function will evaluate an expression for each element until an evaluation yields _false_. If that happens, the whole evaluation yields _false_. If no such element is found, the whole evaluation yields _true_. Example: + + $0.all(rc | this ~= rc) + + Assuming $0 is a list of required capabilities, this query asks for items that fulfill all requirements. +* **First** + The **first** function will evaluate an expression for each element in the collection until an evaluation yields _true_. If that happens, the result of the whole evaluation yields that element. If no such element is found, the whole evaluation yields _null_. Example: + + providedCapabilities.first(p | p.namespace == 'java.package') + + Returns the first provided capability with namespace 'java.package'. +* **Flatten** + Intended to be applied on collections of collections. Yields a single collection with all elements from the source collections, in the order they are evaluated.Example: + + collect(x | x.requirements).flatten() + + Yields a collection with all required capabilities from all iu's that the collect was applied on. +* **Latest** + Some queries must make sure that the result only contains the latest version of each found IU. The special function _latest_ to makes that possible. The function will require that the collection elements implement the IVersionedId interface. Here is an example of how to use it: + + select(x | x.id == $0).latest() + + As a convenience, it is also possible to write: + + latest(x | x.id == $0) + +* **Limit** + It is sometimes desirable to limit the number of rows returned by a query. The function 'limit' can be used for that: + + select(...).limit(100) + +* **Unique** + This function will ensure that the resulting collection contains unique elements. The function can operate in two modes, with or without a cache argument. I.e. + + x.unique() + + or + + x.unique(cache) + + The latter expects the argument to be a _java.util.Set_ that it can use to enforce the uniqueness of the element. This enables the result to be unique in a larger scope then the collection itself. +* **Traverse** + A common scenario in p2 is that you want to start with a set of roots and then find all items that fulfill the root requirements. Those items in turn introduce new requirements so you want to find them too. The process continues until no more requirements can be satisfied. This type of query can be performed using the **traverse** function. The function will evaluate an expression, once for each element, collect elements for which the evaluation returned _true_, then then re-evaluate using the collected result as source of elements. No element is evaluated twice. This continues until no more elements are found: + + $0.traverse(parent | parent.requirements.collect(rc | select(iu | ~= rc)).flatten()) + + This is of course a fairly naive slicing mechanism. See [#Currying](#Currying) below for more advanced examples. + +#### Query parameters + +The query must be parameterised so that expression parsing can be done once and then executed multiple times. A parameter is expressed as $_n_ where _n_ is either the parameter index, originating from 0. + +### Basic IQuery examples + +Here are some examples of how to use the expressions with IQuery: Query for all IU's that has an id: + +IQuery query = QueryUtil.createMatchQuery("id == $0", id); + +Query for the latest IU of some specific id: + +IQuery query = QueryUtil.createQuery("latest(x | x.id == $0)", id); + +Query an artifact repository for all keys with a specific classifier: + +IQuery query = QueryUtil.createMatchQuery(IArtifactKey.class, "classifier == $0", classifier); + +Query for the latest IU that matches a specific version range. Since the second parameter is a VersionRange, the ~= operator is interpreted as _isIncluded_: + +IQuery query = QueryUtil.createQuery("latest(x | x.id == $0 && x.version ~= $1)", id, range); + +Query for an IU that has a specific property set: + +IQuery query = QueryUtil.createMatchQuery("properties\[$0\] == $1", key, value); + +The same query, but this time for multiple possible values: + +IQuery query = QueryUtil.createMatchQuery("$1.exists(v | properties\[$0\] == v)", key, new Object\[\] { v1, v2, v3 }); + +Query for all categories found in the repository: + +IQuery query = QueryUtil.createMatchQueryx("properties\[$0\] == true", IInstallableUnit.PROP\_TYPE\_CATEGORY); + +Query for all IU's that fulfil at least one of the requirements from another IU. Since the first parameter is a list of IRequirements, the ~= applied to each each IU using _satisfies_. + +IQuery query = QueryUtil.createMatchQuery("$0.exists(rc | this ~= rc)", iu.getRequirements()); + +Query for the latest version of all patches: + +IQuery query = QueryUtil.createQuery("latest(x | x ~= $0)", IInstallableUnitPatch.class); + +Query for all IU's affected by a patch: + +IQuery query = QueryUtil.createMatchQuery("$0.exists(rcs | rcs.all(rc | this ~= rc))", patch.getApplicabilityScope()); + +### Localization + +An installable unit may have locale specific properties. Such properties may be stored in the IU itself or in fragments that provide localization for the IU in the 'org.eclipse.equinox.p2.localization' namespace. p2QL will interpret the member _translations_ applied on an IU as a map of translated properties, hiding all the details. As an example, this query: + +translations\['license'\] ~= /\*kommersiellt bruk\*/ + +when used with a default Locale("sv_SE") would yield all IU's that has a license, translated into Swedish, that contains the exact phrase 'kommersiellt bruk'. + +A special class named **KeyWithLocale** was added to allow queries for entries that does not match the current locale. So the above query can be written as: + +QueryUtil.createMatchQuery("this.translations\[$0\] ~= /\*kommersiellt bruk\*/", new KeyWithLocale("license", new Locale("sv", "SE"))) + +### Currying + +Want some add some spice? Probably... + +Consider the traversal function and the example we had above: + +$0.traverse(parent | parent.requirements.collect(rc | select(iu | iu ~= rc)).flatten()) + +Now let us assume that we want to perform this traversal and filter the requirements based on platform. Our first attempt could look something like this. + +$0.traverse(parent | parent.requirements.select(rc | rc.filter == null || $1 ~= rc.filter).collect(rc | select(iu | iu ~= rc)).flatten()) + +This would however be very inefficient since many requirements exists in several IU's. During the traversal there is no point traversing a requirement more then once so it would be good to "remember" the perused requirements. + +The p2QL syntax permits something generally referred to as _currying_. Currying means that you can provide more parameters then the single one that represents the current element to the expression of a collection function. So far, we've seen examples using the syntax: + +select(x | ) + +This is actually a short form for the longer: + +select(_, {x | }) + +The select provides one parameter to each iteration. It's value is always provided reachable using the special operator '_'. In this case, the variable x maps to the parameter _ since they have the same positional offset. I can add more parameters by declaring: + +select(a, b, _, {x, y, z, }) + +Variable x will now have the value of a, y the value of b, and z the value of _. + +So why is this important? Well, the initializers are just evaluated once for one call to select. The expression however, is evaluated once for each new value of _.

Let us now return to the traversal again. Because with this syntax, we can actually specify a global set that we can pass to the unique function so that no requirement is perused twice: + +$0.traverse(set(), _, { rqCache, parent | parent.requirements.unique(rqCache).select(rc | rc.filter == null || $1 ~= rc.filter).collect(rc | select(iu | iu ~= rc)).flatten()}) + +### Performance comparison: p2QL traversal versus the p2 Slicer + +A test was performed using the query example above on the Galileo repository, using a specific version of the org.eclipse.sdk.feature.group IU as the root. The test produced a result with 411 IU's in < 12 ms average. I compared this with another test that used the p2 Slicer. The number of IU's produced was exactly the same. The slicer however, took > 22 ms (it also uses a capability cache internally). Not very scientific perhaps but repeated tests produce similar results. + +I'm sure the Slicer can be optimized and achieve results that might be even slightly better then the traversal but I still think this proves a point. p2QL Performance is very good at this point. + +### Java API + +The expression tree created by the parser must be well documented and easy to use so that queries can be created programmatically. Since all expressions are immutable and without context, they can be combined freely. Hence, code like this is fully possible: + +// Create some expressions. Note the use of identifiers instead of +// indexes for the parameters + +IExpressionFactory factory = ExpressionUtil.getFactory(); +IExpression item = factory.variable("this"); +IExpression cmp1 = factory.equals(factory.member(item, "id"), factory.parameter(0)); +IExpression cmp2 = factory.equals( + factory.at(factory.member(item, "properties"), factory.parameter(1)), + factory.parameter(2)); + +IExpression everything = factory.variable("everything"); +IExpression lambda = factory.lambda(item, factory.and(new IExpression\[\] {cmp1, cmp2})); +IExpression latest = factory.latest(factory.select(everything, lambda)); + +// Create the query +IQuery query = QueryUtil.createQuery(latest "test.bundle", "org.eclipse.equinox.p2.type.group", Boolean.TRUE); + +### The [BNF](http://en.wikipedia.org/wiki/Backus%E2%80%93Naur_Form) + +condition + : orExpression ( '?' orExpression ':' orExpression )? + ; + +orExpression : andExpression ( '||' andExpression )* ; + +andExpression : binaryExpression ( '&&' binaryExpression )* ; + +binaryExpression : notExpression ( op notExpression )?; + +op : '==' | '!=' | '>' | '>=' | '<' | '<=' | '~=' ; + +notExpression + : '!' notExpression + | collectionExpression + ; + +collectionExpression + : memberExpression ( '.' collectionFunction )* + ; + +memberExpression : function ( ( '.' ID ) | ( '\[' memberExpression '\]' ) )* ; + +function + : ( filter | version | range | class) '(' condition ')' + | set '(' ( condition ( ',' condition )* )? ')' + | unaryExpression + ; + +collectionFunction + : ( select | reject | collect | exists | all | traverse | first ) '(' lambdaDefinition ')' + | limit '(' memberExpression ')' + | unique '(' memberExpression? ')' + | latest '(' lambdaDefinition? ')' + | flatten '(' lambdaDefinition? ')' + ; + +lambdaDefinition + : initializer ( ',' initializer )* ( ',' '{' lambda '}' )? + | '{' lambda '}' + | lambda + ; + +initializer + : '_' + | condition + ; + +lambda + : ( ID ( ',' ID )* )? '|' condition + ; + +unaryExpression + : '(' condition ')' + | '\[' condition ( ',' condition )* '\]' // #array construct + | STRING + | INT + | parameter + | 'null' + | 'true' + | 'false' + | ID + ; + +parameter + : '$' INT | ID + ; +