Skip to content

Commit

Permalink
Add support of remote task on remote Pipeline
Browse files Browse the repository at this point in the history
We now support remote tasks on remote Pipeline, allowing to share
a remote Pipeline across multiple repositories.

User can override tasks from the remote pipeline by adding a task with
the same name.

Signed-off-by: Chmouel Boudjnah <[email protected]>
  • Loading branch information
chmouel authored and savitaashture committed Nov 9, 2023
1 parent d81b08e commit 0edaa50
Show file tree
Hide file tree
Showing 10 changed files with 666 additions and 110 deletions.
82 changes: 71 additions & 11 deletions docs/content/docs/guide/resolver.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ command to learn on how to use it.
location with annotations on PipelineRun.

If the resolver sees a PipelineRun referencing a remote task or a Pipeline in
a PipelineRun or a PipelineSpec it will automatically inlines them.
a PipelineRun or a PipelineSpec it will automatically inline them.

If multiple annotations reference the same task name the resolver will pick the
first one fetched from the annotations.
Expand Down Expand Up @@ -128,18 +128,19 @@ will fetch the task directly from that remote URL :
### Remote HTTP URL from a private GitHub repository

If you are using `GitHub` and If the remote task URL uses the same host as where
the repo CRD is, PAC will use the GitHub token and fetch the URL using the
the repository CRD is, PAC will use the GitHub token and fetch the URL using the
GitHub API.

For example if you have a repo URL looking like this :
For example if you have a repository URL looking like this :

<https://github.com/organization/repository>

and the remote HTTP URLs is a referenced GitHub "blob" URL:

<https://github.com/organization/repository/blob/mainbranch/path/file>

if the remote HTTP url has a slash (/) in the branch name you will need to html encode with the `%2F` character, eg:
if the remote HTTP URL has a slash (/) in the branch name you will need to HTML
encode with the `%2F` character, example:

<https://github.com/organization/repository/blob/feature%2Fmainbranch/path/file>

Expand All @@ -150,14 +151,14 @@ GitHub app token are scoped to the owner or organization where the repository is
If you are using the GitHub webhook method you are able to fetch any private or
public repositories on any organization where the personal token is allowed.

There is settings you can set in the pac `Configmap` to control that behavior, see the
There is settings you can set in the pac `Configmap` to control that behaviour, see the
`secret-github-app-token-scoped` and `secret-github-app-scope-extra-repos` settings in the
[settings documentation](/docs/install/settings).

### Tasks or Pipelines inside the repository

Additionally, you can as well have a reference to a task or pipeline from a YAML file inside
your repo if you specify the relative path to it, for example :
your repository if you specify the relative path to it, for example :

```yaml
pipelinesascode.tekton.dev/task: "[share/tasks/git-clone.yaml]"
Expand All @@ -174,18 +175,77 @@ If the object fetched cannot be parsed as a Tekton `Task` it will error out.

## Remote Pipeline annotations

Remote Pipeline can be referenced by annotation, this allows you to share your Pipeline definition across.
Remote Pipeline can be referenced by annotation, allowing you to share a Pipeline across multiple repositories.

Only one Pipeline is allowed in annotation.
Only one Pipeline is allowed on the `PipelineRun` annotation.

An annotation to a remote pipeline looks like this :
An annotation to a remote pipeline looks like this, using a remote URL:

```yaml
pipelinesascode.tekton.dev/pipeline: "https://git.provider/raw/pipeline.yaml
```

It supports remote URL and files inside the same Git repository.
or from a relative path inside the repository:

```yaml
pipelinesascode.tekton.dev/pipeline: "./tasks/pipeline.yaml
```

Fetching `Pipelines` from the [Tekton Hub](https://hub.tekton.dev) is not currently supported.

### Overriding tasks from a remote pipeline on a PipelineRun

Remote task annotations on the remote pipeline are supported. No other
annotations like `on-target-branch`, `on-event` or `on-cel-expression` are
supported.

If a user wants to override one of the tasks from the remote pipeline, they can do
so by adding a task in the annotations that has the same name In their `PipelineRun` annotations.

For example if the user PipelineRun contains those annotations:

```yaml
kind: PipelineRun
metadata:
annotations:
pipelinesascode.tekton.dev/pipeline: "https://git.provider/raw/pipeline.yaml
pipelinesascode.tekton.dev/task: "./my-git-clone-task.yaml
```

and the Pipeline referenced by the `pipelinesascode.tekton.dev/pipeline` annotation
in "<https://git.provider/raw/pipeline.yaml>" contains those annotations:

```yaml
kind: Pipeline
metadata:
annotations:
pipelinesascode.tekton.dev/task: "git-clone"
```

In this case if the `my-git-clone-task.yaml` file in the root directory is a
task named `git-clone` it will be used instead of the `git-clone` on the remote
pipeline that is coming from the Tekon Hub.

{{< hint info >}}
[Tekton Hub](https://hub.tekton.dev) doesn't currently have support for `Pipeline`.
Task overriding is only supported for tasks that are referenced by a `taskRef`
to a `Name`, no override is done on `Tasks` embedded with a `taskSpec`. See
[Tekton documentation](https://tekton.dev/docs/pipelines/pipelines/#adding-tasks-to-the-pipeline) for the differences between `taskRef` and `taskSpec`:
{{< /hint >}}

### Tasks or Pipelines Precedence

From where tasks or pipelines of the same name takes precedence?

for remote Tasks, when you have a `taskRef` on a task name, pac will try to find the task in this order:

1. A task matched from the PipelineRun annotations
2. A task matched from the remote Pipeline annotations
3. A task matched fetched from the Tekton directory
(the tasks from the `.tekton` directory and its subdirs are automatically included)

for remote Pipelines referenced on a `pipelineRef`, pac will try to match a
pipeline in this order:

1. Pipeline from the PipelineRun annotations
2. Pipeline from the Tekton directory (pipelines are automatically fetched from
the `.tekton` directory and subdirs)
2 changes: 1 addition & 1 deletion pkg/action/patch_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ func TestPatchPipelineRun(t *testing.T) {
observer, _ := zapobserver.New(zap.InfoLevel)
logger := zap.New(observer).Sugar()

testPR := tektontest.MakePR("namespace", "force-me", []pipelinev1.ChildStatusReference{
testPR := tektontest.MakePRStatus("namespace", "force-me", []pipelinev1.ChildStatusReference{
tektontest.MakeChildStatusReference("first"),
tektontest.MakeChildStatusReference("last"),
tektontest.MakeChildStatusReference("middle"),
Expand Down
7 changes: 5 additions & 2 deletions pkg/matcher/annotation_tasks_install.go
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ func (rt RemoteTasks) GetTaskFromAnnotations(ctx context.Context, annotations ma

// GetPipelineFromAnnotations Get pipeline remotely if they are on Annotations
// TODO: merge in a generic between the two
func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotations map[string]string) ([]*tektonv1.Pipeline, error) {
func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotations map[string]string) (*tektonv1.Pipeline, error) {
ret := []*tektonv1.Pipeline{}
pipelinesAnnotation, err := grabValuesFromAnnotations(annotations, pipelineAnnotationsRegexp)
if err != nil {
Expand All @@ -225,6 +225,9 @@ func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotation
if len(pipelinesAnnotation) > 1 {
return nil, fmt.Errorf("only one pipeline is allowed on remote resolution, we have received multiple of them: %+v", pipelinesAnnotation)
}
if len(pipelinesAnnotation) == 0 {
return nil, nil
}
for _, v := range pipelinesAnnotation {
data, err := rt.getRemote(ctx, v, false)
if err != nil {
Expand All @@ -239,7 +242,7 @@ func (rt RemoteTasks) GetPipelineFromAnnotations(ctx context.Context, annotation
}
ret = append(ret, pipeline)
}
return ret, nil
return ret[0], nil
}

// getTaskFromLocalFS get task locally if file exist
Expand Down
4 changes: 2 additions & 2 deletions pkg/matcher/annotation_tasks_install_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -463,10 +463,10 @@ func TestGetPipelineFromAnnotations(t *testing.T) {
assert.Assert(t, len(fakelog.FilterMessageSnippet(tt.wantLog).TakeAll()) > 0, "could not find log message: got ", fakelog)
}
assert.NilError(t, err)
assert.Assert(t, len(got) > 0, "GetPipelineFromAnnotations() error no pipelines has been processed")
assert.Assert(t, got != nil, "GetPipelineFromAnnotations() error no pipelines has been processed")

if tt.gotPipelineName != "" {
assert.Equal(t, tt.gotPipelineName, got[0].GetName())
assert.Equal(t, tt.gotPipelineName, got.GetName())
}
})
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/pipelineascode/pipelineascode_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -517,7 +517,7 @@ func TestRun(t *testing.T) {
},
Repositories: tt.repositories,
PipelineRuns: []*pipelinev1.PipelineRun{
tektontest.MakePR("namespace", "force-me", []pipelinev1.ChildStatusReference{
tektontest.MakePRStatus("namespace", "force-me", []pipelinev1.ChildStatusReference{
tektontest.MakeChildStatusReference("first"),
tektontest.MakeChildStatusReference("last"),
tektontest.MakeChildStatusReference("middle"),
Expand Down
2 changes: 1 addition & 1 deletion pkg/reconciler/controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ func TestCheckStateAndEnqueue(t *testing.T) {
})

// Create a new PipelineRun object with the "started" state label.
testPR := tektontest.MakePR("namespace", "force-me", []pipelinev1.ChildStatusReference{
testPR := tektontest.MakePRStatus("namespace", "force-me", []pipelinev1.ChildStatusReference{
tektontest.MakeChildStatusReference("first"),
tektontest.MakeChildStatusReference("last"),
tektontest.MakeChildStatusReference("middle"),
Expand Down
105 changes: 105 additions & 0 deletions pkg/resolve/remote.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
package resolve

import (
"context"
"fmt"

"github.com/openshift-pipelines/pipelines-as-code/pkg/matcher"
)

type NamedItem interface {
GetName() string
}

func alreadySeen[T NamedItem](items []T, item T) bool {
for _, value := range items {
if value.GetName() == item.GetName() {
return true
}
}
return false
}

// getRemotes will get remote tasks or Pipelines from annotations.
//
// It already has some tasks or pipeline coming from the tekton directory stored in [types]
//
// The precedence logic for tasks is in this order:
//
// * Tasks from the PipelineRun annotations
// * Tasks from the Pipeline annotations
// * Tasks from the Tekton directory
//
// The precedence logic for Pipeline is first from PipelineRun annotations and
// then from Tekton directory
func getRemotes(ctx context.Context, rt *matcher.RemoteTasks, types TektonTypes) (TektonTypes, error) {
remoteType := &TektonTypes{}
for _, pipelinerun := range types.PipelineRuns {
if len(pipelinerun.GetObjectMeta().GetAnnotations()) == 0 {
continue
}

// get first all the tasks from the pipelinerun annotations
remoteTasks, err := rt.GetTaskFromAnnotations(ctx, pipelinerun.GetObjectMeta().GetAnnotations())
if err != nil {
return TektonTypes{}, fmt.Errorf("error getting remote task from pipelinerun annotations: %w", err)
}

for _, task := range remoteTasks {
if alreadySeen(remoteType.Tasks, task) {
rt.Logger.Infof("skipping duplicated task %s in annotations on pipelinerun %s", task.GetName(), pipelinerun.GetName())
continue
}
remoteType.Tasks = append(remoteType.Tasks, task)
}

// get the pipeline from the remote annotation if any
remotePipeline, err := rt.GetPipelineFromAnnotations(ctx, pipelinerun.GetObjectMeta().GetAnnotations())
if err != nil {
return TektonTypes{}, fmt.Errorf("error getting remote pipeline from pipelinerun annotation: %w", err)
}

if remotePipeline != nil {
remoteType.Pipelines = append(remoteType.Pipelines, remotePipeline)
}
}

// grab the tasks from the remote pipeline
for _, pipeline := range remoteType.Pipelines {
if pipeline.GetObjectMeta().GetAnnotations() == nil {
continue
}
remoteTasks, err := rt.GetTaskFromAnnotations(ctx, pipeline.GetObjectMeta().GetAnnotations())
if err != nil {
return TektonTypes{}, fmt.Errorf("error getting remote tasks from remote pipeline %s: %w", pipeline.GetName(), err)
}

for _, remoteTask := range remoteTasks {
if alreadySeen(remoteType.Tasks, remoteTask) {
rt.Logger.Infof("skipping remote task %s from remote pipeline %s as already defined in pipelinerun", remoteTask.GetName(), pipeline.GetName())
continue
}
remoteType.Tasks = append(remoteType.Tasks, remoteTask)
}
}

ret := TektonTypes{
PipelineRuns: types.PipelineRuns,
}
// first get the remote types and then the local ones so remote takes precedence
for _, task := range append(remoteType.Tasks, types.Tasks...) {
if alreadySeen(ret.Tasks, task) {
rt.Logger.Infof("overriding task %s coming from tekton directory by an annotation task on the pipeline or pipelinerun", task.GetName())
continue
}
ret.Tasks = append(ret.Tasks, task)
}
for _, remotePipeline := range append(remoteType.Pipelines, types.Pipelines...) {
if alreadySeen(ret.Pipelines, remotePipeline) {
rt.Logger.Infof("overriding pipeline %s coming from tekton directory by the annotation pipelinerun", remotePipeline.GetName())
continue
}
ret.Pipelines = append(ret.Pipelines, remotePipeline)
}
return ret, nil
}
Loading

0 comments on commit 0edaa50

Please sign in to comment.