Skip to content

Commit

Permalink
fix: improve best practices section
Browse files Browse the repository at this point in the history
  • Loading branch information
wrussell1999 committed Nov 5, 2024
1 parent 6cd5e82 commit 21bed2f
Show file tree
Hide file tree
Showing 9 changed files with 153 additions and 100 deletions.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
title: Naming conventions
title: Naming Conventions
icon: /docs/icons/best-practices.svg
---

This page describes common naming conventions to keep your flows and tasks well-organized and consistent.
Common naming conventions to keep your flows and tasks well-organized and consistent.

## Namespace Naming Conventions

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Managing pip package dependencies
title: Managing pip Package Dependencies
icon: /docs/icons/best-practices.svg
---

Expand Down
149 changes: 149 additions & 0 deletions content/docs/14.best-practices/5.expressions-with-namespace-files.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
---
title: Expressions with Namespace Files
icon: /docs/icons/best-practices.svg
---

How to pass expressions to Namespace Files.

You can write the script inline in your flow, and use expressions as part of this inline script. Here is an example flow whih has an expression within the inline script:

```yaml
id: expressions_inline
namespace: company.team

inputs:
- id: uri
type: URI
defaults: https://www.google.com/

tasks:
- id: inline_script
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
script: |
import requests
url = "{{ inputs.uri }}"
response = requests.get(url)
if response.status_code == 200:
print(response.text)
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
```
This approach is very convenient for scripts that are specific to the flow, and does not allow us to have a separate file for our code. Having a separate file has multiple benefits:
- There's multiple files of code being used - common with larger projects
- The code is long and will be hard to maintain directly inside of the workflow - this makes the workflow harder to maintain too
- The files are written and tested locally and synced to Kestra through Git
- The files are used across multiple flows - this avoids repeating the code in each workflow
You cannot directly use [Expressions](../expressions/index.md) inside of [Namespace Files](../05.concepts/02.namespace-files.md) as they will not be rendered, or allow you to run your code outside of Kestra. There are 2 ways to allow you to pass expressions to your code:
1. [Using Namespace Files with environment variables](#using-namespace-files-with-environment-variables)
2. [Using Namespace Files with CLI arguments](#using-namespace-files-with-cli-arguments)
In either case, we need to add our code as a [Namespace File](../05.concepts/02.namespace-files.md). To do this, you can use the [Editor](../08.ui/01.flows.md#editor) or import it directly.
![namespace_file](/docs/best-practices/namespace_file.png)
## Using Namespace Files with Environment Variables
You can pass inputs as environment variables using an expression.
This example uses the input `uri` and passes it to the task `code` as an environment variable so the Python code can access it

```yaml
id: pebble_templating_env_vars
namespace: company.team
inputs:
- id: uri
type: URI
defaults: https://www.google.com/
tasks:
- id: code
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python main.py "{{ inputs.uri }}"
env:
URI: "{{ inputs.uri }}"
```

Inside of the Python code, we use `os.environ` to fetch the environment variable:

```python
import requests
import os
# Perform the GET request
response = requests.get(os.environ['URI'])
# Check if the request was successful
if response.status_code == 200:
# Print the content of the page
print(response.text)
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
```

Using this method, it allows us to keep our code in a separate file, while also preventing our code from getting much longer than the inline example.

## Using Namespace Files with CLI arguments

We can pass arguments directly to our code at execution. In Python, we can use `argparse` to achieve this.

Before we can execute our code, we will need to modify it to receive our argument correctly using the following:

```python
import argparse
import requests
# Setup command line argument parsing
parser = argparse.ArgumentParser(description="Fetch the content of a given URL")
parser.add_argument("url", type=str, help="The URL to fetch")
args = parser.parse_args()
# Perform the GET request
response = requests.get(args.url)
# Check if the request was successful
if response.status_code == 200:
# Print the content of the page
print(response.text)
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
```

You can pass the arguments to your code using an expression. The expression will be rendered, and the evaluated values will be passed to the script via `argparse`:

```yaml
id: pebble_templating_argparse
namespace: company.team
inputs:
- id: uri
type: URI
defaults: https://www.google.com/
tasks:
- id: hello
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python main.py "{{ inputs.uri }}"
```

Using this method, the code has grows longer due to handling of arguments using argparse. This can be clearly seen by comparing the script in the Editor to the inline script used in the first approach. However, this method is very helpful from reusability perspective where the same script can be used in multiple flows without any code duplication of the script.
File renamed without changes.

This file was deleted.

1 change: 1 addition & 0 deletions nuxt.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ export default defineNuxtConfig({
'/docs/developer-guide/error-handling': {redirect: '/docs/workflow-components/errors'},
'/docs/developer-guide/scripts/output-directory': {redirect: '/docs/developer-guide/scripts/input-output-files'},
'/docs/best-practice': {redirect: '/docs/best-practices'},
'/docs/best-practices/pebble-templating-with-namespace-files': {redirect: '/docs/best-practices/expressions-with-namespace-files'},
'/docs/workflow-components/trigger': {redirect: '/docs/workflow-components/triggers'},
'/docs/workflow-components/realtime-triggers': {redirect: '/docs/workflow-components/realtime-trigger'},
'/docs/workflow-components/triggers/conditions': {redirect: '/docs/workflow-components/triggers#conditions'},
Expand Down

0 comments on commit 21bed2f

Please sign in to comment.