Skip to content

Commit

Permalink
feat(site): use nuxt image on doc & blogs (#444)
Browse files Browse the repository at this point in the history
  • Loading branch information
tchiotludo authored Sep 11, 2023
1 parent bd9d98d commit 5ac190d
Show file tree
Hide file tree
Showing 15 changed files with 118 additions and 384 deletions.
2 changes: 1 addition & 1 deletion assets/styles/docs.scss
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ $bd-gutter-x: 3rem;
}
}

img {
img, video {
max-width: 100%;
}

Expand Down
2 changes: 1 addition & 1 deletion components/blogs/BlogCard.vue
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<template>
<div class="mb-5 mt-1" role="button">
<NuxtLink class="text-dark" :href="blog._path">
<img :alt="blog.image" :src="blog.image" class="w-100 rounded-3">
<NuxtImg width="512" loading="lazy" format="webp" quality="80" :alt="blog.image" :src="blog.image" class="w-100 rounded-3" />
<div class="mt-3">
<span class="small-text category">{{ blog.category }}</span>
<h6 class="mt-2">{{ blog.title }}</h6>
Expand Down
2 changes: 1 addition & 1 deletion components/blogs/BlogDetails.vue
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
</span>
</p>

<img class="rounded-circle mt-5" :src="'/landing/company/teams/' + blog.author.image + '-sm.png'" :alt="blog.author.name"/><br />
<NuxtImg loading="lazy" format="webp" width="68" class="rounded-circle mt-5" :src="'/landing/company/teams/' + blog.author.image + '-sm.png'" :alt="blog.author.name"/><br />
<strong>{{ blog.author.name }}</strong>

</div>
Expand Down
2 changes: 1 addition & 1 deletion components/blogs/HighlightBlogCard.vue
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<NuxtLink class="text-dark" :href="blog._path">
<div class="row g-0">
<div class="col-md-6">
<img class="col-md-12 rounded-3 img-fluid" :alt="blog.image" :src="blog.image">
<NuxtImg width="512" loading="lazy" format="webp" quality="80" class="col-md-12 rounded-3 img-fluid" :alt="blog.image" :src="blog.image" />
</div>
<div class="col-md-6 description">
<span class="small-text category">{{ blog.category }}</span>
Expand Down
4 changes: 3 additions & 1 deletion components/content/ProseImg.vue
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<template>
<img :src="refinedSrc" :alt="alt" :width="width" :height="height" :class="classWithZoom">
<span class="text-center d-block">
<NuxtImg :src="refinedSrc" :alt="alt" :width="width" :height="height" :class="classWithZoom" loading="lazy" format="webp" quality="80" />
</span>
</template>

<script setup lang="ts">
Expand Down
2 changes: 1 addition & 1 deletion components/layout/Blogs.vue
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<div class="col-md-4 mb-4">
<div class="card" data-aos="fade-right">
<NuxtLink class="text-dark" :href="blog._path">
<img :src="blog.image" class="card-img-top rounded-3" :alt="blog.image">
<NuxtImg loading="lazy" format="webp" quality="80" :src="blog.image" class="card-img-top rounded-3" :alt="blog.image" />
<div class="card-body">
<p class="type mt-3 mb-2">{{ blog.category }}</p>
<h4 class="card-title">{{ blog.title }}</h4>
Expand Down
4 changes: 1 addition & 3 deletions content/blogs/2022-02-01-kestra-opensource.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,11 @@ As you can see, Kestra will handle **all your pipelines** !
## The History of Kestra!
Kestra started in 2019 with this [initial commit](https://github.com/kestra-io/kestra/commit/d57e30c0c0d450590a1eaac5df0e82e1ea94e562). At this time, Kestra was at the proof-of-concept stage.

<img src="/blogs/2022-02-01-kestra-opensource/initial-commit.jpg" class="rounded img-thumbnail float-left mr-4 mb-4" alt="Initial commit" style="max-width: 450px">
![Initial commit](/blogs/2022-02-01-kestra-opensource/initial-commit.jpg)


To provide a bit of a background: I was working for Leroy Merlin as a consultant. We needed to build a new cloud-based data platform from scratch (destination: mostly Google Cloud Platform). We tried a [lot of things](../blogs/2022-02-22-leroy-merlin-usage-kestra) and failed with some of our attempts. The **biggest setback was the orchestration** software that we tried to deliver with Apache Airflow: a lot of instability (tasks that failed simply due to the Airflow scheduler), performance issues (unable to handle a light workload), and a lack of features (scaling, data processing). After many tests (Google Composer, Open source Airflow on Kubernetes), the decision was final: **Airflow was rejected by Leroy Merlin**.

<div class="clearfix" />

I did some research on the orchestrator ecosystem; most are **proprietary and license based** (far from my mindset), some are open source (at this time, only Apache Airflow seemed to be active — and it was rejected). I was really surprised by this discovery and faced this challenge from a co-worker:
> If you think Airflow is bad, do better!
Expand Down
11 changes: 4 additions & 7 deletions content/blogs/2022-02-22-leroy-merlin-usage-kestra.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,7 @@ Thanks to Kestra, they moved **to full autonomy** and **reduced the time to mark
:::
### Go Production
<img src="/blogs/2022-02-22-leroy-merlin-usage-kestra/architecture.jpg" class="rounded img-thumbnail float-left mr-4 mb-4" alt="Kestra user interface">
<div class="clearfix" />
![Kestra user interface](/blogs/2022-02-22-leroy-merlin-usage-kestra/architecture.jpg)
After a few months, **Kestra was confirmed** and was deployed to a production environment in May of 2020. They also decided to move to the Enterprise Edition of Kestra in August 2020 as they needed to have fine-grained security, role-based access control and single sign-on.
Expand All @@ -158,11 +157,9 @@ Here are some numbers that show the current usage for January 2022 ( for product
- **3,000,000+ tasks** every month
- **Equivalent of 2,300 days of task processing time** every month (yep, that’s the equivalent of seventy-five days of task processing every single day)
<img src="/blogs/2022-02-22-leroy-merlin-usage-kestra/executions.svg" class="rounded img-thumbnail float-left mr-4 mb-4" alt="Initial commit" style="max-width: 370px">
<img src="/blogs/2022-02-22-leroy-merlin-usage-kestra/taskruns.svg" class="rounded img-thumbnail float-left mr-4 mb-4" alt="Initial commit" style="max-width: 370px">
<img src="/blogs/2022-02-22-leroy-merlin-usage-kestra/execution-duration.svg" class="rounded img-thumbnail float-left mr-4 mb-4" alt="Initial commit" style="max-width: 370px">
<div class="clearfix" />
![Kestra user interface](/blogs/2022-02-22-leroy-merlin-usage-kestra/executions.svg)
![Kestra user interface](/blogs/2022-02-22-leroy-merlin-usage-kestra/taskruns.svg)
![Kestra user interface](/blogs/2022-02-22-leroy-merlin-usage-kestra/execution-duration.svg)
### Some Testimonials
Leroy Merlin has strongly supported the development of Kestra. As with any software, its young age could be a source of frustration for users. They needed to handle missing features, report some bugs, suffer some time loss from instability. But what was astounding was the realization that the **pain was far less than the gain**, and we obtained many good reports on software internal notation tools. It proved to be a solution that users really love and uphold.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,56 +22,42 @@ We have done a lot of work to **reduce CPU usage and latency**.
As you can see, the same workload previously used 3 executors (we used autoscaling), with a total usage of 2.5 cores for over 6 hours.
After, only **0.5 cores** were necessary through most of the runtime, with a peak of only 1.5 cores for 1 hour, the whole workload was managed by only 2 executors (minimum autoscaling).

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/executor-cpu-before.png" alt="before" />
<figcaption class="figure-caption text-center">Before</figcaption>
</figure>

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/executor-cpu-after.png" alt="after" />
<figcaption class="figure-caption text-center">After</figcaption>
</figure>
![Before](/blogs/2022-03-14-kestra-0.4.0/executor-cpu-before.png)
<p class="figure-caption text-center">Before</p>

![After](/blogs/2022-03-14-kestra-0.4.0/executor-cpu-after.png)
<p class="figure-caption text-center">After</p>


On the Kafka side, we also see **improved latency**.
The system had a lot of [flow triggers](../docs/05.developer-guide/08.triggers/02.flow.md), which needed to be analyzed for each completed execution. When you have a high volume of execution, a lot of messages are sent by Kafka and if your consumer is too slow, the queue fills up and increases latency.
Previously, we had a large message lag that led to a late start of flow executions (a few minutes). The optimization allowed us to keep the start of the flow to within a few seconds.

![Before](/blogs/2022-03-14-kestra-0.4.0/lag-before.png)
<p class="figure-caption text-center">Before</p>

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/lag-before.png" alt="before" />
<figcaption class="figure-caption text-center">Before</figcaption>
</figure>

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/lag-after.png" alt="after" />
<figcaption class="figure-caption text-center">After</figcaption>
</figure>
![After](/blogs/2022-03-14-kestra-0.4.0/lag-after.png)
<p class="figure-caption text-center">After</p>


The last graph shows the delay between the creation of the task and the task launched by the worker. Since we have optimized Kafka processing globally, we avoid queuing messages on Kafka and reduce the delay between task creation and worker processing.

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/queue-before.png" alt="before" />
<figcaption class="figure-caption text-center">Before</figcaption>
</figure>
![Before](/blogs/2022-03-14-kestra-0.4.0/queue-before.png)
<p class="figure-caption text-center">Before</p>

![After](/blogs/2022-03-14-kestra-0.4.0/queue-after.png)
<p class="figure-caption text-center">After</p>

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/queue-after.png" alt="after" />
<figcaption class="figure-caption text-center">After</figcaption>
</figure>

All of these improvements also provide a **significant reduction in total execution time**. As all messages are consumed quickly, the time between each task is reduced and the total runtime is reduced.

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/executor-duration-before.png" alt="before" />
<figcaption class="figure-caption text-center">Before</figcaption>
</figure>
![Before](/blogs/2022-03-14-kestra-0.4.0/executor-duration-before.png)
<p class="figure-caption text-center">Before</p>

<figure class="figure" style="width: 48%">
<img class="figure-img img-fluid rounded" src="/blogs/2022-03-14-kestra-0.4.0/executor-duration-after.png" alt="after" />
<figcaption class="figure-caption text-center">After</figcaption>
</figure>
![After](/blogs/2022-03-14-kestra-0.4.0/executor-duration-after.png)
<p class="figure-caption text-center">After</p>

Keep in mind that Leroy Merlin's workflow is unbalanced, and all executions take place at the same time, more than 3000 executions and more than 35,000 tasks in a short period of time, making up 50% of the entire day's workload. We have to provide the same requirement for night processing, even if we are all asleep.
We have more optimizations pending, but this provided a big improvement that will allow us to handle more complex clusters with lots of streams and concurrent executions. A full blog post is coming soon to expose what we discovered while scaling a [Kafka Streams](https://kafka.apache.org/documentation/streams/) application.
Expand Down Expand Up @@ -236,10 +222,8 @@ For the Enterprise Edition, we delivered the most requested features: the abilit

We also added a confirmation when deleting a flow that has dependencies, which warns the user that deleting it might break the whole production plan.

<p class="text-center">
<img src="/blogs/2022-03-14-kestra-0.4.0/deps.png" class="rounded img-thumbnail" alt="Kestra user interface">
</p>
<div class="clearfix" />
![Kestra user interface](/blogs/2022-03-14-kestra-0.4.0/deps.png)


This is a valuable feature that provides a complete view of your entire data pipeline across teams. With many teams consuming data from other teams, no one can be sure whether if this flow is changed, another flow will be impacted. Impact analysis is greatly simplified with this powerful user interface.

Expand Down
11 changes: 4 additions & 7 deletions content/blogs/2022-04-05-debezium-without-kafka-connect.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,7 @@ The importance of data pipeline management and orchestration in today's data lan

### Breaking Down Debezium's Architecture ###

<p align="center">
<img src="/blogs/2022-04-05-debezium-without-kafka-connect/debezium-architecture.png" alt="debezium architecture">
</p>
![debezium architecture](/blogs/2022-03-14-kestra-0.4.0/executor-cpu-before.png)

Debezium is an open-source platform developed by Red Hat that stands out in the [Change Data Capture (CDC)](https://en.wikipedia.org/wiki/Change_data_capture) field. It tracks and records row-level changes in your databases, enabling connected applications to respond in real time. Debezium's strength lies in ensuring exactly-once delivery of all changes in the precise order they were made. Furthermore, Debezium's ability to monitor changes in data from multiple sources and various types of databases makes it a highly adaptable tool in data pipeline management.

Expand All @@ -39,9 +37,8 @@ Although Debezium shines as an efficient and performance-oriented solution in re
For example, from [Amazon MSK connect](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-connectors.html) documentation:
> Each MCU represents 1 vCPU of compute and 4 GiB of memory.
<p align="center">
<img src="/blogs/2022-04-05-debezium-without-kafka-connect/msk.png" alt="MSK connect pricing on aws">
</p>

![MSK connect pricing on aws](/blogs/2022-03-14-kestra-0.4.0/msk.png)

This lead to $160 for 1 source and 1 destination per month.

Expand All @@ -58,7 +55,7 @@ Kestra is designed to streamline the construction, execution, scheduling, and mo
It offers batch or micro-batch processing capabilities, thereby scaling performance based on specific requirements from periodic updates to near-real-time scenarios. This adaptability optimizes resource usage, particularly when real-time performance isn't required.
Kestra's Role in Resource Efficiency

When combined with Debezium, Kestra can offer a solution that uses resources precisely based on specific use cases. For example, a dashboard or KPI might only need to be **refreshed once a day**, or every few hours, for example. Whether it is cloud services or on-premises, resources cost money, and the challenge is to make the **most efficient use of resources** based on your requirements. Bandwidth, compute resources, or services based on throughput (such as BigQuery) are all expensive commodities, especially if they are always running.
When combined with Debezium, Kestra can offer a solution that uses resources precisely based on specific use cases. For example, a dashboard or KPI might only need to be **refreshed once a day**, or every few hours, for example. Whether it is cloud services or on-premises, resources cost money, and the challenge is to make the **most efficient use of resources** based on your requirements. Bandwidth, compute resources, or services based on throughput (such as BigQuery) are all expensive commodities, especially if they are always running.

If real-time performance is not necessary, then why waste money on resources you do not need? This approach avoids the application of resource-intensive streaming resources to every process. The result is an efficient solution that maximizes resource usage and saves money, while still ensuring comprehensive capture of all row-level changes.

Expand Down
9 changes: 3 additions & 6 deletions content/blogs/2022-06-21-light-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,8 @@ We have interview many people on our slack community, twitter, ... also that **c

For now, we continue to think a high-availability solution is a **must-have**, a solution that can ensure you that if your needs increase, you will have a solution to scale without having to do without the features. So we decide to create a new version of Kestra (in Beta for now) that will work for a Medium-sized environment.

<p style="text-align:center">
<img src="/docs/architecture/architecture-sql.svg" class="rounded img-thumbnail mt-4 mb-4" alt="Kestra Architecture">
</p>
![Kestra Architecture](/docs/architecture/architecture-sql.svg)


The solution is to **remove the dependencies of Kafka & ElasticSearch** and to allow to replace them with a **simple MySQL of Postgres** database for the both of them.

Expand All @@ -32,9 +31,7 @@ The new version is possible since Kestra thought since its inception as **plugga

## Tradeoff using a Database

<p style="text-align:center">
<img src="/blogs/2022-06-21-light-architecture/warning.jpg" class="rounded img-thumbnail mt-4 mb-4" alt="Tradeoff using a Database">
</p>
![Tradeoff using a Database](/blogs/2022-06-21-light-architecture/warning.jpg)

We have worked harder to be able to have the **more fluent change** using a database, but we have made some tradeoffs for now (maybe you can go deeper depending on your feedback for a certain point).

Expand Down
3 changes: 2 additions & 1 deletion nuxt.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ export default defineNuxtConfig({
modules: [
'@nuxt/devtools',
'@nuxt/content',
'@nuxt/image',
'nuxt-gtag',
'nuxt-simple-sitemap',
'nuxt-multi-cache'
Expand All @@ -27,7 +28,7 @@ export default defineNuxtConfig({
{property: 'og:image', content:'/og-image.png'}
],
script: [
{src: '//js-eu1.hsforms.net/forms/embed/v2.js'}
{src: 'https://js-eu1.hsforms.net/forms/embed/v2.js'}
],
}
},
Expand Down
Loading

1 comment on commit 5ac190d

@vercel
Copy link

@vercel vercel bot commented on 5ac190d Sep 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Successfully deployed to the following URLs:

kestra-io – ./

kestra-io-git-main-kestra.vercel.app
kestra-io.vercel.app
kestra-io-kestra.vercel.app

Please sign in to comment.