Merge pull request #2279 from aiven/harshini-aiven-dragonfly

WIP- Aiven-for-Dragonfly-service

.github/vale/dicts/aiven.dic

@@ -69,6 +69,7 @@ Docker
 dockerized
 downsampled
 downsampling
+Dragonfly
 DSBulk
 DZone
 Elasticsearch
@@ -208,6 +209,7 @@ rebalance
 rebalances
 rebalancing
 Redis
+Redis OSS
 redis_timeout
 Redli
 refcard
@@ -228,6 +230,7 @@ Savepoints
 savepoints
 schemaless
 scriptable
+SDKs?
 Security
 serde
 serializer

.github/vale/styles/Aiven/first_Dragonfly_is_registered.yml

@@ -0,0 +1,8 @@
+extends: conditional
+message: "At least one '%s' must be marked as ®"
+level: error
+scope: text
+ignorecase: false
+
+first: '\b(Dragonfly)(?!®)'
+second: '(Dragonfly)(?:®)'

.github/workflows/advanced-params-dragonfly.yaml

@@ -0,0 +1,39 @@
+name: Dragonfly - Create PR to Update Advanced parameters
+
+on:
+  schedule:
+    - cron: "0 6 * * 2"
+  workflow_dispatch:
+
+jobs:
+  advanced_params_dragonfly:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout the repo
+        uses: actions/checkout@v2
+
+      - name: Set up Python 3.8
+        uses: actions/setup-python@v2
+        with:
+          python-version: "3.8"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Dragonfly - Update Advanced parameters
+        run: make service-type-config-dragonfly
+
+      - name: Create Pull Request
+        uses: peter-evans/create-pull-request@v4
+        with:
+          commit-message: Dragonfly - Update Advanced parameters
+          committer: GitHub <[email protected]>
+          author: GitHub <[email protected]>
+          title: Dragonfly - Update Advanced parameters
+          body: Dragonfly - update advanced parameters file
+          base: main
+          branch: dragonfly-update-advanced-params
+          labels: dragonfly, automated-pr
+          delete-branch: true

_toc.yml

@@ -878,8 +878,46 @@ entries:
             title: Formats for ClickHouse-Kafka data exchange
           - file: docs/products/clickhouse/reference/advanced-params
             title: Advanced parameters
+
+# ---------Dragonfly --------------
+
+  - file: docs/products/dragonfly
+    title: Dragonfly
+    entries:
+      - file: docs/products/dragonfly/concepts/overview
+        title: Overview
+      - file: docs/products/dragonfly/get-started
+        title: Quickstart
+      - file: docs/products/dragonfly/concepts
+        title: Concepts
+        entries:
+          - file:  docs/products/dragonfly/concepts/ha-dragonfly
+      - file: docs/products/dragonfly/howto
+        title: HowTo
+        entries:
+        - file: docs/products/dragonfly/howto/list-code-samples
+          title: Connect to service
+          entries:
+            - file: docs/products/dragonfly/howto/connect-redis-cli
+            - file: docs/products/dragonfly/howto/connect-go
+            - file: docs/products/dragonfly/howto/connect-node
+            - file: docs/products/dragonfly/howto/connect-python
+
+        - file: docs/products/dragonfly/howto/list-migrate-data
+          title: Data migration
+          entries:
+          - file: docs/products/dragonfly/howto/migrate-aiven-redis-df-console
+            title: Migrate Aiven for Redis
+          - file: docs/products/dragonfly/howto/migrate-ext-redis-df-console
+            title: Migrate external Redis
+
+
+
+
+
 
   # -------- GRAFANA --------
+
   - file: docs/products/grafana
     title: Grafana
     entries:

conf.py

@@ -312,6 +312,15 @@
    :width: 24px
    :class: no-scaled-link
 
+.. |icon-dragonfly| image:: /images/icon-dragonfly.svg
+   :width: 24px
+   :class: no-scaled-link   
+
+.. |tick| image:: /images/icon-tick.png
+   :width: 24px
+   :class: no-scaled-link
+
+
 .. |beta| replace:: :bdg-secondary:`beta`
 
 .. |preview| replace:: :bdg-secondary:`preview`

docs/products/dragonfly.rst

@@ -0,0 +1,35 @@
+Aiven for Dragonfly®
+====================
+
+Aiven for Dragonfly is an advanced, **high-scale, and Redis®* compatible in-memory database service** that can be easily deployed in your preferred cloud environment. It provides lightning-fast data storage and retrieval capabilities, making it ideal for businesses that handle large-scale data operations.
+
+.. important:: 
+    Aiven for Dragonfly® is currently a :doc:`limited availability </docs/platform/concepts/beta_services>` service. If you are interested in exploring this offering, reach out to our sales team at [email protected] for more information and access.
+
+-------------------------
+
+.. grid:: 1 2 2 2
+
+    .. grid-item-card:: :doc:`Quickstart </docs/products/dragonfly/get-started>`
+        :shadow: md
+        :margin: 2 2 0 0
+
+        Set up your Aiven for Dragonfly service and learn how to connect to it.
+
+    .. grid-item-card:: :doc:`Overview </docs/products/dragonfly/concepts/overview>`
+        :shadow: md
+        :margin: 2 2 0 0
+
+        Explore the unique features and benefits of Aiven for Dragonfly.
+
+    .. grid-item-card:: :doc:`Concepts </docs/products/dragonfly/concepts>`
+        :shadow: md
+        :margin: 2 2 0 0
+
+        Understand the core concepts behind Aiven for Dragonfly's technology.
+
+    .. grid-item-card:: :doc:`How-Tos </docs/products/dragonfly/howto>`
+        :shadow: md
+        :margin: 2 2 0 0
+
+        Discover step-by-step instructions on how to use Aiven for Dragonfly and tips for various use cases.

docs/products/dragonfly/concepts.rst

@@ -0,0 +1,6 @@
+Concepts
+========
+
+Learn more about some of the key concepts for working with Aiven for Dragonfly®:
+
+.. tableofcontents::

docs/products/dragonfly/concepts/ha-dragonfly.rst

@@ -0,0 +1,73 @@
+High availability in Aiven for Dragonfly®
+==========================================
+
+Aiven for Dragonfly® offers different plans with varying levels of high availability. The available features depend on the selected plan. Refer to the table below for a summary of these plans:
+
+.. important:: 
+    Aiven for Dragonfly® is currently a :doc:`limited availability </docs/platform/concepts/beta_services>` service. If you are interested in exploring this offering, reach out to our sales team at [email protected] for more information and access.
+
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 20 30 30 10
+
+   * - Plan
+     - Node configuration
+     - High availability & failover features
+     - Backup features
+     - Backup history
+   * - **Startup**
+     - Single-node
+     - Limited availability. No automatic failover.
+     - During limited availability, only one latest snapshot stored.
+     - 1 day
+   * - **Business**
+     - Two-node (primary + standby)
+     - High availability with automatic failover to a standby node if the primary fails.
+     - During limited availability, only one latest snapshot stored.
+     - 3 days
+   * - **Premium**
+     - Three-node (primary + standby + standby)
+     - Enhanced high availability with automatic failover among multiple standby nodes if the primary fails.
+     - During limited availability, only one latest snapshot stored.
+     - 13 days
+   * - **Custom**
+     - Custom configurations
+     - Custom high availability and failover features based on user requirements.
+     - During limited availability, only one latest snapshot stored.
+     - Custom based on user requirements
+
+
+
+Failure handling
+----------------
+
+- **Minor failures**: Aiven automatically handles minor failures, such as service process crashes or temporary loss of network access, without any significant changes to the service deployment. In all plans, the service automatically restores regular operation by restarting crashed processes or restoring network access when available.
+- **Severe failures**: In case of severe hardware or software problems, such as losing an entire node, more drastic recovery measures are required. Aiven's monitoring infrastructure automatically detects a failing node when it reports problems with its self-diagnostics or stops communicating altogether. The monitoring infrastructure then schedules the creation of a new replacement node.
+
+.. note::
+
+   In case of database failover, your service's **Service URI** remains the same—only the IP address changes to point to the new primary node.
+
+High availability for business, premium, and custom plans
+------------------------------------------------------------
+
+If a standby Dragonfly node fails, the primary node continues running. The system prepares the replacement standby node and synchronizes it with the primary for normal operations to resume.
+
+In case the primary Dragonfly node fails, the standby node is evaluated for promotion to the new primary based on data from the Aiven monitoring infrastructure. Once promoted, this node starts serving clients, and a new node is scheduled to become the standby. However, during this transition, there may be a brief service interruption.
+
+If the primary and standby nodes fail simultaneously, new nodes will be created automatically to replace them. However, this may lead to data loss as the primary node is restored from the latest backup. As a result, any database writes made since the last backup could be lost.
+
+.. note::
+
+   The duration for replacing a failed node depends mainly on the **cloud region** and the **amount of data** to be restored. For Business, Premium, and Custom plans with multiple nodes, this process is automatic and requires no administrator intervention, but service interruptions may occur during the recreation of nodes.
+
+
+Single-node startup service plans
+----------------------------------------------
+
+Losing the only node in the service triggers an automatic process of creating a new replacement node. The new node then restores its state from the latest available backup and resumes serving customers.
+
+Since there was just a single node providing the service, the service will be unavailable for the duration of the restore operation. All the write operations made since the last backup are lost.
+
+

docs/products/dragonfly/concepts/overview.rst

@@ -0,0 +1,44 @@
+Aiven for Dragonfly® overview
+================================
+
+Aiven for Dragonfly is an advanced, **high-scale, and Redis®* compatible in-memory database service** that can be easily deployed in your preferred cloud environment. It provides lightning-fast data storage and retrieval capabilities, making it ideal for businesses that handle large-scale data operations.
+
+.. important:: 
+    Aiven for Dragonfly® is currently a :doc:`limited availability </docs/platform/concepts/beta_services>` service. If you are interested in exploring this offering, reach out to our sales team at [email protected] for more information and access.
+
+Why Dragonfly?
+--------------
+
+Dragonfly is designed to overcome the limitations of Redis Open Source Software (Redis OSS), especially under high-load conditions. While renowned for its speed and adaptability as an in-memory data repository, Redis encounters limitations when handling large-scale data management and achieving high throughput. Dragonfly addresses these challenges by enabling vertical scaling, optimizing hardware resources, and supporting larger memory footprints.
+
+With Aiven for Dragonfly, businesses can handle workloads exceeding 1TB with more than 10 times the throughput performance of Redis OSS, along with reduced latency. This makes it an ideal solution for enterprises with growing data needs.
+
+
+Features and benefits
+---------------------
+
+Aiven for Dragonfly offers numerous features and benefits:
+
+* **Redis compatibility at scale:** It is a seamless drop-in replacement for Redis, capable of handling extensive workloads with enhanced performance.
+
+* **Optimized for large-scale operations:** Dragonfly is specifically built to address the scalability and resource utilization limitations of Redis Open Source Software (Redis OSS).
+
+* **Advanced performance:** Dragonfly's unique threading model and shared-nothing architecture allow it to scale vertically, enhancing its performance efficiency, especially in environments with heavy data loads.
+
+
+* **Efficient backup and memory management:** Improved snapshot capabilities lead to more efficient memory usage during backups.
+
+* **High availability and replication:** It includes active-passive replication and persistence capabilities, ensuring data reliability and consistency.
+
+* **Ease of integration:** Dragonfly integrates smoothly with existing systems, requiring no code changes, simplifying the adoption process.
+
+Use cases
+---------------------
+
+* **Data-intensive enterprises:** Ideal for businesses that necessitate robust, high-performance in-memory data storage and processing capabilities.
+
+* **Scaling and performance needs:** Perfectly suited for situations where the need for greater scalability and higher throughput goes beyond what Redis OSS can handle.
+
+Related reading
+----------------
+* For detailed information about Dragonfly, refer to `Dragonfly documentation <https://www.dragonflydb.io/docs>`_.

docs/products/dragonfly/get-started.rst

@@ -0,0 +1,59 @@
+Get started with Aiven for Dragonfly®
+=======================================
+
+The first step in using Aiven for Dragonfly is to create a service. You can do so either using the `Aiven Console <https://console.aiven.io/>`_ or the `Aiven CLI <https://github.com/aiven/aiven-client>`_.
+
+.. important:: 
+    Aiven for Dragonfly® is currently a :doc:`limited availability </docs/platform/concepts/beta_services>` service. If you are interested in exploring this offering, reach out to our sales team at [email protected] for more information and access.
+
+Create a service using the Aiven Console
+----------------------------------------------
+1. Log in to the `Aiven Console <https://console.aiven.io/>`_.
+
+2. Follow :doc:`these instructions </docs/platform/howto/create_new_service>` to create a new Dragonfly service.
+
+Once the service is ready, the status changes to *Running*. Depending on your selected cloud provider and region, this generally takes a couple of minutes.
+
+
+Create a service using the Aiven CLI
+------------------------------------------------
+
+`Aiven CLI <https://github.com/aiven/aiven-client>`_ provides a simple and efficient way to create an Aiven for Dragonfly® service. If you prefer launching a new service from the CLI, follow these steps:
+
+1. Determine the service plan, cloud provider, and region you want to use for your Dragonfly service.
+2. Run the following command to create Dragonfly service named dragonfly-demo:
+
+.. code:: 
+
+    avn service create dragonfly-demo   \
+        --service-type dragonfly                 \
+        --cloud google-europe-north1             \
+        --plan startup-4                         \
+        --project dev-sandbox
+
+.. note::
+    There are additional options available to you, which you can view by running the following commands:
+
+    * For a full list of default flags: ``avn service create -h``
+    * For type-specific options: ``avn service types -v`` 
+
+
+Connect to Aiven for Dragonfly
+-------------------------------
+
+Learn how to connect to Aiven for Dragonfly using different programming languages:
+
+* :doc:`redis-cli </docs/products/dragonfly/howto/connect-redis-cli>`
+* :doc:`Go </docs/products/dragonfly/howto/connect-go>`
+* :doc:`Node </docs/products/dragonfly/howto/connect-node>`
+* :doc:`Python </docs/products/dragonfly/howto/connect-python>`
+
+
+Explore other resources 
+------------------------
+
+* Learn about how Aiven for Dragonfly supports :doc:`high availability </docs/products/dragonfly/concepts/ha-dragonfly>`.
+* Migrate data from :doc:`Aiven for Redis®* to Aiven for Dragonfly </docs/products/dragonfly/howto/migrate-aiven-redis-df-console>`.
+* Migrate data from :doc:`external Dragonfly to Aiven for Dragonfly </docs/products/dragonfly/howto/migrate-ext-redis-df-console>`.
+
+

docs/products/dragonfly/howto.rst

@@ -0,0 +1,7 @@
+HowTo
+=====
+
+Check out the common tasks for working with Aiven for Dragonfly®:
+
+.. tableofcontents::
+