Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Update math in PQ size example #16411

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[DOC] Update math in PQ size example #16411

wants to merge 3 commits into from

Conversation

a03nikki
Copy link
Contributor

@a03nikki a03nikki commented Aug 29, 2024

Corrected PQ multiplication factor values for JSON documents and the example.

PREVIEW: https://logstash_bk_16411.docs-preview.app.elstc.co/guide/en/logstash/master/persistent-queues.html

Release notes

[rn:skip]

What does this PR do?

Corrects the math in a number of places in the documentation at https://www.elastic.co/guide/en/logstash/8.15/persistent-queues.html.

Why is it important/What is the impact to the user?

The numbers listed on the page did not align with the values when plugged into a calculator. This could lead to confusion when trying to replicate the math.

 | Plaintext size (bytes) | Serialized {ls} event size (bytes) | Overhead (bytes) | Overhead (%) | Multiplication Factor
-| 11 | 213 | 202 | 1836% | 19.4
+| 11 | 213 | 202 | 1836% | 18.36

202/11 = 18.3636363636

| JSON document size (bytes) | Serialized {ls} event size (bytes) | Overhead (bytes) | Overhead (%) | Multiplication Factor
-| 6751 | 7388 | 637 | 9% | 1.9
-| 58901 | 59693 | 792 | 1% | 1.1
+| 6751 | 7388 | 637 | 9% | 1.09
+| 58901 | 59693 | 792 | 1% | 1.01

637/6751 = 0.09435639165
792/58901 = 0.01344629123

*Example*

-Let's consider a {ls} instance that receives 1000 EPS and each event is 1KB,
-or 3.5GB every hour. In order to tolerate a downstream component being unavailable
+Let's consider a {ls} instance that receives 1000 EPS and each event is 1KB (1024 bytes),
+which is 1000 KB per second or 3.5GB (1*1000*3600 KB) every hour.
+In order to tolerate a downstream component being unavailable
 for 12h without {ls} exerting back-pressure upstream, the persistent queue's
-`max_bytes` would have to be set to 3.6*12*1.10 = 47.25GB, or about 50GB.
+`max_bytes` would have to be set to 3.5*12*1.10 = 46.2GB, or about 50GB.

1000 events per second * 1KB = 1000 KB per second
1000 KB *60 seconds per minute * 60 minutes per hour = 3,600,000 KB pr hour
3,600,000 / 1024 / 1024 = 3.4332275391 GB per hour

Round up as we are dealing with disk space to 3.5GB per hour.

3.5GB per hour * 12 hours * 1.10 default multiplication factor = 46.2 GB

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files (and/or docker env variables)
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • Double check math and grammar

How to test this PR locally

Redo the math yourself.

Related issues

N/A

Use cases

User trying to determine how much disk space is needed for PQ and doing the calculations based on the documentation.

Screenshots

image

Logs

N/A

Copy link
Contributor

@karenzone karenzone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builds cleanly and LGTM.

@karenzone
Copy link
Contributor

@robbavey, keep me honest here please?

@robbavey
Copy link
Member

@a03nikki @karenzone Do we need to be concerned about KiB vs KB and GiB vs GB here in the calculations and numbers?

I prefer to use the right terms where appropriate, but Logstash itself only takes in gb, etc, and translates them to GiB so presenting this is interesting...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants