Scalability/performance testing plan

[dejanb]

We started with basic scalability testing with manually ingesting pre-planned number of documents and then again manually verifying that UI is responsive. This is great effort. Going forward it would be great to take this practice to the next level and being able to automate this kind of testing and get exact results that we can compare with previous tests, versions and deployment options. This discussion is to collect ideas of what we wanna do, how and share currently known best practices.

[dejanb]

I think there are a few tasks that we can perform right now, like:

• document current procedure for data ingestion
• Automate data ingestion, preferably dockerized workload that can be run locally or deployed as a job in the cluster
• Start working on API/UI testing client that we can use for testing

Open questions:

• How to create client API? Do we already have what we need from other testing methods?
• What are we testing? Throughput? Response time? Resources?
• Do we need any kind of tool for this, like JMeter? Are there any better "cloud-native" tools available?

[helio-frota]

What are we testing? Throughput? Response time? Resources?

Are there any better "cloud-native" tools available?

yes

for api endpoints we can [probably] eat own dog food with hyperfoil
https://github.com/Hyperfoil

https://github.com/obs-nebula/hyperfoil-test

[helio-frota]

[gildub]

Regarding data set loading, from the volume testing experience, using cronjobs (follow the same approach as post-installation) is likely the most efficient and consistent way of loading data in RHTPA and it can be entirely automated. For each category of data set (i.e CSAF, SBOM, CEV, etc) the test environment must provide the data set to be available as a URL.

The other alternative which is a combination of a host (pod or remote machine) running the file walkers with data set sources (files or URL) are either less efficient or more complex to automate.

No matter the chosen solution we need to have a place where we maintain consistent CSAF, SBOM, CEV, Packages, data sets which will be used those automated tests.

[JimFuller-RedHat]

To add - there are a few other notable load testing tools in rust land:

• https://github.com/fcsonline/drill
• https://github.com/tag1consulting/goose

though it can sometimes be useful to use something else (other then rust) for load testing (ex. python locust).

[helio-frota]

We have UI and soon sample data for PM-mode #297
UI + sample data + hyperfoil (in-vm) sounds like a good thing to create some scenarios in case team wants to follow this path. (probably)

[helio-frota]

@ctron hi, any possibility to use 0.0.0.0:8080 instead of [::1]:8080 ?

[helio-frota]

🤔

any possibility to use 0.0.0.0:8080 instead of [::1]:8080 ?

maybe we can't do it ... hyperfoil worked with trustification because localhost:8080 wasn't handled by the same code as the project uses a different architecture.

And we hit spog first with hyperfoil tests that constructs a SocketAddr with ipv4 format

[helio-frota]

So https://github.com/franz1981 found the issue and this works with ipv6 as well

[hyperfoil]$ start-local
Starting controller in default directory (/tmp/hyperfoil)
Controller started, listening on 127.0.0.1:45285
Connecting to the controller...
Connected to 127.0.0.1:45285!
[hyperfoil@in-vm]$ upload ../hf.yml
Loaded benchmark foo-foil, uploading...
... done.
[hyperfoil@in-vm]$ run
Started run 0001
Run 0001, benchmark foo-foil
Agents: in-vm[STOPPED]
Started: 2024/05/16 13:44:20.591    Terminated: 2024/05/16 13:44:30.601
NAME  STATUS      STARTED       REMAINING  COMPLETED     TOTAL DURATION               DESCRIPTION
main  TERMINATED  13:44:20.592             13:44:30.600  10008 ms (exceeded by 8 ms)  50.00 users per second
[hyperfoil@in-vm]$

[franz1981]

The problem seems related to a system property we set in hyperfoil to forcibly use the ipv4 stack, likely to not break Infinispan clustering.
I have to double check with the data grid team if the problem is legit or I should find a way to fix it differently

[helio-frota]

@franz1981 we have a PR ( sort of draft ) #309 thanks for the support!

[helio-frota]

@dejanb Do we have any expectations regarding the number of acceptable sboms or requests?
thanks

[helio-frota]

1GiB sboms ?
20k requests ?

[helio-frota]

We have this currently:

[dejanb]

That's a great starting point. I think for starters we should not focus on the huge number of requests, but gather some numbers on response times even for small number of requests.

[helio-frota]

Agreed

[helio-frota]

@gildub ^