FIO just slow

[kookin]

Please acknowledge the following before creating a ticket

yes, although these was a previous issue like this but unresolved and closed

Description of the bug:

FIO very slow on enterprise hardware. Very slow on the initial layout and the subsequent performance results are radically different from an FIO installation on the exact same hardware of a previous installation.

The FIO params are: fio --name=blockchain_test --ioengine=libaio --iodepth=64 --rw=randrw --rwmixread=70 --bs=8k --direct=1 --size=5G --numjobs=8 --runtime=600 --group_reporting

We previously ran the exact same FIO parameters on the exact same hardware with a different installation of Proxmox/Ceph (with even less disks actually) and FIO was much faster in the layout and the overall storage performance was about 3 times better. How is this possible. Ceph itself is showing 100% healthy.

Environment: The environment is a 3 node Proxmox Ceph PoC with a RDB pool backed by 10x 3TB NVMe's. Test VM is running Ubuntu 22.04.

fio version: fio-3.28

Reproduction steps
Replicated env above.

[kookin]

Upgraded to fio-3.38-19-gd1ebb and still the same issue

[sitsofe]

@kookin can you simplify your job further by removing as much of it as possible. For example do you still see the problem with --rw=write? Can you remove --numjobs=8 and up the iodepth and still see the problem etc?

[sitsofe]

@kookin In the other ticket you mention that layout is slow... Is your same job slow when you use --fallocate=none?

[kookin]

@sitsofe - with allocate=none

I timed it and it took 6m51s for the 8 files to be created ->

fio --name=blockchain_test --ioengine=libaio --iodepth=64 --rw=randrw --rwmixread=70 --bs=8k --direct=1 --size=5G --numjobs=8 --runtime=600 --group_reporting --fallocate=none
blockchain_test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
fio-3.38-19-gd1ebb
Starting 8 processes
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)
blockchain_test: Laying out IO file (1 file / 5120MiB)

[sitsofe]

@kookin:

I timed it and it took 6m51s for the 8 files to be created ->

We will also need to know how long it is supposed to take to know how much worse it is.

Also did you see this request in a prior comment?

@kookin can you simplify your job further by removing as much of it as possible. For example do you still see the problem with --rw=write? Can you remove --numjobs=8 and up the iodepth and still see the problem etc?

Please try and remove/reduce as many individual pieces as possible to help narrow down where the problem lies.

[kookin]

@sitsofe - i am suspecting it is my ceph configuration as I've tested it on VMware and it acts normally. I'll keep you informed if anything changes. Thanks for now.