Best Practices for Using Aerospike C Client with a Separate UV Loop

[luyi404]

Hello Aerospike community,

We have been using Aerospike in conjunction with our server's main uv_loop for several years without issues. However, we are now exploring the option of assigning Aerospike its own thread and loop to reduce the load on our main loop. While attempting this, we encountered several challenges.

Initially, we tried using as_event_create_loops, and it seemed to work fine locally. However, when we deployed it to our development environment, we experienced significant issues. The loop appeared to become busy, consuming an entire CPU core, which effectively busted our online environment.

The documentation for as_event_create_loops mentions:

This method should only be called when async client commands will be used, and the calling program itself is not async.

We are trying to understand if this implies that our server, which is based on uv_loop, cannot use this method to create a separate loop for Aerospike. We also tried managing a separate thread and running a uv_loop ourselves with as_event_set_external_loop, but this led to issues with dangling objects and other hard-to-resolve problems.

Could anyone advise on the best practices for assigning Aerospike its own event loop, while allowing our main loop to focus on other tasks without overwhelming the CPU? We would greatly appreciate any guidance.

Thanks

[BrianNichols]

That api doc quote assumed that async applications always benefit from sharing event loops and suggested that creating dedicated aerospike event loops should only be necessary when the application is not async. In your case, creating dedicated aerospike event loops using as_event_create_loops() does seem to make sense. I will change the documentation to be more clear.

We use as_event_create_loops() in our benchmarks, examples and test applications and have not observed that busy with no workload scenario. My suggestions are:

• Upgrade to at least C client 6.6.2. That version contains a critical bug fix when using libuv and TLS.

• One drawback to using as_event_create_loops() is that the event loop thread(s) are not assignable to a specific cpu. This means an aerospike event loop and your application event loop may be running on the same cpu, thus reducing performance. Try creating external event loops as your original design and assign a cpu each event loop thread using as_cpu_assign_thread_attr() or as_cpu_assign_thread(). Distribute each thread to a dedicated cpu. Then, only run aerospike commands on those aerospike event loops.

[luyi404]

@BrianNichols Thank you for your response!

Just to clarify, our goal is to call as_event_create_loops in our main thread, where our main uv_loop also resides. We’d prefer to have the Aerospike C client manage its own threads and a separate event loops without interfering with our existing uv_loop.

Would this approach be available and if there is some potential issues? And if so is there an example for doing that?

We greatly appreciate your insights!

Versions:

• C Client: 6.6.3
• Server: Aerospike Enterprise Edition (ver. 6.4.0.21)

[BrianNichols]

Yes, this approach can work. as_event_create_loops() will spawn a new thread for each new event loop. Each thread will be assigned to a cpu by the operating system (usually by round-robin) and some threads might run on the same cpu as your application's uv_loop. Running multiple threads on the same cpu can cause performance issues if one of the threads has heavy cpu usage.

Another potential issue is running Aerospike commands from your application's uv_loop. Since the threads are now separate, the client must queue the command to the Aerospike event loop thread via a mutex lock. The Aerospike event loop thread also retrieves the command from the queue via the same lock.

Conversely, your application might have to queue Aerospike command results back to your application's uv_loop thread if it needs to process it there. Passing commands and/or data between threads must be done atomically and does incur an added cost.

[luyi404]

Thanks again, Brian.

1. Just to confirm, if we have the new uv_loop running on another thread, am I correct in thinking that the only time we use this separate thread is when we're invoking aerospike_key_get_async or similar async Aerospike calls?
2. Also, when you mentioned that passing commands and data between threads needs to be done atomically, could you clarify what specific data or commands you're referring to in this case?
3. Is it possible that we only use a separate uv_loop for the aerospike operation, and the callback function we passed into the async operation go back to main loop?

[BrianNichols]

1. Correct
2. If your application shares data between threads, the code accessing this data must be thread-safe.
3. Yes, as long as the code is thread-safe. If the client creates and returns a Record instance in the callback, then that Record instance can be used in your uv_loop normally and is thread-safe. However, if your callback code accesses existing shared data (like a counter that is incremented in both your application uv_loop thread and the client's callback uv_loop thread), then the code handling the shared data must be thread-safe.