HammerDB on AWS crashes with Segmentation Fault or Bus Error

[smutluru2020]

Hi Community,
Background:
I am trying to use Oracle Trace file to record all the SQL statements from my production workload and then converting the same in to tcl script using hammerdb's default converter.
Once the script is generated all i add is a try catch block around the orafetch or orasql statements.

I have attached the tcl script and logs from hammerdb strace for more details.

Platform:
AWS EC2 c5.12xlarge/c5n.12.xlarge running HammerDB (3.4) on RHEL 7
AWS RDS with Oracle 12.1.0.2 on r5.4xlarge

Problem:
When i run this script using 1 vuser the scripts takes 8-9 mins to complete and runs successfully. but as soon as i add multiple vusers to run the script simultaneously hammerdb crashes with segmentation fault or bus error. I have tried to debug this using following 2 approaches

1. Added 94 GB memory to my EC2 machine with 36 cores
2. made my EC2 c5n to allow higher network throughput to Oracle DB

I am constantly monitoring the EC2 instance and RDS instance using cloud watch and rds performance insights. I could not find any resource constraints or error logs/alert logs on RDS side.

From my observation it looks like an HammerDB issue. Did anyone else faced the similar problems running hammerdb on AWS for RDS?

Attached:

1. HammerDB Log
2. strace of hammerdb
3. tcl script

Log_Files.zip

[sm-shaw]

Many thanks for the issue and the log files I can take a look. A key aspect is where hammerdb is crashing - in the hammerdb engine itself or the 3rd party library. Are there any system logs or other information that could suggest a cause. One immediate thought is that with the TPC-C/TPC-H workloads it is carefully planned and tested not to have memory leaks from opening cursors and not closing them. So a key check is that the HammerDB client is not running out of memory from the statements that are being run from the trace files. The intention when replaying trace files is that the converted trace is used as a framework to build your own tests rather than a simple capture and re-run.

[sm-shaw]

Hi looking in the logfile there are a number of errors when you run the script. You would need to resolve these before looking at any crashing issues. As noted before if cursors are being opened and not closed you may run out of memory on the HammerDB client that would cause this effect.

Also there are some scenarios that are not currently converted. For example ref cursors and memory dumps. This has been noted and work was started on it however not completed Issue #27 and there is no current commitment to as and when work on trace file conversion will be done (although this would a good project for someone to pick up and work on.)

grep ORA- hammerdb_5E2889DE59CB03E203430353.log | sort --unique
Vuser 1:ORA-01008: not all variables bound
Vuser 1:ORA-01036: illegal variable name/number
Vuser 1:ORA-01843: not a valid month
Vuser 2:ORA-01008: not all variables bound
Vuser 2:ORA-01036: illegal variable name/number
Vuser 2:ORA-01843: not a valid month

[sm-shaw]

Closing as no further update