8192647: GClocker induced GCs can starve threads requiring memory leading to OOME

[albertnetymk]

Here is an attempt to simplify GCLocker implementation for Serial and Parallel.

GCLocker prevents GC when Java threads are in a critical region (i.e., calling JNI critical APIs). JDK-7129164 introduces an optimization that updates a shared variable (used to track the number of threads in the critical region) only if there is a pending GC request. However, this also means that after reaching a GC safepoint, we may discover that GCLocker is active, preventing a GC cycle from being invoked. The inability to perform GC at a safepoint adds complexity -- for example, a caller must retry allocation if the request fails due to GC being inhibited by GCLocker.

The proposed patch uses a readers-writer lock to ensure that all Java threads exit the critical region before reaching a GC safepoint. This guarantees that once inside the safepoint, we can successfully invoke a GC cycle. The approach takes inspiration from ZJNICritical, but some regressions were observed in j2dbench (on Windows) and the micro-benchmark in JDK-8232575. Therefore, instead of relying on atomic operations on a global variable when entering or leaving the critical region, this PR uses an existing thread-local variable with a store-load barrier for synchronization.

Performance is neutral for all benchmarks tested: DaCapo, SPECjbb2005, SPECjbb2015, SPECjvm2008, j2dbench, and CacheStress.

Test: tier1-8

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Error

 ⚠️ 8192647 is used in problem lists: [test/hotspot/jtreg/ProblemList.txt]

Issue

• JDK-8192647: GClocker induced GCs can starve threads requiring memory leading to OOME (Bug - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23367/head:pull/23367
$ git checkout pull/23367

Update a local copy of the PR:
$ git checkout pull/23367
$ git pull https://git.openjdk.org/jdk.git pull/23367/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 23367

View PR using the GUI difftool:
$ git pr show -t 23367

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23367.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back ayang! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@albertnetymk The following labels will be automatically applied to this pull request:

• hotspot
• serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[albertnetymk]

/cc hotspot-gc

[openjdk]

@albertnetymk
The hotspot-gc label was successfully added.

[mlbridge]

Webrevs

• 00: Full (6283a19c)