Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] Slow DB locks caused compaction to stop, resulting in many slow queries. #56344

Closed
bigdata-kuxingseng opened this issue Feb 27, 2025 · 4 comments
Assignees
Labels
type/bug Something isn't working

Comments

@bigdata-kuxingseng
Copy link
Contributor

version:3.2.12,shard_data
fe log:
2025-02-24 04:39:44.486+08:00 INFO (DeadlockChecker|44) [LockChecker.checkSlowLocks():109] slow db locks: [{“lockState”:“readLocked”,“slowReadLockCount”:1,“dumpThreads”:“lockHoldTime: 2327990 ms;dump thread: StarMgrMetaSyncer, id: 50\n [email protected]/jdk.internal.misc.Unsafe.park(Native Method)\n [email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)\n app//io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:717)\n app//io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:159)\n app//com.staros.proto.StarManagerGrpc$StarManagerBlockingStub.listShard(StarManagerGrpc.java:2117)\n app//com.staros.client.StarClient.lambda$listShard$21(StarClient.java:521)\n app//com.staros.client.StarClient$$Lambda$2162/0x00000008018ff338.call(Unknown Source)\n app//com.staros.client.StarClient.internalRpcCall(StarClient.java:770)\n app//com.staros.client.StarClient.listShard(StarClient.java:521)\n app//com.starrocks.lake.StarOSAgent.listShard(StarOSAgent.java:499)\n app//com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaInternal(StarMgrMetaSyncer.java:301)\n app//com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfoInternal(StarMgrMetaSyncer.java:365)\n app//com.starrocks.lake.StarMgrMetaSyncer.syncTableMetaAndColocationInfo(StarMgrMetaSyncer.java:262)\n app//com.starrocks.lake.StarMgrMetaSyncer.runAfterCatalogReady(StarMgrMetaSyncer.java:376)\n app//com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72)\n app//com.starrocks.common.util.Daemon.run(Daemon.java:107)\n;”,“lockDbName”:“sl_attribution_preview”,“lockWaiters”:[{“threadId”:6540484,“threadName”:“thrift-server-pool-227883”},{“threadId”:6467634,“threadName”:“thrift-server-pool-227276”},{“threadId”:6536163,“threadName”:“starrocks-mysql-nio-pool-106760”},{“threadId”:6535726,“threadName”:“starrocks-mysql-nio-pool-106759”},{“threadId”:6538262,“threadName”:“thrift-server-pool-227860”},{“threadId”:6507676,“threadName”:“thrift-server-pool-227620”},{“threadId”:48,“threadName”:“DynamicPartitionScheduler”},{“threadId”:28,“threadName”:“UpdateDbUsedDataQuota”},{“threadId”:56,“threadName”:“MemoryUsageTracker”},{“threadId”:24,“threadName”:“consistency checker”},{“threadId”:98,“threadName”:“COMPACTION_DISPATCH”},{“threadId”:54,“threadName”:“autovacuum”},{“threadId”:33,“threadName”:“tablet stat mgr”}]}]

Image

@bigdata-kuxingseng bigdata-kuxingseng added the type/bug Something isn't working label Feb 27, 2025
@bigdata-kuxingseng bigdata-kuxingseng changed the title Slow DB locks caused compaction to stop, resulting in many slow queries. 【bug】Slow DB locks caused compaction to stop, resulting in many slow queries. Feb 27, 2025
@starrocks-xupeng
Copy link
Contributor

what's the last start time of this FE

@starrocks-xupeng
Copy link
Contributor

it's been fixed in 3.3.9.
and if you using 3.2 version, this problem only appears when FE doesn't restart for a long time(like several months), so you can restart for now and upgrade in some time.

@bigdata-kuxingseng
Copy link
Contributor Author

the last start time of this FE is :2025-01-09, could you please send me the PR,thanks

@kevincai kevincai changed the title 【bug】Slow DB locks caused compaction to stop, resulting in many slow queries. [bug] Slow DB locks caused compaction to stop, resulting in many slow queries. Feb 28, 2025
@kevincai
Copy link
Contributor

the last start time of this FE is :2025-01-09, could you please send me the PR,thanks

#54063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants