Resolve three scenarios that may cause workload to fail #422

qinjingyuan · 2023-08-25T02:41:09Z

resolve problems in describing #403.
using mothed：https://github.com/qinjingyuan/pgreplay-for-polardb

polardb-bot · 2023-08-25T02:42:23Z

Hi @qinjingyuan ~ Thanks for your contribution in this PR. ❤️

Please make sure that your PR conforms the standard, and has passed all the checks.

We will review your PR as soon as possible.

polardb-bot · 2023-08-25T03:45:00Z

Hey @qinjingyuan :

Congratulations~ 🎉 Your commit has passed all the checks. Please wait for further manual review.

polardb-bot · 2023-08-31T15:06:51Z

Hey @qinjingyuan :

Congratulations~ 🎉 Your commit has passed all the checks. Please wait for further manual review.

ocean-dot-li · 2023-10-19T16:59:02Z

src/backend/utils/misc/guc.c

+		true,
+		NULL,NULL,NULL
+	},
+


新增参数一般放在末尾，同时sql 长度已经有参数 polar_auditlog_max_query_length 控制，所以这个参数可以去除

ocean-dot-li · 2023-10-19T16:59:44Z

src/backend/utils/error/elog.c

@@ -4333,6 +4352,28 @@ polar_write_audit_log(ErrorData *edata, const char *fmt, ...)
 								log_channel_write_buffer_pos);
 			log_channel_write_buffer_pos = 0;
 			polar_last_audit_log_flush_time = GetCurrentTimestamp();
+			// solving incomplete audit sql log problem


长度限制，直接去除

ocean-dot-li · 2023-10-19T17:08:30Z

src/include/commands/prepare.h

@@ -56,5 +57,5 @@ extern TupleDesc FetchPreparedStatementResultDesc(PreparedStatement *stmt);
 extern List *FetchPreparedStatementTargetList(PreparedStatement *stmt);

 extern void DropAllPreparedStatements(void);
-
+extern char current_prepared_params_string[MAX_PREPARED_PARAMS_LEN];


这里采用固定长度的数组存储参数信息，存在不能动态扩容的限制。建议采用 MemoryContext 机制进行实现动态扩容，内存可以存放在 ErrorContext 中，正常情况下 log 完手动释放，报错时由系统自动释放。

对于 simple 协议的prepared 语句，sql 的执行会强依赖定义的SQL。如果pgreplay 中没有这个定义语句，则相关的 SQL 都不能回放。为了解决这个限制，需要在输出参数的同时，也要附带语句的定义 SQL。

ocean-dot-li · 2023-10-19T17:09:22Z

src/backend/utils/error/elog.c

@@ -4246,7 +4262,8 @@ polar_construct_logdata_postprocess(StringInfoData *logbuf, ErrorData *edata)
 	else
 	{
 		// NOTE(wormhole.gl): make sure logbuf length is not larger than PIPE_MAX_PAYLOAD
-		polar_shrink_audit_log(logbuf, 0);
+		if (polar_auditlog_max_query_length_limit)


回退，不需要更改

ocean-dot-li · 2023-10-19T17:14:49Z

src/backend/utils/error/elog.c

@@ -2603,6 +2606,19 @@ log_line_prefix(StringInfo buf, ErrorData *edata)
 					appendStringInfoString(buf, formatted_log_time);
 				break;
 			/* POLAR: new format for polar */
+			case 'L':


这里利用 log_line_prefix 增加两个参数用来支持log search path 和参数信息理论上是可以的。

但是考虑到新老版本兼容，以及周边生态的适配，如日志解析工具等，建议采用直接将上面两类信息和SQL文本一起输出。

在解析工具视角看，它就是一个SQL 文本，如果要适配，只需适配SQL 解析借口即可

利用两个 GUC 参数分别控制

ocean-dot-li · 2023-10-19T17:15:06Z

src/backend/utils/error/elog.c

@@ -116,8 +118,9 @@ char	   *Log_destination_string = NULL;
 bool		syslog_sequence_numbers = true;
 bool		syslog_split_messages = true;
 /* POLAR */
-#define LOG_CHANNEL_WRITE_BUFFER_SIZE 128 * 1024 /* 128k */
+#define LOG_CHANNEL_WRITE_BUFFER_SIZE 64 * 1024 /* 64k */


无效更改，回退

ocean-dot-li · 2023-10-19T17:18:33Z

src/backend/commands/prepare.c

@@ -226,6 +228,29 @@ ExecuteQuery(ExecuteStmt *stmt, IntoClause *intoClause,
 	/* Look it up in the hash table */
 	entry = FetchPreparedStatement(stmt->name, true);

+	// get prepare params for solving parameterization problem 


这里需要改变实现方式，见 prepare.h 中的评论。

同时对于 explain (analyze) execute s(1); 场景，也需要支持。

这里只是针对走simple 协议的 prepared 场景，对于走 extended 协议的场景，也需要输出参数信息，见 exec_execute_message 函数

src/backend/commands/prepare.c

polardb-bot · 2023-10-29T03:59:56Z

Hey @qinjingyuan :

Something wrong occuried during the checks of your commit 😟, please check the detail:

⚠️ build (centos7, oltp) View more details

polardb-bot · 2023-10-29T09:47:49Z

Hey @qinjingyuan :

Something wrong occuried during the checks of your commit 😟, please check the detail:

⚠️ build (centos7, oltp) View more details

qinjingyuan · 2023-10-29T14:38:06Z

reopen

qinjingyuan · 2023-10-29T14:38:25Z

reopen pr

polardb-bot · 2023-10-29T15:04:36Z

Hey @qinjingyuan :

Something wrong occuried during the checks of your commit 😟, please check the detail:

⚠️ build (centos7, oltp) View more details

polardb-bot · 2023-10-30T06:00:25Z

Hey @qinjingyuan :

Congratulations~ 🎉 Your commit has passed all the checks. Please wait for further manual review.

analyze poalrdb log file and monitor db system info. 1. support log search_path info to log file. 2. support log paremeters type for parameterization SQL. Only parameters type is logged for extended protocol. Parameters type and definnitation of SQL are logged for PREPARED simple protocol. 3. Add pgreplay tool. currently, this version pgreplay can support polardb audit log and workload replay monitor function 4. Support extension protocol of pg execution and normal execution when prepare is not present on pgreplay.

polardb-bot · 2023-10-30T07:26:15Z

Hey @qinjingyuan :

Congratulations~ 🎉 Your commit has passed all the checks. Please wait for further manual review.

mrdrivingduck · 2023-10-30T07:52:44Z

/rebase-and-merge

polardb-bot · 2023-10-30T07:53:07Z

@mrdrivingduck PR has been merged. 🍗

Thanks again for the contributors involved in this thread: ❤️

@qinjingyuan

polardb-bot bot added the ci/success CI status is success label Aug 25, 2023

qinjingyuan force-pushed the summer branch from 7ba8cd4 to 0919463 Compare August 31, 2023 13:56

polardb-bot bot added ci/success CI status is success and removed ci/success CI status is success labels Aug 31, 2023

ocean-dot-li approved these changes Oct 19, 2023

View reviewed changes

qinjingyuan force-pushed the summer branch from 0919463 to c10640e Compare October 29, 2023 03:49

polardb-bot bot added ci/failure CI status is failure and removed ci/success CI status is success labels Oct 29, 2023

qinjingyuan force-pushed the summer branch from c10640e to 309b20f Compare October 29, 2023 09:37

polardb-bot bot added ci/failure CI status is failure and removed ci/failure CI status is failure labels Oct 29, 2023

qinjingyuan closed this Oct 29, 2023

qinjingyuan deleted the summer branch October 29, 2023 14:31

qinjingyuan reopened this Oct 29, 2023

polardb-bot bot added ci/failure CI status is failure and removed ci/failure CI status is failure labels Oct 29, 2023

qinjingyuan force-pushed the summer branch from 309b20f to 6dba7b8 Compare October 30, 2023 05:13

polardb-bot bot added ci/success CI status is success and removed ci/failure CI status is failure labels Oct 30, 2023

qinjingyuan force-pushed the summer branch from 6dba7b8 to 3357dff Compare October 30, 2023 06:40

polardb-bot bot added ci/success CI status is success and removed ci/success CI status is success labels Oct 30, 2023

polardb-bot bot merged commit 6071c25 into ApsaraDB:POLARDB_11_DEV Oct 30, 2023
5 checks passed

open-digger-bot bot mentioned this pull request Nov 6, 2023

[WeeklyReport] Weekly report for PolarDB-for-PostgreSQL 10/30/2023 to 11/6/2023 #444

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolve three scenarios that may cause workload to fail #422

Resolve three scenarios that may cause workload to fail #422

qinjingyuan commented Aug 25, 2023

polardb-bot bot commented Aug 25, 2023

polardb-bot bot commented Aug 25, 2023

polardb-bot bot commented Aug 31, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

ocean-dot-li Oct 19, 2023

polardb-bot bot commented Oct 29, 2023

polardb-bot bot commented Oct 29, 2023

qinjingyuan commented Oct 29, 2023

qinjingyuan commented Oct 29, 2023

polardb-bot bot commented Oct 29, 2023

polardb-bot bot commented Oct 30, 2023

polardb-bot bot commented Oct 30, 2023

mrdrivingduck commented Oct 30, 2023

polardb-bot bot commented Oct 30, 2023

Resolve three scenarios that may cause workload to fail #422

Resolve three scenarios that may cause workload to fail #422

Conversation

qinjingyuan commented Aug 25, 2023

polardb-bot bot commented Aug 25, 2023

polardb-bot bot commented Aug 25, 2023

polardb-bot bot commented Aug 31, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

polardb-bot bot commented Oct 29, 2023

polardb-bot bot commented Oct 29, 2023

qinjingyuan commented Oct 29, 2023

qinjingyuan commented Oct 29, 2023

polardb-bot bot commented Oct 29, 2023

polardb-bot bot commented Oct 30, 2023

polardb-bot bot commented Oct 30, 2023

mrdrivingduck commented Oct 30, 2023

polardb-bot bot commented Oct 30, 2023