perf: 本地文件准备过程优化 #2611 #2612

jsonwan · 2023-11-11T13:31:46Z

处理traceId断链问题；
优化日志，重构部分代码；
支持发布过程中文件准备过程取消后重新准备文件。

1. 处理traceId断链问题； 2. 优化日志，重构部分代码。

1. 支持发布过程中文件准备过程取消后重新准备文件； 2. 修复StopTask中traceId断链问题。

1. 修复重新准备文件导致的多条任务记录查询报错问题。

1. 第三方文件源临时文件清理优化。

wangyu096 · 2023-11-13T06:28:25Z

...-job-execute/src/main/java/com/tencent/bk/job/execute/dao/impl/FileSourceTaskLogDAOImpl.java

@@ -93,18 +80,21 @@ public void saveFileSourceTaskLog(FileSourceTaskLogDTO fileSourceTaskLog) {
            .set(t.START_TIME, fileSourceTaskLog.getStartTime())
            .set(t.END_TIME, fileSourceTaskLog.getEndTime())
            .set(t.TOTAL_TIME, fileSourceTaskLog.getTotalTime())
-            .set(t.STATUS, JooqDataTypeUtil.toByte(fileSourceTaskLog.getStatus())).set(t.FILE_SOURCE_BATCH_TASK_ID, fileSourceTaskLog.getFileSourceBatchTaskId())
+            .set(t.STATUS, JooqDataTypeUtil.toByte(fileSourceTaskLog.getStatus())).set(t.FILE_SOURCE_BATCH_TASK_ID,


换行格式不对

wangyu096 · 2023-11-13T06:29:59Z

...-job-execute/src/main/java/com/tencent/bk/job/execute/dao/impl/FileSourceTaskLogDAOImpl.java

            .execute();
    }

    @Override
-    public FileSourceTaskLogDTO getFileSourceTaskLog(long stepInstanceId, int executeCount) {
+    public FileSourceTaskLogDTO getLatestFileSourceTaskLog(long stepInstanceId, int executeCount) {
        FileSourceTaskLog t = FileSourceTaskLog.FILE_SOURCE_TASK_LOG;
        Record record = defaultContext.select(t.STEP_INSTANCE_ID, t.EXECUTE_COUNT, t.START_TIME, t.END_TIME,
            t.TOTAL_TIME,


换行格式有点问题

wangyu096 · 2023-11-13T06:33:46Z

...b-file-worker-sdk/src/main/java/com/tencent/bk/job/file/worker/task/clear/ClearFileTask.java

@@ -144,6 +144,10 @@ public void checkVolumeAndClear() {
    private void doCheckVolumeAndClear() {
        long maxSizeBytes = workerConfig.getMaxSizeGB() * 1024L * 1024L * 1024L;
        File workDirFile = new File(workerConfig.getWorkspaceDirPath());
+        if (!workDirFile.exists()) {
+            log.debug("Workspace({}) not exists yet, ignore clear", workerConfig.getWorkspaceDirPath());


debug 日志需要加上 isDebugEnable 判断

wangyu096 · 2023-11-13T06:44:33Z

...ob-execute/src/main/java/com/tencent/bk/job/execute/engine/listener/GseStepEventHandler.java

@@ -139,6 +139,9 @@ public void handleEvent(StepEvent stepEvent,
                case CONTINUE_FILE_PUSH:
                    continueGseFileStep(stepInstance);
                    break;
+                case PREPARE_FILE_AGAIN:


PREPARE_FILE_AGAIN 命名有点奇怪。执行引擎的事件都应该是幂等的，这里不需要加上 AGAIN，与其它地方不统一。建议改成 PREPARE_FILE

wangyu096 · 2023-11-13T06:46:27Z

...-job-execute/src/main/java/com/tencent/bk/job/execute/dao/impl/FileSourceTaskLogDAOImpl.java

        FileSourceTaskLog t = FileSourceTaskLog.FILE_SOURCE_TASK_LOG;
        Record record = defaultContext.select(t.STEP_INSTANCE_ID, t.EXECUTE_COUNT, t.START_TIME, t.END_TIME,
            t.TOTAL_TIME,
            t.STATUS, t.FILE_SOURCE_BATCH_TASK_ID).from(t)
            .where(t.STEP_INSTANCE_ID.eq(stepInstanceId))
            .and(t.EXECUTE_COUNT.eq(executeCount))
+            .orderBy(t.ID.desc())


逻辑上有问题，依赖了主键自增长的条件。后续分库分表改造之后，使用分布式自增长主键就没法保证这种连续性了。

wangyu096 · 2023-11-13T09:43:17Z

...ob-execute/src/main/java/com/tencent/bk/job/execute/engine/listener/GseStepEventHandler.java

@@ -139,6 +139,9 @@ public void handleEvent(StepEvent stepEvent,
                case CONTINUE_FILE_PUSH:
                    continueGseFileStep(stepInstance);
                    break;
+                case PREPARE_FILE:
+                    prepareFileAgain(stepInstance);


方法名需要修改为 prepareFile(), 与 action 对应起来

wangyu096 · 2023-11-13T09:43:39Z

...ob-execute/src/main/java/com/tencent/bk/job/execute/engine/listener/GseStepEventHandler.java

+     *
+     * @param stepInstance 步骤实例
+     */
+    private void prepareFileAgain(StepInstanceDTO stepInstance) {


方法名需要修改为 prepareFile(), 与 action 对应起来。注释需要改成"准备要分发的文件"

处理Review意见。

jsonwan added 4 commits November 11, 2023 14:50

perf: 本地文件准备过程优化 TencentBlueKing#2611

3b8ee6f

1. 支持发布过程中文件准备过程取消后重新准备文件； 2. 修复StopTask中traceId断链问题。

perf: 本地文件准备过程优化 TencentBlueKing#2611

60f449b

1. 修复重新准备文件导致的多条任务记录查询报错问题。

perf: 本地文件准备过程优化 TencentBlueKing#2611

2bbb146

1. 第三方文件源临时文件清理优化。

wangyu096 requested changes Nov 13, 2023

View reviewed changes

wangyu096 approved these changes Nov 13, 2023

View reviewed changes

perf: 本地文件准备过程优化 TencentBlueKing#2611

5ff450e

处理Review意见。

jsonwan force-pushed the github_perf/localfile branch from c198f68 to 5ff450e Compare November 13, 2023 09:46

jsonwan merged commit 3c0e227 into TencentBlueKing:master Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: 本地文件准备过程优化 #2611 #2612

perf: 本地文件准备过程优化 #2611 #2612

jsonwan commented Nov 11, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

wangyu096 Nov 13, 2023

jsonwan Nov 13, 2023

perf: 本地文件准备过程优化 #2611 #2612

perf: 本地文件准备过程优化 #2611 #2612

Conversation

jsonwan commented Nov 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment