-
Notifications
You must be signed in to change notification settings - Fork 93
/
atom.xml
513 lines (301 loc) · 506 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>zhisheng的博客</title>
<subtitle>坑要一个个填，路要一步步走！</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://www.54tianzhisheng.cn/"/>
<updated>2024-02-21T14:18:02.012Z</updated>
<id>http://www.54tianzhisheng.cn/</id>
<author>
<name>zhisheng</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>Apache Paimon Primary Key 表</title>
<link href="http://www.54tianzhisheng.cn/2023/12/27/apache_paimon_primary_key_table/"/>
<id>http://www.54tianzhisheng.cn/2023/12/27/apache_paimon_primary_key_table/</id>
<published>2023-12-26T16:00:00.000Z</published>
<updated>2024-02-21T14:18:02.012Z</updated>
<content type="html"><![CDATA[<p>创建表时，默认的表类型是 Changelog 表。用户可以在该表中插入、更新或删除记录。</p><a id="more"></a><p>主键由一组包含每个记录唯一值的列组成。Paimon 通过在每个 Bucket 内对主键进行排序来强制执行数据排序，使用户能够通过在主键上应用过滤条件来实现高性能。</p><p>通过在 Changelog 表上定义主键，用户可以获得以下功能。</p><h2 id="Bucket"><a href="#Bucket" class="headerlink" title="Bucket"></a>Bucket</h2><p>Bucket 是读写的最小存储单元，每个 Bucket 目录包含一个 LSM 树。</p><h3 id="Fixed-Bucket"><a href="#Fixed-Bucket" class="headerlink" title="Fixed Bucket"></a>Fixed Bucket</h3><p>配置一个大于 0 的 Bucket 数量，对 Bucket 进行重新划分只能通过离线方式处理，参见<a href="https://paimon.apache.org/docs/master/maintenance/rescale-bucket/">Rescale Bucket</a>，过多的 Bucket 会导致过多的小文件，而过少的 Bucket 会导致写入性能下降。</p><h3 id="Dynamic-Bucket"><a href="#Dynamic-Bucket" class="headerlink" title="Dynamic Bucket"></a>Dynamic Bucket</h3><p>配置 <code>'bucket' = '-1'</code>的话，Paimon 会动态维护索引，自动扩展 Bucket 的数量。</p><ul><li><code>dynamic-bucket.target-row-num</code>：控制一个 Bucket 的目标行数。</li><li><code>dynamic-bucket.assigner-parallelism</code>：分配器操作符的并行度，控制初始化 Bucket 的数量。</li></ul><p>注：动态 Bucket 只支持单个写入作业。请不要启动多个作业同时写入同一个分区。</p><h4 id="普通的动态-Bucket-模式"><a href="#普通的动态-Bucket-模式" class="headerlink" title="普通的动态 Bucket 模式"></a>普通的动态 Bucket 模式</h4><p>当你的更新操作不跨越分区（没有分区，或者主键包含所有分区字段）时，动态 Bucket 模式使用哈希索引来维护从键到 Bucket 的映射关系，它需要比固定 Bucket 模式更多的内存。<br>性能：</p><ul><li>一般而言，性能不会有损失，但会有一些额外的内存消耗。每个分区中的 1 亿条记录会增加约 1 GB 的内存消耗，不再活跃的分区不会占用内存。</li><li>对于更新频率较低的表，建议使用此模式来显著提高性能。</li></ul><h4 id="普通的动态-Bucket-模式的排序压缩"><a href="#普通的动态-Bucket-模式的排序压缩" class="headerlink" title="普通的动态 Bucket 模式的排序压缩"></a>普通的动态 Bucket 模式的排序压缩</h4><p>普通动态 Bucket 模式支持<a href="https://paimon.apache.org/docs/master/maintenance/dedicated-compaction/#sort-compact">排序压缩</a>以加速查询。</p><h4 id="跨分区-Upsert-动态-Bucket-模式"><a href="#跨分区-Upsert-动态-Bucket-模式" class="headerlink" title="跨分区 Upsert 动态 Bucket 模式"></a>跨分区 Upsert 动态 Bucket 模式</h4><p>这是一个实验性的功能。</p><p>当需要进行跨分区的 Upsert 操作（主键不包含所有分区字段）时，动态 Bucket 模式直接维护键到分区和 Bucket 的映射关系，使用本地磁盘，并在启动流式写入作业时通过读取表中所有现有键来初始化索引。不同的合并引擎具有不同的行为：</p><ul><li>Deduplicate：从旧的分区删除数据，并将新数据插入到新的分区中。</li><li>PartialUpdate & Aggregation：将新数据插入到旧的分区中。</li><li>FirstRow：如果存在旧值，则忽略新数据。</li></ul><p>性能：对于数据量较大的表，性能会有显著损失。此外，初始化过程需要很长时间。</p><p>如果 Upsert 操作不依赖于过旧的数据，可以考虑配置索引的 TTL 以减少索引和初始化时间：<br><code>cross-partition-upsert.index-ttl</code>：RocksDB 索引和初始化的 TTL，这可以避免维护过多的索引并导致性能逐渐变差。但请注意，这也可能会导致数据重复。</p><h2 id="Merge-Engines"><a href="#Merge-Engines" class="headerlink" title="Merge Engines"></a>Merge Engines</h2><p>当 Paimon Sink 接收到两条或者两条以上相同主键的数据，它会将它们合并成一条保证主键唯一，通过指定表的 merge-engine 属性，用户可以选择如何将数据合并在一起。</p><p>注：在 Flink SQL TableConfig 中始终将 <code>table.exec.sink.upsert-materialize</code> 设置为 NONE，因为启用 sink upsert-materialize 可能会导致奇怪的行为。当输入数据无序时，建议使用<a href="https://paimon.apache.org/docs/master/concepts/primary-key-table/#sequence-field">序列字段</a>来纠正无序性。</p><h3 id="Deduplicate"><a href="#Deduplicate" class="headerlink" title="Deduplicate"></a>Deduplicate</h3><p>Deduplicate 是默认的 merge engine。Paimon 只会保留最新的记录，并丢弃具有相同主键的其他记录。<br>具体而言，如果最新的记录是一个 DELETE 记录，那么所有具有相同主键的记录都将被删除。</p><h3 id="Partial-Update"><a href="#Partial-Update" class="headerlink" title="Partial Update"></a>Partial Update</h3><p>通过指定 ‘merge-engine’ = ‘partial-update’，用户可以通过多次更新来更新记录的列，直到记录完整为止。这是通过逐个更新值字段，使用相同主键下的最新数据来实现的。但在这个过程中，空值不会被覆盖。<br>例如，假设 Paimon 收到三条记录：</p><p><1, 23.0, 10, NULL></p><p><1, NULL, NULL, 'This is a book'></p><1, 25.2, NULL, NULL><p>假设第一列是主键，最终的结果将是 <1, 25.2, 10, 'This is a book'>。</p><blockquote><p>对于流式查询，partial-update 合并引擎必须与查找（lookup）或完全合并（full-compaction）的 <a href="https://paimon.apache.org/docs/master/concepts/primary-key-table/#changelog-producers">changelog producer</a> 一起使用。（’input’ <a href="https://paimon.apache.org/docs/master/concepts/primary-key-table/#changelog-producers">changelog producer</a> 也支持，但只返回输入记录。）</p><p>默认情况下，部分更新不接受删除记录，可以选择以下解决方案之一：</p><ul><li>配置 ‘partial-update.ignore-delete’ 以忽略删除记录。</li><li>配置 ‘sequence-group’ 来撤销部分列。</li></ul></blockquote><h4 id="Sequence-Group"><a href="#Sequence-Group" class="headerlink" title="Sequence Group"></a>Sequence Group</h4><p>一个序列字段可能无法解决具有多个流式更新的 partial-update 表的乱序问题，因为在多流更新期间，序列字段可能会被另一个流的最新数据覆盖。</p><p>因此，我们为 partial-update 表引入了序列组机制。它可以解决以下问题：</p><ul><li>多流更新期间的乱序问题。每个流定义自己的序列组。</li><li>真正的 partial-update，而不仅仅是非空更新。</li></ul><p>请参考以下示例：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> T (</span><br><span class="line"> k <span class="built_in">INT</span>,</span><br><span class="line"> a <span class="built_in">INT</span>,</span><br><span class="line"> b <span class="built_in">INT</span>,</span><br><span class="line"> g_1 <span class="built_in">INT</span>,</span><br><span class="line"> c <span class="built_in">INT</span>,</span><br><span class="line"> d <span class="built_in">INT</span>,</span><br><span class="line"> g_2 <span class="built_in">INT</span>,</span><br><span class="line"> PRIMARY <span class="keyword">KEY</span> (k) <span class="keyword">NOT</span> <span class="keyword">ENFORCED</span></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'merge-engine'</span>=<span class="string">'partial-update'</span>,</span><br><span class="line"> <span class="string">'fields.g_1.sequence-group'</span>=<span class="string">'a,b'</span>,</span><br><span class="line"> <span class="string">'fields.g_2.sequence-group'</span>=<span class="string">'c,d'</span></span><br><span class="line">);</span><br><span class="line"></span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">1</span>);</span><br><span class="line"></span><br><span class="line"><span class="comment">-- g_2 is null, c, d should not be updated</span></span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="number">2</span>, <span class="number">2</span>, <span class="number">2</span>, <span class="number">2</span>, <span class="number">2</span>, <span class="keyword">CAST</span>(<span class="literal">NULL</span> <span class="keyword">AS</span> <span class="built_in">INT</span>));</span><br><span class="line"></span><br><span class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> T; <span class="comment">-- output 1, 2, 2, 2, 1, 1, 1</span></span><br><span class="line"></span><br><span class="line"><span class="comment">-- g_1 is smaller, a, b should not be updated</span></span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="number">3</span>, <span class="number">3</span>, <span class="number">1</span>, <span class="number">3</span>, <span class="number">3</span>, <span class="number">3</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> T; <span class="comment">-- output 1, 2, 2, 2, 3, 3, 3</span></span><br></pre></td></tr></table></figure><p>对于字段 sequence-group，有效的比较数据类型包括：DECIMAL、TINYINT、SMALLINT、INTEGER、BIGINT、FLOAT、DOUBLE、DATE、TIME、TIMESTAMP 和 TIMESTAMP_LTZ。</p><h4 id="Default-Value"><a href="#Default-Value" class="headerlink" title="Default Value"></a>Default Value</h4><p>如果无法保证数据的顺序，并且字段仅通过覆盖 null 进行写入，那么在读取表时，未被覆盖的字段将显示为 null。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> T (</span><br><span class="line"> k <span class="built_in">INT</span>,</span><br><span class="line"> a <span class="built_in">INT</span>,</span><br><span class="line"> b <span class="built_in">INT</span>,</span><br><span class="line"> c <span class="built_in">INT</span>,</span><br><span class="line"> PRIMARY <span class="keyword">KEY</span> (k) <span class="keyword">NOT</span> <span class="keyword">ENFORCED</span></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'merge-engine'</span>=<span class="string">'partial-update'</span></span><br><span class="line"> );</span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="number">1</span>,<span class="literal">null</span>,<span class="literal">null</span>);</span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="literal">null</span>,<span class="literal">null</span>,<span class="number">1</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> T; <span class="comment">-- output 1, 1, null, 1</span></span><br></pre></td></tr></table></figure><p>如果期望在读取表时，未被覆盖的字段具有默认值而不是 null，则需要使用 ‘fields.name.default-value’。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> T (</span><br><span class="line"> k <span class="built_in">INT</span>,</span><br><span class="line"> a <span class="built_in">INT</span>,</span><br><span class="line"> b <span class="built_in">INT</span>,</span><br><span class="line"> c <span class="built_in">INT</span>,</span><br><span class="line"> PRIMARY <span class="keyword">KEY</span> (k) <span class="keyword">NOT</span> <span class="keyword">ENFORCED</span></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'merge-engine'</span>=<span class="string">'partial-update'</span>,</span><br><span class="line"> <span class="string">'fields.b.default-value'</span>=<span class="string">'0'</span></span><br><span class="line">);</span><br><span class="line"></span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="number">1</span>,<span class="literal">null</span>,<span class="literal">null</span>);</span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> (<span class="number">1</span>, <span class="literal">null</span>,<span class="literal">null</span>,<span class="number">1</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">SELECT</span> * <span class="keyword">FROM</span> T; <span class="comment">-- output 1, 1, 0, 1</span></span><br></pre></td></tr></table></figure><h3 id="Aggregation"><a href="#Aggregation" class="headerlink" title="Aggregation"></a>Aggregation</h3><p>注意：在 Flink SQL TableConfig 中，始终将 table.exec.sink.upsert-materialize 设置为 NONE。<br>有时用户只关心聚合结果。aggregation 合并引擎根据聚合函数，逐个将每个值字段与相同主键下的最新数据进行聚合。</p><p>非主键字段可以指定聚合函数，通过 fields.<field-name>.aggregate-function 表属性来指定，否则将默认使用 last_non_null_value 聚合。例如，考虑以下表定义。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<p>创建表时，默认的表类型是 Changelog 表。用户可以在该表中插入、更新或删除记录。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Apache Paimon" scheme="http://www.54tianzhisheng.cn/tags/Apache-Paimon/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="数据湖" scheme="http://www.54tianzhisheng.cn/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/"/>
</entry>
<entry>
<title>Apache Paimon 文件管理</title>
<link href="http://www.54tianzhisheng.cn/2023/12/26/apache_paimon_file_manager/"/>
<id>http://www.54tianzhisheng.cn/2023/12/26/apache_paimon_file_manager/</id>
<published>2023-12-25T16:00:00.000Z</published>
<updated>2024-02-21T14:14:54.774Z</updated>
<content type="html"><![CDATA[<h3 id="管理小文件"><a href="#管理小文件" class="headerlink" title="管理小文件"></a>管理小文件</h3><p>许多用户关注小文件问题，可能导致以下情况：</p><a id="more"></a><ul><li>稳定性问题：HDFS 中如果存在太多小文件的话会导致 NameNode 压力过大</li><li>成本问题：在 HDFS 中，每个小文件都会占用至少一个数据块的大小，例如 128 MB</li><li>查询效率：查询过多小文件会影响查询效率<h3 id="理解-Checkpoint"><a href="#理解-Checkpoint" class="headerlink" title="理解 Checkpoint"></a>理解 Checkpoint</h3></li></ul><p>假设你正在使用 Flink Writer，每个 Checkpoint 会生成 1 ～ 2 个 snapshot，并且 Checkpoint 时会强制将文件生成在分布式文件系统（DFS）上，因此 Checkpoint 间隔越小，生成的小文件就越多。</p><p>1、所以先要增加 Checkpoint 间隔时间</p><p>默认情况下，不仅 Checkpoint 会生成文件，写入器（Writer）的内存（<code>write-buffer-size</code>）耗尽时也会将数据刷新到 DFS 并生成相应的文件。你可以启用 <code>write-buffer-spillable</code> 在写入器中生成溢出文件，以生成更大的文件在 DFS 上。</p><p>2、其次增加 <code>write-buffer-size</code> 或启用 <code>write-buffer-spillable</code></p><h3 id="理解-Snapshot"><a href="#理解-Snapshot" class="headerlink" title="理解 Snapshot"></a>理解 Snapshot</h3><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141409.png" alt=""></p><p>Paimon 维护文件的多个版本，文件的合并和删除是逻辑上的操作，并不实际删除文件。只有在 snapshot 过期时，文件才会真正被删除，所以减少文件的一种方法是缩短 snapshot 过期的时间。Flink Writer 会自动处理过期的 snapshot。</p><h3 id="理解-分区-和-Buckets"><a href="#理解-分区-和-Buckets" class="headerlink" title="理解 分区 和 Buckets"></a>理解 分区 和 Buckets</h3><p>Paimon 的文件以分层方式组织。下图展示了文件布局。从 snapshot 文件开始，Paimon 的读取器可以递归地访问表中的所有记录。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141428.png" alt=""></p><p>举个例子：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> MyTable (</span><br><span class="line"> user_id <span class="built_in">BIGINT</span>,</span><br><span class="line"> item_id <span class="built_in">BIGINT</span>,</span><br><span class="line"> behavior <span class="keyword">STRING</span>,</span><br><span class="line"> dt <span class="keyword">STRING</span>,</span><br><span class="line"> hh <span class="keyword">STRING</span>,</span><br><span class="line"> PRIMARY <span class="keyword">KEY</span> (dt, hh, user_id) <span class="keyword">NOT</span> <span class="keyword">ENFORCED</span></span><br><span class="line">) PARTITIONED <span class="keyword">BY</span> (dt, hh) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'bucket'</span> = <span class="string">'10'</span></span><br><span class="line">);</span><br></pre></td></tr></table></figure><p>表数据会被物理分片到不同的分区，里面有不同的 Bucket ，所以如果整体数据量太小，单个 Bucket 中至少有一个文件，建议你配置较少的 Bucket 数量，否则会出现也有很多小文件。</p><h3 id="理解-Primary-Table-的-LSM"><a href="#理解-Primary-Table-的-LSM" class="headerlink" title="理解 Primary Table 的 LSM"></a>理解 Primary Table 的 LSM</h3><p>LSM 树将文件组织成多个 sorted run。一个 sorted run 由一个或多个数据文件组成，每个数据文件都属于且仅属于一个 sorted run。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141442.png" alt=""></p><p>默认情况下，sorted run 的数量取决于 <code>num-sorted-run.compaction-trigger</code> <a href="https://paimon.apache.org/docs/master/maintenance/write-performance/#number-of-sorted-runs-to-trigger-compaction">参数</a>，这意味着一个Bucket 中至少有 5 个文件。如果你想减少这个数量，可以保留较少的文件，但写入性能可能会受到影响。如果该值变得过大，在查询表时会需要更多的内存和 CPU，这是写入性能和查询性能之间的权衡。</p><h3 id="理解-Append-Only-表的文件"><a href="#理解-Append-Only-表的文件" class="headerlink" title="理解 Append-Only 表的文件"></a>理解 Append-Only 表的文件</h3><p>默认情况下 Append Only 表也会进行自动合并以减少小文件的数量。</p><p>然而，对于 Bucket 的 Append Only 表来说，它会出于顺序目的而只<a href="https://paimon.apache.org/docs/master/concepts/append-only-table/#compaction">压缩</a> Bucket 内的文件，这可能会保留更多的小文件。</p><h3 id="理解-Full-Compaction"><a href="#理解-Full-Compaction" class="headerlink" title="理解 Full Compaction"></a>理解 Full Compaction</h3><p>也许你认为 Primary Key 表中的 5 个文件还可以接受，但 Append Only 表（Bucket）可能在一个单独的 Bucket 中就会有 50 个小文件，这是很难接受的。更糟糕的是，不再活跃的分区也会保留这么多小文件。</p><p>建议你配置<a href="https://paimon.apache.org/docs/master/maintenance/read-performance/#full-compaction">全量合并</a>（Full-Compaction），通过设置 <code>full-compaction.delta-commits</code>参数，在Flink 写入过程中定期执行全量合并，这样可以确保在写入结束之前对分区进行完全合并。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h3 id="管理小文件"><a href="#管理小文件" class="headerlink" title="管理小文件"></a>管理小文件</h3><p>许多用户关注小文件问题，可能导致以下情况：</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Apache Paimon" scheme="http://www.54tianzhisheng.cn/tags/Apache-Paimon/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="数据湖" scheme="http://www.54tianzhisheng.cn/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/"/>
</entry>
<entry>
<title>Apache Paimon 文件操作</title>
<link href="http://www.54tianzhisheng.cn/2023/12/25/apache_paimon_file_operator/"/>
<id>http://www.54tianzhisheng.cn/2023/12/25/apache_paimon_file_operator/</id>
<published>2023-12-24T16:00:00.000Z</published>
<updated>2024-02-21T14:13:59.185Z</updated>
<content type="html"><![CDATA[<p>本文旨在澄清不同文件操作对文件的影响。</p><p>本页面提供具体示例和实用技巧，以有效地管理这些操作。此外，通过对提交（commit）和压实（compact）等操作的深入探讨，我们旨在提供有关文件创建和更新的见解。</p><a id="more"></a><h3 id="创建-catalog"><a href="#创建-catalog" class="headerlink" title="创建 catalog"></a>创建 catalog</h3><p>在 Flink lib 中放入 <a href="https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/0.6-SNAPSHOT/">paimon-flink 依赖包</a>，执行 ./sql-client.sh 启动 Flink SQL Client，然后执行下面的命令去创建 Paimon catlog：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">CREATE CATALOG paimon <span class="title">WITH</span> <span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="string">'type'</span> = <span class="string">'paimon'</span>,</span></span></span><br><span class="line"><span class="function"><span class="params"><span class="string">'warehouse'</span> = <span class="string">'file:///tmp/paimon'</span></span></span></span><br><span class="line"><span class="function"><span class="params">)</span></span>;</span><br><span class="line"></span><br><span class="line">USE CATALOG paimon;</span><br></pre></td></tr></table></figure><p>执行完后会在 <code>file:///tmp/paimon</code> 路径下创建目录 <code>default.db</code></p><h3 id="创建-Table"><a href="#创建-Table" class="headerlink" title="创建 Table"></a>创建 Table</h3><p>执行下面的命令会创建一个带有 3 个属性的 Paimon 表：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">CREATE TABLE <span class="title">T</span> <span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"> id BIGINT,</span></span></span><br><span class="line"><span class="function"><span class="params"> a INT,</span></span></span><br><span class="line"><span class="function"><span class="params"> b STRING,</span></span></span><br><span class="line"><span class="function"><span class="params"> dt STRING COMMENT <span class="string">'timestamp string in format yyyyMMdd'</span>,</span></span></span><br><span class="line"><span class="function"><span class="params"> PRIMARY KEY(id, dt)</span> NOT ENFORCED</span></span><br><span class="line"><span class="function">) PARTITIONED <span class="title">BY</span> <span class="params">(dt)</span></span>;</span><br></pre></td></tr></table></figure><p>执行后，Paimon 表 T 会在 <code>/tmp/paimon/default.db/T</code> 目录下生成目录，它的 schema 会存放在目录 <code>/tmp/paimon/default.db/T/schema/schema-0</code> 下。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140336.png" alt=""></p><h3 id="写入数据到-Table"><a href="#写入数据到-Table" class="headerlink" title="写入数据到 Table"></a>写入数据到 Table</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">INSERT INTO T <span class="title">VALUES</span> <span class="params">(<span class="number">1</span>, <span class="number">10001</span>, <span class="string">'varchar00001'</span>, <span class="string">'20230501'</span>)</span></span>;</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140410.png" alt=""></p><p>用户可以通过执行查询 SELECT * FROM T 来验证这些记录的可见性，该查询将返回一行结果。提交过程会创建一个位于路径 /tmp/paimon/default.db/T/snapshot/snapshot-1 的快照。快照-1 的文件布局如下所述：</p><p>一旦任务运行完成变成 finished 时，提交成功后数据就写入到 Paimon 表中。用户可以通过执行查询 <code>SELECT * FROM T</code> 来验证数据的可见性，该查询将返回一行结果。 </p><p>另外可以发现目录的结构发生如下变化，新增 <code>dt=20230501</code>、<code>manifest</code> 和 <code>snapshot</code>三个目录。三个的目录结构如下：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140431.png" alt=""></p><p>查询目录下的数据如下所示：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140446.png" alt=""></p><p>这是因为提交过程中会创建一个位于 <code>/tmp/paimon/default.db/T/snapshot/snapshot-1</code> 的快照。<code>snapshot-1</code> 的文件布局如下所述：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140508.png" alt=""></p><p><code>snapshot-1</code>的内容包含了这个 snapshot 的元数据，比如 manifest list 和 schema id：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"version"</span> : <span class="number">3</span>,</span><br><span class="line"> <span class="attr">"id"</span> : <span class="number">1</span>,</span><br><span class="line"> <span class="attr">"schemaId"</span> : <span class="number">0</span>,</span><br><span class="line"> <span class="attr">"baseManifestList"</span> : <span class="string">"manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-0"</span>,</span><br><span class="line"> <span class="attr">"deltaManifestList"</span> : <span class="string">"manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-1"</span>,</span><br><span class="line"> <span class="attr">"changelogManifestList"</span> : <span class="literal">null</span>,</span><br><span class="line"> <span class="attr">"commitUser"</span> : <span class="string">"5132ef16-41ec-4172-8bf4-01a304507b36"</span>,</span><br><span class="line"> <span class="attr">"commitIdentifier"</span> : <span class="number">9223372036854775807</span>,</span><br><span class="line"> <span class="attr">"commitKind"</span> : <span class="string">"APPEND"</span>,</span><br><span class="line"> <span class="attr">"timeMillis"</span> : <span class="number">1697080282120</span>,</span><br><span class="line"> <span class="attr">"logOffsets"</span> : { },</span><br><span class="line"> <span class="attr">"totalRecordCount"</span> : <span class="number">1</span>,</span><br><span class="line"> <span class="attr">"deltaRecordCount"</span> : <span class="number">1</span>,</span><br><span class="line"> <span class="attr">"changelogRecordCount"</span> : <span class="number">0</span>,</span><br><span class="line"> <span class="attr">"watermark"</span> : <span class="number">-9223372036854775808</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>需要提醒的是，manifest list 包含了 snapshot 的所有更改，baseManifestList 是应用在 deltaManifestList 中的更改所基于的基本文件。第一次提交将导致生成 1 个清单文件，并创建了 2 个清单列表（文件名可能与你的实验中的不同）：</p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">➜ T </span><br><span class="line">manifest-232271e3-f294-4556-8947-ee87483c3bfd-0</span><br><span class="line">manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-0</span><br><span class="line">manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-1</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140530.png" alt=""></p><p><code>manifest-232271e3-f294-4556-8947-ee87483c3bfd-0</code>: 如前面文件布局图中的 <code>manifest-1-0</code>，存储了关于 snapshot 中数据文件信息的 manifest。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140554.png" alt=""></p><p><code>manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-0</code>: 是基础的 baseManifestList，如前面文件布局图中的 <code>manifest-list-1-base</code>，实际上是空的。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140612.png" alt=""></p><p><code>manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-1</code>：是deltaManifestList，如前面文件布局图中的 <code>manifest-list-1-delta</code>，其中包含一系列对数据文件执行操作的清单条目，而在这种情况下，清单条目为 <code>manifest-1-0</code>。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140624.png" alt=""></p><p>在不同分区中插入一批记录，在 Flink SQL 中执行以下语句：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> T <span class="keyword">VALUES</span> </span><br><span class="line">(<span class="number">2</span>, <span class="number">10002</span>, <span class="string">'varchar00002'</span>, <span class="string">'20230502'</span>),</span><br><span class="line">(<span class="number">3</span>, <span class="number">10003</span>, <span class="string">'varchar00003'</span>, <span class="string">'20230503'</span>),</span><br><span class="line">(<span class="number">4</span>, <span class="number">10004</span>, <span class="string">'varchar00004'</span>, <span class="string">'20230504'</span>),</span><br><span class="line">(<span class="number">5</span>, <span class="number">10005</span>, <span class="string">'varchar00005'</span>, <span class="string">'20230505'</span>),</span><br><span class="line">(<span class="number">6</span>, <span class="number">10006</span>, <span class="string">'varchar00006'</span>, <span class="string">'20230506'</span>),</span><br><span class="line">(<span class="number">7</span>, <span class="number">10007</span>, <span class="string">'varchar00007'</span>, <span class="string">'20230507'</span>),</span><br><span class="line">(<span class="number">8</span>, <span class="number">10008</span>, <span class="string">'varchar00008'</span>, <span class="string">'20230508'</span>),</span><br><span class="line">(<span class="number">9</span>, <span class="number">10009</span>, <span class="string">'varchar00009'</span>, <span class="string">'20230509'</span>),</span><br><span class="line">(<span class="number">10</span>, <span class="number">10010</span>, <span class="string">'varchar00010'</span>, <span class="string">'20230510'</span>);</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140641.png" alt=""></p><p>等任务执行完成并提交快照后，执行 <code>SELECT * FROM T</code> 将返回 10 行数据。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140653.png" alt=""></p><p>创建了一个新的快照，即 <code>snapshot-2</code>，并给出了以下物理文件布局：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140706.png" alt=""></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">dt=20230501</span><br><span class="line">dt=20230502</span><br><span class="line">dt=20230503</span><br><span class="line">dt=20230504</span><br><span class="line">dt=20230505</span><br><span class="line">dt=20230506</span><br><span class="line">dt=20230507</span><br><span class="line">dt=20230508</span><br><span class="line">dt=20230509</span><br><span class="line">dt=20230510</span><br><span class="line">manifest</span><br><span class="line">schema</span><br><span class="line">snapshot</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">➜ T ll snapshot</span><br><span class="line">EARLIEST</span><br><span class="line">LATEST</span><br><span class="line">snapshot-1</span><br><span class="line">snapshot-2</span><br><span class="line"></span><br><span class="line">➜ T ll manifest</span><br><span class="line">manifest-232271e3-f294-4556-8947-ee87483c3bfd-0 //snapshot-1 manifest file</span><br><span class="line">manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-0 //snapshot-1 baseManifestList</span><br><span class="line">manifest-list-989f0744-4419-48dd-b0fd-955343e6e803-1 //snapshot-1 deltaManifestList</span><br><span class="line"></span><br><span class="line">manifest-ed94ba0c-e71f-4a01-863c-479b34af2551-0 //snapshot-2 manifest file</span><br><span class="line">manifest-list-03bad670-05ae-48b2-b88c-de631ad14333-0 //snapshot-2 baseManifestList</span><br><span class="line">manifest-list-03bad670-05ae-48b2-b88c-de631ad14333-1 ////snapshot-2 baseManifestList</span><br></pre></td></tr></table></figure><p>新的快照文件布局图如下：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140726.png" alt=""> </p><h3 id="删除数据"><a href="#删除数据" class="headerlink" title="删除数据"></a>删除数据</h3><p>接下来删除满足条件 <code>dt>=20230503</code> 的数据。在 Flink SQL Client 中，执行以下语句：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">DELETE</span> <span class="keyword">FROM</span> T <span class="keyword">WHERE</span> dt >= <span class="string">'20230503'</span>;</span><br></pre></td></tr></table></figure><p>注意⚠️：</p><p>1、需使用 Flink 1.17 及以上版本，否则不支持该<a href="https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/delete/">语法</a></p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Flink SQL> DELETE FROM T WHERE dt >= '20230503';</span><br><span class="line">></span><br><span class="line">[ERROR] Could not <span class="keyword">execute</span> <span class="keyword">SQL</span> statement. Reason:</span><br><span class="line">org.apache.flink.table.api.TableException: Unsupported <span class="keyword">query</span>: <span class="keyword">DELETE</span> <span class="keyword">FROM</span> T <span class="keyword">WHERE</span> dt >= <span class="string">'20230503'</span>;</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140743.png" alt=""></p><p>2、使用 batch 模式，否则报错不支持</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">Flink SQL> DELETE FROM T WHERE dt >= '20230503';</span><br><span class="line">[ERROR] Could not <span class="keyword">execute</span> <span class="keyword">SQL</span> statement. Reason:</span><br><span class="line">org.apache.flink.table.api.TableException: <span class="keyword">DELETE</span> <span class="keyword">statement</span> <span class="keyword">is</span> <span class="keyword">not</span> supported <span class="keyword">for</span> streaming <span class="keyword">mode</span> now.</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140800.png" alt=""></p><p>需要设置 <code>SET 'execution.runtime-mode' = 'batch';</code></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140815.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140828.png" alt=""></p><p>第三次提交完成，并生成了快照 <code>snapshot-3</code>。现在，表下的目录，会发现没有分区被删除。相反，为分区 20230503 到 20230510 创建了一个新的数据文件：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140846.png" alt=""></p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">➜ T ll dt=20230510/bucket-0</span><br><span class="line"></span><br><span class="line">data-0531fa1e-6ff1-47ed-aeea-3a01896c9698-0.orc # newer data file created by the delete statement </span><br><span class="line">data-0c8a6a16-13c5-4049-b4ed-e968d34e20ae-0.orc # older data file created by the insert statement</span><br></pre></td></tr></table></figure><p>这是有道理的，因为我们在第二次提交中插入了一条数据（为 <code>+I[10, 10010, 'varchar00010', '20230510']</code> ），然后在第三次提交中删除了数据。现在再执行一下 <code>SELECT * FROM T</code>只能查询到两条数据。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">+I[1, 10001, 'varchar00001', '20230501']</span><br><span class="line">+I[2, 10002, 'varchar00002', '20230502']</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140902.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140939.png" alt=""></p><p><code>snapshot-3</code>后新的文件布局如下图：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140955.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141009.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141021.png" alt=""></p><p>请注意，<code>manifest-3-0</code> 包含了 8 个 ADD 操作类型的清单条目，对应着 8 个新写入的数据文件。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141042.png" alt=""></p><h3 id="Compact-Table"><a href="#Compact-Table" class="headerlink" title="Compact Table"></a>Compact Table</h3><p>你可能已经注意到的，随着连续 snapshot 的增加，小文件的数量会增加，这可能会导致读取性能下降。因此，需要进行全量压缩以减少小文件的数量。</p><p>通过 flink run 运行一个<a href="https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/0.6-SNAPSHOT/">专用的合并小文件任务</a>：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><FLINK_HOME>/bin/flink run \</span><br><span class="line"> -D execution.runtime-mode=batch \</span><br><span class="line"> ./paimon-flink-action-0.6-SNAPSHOT.jar \</span><br><span class="line"> compact \</span><br><span class="line"> <span class="comment">--warehouse <warehouse-path> \</span></span><br><span class="line"> <span class="comment">--database <database-name> \ </span></span><br><span class="line"> <span class="comment">--table <table-name> \</span></span><br><span class="line"> [<span class="comment">--partition <partition-name>] \</span></span><br><span class="line"> [<span class="comment">--catalog-conf <paimon-catalog-conf> [--catalog-conf <paimon-catalog-conf> ...]] \</span></span><br><span class="line"> [<span class="comment">--table-conf <paimon-table-dynamic-conf> [--table-conf <paimon-table-dynamic-conf>] ...]</span></span><br></pre></td></tr></table></figure><p>例如（提前下载好压缩任务 jar 在 Flink 客户端下）：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">./bin/flink run \</span><br><span class="line"> -D execution.runtime-mode=batch \</span><br><span class="line"> ./paimon-flink-action-0.6-20231012.001913-36.jar \</span><br><span class="line"> compact \</span><br><span class="line"> <span class="comment">--path file:///tmp/paimon/default.db/T</span></span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141106.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141120.png" alt=""></p><p>压缩任务结束后，再来看下 snapshot-4 文件结构：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"> <span class="attr">"version"</span> : <span class="number">3</span>,</span><br><span class="line"> <span class="attr">"id"</span> : <span class="number">4</span>,</span><br><span class="line"> <span class="attr">"schemaId"</span> : <span class="number">0</span>,</span><br><span class="line"> <span class="attr">"baseManifestList"</span> : <span class="string">"manifest-list-00af18ef-486d-4046-be60-25533e078333-0"</span>,</span><br><span class="line"> <span class="attr">"deltaManifestList"</span> : <span class="string">"manifest-list-00af18ef-486d-4046-be60-25533e078333-1"</span>,</span><br><span class="line"> <span class="attr">"changelogManifestList"</span> : <span class="literal">null</span>,</span><br><span class="line"> <span class="attr">"commitUser"</span> : <span class="string">"ab094a14-17e0-4215-8e79-cd0651436dee"</span>,</span><br><span class="line"> <span class="attr">"commitIdentifier"</span> : <span class="number">9223372036854775807</span>,</span><br><span class="line"> <span class="attr">"commitKind"</span> : <span class="string">"COMPACT"</span>,</span><br><span class="line"> <span class="attr">"timeMillis"</span> : <span class="number">1697102418177</span>,</span><br><span class="line"> <span class="attr">"logOffsets"</span> : { },</span><br><span class="line"> <span class="attr">"totalRecordCount"</span> : <span class="number">2</span>,</span><br><span class="line"> <span class="attr">"deltaRecordCount"</span> : <span class="number">-16</span>,</span><br><span class="line"> <span class="attr">"changelogRecordCount"</span> : <span class="number">0</span>,</span><br><span class="line"> <span class="attr">"watermark"</span> : <span class="number">-9223372036854775808</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141138.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141154.png" alt=""></p><p><code>manifest-4-0</code> 包含 20 个清单条目（18 个 DELETE 操作和 2 个 ADD 操作）:</p><ul><li>对于分区 20230503 到 20230510，有两个删除操作，对应两个数据文件</li><li>对于分区 20230501 到 20230502，有一个删除操作和一个添加操作，对应同一个数据文件。</li></ul><h3 id="Alter-Table"><a href="#Alter-Table" class="headerlink" title="Alter Table"></a>Alter Table</h3><p>执行以下语句来配置全量压缩：<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">ALTER</span> <span class="keyword">TABLE</span> T <span class="keyword">SET</span> (<span class="string">'full-compaction.delta-commits'</span> = <span class="string">'1'</span>);</span><br></pre></td></tr></table></figure></p><p>这将为 Paimon 表创建一个新的 schema，即 schema-1，但在下一次提交之前，不会有任何 snapshot 使用此模式。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141211.png" alt=""></p><h3 id="过期-snapshot"><a href="#过期-snapshot" class="headerlink" title="过期 snapshot"></a>过期 snapshot</h3><p>请注意，标记为删除的数据文件直到 snapshot 过期且没有任何消费者依赖于该 snapshot 时才会真正被删除。参考 <a href="https://paimon.apache.org/docs/master/maintenance/manage-snapshots/#expiring-snapshots">Manage Snapshots</a> 可以查阅更多信息。</p><p>在 snapshot 过期过程中，首先确定 snapshot 的范围，然后标记这些 snapshot 内的数据文件以进行删除。只有当存在引用特定数据文件的 DELETE 类型的清单条目时，才会标记该数据文件进行删除。这种标记确保文件不会被后续的 snapshot 使用，并且可以安全地删除。</p><p>假设上图中的所有 4 个 snapshot 即将过期。过期过程如下：</p><p>1、首先删除所有标记为删除的数据文件，并记录任何更改的 bucket。<br>2、然后删除所有的 changelog 文件和关联的 manifests。<br>3、最后，删除快照本身并写入最早的提示文件。</p><p>如果删除过程后留下的空目录，也将被删除。</p><p>假设创建了另一个快照 snapshot-5，并触发了快照过期。将删除 snapshot-1 到 snapshot-4。为简单起见，我们只关注以前快照的文件，快照过期后的最终布局如下：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141226.png" alt=""></p><p>因此，分区 20230503 到 20230510 的数据被物理删除了。</p><h3 id="Flink-流式写入"><a href="#Flink-流式写入" class="headerlink" title="Flink 流式写入"></a>Flink 流式写入</h3><p>我们通过利用 CDC 数据的示例来测试 Flink 流式写入，将介绍将变更数据捕获并写入 Paimon 的过程，以及异步压缩、快照提交和过期的机制背后的原理。帮我们更详细地了解 CDC 数据摄取的工作流程以及每个参与组件所扮演的独特角色。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141247.png" alt=""></p><p>1、MySQL CDC Source 统一读取快照数据和增量数据，其中 SnapshotReader 读取快照数据，而 BinlogReader 读取增量数据。<br>2、Paimon Sink 将数据按 Bucket 级别写入 Paimon 表中。其中的 CompactManager 将异步触发压缩操作。<br>3、Committer Operator 是一个单例，负责提交和过期快照。</p><p>接下来，我们将逐步介绍端到端的数据流程：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141300.png" alt=""></p><p>MySQL CDC Source 首先读取快照数据和增量数据，然后对它们进行规范化处理，并将其发送到下游。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141315.png" alt=""></p><p>Paimon Sink 首先将新记录缓存在基于堆的 LSM 树中，并在内存缓冲区满时将其 flush 到磁盘上。请注意，每个写入的数据文件都是一个 sorted run。在这个阶段，还没有创建 manifest 文件和 snapshot。在 Flink 执行 Checkpoint 之前，Paimon Sink 将 flush 所有缓冲的记录并发送可提交的消息到下游，下游会在 Checkpoint 期间由 Committer Operator 读取并提交。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141327.png" alt=""></p><p>在 Checkpoint 期间，Committer Operator 将创建一个新的 snapshot，并将其与 manifest lists 关联，以便snapshot 包含表中所有数据文件的信息。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-141341.png" alt=""></p><p>稍等一会后，可能会进行异步压缩，CompactManager 生成的可提交消息包含有关先前文件和合并文件的信息，以便 Committer Operator 可以构建相应的 manifest entries。在这种情况下，Committer Operator 在Flink Checkpoint 期间可能会生成两个 snapshot，一个用于写入的数据（类型为 Append 的快照），另一个用于压缩（类型为 Compact 的快照）。如果在 Checkpoint 间隔期间没有写入数据文件，则只会创建类型为Compact 的快照。Committer Operator 将检查 snapshot 的过期情况，并对标记为删除的数据文件执行物理删除操作。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<p>本文旨在澄清不同文件操作对文件的影响。</p>
<p>本页面提供具体示例和实用技巧，以有效地管理这些操作。此外，通过对提交（commit）和压实（compact）等操作的深入探讨，我们旨在提供有关文件创建和更新的见解。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Apache Paimon" scheme="http://www.54tianzhisheng.cn/tags/Apache-Paimon/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="数据湖" scheme="http://www.54tianzhisheng.cn/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/"/>
</entry>
<entry>
<title>Apache Paimon 文件布局设计</title>
<link href="http://www.54tianzhisheng.cn/2023/12/24/apache_paimon_file_design/"/>
<id>http://www.54tianzhisheng.cn/2023/12/24/apache_paimon_file_design/</id>
<published>2023-12-23T16:00:00.000Z</published>
<updated>2024-02-21T14:03:21.982Z</updated>
<content type="html"><![CDATA[<p>一张表的所有文件都存储在一个基本目录下，Paimon 文件以分层方式组织。从快照文件开始，可以递归地访问表中的所有记录。</p><a id="more"></a><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140247.png" alt=""></p><h2 id="Snapshot-Files"><a href="#Snapshot-Files" class="headerlink" title="Snapshot Files"></a>Snapshot Files</h2><p>所有的 snapshot 文件都存储在 snapshot 目录下，snapshot file 是一个包含了 snapshot 信息的 JSON 文件：</p><ul><li>使用的 Schema 文件</li><li>manifest 列表包含了 snapshot 的所有变更</li></ul><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Snapshot</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Integer version;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> id;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> schemaId;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// a manifest list recording all changes from the previous snapshots</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String baseManifestList;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// a manifest list recording all new changes occurred in this snapshot</span></span><br><span class="line"> <span class="comment">// for faster expire and streaming reads</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String deltaManifestList;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// a manifest list recording all changelog produced in this snapshot</span></span><br><span class="line"> <span class="comment">// null if no changelog is produced, or for paimon <= 0.2</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String changelogManifestList;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// a manifest recording all index files of this table</span></span><br><span class="line"> <span class="comment">// null if no index file</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String indexManifest;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String commitUser;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Mainly for snapshot deduplication.</span></span><br><span class="line"> <span class="comment">//</span></span><br><span class="line"> <span class="comment">// If multiple snapshots have the same commitIdentifier, reading from any of these snapshots</span></span><br><span class="line"> <span class="comment">// must produce the same table.</span></span><br><span class="line"> <span class="comment">//</span></span><br><span class="line"> <span class="comment">// If snapshot A has a smaller commitIdentifier than snapshot B, then snapshot A must be</span></span><br><span class="line"> <span class="comment">// committed before snapshot B, and thus snapshot A must contain older records than snapshot B.</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> commitIdentifier;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> CommitKind commitKind;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> timeMillis;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Map<Integer, Long> logOffsets;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// record count of all changes occurred in this snapshot</span></span><br><span class="line"> <span class="comment">// null for paimon <= 0.3</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Long totalRecordCount;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// record count of all new changes occurred in this snapshot</span></span><br><span class="line"> <span class="comment">// null for paimon <= 0.3</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Long deltaRecordCount;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// record count of all changelog produced in this snapshot</span></span><br><span class="line"> <span class="comment">// null for paimon <= 0.3</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Long changelogRecordCount;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// watermark for input records</span></span><br><span class="line"> <span class="comment">// null for paimon <= 0.3</span></span><br><span class="line"> <span class="comment">// null if there is no watermark in new committing, and the previous snapshot does not have a</span></span><br><span class="line"> <span class="comment">// watermark</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Long watermark;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="Manifest-Files"><a href="#Manifest-Files" class="headerlink" title="Manifest Files"></a>Manifest Files</h2><p>所有的 manifest lists 和 manifest 文件都存放在 manifest 目录下，manifest list 是一组 manifest 文件名列表。manifest 文件是一个包含有关 LSM 数据文件和变更日志文件的变更信息的文件。例如，它记录了在对应的快照中创建了哪个 LSM 数据文件以及删除了哪个文件。</p><p>Schema：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Schema</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> List<DataField> fields;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> List<String> partitionKeys;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> List<String> primaryKeys;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Map<String, String> options;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String comment;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>FileKind:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">enum</span> FileKind {</span><br><span class="line"> ADD((<span class="keyword">byte</span>) <span class="number">0</span>),</span><br><span class="line"> DELETE((<span class="keyword">byte</span>) <span class="number">1</span>);</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>IndexFileMeta:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">IndexFileMeta</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String indexType;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String fileName;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> fileSize;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> rowCount;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>IndexManifestEntry:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">IndexManifestEntry</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> FileKind kind;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> BinaryRow partition;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">int</span> bucket;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> IndexFileMeta indexFile;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>ManifestFileMeta:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ManifestFileMeta</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String fileName;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> fileSize;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> numAddedFiles;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> numDeletedFiles;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> BinaryTableStats partitionStats;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> schemaId;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>ManifestCommittable:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ManifestCommittable</span> </span>{ <span class="comment">///Manifest commit message</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> identifier;</span><br><span class="line"> <span class="meta">@Nullable</span> <span class="keyword">private</span> <span class="keyword">final</span> Long watermark;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Map<Integer, Long> logOffsets;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> List<CommitMessage> commitMessages;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>ManifestFile:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * This file includes several ManifestEntry, representing the additional changes since last snapshot.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ManifestFile</span> <span class="keyword">extends</span> <span class="title">ObjectsFile</span><<span class="title">ManifestEntry</span>> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> SchemaManager schemaManager;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> RowType partitionType;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> FormatWriterFactory writerFactory;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">long</span> suggestedFileSize;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 有 write 方法将各种 ManifestEntry 写进去 ManifestFile，其中会统计对应的 metadata</span></span><br></pre></td></tr></table></figure></p><p>ManifestList：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// This file includes several ManifestFileMeta, representing all data of the whole table at the corresponding snapshot.</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ManifestList</span> <span class="keyword">extends</span> <span class="title">ObjectsFile</span><<span class="title">ManifestFileMeta</span>> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">public</span> String <span class="title">write</span><span class="params">(List<ManifestFileMeta> metas)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">super</span>.writeWithoutRolling(metas);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><h2 id="Data-Files"><a href="#Data-Files" class="headerlink" title="Data Files"></a>Data Files</h2><p>数据文件按照分区和 bucket 进行分组。每个 bucket 目录包含一个 LSM 树和其对应的变更日志文件。<br>目前，Paimon 支持使用 orc（默认）、parquet 和 avro 作为数据文件的格式。</p><h2 id="LSM-Trees"><a href="#LSM-Trees" class="headerlink" title="LSM Trees"></a>LSM Trees</h2><p>Paimon 采用 LSM 树（日志结构合并树）作为文件存储的数据结构。下面简要介绍了关于 LSM 树的概念。</p><h3 id="Sorted-Runs"><a href="#Sorted-Runs" class="headerlink" title="Sorted Runs"></a>Sorted Runs</h3><p>LSM 树将文件组织成多个 sorted runs。一个 sorted run 由一个或多个数据文件组成，每个数据文件都属于且只属于一个 sorted run。<br>数据文件内的记录按其主键进行排序。在一个 sorted run 内，数据文件的主键范围不会重叠。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140305.png" alt=""></p><p>正如您所看到的，不同的 sorted run 可能具有重叠的主键范围，甚至可能包含相同的主键。在查询 LSM 树时，必须将所有的 sorted run 组合起来，并根据用户指定的合并引擎和每个记录的时间戳进行主键相同的记录合并。</p><p>写入 LSM 树的新记录将首先缓存在内存中。当内存缓冲区满时，所有内存中的记录将被排序并刷新到磁盘上。此时就会创建一个新的 sorted run。</p><h3 id="Compaction"><a href="#Compaction" class="headerlink" title="Compaction"></a>Compaction</h3><p>当越来越多的记录被写入 LSM 树时，sorted run 的数量会增加。因为查询 LSM 树需要将所有 sorted run 组合起来，过多的 sorted run 将导致查询性能下降，甚至可能导致内存不足。<br>为了限制 sorted run 的数量，我们需要定期将几个 sorted run 合并成一个大的 sorted run。这个过程被称为compaction。<br>然而，compaction 是一个资源密集型的过程，会消耗一定的 CPU 时间和磁盘 IO，因此过于频繁的 compaction 可能会导致写入速度变慢。这是查询性能和写入性能之间的权衡。Paimon 目前采用了类似Rocksdb 的 <a href="https://github.com/facebook/rocksdb/wiki/Universal-Compaction">universal compaction</a> 策略。<br>默认情况下，当 Paimon 向 LSM 树追加记录时，会根据需要进行 compaction。用户也可以选择在<a href="https://paimon.apache.org/docs/master/maintenance/dedicated-compaction/#dedicated-compaction-job">单独的 compaction 作业</a>中执行所有的 compaction 操作。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<p>一张表的所有文件都存储在一个基本目录下，Paimon 文件以分层方式组织。从快照文件开始，可以递归地访问表中的所有记录。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Apache Paimon" scheme="http://www.54tianzhisheng.cn/tags/Apache-Paimon/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="数据湖" scheme="http://www.54tianzhisheng.cn/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/"/>
</entry>
<entry>
<title>Apache Paimon 基础概念</title>
<link href="http://www.54tianzhisheng.cn/2023/12/23/apache_paimon_02/"/>
<id>http://www.54tianzhisheng.cn/2023/12/23/apache_paimon_02/</id>
<published>2023-12-22T16:00:00.000Z</published>
<updated>2024-02-21T14:02:34.690Z</updated>
<content type="html"><![CDATA[<h2 id="Snapshot"><a href="#Snapshot" class="headerlink" title="Snapshot"></a>Snapshot</h2><p>快照（Snapshot）是在某个时间点上捕捉表状态的方式。用户可以通过最新的快照访问表的最新数据。通过时间回溯，用户还可以通过较早的快照访问表的先前状态。</p><a id="more"></a><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_VERSION = <span class="string">"version"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_ID = <span class="string">"id"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_SCHEMA_ID = <span class="string">"schemaId"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_BASE_MANIFEST_LIST = <span class="string">"baseManifestList"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_DELTA_MANIFEST_LIST = <span class="string">"deltaManifestList"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_CHANGELOG_MANIFEST_LIST = <span class="string">"changelogManifestList"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_INDEX_MANIFEST = <span class="string">"indexManifest"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_COMMIT_USER = <span class="string">"commitUser"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_COMMIT_IDENTIFIER = <span class="string">"commitIdentifier"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_COMMIT_KIND = <span class="string">"commitKind"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_TIME_MILLIS = <span class="string">"timeMillis"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_LOG_OFFSETS = <span class="string">"logOffsets"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_TOTAL_RECORD_COUNT = <span class="string">"totalRecordCount"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_DELTA_RECORD_COUNT = <span class="string">"deltaRecordCount"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_CHANGELOG_RECORD_COUNT = <span class="string">"changelogRecordCount"</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String FIELD_WATERMARK = <span class="string">"watermark"</span>;</span><br></pre></td></tr></table></figure><h2 id="Partition"><a href="#Partition" class="headerlink" title="Partition"></a>Partition</h2><p>Paimon 采用与 Apache Hive 相同的分区概念来分离数据。分区是根据特定列（如日期、城市和部门）的值将表划分成相关部分的可选方式。每个表可以有一个或多个分区键来标识特定的分区。通过分区，用户可以高效地操作表中的一部分记录，请参考<a href="https://paimon.apache.org/docs/master/concepts/file-layouts/">文档</a>如何划分为多个分区。</p><blockquote><p>如果定义了主键，分区键必须是主键的子集。</p></blockquote><h2 id="Bucket"><a href="#Bucket" class="headerlink" title="Bucket"></a>Bucket</h2><p>未分区的表，或者分区表中的分区，被细分为 bucket，以为数据提供额外的结构，以便进行更高效的查询。</p><p>bucket 的范围由记录中一个或多个列的哈希值确定。用户可以通过提供<a href="https://paimon.apache.org/docs/master/maintenance/configurations/#coreoptions">存储桶键选项</a>来指定bucket 列。如果未指定 bucket 键选项，则主键（如果定义）或完整记录将被用作 bucket 键。</p><p>bucket 是读写的最小存储单元，因此 bucket 的数量限制了最大的处理并行性。然而，这个数量不应该过大，因为会导致大量的小文件和低读取性能。一般来说，每个 bucket 中推荐的数据大小约为1GB。</p><p>请参阅<a href="https://paimon.apache.org/docs/master/concepts/file-layouts/">资料</a>了解文件如何被划分为 bucket。此外，如果您想在创建表后调整 bucket 的数量，请参阅<a href="https://paimon.apache.org/docs/master/maintenance/rescale-bucket/">重新调整 bucket</a>。</p><h2 id="一致性保证"><a href="#一致性保证" class="headerlink" title="一致性保证"></a>一致性保证</h2><p>Paimon 的写操作使用两阶段提交协议来原子地将一批记录提交到表中，每次提交最多会产生两个快照。</p><p>对于同时修改同一表的任意两个写入者，只要它们不修改同一个存储桶，它们的提交可以并行进行。如果它们修改同一个存储桶，只能保证快照隔离性。也就是说，最终的表状态可能是两个提交的混合，但不会丢失任何更改。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h2 id="Snapshot"><a href="#Snapshot" class="headerlink" title="Snapshot"></a>Snapshot</h2><p>快照（Snapshot）是在某个时间点上捕捉表状态的方式。用户可以通过最新的快照访问表的最新数据。通过时间回溯，用户还可以通过较早的快照访问表的先前状态。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Apache Paimon" scheme="http://www.54tianzhisheng.cn/tags/Apache-Paimon/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="数据湖" scheme="http://www.54tianzhisheng.cn/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/"/>
</entry>
<entry>
<title>Apache Paimon 介绍</title>
<link href="http://www.54tianzhisheng.cn/2023/12/22/apache_paimon_introduction/"/>
<id>http://www.54tianzhisheng.cn/2023/12/22/apache_paimon_introduction/</id>
<published>2023-12-21T16:00:00.000Z</published>
<updated>2024-02-21T14:02:13.152Z</updated>
<content type="html"><![CDATA[<p>从 Flink Table Store 演进而来</p><a id="more"></a><h3 id="Flink-table-store"><a href="#Flink-table-store" class="headerlink" title="Flink table store"></a>Flink table store</h3><p>架构如下图：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140047.png" alt=""></p><p>（和今天 Paimon 的架构相比，Log System 不再被推荐使用，Lake Store 的能力大幅强于 Log System，除了延时）</p><p>2021 年 9 月，发布了 0.2 版本，陆续有在生产使用。</p><p>Flink Table Store 是一个数据湖存储，用于实时流式 Changelog 写入 (比如来自 Flink CDC 的数据) 和高性能查询。它创新性的结合湖存储和 LSM 结构，深度对接 Flink，提供实时更新的系统设计，支撑大吞吐量的更新数据摄取，同时提供良好的查询性能。</p><p>0.3 形成了一个 Streaming Lakehouse 的基本雏形，我们可以比较自信的说出，0.3 可以推荐生产可用了。</p><p>基于 Flink Table Store 不仅可以支持数据实时入湖，而且支持 Partial Update 等功能，帮助用户更灵活的在延迟和成本之间做均衡。</p><h3 id="Apache-Paimon"><a href="#Apache-Paimon" class="headerlink" title="Apache Paimon"></a>Apache Paimon</h3><p>在发布了三个版本后，虽然 Flink Table Store 具备了一定的成熟度，但作为 Flink 社区的一个子项目，在生态发展（比如 Spark 用户选择和使用）方面存在比较明显的局限性。为了让 Flink Table Store 能够有更大的发展空间和生态体系， Flink PMC 经过讨论决定将其捐赠 ASF 进行独立孵化。</p><p>2023 年 3 月 12 日，Flink Table Store 项目顺利通过投票，正式进入 Apache 软件基金会 (ASF) 的孵化器，改名为 Apache Paimon (incubating)。</p><p>进入孵化器后，Paimon 得到了众多的关注，包括 阿里云、字节跳动、Bilibili、汽车之家、蚂蚁 等多家公司参与到 Apache Paimon 的贡献，也得到了广大用户的使用。</p><p>Paimon 和 Flink 集成也在继续，Paimon 集成了 Flink CDC，提出了全自动的 数据 + Schema 的同步，整库同步，带来更高性能的入湖、更低的入湖成本、更方便的入湖体验。</p><p><img src="https://tva1.sinaimg.cn/large/008vOhrAly1hn1ec4w9hfj30u00av0ud.jpg" alt=""></p><p>解决问题：<br />1、Paimon 实时 CDC 入湖<br />2、Paimon 实时宽表与流读</p><p>Apache Paimon 架构：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140142.png" alt=""></p><p>Streaming Lakehouse 架构：</p><p>1、数据全链路实时流动，同时沉淀所有数据，提供 AD-HOC 查询</p><p>2、通用的离线数据实时化，流批融合的一套数仓</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2024-02-21-140157.png" alt=""></p><p>参考资料：</p><p><a href="https://mp.weixin.qq.com/s/X1Fcx_og0N6eJ0uduxA4mw">当流计算邂逅数据湖：Paimon 的前生今世</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<p>从 Flink Table Store 演进而来</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="Apache Paimon" scheme="http://www.54tianzhisheng.cn/tags/Apache-Paimon/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="数据湖" scheme="http://www.54tianzhisheng.cn/tags/%E6%95%B0%E6%8D%AE%E6%B9%96/"/>
</entry>
<entry>
<title>如何收集 Yarn/K8s 集群中的 Flink 任务日志？</title>
<link href="http://www.54tianzhisheng.cn/2023/03/17/collector-yarn-k8s-flink-logs/"/>
<id>http://www.54tianzhisheng.cn/2023/03/17/collector-yarn-k8s-flink-logs/</id>
<published>2023-03-16T16:00:00.000Z</published>
<updated>2023-03-17T15:44:37.569Z</updated>
<content type="html"><![CDATA[<p>从一个小需求（任务异常或者失败时排查问题不便）出发，开始调研业界常见的解决方案进行解决我们的需求，接着有对应的代码实现和效果展示。</p><a id="more"></a><h2 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h2><p>不管是 Flink On Yarn 还是 On k8s，如果任务正常运行，我们是可以通过 Flink Web UI 去查看 JobManager 和 TaskManager 日志，虽然日志量大的时候去不同的 TaskManager 找日志有点困难（如何快速知道日志在哪个 TaskManager 上；在 TaskManager 里面可能有多个滚动的日志文件，如何快速找到 root cause 异常；如果 TaskManager OOM 掉了该容器的日志就看不到了），但是起码给了一个可以看日志的途径。</p><p>熟悉 Flink On Yarn 的应该知道 Flink 任务运行结束/失败后，只能去 Yarn UI 看到任务的 Jobmanager 日志，对于 TaskManager 日志这些是看不到的，这对于有时候想排查下任务失败的原因日志会比较困难（不过大多数任务挂掉的原因日志都会在 Jobmanager 存在）。</p><p>熟悉 Flink On K8s 的那更能体验到查看日志的痛苦了，在任务运行失败和结束后，所有的 Pod 都会退出，如果没有收集这些运行日志，那几乎很难知道任务为啥会失败。</p><p>Flink History Server 不像 Spark History Server 一样可以看到任务所有运行的 Excutor 日志，所以对于故障定位 Flink 任务异常日志这个场景，Flink 自带的那些体验不是很友好。因此也有本文的出现，来讲述一下如何针对上面两种运行模式下 Flink 任务的日志收集，来解决我们不方便定位任务异常失败的需求。</p><p>当然了，我们收集到这些日志数据后，可以用来做异常日志告警提醒任务负责人作业异常信息（这个后面可以专门开篇文章来写），也可以收集起来存储到 ES，方便用户排查任务异常日志。</p><h2 id="方案选择"><a href="#方案选择" class="headerlink" title="方案选择"></a>方案选择</h2><p>常见的收集日志方案有下面两种：</p><p>1、统一 LogAgent 收集。不管是使用 Flink On Yarn 还是 Flink On K8s，日志都可以配置一个路径（路径有规则），然后每台计算节点机器专门部署一个 LogAgent （比如有 Filebeat）去收集这些运行日志。K8s 的话会比 Yarn 的日志要收集的话稍微会复杂一些，需要 Flink 任务挂载磁盘，这样日志文件数据路径比较固定，否则日志文件是在容器 Pod 内，会随着 Pod 的生命周期而消失。这种方式需要在每台机器都部署一个专门用来收集日志的 Agent，还要额外维护它的稳定性，不然可能会漏收集到任务的日志。</p><p>2、自定义 Kafka Appender。这种方式要根据日志框架进行自定义一个 Appender，将定义好的 Appender 打包后放到 Flink lib 目录，然后配置好 log4j 配置，任务启动后会自动加载这个依赖，运行过程中会自动实时将日志发送到 Kafka。这个 Appender 定义可以比较灵活（具体的可以看下文的代码实现），比如加入一些过滤条件：只收集 warn 级别以上的日志（因为任务多了的话收集所有的级别日志数据量会很大，但是对排查问题带来的作用有限）。这种方式和任务运行在 Yarn 和 K8s 无关，都可以正常收集日志，不用单独配置，也不用单独去维护什么组件的稳定性，唯一的缺点就是对已经在运行的任务如果想要收集日志需要重启一下即可，相比来说我个人觉得还是这种方式会比较合适。</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-154054.jpg" alt=""></p><p>整个架构如上，你理解图中的 Reporter 包含三个：第一个是自定义的 Kafka Appender，第二个是自定义的 Kafka Metrics Reporter，第三个是根据官方的 Prometheus PushGateway Metrics Reporter 做了内部改造的。前两个是本篇要讲解的，后两个后面也可以单独再开文章来讲。</p><h2 id="自定义-Kafka-Appender"><a href="#自定义-Kafka-Appender" class="headerlink" title="自定义 Kafka Appender"></a>自定义 Kafka Appender</h2><h3 id="需求"><a href="#需求" class="headerlink" title="需求"></a>需求</h3><p>自定义 Kafka Appender，将日志数据发到 kafka，但是如何对日志数据进行标识，需要利用 log.file 环境变量来获取作业的 application id、container id、host、jobmanager/taskmanager 信息等。</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-154801.jpg" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-154823.jpg" alt=""></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">2021-07-27 20:39:55,777 INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -Dout.file=/data/HDATA/yarn/logs/application_1596708697506_12674/container_e11_1596708697506_12674_01_000005/taskmanager.out</span><br><span class="line">2021-07-27 20:39:55,777 INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -Dlog.file=/data/HDATA/yarn/logs/application_1596708697506_12674/container_e11_1596708697506_12674_01_000005/taskmanager.log</span><br><span class="line">2021-07-27 20:39:55,777 INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -Derr.file=/data/HDATA/yarn/logs/application_1596708697506_12674/container_e11_1596708697506_12674_01_000005/taskmanager.err</span><br></pre></td></tr></table></figure><p>上面这个是针对 Yarn 可以根据 log.file 这个环境变量拿到作业的一些维度数据，其实 K8s 也可以在环境变量中拿到这些想要的信息，可能需要改造点 Flink-K8s 模块代码（其实就是添加一点容器运行环境变量，比如 K8s 中的任务 ClusterId、运行的物理机器 IP、Pod IP），如下图所示：</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-155714.png" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-155831.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p><h3 id="修改框架源码"><a href="#修改框架源码" class="headerlink" title="修改框架源码"></a>修改框架源码</h3><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160246.jpg" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160300.jpg" alt=""></p><p>在 1.10 和 1.12 中，Flink 作业在发生状态转换成 FAILED 时，作业打印出来的日志级别竟然是 info，而这种日志是作业异常的根因，对于我们要收集作业的异常日志，那么就得更改源码，提高日志级别。</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160327.jpg" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160343.jpg" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160356.jpg" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160409.jpg" alt=""></p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160420.jpg" alt=""></p><p>提高这些地方的日志级别后，即可收集作业出现 failed 或者 restarting 时的异常日志，从而可以方便用户定位作业异常的原因（当然了，上面这些属于锦上添花修改了，如果不改其实有办法能够在自定义收集器里面去收集到这些数据）。</p><p>因为早期开发的时候公司内部 Flink 1.10 和 1.12 两个版本共存，1.10 使用的是log4j，1.12 使用的是 log4j2，两个版本不一致所以自定义 Kafka Appender 也有点区别，所以分别开发了两个版本，这里也都讲一下，请挑选自己需要的阅读。</p><p>我已经将代码上传至 Github 了，感兴趣可以参考一下：<a href="https://github.com/zhisheng17/flink-learning/tree/master/flink-learning-extends/FlinkLogKafkaAppender">https://github.com/zhisheng17/flink-learning/tree/master/flink-learning-extends/FlinkLogKafkaAppender</a> 如果对你有帮助的话可以帮忙点个 star。代码结构如下图所示：</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-160711.png" alt=""></p><p>FlinkLogKafkaAppender 父模块引入了基础需要的依赖：</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">project</span> <span class="attr">xmlns</span>=<span class="string">"http://maven.apache.org/POM/4.0.0"</span> <span class="attr">xmlns:xsi</span>=<span class="string">"http://www.w3.org/2001/XMLSchema-instance"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xsi:schemaLocation</span>=<span class="string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">parent</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-learning-extends<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.zhisheng.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.0-SNAPSHOT<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">parent</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">modelVersion</span>></span>4.0.0<span class="tag"></<span class="name">modelVersion</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>FlinkLogKafkaAppender<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">packaging</span>></span>pom<span class="tag"></<span class="name">packaging</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>FlinkLogKafkaAppender<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">modules</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">module</span>></span>Log4jKafkaAppender<span class="tag"></<span class="name">module</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">module</span>></span>Log4j2KafkaAppender<span class="tag"></<span class="name">module</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">module</span>></span>KafkaAppenderCommon<span class="tag"></<span class="name">module</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">modules</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">maven.compiler.source</span>></span>8<span class="tag"></<span class="name">maven.compiler.source</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">maven.compiler.target</span>></span>8<span class="tag"></<span class="name">maven.compiler.target</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">kafka.version</span>></span>2.4.1<span class="tag"></<span class="name">kafka.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>provided<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">jackson.version</span>></span>2.11.0<span class="tag"></<span class="name">jackson.version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">properties</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependencies</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.kafka<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>kafka-clients<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${kafka.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.projectlombok<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>lombok<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.18.6<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>provided<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependencies</span>></span></span><br><span class="line"><span class="tag"></<span class="name">project</span>></span></span><br></pre></td></tr></table></figure><ul><li>KafkaAppenderCommon 模块：定义了日志类和工具类</li><li>Log4j2KafkaAppender 模块：针对使用了 log4j2 的 Flink 版本</li><li>Log4jKafkaAppender 模块：针对使用了 log4j 的 Flink 版本</li></ul><h3 id="KafkaAppenderCommon"><a href="#KafkaAppenderCommon" class="headerlink" title="KafkaAppenderCommon"></a>KafkaAppenderCommon</h3><p>定义的要发送到 Kafka 的数据结构类：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">LogEvent</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String source; <span class="comment">// default is flink, maybe others will use this kafka appender in future</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String id; <span class="comment">// log id, default it is UUID</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> Long timestamp;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String content; <span class="comment">// log message</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> Map<String, String> tags = <span class="keyword">new</span> HashMap<>(); <span class="comment">// tags of the log, eg: host_name, application_id, job_name etc</span></span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>序列化工具类：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">JacksonUtil</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">static</span> ObjectMapper mapper = <span class="keyword">new</span> ObjectMapper();</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 将对象转换成普通的 JSON 数据</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> value</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span></span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> JsonProcessingException</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">toJson</span><span class="params">(Object value)</span> <span class="keyword">throws</span> JsonProcessingException </span>{</span><br><span class="line"> <span class="keyword">return</span> mapper.writeValueAsString(value);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 将对象转换成结构化的 JSON 数据</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> value</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span></span></span><br><span class="line"><span class="comment"> * <span class="doctag">@throws</span> JsonProcessingException</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">toFormatJson</span><span class="params">(Object value)</span> <span class="keyword">throws</span> JsonProcessingException </span>{</span><br><span class="line"> <span class="keyword">return</span> mapper.writerWithDefaultPrettyPrinter().writeValueAsString(value);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>对异常日志数据处理的工具类：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">ExceptionUtil</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">char</span> TAB = <span class="string">''</span>;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">char</span> CR = <span class="string">'\r'</span>;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="keyword">char</span> LF = <span class="string">'\n'</span>;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String SPACE = <span class="string">" "</span>;</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String EMPTY = <span class="string">""</span>;</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 堆栈转为单行完整字符串</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> throwable 异常对象</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> limit 限制最大长度</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> 堆栈转为的字符串</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">stacktraceToOneLineString</span><span class="params">(Throwable throwable, <span class="keyword">int</span> limit)</span> </span>{</span><br><span class="line"> Map<Character, String> replaceCharToStrMap = <span class="keyword">new</span> HashMap<>();</span><br><span class="line"> replaceCharToStrMap.put(CR, SPACE);</span><br><span class="line"> replaceCharToStrMap.put(LF, SPACE);</span><br><span class="line"> replaceCharToStrMap.put(TAB, SPACE);</span><br><span class="line"> <span class="keyword">return</span> stacktraceToString(throwable, limit, replaceCharToStrMap);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">stacktraceToString</span><span class="params">(Throwable throwable)</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> OutputStream baos = <span class="keyword">new</span> ByteArrayOutputStream();</span><br><span class="line"> throwable.printStackTrace(<span class="keyword">new</span> PrintStream(baos));</span><br><span class="line"> <span class="keyword">return</span> baos.toString();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 堆栈转为完整字符串</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> throwable 异常对象</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> limit 限制最大长度</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> replaceCharToStrMap 替换字符为指定字符串</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> 堆栈转为的字符串</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">static</span> String <span class="title">stacktraceToString</span><span class="params">(Throwable throwable, <span class="keyword">int</span> limit, Map<Character, String> replaceCharToStrMap)</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> OutputStream baos = <span class="keyword">new</span> ByteArrayOutputStream();</span><br><span class="line"> throwable.printStackTrace(<span class="keyword">new</span> PrintStream(baos));</span><br><span class="line"> String exceptionStr = baos.toString();</span><br><span class="line"> <span class="keyword">int</span> length = exceptionStr.length();</span><br><span class="line"> <span class="keyword">if</span> (limit > <span class="number">0</span> && limit < length) {</span><br><span class="line"> length = limit;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (!replaceCharToStrMap.isEmpty()) {</span><br><span class="line"> <span class="keyword">final</span> StringBuilder sb = <span class="keyword">new</span> StringBuilder();</span><br><span class="line"> <span class="keyword">char</span> c;</span><br><span class="line"> String value;</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < length; i++) {</span><br><span class="line"> c = exceptionStr.charAt(i);</span><br><span class="line"> value = replaceCharToStrMap.get(c);</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">null</span> != value) {</span><br><span class="line"> sb.append(value);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> sb.append(c);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> sb.toString();</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">return</span> sub(exceptionStr, <span class="number">0</span>, limit);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 改进JDK subString<br></span></span><br><span class="line"><span class="comment"> * index从0开始计算，最后一个字符为-1<br></span></span><br><span class="line"><span class="comment"> * 如果from和to位置一样，返回 "" <br></span></span><br><span class="line"><span class="comment"> * 如果from或to为负数，则按照length从后向前数位置，如果绝对值大于字符串长度，则from归到0，to归到length<br></span></span><br><span class="line"><span class="comment"> * 如果经过修正的index中from大于to，则互换from和to example: <br></span></span><br><span class="line"><span class="comment"> * abcdefgh 2 3 =》 c <br></span></span><br><span class="line"><span class="comment"> * abcdefgh 2 -3 =》 cde <br></span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> str String</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> fromIndex 开始的index（包括）</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> toIndex 结束的index（不包括）</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> 字串</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">static</span> String <span class="title">sub</span><span class="params">(CharSequence str, <span class="keyword">int</span> fromIndex, <span class="keyword">int</span> toIndex)</span> </span>{</span><br><span class="line"> <span class="keyword">if</span> (isEmpty(str)) {</span><br><span class="line"> <span class="keyword">return</span> str(str);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">int</span> len = str.length();</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (fromIndex < <span class="number">0</span>) {</span><br><span class="line"> fromIndex = len + fromIndex;</span><br><span class="line"> <span class="keyword">if</span> (fromIndex < <span class="number">0</span>) {</span><br><span class="line"> fromIndex = <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (fromIndex > len) {</span><br><span class="line"> fromIndex = len;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (toIndex < <span class="number">0</span>) {</span><br><span class="line"> toIndex = len + toIndex;</span><br><span class="line"> <span class="keyword">if</span> (toIndex < <span class="number">0</span>) {</span><br><span class="line"> toIndex = len;</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (toIndex > len) {</span><br><span class="line"> toIndex = len;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (toIndex < fromIndex) {</span><br><span class="line"> <span class="keyword">int</span> tmp = fromIndex;</span><br><span class="line"> fromIndex = toIndex;</span><br><span class="line"> toIndex = tmp;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (fromIndex == toIndex) {</span><br><span class="line"> <span class="keyword">return</span> EMPTY;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> str.toString().substring(fromIndex, toIndex);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * {<span class="doctag">@link</span> CharSequence} 转为字符串，null安全</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> cs {<span class="doctag">@link</span> CharSequence}</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> 字符串</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">static</span> String <span class="title">str</span><span class="params">(CharSequence cs)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">null</span> == cs ? <span class="keyword">null</span> : cs.toString();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 字符串是否为空，空的定义如下:<br></span></span><br><span class="line"><span class="comment"> * 1、为null <br></span></span><br><span class="line"><span class="comment"> * 2、为""<br></span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> str 被检测的字符串</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> 是否为空</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="function"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">boolean</span> <span class="title">isEmpty</span><span class="params">(CharSequence str)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> str == <span class="keyword">null</span> || str.length() == <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="自定义-Log4j2-Kafka-Appender"><a href="#自定义-Log4j2-Kafka-Appender" class="headerlink" title="自定义 Log4j2 Kafka Appender"></a>自定义 Log4j2 Kafka Appender</h3><p><strong>引入依赖</strong>：</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br></pre></td><td class="code"><pre><span class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-8"</span><span class="meta">?></span></span></span><br><span class="line"><span class="tag"><<span class="name">project</span> <span class="attr">xmlns</span>=<span class="string">"http://maven.apache.org/POM/4.0.0"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xmlns:xsi</span>=<span class="string">"http://www.w3.org/2001/XMLSchema-instance"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xsi:schemaLocation</span>=<span class="string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">parent</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>LogKafkaAppender<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.example<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.0<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">parent</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">modelVersion</span>></span>4.0.0<span class="tag"></<span class="name">modelVersion</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>Log4j2KafkaAppender<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">log4j.version</span>></span>2.12.1<span class="tag"></<span class="name">log4j.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">flink.shaded.version</span>></span>12.0<span class="tag"></<span class="name">flink.shaded.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">jackson.version</span>></span>2.10.1<span class="tag"></<span class="name">jackson.version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">properties</span>></span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependencies</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.logging.log4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>log4j-slf4j-impl<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${log4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>${scope}<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.logging.log4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>log4j-api<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${log4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>${scope}<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.logging.log4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>log4j-core<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${log4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>${scope}<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-shaded-jackson<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${jackson.version}-${flink.shaded.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>${scope}<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependencies</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">build</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">plugins</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.maven.plugins<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>maven-shade-plugin<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">executions</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">execution</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">id</span>></span>shade-flink<span class="tag"></<span class="name">id</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">phase</span>></span>package<span class="tag"></<span class="name">phase</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">goal</span>></span>shade<span class="tag"></<span class="name">goal</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactSet</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">includes</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">include</span>></span>org.apache.kafka:*<span class="tag"></<span class="name">include</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">includes</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">artifactSet</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">execution</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">executions</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugins</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">build</span>></span></span><br><span class="line"><span class="tag"></<span class="name">project</span>></span></span><br></pre></td></tr></table></figure><p><strong>自定义的 Kafka Appender 类</strong>：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> com.zhisheng.log.appender;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> com.zhisheng.flink.model.LogEvent;</span><br><span class="line"><span class="keyword">import</span> com.zhisheng.flink.util.ExceptionUtil;</span><br><span class="line"><span class="keyword">import</span> com.zhisheng.flink.util.JacksonUtil;</span><br><span class="line"><span class="keyword">import</span> lombok.extern.slf4j.Slf4j;</span><br><span class="line"><span class="keyword">import</span> org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;</span><br><span class="line"><span class="keyword">import</span> org.apache.kafka.clients.producer.KafkaProducer;</span><br><span class="line"><span class="keyword">import</span> org.apache.kafka.clients.producer.Producer;</span><br><span class="line"><span class="keyword">import</span> org.apache.kafka.clients.producer.ProducerRecord;</span><br><span class="line"><span class="keyword">import</span> org.apache.kafka.common.config.ConfigException;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.Filter;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.Layout;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.appender.AbstractAppender;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.config.Property;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.config.plugins.Plugin;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.config.plugins.PluginAttribute;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.config.plugins.PluginElement;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.config.plugins.PluginFactory;</span><br><span class="line"><span class="keyword">import</span> org.apache.logging.log4j.core.config.plugins.validation.constraints.Required;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.File;</span><br><span class="line"><span class="keyword">import</span> java.io.Serializable;</span><br><span class="line"><span class="keyword">import</span> java.net.InetAddress;</span><br><span class="line"><span class="keyword">import</span> java.util.HashMap;</span><br><span class="line"><span class="keyword">import</span> java.util.Map;</span><br><span class="line"><span class="keyword">import</span> java.util.Properties;</span><br><span class="line"><span class="keyword">import</span> java.util.UUID;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Slf</span>4j</span><br><span class="line"><span class="meta">@Plugin</span>(name = <span class="string">"KafkaLog4j2Appender"</span>, category = <span class="string">"Core"</span>, elementType = <span class="string">"appender"</span>, printObject = <span class="keyword">true</span>)</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">KafkaLog4j2Appender</span> <span class="keyword">extends</span> <span class="title">AbstractAppender</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String source;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String topic;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String level;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> Producer<String, String> producer;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String appId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String containerId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String containerType;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String taskName;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> String taskId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String nodeIp;</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">protected</span> <span class="title">KafkaLog4j2Appender</span><span class="params">(String name, Filter filter, Layout<? extends Serializable> layout, <span class="keyword">boolean</span> ignoreExceptions, Property[] properties, String source, String bootstrapServers, String topic, String level)</span> </span>{</span><br><span class="line"> <span class="keyword">super</span>(name, filter, layout, ignoreExceptions, properties);</span><br><span class="line"> <span class="keyword">this</span>.source = source;</span><br><span class="line"> <span class="keyword">this</span>.topic = topic;</span><br><span class="line"> <span class="keyword">this</span>.level = level;</span><br><span class="line"></span><br><span class="line"> Properties envProperties = System.getProperties();</span><br><span class="line"> Map<String, String> envs = System.getenv();</span><br><span class="line"> String clusterId = envs.get(<span class="string">"CLUSTER_ID"</span>);</span><br><span class="line"> <span class="keyword">if</span> (clusterId != <span class="keyword">null</span>) {</span><br><span class="line"> <span class="comment">//k8s cluster</span></span><br><span class="line"> appId = clusterId;</span><br><span class="line"> containerId = envs.get(<span class="string">"HOSTNAME"</span>);</span><br><span class="line"> <span class="keyword">if</span> (envs.get(<span class="string">"HOSTNAME"</span>).contains(<span class="string">"taskmanager"</span>)) {</span><br><span class="line"> containerType = <span class="string">"taskmanager"</span>;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> containerType = <span class="string">"jobmanager"</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">//k8s 物理机器 ip</span></span><br><span class="line"> <span class="keyword">if</span> (envs.get(<span class="string">"_HOST_IP_ADDRESS"</span>) != <span class="keyword">null</span>) {</span><br><span class="line"> nodeIp = envs.get(<span class="string">"_HOST_IP_ADDRESS"</span>);</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">//yarn cluster</span></span><br><span class="line"> String logFile = envProperties.getProperty(<span class="string">"log.file"</span>);</span><br><span class="line"> String[] values = logFile.split(File.separator);</span><br><span class="line"> <span class="keyword">if</span> (values.length >= <span class="number">3</span>) {</span><br><span class="line"> appId = values[values.length - <span class="number">3</span>];</span><br><span class="line"> containerId = values[values.length - <span class="number">2</span>];</span><br><span class="line"> String log = values[values.length - <span class="number">1</span>];</span><br><span class="line"> <span class="keyword">if</span> (log.contains(<span class="string">"jobmanager"</span>)) {</span><br><span class="line"> containerType = <span class="string">"jobmanager"</span>;</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (log.contains(<span class="string">"taskmanager"</span>)) {</span><br><span class="line"> containerType = <span class="string">"taskmanager"</span>;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> containerType = <span class="string">"others"</span>;</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> log.error(<span class="string">"log.file Property ({}) doesn't contains yarn application id or container id"</span>, logFile);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> taskName = envProperties.getProperty(<span class="string">"taskName"</span>, <span class="keyword">null</span>);</span><br><span class="line"> taskId = envProperties.getProperty(<span class="string">"taskId"</span>, <span class="keyword">null</span>);</span><br><span class="line"></span><br><span class="line"> Properties props = <span class="keyword">new</span> Properties();</span><br><span class="line"> <span class="keyword">for</span> (Property property : properties) {</span><br><span class="line"> props.put(property.getName(), property.getValue());</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (bootstrapServers != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"bootstrap.servers"</span>, bootstrapServers);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> ConfigException(<span class="string">"The bootstrap servers property must be specified"</span>);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.topic == <span class="keyword">null</span>) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> ConfigException(<span class="string">"Topic must be specified by the Kafka log4j appender"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> String clientIdPrefix = taskId != <span class="keyword">null</span> ? taskId : appId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (clientIdPrefix != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"client.id"</span>, clientIdPrefix + <span class="string">"_log"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (props.getProperty(<span class="string">"acks"</span>) == <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"acks"</span>, <span class="string">"0"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (props.getProperty(<span class="string">"retries"</span>) == <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"retries"</span>, <span class="string">"0"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (props.getProperty(<span class="string">"batch.size"</span>) == <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"batch.size"</span>, <span class="string">"16384"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (props.getProperty(<span class="string">"linger.ms"</span>) == <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"linger.ms"</span>, <span class="string">"5"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (props.getProperty(<span class="string">"compression.type"</span>) == <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"compression.type"</span>, <span class="string">"lz4"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> props.put(<span class="string">"key.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"> props.put(<span class="string">"value.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"></span><br><span class="line"> producer = <span class="keyword">new</span> KafkaProducer<>(props);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">append</span><span class="params">(org.apache.logging.log4j.core.LogEvent event)</span> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">if</span> (level.contains(event.getLevel().toString().toUpperCase()) && !event.getLoggerName().contains(<span class="string">"xxx"</span>)) { <span class="comment">//控制哪些类的日志不收集</span></span><br><span class="line"> producer.send(<span class="keyword">new</span> ProducerRecord<>(topic, appId, subAppend(event)));</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> log.warn(<span class="string">"Parsing the log event or send log event to kafka has exception"</span>, e);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> String <span class="title">subAppend</span><span class="params">(org.apache.logging.log4j.core.LogEvent event)</span> <span class="keyword">throws</span> JsonProcessingException </span>{</span><br><span class="line"> LogEvent logEvent = <span class="keyword">new</span> LogEvent();</span><br><span class="line"> Map<String, String> tags = <span class="keyword">new</span> HashMap<>();</span><br><span class="line"> String logMessage = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> InetAddress inetAddress = InetAddress.getLocalHost();</span><br><span class="line"> tags.put(<span class="string">"host_name"</span>, inetAddress.getHostName());</span><br><span class="line"> tags.put(<span class="string">"host_ip"</span>, inetAddress.getHostAddress());</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> log.error(<span class="string">"Error getting the ip and host name of the node where the job({}) is running"</span>, appId, e);</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> logMessage = ExceptionUtil.stacktraceToString(event.getThrown());</span><br><span class="line"> logEvent.setContent(logMessage);</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> <span class="keyword">if</span> (logMessage != <span class="keyword">null</span>) {</span><br><span class="line"> logMessage = logMessage + <span class="string">"\n\t"</span> + e.getMessage();</span><br><span class="line"> }</span><br><span class="line"> logEvent.setContent(logMessage);</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> logEvent.setId(UUID.randomUUID().toString());</span><br><span class="line"> logEvent.setTimestamp(event.getTimeMillis());</span><br><span class="line"> logEvent.setSource(source);</span><br><span class="line"> <span class="keyword">if</span> (logMessage != <span class="keyword">null</span>) {</span><br><span class="line"> logMessage = event.getMessage().getFormattedMessage() + <span class="string">"\n"</span> + logMessage;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> logMessage = event.getMessage().getFormattedMessage();</span><br><span class="line"> }</span><br><span class="line"> logEvent.setContent(logMessage);</span><br><span class="line"></span><br><span class="line"> StackTraceElement eventSource = event.getSource();</span><br><span class="line"> tags.put(<span class="string">"class_name"</span>, eventSource.getClassName());</span><br><span class="line"> tags.put(<span class="string">"method_name"</span>, eventSource.getMethodName());</span><br><span class="line"> tags.put(<span class="string">"file_name"</span>, eventSource.getFileName());</span><br><span class="line"> tags.put(<span class="string">"line_number"</span>, String.valueOf(eventSource.getLineNumber()));</span><br><span class="line"></span><br><span class="line"> tags.put(<span class="string">"logger_name"</span>, event.getLoggerName());</span><br><span class="line"> tags.put(<span class="string">"level"</span>, event.getLevel().toString());</span><br><span class="line"> tags.put(<span class="string">"thread_name"</span>, event.getThreadName());</span><br><span class="line"> tags.put(<span class="string">"app_id"</span>, appId);</span><br><span class="line"> tags.put(<span class="string">"container_id"</span>, containerId);</span><br><span class="line"> tags.put(<span class="string">"container_type"</span>, containerType);</span><br><span class="line"> <span class="keyword">if</span> (taskId != <span class="keyword">null</span>) {</span><br><span class="line"> tags.put(<span class="string">"task_id"</span>, taskId);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (taskName != <span class="keyword">null</span>) {</span><br><span class="line"> tags.put(<span class="string">"task_name"</span>, taskName);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (nodeIp != <span class="keyword">null</span>) {</span><br><span class="line"> tags.put(<span class="string">"node_ip"</span>, nodeIp);</span><br><span class="line"> }</span><br><span class="line"> logEvent.setTags(tags);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> JacksonUtil.toJson(logEvent);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="meta">@PluginFactory</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> KafkaLog4j2Appender <span class="title">createAppender</span><span class="params">(@PluginElement(<span class="string">"Layout"</span>)</span> <span class="keyword">final</span> Layout<? extends Serializable> layout,</span></span><br><span class="line"><span class="function"> @<span class="title">PluginElement</span><span class="params">(<span class="string">"Filter"</span>)</span> <span class="keyword">final</span> Filter filter,</span></span><br><span class="line"><span class="function"> @<span class="title">Required</span><span class="params">(message = <span class="string">"No name provided for KafkaLog4j2Appender"</span>)</span> @<span class="title">PluginAttribute</span><span class="params">(<span class="string">"name"</span>)</span> <span class="keyword">final</span> String name,</span></span><br><span class="line"><span class="function"> @<span class="title">PluginAttribute</span><span class="params">(value = <span class="string">"ignoreExceptions"</span>, defaultBoolean = <span class="keyword">true</span>)</span> <span class="keyword">final</span> <span class="keyword">boolean</span> ignoreExceptions,</span></span><br><span class="line"><span class="function"> @<span class="title">Required</span><span class="params">(message = <span class="string">"No bootstrapServers provided for KafkaLog4j2Appender"</span>)</span> @<span class="title">PluginAttribute</span><span class="params">(<span class="string">"bootstrapServers"</span>)</span> <span class="keyword">final</span> String bootstrapServers,</span></span><br><span class="line"><span class="function"> @<span class="title">Required</span><span class="params">(message = <span class="string">"No source provided for KafkaLog4j2Appender"</span>)</span> @<span class="title">PluginAttribute</span><span class="params">(<span class="string">"source"</span>)</span> <span class="keyword">final</span> String source,</span></span><br><span class="line"><span class="function"> @<span class="title">Required</span><span class="params">(message = <span class="string">"No topic provided for KafkaLog4j2Appender"</span>)</span> @<span class="title">PluginAttribute</span><span class="params">(<span class="string">"topic"</span>)</span> <span class="keyword">final</span> String topic,</span></span><br><span class="line"><span class="function"> @<span class="title">Required</span><span class="params">(message = <span class="string">"No level provided for KafkaLog4j2Appender"</span>)</span> @<span class="title">PluginAttribute</span><span class="params">(<span class="string">"level"</span>)</span> <span class="keyword">final</span> String level,</span></span><br><span class="line"><span class="function"> @<span class="title">PluginElement</span><span class="params">(<span class="string">"Properties"</span>)</span> <span class="keyword">final</span> Property[] properties) </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> KafkaLog4j2Appender(name, filter, layout, ignoreExceptions, properties, source, bootstrapServers, topic, level);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">stop</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">super</span>.stop();</span><br><span class="line"> <span class="keyword">if</span> (producer != <span class="keyword">null</span>) {</span><br><span class="line"> producer.close();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><strong>修改 Log4j2 配置</strong>：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br></pre></td><td class="code"><pre><span class="line">################################################################################</span><br><span class="line"># Licensed to the Apache Software Foundation (ASF) under one</span><br><span class="line"># or more contributor license agreements. See the NOTICE file</span><br><span class="line"># distributed with this work for additional information</span><br><span class="line"># regarding copyright ownership. The ASF licenses this file</span><br><span class="line"># to you under the Apache License, Version 2.0 (the</span><br><span class="line"># "License"); you may not use this file except in compliance</span><br><span class="line"># with the License. You may obtain a copy of the License at</span><br><span class="line">#</span><br><span class="line"># http://www.apache.org/licenses/LICENSE-2.0</span><br><span class="line">#</span><br><span class="line"># Unless required by applicable law or agreed to in writing, software</span><br><span class="line"># distributed under the License is distributed on an "AS IS" BASIS,</span><br><span class="line"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span><br><span class="line"># See the License for the specific language governing permissions and</span><br><span class="line"># limitations under the License.</span><br><span class="line">################################################################################</span><br><span class="line"></span><br><span class="line"># monitorInterval=30</span><br><span class="line"></span><br><span class="line"># This affects logging for both user code and Flink</span><br><span class="line">rootLogger.level = INFO</span><br><span class="line">rootLogger.appenderRef.file.ref = MainAppender</span><br><span class="line">rootLogger.appenderRef.kafka.ref = KafkaLog4j2Appender</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># Uncomment this if you want to _only_ change Flink's logging</span><br><span class="line">#logger.flink.name = org.apache.flink</span><br><span class="line">#logger.flink.level = INFO</span><br><span class="line"></span><br><span class="line"># The following lines keep the log level of common libraries/connectors on</span><br><span class="line"># log level INFO. The root logger does not override this. You have to manually</span><br><span class="line"># change the log levels here.</span><br><span class="line">logger.akka.name = akka</span><br><span class="line">logger.akka.level = INFO</span><br><span class="line">logger.kafka.name= org.apache.kafka</span><br><span class="line">logger.kafka.level = INFO</span><br><span class="line">logger.hadoop.name = org.apache.hadoop</span><br><span class="line">logger.hadoop.level = INFO</span><br><span class="line">logger.zookeeper.name = org.apache.zookeeper</span><br><span class="line">logger.zookeeper.level = INFO</span><br><span class="line"></span><br><span class="line"># Log all infos in the given file</span><br><span class="line">appender.main.name = MainAppender</span><br><span class="line">appender.main.type = RollingFile</span><br><span class="line">appender.main.append = true</span><br><span class="line">appender.main.fileName = ${sys:log.file}</span><br><span class="line">appender.main.filePattern = ${sys:log.file}.%i</span><br><span class="line">appender.main.layout.type = PatternLayout</span><br><span class="line">appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n</span><br><span class="line">appender.main.policies.type = Policies</span><br><span class="line">appender.main.policies.size.type = SizeBasedTriggeringPolicy</span><br><span class="line">appender.main.policies.size.size = 100MB</span><br><span class="line">appender.main.policies.startup.type = OnStartupTriggeringPolicy</span><br><span class="line">appender.main.strategy.type = DefaultRolloverStrategy</span><br><span class="line">appender.main.strategy.max = ${env:MAX_LOG_FILE_NUMBER:-10}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">appender.kafka.name = KafkaLog4j2Appender</span><br><span class="line">appender.kafka.type = KafkaLog4j2Appender</span><br><span class="line">appender.kafka.source = flink-1.12.0</span><br><span class="line">appender.kafka.bootstrapServers=http://fat-kafka1.com.cn:9092,http://fat-kafka2.com.cn:9092,http://fat-kafka3.com.cn:9092</span><br><span class="line">appender.kafka.topic = yarn_flink_log</span><br><span class="line">appender.kafka.level = ERROR,WARN</span><br><span class="line"></span><br><span class="line">logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline</span><br><span class="line">logger.netty.level = OFF</span><br></pre></td></tr></table></figure><p>注意，针对 K8s 需要修改的是 log4j-console.properties 才能生效</p><p><strong>发到 Kafka 数据的结构</strong>：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"><span class="string">"source"</span>: <span class="string">"flink-1.12.0"</span>,</span><br><span class="line"><span class="string">"id"</span>: <span class="string">"855986dd-993a-4efe-99ab-658fe6bf1683"</span>,</span><br><span class="line"><span class="string">"timestamp"</span>: <span class="number">1628131862944</span>,</span><br><span class="line"><span class="string">"content"</span>: <span class="string">"Source: Custom Source (1/2) (6302f688e5405815b719bb236634f341) switched from RUNNING to FAILED on container_e12_1626247520347_2269_01_000002 @ uat-hadoopuat-dc01-025187.vm.dc01.tech (dataPort=9916).\njava.lang.Exception: [2021-08-05 10:51:00.641]Container killed on request. Exit code is 137\n[2021-08-05 10:51:00.641]Container exited with a non-zero exit code 137. \n[2021-08-05 10:51:00.643]Killed by external signal\n\n\tat org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager.onWorkerTerminated(ActiveResourceManager.java:219)\n\tat org.apache.flink.yarn.YarnResourceManagerDriver$YarnContainerEventHandler.lambda$onContainersCompleted$0(YarnResourceManagerDriver.java:522)\n\tat org.apache.flink.yarn.YarnResourceManagerDriver$YarnContainerEventHandler.lambda$runAsyncWithFatalHandler$2(YarnResourceManagerDriver.java:549)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:404)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:197)\n\tat org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)\n\tat org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:154)\n\tat akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)\n\tat akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)\n\tat scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)\n\tat akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\n\tat scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)\n\tat akka.actor.Actor$class.aroundReceive(Actor.scala:517)\n\tat akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)\n\tat akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)\n\tat akka.actor.ActorCell.invoke(ActorCell.scala:561)\n\tat akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)\n\tat akka.dispatch.Mailbox.run(Mailbox.scala:225)\n\tat akka.dispatch.Mailbox.exec(Mailbox.scala:235)\n\tat akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)\n\tat akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)\n\tat akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)\n\tat akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)\n"</span>,</span><br><span class="line"><span class="string">"tags"</span>: {</span><br><span class="line"><span class="string">"host_ip"</span>: <span class="string">"10.69.1.10"</span>,</span><br><span class="line"><span class="string">"method_name"</span>: <span class="string">"transitionState"</span>,</span><br><span class="line"><span class="string">"level"</span>: <span class="string">"WARN"</span>,</span><br><span class="line"><span class="string">"file_name"</span>: <span class="string">"Execution.java"</span>,</span><br><span class="line"><span class="string">"line_number"</span>: <span class="string">"1608"</span>,</span><br><span class="line"><span class="string">"thread_name"</span>: <span class="string">"flink-akka.actor.default-dispatcher-3"</span>,</span><br><span class="line"><span class="string">"container_type"</span>: <span class="string">"jobmanager"</span>,</span><br><span class="line"><span class="string">"logger_name"</span>: <span class="string">"org.apache.flink.runtime.executiongraph.ExecutionGraph"</span>,</span><br><span class="line"><span class="string">"class_name"</span>: <span class="string">"org.apache.flink.runtime.executiongraph.Execution"</span>,</span><br><span class="line"><span class="string">"app_id"</span>: <span class="string">"application_1626247520347_2269"</span>,</span><br><span class="line"><span class="string">"host_name"</span>: <span class="string">"FAT-hadoopuat-69110.vm.dc01.tech"</span>,</span><br><span class="line"><span class="string">"container_id"</span>: <span class="string">"container_e12_1626247520347_2269_01_000001"</span></span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="自定义-Log4j-Kafka-Appender"><a href="#自定义-Log4j-Kafka-Appender" class="headerlink" title="自定义 Log4j Kafka Appender"></a>自定义 Log4j Kafka Appender</h3><p><strong>引入依赖</strong>：</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br></pre></td><td class="code"><pre><span class="line"><span class="php"><span class="meta"><?</span>xml version=<span class="string">"1.0"</span> encoding=<span class="string">"UTF-8"</span><span class="meta">?></span></span></span><br><span class="line"><span class="tag"><<span class="name">project</span> <span class="attr">xmlns</span>=<span class="string">"http://maven.apache.org/POM/4.0.0"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xmlns:xsi</span>=<span class="string">"http://www.w3.org/2001/XMLSchema-instance"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xsi:schemaLocation</span>=<span class="string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">parent</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>FlinkLogKafkaAppender<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.zhisheng.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.0-SNAPSHOT<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">parent</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">modelVersion</span>></span>4.0.0<span class="tag"></<span class="name">modelVersion</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>Log4jKafkaAppender<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">slf4j.version</span>></span>1.7.15<span class="tag"></<span class="name">slf4j.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">flink.shaded.version</span>></span>9.0<span class="tag"></<span class="name">flink.shaded.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">jackson.version</span>></span>2.10.1<span class="tag"></<span class="name">jackson.version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">properties</span>></span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependencies</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.zhisheng.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>KafkaAppenderCommon<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.0-SNAPSHOT<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.slf4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>slf4j-log4j12<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${slf4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>${scope}<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-shaded-jackson<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${jackson.version}-${flink.shaded.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>${scope}<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependencies</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">build</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">plugins</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.maven.plugins<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>maven-shade-plugin<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>3.1.1<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">executions</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">execution</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">id</span>></span>shade-flink<span class="tag"></<span class="name">id</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">phase</span>></span>package<span class="tag"></<span class="name">phase</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">goal</span>></span>shade<span class="tag"></<span class="name">goal</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactSet</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">includes</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">include</span>></span>org.apache.kafka:*<span class="tag"></<span class="name">include</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">includes</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">artifactSet</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">execution</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">executions</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugins</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">build</span>></span></span><br><span class="line"><span class="tag"></<span class="name">project</span>></span></span><br></pre></td></tr></table></figure><p><strong>KafkaLog4jAppender: 继承 AppenderSkeleton 抽象类</strong></p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">KafkaLog4jAppender</span> <span class="keyword">extends</span> <span class="title">AppenderSkeleton</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String bootstrapServers;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String source;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String topic;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String level;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String acks;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String compressionType;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String retries;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String batchSize;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String lingerMs;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String maxRequestSize;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String requestTimeoutMs;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> Producer<String, String> producer;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String appId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String containerId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String containerType;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String taskId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">private</span> String taskName;</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">activateOptions</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">super</span>.activateOptions();</span><br><span class="line"></span><br><span class="line"> Properties envProperties = System.getProperties();</span><br><span class="line"></span><br><span class="line"> String logFile = envProperties.getProperty(<span class="string">"log.file"</span>);</span><br><span class="line"> String[] values = logFile.split(File.separator);</span><br><span class="line"> <span class="keyword">if</span> (values.length >= <span class="number">3</span>) {</span><br><span class="line"> appId = values[values.length - <span class="number">3</span>];</span><br><span class="line"> containerId = values[values.length - <span class="number">2</span>];</span><br><span class="line"> String log = values[values.length - <span class="number">1</span>];</span><br><span class="line"> <span class="keyword">if</span> (log.contains(<span class="string">"jobmanager"</span>)) {</span><br><span class="line"> containerType = <span class="string">"jobmanager"</span>;</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (log.contains(<span class="string">"taskmanager"</span>)) {</span><br><span class="line"> containerType = <span class="string">"taskmanager"</span>;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> containerType = <span class="string">"others"</span>;</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> log.error(<span class="string">"log.file Property ({}) doesn't contains yarn application id or container id"</span>, logFile);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> taskId = envProperties.getProperty(<span class="string">"taskId"</span>, <span class="keyword">null</span>);</span><br><span class="line"> taskName = envProperties.getProperty(<span class="string">"taskName"</span>, <span class="keyword">null</span>);</span><br><span class="line"></span><br><span class="line"> Properties props = <span class="keyword">new</span> Properties();</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.bootstrapServers != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"bootstrap.servers"</span>, <span class="keyword">this</span>.bootstrapServers);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> ConfigException(<span class="string">"The bootstrap servers property must be specified"</span>);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.topic == <span class="keyword">null</span>) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> ConfigException(<span class="string">"Topic must be specified by the Kafka log4j appender"</span>);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.source == <span class="keyword">null</span>) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> ConfigException(<span class="string">"Source must be specified by the Kafka log4j appender"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> String clientIdPrefix = taskId != <span class="keyword">null</span> ? taskId : appId;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (clientIdPrefix != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"client.id"</span>, clientIdPrefix + <span class="string">"_log"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.acks != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"acks"</span>, <span class="keyword">this</span>.acks);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> props.setProperty(<span class="string">"acks"</span>, <span class="string">"0"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.retries != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"retries"</span>, <span class="keyword">this</span>.retries);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> props.setProperty(<span class="string">"retries"</span>, <span class="string">"0"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.batchSize != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"batch.size"</span>, <span class="keyword">this</span>.batchSize);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> props.setProperty(<span class="string">"batch.size"</span>, <span class="string">"16384"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.lingerMs != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"linger.ms"</span>, <span class="keyword">this</span>.lingerMs);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> props.setProperty(<span class="string">"linger.ms"</span>, <span class="string">"5"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (<span class="keyword">this</span>.compressionType != <span class="keyword">null</span>) {</span><br><span class="line"> props.setProperty(<span class="string">"compression.type"</span>, <span class="keyword">this</span>.compressionType);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> props.setProperty(<span class="string">"compression.type"</span>, <span class="string">"lz4"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> props.put(<span class="string">"key.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"> props.put(<span class="string">"value.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"></span><br><span class="line"> producer = <span class="keyword">new</span> KafkaProducer<>(props);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">append</span><span class="params">(LoggingEvent loggingEvent)</span> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">if</span> (level.contains(loggingEvent.getLevel().toString().toUpperCase()) && !loggingEvent.getLoggerName().contains(<span class="string">"xxx"</span>)) { <span class="comment">//控制哪些类的日志不收集</span></span><br><span class="line"> producer.send(<span class="keyword">new</span> ProducerRecord<>(topic, appId, subAppend(loggingEvent)));</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> log.warn(<span class="string">"Parsing the log event or send log event to kafka has exception"</span>, e);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">private</span> String <span class="title">subAppend</span><span class="params">(LoggingEvent event)</span> <span class="keyword">throws</span> JsonProcessingException </span>{</span><br><span class="line"> LogEvent logEvent = <span class="keyword">new</span> LogEvent();</span><br><span class="line"> Map<String, String> tags = <span class="keyword">new</span> HashMap<>();</span><br><span class="line"> String logMessage = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> InetAddress inetAddress = InetAddress.getLocalHost();</span><br><span class="line"> tags.put(<span class="string">"host_name"</span>, inetAddress.getHostName());</span><br><span class="line"> tags.put(<span class="string">"host_ip"</span>, inetAddress.getHostAddress());</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> log.error(<span class="string">"Error getting the ip and host name of the node where the job({}) is running"</span>, appId, e);</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> logMessage = ExceptionUtil.stacktraceToString(event.getThrowableInformation().getThrowable());</span><br><span class="line"> logEvent.setContent(logMessage);</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> <span class="keyword">if</span> (logMessage != <span class="keyword">null</span>) {</span><br><span class="line"> logMessage = logMessage + <span class="string">"\n\t"</span> + e.getMessage();</span><br><span class="line"> }</span><br><span class="line"> logEvent.setContent(logMessage);</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> logEvent.setId(UUID.randomUUID().toString());</span><br><span class="line"> logEvent.setTimestamp(event.getTimeStamp());</span><br><span class="line"> logEvent.setSource(source);</span><br><span class="line"> <span class="keyword">if</span> (logMessage != <span class="keyword">null</span>) {</span><br><span class="line"> logMessage = event.getMessage().toString() + <span class="string">"\n"</span> + logMessage;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> logMessage = event.getMessage().toString();</span><br><span class="line"> }</span><br><span class="line"> logEvent.setContent(logMessage);</span><br><span class="line"> LocationInfo locationInformation = event.getLocationInformation();</span><br><span class="line"> tags.put(<span class="string">"class_name"</span>, locationInformation.getClassName());</span><br><span class="line"> tags.put(<span class="string">"method_name"</span>, locationInformation.getMethodName());</span><br><span class="line"> tags.put(<span class="string">"file_name"</span>, locationInformation.getFileName());</span><br><span class="line"> tags.put(<span class="string">"line_number"</span>, locationInformation.getLineNumber());</span><br><span class="line"> tags.put(<span class="string">"logger_name"</span>, event.getLoggerName());</span><br><span class="line"> tags.put(<span class="string">"level"</span>, event.getLevel().toString());</span><br><span class="line"> tags.put(<span class="string">"thread_name"</span>, event.getThreadName());</span><br><span class="line"> tags.put(<span class="string">"app_id"</span>, appId);</span><br><span class="line"> tags.put(<span class="string">"container_id"</span>, containerId);</span><br><span class="line"> tags.put(<span class="string">"container_type"</span>, containerType);</span><br><span class="line"> <span class="keyword">if</span> (taskName != <span class="keyword">null</span>) {</span><br><span class="line"> tags.put(<span class="string">"task_name"</span>, taskName);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (taskId != <span class="keyword">null</span>) {</span><br><span class="line"> tags.put(<span class="string">"task_id"</span>, taskId);</span><br><span class="line"> }</span><br><span class="line"> logEvent.setTags(tags);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> JacksonUtil.toJson(logEvent);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">close</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">if</span> (!<span class="keyword">this</span>.closed) {</span><br><span class="line"> <span class="keyword">this</span>.closed = <span class="keyword">true</span>;</span><br><span class="line"> <span class="keyword">this</span>.producer.close();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">requiresLayout</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p><strong>将上面打出来的 jar，放到 flink 的 lib 目录下面，修改 log4j.properties 配置文件</strong>:</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line"># This affects logging for both user code and Flink</span><br><span class="line">log4j.rootLogger=INFO, RFA, kafka</span><br><span class="line"></span><br><span class="line"># Uncomment this if you want to _only_ change Flink's logging</span><br><span class="line">#log4j.logger.org.apache.flink=INFO</span><br><span class="line"></span><br><span class="line"># The following lines keep the log level of common libraries/connectors on</span><br><span class="line"># log level INFO. The root logger does not override this. You have to manually</span><br><span class="line"># change the log levels here.</span><br><span class="line">log4j.logger.akka=INFO</span><br><span class="line">log4j.logger.org.apache.kafka=INFO</span><br><span class="line">log4j.logger.org.apache.hadoop=INFO</span><br><span class="line">log4j.logger.org.apache.zookeeper=INFO</span><br><span class="line"></span><br><span class="line">log4j.appender.RFA=org.apache.log4j.RollingFileAppender</span><br><span class="line">log4j.appender.RFA.File=${log.file}</span><br><span class="line">log4j.appender.RFA.MaxFileSize=256MB</span><br><span class="line">log4j.appender.RFA.Append=true</span><br><span class="line">log4j.appender.RFA.MaxBackupIndex=10</span><br><span class="line">log4j.appender.RFA.layout=org.apache.log4j.PatternLayout</span><br><span class="line">log4j.appender.RFA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %t %-5p %-60c %x - %m%n</span><br><span class="line"></span><br><span class="line">log4j.logger.org.apache.kafka.clients.Metadata=WARN,kafka# 解决死锁问题</span><br><span class="line">log4j.appender.kafka=com.zhisheng.log.appender.KafkaLog4jAppender</span><br><span class="line">log4j.appender.kafka.source=flink-1.10.0</span><br><span class="line">log4j.appender.kafka.bootstrapServers=http://kafka1.com.cn:9092,http://kafka2.com.cn:9092,http://kafka3.com.cn:9092</span><br><span class="line">log4j.appender.kafka.topic=yarn_flink_log</span><br><span class="line">log4j.appender.kafka.level=ERROR,WARN</span><br><span class="line"></span><br><span class="line"># Suppress the irrelevant (wrong) warnings from the Netty channel handler</span><br><span class="line">log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, RFA</span><br></pre></td></tr></table></figure><p><strong>注意上面的一个解决死锁问题的配置，我们在生产也遇到过，打出来的 jstack 的信息为</strong>：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br></pre></td><td class="code"><pre><span class="line">2021-08-20 01:19:24</span><br><span class="line">Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.201-b09 mixed mode):</span><br><span class="line"></span><br><span class="line">"Attach Listener" #14 daemon prio=9 os_prio=0 tid=0x00007f061c001800 nid=0x728 waiting on condition [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"kafka-producer-network-thread | application_1610962534438_1256_log" #11 daemon prio=5 os_prio=0 tid=0x00007f06394ad800 nid=0x27e waiting for monitor entry [0x00007f060be16000]</span><br><span class="line"> java.lang.Thread.State: BLOCKED (on object monitor)</span><br><span class="line">at org.apache.log4j.Category.callAppenders(Category.java:205)</span><br><span class="line">- waiting to lock <0x00000000ffad7510> (a org.apache.log4j.spi.RootLogger)</span><br><span class="line">at org.apache.log4j.Category.forcedLog(Category.java:391)</span><br><span class="line">at org.apache.log4j.Category.log(Category.java:856)</span><br><span class="line">at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:324)</span><br><span class="line">at org.apache.kafka.clients.Metadata.update(Metadata.java:379)</span><br><span class="line">- locked <0x00000000ff1421d8> (a org.apache.kafka.clients.Metadata)</span><br><span class="line">at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1039)</span><br><span class="line">at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:822)</span><br><span class="line">at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)</span><br><span class="line">at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:312)</span><br><span class="line">at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)</span><br><span class="line">at java.lang.Thread.run(Thread.java:748)</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f063865c800 nid=0x27c runnable [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f063862f000 nid=0x27b waiting on condition [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f063862d000 nid=0x27a waiting on condition [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f063862a800 nid=0x279 waiting on condition [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f0638628800 nid=0x278 runnable [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007f0638626800 nid=0x277 waiting on condition [0x0000000000000000]</span><br><span class="line"> java.lang.Thread.State: RUNNABLE</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f06385f3800 nid=0x276 in Object.wait() [0x00007f0617c07000]</span><br><span class="line"> java.lang.Thread.State: WAITING (on object monitor)</span><br><span class="line">at java.lang.Object.wait(Native Method)</span><br><span class="line">- waiting on <0x00000000fff08ed0> (a java.lang.ref.ReferenceQueue$Lock)</span><br><span class="line">at j ava.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)</span><br><span class="line">- locked <0x00000000fff08ed0> (a java.lang.ref.ReferenceQueue$Lock)</span><br><span class="line">at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)</span><br><span class="line">at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f06385f1000 nid=0x275 in Object.wait() [0x00007f0617d08000]</span><br><span class="line"> java.lang.Thread.State: WAITING (on object monitor)</span><br><span class="line">at java.lang.Object.wait(Native Method)</span><br><span class="line">- waiting on <0x00000000fff06bf8> (a java.lang.ref.Reference$Lock)</span><br><span class="line">at java.lang.Object.wait(Object.java:502)</span><br><span class="line">at java.lang.ref.Reference.tryHandlePending(Reference.java:191)</span><br><span class="line">- locked <0x00000000fff06bf8> (a java.lang.ref.Reference$Lock)</span><br><span class="line">at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"main" #1 prio=5 os_prio=0 tid=0x00007f0638011800 nid=0x253 waiting for monitor entry [0x00007f063e7b7000]</span><br><span class="line"> java.lang.Thread.State: BLOCKED (on object monitor)</span><br><span class="line">at org.apache.kafka.clients.Metadata.fetch(Metadata.java:129)</span><br><span class="line">- waiting to lock <0x00000000ff1421d8> (a org.apache.kafka.clients.Metadata)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:960)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:866)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:846)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:733)</span><br><span class="line">at com.zhisheng.log.appender.KafkaLog4jAppender.append(KafkaLog4jAppender.java:147)</span><br><span class="line">at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)</span><br><span class="line">- locked <0x00000000ff542900> (a com.zhisheng.log.appender.KafkaLog4jAppender)</span><br><span class="line">at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)</span><br><span class="line">at org.apache.log4j.Category.callAppenders(Category.java:206)</span><br><span class="line">- locked <0x00000000ffad7510> (a org.apache.log4j.spi.RootLogger)</span><br><span class="line">at org.apache.log4j.Category.forcedLog(Category.java:391)</span><br><span class="line">at org.apache.log4j.Category.log(Category.java:856)</span><br><span class="line">at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:401)</span><br><span class="line">at org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)</span><br><span class="line">at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.<init>(JniBasedUnixGroupsMappingWithFallback.java:38)</span><br><span class="line">at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)</span><br><span class="line">at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)</span><br><span class="line">at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)</span><br><span class="line">at java.lang.reflect.Constructor.newInstance(Constructor.java:423)</span><br><span class="line">at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)</span><br><span class="line">at org.apache.hadoop.security.Groups.<init>(Groups.java:106)</span><br><span class="line">at org.apache.hadoop.security.Groups.<init>(Groups.java:101)</span><br><span class="line">at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:449)</span><br><span class="line">- locked <0x00000000fec67710> (a java.lang.Class for org.apache.hadoop.security.Groups)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:327)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:294)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:854)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:824)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:693)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</span><br><span class="line">at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)</span><br><span class="line">at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</span><br><span class="line">at java.lang.reflect.Method.invoke(Method.java:498)</span><br><span class="line">at org.apache.flink.runtime.util.EnvironmentInformation.getHadoopUser(EnvironmentInformation.java:96)</span><br><span class="line">at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:293)</span><br><span class="line">at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:96)</span><br><span class="line"></span><br><span class="line"> Locked ownable synchronizers:</span><br><span class="line">- None</span><br><span class="line"></span><br><span class="line">"VM Thread" os_prio=0 tid=0x00007f06385e7000 nid=0x274 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f0638029000 nid=0x256 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f063802a800 nid=0x257 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f063802c800 nid=0x258 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f063802e800 nid=0x259 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007f0638030000 nid=0x25a runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007f0638032000 nid=0x25b runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007f0638034000 nid=0x25c runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007f0638035800 nid=0x25d runnable</span><br><span class="line"></span><br><span class="line">"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f063805e000 nid=0x26b runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f0638060000 nid=0x26f runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f0638061800 nid=0x270 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#2 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f0638063800 nid=0x272 runnable</span><br><span class="line"></span><br><span class="line">"Gang worker#3 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007f0638065800 nid=0x273 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f0638047800 nid=0x26a runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f0638045800 nid=0x265 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f0638043800 nid=0x264 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f0638042000 nid=0x263 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f0638040000 nid=0x262 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007f063803e000 nid=0x261 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007f063803c000 nid=0x260 runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007f063803a800 nid=0x25f runnable</span><br><span class="line"></span><br><span class="line">"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007f0638038800 nid=0x25e runnable</span><br><span class="line"></span><br><span class="line">"VM Periodic Task Thread" os_prio=0 tid=0x00007f0638660000 nid=0x27d waiting on condition</span><br><span class="line"></span><br><span class="line">JNI global references: 652</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">Found one Java-level deadlock:</span><br><span class="line">=============================</span><br><span class="line">"kafka-producer-network-thread | application_1610962534438_1256_log":</span><br><span class="line"> waiting to lock monitor 0x00007f0624005f18 (object 0x00000000ffad7510, a org.apache.log4j.spi.RootLogger),</span><br><span class="line"> which is held by "main"</span><br><span class="line">"main":</span><br><span class="line"> waiting to lock monitor 0x00007f06240047b8 (object 0x00000000ff1421d8, a org.apache.kafka.clients.Metadata),</span><br><span class="line"> which is held by "kafka-producer-network-thread | application_1610962534438_1256_log"</span><br><span class="line"></span><br><span class="line">Java stack information for the threads listed above:</span><br><span class="line">===================================================</span><br><span class="line">"kafka-producer-network-thread | application_1610962534438_1256_log":</span><br><span class="line">at org.apache.log4j.Category.callAppenders(Category.java:205)</span><br><span class="line">- waiting to lock <0x00000000ffad7510> (a org.apache.log4j.spi.RootLogger)</span><br><span class="line">at org.apache.log4j.Category.forcedLog(Category.java:391)</span><br><span class="line">at org.apache.log4j.Category.log(Category.java:856)</span><br><span class="line">at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:324)</span><br><span class="line">at org.apache.kafka.clients.Metadata.update(Metadata.java:379)</span><br><span class="line">- locked <0x00000000ff1421d8> (a org.apache.kafka.clients.Metadata)</span><br><span class="line">at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1039)</span><br><span class="line">at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:822)</span><br><span class="line">at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)</span><br><span class="line">at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:312)</span><br><span class="line">at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)</span><br><span class="line">at java.lang.Thread.run(Thread.java:748)</span><br><span class="line">"main":</span><br><span class="line">at org.apache.kafka.clients.Metadata.fetch(Metadata.java:129)</span><br><span class="line">- waiting to lock <0x00000000ff1421d8> (a org.apache.kafka.clients.Metadata)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:960)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:866)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:846)</span><br><span class="line">at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:733)</span><br><span class="line">at com.zhisheng.log.appender.KafkaLog4jAppender.append(KafkaLog4jAppender.java:147)</span><br><span class="line">at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)</span><br><span class="line">- locked <0x00000000ff542900> (a com.zhisheng.log.appender.KafkaLog4jAppender)</span><br><span class="line">at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)</span><br><span class="line">at org.apache.log4j.Category.callAppenders(Category.java:206)</span><br><span class="line">- locked <0x00000000ffad7510> (a org.apache.log4j.spi.RootLogger)</span><br><span class="line">at org.apache.log4j.Category.forcedLog(Category.java:391)</span><br><span class="line">at org.apache.log4j.Category.log(Category.java:856)</span><br><span class="line">at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:401)</span><br><span class="line">at org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)</span><br><span class="line">at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.<init>(JniBasedUnixGroupsMappingWithFallback.java:38)</span><br><span class="line">at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)</span><br><span class="line">at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)</span><br><span class="line">at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)</span><br><span class="line">at java.lang.reflect.Constructor.newInstance(Constructor.java:423)</span><br><span class="line">at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)</span><br><span class="line">at org.apache.hadoop.security.Groups.<init>(Groups.java:106)</span><br><span class="line">at org.apache.hadoop.security.Groups.<init>(Groups.java:101)</span><br><span class="line">at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:449)</span><br><span class="line">- locked <0x00000000fec67710> (a java.lang.Class for org.apache.hadoop.security.Groups)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:327)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:294)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:854)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:824)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:693)</span><br><span class="line">- locked <0x00000000fec436d0> (a java.lang.Class for org.apache.hadoop.security.UserGroupInformation)</span><br><span class="line">at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)</span><br><span class="line">at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)</span><br><span class="line">at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</span><br><span class="line">at java.lang.reflect.Method.invoke(Method.java:498)</span><br><span class="line">at org.apache.flink.runtime.util.EnvironmentInformation.getHadoopUser(EnvironmentInformation.java:96)</span><br><span class="line">at org.apache.flink.runtime.util.EnvironmentInformation.logEnvironmentInfo(EnvironmentInformation.java:293)</span><br><span class="line">at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:96)</span><br><span class="line"></span><br><span class="line">Found 1 deadlock.</span><br></pre></td></tr></table></figure><p><strong>发到 Kafka 数据的结构</strong>:</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"><span class="attr">"source"</span>: <span class="string">"flink-1.10.0"</span>,</span><br><span class="line"><span class="attr">"id"</span>: <span class="string">"fc527ec2-0a95-4fe8-83d2-d3e889c03658"</span>,</span><br><span class="line"><span class="attr">"timestamp"</span>: <span class="number">1627886187629</span>,</span><br><span class="line"><span class="attr">"content"</span>: <span class="string">"Error registering AppInfo mbean\njavax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=application_1626247520347_2075\n\tat com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)\n\tat org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:64)\n\tat org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:426)\n\tat org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)\n\tat org.apache.flink.metrics.kafka.KafkaReporter.open(KafkaReporter.java:107)\n\tat org.apache.flink.runtime.metrics.ReporterSetup.createReporterSetup(ReporterSetup.java:130)\n\tat org.apache.flink.runtime.metrics.ReporterSetup.lambda$setupReporters$1(ReporterSetup.java:239)\n\tat java.util.Optional.ifPresent(Optional.java:159)\n\tat org.apache.flink.runtime.metrics.ReporterSetup.setupReporters(ReporterSetup.java:236)\n\tat org.apache.flink.runtime.metrics.ReporterSetup.fromConfiguration(ReporterSetup.java:148)\n\tat org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:160)\n\tat org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:338)\n\tat org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$3(TaskManagerRunner.java:362)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)\n\tat org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)\n\tat org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:361)\n\tat org.apache.flink.yarn.YarnTaskExecutorRunner.runTaskManagerSecurely(YarnTaskExecutorRunner.java:90)\n\tat org.apache.flink.yarn.YarnTaskExecutorRunner.main(YarnTaskExecutorRunner.java:70)\n"</span>,</span><br><span class="line"><span class="attr">"tags"</span>: {</span><br><span class="line"><span class="attr">"host_ip"</span>: <span class="string">"10.xx.xx.17"</span>,</span><br><span class="line"><span class="attr">"method_name"</span>: <span class="string">"checkState"</span>,</span><br><span class="line"><span class="attr">"level"</span>: <span class="string">"ERROR"</span>,</span><br><span class="line"><span class="attr">"file_name"</span>: <span class="string">"ConnectionState.java"</span>,</span><br><span class="line"><span class="attr">"line_number"</span>: <span class="string">"288"</span>,</span><br><span class="line"><span class="attr">"thread_name"</span>: <span class="string">"main-EventThread"</span>,</span><br><span class="line"><span class="attr">"container_type"</span>: <span class="string">"taskmanager"</span>,</span><br><span class="line"><span class="attr">"logger_name"</span>: <span class="string">"org.apache.flink.shaded.curator.org.apache.curator.ConnectionState"</span>,</span><br><span class="line"><span class="attr">"class_name"</span>: <span class="string">"org.apache.flink.shaded.curator.org.apache.curator.ConnectionState"</span>,</span><br><span class="line"><span class="attr">"app_id"</span>: <span class="string">"application_1626247520347_1831"</span>,</span><br><span class="line"><span class="attr">"host_name"</span>: <span class="string">"FAT-hadoopuat-69117.vm.dc01.tech"</span>,</span><br><span class="line"><span class="attr">"container_id"</span>: <span class="string">"container_e12_1626247520347_1831_01_000002"</span></span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="Flink-SQL-存储日志到-ES"><a href="#Flink-SQL-存储日志到-ES" class="headerlink" title="Flink SQL 存储日志到 ES"></a>Flink SQL 存储日志到 ES</h2><p>通过 Flink SQL 将异常日志数据存储到 ES，按天索引：</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> yarn_flink_warn_logs (</span><br><span class="line"> <span class="string">`source`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`id`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`timestamp`</span> <span class="built_in">BIGINT</span>,</span><br><span class="line"> <span class="string">`content`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`tags`</span> <span class="keyword">ROW</span><host_ip <span class="keyword">STRING</span>, method_name <span class="keyword">STRING</span>, <span class="string">`level`</span> <span class="keyword">STRING</span>, <span class="string">`file_name`</span> <span class="keyword">STRING</span>, line_number <span class="keyword">STRING</span>, thread_name <span class="keyword">STRING</span>, container_type <span class="keyword">STRING</span>, logger_name <span class="keyword">STRING</span>, class_name <span class="keyword">STRING</span>, app_id <span class="keyword">STRING</span>, <span class="string">`host_name`</span> <span class="keyword">STRING</span>, container_id <span class="keyword">STRING</span>></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'connector'</span> = <span class="string">'kafka'</span>,</span><br><span class="line"> <span class="string">'topic'</span> = <span class="string">'yarn_flink_log'</span>,</span><br><span class="line"> <span class="string">'scan.startup.mode'</span> = <span class="string">'latest-offset'</span>,</span><br><span class="line"> <span class="string">'properties.bootstrap.servers'</span> = <span class="string">'fat-kafka1.com.cn:9092,fat-kafka2.com.cn:9092,fat-kafka3.com.cn:9092'</span>,</span><br><span class="line"> <span class="string">'properties.group.id'</span> = <span class="string">'yarn_flink_warn_logs'</span>,</span><br><span class="line"> <span class="string">'scan.topic-partition-discovery.interval'</span> = <span class="string">'10000 ms'</span>,</span><br><span class="line"> <span class="string">'format'</span> = <span class="string">'json'</span>,</span><br><span class="line"> <span class="string">'json.fail-on-missing-field'</span> = <span class="string">'false'</span>,</span><br><span class="line"> <span class="string">'json.ignore-parse-errors'</span> = <span class="string">'true'</span>,</span><br><span class="line"> <span class="string">'source.parallelism'</span> = <span class="string">'1'</span></span><br><span class="line">);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> yarn_flink_warn_logs_es (</span><br><span class="line"> <span class="string">`source`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`id`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`logTime`</span> <span class="built_in">BIGINT</span>,</span><br><span class="line"> <span class="string">`log_time`</span> <span class="keyword">TIMESTAMP</span>(<span class="number">3</span>),</span><br><span class="line"> <span class="string">`content`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`tags`</span> <span class="keyword">ROW</span><host_ip <span class="keyword">STRING</span>, method_name <span class="keyword">STRING</span>, <span class="string">`level`</span> <span class="keyword">STRING</span>, <span class="string">`file_name`</span> <span class="keyword">STRING</span>, line_number <span class="keyword">STRING</span>, thread_name <span class="keyword">STRING</span>, container_type <span class="keyword">STRING</span>, logger_name <span class="keyword">STRING</span>, class_name <span class="keyword">STRING</span>, app_id <span class="keyword">STRING</span>, <span class="string">`host_name`</span> <span class="keyword">STRING</span>, container_id <span class="keyword">STRING</span>></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'connector'</span> = <span class="string">'elasticsearch-universal'</span>,</span><br><span class="line"> <span class="string">'hosts'</span> = <span class="string">'http://fat-search-es.cn:9200'</span>,</span><br><span class="line"> <span class="string">'index'</span> = <span class="string">'yarn_flink_warn_logs-{log_time|yyyy.MM.dd}'</span>,</span><br><span class="line"> <span class="string">'username'</span> = <span class="string">'test-admin'</span>,</span><br><span class="line"> <span class="string">'password'</span> = <span class="string">'test-admin'</span>,</span><br><span class="line"> <span class="string">'sink.parallelism'</span> = <span class="string">'2'</span>, </span><br><span class="line"> <span class="string">'failure-handler'</span> = <span class="string">'ignore'</span>,</span><br><span class="line"> <span class="string">'sink.bulk-flush.max-actions'</span> = <span class="string">'1000'</span>,</span><br><span class="line"> <span class="string">'sink.bulk-flush.max-size'</span> = <span class="string">'5MB'</span>,</span><br><span class="line"> <span class="string">'sink.bulk-flush.interval'</span> = <span class="string">'10'</span></span><br><span class="line">);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">insert</span> <span class="keyword">into</span> yarn_flink_warn_logs_es <span class="keyword">select</span> <span class="string">`source`</span>, <span class="string">`id`</span>, <span class="string">`timestamp`</span> <span class="keyword">as</span> logTime, TO_TIMESTAMP(FROM_UNIXTIME(<span class="string">`timestamp`</span> / <span class="number">1000</span>,<span class="string">'yyyy-MM-dd HH:mm:ss'</span>)) <span class="keyword">AS</span> log_time, <span class="keyword">content</span>, tags <span class="keyword">from</span> yarn_flink_warn_logs;</span><br></pre></td></tr></table></figure><p>单个作业日志查看，可以通过 application_id/container_type/container_id 过滤，当然你可以看到上面自定义里面我们还加入了 taskId 和 taskName 的维度数据，这两个是我们实时计算平台的维度数据，也可以根据这两个进行过滤。</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-163925.jpg" alt=""></p><p>单条日志查看:</p><p><img src="https://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2023-03-16-163948.jpg" alt=""></p><h2 id="实时监控集群作业异常日志数据量"><a href="#实时监控集群作业异常日志数据量" class="headerlink" title="实时监控集群作业异常日志数据量"></a>实时监控集群作业异常日志数据量</h2><p>方便知道任务的异常日志情况，有的任务如果任务突然报出很多异常日志说明有抖动或者报错，对任务负责人和集群负责人都可以做一个提醒通知，起到预警作用。</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">set</span> table.exec.emit.early-fire.enabled=<span class="literal">true</span>;</span><br><span class="line"><span class="keyword">set</span> table.exec.emit.early-fire.delay=<span class="number">60000</span>ms;</span><br><span class="line"><span class="keyword">set</span> table.exec.state.ttl=<span class="number">90000000</span>ms;</span><br><span class="line"></span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> yarn_flink_warn_logs (</span><br><span class="line"> <span class="string">`source`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> <span class="string">`timestamp`</span> <span class="built_in">BIGINT</span>,</span><br><span class="line"> <span class="string">`tags`</span> <span class="keyword">ROW</span><app_id <span class="keyword">STRING</span>>,</span><br><span class="line"> windows_start_time <span class="keyword">as</span> TO_TIMESTAMP(FROM_UNIXTIME(<span class="string">`timestamp`</span> / <span class="number">1000</span>,<span class="string">'yyyy-MM-dd HH:mm:ss'</span>)),</span><br><span class="line"> WATERMARK <span class="keyword">FOR</span> windows_start_time <span class="keyword">AS</span> windows_start_time - <span class="built_in">INTERVAL</span> <span class="string">'5'</span> <span class="keyword">SECOND</span></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'connector'</span> = <span class="string">'kafka'</span>,</span><br><span class="line"> <span class="string">'topic'</span> = <span class="string">'yarn_flink_log'</span>,</span><br><span class="line"> <span class="string">'scan.startup.mode'</span> = <span class="string">'latest-offset'</span>,</span><br><span class="line"> <span class="string">'properties.bootstrap.servers'</span> = <span class="string">'logs-kafka1.com.cn:9092,logs-kafka2.com.cn:9092,logs-kafka3.com.cn:9092'</span>,</span><br><span class="line"> <span class="string">'properties.group.id'</span> = <span class="string">'yarn_flink_warn_logs_monitor_board'</span>,</span><br><span class="line"> <span class="string">'scan.topic-partition-discovery.interval'</span> = <span class="string">'10000 ms'</span>,</span><br><span class="line"> <span class="string">'format'</span> = <span class="string">'json'</span>,</span><br><span class="line"> <span class="string">'json.fail-on-missing-field'</span> = <span class="string">'false'</span>,</span><br><span class="line"> <span class="string">'json.ignore-parse-errors'</span> = <span class="string">'true'</span>,</span><br><span class="line"> <span class="string">'source.parallelism'</span> = <span class="string">'6'</span></span><br><span class="line">);</span><br><span class="line"></span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> printSink (</span><br><span class="line"> <span class="string">`source`</span> <span class="keyword">STRING</span>,</span><br><span class="line"> app_id <span class="keyword">STRING</span>,</span><br><span class="line"> logCount <span class="built_in">BIGINT</span>,</span><br><span class="line"> <span class="string">`window_start_time`</span> <span class="keyword">TIMESTAMP</span>(<span class="number">3</span>),</span><br><span class="line"> PRIMARY <span class="keyword">KEY</span> (app_id) <span class="keyword">NOT</span> <span class="keyword">ENFORCED</span></span><br><span class="line">) <span class="keyword">WITH</span> (</span><br><span class="line"> <span class="string">'connector'</span> = <span class="string">'print'</span>,</span><br><span class="line"> <span class="string">'sink.parallelism'</span> = <span class="string">'1'</span></span><br><span class="line">);</span><br><span class="line"></span><br><span class="line"><span class="keyword">INSERT</span> <span class="keyword">INTO</span> </span><br><span class="line"> printSink</span><br><span class="line"><span class="keyword">SELECT</span> </span><br><span class="line"> <span class="string">`source`</span>,</span><br><span class="line"> <span class="string">`tags`</span>.app_id <span class="keyword">as</span> app_id ,</span><br><span class="line"> <span class="keyword">count</span>(<span class="string">`tags`</span>.app_id) <span class="keyword">as</span> logTotalCount,</span><br><span class="line"> TUMBLE_START(windows_start_time, <span class="built_in">INTERVAL</span> <span class="string">'1'</span> <span class="keyword">day</span>) <span class="keyword">as</span> window_start_time</span><br><span class="line"><span class="keyword">FROM</span> </span><br><span class="line"> yarn_flink_warn_logs</span><br><span class="line"><span class="keyword">GROUP</span> <span class="keyword">BY</span> TUMBLE(windows_start_time, <span class="built_in">INTERVAL</span> <span class="string">'1'</span> <span class="keyword">day</span>), <span class="string">`tags`</span>.app_id, <span class="string">`source`</span>;</span><br></pre></td></tr></table></figure><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文从一个小需求（任务异常或者失败时排查问题不便）出发，开始调研业界常见的解决方案进行解决我们的需求，接着有对应的代码实现和效果展示。</p>]]></content>
<summary type="html">
<p>从一个小需求（任务异常或者失败时排查问题不便）出发，开始调研业界常见的解决方案进行解决我们的需求，接着有对应的代码实现和效果展示。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>关闭 Flink Checkpoint，引发 P3 故障</title>
<link href="http://www.54tianzhisheng.cn/2022/10/09/flink-disable-checkpoint/"/>
<id>http://www.54tianzhisheng.cn/2022/10/09/flink-disable-checkpoint/</id>
<published>2022-10-08T16:00:00.000Z</published>
<updated>2022-11-09T16:27:04.077Z</updated>
<content type="html"><![CDATA[<p>记录一个比较有意义的故障，没遇到的可以避坑，已经被坑过的只能握手🤝了。</p><a id="more"></a><h3 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h3><p>因阿里云提示机器有故障，会在第二天早高峰自动重启，按照之前运维操作，提前做好通知后，在集群非高峰期将机器踢出集群。踢出集群时该机器上运行的 TaskManager Pod 会挂掉，Flink 会在其他正常机器上申请新的 TaskManager 运行，期间会有任务的 failover。</p><p>操作后 10 来分钟看到公司大群有值班同事响应指标异常问题，加入到问题群聊中发现可能和剔除故障机器有关，于是加入排查，提供了当时受影响的任务信息，发现了异常指标对应的任务当时在消费 10 分钟之前的数据。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-140823.png" alt="Flink 任务消费到的数据时间"></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-141234.png" alt="Flink 任务消费延迟时间"></p><p>而这个消费延迟时长对于该作业的业务场景完全不能接受（延迟 10s 就会造成很大问题），最终延迟导致业务指标下跌，产生资损，故障最终定级 P3。</p><p>解决办法：业务方停了几个同步数据到 ES 索引的任务，然后重建了主链索引任务（任务同时加大并发），下游切换到新索引即恢复。</p><h3 id="故障根因分析过程"><a href="#故障根因分析过程" class="headerlink" title="故障根因分析过程"></a>故障根因分析过程</h3><p>事后找业务方线下讨论了一下，初步分析是 ES 压力大，导致 Flink 任务有反压，从而引起任务 Source 端消费 Kafka 数据导致的延迟。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-143142.png" alt="ES CPU 监控"></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-143238.jpg" alt="Flink 任务反压监控"></p><p>本以为结论就是剔除机器后，受影响任务比较多，任务 failover 后会从上一次 Checkpoint 保存的 Kafka offset 开始消费，这时候写到 ES 的数据量会比正常多 2min 左右的数据（Checkpoint 集群默认设置的 2min 执行一次），从而导致对 ES 有点压力进而影响到 Flink 任务出现反压情况。但是晚上业务方给我提供了一个另外受影响任务（这里称备链任务，主链任务就是那个消费 10 分钟之前数据的任务）的作业监控链接，也就是通过这个任务的监控信息才真正发现问题的根因。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-144829.png" alt=""></p><p>当时发现这个备链任务在故障后有大部分 TaskManager 的 task 是从 22 个小时之前的数据开始消费的，如下图监控所示：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-150438.png" alt=""></p><p>又有少部分 TaskManager 的 task 是正常消费的，如下图所示：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-150640.png" alt=""></p><p>在 failover 之后该备链任务的消费速度 TPS 从正常的两千增长到 12W/s</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-151423.png" alt=""></p><p>主链任务 failover 后所有的 task 消费比较正常（前文已截图），对比几个监控来看该备链任务在 failover 后明显消费有异常，即消费 22 小时之前的数据，同时看到任务启动运行时长就是 22 小时左右，相当于从任务开第一次始启动的时候指定的时间点开始消费了。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-153006.png" alt=""></p><p>既然主链和备链任务 failover 后的消费策略都不一样，所以我当时第一反应觉得是不是备链任务长期 Checkpoint 失败，然后 failover 后从早期完成的 Checkpoint 中保存的 offset 开始消费，从而导致的消费 22 小时之前的数据，但是咨询了业务方发现任务关闭了 Checkpoint。心想那只有可能是没打开 Kafka 的 auto.commit.offset 或者打开了但是自动提交一直失败了，我去看了 Kafka topic groupid 的 committed offset 监控发现正常，一直有在正常的提交 offset，所以也否定了刚才的猜想。另外就是觉得两个任务的消费策略这么大，是不是代码不一致或者配置不一致导致的，因为一开始问了业务方代码是否一致，回复是一致的，这里我也被坑了一下，后面在平台发现两个任务的代码 Git commitId 都不一样，那也就是意味着代码不一致。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-154650.png" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-154815.png" alt=""></p><p>接着就开始 review 业务方的项目代码，发现备链路任务关闭了 Checkpoint，而主链任务是开启 Checkpoint 的，同时代码消费方式是指定了 Offset 开始消费（根据时间配置查询 topic 的 offset，然后从指定的 offset 开始消费，这种消费方式其实和指定 timestamp 开始消费是一致的）。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-155112.jpg" alt=""></p><p>然后我就翻了下 KafkaSource Consumer 源码，如下图所示：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-155503.jpg" alt=""></p><p>Open 方法主要做了状态初始化相关逻辑，可以发现如果关闭了 Checkpoint，那么肯定不会有恢复的状态，也就是走到后面的逻辑，按照用户指定的 offset 开始消费。在踢出故障机器后，会申请新的 TaskManager 做状态初始化其实就是会走到上面的逻辑，也就导致 failover 后消费了很早之前的数据。排查到这，几乎根因已经出来了，接下来就是验证过程。</p><h3 id="根因验证"><a href="#根因验证" class="headerlink" title="根因验证"></a>根因验证</h3><p>测试代码：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Slf</span>4j</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">CheckpointTest</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> ParameterTool parameters = ParameterTool.fromArgs(args);</span><br><span class="line"> <span class="keyword">final</span> StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();</span><br><span class="line"> env.getConfig().setGlobalJobParameters(parameters);</span><br><span class="line"> env.setParallelism(parameters.getInt(<span class="string">"envParallelism"</span>, <span class="number">1</span>));</span><br><span class="line"></span><br><span class="line"> env.setRestartStrategy(RestartStrategies.fixedDelayRestart(</span><br><span class="line"> <span class="number">100</span>,</span><br><span class="line"> Time.of(<span class="number">10</span>, TimeUnit.SECONDS)</span><br><span class="line"> ));</span><br><span class="line"> <span class="comment">//手动关闭 checkpoint</span></span><br><span class="line"> env.getCheckpointConfig().disableCheckpointing();</span><br><span class="line"></span><br><span class="line"> FlinkKafkaConsumer<String> flinkKafkaConsumer = <span class="keyword">new</span> FlinkKafkaConsumer<>(parameters.get(<span class="string">"sourceTopic"</span>, <span class="string">"yarn_flink_log"</span>), <span class="keyword">new</span> SimpleStringSchema(), buildKafkaProps(parameters));</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (parameters.get(<span class="string">"timestamp"</span>) != <span class="keyword">null</span>) {</span><br><span class="line"> log.info(<span class="string">"Start from timestamp "</span> + parameters.getLong(<span class="string">"timestamp"</span>));</span><br><span class="line"> <span class="comment">// 手动指定 timestamp 开始消费，底层也是会根据 timestamp 去找这个 timestamp 下对应的 offset 数据</span></span><br><span class="line"> flinkKafkaConsumer.setStartFromTimestamp(parameters.getLong(<span class="string">"timestamp"</span>));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> env.addSource(flinkKafkaConsumer)</span><br><span class="line"> .setParallelism(parameters.getInt(<span class="string">"sourceParallelism"</span>, <span class="number">1</span>))</span><br><span class="line"> .print();</span><br><span class="line"> env.execute(<span class="string">"test checkpoint not enable"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> Properties <span class="title">buildKafkaProps</span><span class="params">(ParameterTool parameterTool)</span> </span>{</span><br><span class="line"> Properties props = parameterTool.getProperties();</span><br><span class="line"> props.put(<span class="string">"bootstrap.servers"</span>, parameterTool.get(<span class="string">"sourceBootstrapServers"</span>, <span class="string">"localhost:9092"</span>));</span><br><span class="line"> props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, <span class="keyword">true</span>);</span><br><span class="line"> props.put(<span class="string">"group.id"</span>, parameterTool.get(<span class="string">"sourceGroupId"</span>, String.valueOf(UUID.randomUUID())));</span><br><span class="line"> props.put(<span class="string">"flink.partition-discovery.interval-millis"</span>, <span class="string">"10000"</span>);</span><br><span class="line"> props.put(<span class="string">"auto.offset.reset"</span>, parameterTool.get(<span class="string">"sourceAutoOffsetReset"</span>, <span class="string">"latest"</span>));</span><br><span class="line"> <span class="keyword">return</span> props;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>当时指定的 timestamp 是 1666819431000 （对应 2022-10-27 05:23:51），然后在 2022-10-28 15:31 时手动 Kill 掉一个 TaskManager pod，任务发生 failover，新申请的 pod 开始消费 1666819431000 （对应 2022-10-27 05:23:51）时候的数据</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-11-09-160323.png" alt=""></p><p>新申请 TM 启动日志：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">2022-10-28 15:31:05,254 INFO org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] - Consumer subtask 0 will start reading the following 6 partitions from timestamp 1666819431000: [KafkaTopicPartition{topic='yxxx_log', partition=34}, KafkaTopicPartition{topic='yxxx_log', partition=4}, KafkaTopicPartition{topic='yxxx_log', partition=54}, KafkaTopicPartition{topic='yxxx_log', partition=24}, KafkaTopicPartition{topic='yxxx_log', partition=44}, KafkaTopicPartition{topic='yxxx_log', partition=14}]</span><br><span class="line"></span><br><span class="line">2022-10-28 15:31:05,267 INFO org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] - Consumer subtask 0 creating fetcher with offsets {KafkaTopicPartition{topic='yxxx_log', partition=34}=(782507410,-1), KafkaTopicPartition{topic='yxxx_log', partition=4}=(87934879,-1), KafkaTopicPartition{topic='yxxx_log', partition=54}=(1259521975,-1), KafkaTopicPartition{topic='yxxx_log', partition=24}=(4335766087,-1), KafkaTopicPartition{topic='yxxx_log', partition=44}=(4210596124,-1), KafkaTopicPartition{topic='yxxx_log', partition=14}=(350718902,-1)}.</span><br></pre></td></tr></table></figure><p>可以从日志发现是继续从 1666819431000 开始</p><h3 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h3><p>关闭 Checkpoint + 指定 timestamp 或者 Offset 消费 + 故障 failover 时会出现重复读数据问题</p><h3 id="思考"><a href="#思考" class="headerlink" title="思考"></a>思考</h3><p>1、核心任务一定要配置好消费延迟告警，及时发现和定位问题能减少故障损失</p><p>2、不要轻易漏掉某些指标，否则可能会忽略掉根因</p><p>3、关闭掉集群默认开启的配置需要谨慎，多充分测试</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<p>记录一个比较有意义的故障，没遇到的可以避坑，已经被坑过的只能握手🤝了。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>Flink Table Store ——从计算到存储提升流批统一端到端用户体验</title>
<link href="http://www.54tianzhisheng.cn/2022/05/12/flink-table-store/"/>
<id>http://www.54tianzhisheng.cn/2022/05/12/flink-table-store/</id>
<published>2022-05-11T16:00:00.000Z</published>
<updated>2022-05-11T16:06:12.752Z</updated>
<content type="html"><![CDATA[<p>该项目用于在 Flink 中为流处理和批处理构建动态表，支持超大流量的数据提取和及时的数据查询。</p><a id="more"></a><p>注意：该项目仍处于 beta 状态，正在快速发展，不建议直接在生产环境中使用它。</p><h3 id="Flink-Table-Store-介绍"><a href="#Flink-Table-Store-介绍" class="headerlink" title="Flink Table Store 介绍"></a>Flink Table Store 介绍</h3><p>在过去的几年里，得益于 Flink 社区众多的贡献者和用户，Apache Flink 已经成为最好的分布式计算引擎之一，尤其是在大规模有状态流处理方面。然而，当人们试图从他们的数据中实时获取洞察力时，仍然面临着一些挑战。在这些挑战中，一个突出的问题是缺乏满足所有计算模式的存储。</p><p>到目前为止，我们通常会部署一些存储系统来与 Flink 一起用于不同目的是很常见的。典型的设置是用于流处理的消息队列、用于批处理和即席查询的可查询文件系统/对象存储，以及用于查找的 K-V 存储。由于其复杂性和异构性，这种架构在数据质量和系统维护方面提出了挑战。这正在成为影响 Apache Flink 带来的流批统一端到端用户体验的一大问题。</p><p>Flink Table Store 的目标就是解决上述问题，这是该项目的重要一步。它将 Flink 的能力从计算扩展到存储领域。 因此，我们可以为用户提供更好的端到端体验。</p><p>Flink Table Store 旨在提供统一的存储抽象，让用户不必自己构建混合存储。具体来说，Flink Table Store 提供以下核心能力：</p><ul><li><p>支持超大数据集的存储，并允许以批处理和流方式读取和写入</p></li><li><p>支持毫秒级别延迟的流式查询</p></li><li><p>支持秒级别延迟的 Batch/OLAP 查询</p></li><li><p>默认支持流读增量快照，所以用户不需要自己解决组合不同存储的问题</p></li></ul><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/2022-05-11-151450.jpg" alt=""></p><p>在这个版本中，架构如上图所示：</p><ul><li><p>用户可以使用 Flink 将数据写入到 Table Store 中，既可以通过流式将数据库中捕获的变更日志写入，也可以通过从数据仓库等其他存储中批量加载数据后再写入</p></li><li><p>用户可以使用 Flink 以不同的方式查询 Table Store，包括流式查询和 Batch/OLAP 查询。 还值得注意的是，用户也可以使用其他引擎（例如 Apache Hive）从 Table Store 中查询</p></li><li><p>在底层，Table Store 使用混合存储架构，使用 Lake Store 存储历史数据，使用 Queue 系统（目前支持 Apache Kafka 集成）存储增量数据。 它为混合流式读取提供增量快照</p></li><li><p>Table Store 的 Lake Store 将数据作为列文件存储在文件系统/对象存储上，并使用 LSM 结构来支持大量的数据更新和高性能查询</p></li></ul><p>非常感谢以下系统的启发：Apache Iceberg 和 RocksDB。</p><h3 id="后续进展"><a href="#后续进展" class="headerlink" title="后续进展"></a>后续进展</h3><p>社区目前正在努力强化核心逻辑，稳定存储格式等，以使 Flink Table Store 可以投入生产。</p><p>在即将发布的 0.2.0 版本中，可以期待（至少）以下功能：</p><ul><li><p>生态：支持 Apache Hive Engine 的 Flink Table Store Reader</p></li><li><p>核心：支持自适应 Bucket 数量</p></li><li><p>核心：支持仅 append 的数据，Table Store 不仅仅局限于更新场景</p></li><li><p>核心：完善的 schema 变化</p></li><li><p>改进基于预览版得到的反馈</p></li></ul><p>从中期来看，你还可以期待：</p><ul><li><p>生态系统：支持 Trino、PrestoDB 和 Apache Spark 的 Flink Table Store Reader</p></li><li><p>Flink Table Store Service 会加速更新，提升查询性能</p></li></ul><h3 id="尝鲜"><a href="#尝鲜" class="headerlink" title="尝鲜"></a>尝鲜</h3><p>可以通过 <a href="https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/docs/try-table-store/quick-start/">https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/docs/try-table-store/quick-start/</a> 来尝试一下。</p><p>下载链接：<a href="https://flink.apache.org/downloads.html">https://flink.apache.org/downloads.html</a></p><blockquote><p>翻译原文链接：<a href="https://flink.apache.org/news/2022/05/11/release-table-store-0.1.0.html">https://flink.apache.org/news/2022/05/11/release-table-store-0.1.0.html</a></p><p>原文作者：李劲松 & 秦江杰</p></blockquote><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<p>该项目用于在 Flink 中为流处理和批处理构建动态表，支持超大流量的数据提取和及时的数据查询。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>Flink Iceberg Source 并行度推断源码解析</title>
<link href="http://www.54tianzhisheng.cn/2022/05/02/flink-iceberg-source-parallelism/"/>
<id>http://www.54tianzhisheng.cn/2022/05/02/flink-iceberg-source-parallelism/</id>
<published>2022-05-01T16:00:00.000Z</published>
<updated>2022-05-07T13:50:31.173Z</updated>
<content type="html"><![CDATA[<h3 id="批读-Iceberg"><a href="#批读-Iceberg" class="headerlink" title="批读 Iceberg"></a>批读 Iceberg</h3><p>Iceberg 提供了两个配置：</p><a id="more"></a><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Boolean> TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM =</span><br><span class="line"> ConfigOptions.key(<span class="string">"table.exec.iceberg.infer-source-parallelism"</span>)</span><br><span class="line"> .booleanType()</span><br><span class="line"> .defaultValue(<span class="keyword">true</span>)</span><br><span class="line"> .withDescription(<span class="string">"If is false, parallelism of source are set by config.\n"</span> +</span><br><span class="line"> <span class="string">"If is true, source parallelism is inferred according to splits number.\n"</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Integer> TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM_MAX =</span><br><span class="line"> ConfigOptions.key(<span class="string">"table.exec.iceberg.infer-source-parallelism.max"</span>)</span><br><span class="line"> .intType()</span><br><span class="line"> .defaultValue(<span class="number">100</span>)</span><br><span class="line"> .withDescription(<span class="string">"Sets max infer parallelism for source operator."</span>);</span><br></pre></td></tr></table></figure><ul><li>table.exec.iceberg.infer-source-parallelism：默认是 true，意味着 source 的并行度是根据推断来配置的，如果配置的 false 的话，那么并行度的配置是以配置的为准。</li><li>table.exec.iceberg.infer-source-parallelism.max： 默认是 100，source 算子的最大并行度。</li></ul><p>这两个参数只在 FileSource 的 inferParallelism 方法中调用：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">inferParallelism</span><span class="params">(FlinkInputFormat format, ScanContext context)</span> </span>{</span><br><span class="line"> <span class="comment">// 读取 table.exec.resource.default-parallelism 配置，默认值为 -1 </span></span><br><span class="line"> <span class="keyword">int</span> parallelism = readableConfig.get(ExecutionConfigOptions.TABLE_EXEC_RESOURCE_DEFAULT_PARALLELISM);</span><br><span class="line"> <span class="comment">// 读取 table.exec.iceberg.infer-source-parallelism 配置，默认是 true </span></span><br><span class="line"> <span class="keyword">if</span> (readableConfig.get(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM)) {</span><br><span class="line"> <span class="comment">// 读取 table.exec.iceberg.infer-source-parallelism.max 配置，默认是 100</span></span><br><span class="line"> <span class="keyword">int</span> maxInferParallelism = readableConfig.get(FlinkConfigOptions</span><br><span class="line"> .TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM_MAX);</span><br><span class="line"> Preconditions.checkState(</span><br><span class="line"> maxInferParallelism >= <span class="number">1</span>,</span><br><span class="line"> FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM_MAX.key() + <span class="string">" cannot be less than 1"</span>);</span><br><span class="line"> <span class="comment">//获取表的 splitNum </span></span><br><span class="line"> <span class="keyword">int</span> splitNum;</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> FlinkInputSplit[] splits = format.createInputSplits(<span class="number">0</span>);</span><br><span class="line"> splitNum = splits.length;</span><br><span class="line"> } <span class="keyword">catch</span> (IOException e) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> UncheckedIOException(<span class="string">"Failed to create iceberg input splits for table: "</span> + table, e);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> parallelism = Math.min(splitNum, maxInferParallelism);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (context.limit() > <span class="number">0</span>) {</span><br><span class="line"> <span class="keyword">int</span> limit = context.limit() >= Integer.MAX_VALUE ? Integer.MAX_VALUE : (<span class="keyword">int</span>) context.limit();</span><br><span class="line"> parallelism = Math.min(parallelism, limit);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// parallelism must be positive.</span></span><br><span class="line"> parallelism = Math.max(<span class="number">1</span>, parallelism);</span><br><span class="line"> <span class="keyword">return</span> parallelism;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>在下面代码获取到表的 splitNum</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line">FlinkInputSplit[] splits = format.createInputSplits(<span class="number">0</span>);</span><br><span class="line">splitNum = splits.length;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">public</span> FlinkInputSplit[] createInputSplits(<span class="keyword">int</span> minNumSplits) <span class="keyword">throws</span> IOException {</span><br><span class="line"> <span class="comment">// Called in Job manager, so it is OK to load table from catalog.</span></span><br><span class="line"> <span class="comment">// 加载 catalog </span></span><br><span class="line"> tableLoader.open();</span><br><span class="line"> <span class="keyword">final</span> ExecutorService workerPool = ThreadPools.newWorkerPool(<span class="string">"iceberg-plan-worker-pool"</span>, context.planParallelism());</span><br><span class="line"> <span class="keyword">try</span> (TableLoader loader = tableLoader) {</span><br><span class="line"> <span class="comment">// 加载表 </span></span><br><span class="line"> Table table = loader.loadTable();</span><br><span class="line"> <span class="comment">// 调用 </span></span><br><span class="line"> <span class="keyword">return</span> FlinkSplitPlanner.planInputSplits(table, context, workerPool);</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> workerPool.shutdown();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">static</span> FlinkInputSplit[] planInputSplits(Table table, ScanContext context, ExecutorService workerPool) {</span><br><span class="line"> <span class="comment">// 主要通过 planTasks 方法</span></span><br><span class="line"> <span class="keyword">try</span> (CloseableIterable<CombinedScanTask> tasksIterable = planTasks(table, context, workerPool)) {</span><br><span class="line"> List<CombinedScanTask> tasks = Lists.newArrayList(tasksIterable);</span><br><span class="line"> FlinkInputSplit[] splits = <span class="keyword">new</span> FlinkInputSplit[tasks.size()];</span><br><span class="line"> <span class="keyword">boolean</span> exposeLocality = context.exposeLocality();</span><br><span class="line"></span><br><span class="line"> Tasks.range(tasks.size())</span><br><span class="line"> .stopOnFailure()</span><br><span class="line"> .executeWith(exposeLocality ? workerPool : <span class="keyword">null</span>)</span><br><span class="line"> .run(index -> {</span><br><span class="line"> CombinedScanTask task = tasks.get(index);</span><br><span class="line"> String[] hostnames = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">if</span> (exposeLocality) {</span><br><span class="line"> hostnames = Util.blockLocations(table.io(), task);</span><br><span class="line"> }</span><br><span class="line"> splits[index] = <span class="keyword">new</span> FlinkInputSplit(index, task, hostnames);</span><br><span class="line"> });</span><br><span class="line"> <span class="keyword">return</span> splits;</span><br><span class="line"> } <span class="keyword">catch</span> (IOException e) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> UncheckedIOException(<span class="string">"Failed to process tasks iterable"</span>, e);</span><br><span class="line"> }</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>planTasks 方法中主要靠 scan.planTasks()，代码如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> CloseableIterable<CombinedScanTask> <span class="title">planTasks</span><span class="params">()</span> </span>{</span><br><span class="line"> CloseableIterable<FileScanTask> fileScanTasks = planFiles();</span><br><span class="line"> CloseableIterable<FileScanTask> splitFiles = TableScanUtil.splitFiles(fileScanTasks, targetSplitSize());</span><br><span class="line"> <span class="keyword">return</span> TableScanUtil.planTasks(splitFiles, targetSplitSize(), splitLookback(), splitOpenFileCost());</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 获取文件情况 </span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> CloseableIterable<FileScanTask> <span class="title">planFiles</span><span class="params">()</span> </span>{</span><br><span class="line"> Snapshot snapshot = snapshot();</span><br><span class="line"> <span class="keyword">if</span> (snapshot != <span class="keyword">null</span>) {</span><br><span class="line"> LOG.info(<span class="string">"Scanning table {} snapshot {} created at {} with filter {}"</span>, table,</span><br><span class="line"> snapshot.snapshotId(), DateTimeUtil.formatTimestampMillis(snapshot.timestampMillis()),</span><br><span class="line"> context.rowFilter());</span><br><span class="line"></span><br><span class="line"> Listeners.notifyAll(</span><br><span class="line"> <span class="keyword">new</span> ScanEvent(table.name(), snapshot.snapshotId(), context.rowFilter(), schema()));</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> planFiles(ops, snapshot,</span><br><span class="line"> context.rowFilter(), context.ignoreResiduals(), context.caseSensitive(), context.returnColumnStats());</span><br><span class="line"></span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> LOG.info(<span class="string">"Scanning empty table {}"</span>, table);</span><br><span class="line"> <span class="keyword">return</span> CloseableIterable.empty();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// split </span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> CloseableIterable<FileScanTask> <span class="title">splitFiles</span><span class="params">(CloseableIterable<FileScanTask> tasks, <span class="keyword">long</span> splitSize)</span> </span>{</span><br><span class="line"> Preconditions.checkArgument(splitSize > <span class="number">0</span>, <span class="string">"Invalid split size (negative or 0): %s"</span>, splitSize);</span><br><span class="line"></span><br><span class="line"> Iterable<FileScanTask> splitTasks = FluentIterable</span><br><span class="line"> .from(tasks)</span><br><span class="line"> .transformAndConcat(input -> input.split(splitSize));</span><br><span class="line"> <span class="comment">// Capture manifests which can be closed after scan planning</span></span><br><span class="line"> <span class="keyword">return</span> CloseableIterable.combine(splitTasks, tasks);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> CloseableIterable<CombinedScanTask> <span class="title">planTasks</span><span class="params">(CloseableIterable<FileScanTask> splitFiles,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">long</span> splitSize, <span class="keyword">int</span> lookback, <span class="keyword">long</span> openFileCost)</span> </span>{</span><br><span class="line"> Preconditions.checkArgument(splitSize > <span class="number">0</span>, <span class="string">"Invalid split size (negative or 0): %s"</span>, splitSize);</span><br><span class="line"> Preconditions.checkArgument(lookback > <span class="number">0</span>, <span class="string">"Invalid split planning lookback (negative or 0): %s"</span>, lookback);</span><br><span class="line"> Preconditions.checkArgument(openFileCost >= <span class="number">0</span>, <span class="string">"Invalid file open cost (negative): %s"</span>, openFileCost);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Check the size of delete file as well to avoid unbalanced bin-packing</span></span><br><span class="line"> Function<FileScanTask, Long> weightFunc = file -> Math.max(</span><br><span class="line"> file.length() + file.deletes().stream().mapToLong(ContentFile::fileSizeInBytes).sum(),</span><br><span class="line"> (<span class="number">1</span> + file.deletes().size()) * openFileCost);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> CloseableIterable.transform(</span><br><span class="line"> CloseableIterable.combine(</span><br><span class="line"> <span class="keyword">new</span> BinPacking.PackingIterable<>(splitFiles, splitSize, lookback, weightFunc, <span class="keyword">true</span>),</span><br><span class="line"> splitFiles),</span><br><span class="line"> BaseCombinedScanTask::<span class="keyword">new</span>);</span><br><span class="line"> }</span><br></pre></td></tr></table></figure><p>推断到表文件 split 的数量后，那么接下来是决定并行度大小的时候了：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">parallelism = Math.min(splitNum, maxInferParallelism);</span><br></pre></td></tr></table></figure><p>取配置的最大并行度和 split 数量的最小值，eg：设置的 source 最大并行度为 50，但是根据表文件划分出来的 split 数量为 40，那么 source 的并行度为 40。</p><p>如果没有配置 table.exec.iceberg.infer-source-parallelism 为 true 的话，那么就以 table.exec.resource.default-parallelism 的并行度为准（默认值是 -1）。</p><p>继续分析接下来的代码：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> (context.limit() > <span class="number">0</span>) {</span><br><span class="line"> <span class="keyword">int</span> limit = context.limit() >= Integer.MAX_VALUE ? Integer.MAX_VALUE : (<span class="keyword">int</span>) context.limit();</span><br><span class="line"> parallelism = Math.min(parallelism, limit);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>limit 像是 SQL 查询语句里面的 limit 的值，如果配置了 limit，那么也会参与并行度配置的计算的，eg：如果 limti 为 1，那么 parallelism 取前面 parallelism 的值与 1 两者的最小值。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// parallelism must be positive.</span></span><br><span class="line">parallelism = Math.max(<span class="number">1</span>, parallelism);</span><br><span class="line"><span class="keyword">return</span> parallelism;</span><br></pre></td></tr></table></figure><p>最后代码保证并行度的最小值为 1。</p><p>来看下 inferParallelism 方法的调用情况，只在 FileSource 类的 build() 方法调用：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> DataStream<RowData> <span class="title">build</span><span class="params">()</span> </span>{</span><br><span class="line"> Preconditions.checkNotNull(env, <span class="string">"StreamExecutionEnvironment should not be null"</span>);</span><br><span class="line"> FlinkInputFormat format = buildFormat();</span><br><span class="line"></span><br><span class="line"> ScanContext context = contextBuilder.build();</span><br><span class="line"> TypeInformation<RowData> typeInfo = FlinkCompatibilityUtil.toTypeInfo(FlinkSchemaUtil.convert(context.project()));</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (!context.isStreaming()) {</span><br><span class="line"> <span class="comment">// 只在 批 模式下生效 </span></span><br><span class="line"> <span class="keyword">int</span> parallelism = inferParallelism(format, context);</span><br><span class="line"> <span class="keyword">return</span> env.createInput(format, typeInfo).setParallelism(parallelism);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> StreamingMonitorFunction function = <span class="keyword">new</span> StreamingMonitorFunction(tableLoader, context);</span><br><span class="line"></span><br><span class="line"> String monitorFunctionName = String.format(<span class="string">"Iceberg table (%s) monitor"</span>, table);</span><br><span class="line"> String readerOperatorName = String.format(<span class="string">"Iceberg table (%s) reader"</span>, table);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> env.addSource(function, monitorFunctionName)</span><br><span class="line"> .transform(readerOperatorName, typeInfo, StreamingReaderOperator.factory(format));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>可以看 TestFlinkScanSql 测试类的 testInferedParallelism 方法进行测试：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testInferedParallelism</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> Table table = catalog.createTable(TableIdentifier.of(<span class="string">"default"</span>, <span class="string">"t"</span>), TestFixtures.SCHEMA, TestFixtures.SPEC);</span><br><span class="line"></span><br><span class="line"> TableLoader tableLoader = TableLoader.fromHadoopTable(table.location());</span><br><span class="line"> FlinkInputFormat flinkInputFormat = FlinkSource.forRowData().tableLoader(tableLoader).table(table).buildFormat();</span><br><span class="line"> ScanContext scanContext = ScanContext.builder().build();</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Empty table, infer parallelism should be at least 1</span></span><br><span class="line"> <span class="keyword">int</span> parallelism = FlinkSource.forRowData().inferParallelism(flinkInputFormat, scanContext);</span><br><span class="line"> Assert.assertEquals(<span class="string">"Should produce the expected parallelism."</span>, <span class="number">1</span>, parallelism);</span><br><span class="line"></span><br><span class="line"> GenericAppenderHelper helper = <span class="keyword">new</span> GenericAppenderHelper(table, fileFormat, TEMPORARY_FOLDER);</span><br><span class="line"> DataFile dataFile1 = helper.writeFile(TestHelpers.Row.of(<span class="string">"2020-03-20"</span>, <span class="number">0</span>),</span><br><span class="line"> RandomGenericData.generate(TestFixtures.SCHEMA, <span class="number">2</span>, <span class="number">0L</span>));</span><br><span class="line"> DataFile dataFile2 = helper.writeFile(TestHelpers.Row.of(<span class="string">"2020-03-21"</span>, <span class="number">0</span>),</span><br><span class="line"> RandomGenericData.generate(TestFixtures.SCHEMA, <span class="number">2</span>, <span class="number">0L</span>));</span><br><span class="line"> helper.appendToTable(dataFile1, dataFile2);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Make sure to generate 2 CombinedScanTasks</span></span><br><span class="line"> <span class="keyword">long</span> maxFileLen = Math.max(dataFile1.fileSizeInBytes(), dataFile2.fileSizeInBytes());</span><br><span class="line"> sql(<span class="string">"ALTER TABLE t SET ('read.split.open-file-cost'='1', 'read.split.target-size'='%s')"</span>, maxFileLen);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 2 splits (max infer is the default value 100 , max > splits num), the parallelism is splits num : 2</span></span><br><span class="line"> parallelism = FlinkSource.forRowData().inferParallelism(flinkInputFormat, scanContext);</span><br><span class="line"> Assert.assertEquals(<span class="string">"Should produce the expected parallelism."</span>, <span class="number">2</span>, parallelism);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 2 splits and limit is 1 , max infer parallelism is default 100，</span></span><br><span class="line"> <span class="comment">// which is greater than splits num and limit, the parallelism is the limit value : 1</span></span><br><span class="line"> parallelism = FlinkSource.forRowData().inferParallelism(flinkInputFormat, ScanContext.builder().limit(<span class="number">1</span>).build());</span><br><span class="line"> Assert.assertEquals(<span class="string">"Should produce the expected parallelism."</span>, <span class="number">1</span>, parallelism);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 2 splits and max infer parallelism is 1 (max < splits num), the parallelism is 1</span></span><br><span class="line"> Configuration configuration = <span class="keyword">new</span> Configuration();</span><br><span class="line"> configuration.setInteger(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM_MAX, <span class="number">1</span>);</span><br><span class="line"> parallelism = FlinkSource.forRowData()</span><br><span class="line"> .flinkConf(configuration)</span><br><span class="line"> .inferParallelism(flinkInputFormat, ScanContext.builder().build());</span><br><span class="line"> Assert.assertEquals(<span class="string">"Should produce the expected parallelism."</span>, <span class="number">1</span>, parallelism);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 2 splits, max infer parallelism is 1, limit is 3, the parallelism is max infer parallelism : 1</span></span><br><span class="line"> parallelism = FlinkSource.forRowData()</span><br><span class="line"> .flinkConf(configuration)</span><br><span class="line"> .inferParallelism(flinkInputFormat, ScanContext.builder().limit(<span class="number">3</span>).build());</span><br><span class="line"> Assert.assertEquals(<span class="string">"Should produce the expected parallelism."</span>, <span class="number">1</span>, parallelism);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 2 splits, infer parallelism is disabled, the parallelism is flink default parallelism 1</span></span><br><span class="line"> configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, <span class="keyword">false</span>);</span><br><span class="line"> parallelism = FlinkSource.forRowData()</span><br><span class="line"> .flinkConf(configuration)</span><br><span class="line"> .inferParallelism(flinkInputFormat, ScanContext.builder().limit(<span class="number">3</span>).build());</span><br><span class="line"> Assert.assertEquals(<span class="string">"Should produce the expected parallelism."</span>, <span class="number">1</span>, parallelism);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>注意⚠️：该代码模块是在 Iceberg 项目的 flink module 下</p><h3 id="流读-Iceberg"><a href="#流读-Iceberg" class="headerlink" title="流读 Iceberg"></a>流读 Iceberg</h3><p>并没有针对流读配置 Source 并行度</p><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/6YjyJYR">https://t.zsxq.com/6YjyJYR</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h3 id="批读-Iceberg"><a href="#批读-Iceberg" class="headerlink" title="批读 Iceberg"></a>批读 Iceberg</h3><p>Iceberg 提供了两个配置：</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>Flink Hive Source 并行度推断源码解析</title>
<link href="http://www.54tianzhisheng.cn/2022/05/01/flink-hive-source-parallelism/"/>
<id>http://www.54tianzhisheng.cn/2022/05/01/flink-hive-source-parallelism/</id>
<published>2022-04-30T16:00:00.000Z</published>
<updated>2022-05-07T13:50:31.165Z</updated>
<content type="html"><![CDATA[<h3 id="批读-Hive"><a href="#批读-Hive" class="headerlink" title="批读 Hive"></a>批读 Hive</h3><p>HiveOptions 中有两个配置</p><a id="more"></a><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Boolean> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM =</span><br><span class="line"> key(<span class="string">"table.exec.hive.infer-source-parallelism"</span>)</span><br><span class="line"> .defaultValue(<span class="keyword">true</span>)</span><br><span class="line"> .withDescription(</span><br><span class="line"> <span class="string">"If is false, parallelism of source are set by config.\n"</span> +</span><br><span class="line"> <span class="string">"If is true, source parallelism is inferred according to splits number.\n"</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Integer> TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MAX =</span><br><span class="line"> key(<span class="string">"table.exec.hive.infer-source-parallelism.max"</span>)</span><br><span class="line"> .defaultValue(<span class="number">1000</span>)</span><br><span class="line"> .withDescription(<span class="string">"Sets max infer parallelism for source operator."</span>);</span><br></pre></td></tr></table></figure><ul><li>table.exec.hive.infer-source-parallelism：默认值是 true，表示 source 的并行度是根据数据分区数和文件数推断的，如果设置为 false 的话表示并行度是以配置的为准</li><li>table.exec.hive.infer-source-parallelism.max：默认值是 1000，表示读取 Hive 数据的 source 最大并行度</li></ul><p>这两个参数只在 HiveParallelismInference 类中使用，观察到 HiveParallelismInference 类是专门针对 Hive 并行度配置的工具类，代码如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * A utility class to calculate parallelism for Hive connector considering various factors.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">HiveParallelismInference</span> </span>{</span><br><span class="line"></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> Logger LOG = LoggerFactory.getLogger(HiveParallelismInference.class);</span><br><span class="line"></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">final</span> ObjectPath tablePath;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">boolean</span> infer;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">int</span> inferMaxParallelism;</span><br><span class="line"></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">int</span> parallelism;</span><br><span class="line"></span><br><span class="line">HiveParallelismInference(ObjectPath tablePath, ReadableConfig flinkConf) {</span><br><span class="line"><span class="keyword">this</span>.tablePath = tablePath;</span><br><span class="line"> <span class="comment">// 获取 table.exec.hive.infer-source-parallelism 配置并赋值，</span></span><br><span class="line"><span class="keyword">this</span>.infer = flinkConf.get(HiveOptions.TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM);</span><br><span class="line"><span class="comment">// 获取 table.exec.hive.infer-source-parallelism.max 配置并赋值</span></span><br><span class="line"> <span class="keyword">this</span>.inferMaxParallelism = flinkConf.get(HiveOptions.TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MAX);</span><br><span class="line">Preconditions.checkArgument(</span><br><span class="line">inferMaxParallelism >= <span class="number">1</span>,</span><br><span class="line">HiveOptions.TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM_MAX.key() + <span class="string">" cannot be less than 1"</span>);</span><br><span class="line"> <span class="comment">// 获取 table.exec.resource.default-parallelism 配置</span></span><br><span class="line"><span class="keyword">this</span>.parallelism = flinkConf.get(ExecutionConfigOptions.TABLE_EXEC_RESOURCE_DEFAULT_PARALLELISM);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Apply limit to calculate the parallelism.</span></span><br><span class="line"><span class="comment"> * Here limit is the limit in query <code>SELECT * FROM xxx LIMIT [limit]</code>.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">limit</span><span class="params">(Long limit)</span> </span>{</span><br><span class="line"><span class="keyword">if</span> (limit != <span class="keyword">null</span>) {</span><br><span class="line">parallelism = Math.min(parallelism, (<span class="keyword">int</span>) (limit / <span class="number">1000</span>));</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// make sure that parallelism is at least 1</span></span><br><span class="line"><span class="keyword">return</span> Math.max(<span class="number">1</span>, parallelism);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"> <span class="comment">//根据</span></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Infer parallelism by number of files and number of splits.</span></span><br><span class="line"><span class="comment"> * If {<span class="doctag">@link</span> HiveOptions#TABLE_EXEC_HIVE_INFER_SOURCE_PARALLELISM} is not set this method does nothing.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function">HiveParallelismInference <span class="title">infer</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params">SupplierWithException<Integer, IOException> numFiles,</span></span></span><br><span class="line"><span class="function"><span class="params">SupplierWithException<Integer, IOException> numSplits)</span> </span>{</span><br><span class="line"> <span class="comment">//如果设置 table.exec.hive.infer-source-parallelism 为 false，则直接跳过了</span></span><br><span class="line"><span class="keyword">if</span> (!infer) {</span><br><span class="line"><span class="keyword">return</span> <span class="keyword">this</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">try</span> {</span><br><span class="line"><span class="comment">// `createInputSplits` is costly,</span></span><br><span class="line"><span class="comment">// so we try to avoid calling it by first checking the number of files</span></span><br><span class="line"><span class="comment">// which is the lower bound of the number of splits</span></span><br><span class="line"><span class="keyword">int</span> lowerBound = logRunningTime(<span class="string">"getNumFiles"</span>, numFiles);</span><br><span class="line"><span class="keyword">if</span> (lowerBound >= inferMaxParallelism) {</span><br><span class="line">parallelism = inferMaxParallelism;</span><br><span class="line"><span class="keyword">return</span> <span class="keyword">this</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">int</span> splitNum = logRunningTime(<span class="string">"createInputSplits"</span>, numSplits);</span><br><span class="line">parallelism = Math.min(splitNum, inferMaxParallelism);</span><br><span class="line">} <span class="keyword">catch</span> (IOException e) {</span><br><span class="line"><span class="keyword">throw</span> <span class="keyword">new</span> FlinkHiveException(e);</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> <span class="keyword">this</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">private</span> <span class="keyword">int</span> <span class="title">logRunningTime</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params">String operationName, SupplierWithException<Integer, IOException> supplier)</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"><span class="keyword">long</span> startTimeMillis = System.currentTimeMillis();</span><br><span class="line"><span class="keyword">int</span> result = supplier.get();</span><br><span class="line">LOG.info(</span><br><span class="line"><span class="string">"Hive source({}}) {} use time: {} ms, result: {}"</span>,</span><br><span class="line">tablePath,</span><br><span class="line">operationName,</span><br><span class="line">System.currentTimeMillis() - startTimeMillis,</span><br><span class="line">result);</span><br><span class="line"><span class="keyword">return</span> result;</span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>可以看到注释主要是 infer 方法去做的的并行度推断，该方法有两个参数 numFiles 和 numSplits，该方法只在HiveTableSource 类中的 getDataStream 方法中调用，可以查看下图：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-05-07-134123.jpg" alt=""></p><p>那就来看看这两个方法的实现：</p><p>getNumFiles 方法是用来获取 Hive 表分区下面的文件数量的:</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">int</span> <span class="title">getNumFiles</span><span class="params">(List<HiveTablePartition> partitions, JobConf jobConf)</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> <span class="keyword">int</span> numFiles = <span class="number">0</span>;</span><br><span class="line"> FileSystem fs = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">for</span> (HiveTablePartition partition : partitions) {</span><br><span class="line"> StorageDescriptor sd = partition.getStorageDescriptor();</span><br><span class="line"> org.apache.hadoop.fs.Path inputPath = <span class="keyword">new</span> org.apache.hadoop.fs.Path(sd.getLocation());</span><br><span class="line"> <span class="keyword">if</span> (fs == <span class="keyword">null</span>) {</span><br><span class="line"> fs = inputPath.getFileSystem(jobConf);</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// it's possible a partition exists in metastore but the data has been removed</span></span><br><span class="line"> <span class="keyword">if</span> (!fs.exists(inputPath)) {</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> numFiles += fs.listStatus(inputPath).length;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> numFiles;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>createInputSplits 方法是用来将 Hive 表分区下的文件分割成逻辑上的 InputSplit，这里是在 Flink Hive Connector 里面定义了一个 HiveSourceSplit 类来包装 InputSplit，包含了 Hive 表分区的信息。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">static</span> List<HiveSourceSplit> <span class="title">createInputSplits</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">int</span> minNumSplits,</span></span></span><br><span class="line"><span class="function"><span class="params"> List<HiveTablePartition> partitions,</span></span></span><br><span class="line"><span class="function"><span class="params"> JobConf jobConf)</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> List<HiveSourceSplit> hiveSplits = <span class="keyword">new</span> ArrayList<>();</span><br><span class="line"> FileSystem fs = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">for</span> (HiveTablePartition partition : partitions) {</span><br><span class="line"> StorageDescriptor sd = partition.getStorageDescriptor();</span><br><span class="line"> org.apache.hadoop.fs.Path inputPath = <span class="keyword">new</span> org.apache.hadoop.fs.Path(sd.getLocation());</span><br><span class="line"> <span class="keyword">if</span> (fs == <span class="keyword">null</span>) {</span><br><span class="line"> fs = inputPath.getFileSystem(jobConf);</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// it's possible a partition exists in metastore but the data has been removed</span></span><br><span class="line"> <span class="keyword">if</span> (!fs.exists(inputPath)) {</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> InputFormat format;</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> format = (InputFormat)</span><br><span class="line"> Class.forName(sd.getInputFormat(), <span class="keyword">true</span>, Thread.currentThread().getContextClassLoader()).newInstance();</span><br><span class="line"> } <span class="keyword">catch</span> (Exception e) {</span><br><span class="line"> <span class="keyword">throw</span> <span class="keyword">new</span> FlinkHiveException(<span class="string">"Unable to instantiate the hadoop input format"</span>, e);</span><br><span class="line"> }</span><br><span class="line"> ReflectionUtils.setConf(format, jobConf);</span><br><span class="line"> jobConf.set(INPUT_DIR, sd.getLocation());</span><br><span class="line"> <span class="comment">//<span class="doctag">TODO:</span> we should consider how to calculate the splits according to minNumSplits in the future.</span></span><br><span class="line"> org.apache.hadoop.mapred.InputSplit[] splitArray = format.getSplits(jobConf, minNumSplits);</span><br><span class="line"> <span class="keyword">for</span> (org.apache.hadoop.mapred.InputSplit inputSplit : splitArray) {</span><br><span class="line"> Preconditions.checkState(inputSplit <span class="keyword">instanceof</span> FileSplit,</span><br><span class="line"> <span class="string">"Unsupported InputSplit type: "</span> + inputSplit.getClass().getName());</span><br><span class="line"> hiveSplits.add(<span class="keyword">new</span> HiveSourceSplit((FileSplit) inputSplit, partition, <span class="keyword">null</span>));</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> hiveSplits;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>因为上面两个方法的执行可能需要一点时间，所以专门还写了一个 logRunningTime 记录其执行的时间。</p><p>如果文件数大于配置的最大并行度，那么作业的并行度直接以配置的最大并行度为准；否则取 InputSplit 个数与配置的最大并行度两者最小值。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">int</span> lowerBound = logRunningTime(<span class="string">"getNumFiles"</span>, numFiles);</span><br><span class="line"><span class="keyword">if</span> (lowerBound >= inferMaxParallelism) {</span><br><span class="line"> parallelism = inferMaxParallelism;</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">int</span> splitNum = logRunningTime(<span class="string">"createInputSplits"</span>, numSplits);</span><br><span class="line">parallelism = Math.min(splitNum, inferMaxParallelism);</span><br></pre></td></tr></table></figure><p>然后就是 limit 方法的限制并行度了：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Apply limit to calculate the parallelism.</span></span><br><span class="line"><span class="comment"> * Here limit is the limit in query <code>SELECT * FROM xxx LIMIT [limit]</code>.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">limit</span><span class="params">(Long limit)</span> </span>{</span><br><span class="line"> <span class="keyword">if</span> (limit != <span class="keyword">null</span>) {</span><br><span class="line"> parallelism = Math.min(parallelism, (<span class="keyword">int</span>) (limit / <span class="number">1000</span>));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// make sure that parallelism is at least 1</span></span><br><span class="line"> <span class="keyword">return</span> Math.max(<span class="number">1</span>, parallelism);</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>这个方法的注释的意思是根据查询语句的 limit 来配置并行度，判断前面得到的并行度与 limit/1000 的大小，取两者最小值。举个例子，前面判断这个 Hive 表分区有非常多的文件，比如 10001 个，那大于默认的最大值 1000，那么返回的并行度是 1000，但是因为查询 Hive 的 SQL 只是 100 条，那么这里取值得到的最小值是 0，最后通过 Math.max(1, parallelism) 返回的 source 并行度是 1。</p><p>注意⚠️：上面的并行度配置仅仅针对于批作业查 Hive 数据，不针对流读 Hive 数据。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-05-07-134319.jpg" alt=""></p><h3 id="流读-Hive"><a href="#流读-Hive" class="headerlink" title="流读 Hive"></a>流读 Hive</h3><p>在 HiveTableSource 类中的 getDataStream 方法中并没有针对流读配置 Source 并行度。</p><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/E6Mj6uv">https://t.zsxq.com/E6Mj6uv</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h3 id="批读-Hive"><a href="#批读-Hive" class="headerlink" title="批读 Hive"></a>批读 Hive</h3><p>HiveOptions 中有两个配置</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>如何提高 Flink K8s 集群资源使用率？</title>
<link href="http://www.54tianzhisheng.cn/2022/03/26/flink-k8s-pod-add-request-and-limit/"/>
<id>http://www.54tianzhisheng.cn/2022/03/26/flink-k8s-pod-add-request-and-limit/</id>
<published>2022-03-25T16:00:00.000Z</published>
<updated>2022-03-30T08:38:05.716Z</updated>
<content type="html"><![CDATA[<h3 id="问题"><a href="#问题" class="headerlink" title="问题"></a>问题</h3><p>在 flink on k8s 默认提交作业的命令下，我们会指定作业的 JM/TM 的 CPU 和 Memory，最后作业生成的 pod 它的 CPU/Memory 的 request/limit 都是一样的资源，但是作业真实运行时使用的资源远达不到 limit 的值，这样就会<strong>造成机器资源浪费</strong>（水位不高，但是机器又不能再申请 pod）。</p><a id="more"></a><p>比如下面命令：（指定了 TM 的资源，未指定 JM 资源）</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">./bin/flink run-application -p 1 -t kubernetes-application -c com.zhisheng.Test \</span><br><span class="line"> -Dkubernetes.cluster-id=flink-log-alert-test1 \</span><br><span class="line"> -Dtaskmanager.memory.process.size=6g \</span><br><span class="line"> -Djobmanager.memory.process.size=2g \</span><br><span class="line"> -Dkubernetes.jobmanager.cpu=0.5 \</span><br><span class="line"> -Dkubernetes.taskmanager.cpu=1 \</span><br><span class="line"> -Dtaskmanager.numberOfTaskSlots=1 \</span><br><span class="line"> ....</span><br></pre></td></tr></table></figure><p>JM:</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-03-29-144153.jpg" alt=""></p><p>TM:</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-03-29-144217.jpg" alt=""></p><h3 id="解决方案"><a href="#解决方案" class="headerlink" title="解决方案"></a>解决方案</h3><p>1、分别为 JM/TM 的 内存和 CPU 添加参数设置 request 和 limit，如果行得通的话，这种方式要增加 8 个参数才能满足需求，但因 Flink 内存模型使得单独设置内存的 request/limit 变得非常复杂，只能设置 CPU 参数，而且之前的参数也将变得不可以使用。</p><p>2、分别为 JM/TM 的 内存和 CPU 添加参数 limit 因子，用户配置的内存或者 CPU 的值默认为 request 的值，limit 因子必须 >= 1，这种方式需要增加四个参数，相比第一种方法这种方法较为简单，但是目前 YARN 集群用户的资源配置，大多数作业已经是有一定的资源浪费（申请的资源远大于实际使用的资源），如果使用该方式，用户作业无感迁移到 K8s 集群后，其实<strong>资源浪费问题并没有解决</strong>。</p><p>3、分别为 JM/TM 的 内存和 CPU 添加参数 request 因子，用户配置的内存或者 CPU 的值默认为 limit 的值，request 因子必须 <= 1，我们可以根据生产的数据配置一个合理的值，比如为 0.5。这种方式同样需要增加四个参数，但是这种方法对比第二种带来的好处是，<strong>大多数用户作业的资源配置将会更合理，机器同等资源能运行更多的 pod，从而可以提高机器的资源水位</strong>。</p><p>代码开发</p><p>1、在 KubernetesConfigOptions 增加配置 </p><p>KubernetesConfigOptions.java</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// jobmanager</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Double> JOB_MANAGER_CPU_REQUEST_FACTOR =</span><br><span class="line"> key(<span class="string">"kubernetes.jobmanager.cpu.request-factor"</span>)</span><br><span class="line"> .doubleType()</span><br><span class="line"> .defaultValue(<span class="number">1.0</span>)</span><br><span class="line"> .withDescription(</span><br><span class="line"> <span class="string">"The request factor of cpu used by job manager. "</span></span><br><span class="line"> + <span class="string">"The resources request cpu will be set to cpu * request-factor."</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Double> JOB_MANAGER_MEMORY_REQUEST_FACTOR =</span><br><span class="line"> key(<span class="string">"kubernetes.jobmanager.memory.request-factor"</span>)</span><br><span class="line"> .doubleType()</span><br><span class="line"> .defaultValue(<span class="number">1.0</span>)</span><br><span class="line"> .withDescription(</span><br><span class="line"> <span class="string">"The request factor of memory used by job manager. "</span></span><br><span class="line">+ <span class="string">"The resources request memory will be set to memory * request-factor."</span>);</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">// taskmanager</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Double> TASK_MANAGER_CPU_REQUEST_FACTOR =</span><br><span class="line"> key(<span class="string">"kubernetes.taskmanager.cpu.request-factor"</span>)</span><br><span class="line"> .doubleType()</span><br><span class="line"> .defaultValue(<span class="number">1.0</span>)</span><br><span class="line"> .withDescription(</span><br><span class="line"> <span class="string">"The request factor of cpu used by task manager. "</span></span><br><span class="line"> + <span class="string">"The resources request cpu will be set to cpu * request-factor."</span>);</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> ConfigOption<Double> TASK_MANAGER_MEMORY_REQUEST_FACTOR =</span><br><span class="line"> key(<span class="string">"kubernetes.taskmanager.memory.request-factor"</span>)</span><br><span class="line"> .doubleType()</span><br><span class="line"> .defaultValue(<span class="number">1.0</span>)</span><br><span class="line"> .withDescription(</span><br><span class="line"> <span class="string">"The request factor of memory used by task manager. "</span></span><br><span class="line"> + <span class="string">"The resources request memory will be set to memory * request-factor."</span>);</span><br></pre></td></tr></table></figure><p>2、在 KubernetesJobManagerParameters 和 KubernetesTaskManagerParameters 中分别提供获取参数的方法</p><p>KubernetesJobManagerParameters.java</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">getJobManagerCPURequestFactor</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> <span class="keyword">double</span> requestFactor =</span><br><span class="line"> flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_CPU_REQUEST_FACTOR);</span><br><span class="line"> checkArgument(</span><br><span class="line"> requestFactor <= <span class="number">1</span>,</span><br><span class="line"> <span class="string">"%s should be less than or equal to 1."</span>,</span><br><span class="line"> KubernetesConfigOptions.JOB_MANAGER_CPU_REQUEST_FACTOR.key());</span><br><span class="line"> <span class="keyword">return</span> requestFactor;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">getJobManagerMemoryRequestFactor</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> <span class="keyword">double</span> requestFactor =</span><br><span class="line"> flinkConfig.getDouble(KubernetesConfigOptions.JOB_MANAGER_MEMORY_REQUEST_FACTOR);</span><br><span class="line"> checkArgument(</span><br><span class="line"> requestFactor <= <span class="number">1</span>,</span><br><span class="line"> <span class="string">"%s should be less than or equal to 1."</span>,</span><br><span class="line"> KubernetesConfigOptions.JOB_MANAGER_MEMORY_REQUEST_FACTOR.key());</span><br><span class="line"> <span class="keyword">return</span> requestFactor;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>KubernetesTaskManagerParameters.java</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">getTaskManagerCPURequestFactor</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> <span class="keyword">double</span> requestFactor =</span><br><span class="line"> flinkConfig.getDouble(KubernetesConfigOptions.TASK_MANAGER_CPU_REQUEST_FACTOR);</span><br><span class="line"> checkArgument(</span><br><span class="line"> requestFactor <= <span class="number">1</span>,</span><br><span class="line"> <span class="string">"%s should be less than or equal to 1."</span>,</span><br><span class="line"> KubernetesConfigOptions.TASK_MANAGER_CPU_REQUEST_FACTOR.key());</span><br><span class="line"> <span class="keyword">return</span> requestFactor;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">getTaskManagerMemoryRequestFactor</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">final</span> <span class="keyword">double</span> requestFactor =</span><br><span class="line"> flinkConfig.getDouble(KubernetesConfigOptions.TASK_MANAGER_MEMORY_REQUEST_FACTOR);</span><br><span class="line"> checkArgument(</span><br><span class="line"> requestFactor <= <span class="number">1</span>,</span><br><span class="line"> <span class="string">"%s should be less than or equal to 1."</span>,</span><br><span class="line"> KubernetesConfigOptions.TASK_MANAGER_MEMORY_REQUEST_FACTOR.key());</span><br><span class="line"> <span class="keyword">return</span> requestFactor;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>3、KubernetesUtils.getResourceRequirements() 方法做如下改变，增加 request 因子参数</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-03-29-144238.jpg" alt=""></p><p>KubernetesUtils.java</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Get resource requirements from memory and cpu.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> mem Memory in mb.</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> memoryRequestFactor Memory request factor.</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> cpu cpu.</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> cpuRequestFactor cpu request factor.</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> externalResources external resources</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> KubernetesResource requirements.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">static</span> ResourceRequirements <span class="title">getResourceRequirements</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">int</span> mem,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">double</span> memoryRequestFactor,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">double</span> cpu,</span></span></span><br><span class="line"><span class="function"><span class="params"> <span class="keyword">double</span> cpuRequestFactor,</span></span></span><br><span class="line"><span class="function"><span class="params"> Map<String, Long> externalResources)</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="comment">//todo：cpu 和 内存分别设置一个因子，默认是 0.5，用户设置的资源配置为 limit；request = limit * 因子</span></span><br><span class="line"> <span class="keyword">final</span> Quantity cpuQuantity = <span class="keyword">new</span> Quantity(String.valueOf(cpu));</span><br><span class="line"> <span class="keyword">final</span> Quantity cpuRequestQuantity = <span class="keyword">new</span> Quantity(String.valueOf(cpu * cpuRequestFactor));</span><br><span class="line"> <span class="keyword">final</span> Quantity memQuantity = <span class="keyword">new</span> Quantity(mem + Constants.RESOURCE_UNIT_MB);</span><br><span class="line"> <span class="keyword">final</span> Quantity memRequestQuantity =</span><br><span class="line"> <span class="keyword">new</span> Quantity(((<span class="keyword">int</span>) (mem * memoryRequestFactor)) + Constants.RESOURCE_UNIT_MB);</span><br><span class="line"></span><br><span class="line"> ResourceRequirementsBuilder resourceRequirementsBuilder = <span class="keyword">new</span> ResourceRequirementsBuilder()</span><br><span class="line"> .addToRequests(Constants.RESOURCE_NAME_MEMORY, memRequestQuantity)</span><br><span class="line"> .addToRequests(Constants.RESOURCE_NAME_CPU, cpuRequestQuantity)</span><br><span class="line"> .addToLimits(Constants.RESOURCE_NAME_MEMORY, memQuantity)</span><br><span class="line"> .addToLimits(Constants.RESOURCE_NAME_CPU, cpuQuantity);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Add the external resources to resource requirement.</span></span><br><span class="line"> <span class="keyword">for</span> (Map.Entry<String, Long> externalResource: externalResources.entrySet()) {</span><br><span class="line"> <span class="keyword">final</span> Quantity resourceQuantity = <span class="keyword">new</span> Quantity(String.valueOf(externalResource.getValue()));</span><br><span class="line"> resourceRequirementsBuilder</span><br><span class="line"> .addToRequests(externalResource.getKey(), resourceQuantity)</span><br><span class="line"> .addToLimits(externalResource.getKey(), resourceQuantity);</span><br><span class="line"> LOG.info(<span class="string">"Request external resource {} with config key {}."</span>, resourceQuantity.getAmount(), externalResource.getKey());</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> resourceRequirementsBuilder.build();</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>4、在 InitJobManagerDecorator 和 InitTaskManagerDecorator 调用上面方法的地方做相应的修改</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-03-29-144238.jpg" alt=""></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-03-29-144323.jpg" alt=""></p><h3 id="最终效果"><a href="#最终效果" class="headerlink" title="最终效果"></a>最终效果</h3><p>可以在 flink-conf.yaml 中定义配置如下：</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">kubernetes.jobmanager.cpu.request-factor:</span> <span class="number">0.5</span></span><br><span class="line"><span class="string">kubernetes.jobmanager.memory.request-factor:</span> <span class="number">0.8</span></span><br><span class="line"><span class="string">kubernetes.taskmanager.cpu.request-factor:</span> <span class="number">0.5</span></span><br><span class="line"><span class="string">kubernetes.taskmanager.memory.request-factor:</span> <span class="number">0.8</span></span><br></pre></td></tr></table></figure><p>当然，用户的作业提交参数中也可以使用上面的参数进行覆盖，最终效果如下：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2022-03-29-144340.jpg" alt=""></p><h3 id="关注我"><a href="#关注我" class="headerlink" title="关注我"></a>关注我</h3><p>微信公众号：<strong>zhisheng</strong></p><p>另外我自己整理了些 Flink 的学习资料，目前已经全部放到微信公众号（zhisheng）了，你可以回复关键字：<strong>Flink</strong> 即可无条件获取到。另外也可以加我微信 你可以加我的微信：<strong>yuanblog_tzs</strong>，探讨技术！</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-17-143454.jpg" alt=""></p><p>更多私密资料请加入知识星球！</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p><h3 id="Github-代码仓库"><a href="#Github-代码仓库" class="headerlink" title="Github 代码仓库"></a>Github 代码仓库</h3><p><a href="https://github.com/zhisheng17/flink-learning/">https://github.com/zhisheng17/flink-learning/</a></p><p>以后这个项目的所有代码都将放在这个仓库里，包含了自己学习 flink 的一些 demo 和博客</p>]]></content>
<summary type="html">
<h3 id="问题"><a href="#问题" class="headerlink" title="问题"></a>问题</h3><p>在 flink on k8s 默认提交作业的命令下，我们会指定作业的 JM/TM 的 CPU 和 Memory，最后作业生成的 pod 它的 CPU/Memory 的 request/limit 都是一样的资源，但是作业真实运行时使用的资源远达不到 limit 的值，这样就会<strong>造成机器资源浪费</strong>（水位不高，但是机器又不能再申请 pod）。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>宕机一台机器，结果一百多个 Flink 作业挂了</title>
<link href="http://www.54tianzhisheng.cn/2021/11/11/flink-akka-framesize/"/>
<id>http://www.54tianzhisheng.cn/2021/11/11/flink-akka-framesize/</id>
<published>2021-11-10T16:00:00.000Z</published>
<updated>2021-11-11T15:33:45.652Z</updated>
<content type="html"><![CDATA[<h3 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h3><p>因宕机了一台物理机器，实时集群不少作业发生 failover，其中大部分作业都能 failover 成功，某个部门的部分作业一直在 failover，始终未成功，到 WebUI 查看作业异常日志如下：</p><a id="more"></a><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">2021-11-09 16:01:11</span><br><span class="line">java.util.concurrent.CompletionException: java.lang.reflect.UndeclaredThrowableException</span><br><span class="line"> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)</span><br><span class="line"> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)</span><br><span class="line"> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)</span><br><span class="line"> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)</span><br><span class="line"> at java.util.concurrent.FutureTask.run(FutureTask.java:266)</span><br><span class="line"> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)</span><br><span class="line"> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)</span><br><span class="line"> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)</span><br><span class="line"> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)</span><br><span class="line"> at java.lang.Thread.run(Thread.java:748)</span><br><span class="line">Caused by: java.lang.reflect.UndeclaredThrowableException</span><br><span class="line"> at com.sun.proxy.$Proxy54.submitTask(Unknown Source)</span><br><span class="line"> at org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway.submitTask(RpcTaskManagerGateway.java:72)</span><br><span class="line"> at org.apache.flink.runtime.executiongraph.Execution.lambda$deploy$10(Execution.java:756)</span><br><span class="line"> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)</span><br><span class="line"> ... 7 more</span><br><span class="line">Caused by: java.io.IOException: The rpc invocation size 56424326 exceeds the maximum akka framesize.</span><br><span class="line"> at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.createRpcInvocationMessage(AkkaInvocationHandler.java:276)</span><br><span class="line"> at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.invokeRpc(AkkaInvocationHandler.java:205)</span><br><span class="line"> at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.invoke(AkkaInvocationHandler.java:134)</span><br><span class="line"> ... 11 more</span><br></pre></td></tr></table></figure><h3 id="解决异常过程"><a href="#解决异常过程" class="headerlink" title="解决异常过程"></a>解决异常过程</h3><p>从上面的异常日志中我们提取到关键信息：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Caused by: java.io.IOException: The rpc invocation size 56424326 exceeds the maximum akka framesize.</span><br></pre></td></tr></table></figure><p>看起来是 RPC 的消息大小超过了默认的 akka framesize 的最大值了，所以我们来了解一下这个值的默认值，从 <a href="https://nightlies.apache.org/flink/flink-docs-release-1.12/deployment/config.html#akka-framesize">官网</a> 我们可以看的到该值的默认大小为 “10485760b”，并且该参数的描述为：</p><p><img src="https://tva1.sinaimg.cn/large/008i3skNly1gwbm72hedkj31i806imya.jpg" alt=""></p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Maximum size of messages which are sent between the JobManager and the TaskManagers. If Flink fails because messages exceed this limit, then you should increase it. The message size requires a size-unit specifier.</span><br></pre></td></tr></table></figure><p>翻译过来的意思就是：这个参数是 JobManager 和 TaskManagers 之间通信允许的最大消息大小，如果 Flink 作业因为通信消息大小超过了该值，你可以通过增加该值的大小来解决，该参数需要指定一个单位。</p><h3 id="分析原因"><a href="#分析原因" class="headerlink" title="分析原因"></a>分析原因</h3><p>Flink 使用 Akka 作为组件（JobManager/TaskManager/ResourceManager）之间的 RPC 框架，在 JobManager 和 TaskManagers 之间发送的消息的最大大小默认为 10485760b，如果消息超过这个限制就会失败，报错。这个可以看下抛出异常处的源码：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">protected RpcInvocation createRpcInvocationMessage(String methodName, Class<?>[] parameterTypes, Object[] args) throws IOException {</span><br><span class="line"> Object rpcInvocation;</span><br><span class="line"> if (this.isLocal) {</span><br><span class="line"> rpcInvocation = new LocalRpcInvocation(methodName, parameterTypes, args);</span><br><span class="line"> } else {</span><br><span class="line"> try {</span><br><span class="line"> RemoteRpcInvocation remoteRpcInvocation = new RemoteRpcInvocation(methodName, parameterTypes, args);</span><br><span class="line"> if (remoteRpcInvocation.getSize() > this.maximumFramesize) {</span><br><span class="line"> // 异常所在位置</span><br><span class="line"> throw new IOException("The rpc invocation size exceeds the maximum akka framesize.");</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> rpcInvocation = remoteRpcInvocation;</span><br><span class="line"> } catch (IOException var6) {</span><br><span class="line"> LOG.warn("Could not create remote rpc invocation message. Failing rpc invocation because...", var6);</span><br><span class="line"> throw var6;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> return (RpcInvocation)rpcInvocation;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>至于为什么 JobManager 和 TaskManager 之间的 RPC 消息大小会如此之大，初步的解释是在 task 出现异常之后，它需要调用 updateTaskExecutionState(TaskExecutionState，taskExecutionState) 这个 RPC 接口去通知 Flink Jobmanager 去改变对应 task 的状态并且重启 task。但是呢，taskExecutionState 这个参数里面有个 error 属性，当我的 task 打出来的错误栈太多的时候，在序列化的之后超过了 rpc 接口要求的最大数据大小（也就是 maximum akka framesize），导致调用 updateTaskExecutionState 这个 rpc 接口失败，Jobmanager 无法获知这个 task 已经处于 fail 的状态，也无法重启，然后就导致了一系列连锁反应。</p><h3 id="解决办法"><a href="#解决办法" class="headerlink" title="解决办法"></a>解决办法</h3><p>任务停止，在 <code>flink-conf.yaml</code> 中加入 <code>akka.framesize</code> 参数，调大该值。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">akka.framesize: "62914560b"</span><br></pre></td></tr></table></figure><p>然后将任务重启，可以观察 Jobmanager Configration 看看参数是否生效。</p>]]></content>
<summary type="html">
<h3 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h3><p>因宕机了一台物理机器，实时集群不少作业发生 failover，其中大部分作业都能 failover 成功，某个部门的部分作业一直在 failover，始终未成功，到 WebUI 查看作业异常日志如下：</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>实时平台如何管理多个 Flink 版本？—— 为啥会出现多个版本？</title>
<link href="http://www.54tianzhisheng.cn/2021/09/26/realtime-platform-flink-version/"/>
<id>http://www.54tianzhisheng.cn/2021/09/26/realtime-platform-flink-version/</id>
<published>2021-09-25T16:00:00.000Z</published>
<updated>2021-11-14T03:12:49.099Z</updated>
<content type="html"><![CDATA[<h3 id="为啥会出现多个版本？"><a href="#为啥会出现多个版本？" class="headerlink" title="为啥会出现多个版本？"></a>为啥会出现多个版本？</h3><a id="more"></a><ul><li><p><strong>Flink 社区</strong>本身迭代速度非常快，目前阿里云有一大波的人专职做 Flink 开源，另外还拥有活跃的社区贡献者，所以功能开发较快，bug 修复速度较快，几乎每 4 个月一个大版本，每个大版本之间迭代的功能非常多，代码变动非常大，API 接口变动也大，动不动就干翻自己了。</p></li><li><p>社区迭代快就快呗，为什么<strong>公司</strong>也要要不断跟着社区鼻子走？社区迭代快意味着功能多，修复的 bug 多，相对于早期版本意味着稳定性也高些。除了国内一二线公司有特别多的专职人去负责这块，大多数中小公司最简单最快捷体验到稳定性最高、功能性最多、性能最好的 Flink 版本无非是直接使用最新的 Flink 版本。举个例子：Flink SQL 从最早期（1.9）的功能、性能到目前 1.14，差别真的大很多，优化了特别多的地方，增强了很多功能。原先使用 Flink SQL 完成一个流处理任务非常麻烦，还不如直接写几十行代码来的快，目前我情愿写 SQL 去处理一个流任务。那么自然会跟着升级到新版本。</p></li><li><p><strong>用户 A</strong> 问 Flink SQL 支持单独设置并行度吗？<strong>用户 B</strong> 问实时平台现在支持 Flink 1.13 版本的 Window TVF？这个要 Flink xxx 版本才能支持，要不你升级一下 Flink 版本到 xxx？这样就能支持了，类似的场景还有很多，对于<strong>中小公司的实时平台负责人</strong>来说，这无非最省事；对于<strong>大公司的负责实时开发的人</strong>来说，这无疑是一个噩梦，每次升级新版本都要将在老版本开发的各种功能都想尽办法移植到新版本上来，碰到 API 接口变动大的无非相当于重写了，或者将新版本的某些特别需要的功能通过打 patch 的方式打到老版本里面去。</p></li><li><p>新版本香是真的香，可是为啥有的人不用呢？问题就是，实时作业大多数是长期运行的，如果一个作业没啥错误，在生产运行的好好的，也不出啥故障，稳定性和性能也都能接受（并不是所有作业数据量都很大，会遇到性能问题），那么<strong>用户</strong>为啥要使用新版本？用户才不管你新版本功能多牛逼，性能多屌呢，老子升级还要改依赖版本、改接口代码、测试联调、性能测试（谁知道你说的性能提升是不是吹牛逼的）、稳定性测试（可能上线双跑一段时间验证），这些不需要时间呀，你叫我升级就升级，滚犊子吧，你知道我还有多少业务需求要做吗？</p></li></ul><p>那么就落下这个场地了，又要使用新版本的功能去解决问题，老作业的用户跟他各种扯皮也打动不了他升级作业的版本，那么自然就不断的出现了多个版本了。</p><p>这样，如果不对版本做好规划，那么摊子就逐渐越来越大，越来越难收拾了？</p><p>那么该如何管理公司的 Flink 版本？如果管理和兼容多个 Flink 版本的作业提交？如何兼容 Jar 包和 SQL 作业的提交</p><h3 id="怎么管理多个-Flink-版本的作业提交？"><a href="#怎么管理多个-Flink-版本的作业提交？" class="headerlink" title="怎么管理多个 Flink 版本的作业提交？"></a>怎么管理多个 Flink 版本的作业提交？</h3><p>尽请期待下篇文章</p>]]></content>
<summary type="html">
<h3 id="为啥会出现多个版本？"><a href="#为啥会出现多个版本？" class="headerlink" title="为啥会出现多个版本？"></a>为啥会出现多个版本？</h3>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>《Flink 实战与性能优化》—— 基于 Flink 的实时监控告警系统</title>
<link href="http://www.54tianzhisheng.cn/2021/08/21/flink-in-action-12.3/"/>
<id>http://www.54tianzhisheng.cn/2021/08/21/flink-in-action-12.3/</id>
<published>2021-08-20T16:00:00.000Z</published>
<updated>2022-02-20T13:15:20.296Z</updated>
<content type="html"><![CDATA[<h2 id="12-3-基于-Flink-的实时监控告警系统"><a href="#12-3-基于-Flink-的实时监控告警系统" class="headerlink" title="12.3 基于 Flink 的实时监控告警系统"></a>12.3 基于 Flink 的实时监控告警系统</h2><p>在如今微服务、云原生等技术盛行的时代，当谈到说要从 0 开始构建一个监控系统，大家无非就首先想到三个词：Metrics、Tracing、Logging。</p><a id="more"></a><h3 id="12-3-1-监控系统的诉求"><a href="#12-3-1-监控系统的诉求" class="headerlink" title="12.3.1 监控系统的诉求"></a>12.3.1 监控系统的诉求</h3><p>国外一篇比较火的文章 <a href="http://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html">Metrics, Tracing, and Logging</a> 内有个图很好的总结了一个监控系统的诉求，分别是 Metrics、Logging、Tracing，如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-12-08-132227.png" alt="监控系统的诉求"></p><p>Metrics 的特点：它自己提供了五种基本的度量类型 Gauge、Counter、Histogram、Timer、Meter。</p><p>Tracing 的特点：提供了一个请求从接收到处理完毕整个生命周期的跟踪路径，通常请求都是在分布式的系统中处理，所以也叫做分布式链路追踪。</p><p>Logging 的特点：提供项目应用运行的详细信息，例如方法的入参、运行的异常记录等。</p><p>这三者在监控系统中缺一不可，它们之间的关系是：基于 Metrics 的异常告警事件，然后通过 Tracing 定位问题可疑模块，根据模块详细的日志定位到错误根源，最后再返回来调整 Metrics 的告警规则，以便下次更早的预警，提前预防出现此类问题。</p><h3 id="12-3-2-监控系统包含的内容"><a href="#12-3-2-监控系统包含的内容" class="headerlink" title="12.3.2 监控系统包含的内容"></a>12.3.2 监控系统包含的内容</h3><p>针对提到的三个点，笔者找到国内外的开源监控系统做了对比，发现真正拥有全部功能的比较少，有的系统比较专注于 Logging、有的系统比较专注于 Tracing，而大部分其他的监控系统无非是只是监控系统的一部分，比如是作为一款数据库存储监控数据、作为一个可视化图表的系统去展示各种各样的监控数据信息。</p><p>拿 Logging 来说，开源用的最多最火的技术栈是 ELK，Tracing 这块有 Skywalking、Pinpoint 等技术，它们的对比如 <a href="https://mp.weixin.qq.com/s/_XE-gCJnDY3-yEK4xmWiMg">APM 巅峰对决：Skywalking PK Pinpoint</a> 一文介绍。而存储监控数据的时序数据库那就比较多了，常见的比如 InfluxDB、Prometheus、OpenTSDB 等，它们之间的对比介绍如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-31-%E6%97%B6%E5%BA%8F%E6%95%B0%E6%8D%AE%E5%BA%93%E6%AF%94%E8%BE%83.png" alt="常见时序数据库对比"></p><p>监控可视化图表的开源系统个人觉得最好看的就是 Grafana，在 8.2 节中搭建 Flink 监控系统的数据展示也是用的 Grafana，当然还可以利用 ECharts、BizCharts 等数据图表库做二次开发来适配公司的数据展示图表。</p><p>上面说了这么多，这里笔者根据自己的工作经验先谈谈几点自己对监控系统的心得：</p><ol><li><strong>告警是监控系统第一入口，图表展示体现监控的价值</strong>：告警是唯一可以第一时间反映运行状态，它承担着系统与人之间的沟通桥梁，通常告警消息又会携带链接跳转到图表展示，它作为第一入口并衔接上了整个监控系统。</li><li><strong>数据采集是监控的源泉</strong>：数据采集是监控系统的源泉，如果采集的数据是错误的，将导致后面的链路（告警、数据展示）全处于无效状态，所以千万千万要保证数据采集的准确性和完整性。</li><li><strong>数据存储是监控最大挑战</strong>：当机器、系统应用和监控指标等变得越多来多时，采集上来的数据是爆炸性增长的，将海量的监控数据实时存储到任何一个数据库，挑战都是不小的。</li></ol><p>说完心得再来讲解到底一个监控系统真正该包含哪些东西呢？笔者觉得首先分 6 层：数据采集层、数据传输层、数据计算层、告警、数据存储、数据展示，如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-12-11-012408.png" alt="监控系统分层"></p><p>监控系统这六层的主要功能分别是：</p><ul><li>数据采集层：该层主要功能就是去采集各种各样的数据，比如 Metrics、Logging、Tracing 数据。</li><li>数据传输层：该层主要功能就是传输采集到的监控数据，一般使用消息队列居多，比如 Kafka、RocketMQ 等。</li><li>数据计算层：该层的主要功能是将采集到的数据进行数据清洗和计算，一般采用 Flink、Spark 等计算引擎来处理。</li><li>告警：该层其实也属于数据计算层，但是因为告警涉及的内容太多，比如告警规则、告警计算、告警通知等，所以可以单独作为一个重要点来讲。</li><li>数据存储层：该层的主要功能是存储所有的监控数据，为后面的数据可视化提供数据源。</li><li>数据展示层：该层的主要功能是将监控数据通过可视化图表来展示出来，通过图表可以知道服务器、应用的运行状态。</li></ul><h3 id="12-3-3-Metrics／Tracing／Logging-数据实时采集"><a href="#12-3-3-Metrics／Tracing／Logging-数据实时采集" class="headerlink" title="12.3.3 Metrics／Tracing／Logging 数据实时采集"></a>12.3.3 Metrics／Tracing／Logging 数据实时采集</h3><p>在 12.1 节中讲解了日志数据如何采集，那么对于 Metrics 和 Tracing 数据该怎么采集呢？</p><h4 id="Metrics"><a href="#Metrics" class="headerlink" title="Metrics"></a>Metrics</h4><h4 id="Tracing"><a href="#Tracing" class="headerlink" title="Tracing"></a>Tracing</h4><h3 id="12-3-4-消息队列如何撑住高峰流量"><a href="#12-3-4-消息队列如何撑住高峰流量" class="headerlink" title="12.3.4 消息队列如何撑住高峰流量"></a>12.3.4 消息队列如何撑住高峰流量</h3><h4 id="RabbitMQ"><a href="#RabbitMQ" class="headerlink" title="RabbitMQ"></a>RabbitMQ</h4><h4 id="Kafka"><a href="#Kafka" class="headerlink" title="Kafka"></a>Kafka</h4><h4 id="RocketMQ"><a href="#RocketMQ" class="headerlink" title="RocketMQ"></a>RocketMQ</h4><h3 id="12-3-5-指标数据实时计算"><a href="#12-3-5-指标数据实时计算" class="headerlink" title="12.3.5 指标数据实时计算"></a>12.3.5 指标数据实时计算</h3><h3 id="12-3-6-提供及时且准确的根因分析告警"><a href="#12-3-6-提供及时且准确的根因分析告警" class="headerlink" title="12.3.6 提供及时且准确的根因分析告警"></a>12.3.6 提供及时且准确的根因分析告警</h3><h4 id="告警本质"><a href="#告警本质" class="headerlink" title="告警本质"></a>告警本质</h4><h4 id="告警通知对象"><a href="#告警通知对象" class="headerlink" title="告警通知对象"></a>告警通知对象</h4><h4 id="告警通知方式"><a href="#告警通知方式" class="headerlink" title="告警通知方式"></a>告警通知方式</h4><h4 id="告警规则的设计"><a href="#告警规则的设计" class="headerlink" title="告警规则的设计"></a>告警规则的设计</h4><h4 id="根因分析告警"><a href="#根因分析告警" class="headerlink" title="根因分析告警"></a>根因分析告警</h4><h4 id="告警事件解耦"><a href="#告警事件解耦" class="headerlink" title="告警事件解耦"></a>告警事件解耦</h4><h4 id="告警工单记录告警解决过程"><a href="#告警工单记录告警解决过程" class="headerlink" title="告警工单记录告警解决过程"></a>告警工单记录告警解决过程</h4><h3 id="12-3-7-AIOps-智能运维道路探索"><a href="#12-3-7-AIOps-智能运维道路探索" class="headerlink" title="12.3.7 AIOps 智能运维道路探索"></a>12.3.7 AIOps 智能运维道路探索</h3><h3 id="12-3-8-如何保障高峰流量实时写入存储系统的稳定性"><a href="#12-3-8-如何保障高峰流量实时写入存储系统的稳定性" class="headerlink" title="12.3.8 如何保障高峰流量实时写入存储系统的稳定性"></a>12.3.8 如何保障高峰流量实时写入存储系统的稳定性</h3><h3 id="12-3-9-监控数据使用可视化图表展示"><a href="#12-3-9-监控数据使用可视化图表展示" class="headerlink" title="12.3.9 监控数据使用可视化图表展示"></a>12.3.9 监控数据使用可视化图表展示</h3><h4 id="Grafana-介绍"><a href="#Grafana-介绍" class="headerlink" title="Grafana 介绍"></a>Grafana 介绍</h4><h3 id="12-3-10-小结与反思"><a href="#12-3-10-小结与反思" class="headerlink" title="12.3.10 小结与反思"></a>12.3.10 小结与反思</h3><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/IeAYbEy">https://t.zsxq.com/IeAYbEy</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p><h3 id="本章总结"><a href="#本章总结" class="headerlink" title="本章总结"></a>本章总结</h3><p>本章讲了三个公司常见场景案例：日志处理、去重、监控告警。在日志处理案例中讲解了日志的采集的需求，以及分析了整个日志采集案例的架构设计分层，关于日志采集工具的选型在 11.5 节中有讲过，所以该案例中就直接讲述采集工具的使用和安装，并将采集到的数据发送到 Kafka，然后使用 Flink 清洗日志数据并将异常日志告警通知到相应负责人，接着将数据写入到 ElasticSearch，最后通过 Kibana 可以展示和搜索日志数据。</p><p>在百亿数据实时去重案例中通过对比通用解决方法、使用 BloomFilter、使用 HBase 和使用 Flink KeyedState 几种方案来分析实时去重的解决方案，并在该案例中还提及到如何去优化去重的效果。</p><p>在实时监控告警案例中讲述了公司通用的监控告警需求，包括 Metrics、Logging、Tracing，然后设计出整个监控告警系统的架构分层和技术选型，其中分层包含数据采集层、数据传输层、数据计算层、数据存储层、数据展示层。关于数据采集工具、消息队列、数据存储中间件、数据展示的技术选型，笔者也分别做了对比介绍，以便大家可以根据自己公司的情况做架构选型。另外针对实时告警这块，笔者也详述了很多，包括告警规则的设计、告警通知对象、告警通知方式、告警消息收敛、告警消息根因分析等，最后还讲了对监控告警的展望，希望能够在 AIOps 做更多的探索，对于这块的内容，笔者在社区钉钉群做过视频直播，大家可以去笔者的博客查看录播的视频。</p>]]></content>
<summary type="html">
<h2 id="12-3-基于-Flink-的实时监控告警系统"><a href="#12-3-基于-Flink-的实时监控告警系统" class="headerlink" title="12.3 基于 Flink 的实时监控告警系统"></a>12.3 基于 Flink 的实时监控告警系统</h2><p>在如今微服务、云原生等技术盛行的时代，当谈到说要从 0 开始构建一个监控系统，大家无非就首先想到三个词：Metrics、Tracing、Logging。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>《Flink 实战与性能优化》—— 基于 Flink 的百亿数据去重实践</title>
<link href="http://www.54tianzhisheng.cn/2021/08/20/flink-in-action-12.2/"/>
<id>http://www.54tianzhisheng.cn/2021/08/20/flink-in-action-12.2/</id>
<published>2021-08-19T16:00:00.000Z</published>
<updated>2022-02-20T13:10:49.299Z</updated>
<content type="html"><![CDATA[<h2 id="12-2-基于-Flink-的百亿数据去重实践"><a href="#12-2-基于-Flink-的百亿数据去重实践" class="headerlink" title="12.2 基于 Flink 的百亿数据去重实践"></a>12.2 基于 Flink 的百亿数据去重实践</h2><p>在工作中经常会遇到去重的场景，例如基于 APP 的用户行为日志分析系统：用户的行为日志从手机 APP 端上报到 Nginx 服务端，然后通过 Logstash、Flume 或其他工具将日志从 Nginx 写入到 Kafka 中。由于用户手机客户端的网络可能出现不稳定，所以手机 APP 端上传日志的策略是：宁可重复上报，也不能漏报日志，所以导致 Kafka 中可能会出现日志重复的情况，即：同一条日志出现了 2 条或 2 条以上。通常情况下，Flink 任务的数据源都是 Kafka，若 Kafka 中数据出现了重复，在实时 ETL 或者流计算时都需要考虑基于日志主键对日志进行去重，否则会导致流计算结果偏高或结果不准确的问题，例如用户 a 在某个页面只点击了一次，但由于日志重复上报，所以用户 a 在该页面的点击日志在 Kafka 中出现了 2 次，最后统计该页面的点击数时，结果就会偏高。这里只阐述了一种可能造成 Kafka 中数据重复的情况，在生产环境中很多情况都可能造成 Kafka 中数据重复，这里不一一列举，本节主要讲述出现了数据重复后，该如何处理。</p><a id="more"></a><h3 id="12-2-1-去重的通用解决方案"><a href="#12-2-1-去重的通用解决方案" class="headerlink" title="12.2.1 去重的通用解决方案"></a>12.2.1 去重的通用解决方案</h3><p>Kafka 中数据出现重复后，各种解决方案都比较类似，一般需要一个全局 Set 集合来维护历史所有数据的主键。当处理新日志时，需要拿到当前日志的主键与历史数据的 Set 集合按照规则进行比较，若 Set 集合中已经包含了当前日志的主键，说明当前日志在之前已经被处理过了，则当前日志应该被过滤掉，否则认为当前日志不应该被过滤应该被处理，而且处理完成后需要将新日志的主键加入到 Set 集合中，Set 集合永远存放着所有已经被处理过的数据。这种去重的通用解决方案的流程图如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-12-11-151455.png" alt="去重的通用解决方案的流程图"></p><p>处理流程很简单，关键在于如何维护这个 Set 集合，可以简单估算一下这个 Set 集合需要占用多大空间。本小节要解决的问题是百亿数据去重，所以就按照每天 1 百亿的数据量来计算。由于每天数据量巨大，因此主键占用空间通常会比较大，如果主键占用空间小意味着表示的数据范围就比较小，就可能导致主键冲突，例如：4 个字节的 int 类型表示数据范围是为 -2147483648 ~ 2147483647，总共可以表示 42 亿个数，如果这里每天百亿的数据量选用 int 类型做为主键的话，很明显会有大量的主键发生冲突，会将不重复的数据认为是发生了重复。用户的行为日志是在手机客户端生成的，没有全局发号器，一般会选取 UUID 做为日志的主键，UUID 会生成 36 位的字符串，例如：”f106c4a1-4c6f-41c1-9d30-bbb2b271284a”。每个主键占用 36 字节，每天 1 百亿数据，36字节 <em> 100亿 ≈ 360 GB。这仅仅是一天的数据量，所以该 Set 集合要想存储空间不发生持续地爆炸式增长，必须增加一个功能，那就是给所有的主键增加 TTL（过期时间）。如果不增加 TTL，10 天数据量的主键占用空间就 3.6T，100 天数据量的主键占用空间 36T，所以在设计之初必须考虑为主键设定 TTL。如果要求按天进行去重或者认为日志发生重复上报的时间间隔不可能大于 24 小时，那么为了系统的可靠性 TTL 可以设置为 36 小时。每天数据量 1 百亿，且 Set 集合中存放着 36 小时的数据量，即 100 亿 </em> 1.5 = 150 亿，所以 Set 集合中需要维护 150 亿的数据量，且 Set 集合中每条数据都增加了 TTL，意味着 Set 集合需要为每条数据再附带保存一个时间戳，来确定该数据什么时候过期。例如 Redis 中为一个 key 设置了 TTL，如果没有为这个 key 附带时间戳，那么根本无法判断该 key 什么时候应该被清理。所以在考虑每条数据占用空间时，不仅要考虑数据本身，还需要考虑是否需要其他附带的存储。主键本身占用 36 字节加上 long 类型的时间戳 8 字节，所以每条数据至少需要占用 44 字节，150 亿 * 44 字节 = 660 GB。所以每天百亿的数据量，如果我们使用 Set 集合的方案来实现，至少需要占用 660 GB 以上的存储空间。</p><h3 id="12-2-2-使用-BloomFilter-实现去重"><a href="#12-2-2-使用-BloomFilter-实现去重" class="headerlink" title="12.2.2 使用 BloomFilter 实现去重"></a>12.2.2 使用 BloomFilter 实现去重</h3><p>有些流计算的场景对准确性要求并不是很高，例如传统的 Lambda 架构中，都会有离线去矫正实时计算的结果，所以根据业务场景，当业务要求可以接受结果有小量误差时，可以选择使用一些低成本的数据结构。BloomFilter 和 HyperLogLog 都是相对低成本的数据结构，分别有自己的应用场景，且两种数据结构都有一定误差。HyperLogLog 可以估算出 HyperLogLog 中插入了多少个不重复的元素，而不能告诉我们之前是否插入了哪些元素。BloomFilter 则恰好相反，相对而言 BloomFilter 更像是一个 Set 集合，BloomFilter 可以告诉你 BloomFilter 中<strong>肯定不包含</strong>元素 a，或者告诉你 BloomFilter 中<strong>可能包含</strong>元素 b，但 BloomFilter 不能告诉你 BloomFilter 中插入了多少个元素。接下来了解一下 BloomFilter 的实现原理。</p><h4 id="bitmap-位图"><a href="#bitmap-位图" class="headerlink" title="bitmap 位图"></a>bitmap 位图</h4><p>了解 BloomFilter，从 bitmap（位图）开始说起。现在有 1 千万个整数，数据范围在 0 到 2 千万之间。如何快速查找某个整数是否在这 1 千万个整数中呢？可以将这 1 千万个数保存在 HashMap 中，不考虑对象头及其他空间，1000 万个 int 类型数据需要占用大约 1000万 * 4 字节 ≈ 40 MB 存储空间。有没有其他方案呢？因为数据范围是 0 到 2 千万，所以可以申请一个长度为 2000 万、boolean 类型的数组，将这 2 千万个整数作为数组下标，将其对应的数组默认值设置成 false，如下图所示，数组下标为 2、666、999 的位置存储的数据为 true，表示 1 千万个数中包含了 2、666、999 等。当查询某个整数 K 是否在这 1 千万个整数中时，只需要将对应的数组值 <code>array[K]</code> 取出来，看是否等于 true。如果等于 true，说明 1 千万整数中包含这个整数 K，否则表示不包含这个整数 K。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-11-03-103907.jpg" alt=""></p><p>Java 的 boolean 基本类型占用一个字节（8bit）的内存空间，所以上述方案需要申请 2000 万字节。如下图所示，可以通过编程语言用二进制位来模拟布尔类型，二进制的 1 表示true、二进制的 0 表示false。通过二进制模拟布尔类型的方案，只需要申请 2000 万 bit 即可，相比 boolean 类型而言，存储空间占用仅为原来的 1/8。2000 万 bit ≈ 2.4 MB，相比存储原始数据的方案 40 MB 而言，占用的存储空间少了很多。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-11-03-103905.jpg" alt=""></p><p>假如这 1 千万个整数的数据范围是 0 到 100 亿，那么就需要申请 100 亿个 bit 约等于 1200 MB，比存储原始数据方案的 40MB 还要大很多。该情况下，直接使用位图使用的存储空间更多了，怎么解决呢？可以只申请 1 亿 bit 的存储空间，对 1000 万个数求hash，映射到 1 亿的二进制位上，最后大约占用 12 MB 的存储空间，但是可能存在 hash 冲突的情况。例如 3 和 100000003（一亿零三）这两个数对一亿求余都为 3，所以映射到长度为 1 亿的位图上，这两个数会占用同一个 bit，就会导致一个问题：1 千万个整数中包含了一亿零三，所以位图中下标为 3 的位置存储着二进制 1。当查询 1 千万个整数中是否包含数字 3 时，同样也是去位图中下标 3 的位置去查找，发现下标为 3 的位置存储着二进制 1，所以误以为 1 千万个整数中包含数字 3。为了减少 hash 冲突，于是诞生了 BloomFilter。</p><h4 id="BloomFilter-原理介绍"><a href="#BloomFilter-原理介绍" class="headerlink" title="BloomFilter 原理介绍"></a>BloomFilter 原理介绍</h4><p>hash 存在 hash 冲突（碰撞）的问题，两个不同的 key 通过同一个 hash 函数得到的值有可能相同。为了减少冲突，可以引入多个 hash 函数，如果通过其中的一个 hash 函数发现某元素不在集合中，那么该元素肯定不在集合中。当所有的 hash 函数告诉我们该元素在集合中时，才能确定该元素存在于集合中，这便是BloomFilter的基本思想。</p><p>如下图所示，是往 BloomFilter 中插入元素 a、b 的过程，有 3 个 hash 函数，元素 a 经过 3 个 hash 函数后对应的 2、8、10 这三个二进制位，所以将这三个二进制位置为 1，元素 b 经过 3 个 hash 函数后，对应的 5、10、14 这三个二进制位，将这三个二进制位也置为 1，其中下标为 10 的二进制位被 a、b 元素都涉及到。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-11-03-103858.jpg" style="zoom:20%;" /></p><p>如下图所示，是从 BloomFilter 中查找元素 c、d 的过程，同样包含了 3 个 hash 函数，元素 c 经过 3 个 hash 函数后对应的 2、6、9 这三个二进制位，其中下标 6 和 9 对应的二进制位为 0，所以会认为 BloomFilter 中不存在元素 c。元素 d 经过 3 个 hash 函数后对应的 5、8、14 这三个二进制位，这三个位对应的二进制位都为 1，所以会认为 BloomFilter 中存在元素 d，但其实 BloomFilter 中并不存在元素 d，是因为元素 a 和元素 b 也对应到了 5、8、14 这三个二进制位上，所以 BloomFilter 会有误判。但是从实现原理来看，当 BloomFilter 告诉你不包含元素 c 时，BloomFilter 中<strong>肯定不包含</strong>元素 c，当 BloomFilter 告诉你 BloomFilter 中包含元素 d 时，它只是<strong>可能包含</strong>，也有可能不包含。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-11-03-103906.jpg" style="zoom:20%;" /></p><h4 id="使用-BloomFilter-实现数据去重"><a href="#使用-BloomFilter-实现数据去重" class="headerlink" title="使用 BloomFilter 实现数据去重"></a>使用 BloomFilter 实现数据去重</h4><p>Redis 4.0 之后 BloomFilter 以插件的形式加入到 Redis 中，关于 api 的具体使用这里不多赘述。BloomFilter 在创建时支持设定一个预期容量和误判率，预期容量即预计插入的数据量，误判率即：当 BloomFilter 中插入的数据达到预期容量时，误判的概率，如果 BloomFilter 中插入数据较少的话，误判率会更低。</p><p>经笔者测试，申请一个预期容量为 10 亿，误判率为千分之一的 BloomFilter，BloomFilter 会申请约 143 亿个 bit，即：14G左右，相比之前 660G 的存储空间小太多了。但是在使用过程中，需要记录 BloomFilter 中插入元素的个数，当插入元素个数达到 10 亿时，为了保障误差率，可以将当前 BloomFilter 清除，重新申请一个新的 BloomFilter。</p><p>通过使用 Redis 的 BloomFilter，我们可以通过相对较小的内存实现百亿数据的去重，但是 BloomFilter 有误差，所以只能使用在那些对结果能承受一定误差的应用场景，对于广告计费等对数据精度要求非常高的场景，极力推荐大家使用精准去重的方案来实现。</p><h3 id="12-2-3-使用-HBase-维护全局-Set-实现去重"><a href="#12-2-3-使用-HBase-维护全局-Set-实现去重" class="headerlink" title="12.2.3 使用 HBase 维护全局 Set 实现去重"></a>12.2.3 使用 HBase 维护全局 Set 实现去重</h3><p>通过之前分析，我们知道要想实现百亿数据量的精准去重，需要维护 150 亿数据量的 Set 集合，每条数据占用 44 KB，总共需要 660 GB 的存储空间。注意这里说的是存储空间而不是内存空间，为什么呢？因为 660 G 的内存实在是太贵了，660G 的 Redis 云服务一个月至少要 2 万 RMB 以上，俗话说设计架构不考虑成本等于耍流氓。这里使用 Redis 确实可以解决问题，但是成本较高。HBase 基于 RowKey Get 的效率比较高，所以这里可以考虑将这个大的 Set 集合以 HBase RowKey 的形式存放到 HBase 中。HBase 表设置 TTL 为 36 小时，最近 36 小时的 150 亿条日志的主键都存放到 HBase 中，每来一条数据，先拿到主键去 HBase 中查询，如果 HBase 表中存在该主键，说明当前日志已经被处理过了，当前日志应该被过滤。如果 HBase 表中不存在该主键，说明当前日志之前没有被处理过，此时应该被处理，且处理完成后将当前主键 Put 到 HBase 表中。由于数据量比较大，所以一定要提前对 HBase 表进行预分区，将压力分散到各个 RegionServer 上。</p><h4 id="使用-HBase-RowKey-去重带来的问题"><a href="#使用-HBase-RowKey-去重带来的问题" class="headerlink" title="使用 HBase RowKey 去重带来的问题"></a>使用 HBase RowKey 去重带来的问题</h4><h3 id="12-2-4-使用-Flink-的-KeyedState-实现去重"><a href="#12-2-4-使用-Flink-的-KeyedState-实现去重" class="headerlink" title="12.2.4 使用 Flink 的 KeyedState 实现去重"></a>12.2.4 使用 Flink 的 KeyedState 实现去重</h3><p>下面就教大家如何使用 Flink 的 KeyedState 实现去重。</p><h4 id="使用-Flink-状态来维护-Set-集合的优势"><a href="#使用-Flink-状态来维护-Set-集合的优势" class="headerlink" title="使用 Flink 状态来维护 Set 集合的优势"></a>使用 Flink 状态来维护 Set 集合的优势</h4><h4 id="如何使用-KeyedState-维护-Set-集合"><a href="#如何使用-KeyedState-维护-Set-集合" class="headerlink" title="如何使用 KeyedState 维护 Set 集合"></a>如何使用 KeyedState 维护 Set 集合</h4><h4 id="优化主键来减少状态大小，且提高吞吐量"><a href="#优化主键来减少状态大小，且提高吞吐量" class="headerlink" title="优化主键来减少状态大小，且提高吞吐量"></a>优化主键来减少状态大小，且提高吞吐量</h4><h3 id="12-2-5-使用-RocksDBStateBackend-的优化方法"><a href="#12-2-5-使用-RocksDBStateBackend-的优化方法" class="headerlink" title="12.2.5 使用 RocksDBStateBackend 的优化方法"></a>12.2.5 使用 RocksDBStateBackend 的优化方法</h3><p>在使用上述方案的过程中，可能会出现吞吐量时高时低，或者吞吐量比笔者的测试性能要低一些，当出现这类问题的时候，可以尝试从以下几个方面进行优化。</p><h4 id="设置本地-RocksDB-的数据目录"><a href="#设置本地-RocksDB-的数据目录" class="headerlink" title="设置本地 RocksDB 的数据目录"></a>设置本地 RocksDB 的数据目录</h4><h4 id="Checkpoint-参数相关配置"><a href="#Checkpoint-参数相关配置" class="headerlink" title="Checkpoint 参数相关配置"></a>Checkpoint 参数相关配置</h4><h4 id="RocksDB-参数相关配置"><a href="#RocksDB-参数相关配置" class="headerlink" title="RocksDB 参数相关配置"></a>RocksDB 参数相关配置</h4><h3 id="12-2-6-小结与反思"><a href="#12-2-6-小结与反思" class="headerlink" title="12.2.6 小结与反思"></a>12.2.6 小结与反思</h3><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/IeAYbEy">https://t.zsxq.com/IeAYbEy</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h2 id="12-2-基于-Flink-的百亿数据去重实践"><a href="#12-2-基于-Flink-的百亿数据去重实践" class="headerlink" title="12.2 基于 Flink 的百亿数据去重实践"></a>12.2 基于 Flink 的百亿数据去重实践</h2><p>在工作中经常会遇到去重的场景，例如基于 APP 的用户行为日志分析系统：用户的行为日志从手机 APP 端上报到 Nginx 服务端，然后通过 Logstash、Flume 或其他工具将日志从 Nginx 写入到 Kafka 中。由于用户手机客户端的网络可能出现不稳定，所以手机 APP 端上传日志的策略是：宁可重复上报，也不能漏报日志，所以导致 Kafka 中可能会出现日志重复的情况，即：同一条日志出现了 2 条或 2 条以上。通常情况下，Flink 任务的数据源都是 Kafka，若 Kafka 中数据出现了重复，在实时 ETL 或者流计算时都需要考虑基于日志主键对日志进行去重，否则会导致流计算结果偏高或结果不准确的问题，例如用户 a 在某个页面只点击了一次，但由于日志重复上报，所以用户 a 在该页面的点击日志在 Kafka 中出现了 2 次，最后统计该页面的点击数时，结果就会偏高。这里只阐述了一种可能造成 Kafka 中数据重复的情况，在生产环境中很多情况都可能造成 Kafka 中数据重复，这里不一一列举，本节主要讲述出现了数据重复后，该如何处理。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>《Flink 实战与性能优化》—— 基于 Flink 实时处理海量日志</title>
<link href="http://www.54tianzhisheng.cn/2021/08/19/flink-in-action-12.1/"/>
<id>http://www.54tianzhisheng.cn/2021/08/19/flink-in-action-12.1/</id>
<published>2021-08-18T16:00:00.000Z</published>
<updated>2022-02-20T13:07:36.232Z</updated>
<content type="html"><![CDATA[<h1 id="第十二章-——-Flink-案例"><a href="#第十二章-——-Flink-案例" class="headerlink" title="第十二章 —— Flink 案例"></a>第十二章 —— Flink 案例</h1><p>本章将介绍 Flink 在多个场景下落地实现的大型案例，第一个是实时处理海量的日志，将从日志的收集、日志的传输、日志的实时清洗和异常检测、日志存储、日志展示等方面去介绍 Flink 在其中起的作用，希望整个日志处理的架构大家可以灵活的运用在自己的公司；第二个是百亿数据量的情况下如何使用 Flink 实时去重，在这个案例中将对比介绍其他几种常见的去重实现方案；第三个是 Flink 在监控告警系统中的落地实现，在这个案例中同样很详细的介绍了一个监控告警系统的全链路，每一个关节都不可或缺，并且还介绍了 Flink 在未来结合机器学习算法做一些 AIOps 的事情。三个案例都比较典型，如果你也在做类似的项目，希望对你们的技术选型有一定的帮助。</p><h2 id="12-1-基于-Flink-实时处理海量日志"><a href="#12-1-基于-Flink-实时处理海量日志" class="headerlink" title="12.1 基于 Flink 实时处理海量日志"></a>12.1 基于 Flink 实时处理海量日志</h2><p>在 11.5 节中讲解了 Flink 如何实时处理异常的日志，并且对比分析了几种常用的日志采集工具。我们也知道通常在排查线上异常故障的时候，日志是必不可缺的一部分，通过异常日志我们可以快速的定位到问题的根因。那么通常在公司对于日志处理有哪些需求呢？</p><a id="more"></a><h3 id="12-1-1-实时处理海量日志需求分析"><a href="#12-1-1-实时处理海量日志需求分析" class="headerlink" title="12.1.1 实时处理海量日志需求分析"></a>12.1.1 实时处理海量日志需求分析</h3><p>现在公司都在流行构建分布式、微服务、云原生的架构，在这类架构下，项目应用的日志都被分散到不同的机器上，日志查询就会比较困难，所以统一的日志收集几乎也是每家公司必不可少的。据笔者调研，不少公司现在是有日志统一的收集，也会去做日志的实时 ETL，利用一些主流的技术比如 ELK 去做日志的展示、搜索和分析，但是却缺少了日志的实时告警。总结来说，大部分公司对于日志这块的现状是：</p><ul><li><strong>日志分布零散</strong>：分布式应用导致日志分布在不同的机器上，人肉登录到机器上操作复杂，需要统一的日志收集工具。</li><li><strong>异常日志无告警</strong>：出错时无异常日志告警，导致错过最佳定位问题的时机，需要异常错误日志的告警。</li><li><strong>日志查看不友好</strong>：登录服务器上在终端查看日志不太方便，需要一个操作友好的页面去查看日志。</li><li><strong>无日志搜索分析</strong>：历史日志文件太多，想找某种日志找不到了，需要一个可以搜索日志的功能。</li></ul><p>在本节中，笔者将为大家讲解日志的全链路，包含了日志的实时采集、日志的 ETL、日志的实时监控告警、日志的存储、日志的可视化图表展示与搜索分析等。</p><h3 id="12-1-2-实时处理海量日志架构设计"><a href="#12-1-2-实时处理海量日志架构设计" class="headerlink" title="12.1.2 实时处理海量日志架构设计"></a>12.1.2 实时处理海量日志架构设计</h3><p>分析完我们这个案例的需求后，接下来对整个项目的架构做一个合理的设计。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-27-145059.png" alt=""></p><p>整个架构分为五层：日志接入层、日志削峰层、日志处理层、日志存储层、日志展示层。</p><ul><li>日志接入层：日志采集的话使用的是 Filebeat 组件，需要在每台机器上部署一个 Filebeat。</li><li>日志削峰层：防止日志流量高峰，使用 Kafka 消息队列做削峰。</li><li>日志处理层：Flink 作业同时消费 Kafka 数据做日志清洗、ETL、实时告警。</li><li>日志存储层：使用 ElasticSearch 做日志的存储。</li><li>日志展示层：使用 Kibana 做日志的展示与搜索查询界面。</li></ul><h3 id="12-1-3-日志实时采集"><a href="#12-1-3-日志实时采集" class="headerlink" title="12.1.3 日志实时采集"></a>12.1.3 日志实时采集</h3><p>在 11.5.1 中对比了这几种比较流行的日志采集工具（Logstash、Filebeat、Fluentd、Logagent），从功能完整性、性能、成本、使用难度等方面综合考虑后，这里演示使用的是 Filebeat。</p><h4 id="安装-Filebeat"><a href="#安装-Filebeat" class="headerlink" title="安装 Filebeat"></a>安装 Filebeat</h4><p>在服务器上下载 <a href="https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz">Fliebeat 6.3.2</a> 安装包（请根据自己服务器和所需要的版本进行下载），下载后进行解压。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">tar xzf filebeat-6.3.2-linux-x86_64.tar.gz</span><br></pre></td></tr></table></figure><h4 id="配置-Filebeat"><a href="#配置-Filebeat" class="headerlink" title="配置 Filebeat"></a>配置 Filebeat</h4><p>配置 Filebeat 需要编辑 Filebeat 的配置文件 <code>filebeat.yml</code>，不同安装方式配置文件的存放路径有一些不同，对于解压包安装的方式，配置文件存在解压目录下面；对于 rpm 和 deb 的方式, 配置文件路径的是 <code>/etc/filebeat/filebeat.yml</code> 下。</p><p>因为 Filebeat 是要实时采集日志的，所以得让 Filebeat 知道日志的路径是在哪里，下面在配置文件中定义一下日志文件的路径。通常建议在服务器上固定存放日志的路径，然后应用的日志都打在这个固定的路径中，这样 Filebeat 的日志路径配置只需要填写一次，其他机器上可以拷贝同样的配置就能将 Filebeat 运行起来，配置如下。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">- type: log</span><br><span class="line"> # 配置为 true 表示开启</span><br><span class="line"> enabled: true</span><br><span class="line"> # 日志的路径</span><br><span class="line"> paths:</span><br><span class="line"> - /var/logs/*.log</span><br></pre></td></tr></table></figure><p>上面的配置表示将对 /var/logs 目录下所有以 .log 结尾的文件进行采集，接下来配置日志输出的方式，这里使用的是 Kafka，配置如下。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">output.kafka:</span><br><span class="line"> # 填写 Kafka 地址信息</span><br><span class="line"> hosts: ["localhost:9092"]</span><br><span class="line"> # 数据发到哪个 topic</span><br><span class="line"> topic: zhisheng-log</span><br><span class="line"> partition.round_robin:</span><br><span class="line"> reachable_only: false</span><br><span class="line"> required_acks: 1</span><br></pre></td></tr></table></figure><p>上面讲解的两个配置，笔者这里将它们写在一个新建的配置文件中 kafka.yml，然后启动 Filebeat 的时候使用该配置。</p><figure class="highlight yml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">filebeat.inputs:</span></span><br><span class="line"><span class="attr">- type:</span> <span class="string">log</span></span><br><span class="line"><span class="attr"> enabled:</span> <span class="literal">true</span></span><br><span class="line"><span class="attr"> paths:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">/var/logs/*.log</span></span><br><span class="line"><span class="string">output.kafka:</span></span><br><span class="line"><span class="attr"> hosts:</span> <span class="string">["localhost:9092"]</span></span><br><span class="line"><span class="attr"> topic:</span> <span class="string">zhisheng_log</span></span><br><span class="line"> <span class="string">partition.round_robin:</span></span><br><span class="line"><span class="attr"> reachable_only:</span> <span class="literal">false</span></span><br><span class="line"><span class="attr"> required_acks:</span> <span class="number">1</span></span><br></pre></td></tr></table></figure><h4 id="启动-Filebeat"><a href="#启动-Filebeat" class="headerlink" title="启动 Filebeat"></a>启动 Filebeat</h4><p>日志路径的配置和 Kafka 的配置都写好后，则接下来通过下面命令将 Filebeat 启动：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/filebeat -e -c kafka.yml</span><br></pre></td></tr></table></figure><p>执行完命令后出现的日志如下则表示启动成功了，另外还可以看得到会在终端打印出 metrics 数据出来。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-26-075438.png" alt=""></p><h4 id="验证-Filebeat-是否将日志数据发到-Kafka"><a href="#验证-Filebeat-是否将日志数据发到-Kafka" class="headerlink" title="验证 Filebeat 是否将日志数据发到 Kafka"></a>验证 Filebeat 是否将日志数据发到 Kafka</h4><p>那么此时就得去查看是否真正就将这些日志数据发到 Kafka 了呢，你可以通过 Kafka 的自带命令去消费这个 Topic 看是否不断有数据发出来，命令如下：</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-console-consumer.sh --zookeeper 106.54.248.27:2181 --topic zhisheng_log --from-beginning</span><br></pre></td></tr></table></figure><p>如果出现数据则代表是已经有数据发到 Kafka 了，如果你不喜欢使用这种方式验证，可以自己写个 Flink Job 去读取 Kafka 该 Topic 的数据，比如写了个作业运行结果如下就代表着日志数据已经成功发送到 Kafka。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-27-150039.png" alt=""></p><h4 id="发到-Kafka-的日志结构"><a href="#发到-Kafka-的日志结构" class="headerlink" title="发到 Kafka 的日志结构"></a>发到 Kafka 的日志结构</h4><p>既然数据都已经发到 Kafka 了，通过消费 Kafka 该 Topic 的数据我们可以发现这些数据的格式否是 JSON，结构如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"><span class="attr">"@timestamp"</span>: <span class="string">"2019-10-26T08:18:18.087Z"</span>,</span><br><span class="line"><span class="attr">"@metadata"</span>: {</span><br><span class="line"><span class="attr">"beat"</span>: <span class="string">"filebeat"</span>,</span><br><span class="line"><span class="attr">"type"</span>: <span class="string">"doc"</span>,</span><br><span class="line"><span class="attr">"version"</span>: <span class="string">"6.8.4"</span>,</span><br><span class="line"><span class="attr">"topic"</span>: <span class="string">"zhisheng_log"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"prospector"</span>: {</span><br><span class="line"><span class="attr">"type"</span>: <span class="string">"log"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"input"</span>: {</span><br><span class="line"><span class="attr">"type"</span>: <span class="string">"log"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"beat"</span>: {</span><br><span class="line"><span class="attr">"name"</span>: <span class="string">"VM_0_2_centos"</span>,</span><br><span class="line"><span class="attr">"hostname"</span>: <span class="string">"VM_0_2_centos"</span>,</span><br><span class="line"><span class="attr">"version"</span>: <span class="string">"6.8.4"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"host"</span>: {</span><br><span class="line"><span class="attr">"name"</span>: <span class="string">"VM_0_2_centos"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"source"</span>: <span class="string">"/var/logs/middleware/kafka.log"</span>,</span><br><span class="line"><span class="attr">"offset"</span>: <span class="number">9460</span>,</span><br><span class="line"><span class="attr">"log"</span>: {</span><br><span class="line"><span class="attr">"file"</span>: {</span><br><span class="line"><span class="attr">"path"</span>: <span class="string">"/var/logs/middleware/kafka.log"</span></span><br><span class="line">}</span><br><span class="line">},</span><br><span class="line"><span class="attr">"message"</span>: <span class="string">"2019-10-26 16:18:11 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>这个日志结构里面包含了很多字段，比如 timestamp、metadata、host、source、message 等，但是其中某些字段我们其实根本不需要的，你可以根据公司的需求丢弃一些字段，把要丢弃的字段也配置在 kafka.yml 中，如下所示。</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">processors:</span><br><span class="line">- drop_fields:</span><br><span class="line"> fields: ["prospector","input","beat","log","offset","@metadata"]</span><br></pre></td></tr></table></figure><p>然后再次启动 Filebeat ，发现上面配置的字段在新的数据中没有了（除 @metadata 之外），另外经笔者验证：不仅 @metadata 字段不能丢弃，如果 @timestamp 这个字段在 drop_fields 中配置了，也是不起作用的，它们两不允许丢弃。通常来说一行日志已经够长了，再加上这么多我们不需要的字段，就会增加数据的大小，对于生产环境的话，日志数据量非常大，那无疑会对后面所有的链路都会造成一定的影响，所以一定要在底层数据源头做好精简。另外还可以在发送 Kafka 的时候对数据进行压缩，可以在配置文件中配置一个 <code>compression: gzip</code>。精简后的日志数据结构如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"><span class="attr">"@timestamp"</span>: <span class="string">"2019-10-26T09:23:16.848Z"</span>,</span><br><span class="line"><span class="attr">"@metadata"</span>: {</span><br><span class="line"><span class="attr">"beat"</span>: <span class="string">"filebeat"</span>,</span><br><span class="line"><span class="attr">"type"</span>: <span class="string">"doc"</span>,</span><br><span class="line"><span class="attr">"version"</span>: <span class="string">"6.8.4"</span>,</span><br><span class="line"><span class="attr">"topic"</span>: <span class="string">"zhisheng_log"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"host"</span>: {</span><br><span class="line"><span class="attr">"name"</span>: <span class="string">"VM_0_2_centos"</span></span><br><span class="line">},</span><br><span class="line"><span class="attr">"source"</span>: <span class="string">"/var/logs/middleware/kafka.log"</span>,</span><br><span class="line"><span class="attr">"message"</span>: <span class="string">"2019-10-26 17:23:11 TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="12-1-4-日志格式统一"><a href="#12-1-4-日志格式统一" class="headerlink" title="12.1.4 日志格式统一"></a>12.1.4 日志格式统一</h3><p>因为 Filebeat 是在机器上采集的日志，这些日志的种类比较多，常见的有应用程序的运行日志、作业构建编译打包的日志、中间件服务运行的日志等。通常在公司是可以给开发约定日志打印的规则，但是像中间件这类服务的日志是不固定的，如果将 Kafka 中的消息直接存储到 ElasticSearch 的话，后面如果要做区分筛选的话可能会有问题。为了避免这个问题，我们得在日志存入 ElasticSearch 之前做一个数据格式化和清洗的工作，因为 Flink 处理数据的速度比较好，而且可以做到实时，所以选择在 Flink Job 中完成该工作。</p><p>在该作业中的要将 message 解析，一般该行日志信息会包含很多信息，比如日志打印时间、日志级别、应用名、唯一性 ID（用来关联各个请求）、请求上下文。那么我们就需要一个新的日志结构对象来统一日志的格式，定义如下：</p><h3 id="12-1-5-日志实时清洗"><a href="#12-1-5-日志实时清洗" class="headerlink" title="12.1.5 日志实时清洗"></a>12.1.5 日志实时清洗</h3><h3 id="12-1-6-日志实时告警"><a href="#12-1-6-日志实时告警" class="headerlink" title="12.1.6 日志实时告警"></a>12.1.6 日志实时告警</h3><h3 id="12-1-7-日志实时存储"><a href="#12-1-7-日志实时存储" class="headerlink" title="12.1.7 日志实时存储"></a>12.1.7 日志实时存储</h3><h3 id="12-1-8-日志实时展示"><a href="#12-1-8-日志实时展示" class="headerlink" title="12.1.8 日志实时展示"></a>12.1.8 日志实时展示</h3><h3 id="12-1-9-小结与反思"><a href="#12-1-9-小结与反思" class="headerlink" title="12.1.9 小结与反思"></a>12.1.9 小结与反思</h3><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/IeAYbEy">https://t.zsxq.com/IeAYbEy</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h1 id="第十二章-——-Flink-案例"><a href="#第十二章-——-Flink-案例" class="headerlink" title="第十二章 —— Flink 案例"></a>第十二章 —— Flink 案例</h1><p>本章将介绍 Flink 在多个场景下落地实现的大型案例，第一个是实时处理海量的日志，将从日志的收集、日志的传输、日志的实时清洗和异常检测、日志存储、日志展示等方面去介绍 Flink 在其中起的作用，希望整个日志处理的架构大家可以灵活的运用在自己的公司；第二个是百亿数据量的情况下如何使用 Flink 实时去重，在这个案例中将对比介绍其他几种常见的去重实现方案；第三个是 Flink 在监控告警系统中的落地实现，在这个案例中同样很详细的介绍了一个监控告警系统的全链路，每一个关节都不可或缺，并且还介绍了 Flink 在未来结合机器学习算法做一些 AIOps 的事情。三个案例都比较典型，如果你也在做类似的项目，希望对你们的技术选型有一定的帮助。</p>
<h2 id="12-1-基于-Flink-实时处理海量日志"><a href="#12-1-基于-Flink-实时处理海量日志" class="headerlink" title="12.1 基于 Flink 实时处理海量日志"></a>12.1 基于 Flink 实时处理海量日志</h2><p>在 11.5 节中讲解了 Flink 如何实时处理异常的日志，并且对比分析了几种常用的日志采集工具。我们也知道通常在排查线上异常故障的时候，日志是必不可缺的一部分，通过异常日志我们可以快速的定位到问题的根因。那么通常在公司对于日志处理有哪些需求呢？</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>《Flink 实战与性能优化》—— 如何实时将应用 Error 日志告警？</title>
<link href="http://www.54tianzhisheng.cn/2021/08/18/flink-in-action-11.5/"/>
<id>http://www.54tianzhisheng.cn/2021/08/18/flink-in-action-11.5/</id>
<published>2021-08-17T16:00:00.000Z</published>
<updated>2022-02-07T14:13:56.910Z</updated>
<content type="html"><![CDATA[<h2 id="11-5-如何实时将应用-Error-日志告警？"><a href="#11-5-如何实时将应用-Error-日志告警？" class="headerlink" title="11.5 如何实时将应用 Error 日志告警？"></a>11.5 如何实时将应用 Error 日志告警？</h2><p>大数据时代，随着公司业务不断的增长，数据量自然也会跟着不断的增长，那么业务应用和集群服务器的的规模也会逐渐扩大，几百台服务器在一般的公司已经是很常见的了。那么将应用服务部署在如此多的服务器上，对开发和运维人员来说都是一个挑战。一个优秀的系统运维平台是需要将部署在这么多服务器上的应用监控信息汇总成一个统一的数据展示平台，方便运维人员做日常的监测、提升运维效率，还可以及时反馈应用的运行状态给应用开发人员。举个例子，应用的运行日志需要按照时间排序做一个展示，并且提供日志下载和日志搜索等服务，这样如果应用出现问题开发人员首先可以根据应用日志的错误信息进行问题的排查。那么该如何实时的将应用的 Error 日志推送给应用开发人员呢，接下来我们将讲解日志的处理方案。</p><a id="more"></a><h3 id="11-5-1-日志处理方案的演进"><a href="#11-5-1-日志处理方案的演进" class="headerlink" title="11.5.1 日志处理方案的演进"></a>11.5.1 日志处理方案的演进</h3><p>日志处理的方案也是有一个演进的过程，要想弄清楚整个过程，我们先来看下日志的介绍。</p><h4 id="什么是日志？"><a href="#什么是日志？" class="headerlink" title="什么是日志？"></a>什么是日志？</h4><p>日志是带时间戳的基于时间序列的数据，它可以反映系统的运行状态，包括了一些标识信息（应用所在服务器集群名、集群机器 IP、机器设备系统信息、应用名、应用 ID、应用所属项目等）</p><h4 id="日志处理方案演进"><a href="#日志处理方案演进" class="headerlink" title="日志处理方案演进"></a>日志处理方案演进</h4><p>日志处理方案的演进过程：</p><ul><li>日志处理 v1.0: 应用日志分布在很多机器上，需要人肉手动去机器查看日志信息。</li><li>日志处理 v2.0: 利用离线计算引擎统一的将日志收集，形成一个日志搜索分析平台，提供搜索让用户根据关键字进行搜索和分析，缺点就是及时性比较差。</li><li>日志处理 v3.0: 利用 Agent 实时的采集部署在每台机器上的日志，然后统一发到日志收集平台做汇总，并提供实时日志分析和搜索的功能，这样从日志产生到搜索分析出结果只有简短的延迟（在用户容忍时间范围之内），优点是快，但是日志数据量大的情况下带来的挑战也大。</li></ul><h3 id="11-5-2-日志采集工具对比"><a href="#11-5-2-日志采集工具对比" class="headerlink" title="11.5.2 日志采集工具对比"></a>11.5.2 日志采集工具对比</h3><p>上面提到的日志采集，其实现在已经有很多开源的组件支持去采集日志，比如 Logstash、Filebeat、Fluentd、Logagent 等，这里简单做个对比。</p><h4 id="Logstash"><a href="#Logstash" class="headerlink" title="Logstash"></a>Logstash</h4><p>Logstash 是一个开源数据收集引擎，具有实时管道功能。Logstash 可以动态地将来自不同数据源的数据统一起来，并将数据标准化到你所选择的目的地。如下图所示，Logstash 将采集到的数据用作分析、监控、告警等。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-13-025214.jpg" alt=""></p><p><strong>优势</strong>：Logstash 主要的优点就是它的灵活性，它提供很多插件，详细的文档以及直白的配置格式让它可以在多种场景下应用。而且现在 ELK 整个技术栈在很多公司应用的比较多，所以基本上可以在往上找到很多相关的学习资源。</p><p><strong>劣势</strong>：Logstash 致命的问题是它的性能以及资源消耗(默认的堆大小是 1GB)。尽管它的性能在近几年已经有很大提升，与它的替代者们相比还是要慢很多的，它在大数据量的情况下会是个问题。另一个问题是它目前不支持缓存，目前的典型替代方案是将 Redis 或 Kafka 作为中心缓冲池：</p><h4 id="Filebeat"><a href="#Filebeat" class="headerlink" title="Filebeat"></a>Filebeat</h4><p>作为 Beats 家族的一员，Filebeat 是一个轻量级的日志传输工具，它的存在正弥补了 Logstash 的缺点，Filebeat 作为一个轻量级的日志传输工具可以将日志推送到 Kafka、Logstash、ElasticSearch、Redis。它的处理流程如下图所示：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-13-030138.jpg" alt=""></p><p><strong>优势</strong>：Filebeat 只是一个二进制文件没有任何依赖。它占用资源极少，尽管它还十分年轻，正式因为它简单，所以几乎没有什么可以出错的地方，所以它的可靠性还是很高的。它也为我们提供了很多可以调节的点，例如：它以何种方式搜索新的文件，以及当文件有一段时间没有发生变化时，何时选择关闭文件句柄。</p><p><strong>劣势</strong>：Filebeat 的应用范围十分有限，所以在某些场景下我们会碰到问题。例如，如果使用 Logstash 作为下游管道，我们同样会遇到性能问题。正因为如此，Filebeat 的范围在扩大。开始时，它只能将日志发送到 Logstash 和 Elasticsearch，而现在它可以将日志发送给 Kafka 和 Redis，在 5.x 版本中，它还具备过滤的能力。</p><h4 id="Fluentd"><a href="#Fluentd" class="headerlink" title="Fluentd"></a>Fluentd</h4><p>Fluentd 创建的初衷主要是尽可能的使用 JSON 作为日志输出，所以传输工具及其下游的传输线不需要猜测子字符串里面各个字段的类型。这样它为几乎所有的语言都提供库，这也意味着可以将它插入到自定义的程序中。它的处理流程如下图所示：</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-13-031337.png" alt=""></p><p><strong>优势</strong>：和多数 Logstash 插件一样，Fluentd 插件是用 Ruby 语言开发的非常易于编写维护。所以它数量很多，几乎所有的源和目标存储都有插件(各个插件的成熟度也不太一样)。这也意味这可以用 Fluentd 来串联所有的东西。</p><p><strong>劣势</strong>：因为在多数应用场景下得到 Fluentd 结构化的数据，它的灵活性并不好。但是仍然可以通过正则表达式来解析非结构化的数据。尽管性能在大多数场景下都很好，但它并不是最好的，它的缓冲只存在与输出端，单线程核心以及 Ruby GIL 实现的插件意味着它大的节点下性能是受限的。</p><h4 id="Logagent"><a href="#Logagent" class="headerlink" title="Logagent"></a>Logagent</h4><p>Logagent 是 Sematext 提供的传输工具，它用来将日志传输到 Logsene(一个基于 SaaS 平台的 Elasticsearch API)，因为 Logsene 会暴露 Elasticsearch API，所以 Logagent 可以很容易将数据推送到 Elasticsearch 。</p><p><strong>优势</strong>：可以获取 /var/log 下的所有信息，解析各种格式的日志，可以掩盖敏感的数据信息。它还可以基于 IP 做 GeoIP 丰富地理位置信息。同样，它轻量又快速，可以将其置入任何日志块中。Logagent 有本地缓冲，所以在数据传输目的地不可用时不会丢失日志。</p><p><strong>劣势</strong>：没有 Logstash 灵活。</p><h3 id="11-5-3-日志结构设计"><a href="#11-5-3-日志结构设计" class="headerlink" title="11.5.3 日志结构设计"></a>11.5.3 日志结构设计</h3><p>前面介绍了日志和对比了常用日志采集工具的优势和劣势，通常在不同环境，不同机器上都会部署日志采集工具，然后采集工具会实时的将新的日志采集发送到下游，因为日志数据量毕竟大，所以建议发到 MQ 中，比如 Kafka，这样再想怎么处理这些日志就会比较灵活。假设我们忽略底层采集具体是哪种，但是规定采集好的日志结构化数据如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">LogEvent</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> String type;<span class="comment">//日志的类型(应用、容器、...)</span></span><br><span class="line"> <span class="keyword">private</span> Long timestamp;<span class="comment">//日志的时间戳</span></span><br><span class="line"> <span class="keyword">private</span> String level;<span class="comment">//日志的级别(debug/info/warn/error)</span></span><br><span class="line"> <span class="keyword">private</span> String message;<span class="comment">//日志内容</span></span><br><span class="line"> <span class="comment">//日志的标识(应用 ID、应用名、容器 ID、机器 IP、集群名、...)</span></span><br><span class="line"> <span class="keyword">private</span> Map<String, String> tags = <span class="keyword">new</span> HashMap<>();</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>然后上面这种 LogEvent 的数据（假设采集发上来的是这种结构数据的 JSON 串，所以需要在 Flink 中做一个反序列化解析）就会往 Kafka 不断的发送数据，样例数据如下：</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">{</span><br><span class="line"><span class="attr">"type"</span>: <span class="string">"app"</span>,</span><br><span class="line"><span class="attr">"timestamp"</span>: <span class="number">1570941591229</span>,</span><br><span class="line"><span class="attr">"level"</span>: <span class="string">"error"</span>,</span><br><span class="line"><span class="attr">"message"</span>: <span class="string">"Exception in thread \"main\" java.lang.NoClassDefFoundError: org/apache/flink/api/common/ExecutionConfig$GlobalJobParameters"</span>,</span><br><span class="line"><span class="attr">"tags"</span>: {</span><br><span class="line"><span class="attr">"cluster_name"</span>: <span class="string">"zhisheng"</span>,</span><br><span class="line"><span class="attr">"app_name"</span>: <span class="string">"zhisheng"</span>,</span><br><span class="line"><span class="attr">"host_ip"</span>: <span class="string">"127.0.0.1"</span>,</span><br><span class="line"><span class="attr">"app_id"</span>: <span class="string">"21"</span></span><br><span class="line">}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>那么在 Flink 中如何将应用异常或者错误的日志做实时告警呢？</p><h3 id="11-5-4-异常日志实时告警项目架构"><a href="#11-5-4-异常日志实时告警项目架构" class="headerlink" title="11.5.4 异常日志实时告警项目架构"></a>11.5.4 异常日志实时告警项目架构</h3><p>整个异常日志实时告警项目的架构如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-13-035811.png" alt=""></p><p>应用日志散列在不同的机器，然后每台机器都有部署采集日志的 Agent（可以是上面的 Filebeat、Logstash 等），这些 Agent 会实时的将分散在不同机器、不同环境的应用日志统一的采集发到 Kafka 集群中，然后告警这边是有一个 Flink 作业去实时的消费 Kafka 数据做一个异常告警计算处理。如果还想做日志的搜索分析，可以起另外一个作业去实时的将 Kafka 的日志数据写入进 ElasticSearch，再通过 Kibana 页面做搜索和分析。</p><h3 id="11-5-5-日志数据发送到-Kafka"><a href="#11-5-5-日志数据发送到-Kafka" class="headerlink" title="11.5.5 日志数据发送到 Kafka"></a>11.5.5 日志数据发送到 Kafka</h3><p>上面已经讲了日志数据 LogEvent 的结构和样例数据，因为要在服务器部署采集工具去采集应用日志数据对于本地测试来说可能稍微复杂，所以在这里就只通过代码模拟构造数据发到 Kafka 去，然后在 Flink 作业中去实时消费 Kafka 中的数据，下面演示构造日志数据发到 Kafka 的工具类，这个工具类主要分两块，构造 LogEvent 数据和发送到 Kafka。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Slf</span>4j</span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">BuildLogEventDataUtil</span> </span>{</span><br><span class="line"> <span class="comment">//Kafka broker 和 topic 信息</span></span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String BROKER_LIST = <span class="string">"localhost:9092"</span>;</span><br><span class="line"> <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">final</span> String LOG_TOPIC = <span class="string">"zhisheng_log"</span>;</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">writeDataToKafka</span><span class="params">()</span> </span>{</span><br><span class="line"> Properties props = <span class="keyword">new</span> Properties();</span><br><span class="line"> props.put(<span class="string">"bootstrap.servers"</span>, BROKER_LIST);</span><br><span class="line"> props.put(<span class="string">"key.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"> props.put(<span class="string">"value.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"> KafkaProducer producer = <span class="keyword">new</span> KafkaProducer<String, String>(props);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < <span class="number">10000</span>; i++) {</span><br><span class="line"> <span class="comment">//模拟构造 LogEvent 对象</span></span><br><span class="line"> LogEvent logEvent = <span class="keyword">new</span> LogEvent().builder()</span><br><span class="line"> .type(<span class="string">"app"</span>)</span><br><span class="line"> .timestamp(System.currentTimeMillis())</span><br><span class="line"> .level(logLevel())</span><br><span class="line"> .message(message(i + <span class="number">1</span>))</span><br><span class="line"> .tags(mapData())</span><br><span class="line"> .build();</span><br><span class="line"><span class="comment">// System.out.println(logEvent);</span></span><br><span class="line"> ProducerRecord record = <span class="keyword">new</span> ProducerRecord<String, String>(LOG_TOPIC, <span class="keyword">null</span>, <span class="keyword">null</span>, GsonUtil.toJson(logEvent));</span><br><span class="line"> producer.send(record);</span><br><span class="line"> }</span><br><span class="line"> producer.flush();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> </span>{</span><br><span class="line"> writeDataToKafka();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">message</span><span class="params">(<span class="keyword">int</span> i)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"这是第 "</span> + i + <span class="string">" 行日志！"</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">logLevel</span><span class="params">()</span> </span>{</span><br><span class="line"> Random random = <span class="keyword">new</span> Random();</span><br><span class="line"> <span class="keyword">int</span> number = random.nextInt(<span class="number">4</span>);</span><br><span class="line"> <span class="keyword">switch</span> (number) {</span><br><span class="line"> <span class="keyword">case</span> <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"debug"</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">1</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"info"</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"warn"</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">3</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"error"</span>;</span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"info"</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> String <span class="title">hostIp</span><span class="params">()</span> </span>{</span><br><span class="line"> Random random = <span class="keyword">new</span> Random();</span><br><span class="line"> <span class="keyword">int</span> number = random.nextInt(<span class="number">4</span>);</span><br><span class="line"> <span class="keyword">switch</span> (number) {</span><br><span class="line"> <span class="keyword">case</span> <span class="number">0</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"121.12.17.10"</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">1</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"121.12.17.11"</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">2</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"121.12.17.12"</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">3</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"121.12.17.13"</span>;</span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> <span class="keyword">return</span> <span class="string">"121.12.17.10"</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> Map<String, String> <span class="title">mapData</span><span class="params">()</span> </span>{</span><br><span class="line"> Map<String, String> map = <span class="keyword">new</span> HashMap<>();</span><br><span class="line"> map.put(<span class="string">"app_id"</span>, <span class="string">"11"</span>);</span><br><span class="line"> map.put(<span class="string">"app_name"</span>, <span class="string">"zhisheng"</span>);</span><br><span class="line"> map.put(<span class="string">"cluster_name"</span>, <span class="string">"zhisheng"</span>);</span><br><span class="line"> map.put(<span class="string">"host_ip"</span>, hostIp());</span><br><span class="line"> map.put(<span class="string">"class"</span>, <span class="string">"BuildLogEventDataUtil"</span>);</span><br><span class="line"> map.put(<span class="string">"method"</span>, <span class="string">"main"</span>);</span><br><span class="line"> map.put(<span class="string">"line"</span>, String.valueOf(<span class="keyword">new</span> Random().nextInt(<span class="number">100</span>)));</span><br><span class="line"> <span class="comment">//add more tag</span></span><br><span class="line"> <span class="keyword">return</span> map;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>如果之前 Kafka 中没有 zhisheng_log 这个 topic，运行这个工具类之后也会自动创建这个 topic 了。</p><h3 id="11-5-6-Flink-实时处理日志数据"><a href="#11-5-6-Flink-实时处理日志数据" class="headerlink" title="11.5.6 Flink 实时处理日志数据"></a>11.5.6 Flink 实时处理日志数据</h3><h3 id="11-5-7-处理应用异常日志"><a href="#11-5-7-处理应用异常日志" class="headerlink" title="11.5.7 处理应用异常日志"></a>11.5.7 处理应用异常日志</h3><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/RBYj66M">https://t.zsxq.com/RBYj66M</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p><p>本章属于 Flink 实战篇，前面章节讲了很多 Flink 相关的技术知识点，这章主要是通过技术点来教大家如何去完成一些真实的需求，比如通过 State 去实时统计网站各页面一天的 PV 和 UV、通过 ProcessFunction 去做定时器处理一些延迟的事件（宕机告警）、通过 Async IO 读取告警规则、通过广播变量动态的更新告警规则、如何实时的做到日志告警。</p><p>虽然这些需求换到你们公司去可能不一样，但是这些技术知识点是可以运用到你的项目需求中去的，这里介绍的这些需求，你要学会去分析，然后去判断这些需求到底该使用什么技术来实现会更好，这样才可以做到活学活用。</p>]]></content>
<summary type="html">
<h2 id="11-5-如何实时将应用-Error-日志告警？"><a href="#11-5-如何实时将应用-Error-日志告警？" class="headerlink" title="11.5 如何实时将应用 Error 日志告警？"></a>11.5 如何实时将应用 Error 日志告警？</h2><p>大数据时代，随着公司业务不断的增长，数据量自然也会跟着不断的增长，那么业务应用和集群服务器的的规模也会逐渐扩大，几百台服务器在一般的公司已经是很常见的了。那么将应用服务部署在如此多的服务器上，对开发和运维人员来说都是一个挑战。一个优秀的系统运维平台是需要将部署在这么多服务器上的应用监控信息汇总成一个统一的数据展示平台，方便运维人员做日常的监测、提升运维效率，还可以及时反馈应用的运行状态给应用开发人员。举个例子，应用的运行日志需要按照时间排序做一个展示，并且提供日志下载和日志搜索等服务，这样如果应用出现问题开发人员首先可以根据应用日志的错误信息进行问题的排查。那么该如何实时的将应用的 Error 日志推送给应用开发人员呢，接下来我们将讲解日志的处理方案。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>《Flink 实战与性能优化》—— 如何利用广播变量动态更新告警规则？</title>
<link href="http://www.54tianzhisheng.cn/2021/08/17/flink-in-action-11.4/"/>
<id>http://www.54tianzhisheng.cn/2021/08/17/flink-in-action-11.4/</id>
<published>2021-08-16T16:00:00.000Z</published>
<updated>2022-02-07T14:05:22.257Z</updated>
<content type="html"><![CDATA[<h2 id="11-4-如何利用广播变量动态更新告警规则？"><a href="#11-4-如何利用广播变量动态更新告警规则？" class="headerlink" title="11.4 如何利用广播变量动态更新告警规则？"></a>11.4 如何利用广播变量动态更新告警规则？</h2><p>一个在生产环境运行的流作业有时候会想变更一些作业的配置或者数据流的配置，然后作业可以读取并使用新的配置，而不是通过修改配置然后重启作业来读取配置，毕竟重启一个有状态的流作业代价挺大，本节将带你熟悉 Broadcast，并通过一个案例来教会你如何去动态的更新作业的配置。</p><a id="more"></a><h3 id="11-4-1-BroadcastVariable-简介"><a href="#11-4-1-BroadcastVariable-简介" class="headerlink" title="11.4.1 BroadcastVariable 简介"></a>11.4.1 BroadcastVariable 简介</h3><p>BroadcastVariable 中文意思是广播变量，其实可以理解是一个公共的共享变量（可能是固定不变的数据集合，也可能是动态变化的数据集合），在作业中将该共享变量广播出去，然后下游的所有任务都可以获取到该共享变量，这样就可以不用将这个变量拷贝到下游的每个任务中。之所以设计这个广播变量的原因主要是因为在 Flink 中多并行度的情况下，每个算子或者不同算子运行所在的 Slot 不一致，这就导致它们不会共享同一个内存，也就不可以通过静态变量的方式去获取这些共享变量值。对于这个问题，有不少读者在问过我为啥我设置的静态变量值在本地运行是可以获取到的，在集群环境运行作业就出现空指针啊，该问题其实笔者自己也在生产环境遇到过，所以接下来好好教大家使用！</p><h3 id="11-4-2-如何使用-BroadcastVariable-？"><a href="#11-4-2-如何使用-BroadcastVariable-？" class="headerlink" title="11.4.2 如何使用 BroadcastVariable ？"></a>11.4.2 如何使用 BroadcastVariable ？</h3><p>在 3.4 节中讲过如何 broadcast 算子和 BroadcastStream 如何使用，在 4.1 节中讲解了 Broadcast State 如何使用以及需要注意的地方，注意 BroadcastVariable 只能应用在批作业中，如果要应用在流作业中则需要要使用 BroadcastStream。</p><p>在批作业中通过使用 <code>withBroadcastSet(DataSet, String)</code> 来广播一个 DataSet 数据集合，并可以给这份数据起个名字，如果要获取数据的时候，可以通过 <code>getRuntimeContext().getBroadcastVariable(String)</code> 获取广播出去的变量数据。下面演示一下广播一个 DataSet 变量和获取变量的样例。</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">final</span> ParameterTool params = ParameterTool.fromArgs(args);</span><br><span class="line"><span class="keyword">final</span> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();</span><br><span class="line"></span><br><span class="line"><span class="comment">//1. 待广播的数据</span></span><br><span class="line">DataSet<Integer> toBroadcast = env.fromElements(<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>);</span><br><span class="line"></span><br><span class="line">env.fromElements(<span class="string">"a"</span>, <span class="string">"b"</span>)</span><br><span class="line"> .map(<span class="keyword">new</span> RichMapFunction<String, String>() {</span><br><span class="line"> List<Integer> broadcastData;</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">open</span><span class="params">(Configuration parameters)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> <span class="comment">// 3. 获取广播的 DataSet 数据 作为一个 Collection</span></span><br><span class="line"> broadcastData = getRuntimeContext().getBroadcastVariable(<span class="string">"zhisheng"</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> String <span class="title">map</span><span class="params">(String value)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> <span class="keyword">return</span> broadcastData.get(<span class="number">1</span>) + value;</span><br><span class="line"> }</span><br><span class="line"> }).withBroadcastSet(toBroadcast, <span class="string">"zhisheng"</span>)<span class="comment">// 2. 广播 DataSet</span></span><br><span class="line"> .print();</span><br></pre></td></tr></table></figure><p>注意广播的时候设置的名称和获取的名称要一致，然后运行的结果如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-17-013429.png" alt=""></p><p>流作业中通常使用 BroadcastStream 的方式将变量集合在数据流中传递，可能数据集合会做修改更新，但是修改后其实并不想重启作业去读取这些新修改的配置，因为对于一个流作业来说重启带来的代价很高（需要考虑数据堆积和如何恢复至重启前的状态等问题），那么这种情况下就可以在广播数据流处定时查询数据，这样就能够获取更改后的数据，通常在这种广播数据处获取数据只需要设置一个并行度就好，时间根据需求来判断及时性，一般 1 分钟内的数据变更延迟都是在容忍范围之内。广播流中的元素保证流所有的元素最终都会发到下游的所有并行实例，但是元素到达下游的并行实例的顺序可能不相同。因此，对广播状态的修改不能依赖于输入数据的顺序。在进行 Checkpoint 时，所有的任务都会 Checkpoint 下它们的广播状态。</p><p>另外需要注意的是：广播出去的变量存在于每个节点的内存中，所以这个数据集不能太大，因为广播出去的数据，会一致在内存中存在，除非程序执行结束。个人建议：如果数据集在几十兆或者百兆的时候，可以选择进行广播，如果数据集的大小上 G 的话，就不建议进行广播了。</p><p>上面介绍了下广播变量的在批作业的使用方式，下面通过一个案例来教大家如何在流作业中使用广播变量。</p><h3 id="11-4-3-利用广播变量动态更新告警规则数据需求分析"><a href="#11-4-3-利用广播变量动态更新告警规则数据需求分析" class="headerlink" title="11.4.3 利用广播变量动态更新告警规则数据需求分析"></a>11.4.3 利用广播变量动态更新告警规则数据需求分析</h3><p>在 11.3.3 节中有设计一张简单的告警规则表，通常告警规则是会对外提供接口进行增删改查的，那么随着业务应用上线，开发人员会对其应用服务新增或者修改告警规则（更改之前规则中的阈值），那么更改之后就需要让告警的作业能够去感知到之前的规则发生了变动，所以就需要在作业中想个什么办法去获取到更改后的数据。有两种方式可以让作业知道规则的变更： push 和 pull 模式。</p><p>push 模式则需要在更新、删除、新增接口中不仅操作数据库，还需要额外的发送更新、删除、新增规则的事件到消息队列中，然后作业消费消息队列的数据再去做更新、删除、新增规则，这种及时性有保证，但是可能会有数据不统一的风险（如果消息队列的数据丢了，但是在接口中还是将规则的数据变更存储到数据库）；pull 模式下就需要作业定时去查找一遍所有的告警规则数据，然后存在作业内存中，这个时间可以设置的比较短，比如 1 分钟，这样就能既保证数据的一致性，时间延迟也是在容忍范围之内。</p><p>对于这种动态变化的规则数据，在 Flink 中通常是使用广播流来处理的。那么接下来就演示下如何利用广播变量动态更新告警规则数据，假设我们在数据库中新增告警规则或者修改告警规则指标的阈值，然后看作业中是否会出现相应的变化。</p><h3 id="11-4-4-读取告警规则数据"><a href="#11-4-4-读取告警规则数据" class="headerlink" title="11.4.4 读取告警规则数据"></a>11.4.4 读取告警规则数据</h3><h3 id="11-4-5-监控数据连接规则数据"><a href="#11-4-5-监控数据连接规则数据" class="headerlink" title="11.4.5 监控数据连接规则数据"></a>11.4.5 监控数据连接规则数据</h3><h3 id="11-4-6-小结与反思"><a href="#11-4-6-小结与反思" class="headerlink" title="11.4.6 小结与反思"></a>11.4.6 小结与反思</h3><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/RBYj66M">https://t.zsxq.com/RBYj66M</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h2 id="11-4-如何利用广播变量动态更新告警规则？"><a href="#11-4-如何利用广播变量动态更新告警规则？" class="headerlink" title="11.4 如何利用广播变量动态更新告警规则？"></a>11.4 如何利用广播变量动态更新告警规则？</h2><p>一个在生产环境运行的流作业有时候会想变更一些作业的配置或者数据流的配置，然后作业可以读取并使用新的配置，而不是通过修改配置然后重启作业来读取配置，毕竟重启一个有状态的流作业代价挺大，本节将带你熟悉 Broadcast，并通过一个案例来教会你如何去动态的更新作业的配置。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
<entry>
<title>《Flink 实战与性能优化》—— 如何利用 Async I/O 读取告警规则？</title>
<link href="http://www.54tianzhisheng.cn/2021/08/16/flink-in-action-11.3/"/>
<id>http://www.54tianzhisheng.cn/2021/08/16/flink-in-action-11.3/</id>
<published>2021-08-15T16:00:00.000Z</published>
<updated>2022-02-07T14:03:38.105Z</updated>
<content type="html"><![CDATA[<h2 id="11-3-如何利用-Async-I-O-读取告警规则？"><a href="#11-3-如何利用-Async-I-O-读取告警规则？" class="headerlink" title="11.3 如何利用 Async I/O 读取告警规则？"></a>11.3 如何利用 Async I/O 读取告警规则？</h2><p>Async 中文是异步的意思，在流计算中，使用异步 I/O 能够提升作业整体的计算能力，本节中不仅会讲解异步 I/O 的 API 原理，还会通过一个实战需求（读取告警规则）来讲解异步 I/O 的使用。</p><a id="more"></a><h3 id="11-3-1-为什么需要-Async-I-O？"><a href="#11-3-1-为什么需要-Async-I-O？" class="headerlink" title="11.3.1 为什么需要 Async I/O？"></a>11.3.1 为什么需要 Async I/O？</h3><p>在大多数情况下，IO 操作都是一个耗时的过程，尤其在流计算中，如果在具体的算子里面还有和第三方外部系统（比如数据库、Redis、HBase 等存储系统）做交互，比如在一个 MapFunction 中每来一条数据就要去查找 MySQL 中某张表的数据，然后跟查询出来的数据做关联（同步交互）。查询请求到数据库，再到数据库响应返回数据的整个流程的时间对于流作业来说是比较长的。那么该 Map 算子处理数据的速度就会降下来，在大数据量的情况下很可能会导致整个流作业出现反压问题（在 9.1 节中讲过），那么整个作业的消费延迟就会增加，影响作业整体吞吐量和实时性，从而导致最终该作业处于不可用的状态。</p><p>这种同步（Sync）的与数据库做交互操作，会因耗时太久导致整个作业延迟，如果换成异步的话，就可以同时处理很多请求并同时可以接收响应，这样的话，等待数据库响应的时间就会与其他发送请求和接收响应的时间重叠，相同的等待时间内会处理多个请求，从而比同步的访问要提高不少流处理的吞吐量。虽然也可以通过增大该算子的并行度去执行查数据库，但是这种解决办法需要消耗更多的资源（并行度增加意味着消费的 slot 个数也会增加），这种方法和使用异步处理的方法对比一下，还是使用异步的查询数据库这种方法值得使用。同步操作（Sync I/O）和异步操作（Async I/O）的处理流程如下图所示。</p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-10-12-121929.png" alt=""></p><p>左侧表示的是在流处理中同步的数据库请求，右侧是异步的数据库请求。假设左侧是数据流中 A 数据来了发送一个查询数据库的请求看是否之前存在 A，然后等待查询结果返回，只有等 A 整个查询请求响应后才会继续开始 B 数据的查询请求，依此继续；而右侧是连续的去数据库查询是否存在 A、B、C、D，后面哪个请求先响应就先处理哪个，不需要和左侧的一样要等待上一个请求全部完成才可以开始下一个请求，所以异步的话吞吐量自然就高起来了。但是得注意的是：使用异步这种方法前提是要数据库客户端支持异步的请求，否则可能需要借助线程池来实现异步请求，但是现在主流的数据库通常都支持异步的操作，所以不用太担心。</p><h3 id="11-3-2-Async-I-O-API"><a href="#11-3-2-Async-I-O-API" class="headerlink" title="11.3.2 Async I/O API"></a>11.3.2 Async I/O API</h3><p>Flink 的 Async I/O API 允许用户在数据流处理中使用异步请求，并且还支持超时处理、处理顺序、事件时间、容错。在 Flink 中，如果要使用 Async I/O API，是非常简单的，需要通过下面三个步骤来执行对数据库的异步操作。</p><ul><li>继承 RichAsyncFunction 抽象类或者实现用来分发请求的 AsyncFunction 接口</li><li>返回异步请求的结果的 Future</li><li>在 DataStream 上使用异步操作</li></ul><p>官网也给出案例如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">AsyncDatabaseRequest</span> <span class="keyword">extends</span> <span class="title">RichAsyncFunction</span><<span class="title">String</span>, <span class="title">Tuple2</span><<span class="title">String</span>, <span class="title">String</span>>> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="comment">//数据库的客户端，它可以发出带有 callback 的并发请求</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">transient</span> DatabaseClient client;</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">open</span><span class="params">(Configuration parameters)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> client = <span class="keyword">new</span> DatabaseClient(host, post, credentials);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">close</span><span class="params">()</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> client.close();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">asyncInvoke</span><span class="params">(String key, <span class="keyword">final</span> ResultFuture<Tuple2<String, String>> resultFuture)</span> <span class="keyword">throws</span> Exception</span>{</span><br><span class="line"> <span class="comment">//发出异步请求，接收 future 的结果</span></span><br><span class="line"> <span class="keyword">final</span> Future<String> result = client.query(key);</span><br><span class="line"></span><br><span class="line"> <span class="comment">//设置客户端请求完成后执行的 callback，callback 只是将结果转发给 ResultFuture</span></span><br><span class="line"> CompletableFuture.supplyAsync(<span class="keyword">new</span> Supplier<String>() {</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> String <span class="title">get</span><span class="params">()</span> </span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">return</span> result.get();</span><br><span class="line"> } <span class="keyword">catch</span> (InterruptedException | ExecutionException e) {</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">null</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }).thenAccept( (String dbResult) -> {</span><br><span class="line"> resultFuture.complete(Collections.singleton(<span class="keyword">new</span> Tuple2<>(key, dbResult)));</span><br><span class="line"> });</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">//原始数据</span></span><br><span class="line">DataStream<String> stream = ...;</span><br><span class="line"></span><br><span class="line"><span class="comment">//应用异步 I/O 转换</span></span><br><span class="line">DataStream<Tuple2<String, String>> resultStream =</span><br><span class="line"> AsyncDataStream.unorderedWait(stream, <span class="keyword">new</span> AsyncDatabaseRequest(), <span class="number">1000</span>, TimeUnit.MILLISECONDS, <span class="number">100</span>);</span><br></pre></td></tr></table></figure><p>注意：ResultFuture 在第一次调用 resultFuture.complete 时就已经完成了，后面所有 resultFuture.complete 的调用都会被忽略。</p><p>下面两个参数控制了异步操作：</p><ul><li>Timeout：timeout 定义了异步操作过了多长时间后会被丢弃，这个参数是防止了死的或者失败的请求</li><li>Capacity：这个参数定义可以同时处理多少个异步请求。虽然异步请求会带来更好的吞吐量，但是该操作仍然可能成为流作业的性能瓶颈。限制并发请求的数量可确保操作不会不断累积处理请求，一旦超过 Capacity 值，它将触发反压。</li></ul><h4 id="超时处理"><a href="#超时处理" class="headerlink" title="超时处理"></a>超时处理</h4><h4 id="结果顺序"><a href="#结果顺序" class="headerlink" title="结果顺序"></a>结果顺序</h4><h4 id="事件时间"><a href="#事件时间" class="headerlink" title="事件时间"></a>事件时间</h4><h4 id="容错性保证"><a href="#容错性保证" class="headerlink" title="容错性保证"></a>容错性保证</h4><h4 id="实践技巧"><a href="#实践技巧" class="headerlink" title="实践技巧"></a>实践技巧</h4><h4 id="注意点"><a href="#注意点" class="headerlink" title="注意点"></a>注意点</h4><h3 id="11-3-3-利用-Async-I-O-读取告警规则需求分析"><a href="#11-3-3-利用-Async-I-O-读取告警规则需求分析" class="headerlink" title="11.3.3 利用 Async I/O 读取告警规则需求分析"></a>11.3.3 利用 Async I/O 读取告警规则需求分析</h3><h4 id="监控数据样例"><a href="#监控数据样例" class="headerlink" title="监控数据样例"></a>监控数据样例</h4><h4 id="告警规则表设计"><a href="#告警规则表设计" class="headerlink" title="告警规则表设计"></a>告警规则表设计</h4><h4 id="告警规则实体类"><a href="#告警规则实体类" class="headerlink" title="告警规则实体类"></a>告警规则实体类</h4><h3 id="11-3-4-如何使用-Async-I-O-读取告警规则数据"><a href="#11-3-4-如何使用-Async-I-O-读取告警规则数据" class="headerlink" title="11.3.4 如何使用 Async I/O 读取告警规则数据"></a>11.3.4 如何使用 Async I/O 读取告警规则数据</h3><h3 id="11-3-5-小结与反思"><a href="#11-3-5-小结与反思" class="headerlink" title="11.3.5 小结与反思"></a>11.3.5 小结与反思</h3><p>加入知识星球可以看到上面文章：<a href="https://t.zsxq.com/RBYj66M">https://t.zsxq.com/RBYj66M</a></p><p><img src="http://zhisheng-blog.oss-cn-hangzhou.aliyuncs.com/img/2019-09-25-zsxq.jpg" alt=""></p>]]></content>
<summary type="html">
<h2 id="11-3-如何利用-Async-I-O-读取告警规则？"><a href="#11-3-如何利用-Async-I-O-读取告警规则？" class="headerlink" title="11.3 如何利用 Async I/O 读取告警规则？"></a>11.3 如何利用 Async I/O 读取告警规则？</h2><p>Async 中文是异步的意思，在流计算中，使用异步 I/O 能够提升作业整体的计算能力，本节中不仅会讲解异步 I/O 的 API 原理，还会通过一个实战需求（读取告警规则）来讲解异步 I/O 的使用。</p>
</summary>
<category term="大数据" scheme="http://www.54tianzhisheng.cn/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="流式计算" scheme="http://www.54tianzhisheng.cn/tags/%E6%B5%81%E5%BC%8F%E8%AE%A1%E7%AE%97/"/>
<category term="Flink" scheme="http://www.54tianzhisheng.cn/tags/Flink/"/>
</entry>
</feed>