[Bug]: assertion failed: (*header).pd_special >= SizeOfPageHeaderData as u16

[Eli-Airis]

What happened?

I am using the official docker image unchanged:

$ docker images
REPOSITORY                 TAG               IMAGE ID       CREATED        SIZE
timescale/timescaledb-ha   pg17.2-ts2.17.2   16b410ac50dc   3 weeks ago    1.79GB

and I'm frequently getting the following assertion error when (multi-)inserting into a table with a diskann index: assertion failed: (*header).pd_special >= SizeOfPageHeaderData as u16.
My loaded extensions:

# SELECT extname, extversion
FROM pg_extension;
       extname       | extversion
---------------------+------------
 plpgsql             | 1.0
 vector              | 0.8.0
 vectorscale         | 0.5.1
 pg_stat_monitor     | 2.1
 pageinspect         | 1.12
 timescaledb_toolkit | 1.19.0
 timescaledb         | 2.17.2

I did not set any option explicitly when creating the index:

# CREATE INDEX my_index ON my_table USING diskann (my_vector_field vector_cosine_ops);
NOTICE:  Starting index build. num_neighbors=-1 search_list_size=100, max_alpha=1.2, storage_layout=SbqCompression

I only changed the following parameters in my configuration based on https://pgtune.leopard.in.ua/?dbVersion=17&osType=linux&dbType=dw&cpuNum=32&totalMemory=32&totalMemoryUnit=GB&connectionNum=&hdType=ssd:

# DB Version: 17
# OS Type: linux
# DB Type: dw
# Total Memory (RAM): 32 GB
# CPUs num: 32
# Data Storage: ssd

#max_connections = 40  # Why was this even suggested for my strong machine?
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 6553kB
huge_pages = try
min_wal_size = 4GB
max_wal_size = 16GB
max_worker_processes = 32
max_parallel_workers_per_gather = 16
max_parallel_workers = 32
max_parallel_maintenance_workers = 4

I have also tried running without any change to the default postgresql.conf, and still got the same assertion error.
SHOW blocksize returns 8192

I would appreciate any help resolving this.

pgvectorscale extension affected

0.5.1

PostgreSQL version used

17.2

What operating system did you use?

Linux my-db-machine 6.8.0-1020-gcp #22-Ubuntu SMP Mon Dec 9 17:09:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

What installation method did you use?

Docker

What platform did you run on?

Google Cloud Platform (GCP)

Relevant log output and stack trace

All I'm getting in the container's log is:
2025-01-16 18:49:08.093 UTC [538] ERROR:  assertion failed: (*header).pd_special >= SizeOfPageHeaderData as u16
followed by something like
2025-01-16 18:49:08.093 UTC [538] STATEMENT:  INSERT INTO my_table (field1, field2, field3, my_vector_field, final_field) VALUES ($1::VARCHAR, $2::VARCHAR, $3, $4, $5::VARCHAR), ($6::VARCHAR, $7::VARCHAR, $8, $9, $10::VARCHAR), ($11::VARCHAR, $12::VARCHAR, $13, $14, $15::VARCHAR), ($16::VARCHAR, $17::VARCHAR, $18, $19, $20::VARCHAR), ($21::VARCHAR, $22::VARCHAR, $23, $24, $25::VARCHAR), ($26::VARCHAR, $27::VARCHAR, $28, $29, $30::VARCHAR), ($31::VARCHAR, $32::VARCHAR, $33, $34, $35::VARCHAR), ($36::VARCHAR, $37::VARCHAR, $38, $39, $40::VARCHAR), ($41::VARCHAR, $42::VARCHAR, $43, $44, $45::VARCHAR), ($46::VARCHAR, $47::VARCHAR, $48, $49, $50::VARCHAR), ($51::VARCHAR, $52::VARCHAR, $53, $54, $55::VARCHAR), ($56::VARCHAR, $57::VARCHAR, $58, $59, $60::VARCHAR), ($61::VARCHAR, $62::VARCHAR, $63, $64, $65::VARCHAR), ($66::VARCHAR, $67::VARCHAR, $68, $69, $70::VARCHAR), ($71::VARCHAR, $72::VARCHAR, $73, $74, $75::VARCHAR), ($76::VARCHAR, $77::VARCHAR, $78, $79, $80::VARCHAR), ($81::VARCHAR, $82::VARCHAR, $83, $84, $85::VARCHAR), ($86::VARCHAR, $87::VARCHAR, $88, $89, $90::VARCHAR), ($91::VARCHAR, $92::VARCHAR, $93, $94, $95::VARCHAR), ($96::VARCHAR, $97::VARCHAR, $98, $99, $100::VARCHAR), ($101::VARCHAR, $102::VARCHAR, $103, $104, $105::VARCHAR), ($106::VARCHAR, $107::VARCHAR, $108, $109, $110::VARCHAR), ($111::VARCHAR, $112::VARCHAR, $113, $114, $115::VARCHAR), ($116::VARCHAR, $117::VARCHAR, $118, $119, $120::VARCHAR), ($121::VARCHAR, $122::VARCHAR, $123, $124, $125::VARCHAR), ($126::VARCHAR, $127::VARCHAR, $128, $129, $130::VARCHAR), ($131::VARCHAR, $132::VARCHAR, $133, $134, $135::VARCHAR), ($136::VARCHAR, $137::VARCHAR, $138, $139, $140::VARCHAR), ($141::VARCHAR, $142::VARCHAR, $143, $144, $145::VARCHAR), ($146::VARCHAR, $147::VARCHAR, $148, $149, $150::VARCHAR), ($151::VARCHAR, $152::VARCHAR, $153, $154, $155::VARCHAR), ($156::VARCHAR, $157::VARCHAR, $158, $159, $160::VARCHAR), ($161::VARCHAR, $162::VARCHAR, $163, $164, $165::VARCHAR), ($166::VARCHAR, $167::VARCHAR, $168, $169, $170::VARCHAR), ($171::VARCHAR, $172::VARCHAR, $173, $174, $175::VARCHAR), ($176::VARCHAR, $177::VARCHAR, $178, $179, $180::VARCHAR), ($181::VARCHAR, $182::VARCHAR, $183, $184, $185::VARCHAR), ($186::VARCHAR, $187::VARCHAR, $188, $189, $190::VARCHAR), ($191::VARCHAR, $192::VARCHAR, $193, $194, $195::VARCHAR), ($196::VARCHAR, $197::VARCHAR, $198, $199, $200::VARCHAR), ($201::VARCHAR, $202::VARCHAR, $203, $204, $205::VARCHAR), ($206::VARCHAR, $207::VARCHAR, $208, $209, $210::VARCHAR), ($211::VARCHAR, $212::VARCHAR, $213, $214, $215::VARCHAR), ($216::VARCHAR, $217::VARCHAR, $218, $219, $220::VARCHAR), ($221::VARCHAR, $222::VARCHAR, $223, $224, $225::VARCHAR), ($226::VARCHAR, $227::VARCHAR, $228, $229, $230::VARCHAR), ($231::VARCHAR, $232::VARCHAR, $233, $234, $235::VARCHAR), ($236::VARCHAR, $237::VARCHAR, $238, $239, $240::VARCHAR), ($241::VARCHAR, $242::VARCHAR, $243, $244, $245::VARCHAR), ($246::VARCHAR, $247::VARCHAR, $248, $249, $250::VARCHAR), ($251::VARCHAR, $252::VARCHAR, $253, $254, $255::VARCHAR), ($256::VARCHAR, $257::VARCHAR, $258, $259, $260::VARCHAR), ($261::VARCHAR, $262::VARCHAR, $263, $264, $265::VARCHAR), ($266::VARCHAR, $267::VARCHAR, $268, $269, $270::VARCHAR), ($271::VARCHAR, $272::VARCHAR, $273, $274, $275::VARCHAR), ($276::VARCHAR, $277::VARCHAR, $278, $279, $280::VARCHAR), ($281::VARCHAR, $282::VARCHAR, $283, $284, $285::VARCHAR), ($286::VARCHAR, $287::VARCHAR, $288, $289, $290::VARCHAR), ($291::VARCHAR, $292::VARCHAR, $293, $294, $295::VARCHAR), ($296::VARCHAR, $297::VARCHAR, $298, $299, $300::VARCHAR), ($301::VARCHAR, $302::VARCHAR, $303, $304, $305::VARCHAR), ($306::VARCHAR, $307::VARCHAR, $308, $309, $310::VARCHAR), ($311::VARCHAR, $312::VARCHAR, $313, $314, $315::VARCHAR), ($316::VARCHAR, $317::VARCHAR, $318, $319, $320::VARCHAR), ($321::VARCHAR, $322::VARCHAR, $323, $324, $325::VARCHAR), ($326::VARCHAR, $327::VARCHAR, $328, $329, $330::VARCHAR), ($331::VARCHAR, $332::VARCHAR, $333, $334, $335::VARCHAR), ($336::VARCHAR, $337::VARCHAR, $338, $339, $340::VARCHAR), ($341::VARCHAR, $342::VARCHAR, $343, $344, $345::VARCHAR), ($346::VARCHAR, $347::VARCHAR, $348, $349, $350::VAR...

How can we reproduce the bug?

The only unusual thing about my vector field is its size isn't a power of 2, it's 513.

I run a multi-insert from Sqlalchemy of batches of size 1000.

Are you going to work on the bugfix?

None

[Eli-Airis]

I also checked the page size:

# CREATE EXTENSION pageinspect;
CREATE EXTENSION
# SELECT page_header(get_raw_page('pg_class', 0));
              page_header
----------------------------------------
 (0/278CFA0,0,1,212,4400,8192,8192,4,0)

[tjgreen42]

Ouch, page corruption. @Eli-Airis would you be willing to share the workload you're running that triggers this? Likely me or @syvb will need to try to repro this ourselves to make progress. (I can share contact info if you're not comfortable posting it somewhere publicly accessible).

[Eli-Airis]

I'll happily share, pending approval on Sunday morning.

[Eli-Airis]

Hi @tjgreen42 and @smoya, I have approval for sharing the workload. Could you please provide the email address that I should give access to? The workload isn't sensitive but we would still prefer you don't share it outside of Timescale.

[Eli-Airis]

Looks like the regression was introduced between versions 2.17.1 and 2.17.2:

I COULD NOT reproduce the assertion error with the docker image tagged timescale/timescaledb-ha:pg16.4-ts2.17.1 (image ID 7f9533ca34d7),
and COULD reproduce the error with the docker images tagged timescale/timescaledb-ha:pg16 (image ID 2bfd8ae73876), timescale/timescaledb-ha:pg17-ts2.17-oss (image ID f512c54b681c), or timescale/timescaledb-ha:pg17.2-ts2.17.2 (image ID 16b410ac50dc)

Note that the issue is reproduced with version 2.17.2 both on pg16 and on pg17.

[tjgreen42]

Hi @Eli-Airis, my work email address is [email protected]. Thanks much for being willing to share the workload.

[Eli-Airis]

Great! I shared the payload with you.