Issues when rendering some chunks sections

[Paulomart]

Hey!

Thanks for the awesome work on maintaining this iconic project! Im currently trying to move my rendering pipeline to use this version of mcmap.

In that process I discovered a very strange problem: Some chunk sections seem 'scrambled up'. All blocks inside are positioned quite random.

See for example see this render (1.16.4 world):

Note: The images in this issue have been scaled by GitHub, open the image in a new tab to see the full details.

This happens for all sorts of version and server software combinations. But I should note here, that the world savefile that was used here was written by a mineflayer bot. I'm also not sure if its an issue with their implementation of the anvil storage format or with mcmap itself.

---

I tried to write down how I approached this and where I think might be the cause of the issue.

My first impression was, that something wrote the chunks wrong to disk. So I started up a Minecraft client and loaded the world.

As you can see the red marked section was loaded correctly. This was the same chunk section that rendered wrong before. Its the scrambled up section on the right. Same savefile. Same chunks.

Now I was sure that the chunks wore at least in some kind of readable format on disk. I saved and quitted Minecraft and rendered the world again.

Before	After load & save

As you can see Minecraft somehow fixed these chunks and as result mcmap was able to render them like it should. You can still see some chunks that are 'scrambled up', these chunks were just not loaded by the client. Note: The Minecraft client generated some new terrain that I cropped away.

The Minecraft client must have altered the chunk section in some form. I opened both worlds using NBT Studio and compared the chunk section in question.

Before	After load & save
value 37191016277606400 33825 0 68719476736 0 2357352929953794 68719575043 4398046511104 5242880 5497558138882 4194304 4398052802624 67108864 2052 4196416 68721641632 2206541545472 2322168557862912 2251799813685248 70368744180739 72130161805361152 14338 2147483712 0 0 0 0 103082360928 0 5497558138884 0 4398046511109 6597069766656 4 4194304 4398046511104 167772160 7881299347898368 7 0 106404249600 0 0 0 0 0 0 0 108089689692438528 370702544327933952 4465995 320357116197 373109799749943296 482272188221620228 4567462 4753871744320 335918783468142592 370596991211667460 271691 7347200 0 111572942428569600 0 0 0 0 0 0 0 105553116266496 98304 4682935510272 335918783473385472 445082306923724800 339341 4789331472736 373109799744700416 407892425625829376 4499820 147450045600 7696581394432 0 0 101475 0 0 0 0 0 0 0 0 3298534883328 401946266742816768 141676 12227849216 158090440594161664 444906385063280640 11661 4677019648 7937099563008 0 0 0 106404249600 373034608490446848 265 0 0 0 0 0 0 0 3 347411456 11659609995476992 444730463202836480 11661 11119470752 364281946177536 0 0 0 0 108195242708041728 0 278211913 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 106300440576 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 111464090777419776 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0	value 1229782938247299072 0 35184372088832 144678138029277184 2305895798693429248 1125899906842624 562949953748992 1125899907170304 1125925678743552 145241087984795648 1128236369051648 35193537691648 8707506176 35651584 9007199305269760 577059032657821696 144115325514809344 0 0 0 52789446180864 1125899906842624 327680 1125899907170304 1125925676646400 1125899906842624 1125899906842624 5242880 30182211584 0 858980352 0 0 0 0 0 52789446180864 1296035281371136 188900373233664 1333565278453760 1352291621797888 1333565278126080 1314800279748608 170135370334208 30182211584 0 858980352 0 0 0 0 0 52776561278976 1296035281371136 188900373233664 207665371283456 1352291621797888 207665371283456 1314800279748608 82174440112128 30064771072 0 858980352 0 0 0 0 0 0 12884901888 83347145490432 14151313260544 85654219259904 14151318503424 5281920581632 31943819264 0 0 858980352 664591269888 0 0 0 0 0 12884901888 810490527744 12978501976064 14151313260544 12978507218944 810406641664 0 0 0 808648704 664591269888 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 855638016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 855638016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

It seems pretty clear what happened here: The chunk section must have been stored in some inefficiency way, the Minecraft client compacted it from 342 down to 256 longs.

I looked at the post116 function. Maybe this function could not deal with this strange case?

mcmap/src/chunk_format_versions/section_format.cpp

Lines 6 to 30 in e4440a7

	void post116(const uint8_t index_length,
	const std::vector<int64_t> *blockStates,
	Section::block_array &buffer) {
	// NEW in 1.16, longs are padded by 0s when a block cannot fit, so no more
	// overflow to deal with !
	for (uint16_t index = 0; index < 4096; index++) {
	// Determine how many indexes each long holds
	const uint8_t blocksPerLong = 64 / index_length;
	// Calculate where in the long array is the long containing the right index.
	const uint16_t longIndex = index / blocksPerLong;
	// Once we located a long, we have to know where in the 64 bits
	// the relevant block is located.
	const uint8_t padding = (index - longIndex * blocksPerLong) * index_length;
	// Bring the data to the first bits of the long, then extract it by bitwise
	// comparison
	const uint16_t blockIndex = ((*blockStates)[longIndex] >> padding) &
	((uint64_t(1) << index_length) - 1);
	buffer[index] = blockIndex;
	}
	}

I added a debug statement to analyse the different blockstates lengths and index_lengths. This were the counts and the lengths that occurred while rendering the world.

1876 [Deep Debug] blockStates length: 256; index_length: 4;
  35 [Deep Debug] blockStates length: 342; index_length: 4;
 508 [Deep Debug] blockStates length: 342; index_length: 5;

When ignoring all chunk sections that have blockStates->size() == 342 you can see that more then the scrambled sections get removed. When also filtering on the index_length == 4, we get only the scrambled sections removed.

blockStates->size() == 342	&& index_length == 4	(inverted)

I did not test this behaviour with other blockStates lengths. From my testing with very different worlds and versions, some sort of error happens in all versions from 1.13 to 1.16.5.

Expand for a test matrix with different versions

Note: This also contains worlds (1.8 - 1.12) that were rendered using a legacy version of mcmap. These dont show this error.
Reminder: These worlds were generated using the specified server software, then 'downloaded' by mineflayer and then rendered.

---

I tried 'fixing' the chunk reading in mcmap myself, even reimplemented the function heavily inspired from nms code:

  for (uint16_t index = 0; index < 4096; index++) {
    const uint16_t j = index * index_length;
    const uint16_t k = j >> 6;
    const uint16_t l = (index + 1) * index_length - 1 >> 6;
    const uint16_t i1 = j ^ k << 6;

    uint16_t blockIndex = 0;

    if (k == l) {
        blockIndex = (int) ((unsigned((*blockStates)[k])) >> i1 & mask);
    } else {
        int j1 = 64 - i1;

        blockIndex = (int) (((unsigned((*blockStates)[k])) >> i1 | (*blockStates)[l] << j1) & mask);
    }

    buffer[index] = blockIndex;
  }

Sadly this didnt bring any success and I'm not really sure how to go on from here: It seems that there is an issue how mineflayer writes the chunks to disk. But at the same time Minecraft somehow is able to handle this. I would assume that it would be best to fix this in mineflayer, but I'm not really sure I can pinpoint the bug over there.

Maybe you could give some hints or help getting this resolved? Any help is really appreciated. Thanks for reading this really long issue.

[spoutn1k]

Wow man thanks for such a detailed issue ! You narrowed down perfectly what I suspect is happening.

I have seen this before. Minecraft changed the way the chunks are saved in 1.16, to forego space for efficiency, and those scrambled chunks used to appear everywhere at first: they are the interpretation of data written in one format with the expectation of the other. This format is linked to the DataVersion tag that appears in your screenshots.

What I suspect is happening here is that there is a mismatch between the version and the format used by the mod, and this is the result. What can also happen is that I mislabeled the Dataversion number that introduced this format, and mcmap is blundering hard.

I can look into it, do you have a region file where this is happening ? I can also get into more details about those format changes to help if this is not on mcmap. Probably will be though.

Edit: It would seem the format is wrong on the index size.

On your screenshots, the palette has 15 elements -> the index size should be 4, as you can fit up to 16 on 4bits.

This is the binary value of the first long in the output of the mod:

0000000010000100001000010000100001000010000100000000000000000000

And this is the output of the first long of the minecraft output:

0001000100010001000100010001000100010001000100010000000000000000

You can clearly see repeating patterns (similar adjacent blocks), but of different sizes ! This clearly confuses mcmap. So, two possibilities:

• The chunks are considered as corrupted, and Minecraft re-generates them on open & save;
• I am missing something and Minecraft is able to understand this and correct it.

[Paulomart]

Thanks for the reply.

I have seen this before. Minecraft changed the way the chunks are saved in 1.16, to forego space for efficiency, and those scrambled chunks used to appear everywhere at first: they are the interpretation of data written in one format with the expectation of the other. This format is linked to the DataVersion tag that appears in your screenshots.

I assume you mean the zero padding for the long arrays: I have tested this with different DataVersions: The problem also appears when mcmap is using the pre116 method to load chunk sections. So this happens on chunk sections that are zero padded and non zero padded.

I can look into it, do you have a region file where this is happening ? I can also get into more details about those format changes to help if this is not on mcmap. Probably will be though.

Sure, I just forgot to attach it. Here it is: world-1.16.4-bukkit.zip

On your screenshots, the palette has 15 elements -> the index size should be 4, as you can fit up to 16 on 4bits.

Yes. That confused me as well. I tried to adjust the decoding function to use the other length, so it would respect that.

You can also see that Mineflayer writes 4 bits of zeros and then continues to use 5 bits per block. I dont see how any code would be able to guess the 4 bits at the beginning.

The chunks are considered as corrupted, and Minecraft re-generates them on open & save;

Im really really sure this is not the case: As you can see in the rendered images after load and save, there is a world generation border with the surrounding chunks. If Minecraft was not able to read some chunk section, these borders would also happen inside the area of question.

Another point on this: The level.dat file, which stores the seed, was just copied over from another world, so the seeds could never match. There is no per chunk seed, as far as I know, right?

Minecraft is able to understand this and correct it.

I think this is whats happening. But I cannot understand why. I will look at Mineflayers implementation later today and check again if there is some funky business going on. Maybe its possible to just fix it on their side, as this seems the only case it issues arises.

Thanks again for looking into this :)

[spoutn1k]

Hey is this still an issue ? Has it been fixed on the Mineflayer side ? I looked at the data again today and it does seem like a format error of the world and not mcmap. For reference, one of the erroneous chunks uses pre-1.16 format and states a DataVersion of 2567, when the 1.16 format has been used from DataVersions of 2534 onwards. I may be able to create a patch to check format errors, but I certainly do not want to make mcmap more complex than it already is ...

[Paulomart]

Hey, sorry for leaving this untouched so long. It really seems that there is an issue in Mineflayer. Sadly I was not able to nail it down further then described here, but I tested that Mineflayer is able to load and write chunks in this corrupt format (Even asserting the blocks one by one worked out).

I certainly do not want to make mcmap more complex than it already is

I agree 100% with you. But maybe Im or you are able to submit an fix based on such patch? My expertise in the world region format is pretty limited and so far I havent been able to figure it out...

[spoutn1k]

Can you verify the mineflayer branch fixes the issue ? It works for the world you provided.

Now we need to discuss how this should be bundled. First of all, what code does bukkit use to generate regions ? Is it using minecraft binaries or all homebrew code ? Because if this is an issue in Minecraft itself, adding this to the project would not be an issue.