GCC gcc.dg/pr118224.c fails for AVR

[dinuxbg]

The gcc.dg/pr118224.c GCC test case fails for AVR backend with an abort when executing with a simulator.

Since the test case calls calloc, I'm filing this as a potential issue in avr-libc.

The failing snippet is:

  if (__builtin_calloc ((~(__SIZE_TYPE__) 0) / 2, 3))
    __builtin_abort ();

The multiplication of the two calloc arguments overflows, and calloc returns non-zero value.

[sprintersb]

As far as I understand, the purpose of that GCC test case is to make sure that the compiler doesn't call calloc when it knows in advance that too much memory is being requested.

So the question is why the compiler is mapping __builtin_calloc to a calloc call instead of replacing the built-in call with NULL.

[dinuxbg]

So the question is why the compiler is mapping __builtin_calloc to a calloc call instead of replacing the built-in call with NULL.

My understanding is the opposite - the call is left on purpose. See this comment in GCC bugzilla. Also the commit message describes the intention to avoid dead-code elimination:

... the following patch will fail to DCE
allocation calls if they have constant size which is too large ..

From the following quote in PR118224 I understand that calloc implementation must reject invalid sizes:

The calloc function returns either a pointer to the allocated space or a null
pointer if the space cannot be allocated or if the product nmemb * size would
wraparound size_t.

Section 7.24.3.1 "The aligned_alloc function" Paragraph 3 ISO/IEC 9899:2024

[sprintersb]

Ok, in that case maybe the best is to XFAIL the test case or similar, rather than turning calloc's MULT into a widening one.

Also @dl8dtl may have an opinion on it.