i386: Return wider vectors in registers for x86_64 MS ABI.
Enables returning 256-bit and 512-bit vectors in registers for x86_64 MS ABI, improving code generation for AVX/AVX512.
This patch modifies GCC to return 256-bit and 512-bit vectors in registers (YMM0/ZMM0) when compiling for x86_64 Windows targets using the MS ABI. Previously, these vector returns were incorrectly classified as memory returns, leading to inefficient code generation with hidden sret pointer returns. This change improves code generation when AVX/AVX512 is enabled, ensuring that vector return values are passed in registers as expected.
In Details
On x86_64 Windows targets using MS ABI, GCC classified 256-bit and 512-bit vector returns as memory returns, which caused hidden sret pointer returns where YMM0/ZMM0 returns are expected. The patch modifies function_value_ms_64 to handle 32-byte and 64-byte vector returns in registers when supported, and also updates ix86_return_in_memory to avoid forcing eligible vector returns to memory.
For Context
The Application Binary Interface (ABI) defines how function arguments and return values are passed between compiled programs. The MS ABI is the calling convention used on Windows. x86_64 is the 64-bit extension of the x86 instruction set architecture. AVX (Advanced Vector Extensions) and AVX512 are instruction set extensions that enable SIMD (Single Instruction, Multiple Data) operations on 256-bit and 512-bit vectors, respectively. This change ensures that vector return values are handled correctly according to the MS ABI, leading to more efficient code generation for AVX/AVX512.