Avoid invalid alloc size error in shm_mq

Avoid invalid alloc size error in shm_mq

In shm_mq_receive(), a huge payload could trigger an unjustified
"invalid memory alloc request size" error due to the way the buffer
size is increased.

Add error checks (documenting the upper limit) and avoid the error by
limiting the allocation size to MaxAllocSize.

Author: Markus Wanner <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/3bb363e7-ac04-0ac4-9fe8-db1148755bfa%402ndquadrant.com