Improve performance of and reduce overheads of memory management

Whenever we palloc a chunk of memory, traditionally, we prefix the
returned pointer with a pointer to the memory context to which the chunk
belongs.  This is required so that we're able to easily determine the
owning context when performing operations such as pfree() and repalloc().

For the AllocSet context, prior to this commit we additionally prefixed
the pointer to the owning context with the size of the chunk.  This made
the header 16 bytes in size.  This 16-byte overhead was required for all
AllocSet allocations regardless of the allocation size.

For the generation context, the problem was worse; in addition to the
pointer to the owning context and chunk size, we also stored a pointer to
the owning block so that we could track the number of freed chunks on a
block.

The slab allocator had a 16-byte chunk header.

The changes being made here reduce the chunk header size down to just 8
bytes for all 3 of our memory context types.  For small to medium sized
allocations, this significantly increases the number of chunks that we can
fit on a given block which results in much more efficient use of memory.

Additionally, this commit completely changes the rule that pointers to
palloc'd memory must be directly prefixed by a pointer to the owning
memory context and instead, we now insist that they're directly prefixed
by an 8-byte value where the least significant 3-bits are set to a value
to indicate which type of memory context the pointer belongs to.  Using
those 3 bits as an index (known as MemoryContextMethodID) to a new array
which stores the methods for each memory context type, we're now able to
pass the pointer given to functions such as pfree() and repalloc() to the
function specific to that context implementation to allow them to devise
their own methods of finding the memory context which owns the given
allocated chunk of memory.

The reason we're able to reduce the chunk header down to just 8 bytes is
because of the way we make use of the remaining 61 bits of the required
8-byte chunk header.  Here we also implement a general-purpose MemoryChunk
struct which makes use of those 61 remaining bits to allow the storage of
a 30-bit value which the MemoryContext is free to use as it pleases, and
also the number of bytes which must be subtracted from the chunk to get a
reference to the block that the chunk is stored on (also 30 bits).  The 1
additional remaining bit is to denote if the chunk is an "external" chunk
or not.  External here means that the chunk header does not store the
30-bit value or the block offset.  The MemoryContext can use these
external chunks at any time, but must use them if any of the two 30-bit
fields are not large enough for the value(s) that need to be stored in
them.  When the chunk is marked as external, it is up to the MemoryContext
to devise its own means to determine the block offset.

Using 3-bits for the MemoryContextMethodID does mean we're limiting
ourselves to only having a maximum of 8 different memory context types.
We could reduce the bit space for the 30-bit value a little to make way
for more than 3 bits, but it seems like it might be better to do that only
if we ever need more than 8 context types.  This would only be a problem
if some future memory context type which does not use MemoryChunk really
couldn't give up any of the 61 remaining bits in the chunk header.

With this MemoryChunk, each of our 3 memory context types can quickly
obtain a reference to the block any given chunk is located on.  AllocSet
is able to find the context to which the chunk is owned, by first
obtaining a reference to the block by subtracting the block offset as is
stored in the 'hdrmask' field and then referencing the block's 'aset'
field.  The Generation context uses the same method, but GenerationBlock
did not have a field pointing back to the owning context, so one is added
by this commit.

In aset.c and generation.c, all allocations larger than allocChunkLimit
are stored on dedicated blocks.  When there's just a single chunk on a
block like this, it's easy to find the block from the chunk, we just
subtract the size of the block header from the chunk pointer.  The size of
these chunks is also known as we store the endptr on the block, so we can
just subtract the pointer to the allocated memory from that.  Because we
can easily find the owning block and the size of the chunk for these
dedicated blocks, we just always use external chunks for allocation sizes
larger than allocChunkLimit.  For generation.c, this sidesteps the problem
of non-external MemoryChunks being unable to represent chunk sizes >= 1GB.
This is less of a problem for aset.c as we store the free list index in
the MemoryChunk's spare 30-bit field (the value of which will never be
close to using all 30-bits).  We can easily reverse engineer the chunk size
from this when needed.  Storing this saves AllocSetFree() from having to
make a call to AllocSetFreeIndex() to determine which free list to put the
newly freed chunk on.

For the slab allocator, this commit adds a new restriction that slab
chunks cannot be >= 1GB in size.  If there happened to be any users of
slab.c which used chunk sizes this large, they really should be using
AllocSet instead.

Here we also add a restriction that normal non-dedicated blocks cannot be
1GB or larger.  It's now not possible to pass a 'maxBlockSize' >= 1GB
during the creation of an AllocSet or Generation context.  Allocations can
still be larger than 1GB, it's just these will always be on dedicated
blocks (which do not have the 1GB restriction).

Author: Andres Freund, David Rowley
Discussion: https://postgr.es/m/CAApHDvpjauCRXcgcaL6+e3eqecEHoeRm9D-kcbuvBitgPnW=vw@mail.gmail.com

1 files changed, 127 insertions, 0 deletions

diff --git a/src/include/utils/memutils_internal.h b/src/include/utils/memutils_internal.h
new file mode 100644
index 00000000000..9a9f52ef164
--- /dev/null
+++ b/src/include/utils/memutils_internal.h
@@ -0,0 +1,127 @@
+/*-------------------------------------------------------------------------
+ *
+ * memutils_internal.h
+ *	  This file contains declarations for memory allocation utility
+ *	  functions for internal use.
+ *
+ *
+ * Portions Copyright (c) 2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/memutils_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef MEMUTILS_INTERNAL_H
+#define MEMUTILS_INTERNAL_H
+
+#include "utils/memutils.h"
+
+/* These functions implement the MemoryContext API for AllocSet context. */
+extern void *AllocSetAlloc(MemoryContext context, Size size);
+extern void AllocSetFree(void *pointer);
+extern void *AllocSetRealloc(void *pointer, Size size);
+extern void AllocSetReset(MemoryContext context);
+extern void AllocSetDelete(MemoryContext context);
+extern MemoryContext AllocSetGetChunkContext(void *pointer);
+extern Size AllocSetGetChunkSpace(void *pointer);
+extern bool AllocSetIsEmpty(MemoryContext context);
+extern void AllocSetStats(MemoryContext context,
+						  MemoryStatsPrintFunc printfunc, void *passthru,
+						  MemoryContextCounters *totals,
+						  bool print_to_stderr);
+#ifdef MEMORY_CONTEXT_CHECKING
+extern void AllocSetCheck(MemoryContext context);
+#endif
+
+/* These functions implement the MemoryContext API for Generation context. */
+extern void *GenerationAlloc(MemoryContext context, Size size);
+extern void GenerationFree(void *pointer);
+extern void *GenerationRealloc(void *pointer, Size size);
+extern void GenerationReset(MemoryContext context);
+extern void GenerationDelete(MemoryContext context);
+extern MemoryContext GenerationGetChunkContext(void *pointer);
+extern Size GenerationGetChunkSpace(void *pointer);
+extern bool GenerationIsEmpty(MemoryContext context);
+extern void GenerationStats(MemoryContext context,
+							MemoryStatsPrintFunc printfunc, void *passthru,
+							MemoryContextCounters *totals,
+							bool print_to_stderr);
+#ifdef MEMORY_CONTEXT_CHECKING
+extern void GenerationCheck(MemoryContext context);
+#endif
+
+
+/* These functions implement the MemoryContext API for Slab context. */
+extern void *SlabAlloc(MemoryContext context, Size size);
+extern void SlabFree(void *pointer);
+extern void *SlabRealloc(void *pointer, Size size);
+extern void SlabReset(MemoryContext context);
+extern void SlabDelete(MemoryContext context);
+extern MemoryContext SlabGetChunkContext(void *pointer);
+extern Size SlabGetChunkSpace(void *pointer);
+extern bool SlabIsEmpty(MemoryContext context);
+extern void SlabStats(MemoryContext context,
+					  MemoryStatsPrintFunc printfunc, void *passthru,
+					  MemoryContextCounters *totals,
+					  bool print_to_stderr);
+#ifdef MEMORY_CONTEXT_CHECKING
+extern void SlabCheck(MemoryContext context);
+#endif
+
+/*
+ * MemoryContextMethodID
+ *		A unique identifier for each MemoryContext implementation which
+ *		indicates the index into the mcxt_methods[] array. See mcxt.c.
+ */
+typedef enum MemoryContextMethodID
+{
+	MCTX_ASET_ID,
+	MCTX_GENERATION_ID,
+	MCTX_SLAB_ID,
+} MemoryContextMethodID;
+
+/*
+ * The number of bits that 8-byte memory chunk headers can use to encode the
+ * MemoryContextMethodID.
+ */
+#define MEMORY_CONTEXT_METHODID_BITS 3
+#define MEMORY_CONTEXT_METHODID_MASK \
+	UINT64CONST((1 << MEMORY_CONTEXT_METHODID_BITS) - 1)
+
+/*
+ * This routine handles the context-type-independent part of memory
+ * context creation.  It's intended to be called from context-type-
+ * specific creation routines, and noplace else.
+ */
+extern void MemoryContextCreate(MemoryContext node,
+								NodeTag tag,
+								MemoryContextMethodID method_id,
+								MemoryContext parent,
+								const char *name);
+
+/*
+ * GetMemoryChunkMethodID
+ *		Return the MemoryContextMethodID from the uint64 chunk header which
+ *		directly precedes 'pointer'.
+ */
+static inline MemoryContextMethodID
+GetMemoryChunkMethodID(void *pointer)
+{
+	uint64		header;
+
+	/*
+	 * Try to detect bogus pointers handed to us, poorly though we can.
+	 * Presumably, a pointer that isn't MAXALIGNED isn't pointing at an
+	 * allocated chunk.
+	 */
+	Assert(pointer != NULL);
+	Assert(pointer == (void *) MAXALIGN(pointer));
+
+	header = *((uint64 *) ((char *) pointer - sizeof(uint64)));
+
+	return (MemoryContextMethodID) (header & MEMORY_CONTEXT_METHODID_MASK);
+}
+
+#endif							/* MEMUTILS_INTERNAL_H */