meson: Add initial version of meson based build system

Autoconf is showing its age, fewer and fewer contributors know how to wrangle
it. Recursive make has a lot of hard to resolve dependency issues and slow
incremental rebuilds. Our home-grown MSVC build system is hard to maintain for
developers not using Windows and runs tests serially. While these and other
issues could individually be addressed with incremental improvements, together
they seem best addressed by moving to a more modern build system.

After evaluating different build system choices, we chose to use meson, to a
good degree based on the adoption by other open source projects.

We decided that it's more realistic to commit a relatively early version of
the new build system and mature it in tree.

This commit adds an initial version of a meson based build system. It supports
building postgres on at least AIX, FreeBSD, Linux, macOS, NetBSD, OpenBSD,
Solaris and Windows (however only gcc is supported on aix, solaris). For
Windows/MSVC postgres can now be built with ninja (faster, particularly for
incremental builds) and msbuild (supporting the visual studio GUI, but
building slower).

Several aspects (e.g. Windows rc file generation, PGXS compatibility, LLVM
bitcode generation, documentation adjustments) are done in subsequent commits
requiring further review. Other aspects (e.g. not installing test-only
extensions) are not yet addressed.

When building on Windows with msbuild, builds are slower when using a visual
studio version older than 2019, because those versions do not support
MultiToolTask, required by meson for intra-target parallelism.

The plan is to remove the MSVC specific build system in src/tools/msvc soon
after reaching feature parity. However, we're not planning to remove the
autoconf/make build system in the near future. Likely we're going to keep at
least the parts required for PGXS to keep working around until all supported
versions build with meson.

Some initial help for postgres developers is at
https://wiki.postgresql.org/wiki/Meson

With contributions from Thomas Munro, John Naylor, Stone Tickle and others.

Author: Andres Freund <[email protected]>
Author: Nazir Bilal Yavuz <[email protected]>
Author: Peter Eisentraut <[email protected]>
Reviewed-By: Peter Eisentraut <[email protected]>
Discussion: https://postgr.es/m/[email protected]

4 files changed, 243 insertions, 0 deletions

diff --git a/src/tools/find_meson b/src/tools/find_meson
new file mode 100755
index 00000000000..50e501a8011
--- /dev/null
+++ b/src/tools/find_meson
@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+#
+# Returns the path to the meson binary, for cases where we need to call it as
+# part of the build, e.g. to install into tmp_install/ for the tests.
+
+import os
+import shlex
+import sys
+
+to_print = []
+
+if 'MUON_PATH' in os.environ:
+    to_print += ['muon', os.environ['MUON_PATH']]
+else:
+    mesonintrospect = os.environ['MESONINTROSPECT']
+    components = shlex.split(mesonintrospect)
+
+    if len(components) < 2:
+        print('expected more than two components, got: %s' % components)
+        sys.exit(1)
+
+    if components[-1] != 'introspect':
+        print('expected introspection at the end')
+        sys.exit(1)
+
+    to_print += ['meson'] + components[:-1]
+
+print('\n'.join(to_print), end='')
+
+sys.exit(0)
diff --git a/src/tools/gen_export.pl b/src/tools/gen_export.pl
new file mode 100644
index 00000000000..68b3ab86614
--- /dev/null
+++ b/src/tools/gen_export.pl
@@ -0,0 +1,81 @@
+use strict;
+use warnings;
+use Getopt::Long;
+
+my $format;
+my $libname;
+my $input;
+my $output;
+
+GetOptions(
+	'format:s'   => \$format,
+	'libname:s'    => \$libname,
+	'input:s' => \$input,
+	'output:s'  => \$output) or die "wrong arguments";
+
+if (not ($format eq 'aix' or $format eq 'darwin' or $format eq 'gnu' or $format eq 'win'))
+{
+	die "$0: $format is not yet handled (only aix, darwin, gnu, win are)\n";
+}
+
+open(my $input_handle, '<', $input)
+  or die "$0: could not open input file '$input': $!\n";
+
+open(my $output_handle, '>', $output)
+  or die "$0: could not open output file '$output': $!\n";
+
+
+if ($format eq 'gnu')
+{
+	print $output_handle "{
+  global:
+";
+}
+elsif ($format eq 'win')
+{
+	# XXX: Looks like specifying LIBRARY $libname is optional, which makes it
+	# easier to build a generic command for generating export files...
+	if ($libname)
+	{
+		print $output_handle "LIBRARY $libname\n";
+	}
+	print $output_handle "EXPORTS\n";
+}
+
+while (<$input_handle>)
+{
+	if (/^#/)
+	{
+		# don't do anything with a comment
+	}
+	elsif (/^(\S+)\s+(\S+)/)
+	{
+		if ($format eq 'aix')
+		{
+			print $output_handle "$1\n";
+		}
+		elsif ($format eq 'darwin')
+		{
+			print $output_handle "_$1\n";
+		}
+		elsif ($format eq 'gnu')
+		{
+			print $output_handle "    $1;\n";
+		}
+		elsif ($format eq 'win')
+		{
+			print $output_handle "$1 @ $2\n";
+		}
+	}
+	else
+	{
+		die "$0: unexpected line $_\n";
+	}
+}
+
+if ($format eq 'gnu')
+{
+	print $output_handle "  local: *;
+};
+";
+}
diff --git a/src/tools/pgflex b/src/tools/pgflex
new file mode 100755
index 00000000000..baabe2df1c8
--- /dev/null
+++ b/src/tools/pgflex
@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+
+#
+# Wrapper around flex that:
+# - ensures lex.backup is created in a private directory
+# - can error out if lex.backup is created (--no-backup)
+# - can fix warnings (--fix-warnings)
+# - works around concurrency issues with win_flex.exe:
+#   https://github.com/lexxmark/winflexbison/issues/86
+
+import argparse
+import os
+import subprocess
+import sys
+from os.path import abspath
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument('--flex', type=abspath, required=True)
+parser.add_argument('--perl', type=abspath, required=True)
+parser.add_argument('--builddir', type=abspath, required=True)
+parser.add_argument('--srcdir', type=abspath, required=True)
+parser.add_argument('--privatedir', type=abspath, required=True,
+                    help='private directory for target')
+
+parser.add_argument('-o', dest='output_file', type=abspath, required=True,
+                    help='output file')
+parser.add_argument('-i', dest='input_file', type=abspath, help='input file')
+
+
+parser.add_argument('--fix-warnings', action='store_true',
+                    help='whether to fix warnings in generated file')
+parser.add_argument('--no-backup', action='store_true',
+                    help='whether no_backup is enabled or not')
+
+parser.add_argument('flex_flags', nargs='*', help='flags passed on to flex')
+
+args = parser.parse_args()
+
+# Since 'lex.backup' is always named that and ninja uses the top level build
+# directory as current directory for all commands, change directory to
+# temporary directory to avoid conflicts between concurrent flex
+# invocations. Only unreleased versions of flex have an argument to change
+# lex.filename to be named differently.
+if not os.path.isdir(args.privatedir):
+    os.mkdir(args.privatedir)
+os.chdir(args.privatedir)
+
+# win_flex.exe generates names in a racy way, sometimes leading to random
+# "error deleting file" failures and sometimes to intermingled file
+# contents. Set FLEX_TMP_DIR to the target private directory to avoid
+# that. That environment variable isn't consulted on other platforms, so we
+# don't even need to make this conditional.
+env = {'FLEX_TMP_DIR': args.privatedir}
+
+# build flex invocation
+command = [args.flex, '-o', args.output_file]
+if args.no_backup:
+    command += ['-b']
+command += args.flex_flags
+command += [args.input_file]
+
+# create .c file from .l file
+sp = subprocess.run(command, env=env)
+if sp.returncode != 0:
+    sys.exit(sp.returncode)
+
+# check lex.backup
+if args.no_backup:
+    with open('lex.backup') as lex:
+        if len(lex.readlines()) != 1:
+            sys.exit('Scanner requires backup; see lex.backup.')
+    os.remove('lex.backup')
+
+# fix warnings
+if args.fix_warnings:
+    fix_warning_script = os.path.join(args.srcdir,
+                                      'src/tools/fix-old-flex-code.pl')
+
+    command = [args.perl, fix_warning_script, args.output_file]
+    sp = subprocess.run(command)
+    if sp.returncode != 0:
+        sys.exit(sp.returncode)
+
+sys.exit(0)
diff --git a/src/tools/testwrap b/src/tools/testwrap
new file mode 100755
index 00000000000..7a64fe76a2d
--- /dev/null
+++ b/src/tools/testwrap
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+
+import argparse
+import shutil
+import subprocess
+import os
+import sys
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument('--srcdir', help='source directory of test', type=str)
+parser.add_argument('--basedir', help='base directory of test', type=str)
+parser.add_argument('--testgroup', help='test group', type=str)
+parser.add_argument('--testname', help='test name', type=str)
+parser.add_argument('test_command', nargs='*')
+
+args = parser.parse_args()
+
+testdir = '{}/testrun/{}/{}'.format(
+    args.basedir, args.testgroup, args.testname)
+
+print('# executing test in {} group {} test {}'.format(
+    testdir, args.testgroup, args.testname))
+sys.stdout.flush()
+
+if os.path.exists(testdir) and os.path.isdir(testdir):
+    shutil.rmtree(testdir)
+os.makedirs(testdir)
+
+os.chdir(args.srcdir)
+
+# mark test as having started
+open(os.path.join(testdir, 'test.start'), 'x')
+
+env_dict = {**os.environ,
+            'TESTDATADIR': os.path.join(testdir, 'data'),
+            'TESTLOGDIR': os.path.join(testdir, 'log')}
+
+sp = subprocess.run(args.test_command, env=env_dict)
+
+if sp.returncode == 0:
+    print('# test succeeded')
+    open(os.path.join(testdir, 'test.success'), 'x')
+else:
+    print('# test failed')
+    open(os.path.join(testdir, 'test.fail'), 'x')
+sys.exit(sp.returncode)