Fix datatype confusion with the new lossy GiST distance functions.

We can only support a lossy distance function when the distance function's
datatype is comparable with the original ordering operator's datatype.
The distance function always returns a float8, so we are limited to float8,
and float4 (by a hard-coded cast of the float8 to float4).

In light of this limitation, it seems like a good idea to have a separate
'recheck' flag for the ORDER BY expressions, so that if you have a non-lossy
distance function, it still works with lossy quals. There are cases like
that with the build-in or contrib opclasses, but it's plausible.

There was a hidden assumption that the ORDER BY values returned by GiST
match the original ordering operator's return type, but there are plenty
of examples where that's not true, e.g. in btree_gist and pg_trgm. As long
as the distance function is not lossy, we can tolerate that and just not
return the distance to the executor (or rather, always return NULL). The
executor doesn't need the distances if there are no lossy results.

There was another little bug: the recheck variable was not initialized
before calling the distance function. That revealed the bigger issue,
as the executor tried to reorder tuples that didn't need reordering, and
that failed because of the datatype mismatch.

1 files changed, 1 insertions, 0 deletions

diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h
index 865d36403a2..5a0d724aca5 100644
--- a/src/include/access/relscan.h
+++ b/src/include/access/relscan.h
@@ -99,6 +99,7 @@ typedef struct IndexScanDescData
 	 */
 	Datum	   *xs_orderbyvals;
 	bool	   *xs_orderbynulls;
+	bool		xs_recheckorderby;	/* T means ORDER BY exprs must be rechecked */
 
 	/* state data for traversing HOT chains in index_getnext */
 	bool		xs_continue_hot;	/* T if must keep walking HOT chain */