Add support for sparse_vector queries against semantic_text fields

[kderusso]

Adds support for running sparse_vector queries directly against semantic_text fields without having to pass in nested queries from the client or knowing the semantic subfields.

Example script to test:

PUT _inference/sparse_embedding/my-elser-endpoint
{
  "service": "elser",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1
  },
  "task_settings": {}
}

PUT my-index-1
{
    "mappings": {
        "properties": {
            "inference_field": {
                "type": "semantic_text",
                "inference_id": "my-elser-endpoint"
            },
            "text_field": {
                "type": "text"
            }
        }
    }
}

PUT my-index-2
{
  "mappings": {
    "properties": {
      "inference_field": {
        "type": "sparse_vector"
      },
      "text_field": { 
        "type": "text"
      }
    }
  }
}

POST my-index-1/_doc/1
{
  "inference_field": "a test value",
  "text_field": "a test value"
}

POST my-index-2/_doc/1
{
  "inference_field": { "test": 1, "value": 2, "foo": 3 }, 
  "text_field": "another test value"
}

// Vanilla nested sparse vector query with inference still works
GET my-index-1/_search
{
  "query": {
    "nested": {
      "path": "inference_field.inference.chunks",
      "query": {
        "sparse_vector": {
          "field": "inference_field.inference.chunks.embeddings",
          "inference_id": "my-elser-endpoint",
          "query": "test"
        }
      }
    }
  }
}

// Vanilla nested sparse vector query with query vectors still works
GET my-index-1/_search
{
  "query": {
    "nested": {
      "path": "inference_field.inference.chunks",
      "query": {
        "sparse_vector": {
          "field": "inference_field.inference.chunks.embeddings",
          "query_vector": { "test": 2.9, "value": 2.7 }
        }
      }
    }
  }
}

// Sparse vector against a semantic text field using inference
GET my-index-1/_search
{
  "query": {
    "sparse_vector": {
      "field": "inference_field",
      "inference_id": "my-elser-endpoint",
      "query": "test"
    }
  }
}

// Sparse vector against a semantic text field using query vectors
GET my-index-1/_search
{
  "query": {
    "sparse_vector": {
      "field": "inference_field",
      "query_vector": {
        "test": 2.9,
        "value": 2.7
      }
    }
  }
}

// Combined search using inference
GET my-index-*/_search
{
  "query": {
    "sparse_vector": {
      "field": "inference_field",
      "inference_id": "my-elser-endpoint",
      "query": "test"
    }
  }
}

// Combined search using query vectors 
GET my-index-*/_search
{
  "query": {
    "sparse_vector": {
      "field": "inference_field",
      "query_vector": {
        "test": 2.9,
        "value": 2.7
      }
    }
  }
}

[elasticsearchmachine]

Hi @kderusso, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/search-eng (Team:SearchOrg)

[elasticsearchmachine]

Pinging @elastic/search-relevance (Team:Search - Relevance)

[elasticsearchmachine]

Hi @kderusso, I've created a changelog YAML for you.

[carlosdelest]

Looks great! Love how the rewriting mechanism makes this soooo easy!

++ to Ben's comment on removing the need for inference_id for semantic_text.

A refactoring question about introducing a base class for your consideration.

[carlosdelest]

This should be retrieved from the index metadata, as per Ben's comment this is not necessary

[kderusso]

For some use cases we will still need it, for example if the request is sending in query vectors. However, I will address this

[Mikep86]

Great work, I think the overall mechanism for implementing this looks good. I agree with Ben and Carlos that we need to work on making inference_id optional when querying a semantic_text field in this PR, as it is a big part of the user experience.

[Mikep86]

Why nest two sub-queries like this?

[kderusso]

Discussed this offline with the team, and we decided that the use case of not specifying inference ID should happen in this PR. So, I am moving it back to Draft status until it's ready for re-review.

[carlosdelest]

LGTM 👍

[carlosdelest]

Including the field name and the (wrong) field type might help users diagnose the problem

Suggested change

	throw new IllegalArgumentException("inference_id required to perform vector search on query string");
	throw new IllegalArgumentException("inference_id required to perform a sparse_vector query on sparse_vector field [" + fieldName + "]");

[carlosdelest]

Nice refactoring! 👏

[benwtrent]

I really like the refactoring. I think the change overall is good.

I would REALLY like a QueryInterceptorTestCase and then two sets of tests

SemanticMatchQueryRewriteInterceptorTests and SemanticQueryRewriteInterceptorTests

I know all the yaml tests SHOULD be covering every branch, but I think having purpose driven interceptor tests that assert rewritten query types and shapes would be really nice.

[benwtrent]

❤️

[kderusso]

I would REALLY like a QueryInterceptorTestCase and then two sets of tests

SemanticMatchQueryRewriteInterceptorTests and SemanticQueryRewriteInterceptorTests

@benwtrent As a compromise, I'm adding some Java tests here with the expectation that they'll be cleaned up and fleshed out more in a followup PR.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 118617

[kderusso]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation