[ML] inference hits error during reindex: Error in inference process: [inference canceled as process is stopping]

[wwang500]

Environment:

Serverless ElasticSearch project (Oct 29, 2024)

Step to reproduce:

1. Add an inference endpoint

PUT _inference/sparse_embedding/elser-endpoint
{
  "service": "elasticsearch", 
  "service_settings": {"model_id": ".elser_model_2_linux-x86_64", "num_threads": 4, "adaptive_allocations": {"enabled": true, "min_number_of_allocations":2}}
}

2. create an index with semantic_text field mapping

PUT semantic-embeddings-2
{
  "mappings": {
    "properties": {
      "message": { 
        "type": "semantic_text", 
        "inference_id": "elser-endpoint" 
      }
    }
  }
}

3. Restore a test dataset filebeat-length which has a message field, (message me if need the dataset)
4. Run reindex with size 50

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "filebeat-length",
    "query": {
      "exists": {
        "field": "message"
      }
    }, 
    "size": 50
  },
  "dest": {
    "index": "semantic-embeddings-2"
  }
}

Observed:

After ~20 minutes, reindex stopped, the task contains errors:

    "failures": [
      {
        "index": "semantic-embeddings-2",
        "id": "a76p05IBEL3KhHcxIADt",
        "cause": {
          "type": "exception",
          "reason": "Exception when running inference id [elser-endpoint] on field [message]",
          "caused_by": {
            "type": "status_exception",
            "reason": "Error in inference process: [inference canceled as process is stopping]"
          }
        },
        "status": 500
      },

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)