8.4.2 (in Docker)
Whatever comes in the Elasticsearch Docker image
java -version doesn't work inside the Docker image)
Whatever's in the Elasticsearch Docker image:
$ uname -a Linux c8dae8d214e8 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
We're seeing the following error from queries in Elasticsearch 8.4.2:
totalTermFreq must be at least docFreq, totalTermFreq: 413958, docFreq: 413959
These are queries that worked successfully in Elasticsearch 8.4.1.
The errors are reproducible, but don't affect all queries.
The numbers in the error message seem to be stable.
Start an instance of the Elasticsearch Docker container:
docker run \ --env xpack.security.enabled=false \ --env discovery.type=single-node \ --publish 9200:9200 \ -it docker.elastic.co/elasticsearch/elasticsearch:8.4.2
(These settings may not all be required, but they're how I usually the local Docker image and they avoid having to do the password/CA certs dance. I’m running Docker on macOS, although I don’t think it makes a difference.)
Download the following data set from S3: https://wellcomecollection-data-public-delta.s3.eu-west-1.amazonaws.com/elasticsearch-issue-files/works.json.gz (40MB)
This contains ~900k documents with a minimal set of fields to cause this error – if I reduce the number of documents or fields, the error goes away.
(Corpus is CC-BY 4.0 licensed Wellcome Collection, similar to https://developers.wellcomecollection.org/docs/datasets)
Run the attached Python script, which will:
works.json.gzinto the index
The search request returns a list of results.
We get an error from the Elasticsearch Python library:
elasticsearch.BadRequestError: BadRequestError(400, 'search_phase_execution_exception', 'totalTermFreq must be at least docFreq, totalTermFreq: 287039, docFreq: 287040')
If I run my Python script against the 8.4.1 Docker image, the error doesn't reproduce.
I found an issue with a similar error message from 7.0.1: https://github.com/elastic/elasticsearch/issues/41934
Notably, the fix for that issue (https://github.com/elastic/elasticsearch/pull/41938) mentions
cross_fields, which we're using in our query.
If I remove any of the fields from the query, the error goes away.
I’m using the Elasticsearch Docker image for the sake of an easy reproduction case, but we're seeing this issue in our managed Elastic Cloud clusters also. We actually see the error in two different clusters:
Our actual documents are quite a bit larger, and the query more complicated. We can share more details if it would be useful, but I figured you'd prefer the minimal version.
The numbers in the error message seem to vary depending on which Elastic node handles the request, but each node returns a consistent set of numbers. e.g. if I run this query in our Elastic Cloud cluster, it gets handled by one of two nodes:
error.failed_shards.node = ZTz1Ump1THGC5lMd9VL3XQgets the error "totalTermFreq must be at least docFreq, totalTermFreq: 413958, docFreq: 413959"
error.failed_shards.node = qhmqpnpCQhe9KKSKYWQfNwgets the error "totalTermFreq must be at least docFreq, totalTermFreq: 437539, docFreq: 437540"
If I run the query repeatedly, each node returns consistent numbers.
Elasticsearch has a few features designed for scalability, but you can free up resources on your Elasticsearch servers by offloading the load balancing of requests to NGINX Open Source or NGINX Plus, which has even more enterprise‑grade features.
|Last Update||11 hours ago|