Search to exclude words like "with" and "and"

[ntmhung]

We are using version 2.4.0 of Magento and ElasticSuite for searches. We would like our search to exclude stopwords like "with" and "and". Is there any configuration or adjustment in the code we can do?

Ex :
For example, the search "lidocaine epinephrine", "lidocaine with epinephrine", and "lidocaine and epinephrine" should all show the same exact search results.

Additional context
This is our website: https://shop.tccpharma.com/

[vahonc]

Hello @ntmhung,

When the user enters a query with several words, the short words like "and, or, to, for, are, be, by, in,… " are most of the time very frequently found in the catalog, so the short words might have a strong weight in the search result despite their lack of relevance. Most search engines (this includes Elasticsearch) offer to filter out those using a static list of "stop words". The search engine will ignore any word in this list.

In the ElasticSuite we use a different way: the search engine calculates the frequency of each word in the catalog, and ignores the words with a frequency above a given limit, called Cutoff Frequency. If all words in the request are above below the cutoff frequency, then the query is searched as a normal query. The words that appear in the index with a frequency above that cut-off frequency will be ignored. This feature allows a more accurate and automatic detection of meaningless words. And, it also allows to process queries with only stop words, like "to be or not to be" for example.

You can go further with the official documentation here : ElasticSearch Cutoff Frequency

The Cutoff Frequency can be set up in the back-office:

1. Open the menu Elastic Suite -> Search Relevance
2. Open the tab Relevance configuration -> Cuttoff frequency configuration
3. Enter a value between 0 and 1 in the setting Cutoff frequency. Any word included in more than this percentage of indexed entities will be ignored by the search engine.

The default value for the Cutoff Frequency is 0.15. It means that a word will not be significant for the search engine if this word is found in more than 15% of the indexed entities index documents. An indexed entity is a product or a category.
Edit: An index document can be a top-level indexed entity (category or product) OR one if its nested sub-document (category names and positions or prices for products).

Also, check our Wiki for more information about Fulltext base settings and Cutoff frequency configuration.

So, you need to do some tests with the cutoff frequency value and check if it's enough for your purposes or if this value should be higher or lower than the default value.

Also, pay attention to the Minimum Should Match value in the Fulltext base settings it should be 100% (or at least 50%) Edit: when you try to find the correct value.

BR,
Vadym

[ntmhung]

Thanks @vahonc

Let me try editing those settings

[rbayet]

Hello @ntmhung,

I updated a bit @vahonc's comment.

Basically the cut-off frequency is expressed as a percentage of indexed documents. Not indexed top entities.

In the catalog product index, there are documents that contain relevant text data and those that do not.
For instance :

• for a product, the "top" document with the searchable attributes : relevant
• for a product, every "categories" (nested) sub-document with a category name (and/or position) : somewhat relevant
• for a product, every "price" (nested) sub-document : absolutely no text data

So, in a default Magento2 installation, you have 4 customer groups, hence 4 prices per product top document.
Add the categories (2 to 4), that makes, let's say 10 documents (top or sub) per indexed product on average for a small sites.

If you have a lot of products, you probably have also a lot of categories, so you increase the number of "categories".
Most of them should have "Use category name in the product search", so those count toward "good" sub-document and the cut-off frequency probably does not need tweaking.

Now, if you have a lot of customer prices or a multi-stock feature where you end up with, say, 300 documents per indexed product on average with a majority of those, say 250 not containing text, then the cut-off frequency must be made 25 times lower than 0.15, so around 0.006

PS: You can find the total number of documents for a given index through the Elasticsuite > System > Indices.
Below screenshots taken on a Luma with the sample catalog.

[ntmhung]

Hey @rbayet

Is there a formula to calculate the cut-off frequency based on the number of indexed documents? I don't really understand how "the cut-off frequency must be made 25 times lower than 0.15, so around 0.006" part is calculated.

[rbayet]

Hello @ntmhung,

I said 25 times, because in my example, we go from 10 documents per product (higher bracker on a Luma sample catalog) to 250 - 300 documents per product with most of those not containing actual text data (being only physical shops related stock and price data).
So you could say that the number of documents containing actual text data has been "diluted" 25 times.

A more accurate formula would be "Divide by (number of index documents - number of non-text holding subdocuments) / number of index documents".

Another example: Imagine you have 10K products indexed and for each product, price and stock availability for 100 physical stores, and an approximate total of 2 050 000 indexed documents (10K x (1 top document + ~4 categories records + 100 stock records+ 100 price records) = 10k x 205).
Your approximate number of documents with actual text data is 2 050 000 - (10K x (100 + 100)) ~= 50K.

So the ratio of documents with actual text data is "50 K / 2 050K = 1/41".
In that case a good starting point for a lower cut-off frequency would be 0.15 / 40 (or 41), hence 0.00375 (or 0.0365).

I hope it's a bit clearer.

Regards,

[ntmhung]

Thanks @rbayet ,

To confirm if I understand correctly.

My store has 6600 products, 7 customer groups, and 1 physical store. So the total indexed documents should be 6600 x (1 top document + ~4 categories records + 7 stock records+ 7 price records) = 125 400.
My approximate number of documents with actual text data is 125 400 - (6600 x (7 + 7)) ~= 33K.

So the ratio of documents with actual text data is "33 K / 125 400 ~= 1/4".
In this case, the cut-off frequency would be 0.15 / 4 = 0.0375.

Is this correct?

[ntmhung]

Hi @rbayet

Looks like the cutoff frequency doesn’t work with my case.

I asked google bard for a solution and I am wondering if this could work? https://g.co/bard/share/aba745335e37 (see the code toward the bottom).

[vahonc]

Hello @ntmhung,

I asked google bard for a solution and I am wondering if this could work? https://g.co/bard/share/aba745335e37 (see the code toward the bottom).

Have you seen Elasticsuite admin panel or Settings > Search > Queries section in your installation?

Nope, it will not work as described by Bard, I'd say it's a stupid suggestion.

BR,
Vadym

[ntmhung]

Hi @vahonc

Is there any way I can add that query programmatically? E.g, create a plugin to Elasticsuite class? I don't know the correct class to add that query to

[rbayet]

Thanks @rbayet ,

To confirm if I understand correctly.

My store has 6600 products, 7 customer groups, and 1 physical store. So the total indexed documents should be 6600 x (1 top document + ~4 categories records + 7 stock records+ 7 price records) = 125 400. My approximate number of documents with actual text data is 125 400 - (6600 x (7 + 7)) ~= 33K.

Hello @ntmhung,

If you're not using something like Retailer Suite with the concept of shops inside a store

• you should not really have to tweak the cut-off frequency that much
• you will only have one "stock" record inside the top product document and it's not a nested document, just an object

So in your case, the approximate number of documents should be 6600 x (1 top document + ~4 categories records + 7 price records) = 79200.
Your approximate number of documents with actual text data should be 79200 - (6600 x 7) = 33K.
But please you directly the number of total documents that you can read in Elasticsuite > System > Indices and then substract the (6600 x 7).

So the ratio of documents with actual text data is "33 K / 125 400 ~= 1/4". In this case, the cut-off frequency would be 0.15 / 4 = 0.0375.

Is this correct?
As seen above, your total number of documents per index should pretty much be closer to 80K.
Look at what's visible for the catalog_product index in Elasticsuite > System > Indices.
For 79200 total documents, the ratio would be 33K / 79 200 ~ 0.41 so that would make a cut-off frequency of 0.0625.

But if 0.0375 is not already working then the valid cut-off frequency must be lower still.

As a side note, have you changed the "minimum should match" in Elasticsuite > Search Relevance > Fulltext base settings" ?

Regards,

[ntmhung]

Hi @rbayet

I tried to change the "minimum should match" setting to 5%, but the results were not as expected. The expected result is, for example, when I search "lidocaine and epinephrine" on the site https://shop.tccpharma.com/, it is only showing results with that exact match when we would like it to show results for "lidocaine and epinephrine", "lidocaine with epinephrine", and "lidocaine epinephrine".

Regards,

[vahonc]

@ntmhung,

The setting Minimum Should Match allows you to choose the minimum percentage of words that should match in the user query in order to suggest a result. Stopwords are not taken into account when calculating this percentage.

When a user enters a query with several words, several approaches could be used:

• Display the products that include ALL the words of the query

or

• Display the products that include AT LEAST one word of the query

The first approach is better for relevancy but may return no result for queries with a lot of words.
The second approach will return more results, but the results will be less relevant.

We recommend using a value of 100% (all words should match). Optionally, a slightly lower value may improve the results for long queries.

EXAMPLE:
If the Minimum Should Match is set to 60%, then:

• For a query with 3 words, a product will be suggested if at least 2 words are found in the datas of the product.
• For a query with 5 words, a product will be suggested if at least 3 words are found in the datas of the product.

BR,
Vadym

[ntmhung]

Hi @vahonc

I tried to set that setting to 30%, and the number of suggested products was increased. However, only products that match the search text were shown on the search result page. Using the same approach, can we increase the number of products on the search result page?

Regards,

[rbayet]

Hello @ntmhung,

Could you clarify your question ?

Using the same approach, can we increase the number of products on the search result page?

Just to be clear: the Minimum Should Match operates both on autocomplete and fulltext search.
Setting it at 30% means basically that if the user searches for 3 words, for instance "red race car", the will find products (or categories in the autocomplete) matching either 1 or 2 of 3 of those words.

So yes, lowering the Minimum Should Match usually augments the number of products found, but it might not be exactly those that the user is searching for (since a product titled "red plane" will match, as well as a "yellow race car").
The "Phrase Match" component will boost products that match couple of words close by, so in theory, if you have a "red race car" product, it should be in the first results.

PS: moving this issue as a discussion as there is not technical problem reported.