Does Spryker plan to improve pagination support for categories with 10k+ products?

Roland
Roland Senior Application Engineer Posts: 3 🧑🏻‍🚀 - Cadet
in Spryker Development

Currently if there is more than 10k products in category and you try to switch to category page which shows product list ordered >10k you will see `fail whale` and get such error log:
```
Yves.CRITICAL: Spryker\Client\SearchElasticsearch\Exception\SearchResponseException - Search failed with the following reason: Result window is too large, from + size must be less than or equal to: [10000] but was [10008]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting. [reason: all shards failed].
```

From my research increasing the `index.max_result_window` is not recommended approach due to potential large CPU and memory resource needs.
Scroll API docs say:

```
The scroll API is no longer recommend for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).
```

It seems that `search_after` with PIT is currently the most recommended approach which seems not supported by SearchElasticsearch client yet (or am I wrong?). Hence my question.
Do Spryker plan to improve pagination approach for such case in near future?

Cheers.

Answers

  • profuel
    profuel SSA Sprykee Posts: 106 🪐 - Explorer

    Although it's not convenient to see an error, may I ask why is it so important to use so deep pagination?

    It's not the first time we hear about the 10000 limit in the ES query, though every time the resolution was tied to the specific business case.

  • Roland
    Roland Senior Application Engineer Posts: 3 🧑🏻‍🚀 - Cadet
    https://forum.commercequest.space/discussion/comment/29519#Comment_29519

    Thank you for your reply,

    One of our customers has more than 10k products in a single category branch. On PCP, every time someone enters on the pages which are supposed to present products in the category above 10k - application returns 500 (fail whale). We get notifications from our QA department on this bug discovery.

    Altho solution can be provided on the project level, workload required to solve this issue seems significant. As this issue applies potentially to all Spryker projects which includes more than 10k products in category, the best solution would be to implement new pagination approach on the Spryker level IMHO. Hence my question here.

Categories