Skip to content

PoC: Connect GLQL WorkItems to ES

Alisa Frunzarequested to merge
524326-poc-connect-glql-to-es into master

What does this MR do and why?

This MR introduces a proof-of-concept (PoC) integration for connecting GLQL to Work Items API using Elasticsearch. Its scope is limited to demonstrating the ability to search Work Items via ES and is currently experimental.

Scope & Limitations

  • Supported ES Fields:
    The PoC supports a limited set of ES fields. Specifically, it uses:
    • :label_name
    • :group_id
    • :project_id
    • :state
    • :confidential

See the comparison of GLQL and ES fields in this Issue #525570. The next step would be to add the missing fields to ES and expand the searching capabilities of current MR's implementation.

  • Search Scope:

    • Group vs. Project Level:
      If a group is not specified in the GLQL query, the search will be executed at the project level.

  • Sorting & Pagination:

    • Items are sorted in descending order.
    • The search is limited to returning 100 items.

  • Operators & Wildcards:

    • Only AND operators are supported for now.
    • Only = or in (the later only for labels) operators are supported.
    • Label wildcard search in labels does not work at the moment.

Implementation Details

  • A glql_es_integration feature flag has been added, which is applied per user.
  • The finder checks that the request is coming from the GLQL controller.
  • We also allow a URL parameter (useES=true/false) to explicitly switch between the new ES flow and the legacy PostgreSQL flow.
  • If the useES parameter is not provided (and it’s a GLQL request with the feature flag enabled), the new ES flow is used by default.
  • If any unsupported filter is used, even if all flags are enabled, the search will fallback to using PostgreSQL.
  • The new logic is moved to EE since Elasticsearch is a licensed feature.

Known Limitations

  • If a label specified in the query does not exist, the ES search service ignores it—resulting in a search as if no label was applied.
  • This behavior is expected to improve when we migrate from using label IDs to label names, as discussed in Issue #525883.
  • This migration will also enable label wildcard searches.

References

Screenshots or screen recordings

Nothing should change in the UI.

How to set up and validate locally

  1. Follow this guide to have ES enabled locally for GDK
  2. Enable the following two feature flags in rails c:
=> Feature.enable(:glql_integration)
=> Feature.enable(:glql_work_items)
  1. Navigate to any issue comment or description field where Markdown is supported
  2. Add the following GLQL query into the field (make sure to update labels based on what you have locally):
```glql
display: table
fields: title, labels("workflow::*"), author, labels
query: group = "gitlab-org" and label = "Brantforge" and state = opened and confidential = false
```
  1. Verify that when using only the supported filters (i.e. group, label, state, confidential), the search returns the expected work items
  2. Try adding an unsupported query filter (for example, assignee = @ username) and check that the system falls back to the old search
  3. Also, remove the group filter entirely and in that case, the search should run on the current project instead of a group level search.
  4. If you want to test between old and new behavious, you can add the useES=true/false param to url

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #524326 (closed)

Edited by Alisa Frunza

Merge request reports