ES|QL Joins are here! Yes, Joi... Note

ES|QL Joins are here! Yes, Joins!

Elasticsearch 8.18 introduces a new SQL-style JOIN feature called LOOKUP JOIN, which enables data correlation and enrichment with easily updatable lookup datasets. This feature is available in tech preview and allows users to add host and asset information to events, check IP addresses against threat intelligence lists, and more. Lookup Join is a LEFT OUTER JOIN that relies on a new index mode called "lookup" for the right side, which can be updated directly. The lookup index can hold various types of data, such as assets, threat intel data, order info, employee or customer info, and more. Historically, Elasticsearch lacked join capability, but Lookup Join addresses this limitation. To enable Lookup Joins, a new index mode called "lookup" was created, which is limited to 2 billion documents and is directly updateable. There are no restrictions on the source data, and no data preparation is required to perform a join. Lookup Join is easier to set up and manage than the ENRICH command in ES|QL, with benefits such as no enrich policies to create, no policy execution, and better handling of multiple matches. Users can create lookup indices in various ways, including through Index Management or the ML File uploader in Kibana. The possibilities for using Lookup Join are endless, and future developments may include other types of joins, such as INNER joins or subqueries, and joining against any index.