[RFC 0050] [Update] Stage 2: Introducing GenAI fields

[susan-shu-c]

Overview

This PR proposes changes on top of

• [RFC 0050] Stage 2: Introducing GenAI fields #2475

based on discussion with @felixbarny @eyalkoren in this PR:

• Update ecs@mappings.json with new GenAI fields elasticsearch#129122

namely, updating gen_ai.request.model and gen_ai.agent.description from text to keyword.

The rationale:

• gen_ai.request.model: doesn't require text analyzing
• gen_ai.agent.description: phrase queries may not be used often for this field, and may be expensive from storage and indexing - via @felixbarny

While it has a limit on how many characters it can store in doc_values it falls back to storing the value in a stored field if ignore_above is set, which seems like a good compromise here.

I'm looking for feedback if this is covered, as it's part of the generated files and I'm not completely sure which changes are expected for ignore_above to work.

Checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?
• For proposing substantial changes or additions to the schema, have you reviewed the RFC process?
• If submitting code/script changes, have you verified all tests pass locally using make test?
• If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes?
• Is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
• Have you added an entry to the CHANGELOG.next.md?

[github-actions]

Documentation changes preview: https://docs-v3-preview.elastic.dev/elastic/ecs/pull/2489/reference/

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[eyalkoren]

LGTM

While it has a limit on how many characters it can store in doc_values it falls back to storing the value in a stored field if ignore_above is set, which seems like a good compromise here.

I'm looking for feedback if this is covered , as it's part of the generated files and I'm not completely sure which changes are expected for ignore_above to work.

I am not sure what you mean. Most (or all) keyword fields in ECS are defined with ignore_above: 1024.
In addition, this is the setting we use in ecs@mappings by default for all string fields that are not mapped otherwise.

[felixbarny]

Optional suggestion to disable indexing for the description field, otherwise LGTM.

[susan-shu-c]

Thanks @eyalkoren this is clear enough to me! I just wanted to make sure I wasn't missing something

Most (or all) keyword fields in ECS are defined with ignore_above: 1024.

[susan-shu-c]

cc @trisch-me @mjwolf regarding updating gen_ai.request.model and gen_ai.agent.description from text to keyword, as the former doesn't require text analyzing and and latter is not often used for direct phrase queries; gen_ai.agent.name can be used instead.

This came up as part of:
Update ecs@mappings.json

[susan-shu-c]

Thank you, @trisch-me !

[felixbarny]

It's curious that doc_values is also false now but just setting index: false in the schema.

Is it possible to explicitly set doc_values: true in here to retain it in the mappings (and to keep ignore_above)?

ecs/schemas/gen_ai.yml

Lines 29 to 37 in a1237a3

	- name: agent.description
	type: keyword
	index: false
	description: Free-form description of the GenAI agent provided by the application.
	example: Helps with math problems; Generates fiction stories
	level: extended
	beta: This field is beta and subject to change.
	otel:
	- relation: match

[susan-shu-c]

Thanks for pointing this out! - To summarize what we discussed on the Elasticsearch issue we'll merge first and then continue to adjust in the upcoming weeks.

[mjwolf]

@felixbarny It's possible to explicitly set doc_values: true, and it will be applied in the generated files.

I've created an ECS issue to update the generator: #2492.

Should ignore_above: 1024 be set as the default value for all keywords? I can add that to the issue to update the generator too.