[Request]: Entity-Centric Services

[roshan-elastic]

Description

Note : This is a placeholder for awareness for a 9th Jul deliverable we will be releasing to customers. Detail TBC but wanted to put this in on the radar

We are delivering our first step in a new Observability Experience which will focus on showing users entities (e.g. Hosts, Containers, Services, K8s deployments etc)...even if they only have logs for them.

Our first step will be an updated Services Inventory which will show not only services instrumented with APM but also services we found in logs (reliant on service.name being declared in the data).

What do we need?

We potentially need docs to go live after 9th July (serverless first) which explain the following concepts to users:

Requirements

Beta Give feedback

1. Tech preview documentation explaining the new 'Experience' to users
2. ==> within these docs : Explain What it is and how it works?
3. ==> within these docs : Explain how users can add services to their Inventory - covered via [Request] Document how to add service.name to logs #4037
4. Tech preview docs explaining Explain what the Elastic-Entity Model (EEM) is - covered via [Request]: Elastic Entity Model ("EEM") Technical Preview docs #4028

Background

What are we releasing?

We will be releasing a new experience to our users that will allow us to be far more 'entity-centric' with Observability . Eventually, we be able to have faster loading experiences, be able to show relationships between entities better and generally have a more opinionated Observability experience talking less and less about ECS field names and signals and focused on the entities first (entities first...signals second).

What are entities?

Entities include things like Hosts, Containers, Services, S3 buckets, Kubernetes Clusters - the 'things' that customers want to observe.

What is the benefit to the customer in this release?

One huge benefit to customers is that we will start to show these entities regardless of what signal we have available (e.g. detecting services from log files, i.e. you wouldn't need to add APM into your services to populate the services inventory)

What are we releasing first?

Our first step in this direction will be to promote the new experience to users by offering the ability to view services not just by instrumenting with APM or OTel but by declaring service.name in a logs-* index. This would populate an updated Service Inventory which will show services from APM, from custom logs or both:

Note : See link to designs for how users interact with this experience

Promoting the new experience

Entity-Centric Services Inventory

Note : We will progressively adding different parts of the UI to this experience. We're starting with Services but we'll move onto Hosts, then Containers and then a new Inventory.

How will users turn this on and off?
The UI will let any user toggle the experience on or off in their browser (it won't be a feature flag so anyone can try it out without affecting other users):

Turning it on

Turning it off

Under the hood though, it will require EEM (below) to be turned on. We'll likely enable this immediately upon 'trying out' the feature in the UI.

If they are admins, we'll just turn on EEM for the cluster and let them know:

Toast showing on the new experience when we enable EEM for them upon first 'try'

If they don't have the right permissions, we'll tell them they need to speak to someone with the right permissions to turn it on:

Experience if you try the new UI but you don't have permissions for us to turn on EEM for your cluster

Note : EEM will be able to be controlled via a feature flag in case someone wants to turn it off. However, as soon as we can, we want to turn this on by default.

What will users see when they open the experience within the services inventory?
If they have APM services, we'll show them as per usual. If they have declared service.name in any of their logs-* indices - we'll also show those services too. Where there are services which have both APM and custom logs, they'll show as the same service:

Services from multiple signal types in a single inventory

If they click on a logs-only service, they'll have a service view they'll see a minimal service view based on logs:

Service views (logs only)

The service inventory will need to explain to users how to add more services to their service inventory via an empty state or call to action for a populated service inventory:

Example empty state

How will users add new services to this inventory?
The three options for declaring services will be:

1. Add APM (e.g. Elastic APM Agent, Otel or Synthetics) (existing journey)
2. Declaring new logs with service.name populated (existing journey)
3. Updating their existing logs to have service.name (e.g. maybe via ingest pipelines or logstash) - we will need a documentation to explain how users can do this:

Mock - just illustrative

What is EEM?
We are using a new platform we call the Elastic-Entity Model (EEM). This effectively processes data using transforms into pre-aggregated indices which store entities and their metadata, relationships and metrics (see sample entity - it was called OAM, renamed to EEM)

The Observability solution will then effectively query these indices instead of constructing these entities on-the-fly by parsing thousands or maybe millions of documents.

What do we need?

We need a set of documentation we can point users to from the UI which will explain:

1. What is our new Observability Experience?
2. How can users populate the new service inventory with services?
3. What is the Elastic-Entity Model (EEM)?

1. What is our new Observability Experience?
We need documentation which will explain what this new experience is. We'll refer to this in various parts of the UI:

2. How can users populate the new service inventory with services?
The above docs will need a section which we can point users to in order to explain how services get populated. Effectively, this is either:

a. APM instrumented services
b. Services we found in documents matching a logs-* index pattern which contain a value for service.name

For (a), users can do this via the application onboarding journey

For (b), users can do this by either:

• Collecting new logs via our custom logs onboarding journey

• Declaring service.name in their logs using things like Logstash, processors in beats/agent, mappings or ingest pipelines.

Important! We'll need documentation we can link to how users can declare service.name in their existing logs! We assume you will need some expertise to help on this.

3. What is the Elastic-Entity Model (EEM)?
This is handled in a separate request:

• [Request]: Elastic Entity Model ("EEM") Technical Preview docs #4028

Resources

See epic (and parent issue):

• https://github.com/elastic/observability-dev/issues/3350

Designs - in progress

Which documentation set does this change impact?

Stateful and Serverless

Feature differences

Same across both (released in serverless first though)

What release is this request related to?

Stateful : 8.16
Serverless : As soon as main is deployed after 9th July

Collaboration model

The documentation team

Point of contact.

Main contact: @roshan-elastic @chrisdistasio

Stakeholders: @kkurstak @tommyers-elastic @kpatticha @akhileshpok

[roshan-elastic]

Just FYI @bmorelli25 - FYI for now
(cc @chrisdistasio @kpatticha and @kkurstak)

[chrisdistasio]

Can replace

What is the Elastic-Entity Model (EEM)? (might not want to do this)

with #4028

[roshan-elastic]

Declaring service.name in their logs using things like Logstash, processors in beats/agent, mappings or ingest pipelines.

Hey @dedemorton, I just wanted to call out that I assume we'll need some pretty heavy expertise from our side in how users can add service.name to existing logs using things like:

• Mapping
• Logstash
• Processors : filebeat | elastic agent

I think we can probably point to these kind of docs but it might be helpful to give a simple example just to help someone get started?

Our expectation is that only really technical users will probably be able to do this...we doubt a brand new user will be able to follow this and do it really easily.

Note : We really want to figure out a way services can be discovered/declared with a specific feature in Observability but at the moment, we don't have anything like this

I'm on PTO until Thu 4th Jul but I'll dedicate my time after this support any efforts on this!

[bmorelli25]

Thanks, @roshan-elastic. I'm going to break the service.name request into its own issue (#4037) and have @mdbirnstiehl (our award-winning log writer) work on that part.

[roshan-elastic]

Sounds like a great idea @bmorelli25 - I was hoping you would!

[roshan-elastic]

Hey @bmorelli25, I was wondering when you might have a draft of the docs we need for the new entity-centric Observability experience?

I think there are three sets of docs we need:

1. Introduction and explanation of our new experience for services (this issue)

We'll link to that from the UI in few places like below:

Popovers in the UI which explains to users what the new experience is

2. How to declare services : #4037

That will be linked to from the above doc and also from the UI as well.

3. Link to EEM: #4028

This will be referred to in the (1) doc but also the UI:

If it helps, I'm happy to jump on a call - especially around (1).

Note : We were hoping to go live tomorrow but we're running a few days behind so we have some slack.

[roshan-elastic]

As per my other comment @bmorelli25, we'll most likely point towards this document via a short-link from the UI:

https://ela.st/new-experience-services

However, we'd make sure this doesn't get in front of customers until the docs are ready (e.g. by putting it behind a feature flag which is off by default).

[dedemorton]

Met with @bmorelli25 and @mdbirnstiehl to discuss the location of this new content in the docs. Decisions:

• Topic about our new Observability experience will appear in the Observability Guide at the same level as the Obs overview for now. We'll revisit this decision later when we work on the AI for the serverless GA (or if reviewers have a strong preference for some other location).
• Topic about how to populate the new service inventory with services will appear under Logging for now since it currently only pertains to logs.
• Topic about EEM will appear nested below the topic about our new Observability experience.

We'll make sure that there are links between the topics so users can find this information quickly. We'll also make sure the current topic about the Service inventory (in the Kibana Guide) points to the topic about the new experience.

[dedemorton]

@roshan-elastic I went through all the resources today and have a few questions plus some ideas about what we should cover in the docs. I'm going to drop my questions and rough ideas here in case we aren't able to touch base tomorrow (scheduling is difficult with the timezone difference).

QUESTIONS:

• What are we calling this thing? Entity-centric Observability?
• Where can I see this feature in action? (Is there an oblt environment that has these changes?) I have the Figma link, but in my experience, it's better to work with the product.
• Should the topic focus on what we are delivering for the technical preview? Or should we tell users about our future vision?
• Which privileges are required to turn on the new experience? Which privileges are required to enable the EEM?

ROUGH OUTLINE:

• [Not sure if we want to cover this] Describe why we are introducing a new experience (limitations of signal-based observability)
• Describe the new experience and what it provides over the traditional, signals-based approach.
• Explain what capabilities the technical preview provides: updated Services inventory that shows services instrumented with APM plus services found in logs
• Describe how to turn the new experience on (and off)
• Describe how to add services to the inventory (include a link to details about adding the service name to logs)

WDYT? Am I on the right track here?

[roshan-elastic]

Hey @dedemorton - thanks for picking this up:

What are we calling this thing? Entity-centric Observability?

Good question. I was hoping to just call it our new experience but I can't see a way of getting around giving it a name.

Maybe we should put this under an umbrella of 'Entity-Centric Observability' and as we start adding to this experience we start rolling specific docs for each one?

@chrisdistasio - I know Abhishek was giving us a steer on just calling it our 'new experience' but I can't think of a way to talk about this in the docs without giving it some kind of a name. WDYT?

@dedemorton - I was thinking of maybe this kind of structure?

Entity-Centric Observability

1. What is it?
2. Services*
3. Further reading
   a. Elastic Entity Model
   b. Declaring Services

*the services section could focus on what we're delivering in this first release (e.g. service inventory) and cross-link through to the general APM documentation it needs to

Then, as we start rolling in more functionality to it it would grow?

Entity-Centric Observability

1. What is it?
2. Services (include Services things like service inventory etc)
3. Hosts*
4. Containers*
5. Inventory*
6. Further reading
   a. Elastic Entity Model
   b. Declaring Services

*new things we would incrementally add

Eventually, all of this entity-centric stuff will go to GA and will replace the existing functionality (e.g. services, hosts etc) so we'd prob just merge all this new stuff into the normal docs (i.e. nothing would need to be nested under 'Entity-Centric Observability'.

Just my thoughts....I'm not 100% sure.

Where can I see this feature in action? (Is there an oblt environment that has these changes?) I have the Figma link, but in my experience, it's better to work with the product.

Just noting that we've slacked about this and you should get the help you need to play around with this.

Should the topic focus on what we are delivering for the technical preview? Or should we tell users about our future vision?

I'm thinking maybe if we had an entity-centric observability section then we could start by introducing it a little bit (and refer to the EEM docs being made)?

Which privileges are required to turn on the new experience? Which privileges are required to enable the EEM?

@chrisdistasio / @tommyers-elastic - I think you know this?

ROUGH OUTLINE:

I think you're on the right lines here. I think this broadly aligns with what I was thinking at the top of my comment?

@chrisdistasio / @tommyers-elastic - Just as an aside, I wonder whether it's worth co-authoring a blog post which talks about Entity-Centric Observability? Could be something a bit less formal which could be useful?

[chrisdistasio]

• Topic about EEM will appear nested below the topic about our new Observability experience.

In the very near future, "EEM" will power more than this experience (and likely even more than just Observability). I would recommend up-leveling EEM a notch in the IA.

[chrisdistasio]

Eventually, all of this entity-centric stuff will go to GA and will replace the existing functionality (e.g. services, hosts etc) so we'd prob just merge all this new stuff into the normal docs (i.e. nothing would need to be nested under 'Entity-Centric Observability'.

I agree with this, and it gives me pause to introduce it as "Entity-centric Observability."

As this is currently an extension of the Services (Inventory), I would propose it be linked to from the existing Services Inventory page and not even giving it a new place in the IA (until there is a real notion of a new comprehensive inventory experience)

[roshan-elastic]

Thanks for the quick feedback @chrisdistasio

So if we ditch it from the IA I'm thinking we'd have two or three docs basically?

1. New experience doc focused on 'services' for now
2. How to declare services
3. Elastic Entity Model doc

Then perhaps:

(1) would be nested somewhere within existing Services doc (perhaps we should rename 'APM' to 'Applications' in the docs too - we'll be updating the UI nav to show 'applications' instead of 'APM').

(2) might be a live somewhere else but be referred to from (1)

(3) would probably live somewhere else but again, referred to in (1)

co-authoring a blog post which talks about Entity-Centric Observability?

Is this probably too early do you think? Maybe we need to complete HCS first so people can really see the end-to-end value?

[chrisdistasio]

Is this probably too early do you think? Maybe we need to complete HCS first so people can really see the end-to-end value?

Agreed, a bit premature for now.

On #1-3 agreed; also need to consider placement for Serverless. Calling out because it's similar but slightly different.

[roshan-elastic]

@dedemorton - hopefully this last comment helps give a better steer?

[dedemorton]

Ok so based on your comments:

• I've created a new topic that focuses on the new experience for services. You can see the draft in Add docs about EEM and new Services inventory #4062 and provide feedback on that section now.
• Mike is working on the docs about declaring the service name in logs. Draft is up in Create docs for adding the service name field to logs #4054.
• I've created a placeholder file for EEM topic. Right now, I've added the topic under the Observability overview. Unfortunately there is no good place where we can define this content right now, but if there's a desire to de-emphasize it, I can move it to the bottom of the nav. Just let me know by adding a comment to the PR.

[roshan-elastic]

Nice one @dedemorton.

A question...every time I review docs I kind of get lost and end up looking at the raw files in github.

Is this the best way for me to be reviewing docs or should I be able to see a preview of the doc (i.e. what a user should see)?

I'm sure I just need a bit of education!

[dedemorton]

A question...every time I review docs I kind of get lost and end up looking at the raw files in github.

@roshan-elastic Typically I'll include a link to the preview docs, but I haven't done that in this PR (yet!) because I need to figure out why the preview link is 404ing. I'll drop the link in the PR when I get that figured out.

(Right now the doc previews are kind of confusing because we have two different doc builds taking place...one for asciidoc and one for markdown (mdx files). The markdown build needs to be triggered manually. A preview link gets added to the PR for asciidoc changes, but not mdx, which means I need to remember to run the build and update the link every time I push a commit.)

[roshan-elastic]

Got it @dedemorton - thanks!

[dedemorton]

Closed by #4062 and #4037