Create new IA for observability docs

[dedemorton]

Closes https://github.com/elastic/obs-docs-projects/issues/247

Preview link: https://observability-docs_bk_4134.docs-preview.app.elstc.co/guide/en/observability/master/index.html

Top-level TOC changes

Questions to consider during review

• Is the latest nav design (without verbs or gerunds) effective or too slimmed down? Note that the page titles provide a lot more context for search engines to scoop up.
• Is the topic nesting excessive or does it make the navigation easier to scan and use?
• Should universal profiling live under infrastructure instead of applications?
• Should Kubernetes live under Infrastructure monitoring or as it's own top-level container, maybe at the same level as a container called "Hosts"?
• Are we OK with proceeding with an Information Architecture that exposes our gaps and promotes topics that we know are dated (cloud monitoring, for example)?
• Are we OK with having a set of top-level topics that are not parallel in nature? For example, some top-level topics (like the topic about monitoring applications and services) is simply a jump table that points to other content. Other topics are meaty with all the details in a single topic. Given the limited content we currently have for some topics, this problem is unavoidable.

TODO

• Add the "quickstarts" under "start here" (after Add hosts and K8s quickstarts to stateful docs #4168 is merged). We may want to flatten out the nav since "quickstart" will be redundant with "start here"). I like this idea, but we need to find out if the info that introduces the quickstarts is necessary. We could add this info in the box at the top or something, but it might be a bit repetitive and distracting.
• Add content to placeholder topics:
  • start-here.asciidoc
  • application-and-service-monitoring.asciidoc
  • cloud-monitoring.asciidoc
  • incident-management.asciidoc
• Check for links that point to topics where we've added new abbreviated titles and make sure the link text looks OK.

TODO after merging:

• Set up redirects for any renamed files or anchor IDs:
  • Redirect analyze-metrics.html to monitor-infrastructure-and-hosts
  • Redirect quickstarts-overview.html to observability-get-started.html
  • Look at redirects.asciidoc for other possible redirects to set up.
• Resolve TODOs (inline in source files) related to this effort -- these are mostly related to adding content or deleting old files.
• Make troubleshooting titles consistent across obs docs (to gerund or not to gerund, that is the question)
• Apply similar changes to the serverless docs.

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[mergify]

This pull request does not have a backport label. Could you fix it @dedemorton? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-/d./d is the label to automatically backport to the /d./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[colleenmcginnis]

Lookin' good ✨

[alaudazzi]

@dedemorton thank you for leading this doc refactoring!
A few comments/suggestions on the overall structure:

• The Start here page is a nice entry point. How about making it lighter by removing the various “Learn about, learn how, learn more” with something like this:
  New to Elastic Observability? Discover more about our Observability features and how to get started.

• The vicinity of Quick starts and Get started is confusing. How about promoting the Quick starts one level up and make them more visible in the TOC?

• In Applications and services, it is weird to have “monitoring” for most of the TOC entries, except for APM and Synthetics.

• I love the Cloud section!

• In Monitor AWS with Elastic Agent and Monitor AWS with Amazon Data Firehose, we can alleviate the TOC by removing Monitor from headers?
  

• Reference seems vague and is not adding much value to the navigation. How about moving Fields reference and Elastic Entity Model on level up and removing Reference?

Now to your questions:

1. Is the latest nav design (without verbs or gerunds) effective or too slimmed down? Note that the page titles provide a lot more context for search engines to scoop up.
   Looks OK. Otherwise, a possible alternative could be:
   Start here
   Applications and services
   Cloud
   Infrastructure and hosts
   Logs
   CI/CD
   Kubernetes
   Observability AI Assistant
   Incident management
   Fields reference
   Elastic Entity Model

2. Is the topic nesting excessive or does it make the navigation easier to scan and use?
   It goes down to level 4, but it seems to be unavoidable considering how the TOC is recompacted

3. Should universal profiling live under infrastructure instead of applications?
   I don’t know

4. Should Kubernetes live under Infrastructure monitoring or as it's own top-level container, maybe at the same level as a container called "Hosts"?
   I would leave it there

5. Should the getting started guides live under "start here" or within relevant sections?
   Having them consolidated in one single place makes them more usable.

6. Are we OK with proceeding with an Information Architecture that exposes our gaps and promotes topics that we know are dated (cloud monitoring, for example)?
   Upper management recently claimed that AWS and Azure were not discoverable enough. This reorg exposes them as they should.

7. Are we OK with having a set of top-level topics that are not parallel in nature? For example, some top-level topics (like the topic about monitoring applications and services) is simply a jump table that points to other content. Other topics are meaty with all the details in a single topic. Given the limited content we currently have for some topics, this problem is unavoidable.
   If it’s unavoidable, then it’s OK :-)

[alaudazzi]

Great rework! I've left a few comments/suggestions.

[dedemorton]

@alaudazzi Thanks for your input! Just to clarify, I didn't go very far into the guts of each area because some of them (like APM) are going to need some extra special love. But I can certainly make the changes you want to the firehose section because it's pretty straightforward. :-) Thanks again.

[dedemorton]

We made the following decisions during the team meeting today:

For the getting started content:

• ✅ Remove topics nested under Get started.
• ✅ Consider reframing "Logs and metrics" guide as a tutorial that uses Fleet, and move it to the Infrastructure and hosts section (or remove the topic entirely). I moved this to the beginning of infra and hosts because there's currently no getting started info in that section.
• ✅ Move Nginx to the "Infrastructure and hosts" section and rename it to begin with "Tutorial:"
• ✅ Delete the "Traces and APM" topic and redirect users to the section under APM (@colleenmcginnis which topic do we want to redirect users to? Looking at the source files, the only getting started guide I see is under "Get started")
• Figure out what's going on with the platform docs about getting started: who owns, do we still need them, do we still want to link to them, etc.

For other tutorial-style content:

• ✅ Move topic about Kubernetes under "Infra and hosts" and add "Tutorial:" to the title.
• ✅ Move topic about Java applications under "Applications and services" and add "Tutorial:" to the title.

For troubleshooting topics, we will leave them as-is until a larger docs team plan emerges.

[colleenmcginnis]

Delete the "Traces and APM" topic and redirect users to the section under APM (@colleenmcginnis which topic do we want to redirect users to? Looking at the source files, the only getting started guide I see is under "Get started")

The section that duplicates that content is Fleet-managed APM Server. As a part of https://github.com/elastic/obs-docs-projects/issues/260, I'm thinking we should change the section that contains that content from "Self manage APM Server" to "Get started" or "Set up".

[dedemorton]

@alaudazzi Thanks for your input! Just an FYI--here's how I have addressed your feedback:

The Start here page is a nice entry point. How about making it lighter by removing the various “Learn about, learn how, learn more” with something like this:
New to Elastic Observability? Discover more about our Observability features and how to get started.

Nice idea. Done!

The vicinity of Quick starts and Get started is confusing. How about promoting the Quick starts one level up and make them more visible in the TOC?

I'll look at this after the quickstarts are merged, but I think with the getting started section slimmed down, your suggestion makes sense.

In Applications and services, it is weird to have “monitoring” for most of the TOC entries, except for APM and Synthetics.

It is. Already fixed based on Colleen's feedback.

I love the Cloud section!

Yay

In Monitor AWS with Elastic Agent and Monitor AWS with Amazon Data Firehose, we can alleviate the TOC by removing Monitor from headers?

Done

Reference seems vague and is not adding much value to the navigation. How about moving Fields reference and Elastic Entity Model on level up and removing Reference?

Let's wait and get more input from reviewers on this. I think we might get some mixed opinions about this.

[dedemorton]

The section that duplicates that content is Fleet-managed APM Server. As a part of https://github.com/elastic/obs-docs-projects/issues/260, I'm thinking we should change the section that contains that content from "Self manage APM Server" to "Get started" or "Set up".

@colleenmcginnis So the page I need to link to does not have an ID. I'm going to add an ID that you can change later, if you want.

[colleenmcginnis]

@dedemorton I ran into similar errors recently...

INFO:build_docs:Bad cross-document links:
--
  | INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/guide/master/_fleet_managed_apm_server.html contains broken links to:
  | INFO:build_docs:   - en/observability/master/_fleet_managed_apm_server.html
  | INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/guide/master/_step_2_add_and_configure_the_apm_integration.html contains broken links to:
  | INFO:build_docs:   - en/observability/master/_step_2_add_and_configure_the_apm_integration.html
  | INFO:build_docs:  /tmp/docsbuild/target_repo/html/en/apm/guide/master/apm-quick-start.html contains broken links to:
  | INFO:build_docs:   - en/observability/master/traces-get-started.html

... to fix them, you'll need to update the redirects in docs/en/apm-server/redirects.asciidoc.

[dedemorton]

@colleenmcginnis @alaudazzi @bmorelli25 @mdbirnstiehl I'm curious what you think of merging the "get started" and quickstarts into one section:

That way there is one page that has all the links to quickstarts and getting started.

[alaudazzi]

I'm curious what you think of merging the "get started" and quickstarts into one section:

It makes sense, but I would remove "Quickstart" and reword the headings. It sounds weird to have a quickstart into a get started. How about this:

• Get started with Elastic Agent
  • Monitor your hosts
  • Monitor your Kubernetes cluster
• Get started with data from Splunk

[dedemorton]

It sounds weird to have a quickstart into a get started.

Why? In my mind, a quickstart is just a type of getting started document. I also think the quickstart label makes it clear that we're offering an accelerated path. I'm curious if other writers share your concern. I want to indicate in the nav somehow that these topics are special.

[mergify]

This pull request is now in conflict. Could you fix it @dedemorton? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b issue#247 upstream/issue#247
git merge upstream/main
git push upstream issue#247

[colleenmcginnis]

@dedemorton I'm curious what the distinction is between "get started" and "quickstart". What about the "Get started with data from Splunk" makes it not a quickstart? Could it be added to the "Get started" section as another quickstart?

[alaudazzi]

@colleenmcginnis @dedemorton
The Elastic documentation guidelines cover these content types, where get started and quickstart are not directly referenced but might correspond to how-to and tutorial.

Based on previous conversations we had at Elastic about content types and IA, I recall that it has always been hard to exactly define what is what. My understanding is that:

A Get started is to learn how to cook. More general. For novice users.
A Quickstart is to learn how to cook pasta, fish, meat, and so on. More specific. For expert users.

They are two different content types, they address two different audiences, so it would be more natural to have them at the same level.

[colleenmcginnis]

A Get started is to learn how to cook. More general. For novice users.
A Quickstart is to learn how to cook pasta, fish, meat, and so on. More specific. For expert users.

Interesting. Do you think Get started with data from Splunk is more general than Quickstart: Monitor your Kubernetes cluster with Elastic Agent?

[dedemorton]

I'm curious what the distinction is between "get started" and "quickstart".

@colleenmcginnis Good question! Originally I had all the quickstarts under a "Quickstarts" section that explained what we mean by quickstarts (the term "quickstarts" does appear in the UI and I think the original issue requested that we describe what a quickstart is). Originally the titles of these quickstarts exactly matched the UI, but I was asked to make the title more goal oriented. The section that describes quickstarts is now here, but maybe it's too subtle and buried in the "Get started" topic? I mainly used "quickstart" in the topic titles to map this content to what users see in the UI and also let users know that the topic contains info to get started quickly.

Edited: I forgot to mention splunk in my response. It's really an outlier because I can't more it into a different like I did with the other GS topics, but it really doesn't stand very well alone. Maybe I shouldn't even include "get started" in the title. WDYT?

[dedemorton]

Met with other obs writers today. Decision was to:

• Change "Start here" to "Get started" and use the current content from the getting started topic as the main page.
• Flatten out the quickstart structure and possibly remove "Quick start" from the title. (I'll see how it looks)

[akhileshpok]

@dedemorton - I think that universal profiling should be under infrastructure. IMO, we should have either'Get started' or 'Quick start' in the nav. Please let me know if you would like to me to review any other items. Thanks

[dedemorton]

I've moved Universal Profiling to appear under the infra/hosts monitoring section. It didn't seem right to nest Universal Profiling under a topic that describes the Infrastructure UI, so I added a new container topic with a jump table to the subsections. That's the best I can do for the content as it is today.

[dedemorton]

@akhileshpok Mainly I need you to confirm that the navigation/TOC structure looks good. With the new structure, we'll eventually need to add more content and rewrite some of the existing topics, but that work is out of scope for this PR.

We'll backport this to 8.16, so the nav won't go live right away. That'll give other stakeholders an opportunity to suggest tweaks. Thanks!

Adding the preview link to make it easier for you to see the final TOC: https://observability-docs_bk_4134.docs-preview.app.elstc.co/guide/en/observability/master/index.html

[dedemorton]

I got the go-ahead from Akhilesh over slack. I just need to move the troubleshooting section, and I think we're ready to merge this into main.

Akhilesh Pokhariyal
Yesterday at 3:26 AM
@dede

• I like the new IA for Obs, as it is consistent with the entity centric vision. I would suggest moving Troubleshooting out of the 'Apps and services' section and make it a top-level option. We can always improve the quality, quantity and consistency of the content under various sections, but this is a good starting point in my opinion. Thanks (edited)

[mergify]

This pull request is now in conflict. Could you fix it @dedemorton? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b issue#247 upstream/issue#247
git merge upstream/main
git push upstream issue#247

[alaudazzi]

LGTM