Change index on visualization/dashboard

[hvisage]

Update on April 4th 2018
We really appreciate all of your feedback on this mega issue so far, but we need your help as we break this down into more concrete items to work on. Even if you've already +1'd this specific issue, it would be extremely helpful if you could +1 and describe your use case in the more granular issues that are provided below. We won't ignore all the feedback folks have provided so far in this issue, but more focused feedback on the individual feature proposals would be invaluable.

Here are the three sub issues that I believe make up this mega issue:

Changing index patterns on a visualization: #17542
Dashboard level index patterns: #16917
Improved export/import to include referenced objects. Then you can do a search/replace on an index pattern id in a single file: #16831
Nested dashboards: #16919

Dashboard level index patterns seems to match this initial request best and there is a currently available workaround for it, which is mentioned in #16917.

If those three issues do not cover your particular case, please let us know!

---

Original Issue Request:

I'm in a situation where I have the exact same dashboard and it's visualizations, but want it to use different aliases/indexes.

The idea would be to segregate different user's data/views, but each have the same view/dashboard (just his/her index/alias differs), especially as I read Shield does authorization/access control via aliases/indexes and that is what I need/want to use for the user authorization/access control then.

[rashidkpc]

You can do this by copying the visualization documents in elasticsearch and changing the parameter for index. I don't see us supporting a UI on for this sort of functionality any time soon.

[hvisage]

:( It then becomes an quickly intractable O(M*N) issue (N=the number of indexes, and M the number of visualizations.

[chrisjschultz]

Is it possible with the current architecture? The index appears bound to the visualization, not the dashboard in K4. This would be hugely useful for me. Like most people (I imagine) I have each of my environments split into separate indexes; maintaining the dashboards becomes difficult...

In Kibana 3 I have hacked in a select list box which allows users to switch indexes for each dashboard, meaning only 1 dashboard needs to be maintained for all environments. Kibana 4 however binds the index to the visualisation not the dashboard. It might be helpful if the dashbaord could (optionally) override the index on its visualisations? I can see how it is useful to have a dashboard that is index agnostic in some use cases however.

[wittwerch]

+1

Copying the visualization documents in elasticsearch is not really an option. That generates a lot of duplicated information and updating would be a nightmare.

[AndreLouisCaron]

+1 on this. The "index pattern" concept naturally maps onto a "source" of data and the dashboard currently make it impossible to do this.

At the very least, the search bar at the top of the dashboard page should allow filtering by a "more specific" index pattern but it doesn't seem to accept searches with _index:­...-* patterns (nor specific values for that matter).

Our current workaround is to have a separate document type for each source because it's the only way to quickly filter by source while staying the same dashboard, but this creates Yet Another Configuration Problem (tm) because we need to create new type mappings for each environment...

[ghost]

We face a situation where we will have users from different departments doing different roles accessing the same data via their own dashboards.
e.g. developers vs front line support staff.

How can we restrict the dashboard modifications - so that the set of dashboards created by developers for eg. cannot be modifed by support staff?

If shield is tied solely to the index, it wouldnt work as the underlying data for both dashboards is the same.

Thank you.

[pdjtrifork]

+1 for this.
At this moment I am using Ansible and templates to at least automate a bit of the copy and substite process but this should not be THE way to do it. Redundancy is not cool from a maintenance perspective...

[dfaropennetwork]

+1

[gohoyer]

+1 for this.
It would be very usefull to have this feature when you have one index patter for every environment (dev, homolog, prod) but dashboards and visualizations are the same.

[lclemente]

+1
Having index tied to visualization, embedded in dashboards, is a complete nightmare. It makes impossible to use aliases for recent data and long term logstash data, without rewriting completely the dashboards.

Please have a look at how dashboard interface and design works in Grafana 2, that totally nailed it form my point of view. For example, graphs are not separate entity but created inside a dashboard.

[Pryz]

+1 Right now it is a complete nightmare when you have complex platforms.

[Paris999]

+1 Please add this support fast...

[acs]

+1

[JonahTurnquist]

+1 It's a huge nightmare for us to have to create and then upkeep 3 different dashboard across 3 different environment. The dashboards should be the same for us in each environment but without this feature we have to manually create visualizations and dashboards 3 times over.

[ghost]

+1

[gregoryb]

+1

[ppf2]

+1

[davalente]

+1
Duplicate visualizations is not the way.
Why did you change Kibana 3 way to handle dashboards!?!?

[mattheww-porch]

+1

[peteharverson]

+1

[aschokking]

+1

[DaneGardner]

+1

[byronvoorbach]

+1

[rvrignaud]

+1 this would be really important for us

[darkoc]

+1

[stacey-gammon]

I have a few questions to everyone that is eagerly looking forward to this feature being implemented, as I start to explore different approaches to solving these problems.

First let me try to understand the main problems that have been discussed here:

Problem 1: There is no way to easily exchange one index pattern for another in a given set of existing dashboards and visualizations.

Problem 2: You don't want to maintain a slew of duplicate visualizations and dashboards where the only thing that differs is the index pattern. This situation occurs because viewers of your dashboards have access to data that resides in different index patterns.

Does this sound accurate?

For now I'd like to focus only on Problem 2.

To give a hypothetical scenario, a company creates Kibana dashboards for their clients, client A and client B. client A's data is in client-a-* index pattern and client B's data is in client-b-* index pattern. The company now has two duplicate sets of visualizations and dashboards, where everything is equal but the index pattern used to create the visualizations.

Is that similar to the situation many of you are running into? If so, I'm curious whether anyone has tried to work around that by creating a single set of visualizations that point to index pattern client-*. The two clients data would naturally be filtered because of their permissions, so they would only see their own data.

If the issue is that the dashboard creator wants to be able to easily see the data that each client sees, could you add a client field to filter by?

As a side note, there is an _index meta field but you can't search using a wildcard on it, which means if your pattern is client-a-date you wouldn't be able to see across dates.

If you go this route, there is only one set of visualizations, and one dashboard to maintain. Data is filtered naturally for viewers based on their permissions, but admins who have access to all the index patterns can view everything, or segment the data. Clients will see the client field but will only see their own name as an option to select in it (so they wouldn't see a reference to a at all).

Understanding how that workaround falls short as a solution will help me figure out what concerns need to be addressed as I think about implementing this feature.

[ArtemUstynov]

well what you describe is not the problem for me. Say i index N elements to elastic and name it "approach 1" then i index data that had same Jsons fields, but that data was extracted in a different (hopefully better) and name it "approach 2" way so now i want set of mine visualizations to be applied to a different index. Think of dashboard as of a pointer and index as a data pointer is pointing to, so i want pointer (dashboard with all mine visualizations ) to be applied to a new index. To be honest with you i made a python script to work around that problem. you can check it out i posted it as a solution. Also i added feature to "clone" dashboard applying it to new index, so you will have two dashboards, but they will be referencing different indexes .

…

On Thu, Mar 1, 2018 at 5:11 PM, Stacey Gammon ***@***.***> wrote: I have a few questions to everyone that is eagerly looking forward to this feature being implemented, as I start to explore different approaches to solving these problems. First let me try to understand the main problems that have been discussed here: Problem 1: There is no way to easily exchange one index pattern for another in a given set of existing dashboards and visualizations. Problem 2: You don't want to maintain a slew of duplicate visualizations and dashboards where the only thing that differs is the index pattern. This situation occurs because viewers of your dashboards have access to data that resides in different index patterns. Does this sound accurate? For now I'd like to focus only on Problem 2. To give a hypothetical scenario, a company creates Kibana dashboards for their clients, client A and client B. client A's data is in client-a-* index pattern and client B's data is in client-b-* index pattern. The company now has two duplicate sets of visualizations and dashboards, where everything is equal but the index pattern used to create the visualizations. Is that similar to the situation many of you are running into? If so, I'm curious whether anyone has tried to work around that by creating a single set of visualizations that point to index pattern client-*. The two clients data would naturally be filtered because of their permissions, so they would only see their own data. If the issue is that the dashboard creator wants to be able to easily see the data that each client sees, could you add a client field to filter by? [image: screen shot 2018-03-01 at 10 47 49 am] <https://user-images.githubusercontent.com/16563603/36854121-2f297614-1d3e-11e8-8036-c9f69aa99ea0.png> As a side note, there is an _index meta field but you can't search using a wildcard on it, which means if your pattern is client-a-date you wouldn't be able to see across dates. If you go this route, there is only one set of visualizations, and one dashboard to maintain. Data is filtered naturally for viewers based on their permissions, but admins who have access to all the index patterns can view everything, or segment the data. Clients will see the client field but will only see their own name as an option to select in it (so they wouldn't see a reference to a at all). Understanding how that workaround falls short as a solution will help me figure out what concerns need to be addressed as I think about implementing this feature. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3668 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ae1KBra2IKRCwsvTuL6wLY0UKVBlwDXxks5taB2_gaJpZM4EHRML> .

[stacey-gammon]

Interesting, thanks for the response @ArtemUstynov. I'm not able to visit the link you supplied above, I get a 404 error.

How often do you create new indexes with different approaches? Once you have those two approaches, do you keep both around, or do you pick one and discard the other?

If your second index shared a prefix with the index created by your first approach, and you added a field called 'approach' to both indexes, would my solution above then work for you? You would be able to have a single set of visualizations using the shared prefix as the index pattern, and could switch approaches by filtering.

If we allowed users to filter with a wildcard on the automatically included _index field, then you wouldn't even need to manually add an extra field.

I could definitely see the usefulness of a bulk index change for a selected group of saved objects for one off situations. For instance, you created all your visualizations pointing to approach-a-* and now you want to point them to approach-*.

I'm just not sure what the difference is between changing indexes, and filtering on indexes (assuming the index mappings are the same, which they'd have to be for an index swap to seamlessly work). Both are a way to say "I only want to view this set of data, exclude everything else". Users already know how to filter their data on a dashboard, but changing the index would be a new concept with a new UI. I want to make sure, before we introduce a new UI/UX, that it doesn't over complicate or confuse, and that there isn't something already being used we could take advantage of to solve the problem instead.

[ArtemUstynov]

@stacey-gammon here you go https://github.com/ArtemUstynov/kibana_dashboard_manager/blob/master/kibana_manager.py
`

import os
import string
import requests
import json

nativeURL = "http://localhost:5601/es_admin/.kibana/_mget"
HEADERS = {'Content-Type': 'application/json', 'kbn-xsrf': 'true', 'Host': 'localhost:5601',
'Connection': 'keep-alive', 'Accept': 'application/json'}

visURL = "http://localhost:5601/api/saved_objects/visualization?per_page=2000"

vis_list = requests.get(visURL, headers=HEADERS).json()['saved_objects']
oldName = input("Old index name: ")
newName = input("New index name: ")
for vis in vis_list:
    payload = "{\"docs\":[{\"_id\":\"" + vis['id'] + "\" ,\"_type\": \"visualization\"}]}" `'`

    VIS = json.dumps(requests.post(nativeURL, json=json.loads(payload), headers=HEADERS).json())
    VIS = json.dumps(json.loads(VIS)['docs'][0]['_source']).replace(oldName, newName)

    POSTURL = "http://localhost:5601/es_admin/.kibana/visualization/" + vis['id']
    print("ERRORS: " + str(requests.post(POSTURL, json=json.loads(VIS), headers=HEADERS).raise_for_status()))`

Well i am sorry if i didn't explain myself properly and we ended up talking about two different things. This script pretty much does what i wanted. also i made one for "cloning" dashboard. If there a way to do this natively, then i am sorry for waisting your time.

[juancar1979]

Hi @ArtemUstynov !

I have the problem:
nativeURL = "http://localhost:5601/es_admin/.kibana/_mget"
{"statusCode":404,"error":"Not Found"}'

in line: VIS = json.dumps(requests.post(nativeURL, json=json.loads(payload), headers=HEADERS).json())

Can you help me?

thx!!!

[ArtemUstynov]

i access kibana via ssh, so your url might be different, you can find out what is it by going to your browser and monitor network (right click -> inspect element -> network). then go to kibana management and see and download something , that should give you your "native url" somewhere in the response (it will not be called "native url") but it will be easy to spot it, just change mine url to yours.

…

On Tue, Mar 13, 2018 at 9:17 PM, juancar1979 ***@***.***> wrote: Hi @ArtemUstynov <https://github.com/artemustynov> ! I have the problem: nativeURL = "http://localhost:5601/es_admin/.kibana/_mget" {"statusCode":404,"error":"Not Found"}' in line: VIS = json.dumps(requests.post(nativeURL, json=json.loads(payload), headers=HEADERS).json()) Can you help me? thx!!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3668 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ae1KBmkOB6q1cFJaBnm_s6NXcdv92W86ks5teClUgaJpZM4EHRML> .

[juancar1979]

Thanks @ArtemUstynov, but i can't find this URL anywhere. Do you know how I can look for it? Thank you very much!
:)

[ArtemUstynov]

@juancar1979 as i said above -> open your kibana -> go to management -> saved objects -> right click in browser -> inspect element -> network -> (if there is something clear it(crossed circle)) -> now in kibana click export everything -> some stuff will appear in your networking menu -> under column "name" select any field (first for example) -> there will be "Request URL" -> play with it and you will figure out your url. maybe download something else it was't straight forward for me either, but i tried to make it as general of an approach as i could. Also try to print out intermediate values from script, because for some reason console urls not always match with ui. Sorry, but i don't think i can help you more than this, if you use some unusual configuration for elastic server and so on, i would be able to debug it for you. Ask person who set up the servers he might help. Good luck (also you can just export everything and change index names in text editor and import it back, but this is not as cool )

[juancar1979]

thanks @ArtemUstynov !!!! I try and say u!!! 👍

[ArtemUstynov]

@juancar1979 if you use any sort of proxy, that might be a reason why it fails. just add flag not to use proxies in script (it is easily googleble i don't remember exactly )

[ReanimationXP]

+1. Why the hell is this very basic request 3 years old and receiving no attention?

[stacey-gammon]

@ReanimationXP - this is on our radar. Part of the issue is that there are many requests lumped into this single issue. I've split them out to help us better prioritize the work:

Changing index patterns on a visualization: #17542
Dashboard level index patterns: #16917
Improved export/import to include referenced objects. Then you can do a search/replace on an index pattern id in a single file: #16831

[stacey-gammon]

After thinking about it, I believe the best way forward is to close this mega issue to hopefully steer the community towards giving us more specific feedback on the sub issues. If anyone strongly disagrees with this path forward, I'm certainly open to reconsidering, so feel free to comment!

To reiterate what I just added to the top of the issue:

We really appreciate all of your feedback on this mega issue so far, but we need your help as we break this down into more concrete items to work on. Even if you've already +1'd this specific issue, it would be extremely helpful if you could +1 and describe your use case in the more granular issues that are provided below. We won't ignore all the feedback folks have provided so far in this issue, but more focused feedback on the individual feature proposals would be invaluable.

Here are the three sub issues that I believe make up this mega issue:

Changing index patterns on a visualization: #17542
Dashboard level index patterns: #16917
Improved export/import to include referenced objects. Then you can do a search/replace on an index pattern id in a single file: #16831

Dashboard level index patterns seems to match this initial request best and there is a currently available workaround for it, which is mentioned in #16917.

If those three issues do not cover your particular case, please let us know!

[ReanimationXP]

The only thing not addressed is I could see someone wanting to do the reverse as well - monitor the same field from multiple sites (assuming sites == indexes) in one dashboard, varying some other portion of the search.. i.e. "Show me humidity levels at all sites when the temperature is > 50." "Okay, now > 51." Splunk uses optional dashboard-level "global" user variables and controls which are injected into each visualization's search string (including index) to solve both this and the index issue, as I described in #17542. I don't think current filtering could handle such a request across multiple indexes, but I could be wrong. That said, I think simply changing visuals by site (index) will obviously be far more common. A simple 'global' dropdown for changing the index, or a way of bulk-changing the index across multiple visualizations would both be better, faster to implement intermediate solutions than what currently exists, I suspect.

[stacey-gammon]

Very interesting @ReanimationXP. I think our filters can handle this, at least using the workaround mentioned in #16917.

Here is my attempt - I have two indexes animal-cats and animal-dogs, and I have three visualizations - one to show just the cat sounds, one for the dog sounds, and one for all. I can add a filter and it will filter the visualizations even though they have data from different indexes:

Is that about right?

The downside to this workaround, that I see it, is that you might not want to have to create a visualization for each "site/index pattern", in which case I could see dashboard panel level index patterns be a better solution (although then we run into some of the same issues with dashboard level index patterns).

Does sound like it's worth investigating using some sort of search/replace for variables like you mention splunk does. It would be tough to incorporate into our current infrastructure though, we'd need to change a lot of stuff around to make it user friendly, IMO.

[heris25]

I suffer the same issue.
I am using some trick to solve that.

1. save your visualization.( check the "Save as a new visualization")
2. go Management. => saved objects => Visualizations
   and save you want the visualizations.
3. and delete the visualizations.
4. open the json file. and change "kibanaSavedObjectMeta - searchSourceJSON - index uuid" to any uuid
   i change "5dad88d0-475b-11e8-9a8b-51472dd99c91" to "5dad88d0-475b-11e8-9a8b-51472dd99c92"
5. go Management. => saved objects => Visualizations
   and import that json file.
6. you can choose a new index.

good luck.

[chakrayadavalli]

+1

[alanwds]

+1

[tmedford]

+1

[PritomAhmed]

+1

[FrancoisDB]

+1

[kazizi-swe]

@heris25 , could you please explain why in step 3 you delete the visualization? Also, in step 4, could you please clarify where you found the index uuid, I don't see anything named uuid. I appreciate if you can clarify which field I need to modify in kibanaSavedObjectMeta.searchSourceJSON.

{
  "query": {
    "query": "",
    "language": "kuery"
  },
  "filter": [
    {
      "$state": {
        "store": "appState"
      },
      "meta": {
        "alias": null,
        "disabled": false,
        "key": "cloudwatch_logs.log_group",
        "negate": false,
        "params": {
          "query": "/aws/lambda/b2_record_processor"
        },
        "type": "phrase",
        "value": "/aws/lambda/b2_record_processor",
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
      },
      "query": {
        "match": {
          "cloudwatch_logs.log_group": {
            "query": "/aws/lambda/b2_record_processor",
            "type": "phrase"
          }
        }
      }
    },
    {
      "meta": {
        "alias": null,
        "negate": false,
        "type": "phrase",
        "key": "message",
        "value": "ERROR - RECPROC - PROCESS",
        "params": {
          "query": "ERROR - RECPROC - PROCESS"
        },
        "disabled": false,
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[1].meta.index"
      },
      "query": {
        "match": {
          "message": {
            "query": "ERROR - RECPROC - PROCESS",
            "type": "phrase"
          }
        }
      },
      "$state": {
        "store": "appState"
      }
    }
  ],
  "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index"
}