Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kibana OSS 7.10.2 Connected to INFINI Easysearch #5

Open
liukuijun666 opened this issue Dec 31, 2024 · 0 comments
Open

Kibana OSS 7.10.2 Connected to INFINI Easysearch #5

liukuijun666 opened this issue Dec 31, 2024 · 0 comments
Labels
INFINI Easysearch INFINI Gateway Kibana OSS 7.10.2

Comments

@liukuijun666
Copy link
Owner

As we all know, there is version verification between various software in the ELK ecosystem. So what should I do if I want to use Kibana to connect to INFINI Easysearch?

Today I will try it out.

First, directly request INFINI Easysearch API and confirm the version information.

# curl -ku admin:a9ce40464e259c7d1e58 https://10.0.0.1:9203
{
  "name" : "node-1",
  "cluster_name" : "infini-logs-tmp",
  "cluster_uuid" : "R_u1UaVQQo-wS1MFGOho3A",
  "version" : {
    "distribution" : "easysearch",
    "number" : "1.9.1",
    "distributor" : "INFINI Labs",
    "build_hash" : "da95c34b95dd934779d7ae5c6c1bccd2f1c25954",
    "build_date" : "2024-12-20T21:01:36.991581Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.2",
    "minimum_wire_lucene_version" : "7.7.0",
    "minimum_lucene_index_compatibility_version" : "7.7.0"
  },
  "tagline" : "You Know, For Easy Search!"
}

INFINI Easysearch provides version compatibility parameters.

elasticsearch.api_compatibility: true
elasticsearch.api_compatibility_version: "7.10.2"

Access the API again after the version compatibility parameters is enabled, the version information returned has been modified.

# curl -ku admin:a9ce40464e259c7d1e58 https://10.0.0.1:9203
{
  "name" : "node-1",
  "cluster_name" : "infini-logs-tmp",
  "cluster_uuid" : "R_u1UaVQQo-wS1MFGOho3A",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "da95c34b95dd934779d7ae5c6c1bccd2f1c25954",
    "build_date" : "2024-12-20T21:01:36.991581Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0"
  },
  "tagline" : "You Know, for Search"
}

Configure Kibana:

elasticsearch.hosts: ["https://10.0.0.1:9203"]
elasticsearch.username: "admin"
elasticsearch.password: "a9ce40464e259c7d1e58"

When starting Kibana, I get a version error message.

# ./bin/kibana --allow-root
  log   [01:28:35.150] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [01:28:35.315] [info][plugins-system] Setting up [40] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaUsageCollection,newsfeed,securityOss,mapsLegacy,kibanaLegacy,share,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,bfetch]
  log   [01:28:35.553] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  log   [01:28:35.934] [error][savedobjects-service] This version of Kibana (v7.10.2) is incompatible with the following Elasticsearch nodes in your cluster: v1.9.1 @ 172.80.0.1:9203 (172.80.0.1)

Analyzing the error message, Kibana should have requested the "/_nodes" API interface. After executing the request, you will see that there is indeed corresponding version information.

# curl -ku admin:a9ce40464e259c7d1e58 https://10.0.0.1:9203/_nodes\?pretty
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "infini-logs-tmp",
  "nodes" : {
    "ZZtIpBZpQxWsy3yRC0FKcg" : {
      "name" : "node-1",
      "transport_address" : "172.80.0.1:9303",
      "host" : "172.80.0.1",
      "ip" : "172.80.0.1",
      "version" : "1.9.1",
      "build_flavor" : "default",
      "build_type" : "tar",
      "build_hash" : "da95c34b95dd934779d7ae5c6c1bccd2f1c25954",
      "total_indexing_buffer" : 107374182,
      "roles" : [
        "data",
        "ingest",
        "master",
        "remote_cluster_client"
      ],
.......

Now let’s try to find a way to adjust the returned version information. The 'set_context' filter of INFINI Gateway can achieve this function.

Configuration (gateway.yml):

entry:
  - name: proxy_es
    enabled: true
    router: my_router
    max_concurrency: 10000
    network:
      binding: 0.0.0.0:9002

router:
  - name: my_router
    default_flow: default_flow
    rules:
      - method:
          - "GET"
        pattern:
          - "/_nodes"
        flow:
          - overwrite_flow

flow:
  - name: default_flow
    filter:
      - http: ##another general option to proxy requests
          schema: "https" #https or http
          max_idle_conn_duration: "900s"
          skip_failure_host: false
          hosts:
            - "10.0.0.1:9203"
  - name: overwrite_flow
    filter:
      - http: ##another general option to proxy requests
          schema: "https" #https or http
          max_idle_conn_duration: "900s"
          skip_failure_host: false
          hosts:
            - "10.0.0.1:9203"
      - set_context:
          context:
            _ctx.response.body_json.nodes.ZZtIpBZpQxWsy3yRC0FKcg.version: '"7.10.2"'

Start INFINI Gateway:

# ./gateway-linux-amd64

   ___   _   _____  __  __    __  _
  / _ \ /_\ /__   \/__\/ / /\ \ \/_\ /\_/\
 / /_\///_\\  / /\/_\  \ \/  \/ //_\\\_ _/
/ /_\\/  _  \/ / //__   \  /\  /  _  \/ \
\____/\_/ \_/\/  \__/    \/  \/\_/ \_/\_/

[GATEWAY] A light-weight, powerful and high-performance search gateway.
[GATEWAY] 1.0.0_SNAPSHOT#001, 2024-09-02 08:53:26, 2024-12-31 10:10:10, 250cefeb7c16438f1bf4e6416b7f3eeeec7d9aef
[12-29 09:30:37] [INF] [app.go:285] initializing gateway, pid: 45373
[12-29 09:30:37] [INF] [app.go:286] using config: /opt/demo-environment/gateway_es_tmp/gateway.yml
[12-29 09:30:37] [INF] [instance.go:78] workspace: /opt/demo-environment/gateway_es_tmp/data/gateway/nodes/ctnsiut3q95hgcqen710
[12-29 09:30:37] [INF] [module.go:136] started module: badger
[12-29 09:30:37] [INF] [api.go:196] local ips: 192.168.3.181, 10.0.0.1, 172.80.2.1, 172.80.3.1, 169.254.25.10, 10.233.39.174, 10.233.9.199, 10.233.0.1, 10.233.0.3, 10.233.62.114, 10.233.23.74, 10.233.27.40, 10.233.23.200, 10.233.3.49, 10.233.23.21, 10.233.19.207, 10.233.41.58, 10.233.28.97, 10.233.17.129, 10.233.14.179, 10.233.24.184, 10.233.14.180, 10.233.112.0, 172.80.0.1, 172.34.0.1, 192.168.122.1
[12-29 09:30:37] [INF] [api.go:360] api listen at: http://0.0.0.0:2904
[12-29 09:30:37] [INF] [module.go:136] started module: api
[12-29 09:30:37] [INF] [module.go:136] started module: disk_queue
[12-29 09:30:37] [INF] [module.go:136] started module: elasticsearch
[12-29 09:30:37] [INF] [module.go:136] started module: kafka_queue
[12-29 09:30:37] [INF] [module.go:136] started module: queue
[12-29 09:30:37] [INF] [module.go:136] started module: redis
[12-29 09:30:37] [INF] [module.go:136] started module: s3
[12-29 09:30:37] [INF] [module.go:136] started module: simple_stats
[12-29 09:30:37] [INF] [module.go:136] started module: task
[12-29 09:30:37] [INF] [module.go:136] started module: pipeline
[12-29 09:30:37] [INF] [module.go:155] started plugin: floating_ip
[12-29 09:30:37] [INF] [module.go:155] started plugin: force_merge
[12-29 09:30:37] [INF] [module.go:155] started plugin: metrics
[12-29 09:30:37] [INF] [module.go:155] started plugin: statsd
[12-29 09:30:37] [INF] [entry.go:398] entry [proxy_es] listen at: http://0.0.0.0:9002
[12-29 09:30:37] [INF] [module.go:155] started plugin: gateway
[12-29 09:30:37] [INF] [module.go:161] all modules are started
[12-29 09:30:37] [INF] [app.go:511] gateway is up and running now.

Confirm version information by accessing the API through INFINI Gateway.

# curl -ku admin:a9ce40464e259c7d1e58 http://10.0.0.1:9002
{
  "name" : "node-1",
  "cluster_name" : "infini-logs-tmp",
  "cluster_uuid" : "R_u1UaVQQo-wS1MFGOho3A",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "da95c34b95dd934779d7ae5c6c1bccd2f1c25954",
    "build_date" : "2024-12-20T21:01:36.991581Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0"
  },
  "tagline" : "You Know, for Search"
}

# curl -ku admin:a9ce40464e259c7d1e58 http://10.0.0.1:9002/_nodes\?pretty
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "infini-logs-tmp",
  "nodes" : {
    "ZZtIpBZpQxWsy3yRC0FKcg" : {
      "name" : "node-1",
      "transport_address" : "172.80.0.1:9303",
      "host" : "172.80.0.1",
      "ip" : "172.80.0.1",
      "version" : "7.10.2",
      "build_flavor" : "default",
      "build_type" : "tar",
      "build_hash" : "da95c34b95dd934779d7ae5c6c1bccd2f1c25954",
      "total_indexing_buffer" : 107374182,
      "roles" : [
        "data",
        "ingest",
        "master",
        "remote_cluster_client"
      ],
......

Modify the Kibana configuration file (config/kibana.yml) and access the cluster through INFINI Gateway.

elasticsearch.hosts: ["http://10.0.0.1:9002"]

Kibana was started successfully without any errors.

# ./bin/kibana --allow-root
  log   [01:31:42.733] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [01:31:42.865] [info][plugins-system] Setting up [40] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,share,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,bfetch]
  log   [01:31:43.098] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...
  log   [01:31:43.151] [info][savedobjects-service] Starting saved objects migrations
  log   [01:31:43.228] [info][plugins-system] Starting [40] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,share,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,bfetch]
  log   [01:31:43.477] [info][listening] Server running at http://0.0.0.0:5602
  log   [01:31:43.516] [info][server][Kibana][http] http server running at http://0.0.0.0:5602
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
INFINI Easysearch INFINI Gateway Kibana OSS 7.10.2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@liukuijun666