Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Whale sightings data in Signal K #1784

Open
puzzledcherry opened this issue Sep 4, 2024 · 24 comments
Open

Whale sightings data in Signal K #1784

puzzledcherry opened this issue Sep 4, 2024 · 24 comments

Comments

@puzzledcherry
Copy link

I'm running into errors with having the signalK server parse my JSON file. I've attached a screenshot of the server log and the JSON file. I can't seem to find the issue as to why it's not parsing.

signalkSightings.json
Screenshot 2024-09-04 102341

@sbender9
Copy link
Member

sbender9 commented Sep 4, 2024

How are you sending this?

@puzzledcherry
Copy link
Author

Through a file stream with the data type as SignalK

@sbender9
Copy link
Member

sbender9 commented Sep 4, 2024

It should be one delta per line (not an array), no new lines in the deltas

Also, your "context" is not right, you probably want it to be "vessels.self" ?

{     "context": "vessels.self",        "updates": [          {                "timestamp": "2024-08-28T09:15:00-07:00",                "values": [                    {                        "path": "environment.sightings.whales",                        "value": {                            "id": 0,                            "type": "Orca",                            "latitude": 47.6205,                            "longitude": -122.3493,                            "no_sighted": 2,                            "comment": "Whale breaching near ferry",                            "recent": 1                        }                    }                ]            }        ]    }
{        "context": "vessels.self",        "updates": [            {                "timestamp": "2024-08-28T09:45:00-07:00",                "values": [                    {                        "path": "environment.sightings.whales",                        "value": {                            "id": 0,                            "type": "Orca",                            "latitude": 47.6694,                            "longitude": -122.3866,                            "no_sighted": 2,                            "comment": "Spotted heading north",                            "recent": 1                        }                    }                ]            }        ]    }

@puzzledcherry
Copy link
Author

NDJSON format right? When i try to save that to a JSON file it gives me errors.

@sbender9
Copy link
Member

sbender9 commented Sep 4, 2024

the whole file is not really valid json since it's one json per line

@puzzledcherry
Copy link
Author

what should the file type be?

@sbender9
Copy link
Member

sbender9 commented Sep 4, 2024

It's just text. Extension does not matter.

@puzzledcherry
Copy link
Author

Thank you for the info, the inbound datastream icon is now flashing and there's a yellow bar- is this supposed to happen? How can I see my data on a map now? I've attached an image below:

Screenshot 2024-09-05 104113

@tkurki
Copy link
Member

tkurki commented Sep 5, 2024

I think you are now playing the same datafile over and over again, as fast as the server can process (6000+ messages per second).

What are you actually trying to achieve? Where is your data coming from?

@puzzledcherry
Copy link
Author

I am working on a project to visualize whale sightings from a JSON data source on a map. Currently, I have a Python script that generates a static JSON file with whale sighting data. My immediate goal is to display this data on a map and ensure the visualization correctly represents the sightings.

As a next step, I plan to periodically update the JSON file at regular intervals and reflect these updates on the map automatically. Eventually, I aim to overlay this whale sighting data with an AIS viewer to analyze vessel interactions with the whales, but this is a future development phase.

At the moment, I am seeking guidance on how to get started with displaying the static JSON data on a map and understanding how I can have the map stay up-to-date when I run my python script to update the JSON file.

@panaaj
Copy link
Member

panaaj commented Sep 6, 2024

Given your description of what you are trying to achieve it aligns with the concept of Resources.
11.3 Resource Provider Plugins in the Server Documentation will provide some insight on how this works.

You can use the built-in Resources provider plugin configuration to define a path for your resource files (in your case, Whale Sighting entries) which will then be available to client apps such as Freeboard-SK for display.

image

@tkurki
Copy link
Member

tkurki commented Sep 8, 2024

Adrian has a point. Is this more static in nature, like POIs (points of interest with location) or dynamic (like AIS data: "here's a vessel at this point")?

@puzzledcherry
Copy link
Author

The map is dynamic in that it updates to display new sightings as they are submitted by the community. However, the sightings themselves are more static compared to something like AIS data, since vessel positions are constantly updating but the sightings aren't.

In other words, the sightings are static and don't move on the map but newly submitted sightings should be displayed on the map as they roll in. So the map is dynamic but the sightings are static? If that makes sense.

@puzzledcherry
Copy link
Author

@panaaj following your instructions, would I then save the data file into signalk/v2/api/resources/whales? Will it then automatically display the data to the freeboard-SK?

I appreciate everyone's help, I am new to all of this

@puzzledcherry
Copy link
Author

is there a sample JSON i could use when testing the resource plug-in? im messing with it and im wondering if my file is the issue

@panaaj
Copy link
Member

panaaj commented Sep 11, 2024

For testing the resource plugin, you can keep it simple and start with the Waypoint resource schema.

You can post this schema to the /waypoints path or any custom path you create.
e.g. /signalk/v2/api/resources/whales.

Note: If you use the /waypoints path you will see them displayed on the map in Freeboard-SK.
If you use a custom path you will need to use the resource set schema detailed here.

Example: Create a new sighting location by sending a POST request with the following payload:

{
    "name": "Sighting #1",
    "description": "Whales sighted here!",
    "type": "Whale Sighting",
    "feature": {
        "type": "Feature",
        "geometry": {
            "type": "Point",
            "coordinates": [
                5.47728,
                53.24771
            ]
        },
        "properties": {}
    }
}

Deltas are sent when resources are created, updated or deleted using the API.

@puzzledcherry
Copy link
Author

Ok, i got that to display!
How would I pass in multiple sightings though? I tried using the signalk resource set structure which was mentioned in the documentation you linked me to, and also a standard JSON list but neither worked. I've attached my resource/json data files with mock data- if someone could tell me how to display multiple sightings that would be great.

I feel close to solving this issue.

signalkSightings_JSON.json
signalkSightings_SignalKResourceSet.json

@panaaj
Copy link
Member

panaaj commented Sep 12, 2024

How do you want to show multiple sightings?

I will assume one location and some kind of label indicating the number of sightings.

The easiest way to do that without any code updates is to use a waypoint and set the name to be the number of sightings in that location.
The waypoint label is displayed when zoomed in which means no extra clicks.

Currently the waypoint icon does not change based on the type but that is in the pipeline.
"Whale" could be one of the defined icon types.

@panaaj
Copy link
Member

panaaj commented Sep 12, 2024

I feel close to solving this issue.

signalkSightings_JSON.json
signalkSightings_SignalKResourceSet.json

I will have a look at these when I get back to land... Won't be for a few days though.

@puzzledcherry
Copy link
Author

I want to show multiple sightings as multiple points on the map, such as one near Seattle, Bainbridge Islands, and one near Edmonds in Washington state. Not necessarily how many whales were spotted in a pod- in that case, I would display it in the description or title as you said.

My data pull retrieves whale sightings throughout the west coast of Washington state, and I want to display all those sightings along the coast. I'll attach an example of my previous project that did this.

Eventually, I'd like to implement transparency to show time passed and also the connecting lines between sightings, but I feel as though that's a project development further into the future so no need to worry about that right now.
Defining some icon types once we get these whale sightings on the map sounds awesome.

No rush responding! I'll be out of the office all of next week, so I won't be active again until 09/23.

Screenshot 2024-04-25 121006

@panaaj
Copy link
Member

panaaj commented Sep 18, 2024

I feel close to solving this issue.
signalkSightings_JSON.json
signalkSightings_SignalKResourceSet.json

@puzzledcherry So close.....

  • With regards the file signalkSightings_JSON.json, if you want to use waypoints to display sightings, then you will need to place each waypoint in a separate file with a filename that is a v4 UUID (e.g. d16ebf97-f860-44c8-bc9c-b2d4553af61d). These will then appear in the waypoint list with all the other waypoints and can be selected for display in the usual way.
    The easiest way to do this is via the Signal K Resources API as it will both create the waypoint resource and emit a delta to notify clients.

Example:
curl -d "@data.json" -X POST http://localhost:3000/signalk/v2/api/resources/waypoints

data.json

{
    "name": "Whale Sighting #1",
    "description": "Whale sighting from shore.",
    "type": "whale",
    "feature": {
        "type": "Feature",
        "geometry": {
            "type": "Point",
            "coordinates": [
                4.964859905436475,
                53.38410865639514
            ]
        },
        "properties": {}
    }
}
  • The signalkSightings_SignalKResourceSet.json file is formatted correctly and works just fine.
    To display them in Freeboard-SK on the map you will need to select them from the resource layers menu in the following way:
  1. Place your file in the correct location, this is the folder (with the same name you set up in the plugin configuration) located in the resource provider plugin's configuration i.e. ~/.signalk/plugin-config-data/resources-provider/resources/<my_res_set>
    e.g. I created a resource type called islands in the plugin config and then placed the file you provided in the folder ~/.signalk/plugin-config-data/resources-provider/resources/islands

  2. In Freeboard-SK, select your resource set by:

  • Clicking the resource layers button to display the menu and then select the resource type you configured (In my case..islands)
  • In the displayed list of resource sets, select the desired one.

image
image

  • Once selected, the features will be displayed on the map.
    image

@panaaj
Copy link
Member

panaaj commented Sep 21, 2024

FYI.... the next release of Freeboard-SK will support the "whale" waypoint type.
image

@puzzledcherry
Copy link
Author

@panaaj Thank you so much! Your instructions were clear, and I was able to successfully replicate the results. I also love the addition of the "whale" waypoint type – it's exactly what I need!

Could you please guide me/point me in the direction of some resources on how to display hover information for these points? Each sighting includes some details I'd like to show when users hover over them.

@panaaj
Copy link
Member

panaaj commented Sep 27, 2024

So "hover information" I am going to assume refers to information displayed by the client.

The waypoint resource schema contains a feature.properties attribute which could be populated with relevant data and displayed by the client.

Ultimately the behaviour will be determined by the client, so for example, currently Freeboard-SK will:

  • display the resource name at zoom levels above a configurable minimum
  • show a popup with summary details including name and description.

For an example of how Freeboard-SK could display feature properties for waypoints, click on an AIS target and then click Info

@tkurki tkurki changed the title JSON parsing errors Whale sightings data in Signal K Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants