Skip to content

Latest commit

 

History

History
465 lines (369 loc) · 19.5 KB

README.md

File metadata and controls

465 lines (369 loc) · 19.5 KB

SideSeeing Tools

Version Python License Platform

SideSeeing Tools is a suite of scripts designed to load, preprocess, and analyze data collected using the MultiSensor Data Collection App. These tools facilitate the extraction and visualization of sensor data, making them valuable for urban informatics research and applications. This project is licensed under the MIT License, allowing for open-source distribution and modification. For more details, please refer to the LICENSE file.

SideSeeing Tools is compatible with Python version 3.9 and above. The tools are designed to work across multiple platforms, including Linux, macOS, and Windows. Here are some of the key features of SideSeeing Tools:

  • Data Loading: Easily load data collected using the MultiSensor Data Collection App.
  • Preprocessing: Preprocess the data to make it ready for analysis.
  • Analysis: Perform various analyses on the data, including extracting and visualizing sensor data.
  • Visualization: Generate visual representations of the data, such as plots and maps.
  • Frame Extraction: Extract frames from video files at specified times or positions.
  • Snippet Extraction: Extract snippets from video and sensor data for focused analysis.

Installation

pip install sideseeing-tools

General Usage

Create a dataset

from sideseeing_tools import sideseeing

ds = sideseeing.SideSeeingDS(root_dir='/home/user/my-project')

# Available iterators
#   .instances  // Tip: dictionary of instances (key=name, value=SideSeeingInstance)
#   .iterator   // Tip: for i in ds.iterator: i.name

# Available attributes and methods
#   .metadata() // Tip: generates and prints the dataset metadata
#   .size       // Tip: shows the number of instances  
#   .sensors    // Tip: a dictionary containing the names of the available sensors

Get a random sample from the dataset

my_sample = ds.instance

Check the available sensors by instance

ds.sensors
# This command will produce output like this:

{
    # A key representing the number of available axes
    'sensors1': {
        # A key representing the sensor name
        'lps22h barometer sensor': {
            # Keys representing the instances where the sensor data is found
            'FhdFastest#S10e-2024-08-01-10-42-43-354',
            'FhdGame#S10e-2024-08-01-10-25-08-383',
            'FhdNormal#S10e-2024-08-01-10-02-18-947',
            'FhdUi#S10e-2024-08-01-10-13-50-369'
        },
        'tcs3407 uncalibrated lux sensor': {
            'FhdFastest#S10e-2024-08-01-10-42-43-354',
            ...
        },
        ...
    },
    'sensors3': {
        'ak09918c magnetic field sensor': {...},
        'bmi160_accelerometer accelerometer non-wakeup': {
            'FhdFastest#Mia3-2024-08-01-10-42-44-639',
            'FhdNormal#Mia3-2024-08-01-10-02-22-118',
            ...
        },
        ...
    },
    'sensors6': {
        ...
    }
}

Get accelerometer data from the sample

my_sample = ds.instances['FhdNormal#Mia3-2024-08-01-10-02-22-118']
my_sample_accel_data = my_sample.sensors3['bmi160_accelerometer accelerometer non-wakeup']
my_sample_accel_data
Datetime UTC x y z Time (s)
0 2024-03-21 19:33:01.550000 9.34247 -0.270545 3.10767 0
1 2024-03-21 19:33:01.561000 9.51725 -0.347159 3.00233 0.011
2 2024-03-21 19:33:01.571000 9.46458 -0.407014 2.81079 0.021
3 2024-03-21 19:33:01.581000 9.35205 -0.395043 2.79164 0.031
4 2024-03-21 19:33:01.590000 9.36402 -0.263362 2.77488 0.04

Extract a snippet from a sample (video and sensor data)

my_sample.extract_snippet(
    start_time=2,                        # Start time of the snippet (in seconds)
    end_time=17,                         # End time of the snippet (in seconds)
    output_dir='/home/user/snippet_2_17' # Directory to save the extracted snippet
)

Running the command extract_snippet will generate one file for the video (with audio), one file for consumption data, one file for GPS data, and one file for each sensor type (single-axis, three-axis, three-axis uncalibrated) present in the instance. See an illustrative example in the following file tree.

home/
├─ user/
│  ├─ snippet_2_17/
│  │  ├─ consumption.2_17.csv
│  │  ├─ gps.2_17.csv
│  │  ├─ sensors.one.2_17.csv
│  │  ├─ sensors.three.2_17.csv
│  │  ├─ sensors.three.uncalibrated.2_17.csv
│  │  ├─ video.2_17.mp4

Extract only video snippets

It is possible to extract a snippet from a video (and only the video) using the extract_video_snippet function.

from sideseeing_tools import media

# Extract a 15-second snippet from the video, beginning at the 3-second mark and ending at the 18-second mark
media.extract_video_snippet(
    source_path=my_sample.video,    # Path to the input mp4 file
    start_second=3,                 # Start time of the snippet (in seconds)
    end_second=18,                  # End time of the snippet (in seconds)
    output_path='my_snippet.mp4'    # Path to save the extracted snippet
)

Iterate over the samples

for i in ds.iterator:
    print(i.name, i.video)

Create a plotter

from sideseeing_tools import plot

plotter = plot.SideSeeingPlotter(ds, taxonomy='/home/user/my-project/taxonomy.csv')

# Available methods:
#   .generate_video_for_sensor()
#   .plot_dataset_cities()
#   .plot_dataset_map()
#   .plot_dataset_tags_matrix()
#   .plot_dataset_tags()
#   .plot_instance_audio()
#   .plot_instance_map()
#   .plot_instance_sensors3_and_audio()
#   .plot_instance_video_frames_at_times()
#   .plot_instance_video_frames()
#   .plot_sensor()
#   .plot_sensors()

Frame Extraction Methods

The following methods provide flexible and efficient ways to extract frames from video files, either saving them to disk or returning them in memory for immediate use in your applications. While using the methods below, you can opt for saving the frames to disk or returning them in memory.

extract_frames_at_times

This method extracts frames from a video file at the specified time points and saves them as image files.

Parameters:

  • source_path (str): Path to the source video file.
  • frame_times (list): List of time points (in seconds) at which frames should be extracted.
  • target_dir (str, optional): Path to the target directory where frames will be saved. If None, frames will be returned in memory.
  • prefix (str, optional): Prefix to be added to the frame file names.
  • left_zeros (int, optional): Number of left-padded zeros in the frame file names.

Returns:

  • list: List of paths to the extracted frames or list of frames in memory if target_dir is None.

extract_frames_at_positions

This method extracts frames from a video file at the specified frame positions and saves them as image files.

Parameters:

  • source_path (str): Path to the source video file.
  • frame_positions (list): List of frame positions at which frames should be extracted.
  • target_dir (str, optional): Path to the target directory where frames will be saved. If None, frames will be returned in memory.
  • prefix (str, optional): Prefix to be added to the frame file names.
  • left_zeros (int, optional): Number of left-padded zeros in the frame file names.

Returns:

  • list: List of paths to the extracted frames or list of frames in memory if target_dir is None.

extract_frames_timespan

This method extracts frames from a video file within a specified time span and saves them as image files.

Parameters:

  • source_path (str): Path to the source video file.
  • start_time (float): The start time (in seconds) of the time span from which frames should be extracted.
  • end_time (float): The end time (in seconds) of the time span from which frames should be extracted.
  • target_dir (str, optional): Path to the target directory where frames will be saved. If None, frames will be returned in memory.
  • step (int, optional): The rate at which frames are extracted.
  • prefix (str, optional): Prefix to be added to the frame file names.
  • left_zeros (int, optional): Number of left-padded zeros in the frame file names.

Returns:

  • list: List of paths to the extracted frames or list of frames in memory if target_dir is None.

extract_frames_positionspan

This method extracts frames from a video file within a specified position span and saves them as image files.

Parameters:

  • source_path (str): Path to the source video file.
  • start_frame (int): The start frame position from which frames should be extracted.
  • end_frame (int): The end frame position from which frames should be extracted.
  • target_dir (str, optional): Path to the target directory where frames will be saved. If None, frames will be returned in memory.
  • step (int, optional): The rate at which frames are extracted.
  • prefix (str, optional): Prefix to be added to the frame file names.
  • left_zeros (int, optional): Number of left-padded zeros in the frame file names.

Returns:

  • list: List of paths to the extracted frames or list of frames in memory if target_dir is None.

extract_frames

This method extracts frames from a video file at a specified frame rate and saves them as image files.

Parameters:

  • source_path (str): Path to the source video file.
  • target_dir (str, optional): Path to the target directory where frames will be saved. If None, frames will be returned in memory.
  • step (int, optional): The rate at which frames are extracted.
  • prefix (str, optional): Prefix to be added to the frame file names.
  • left_zeros (int, optional): Number of left-padded zeros in the frame file names.

Returns:

  • list: List of paths to the extracted frames or list of frames in memory if target_dir is None.

Example Usage of Frame Extraction Methods through the SideSeeingInstance:

from sideseeing_tools.sideseeing import SideSeeingDS

# Initialize the SideSeeingDS object
ds = sideseeing.SideSeeingDS(root_dir='/home/user/my-project')

# Get a random instance from the dataset
inst = ds.instance

# Extract frames at specific times and save to disk
inst.extract_frames_at_times(
    frame_times=[1.0, 2.0, 3.0],
    target_dir='output',
    prefix='frame_',
    left_zeros=5
)

Example Usage of Frame Extraction Methods through the media module:

from sideseeing_tools import media

# Extract frames at specific times and save to disk
media.extract_frames_at_times(
    source_path=video_path,
    frame_times=[1.0, 2.0, 3.0],
    target_dir='output',
    prefix='frame_',
    left_zeros=5
)

# Extract frames at specific times and return in memory
frames = media.extract_frames_at_times(
    source_path=video_path,
    frame_times=[1.0, 2.0, 3.0]
)

# Extract frames at specific positions and save to disk
media.extract_frames_at_positions(
    source_path=video_path,
    frame_positions=[30, 60, 90],
    target_dir='output',
    prefix='frame_',
    left_zeros=5
)

# Extract frames at specific positions and return in memory
frames = media.extract_frames_at_positions(
    source_path=video_path,
    frame_positions=[30, 60, 90]
)

# Extract frames within a timespan and save to disk
media.extract_frames_timespan(
    source_path=video_path,
    start_time=10.0,
    end_time=20.0,
    target_dir='output',
    step=30,
    prefix='frame_',
    left_zeros=5
)

# Extract frames within a timespan and return in memory
frames = media.extract_frames_timespan(
    source_path=video_path,
    start_time=10.0,
    end_time=20.0,
    step=30
)

# Extract frames within a position span and save to disk
media.extract_frames_positionspan(
    source_path=video_path,
    start_frame=300,
    end_frame=600,
    target_dir='output',
    step=30,
    prefix='frame_',
    left_zeros=5
)

# Extract frames within a position span and return in memory
frames = media.extract_frames_positionspan(
    source_path=video_path,
    start_frame=300,
    end_frame=600,
    step=30
)

# Extract frames at a specified frame rate and save to disk
media.extract_frames(
    source_path=video_path,
    target_dir='output',
    step=30,
    prefix='frame_',
    left_zeros=5
)

# Extract frames at a specified frame rate and return in memory
frames = media.extract_frames(
    source_path=video_path,
    step=30
)

Additional tips

We suggest implementing the following folder structure: create a directory named data to contain all recordings. By doing so, when instantiating the SideSeeingDataset, a metadata.csv file will be generated in the root directory. Here is the command to instantiate a dataset:

ds = sideseeing.SideSeeingDS('/home/user/my-project', subdir='data', name='MyDataset')

And here is the suggested folder structure:

my-project/
├─ data/
│  ├─ place01/
│  │  ├─ route01/
│  │  │  ├─ consumption.csv
│  │  │  ├─ gps.csv
│  │  │  ├─ metadata.json
│  │  │  ├─ sensors.one.csv
│  │  │  ├─ sensors.three.csv
│  │  │  ├─ sensors.three.uncalibrated.csv
│  │  │  ├─ video.gif
│  │  │  ├─ video.mp4
│  │  │  ├─ video.wav
│  │  ├─ route02/
│  ├─ place02/
│  ├─ place03/
├─ metadata.csv
├─ taxonomy.csv

Sensor data specification before SideSeeing conversion

The following data outlines the specifications of sensor content before SideSeeing conversion, i.e., when accessing them directly through the files generated by the MultSensor Data Collection tool.

File consumption.csv

datetime_utc battery_microamperes
2024-03-21T19:38:04.961Z -1431
2024-03-21T19:38:05.961Z -1011
2024-03-21T19:38:06.961Z -2216

File gps.csv

datetime_utc gps_interval accuracy latitude longitude
2024-03-21T19:38:10.309Z 15 16.0 -23.5645676 -46.7395994
2024-03-21T19:38:38.033Z 15 57.639 -23.5645617 -46.739602
2024-03-21T19:38:54.120Z 15 26.611 -23.5645528 -46.7396658

File sensors.one.csv

timestamp_nano datetime_utc name axis_x accuracy
0 2024-03-13T13:40:27.243Z Palm Proximity Sensor 5.0 3
712657771915658 2024-03-21T19:38:05.015Z TCS3407 Uncalibrated lux Sensor 1810.0 3
712657931915658 2024-03-21T19:38:05.174Z TCS3407 Uncalibrated lux Sensor 1812.0 3

File sensors.three.csv

timestamp_nano datetime_utc name axis_x axis_y axis_z accuracy
712657652031560 2024-03-21T19:38:04.895Z LSM6DSO Acceleration Sensor 9.603442 -0.10295067 3.9959226 3
712657673895658 2024-03-21T19:38:04.916Z LSM6DSO Acceleration Sensor 9.823709 -0.38067806 3.9097314 3
712657652031560 2024-03-21T19:38:04.895Z LSM6DSO Gyroscope Sensor 0.113544576 0.42852196 0.083306745 3

File sensors.three.uncalibrated.csv

timestamp_nano datetime_utc name axis_x axis_y axis_z delta_x delta_y delta_z accuracy
712657851915658 2024-03-21T19:38:05.094Z Uncalibrated Magnetic Sensor 268.56 -9.54 -230.45999 255.48 -2.28 -227.94 3
712657852615658 2024-03-21T19:38:05.096Z Gyroscope sensor UnCalibrated 0.044593163 -0.13439035 0.07086037 -0.003009122 -0.016193425 -0.0026664268 3
712657862615658 2024-03-21T19:38:05.105Z Gyroscope sensor UnCalibrated 0.042760566 -0.05009095 0.100792766 -0.003009122 -0.016193425 -0.0026664268 3
712657874678965 2024-03-21T19:38:05.118Z Uncalibrated Magnetic Sensor 268.62 -8.639999 -231.48 255.48 -2.28 -227.94 3
712657872615658 2024-03-21T19:38:05.116Z Gyroscope sensor UnCalibrated 0.064751714 -0.007330383 0.118507855 -0.003009122 -0.016193425 -0.0026664268 3

List of SideSeeingInstance attributes/methods

Attribute or method Description
geolocation_points Dictionary containing geolocation data, including latitude and longitude coordinates representing geographical points.
geolocation_center Latitude and longitude coordinates representing the geographic center of a specific area.
audio Path to the audio file associated with the collected data.
gif Path to the GIF file associated with the collected data.
video Path to the video file associated with the collected data.
sensors1 Dictionary containing data from one-axis sensors.
sensors3 Dictionary containing data from three-axis sensors.
sensors6 Dictionary containing data from six-axis sensors, including uncalibrated data.
label List of categories and tags representing the taxonomy of sidewalks.
video_start_time Start time of the video associated with the collected data.
video_stop_time Stop time of the video associated with the collected data.
extract_snippet Extracting a snippet from the sample (video, sensor, gps and consumption data).
extract_frames_at_times Extract frames from a video file at specified time points and save them as image files.
extract_frames_at_positions Extract frames from a video file at specified frame positions and save them as image files.
extract_frames_timespan Extract frames from a video file within a specified time span and save them as image files.
extract_frames_positionspan Extract frames from a video file within a specified position span and save them as image files.
extract_frames Extract frames from a video file at a specified frame rate and save them as image files.

Author

Rafael J P Damaceno

About us

The SideSeeing Project aims to develop methods based on Computer Vision and Machine Learning for Urban Informatics applications. Our goal is to devise strategies for obtaining and analyzing data related to urban accessibility. Take a look at our website.