Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WOSAC Why feeding GT can't have likelihood=1.0 for kinematic_metrics? #893

Open
pengzhenghao opened this issue Dec 18, 2024 · 1 comment

Comments

@pengzhenghao
Copy link

I try to feed GT (repeated 32 times) into the WOSAC eval pipe and find:

scenario_metrics/metametric: 0.77725
scenario_metrics/average_displacement_error: 0.00000
scenario_metrics/min_average_displacement_error: 0.00000
scenario_metrics/linear_speed_likelihood: 0.39830
scenario_metrics/linear_acceleration_likelihood: 0.47793
scenario_metrics/angular_speed_likelihood: 0.45490
scenario_metrics/angular_acceleration_likelihood: 0.64707
scenario_metrics/distance_to_nearest_object_likelihood: 0.21332
scenario_metrics/collision_indication_likelihood: 0.99997
scenario_metrics/time_to_collision_likelihood: 0.99965
scenario_metrics/distance_to_road_edge_likelihood: 0.57061
scenario_metrics/offroad_indication_likelihood: 0.99997
aggregate_metrics/realism_meta_metric: 0.77725
aggregate_metrics/kinematic_metrics: 0.49455
aggregate_metrics/interactive_metrics: 0.82509
aggregate_metrics/map_based_metrics: 0.87730
aggregate_metrics/min_ade: 0.00000

Is this a normal behavior?

@pengzhenghao
Copy link
Author

I wonder if there any issue inside the histogram_estimate of
waymo_open_dataset/wdl_limited/sim_agents_metrics/estimators.py:

If I asking for the log likelihood of all zeros vs all zeros:

image

If I asking for the log likelihood of the same array:

image

(in case you need it, I just randomly grab an agent's heading data:
image

You can find the output of histogram_estimate is quite wierd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant