Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If Nan present in run decimation propogates to all nans #328

Open
kujaku11 opened this issue May 29, 2024 · 0 comments
Open

If Nan present in run decimation propogates to all nans #328

kujaku11 opened this issue May 29, 2024 · 0 comments

Comments

@kujaku11
Copy link
Collaborator

I've got some runs that are sampled at 4096, but the channels don't all start and end at the exact same time, there are a few samples missing at the beginning and end of each channel time series, but not always the same ones. So there are a few Nans at the beginning and end of the RunTS channels. Looks like only Ey, Hx, Hy are the channels that have a few Nans.

Decimation level 0 seems to run fine, but on the subsequent decimations basically the entire time series of Ey, Hx, Hy become Nans. So there is a leakage somewhere of Nans between decimation levels. It seems to be with the decimation method prototype_decimation.

In PR #327 I added a prototype_decimation_4 which uses the scipy.signal accessor methods added to xarray from mth5.timeseries.scipy_filters. This accounts for nans and uses the more efficient and accurate resample_poly method. This seems to alleviate the nan issue and is slightly faster.

Example log output

24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392061, data length = (2392057,). Setting end_index to 2392057
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392061, data length = (2392057,). Setting end_index to 2392057
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392061, data length = (2392055,). Setting end_index to 2392055
24:05:29T14:30:12 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T01:09:41.999023437
24:05:29T14:30:14 | WARNING | line:658 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2024-05-18T01:09:41.998779296+00:00 does not match metadata end 2024-05-18T01:09:41.999023437+00:00 updating metatdata value to 2024-05-18T01:09:41.998779296+00:00
24:05:29T14:30:14 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392057, data length = (2392049,). Setting end_index to 2392049
24:05:29T14:30:15 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392057, data length = (2392048,). Setting end_index to 2392048
24:05:29T14:30:15 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392057, data length = (2392052,). Setting end_index to 2392052
24:05:29T14:30:15 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T07:09:41.998047000
24:05:29T14:30:17 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data.  Slice length = 2392055, data length = (2392054,). Setting end_index to 2392054
24:05:29T14:30:17 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T13:09:41.997559000
24:05:29T14:30:18 | WARNING | line:658 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2024-05-18T13:09:41.997559+00:00 does not match metadata end 2024-05-18T13:09:41.999023437+00:00 updating metatdata value to 2024-05-18T13:09:41.997559+00:00
24:05:29T14:30:18 | INFO | line:431 |aurora.transfer_function.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
24:05:29T14:30:18 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 2 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 2 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 4 Null values detected in xrds channel hy. Check if this is unexpected.
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 7 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 8 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 5 Null values detected in xrds channel hz. Check if this is unexpected.
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:05:29T14:30:22 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 1 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc =  False
24:05:29T14:30:25 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.006281s  (159.197990Hz)
24:05:29T14:30:25 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.004866s  (205.523721Hz)
24:05:29T14:30:26 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.003702s  (270.111088Hz)
24:05:29T14:30:27 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.002868s  (348.711915Hz)
24:05:29T14:30:28 | INFO | line:123 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
24:05:29T14:30:29 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel hy. Check if this is unexpected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant