You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've got some runs that are sampled at 4096, but the channels don't all start and end at the exact same time, there are a few samples missing at the beginning and end of each channel time series, but not always the same ones. So there are a few Nans at the beginning and end of the RunTS channels. Looks like only Ey, Hx, Hy are the channels that have a few Nans.
Decimation level 0 seems to run fine, but on the subsequent decimations basically the entire time series of Ey, Hx, Hy become Nans. So there is a leakage somewhere of Nans between decimation levels. It seems to be with the decimation method prototype_decimation.
In PR #327 I added a prototype_decimation_4 which uses the scipy.signal accessor methods added to xarray from mth5.timeseries.scipy_filters. This accounts for nans and uses the more efficient and accurate resample_poly method. This seems to alleviate the nan issue and is slightly faster.
Example log output
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392061, data length = (2392057,). Setting end_index to 2392057
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392061, data length = (2392057,). Setting end_index to 2392057
24:05:29T14:30:12 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392061, data length = (2392055,). Setting end_index to 2392055
24:05:29T14:30:12 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T01:09:41.999023437
24:05:29T14:30:14 | WARNING | line:658 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2024-05-18T01:09:41.998779296+00:00 does not match metadata end 2024-05-18T01:09:41.999023437+00:00 updating metatdata value to 2024-05-18T01:09:41.998779296+00:00
24:05:29T14:30:14 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392057, data length = (2392049,). Setting end_index to 2392049
24:05:29T14:30:15 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392057, data length = (2392048,). Setting end_index to 2392048
24:05:29T14:30:15 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392057, data length = (2392052,). Setting end_index to 2392052
24:05:29T14:30:15 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T07:09:41.998047000
24:05:29T14:30:17 | WARNING | line:1036 |mth5.groups.channel_dataset | time_slice | Requested slice is larger than data. Slice length = 2392055, data length = (2392054,). Setting end_index to 2392054
24:05:29T14:30:17 | INFO | line:345 |mth5.timeseries.run_ts | _align_channels | Channels do not have a common end, using latest: 2024-05-18T13:09:41.997559000
24:05:29T14:30:18 | WARNING | line:658 |mth5.timeseries.run_ts | validate_metadata | end time of dataset 2024-05-18T13:09:41.997559+00:00 does not match metadata end 2024-05-18T13:09:41.999023437+00:00 updating metatdata value to 2024-05-18T13:09:41.997559+00:00
24:05:29T14:30:18 | INFO | line:431 |aurora.transfer_function.kernel_dataset | initialize_dataframe_for_processing | Dataset dataframe initialized successfully
24:05:29T14:30:18 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 0 Successfully
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 2 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 2 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:18 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 4 Null values detected in xrds channel hy. Check if this is unexpected.
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:20 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc = False
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 7 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 8 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:20 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 5 Null values detected in xrds channel hz. Check if this is unexpected.
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:22 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc = False
24:05:29T14:30:22 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 1 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | WARNING | line:223 |mt_metadata.timeseries.filters.frequency_response_table_filter | complex_response | Extrapolating, use values outside calibration frequencies with caution
24:05:29T14:30:25 | INFO | line:263 |aurora.pipelines.process_mth5 | save_fourier_coefficients | Skip saving FCs. dec_level_config.save_fc = False
24:05:29T14:30:25 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.006281s (159.197990Hz)
24:05:29T14:30:25 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.004866s (205.523721Hz)
24:05:29T14:30:26 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.003702s (270.111088Hz)
24:05:29T14:30:27 | INFO | line:31 |aurora.time_series.frequency_band_helpers | get_band_for_tf_estimate | Processing band 0.002868s (348.711915Hz)
24:05:29T14:30:28 | INFO | line:123 |aurora.pipelines.transfer_function_kernel | update_dataset_df | DECIMATION LEVEL 1
24:05:29T14:30:29 | INFO | line:143 |aurora.pipelines.transfer_function_kernel | update_dataset_df | Dataset Dataframe Updated for decimation level 1 Successfully
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel ey. Check if this is unexpected.
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel hx. Check if this is unexpected.
24:05:29T14:30:29 | INFO | line:230 |aurora.pipelines.time_series_helpers | nan_to_mean | 598015 Null values detected in xrds channel hy. Check if this is unexpected.
The text was updated successfully, but these errors were encountered:
I've got some runs that are sampled at 4096, but the channels don't all start and end at the exact same time, there are a few samples missing at the beginning and end of each channel time series, but not always the same ones. So there are a few Nans at the beginning and end of the RunTS channels. Looks like only Ey, Hx, Hy are the channels that have a few Nans.
Decimation level 0 seems to run fine, but on the subsequent decimations basically the entire time series of Ey, Hx, Hy become Nans. So there is a leakage somewhere of Nans between decimation levels. It seems to be with the decimation method
prototype_decimation
.In PR #327 I added a
prototype_decimation_4
which uses thescipy.signal
accessor methods added to xarray frommth5.timeseries.scipy_filters
. This accounts for nans and uses the more efficient and accurateresample_poly
method. This seems to alleviate the nan issue and is slightly faster.Example log output
The text was updated successfully, but these errors were encountered: