Can OpenRVDAS strip blank lines from raw input data? #408
Replies: 2 comments 2 replies
-
|
Hi Paul - I believe you should be able to use the StripTransform (
https://github.com/OceanDataTools/openrvdas/blob/master/logger/transforms/strip_transform.py)
for that.
…On Mon, Dec 9, 2024 at 10:46 AM pauldavidmena ***@***.***> wrote:
We (Woods Hole Oceanographic Institution) have an instrument (a Seabird
Electronics SBE48) whose raw data prepends a blank line before its first
write to a serial port and then adds another blank line after each
successive read. When I started up an OpenRVDAS listener for the serial
data, it timed out immediately without reading the data. I wrote a quick
"prep" script to strip out the whitespace, after which the listener behaved
as expected, but I would prefer not to adopt the precedent of altering raw
input data. Does OpenRVDAS have a way to transform input data - essentially
stripping out whitespace before further processing - so that we don't need
to rely on pre-seasoning the data? Thanks! - Paul Mena
—
Reply to this email directly, view it on GitHub
<#408>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFO7V3SDXFOX7QLJAYX6VVL2EXQRPAVCNFSM6AAAAABTJPCOVKVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZXGY2DINJSGY>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hello Pablo - thank you for your quick response. I apologize for not following up sooner. We should probably implement StripTransform for this sort of thing, but one of the developers on our team rewrote the shell script I used as a wrapper for the listen.py command line as a more functional python script, and simply ran the input through a strip() function to essentially ignore blank lines. After that, we performed the required transformations and directed the output to a UDP port. While satisfying one requirement - that of being able to use OpenRVDAS to send transformed serial input to a UDP port and ultimately to a database - we lose the format of the instrument input. Scientists and researchers, key end users of our data products, expect to see transformed data even if the incoming input stream is blank. Here is an example from a recent cruise, with the data acquisition and transform done by the legacy "dsLog" application: sssg@mnemosyn /home/data/underway/raw $ tail ar20241206_1312.SBE48 Every other line sent by the SBE48 instrument is blank, and that's how the input is currently transformed and saved for Science. Is there a way that OpenRVDAS can ignore blank input lines without "normalizing" it first? Thanks! Paul Mena |
Beta Was this translation helpful? Give feedback.
-
|
Hi Paul,
This probably requires a phone/Zoom call, as I'm not confident that I
understand certain terms you're using in the same way you mean them.
"normalizing," "stripping," "losing the format" etc could mean a bunch of
different things, and without seeing what you're trying to achieve, the
data flow you're trying to achieve it with, and where it's breaking, any
advice I gave would just be barking in a forest.
I'm going to be back home Dec 26th, with family visiting until Jan 1. If
this is urgent, though, I can try to find time between now and then.
…On Thu, Dec 19, 2024 at 5:33 AM pauldavidmena ***@***.***> wrote:
Hello Pablo - thank you for your quick response. I apologize for not
following up sooner. We should probably implement StripTransform for this
sort of thing, but one of the developers on our team rewrote the shell
script I used as a wrapper for the listen.py command line as a more
functional python script, and simply ran the input through a strip()
function to essentially ignore blank lines. After that, we performed the
required transformations and directed the output to a UDP port. While
satisfying one requirement - that of being able to use OpenRVDAS to send
transformed serial input to a UDP port and ultimately to a database - we
lose the format of the instrument input. Scientists and researchers, key
end users of our data products, expect to see transformed data even if the
incoming input stream is blank. Here is an example from a recent cruise,
with the data acquisition and transform done by the legacy "dsLog"
application:
***@***.*** /home/data/underway/raw $ tail ar20241206_1312.SBE48
SSW 2024/12/06 13:17:10.530 SBE48
SSW 2024/12/06 13:17:10.662 SBE48 # 6.4724, 16 Dec 2023, 20:33:20
SSW 2024/12/06 13:17:19.572 SBE48
SSW 2024/12/06 13:17:19.703 SBE48 # 6.4822, 16 Dec 2023, 20:33:20
SSW 2024/12/06 13:17:28.614 SBE48
SSW 2024/12/06 13:17:28.746 SBE48 # 6.4919, 16 Dec 2023, 20:33:20
SSW 2024/12/06 13:17:37.655 SBE48
SSW 2024/12/06 13:17:37.787 SBE48 # 6.5014, 16 Dec 2023, 20:33:20
SSW 2024/12/06 13:17:46.697 SBE48
SSW 2024/12/06 13:17:46.828 SBE48 # 6.5106, 16 Dec 2023, 20:33:20
Every other line sent by the SBE48 instrument is blank, and that's how the
input is currently transformed and saved for Science. Is there a way that
OpenRVDAS can ignore blank input lines without "normalizing" it first?
Thanks!
Paul Mena
—
Reply to this email directly, view it on GitHub
<#408 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFO7V3T7MUAMT4QR2YXWZPL2GK4KHAVCNFSM6AAAAABTJPCOVKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNRRGY4TENI>
.
You are receiving this because you commented.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
Pablo, Thank you for the response. I agree that we probably need to have a larger discussion so that - among other things - we can sync up on terminology and the details of what we're seeing at WHOI as well as the data requirements of our science community. I would say that it isn't so urgent that we should interrupt holiday plans, and that it can wait until you and some of my WHOI colleagues are back in their respective offices. Thanks again, Paul |
Beta Was this translation helpful? Give feedback.
-
We (Woods Hole Oceanographic Institution) have an instrument (a Seabird Electronics SBE48) whose raw data prepends a blank line before its first write to a serial port and then adds another blank line after each successive read. When I started up an OpenRVDAS listener for the serial data, it timed out immediately without reading the data. I wrote a quick "prep" script to strip out the whitespace, after which the listener behaved as expected, but I would prefer not to adopt the precedent of altering raw input data. Does OpenRVDAS have a way to transform input data - essentially stripping out whitespace before further processing - so that we don't need to rely on pre-seasoning the data? Thanks! - Paul Mena
Beta Was this translation helpful? Give feedback.
All reactions