Real-time LF sampling #2173

wh201906 · 2023-11-14T02:50:23Z

#2167 with clear commit history and some minor improvements

Known limitations:

Only bits_per_sample = 8 is supported now.
Only USB connection is supported now. The transfer speed should be greater than 300kByte/s to support this feature.
~~When LF image is not loaded into the FPGA, the first 14 bytes could be the response of CMD_WTX rather than real samples.~~ (Fixed in Fix corrupted data caused by CMD_WTX #2193)

1280000 samples -> 10.24s for 125kHz 8bit sampling Allocate graph related memory on heap

Use longer timeout in WaitForRawDataTimeout() to handle CMD_WTX Fix a wrong type Apply changes to other similar part Remove unused instructions

github-actions · 2023-11-14T02:50:37Z

You are welcome to add an entry to the CHANGELOG.md as well

wh201906 · 2023-11-14T03:03:10Z

@henrygab @iceman1001 Could you please review this PR?

henrygab

I reviewed all files EXCEPT client/src/comms.c.
Note that I'm just a random user ... so any comments from Iceman can totally overrule my own. I only noticed a couple bugs (even if one was repeated) ... which is amazing for the size of this PR.

At a minimum, checking for and handling failed memory allocations is critical. The rest of my comments are just fit and finish.

Well done!

henrygab · 2023-11-14T10:05:27Z

client/src/graph.h

@@ -42,7 +42,7 @@ int GetNrzClock(const char *str, bool verbose);
 int GetFskClock(const char *str, bool verbose);
 bool fskClocks(uint8_t *fc1, uint8_t *fc2, uint8_t *rf1, int *firstClockEdge);

-#define MAX_GRAPH_TRACE_LEN (40000 * 8)
+#define MAX_GRAPH_TRACE_LEN (40000 * 32)


In case others wondered:

The maximum graph trace length being hard-coded at compile time appears to be relied on by at least some code in this system.

See, for example, getFromGraphBuf(), which presumes the buffer's size.

Is this therefore also the limit for obtaining longer traces? (i.e., does this limit long traces on clients?)

I guess this is the only limit for obtaining longer traces in the code. I got stack overflows when just increasing this number. And that's why I made commit 8b6a274 to allocate related arrays on the heap.

client/src/cmdlfio.c

client/src/cmdlfhid.c

client/src/cmdlf.c

armsrc/lfsampling.c

armsrc/util.c

client/src/cmddata.c

Check if memory allocation fails Fix memory leak Initialize struct in declaration Add/Fix some notes Remove unlikely() in favor of readability Remove a hard-coded magic number

iceman1001 · 2023-11-17T03:44:36Z

Gentle reminder that we use calloc instead.

Some external libs still uses malloc which we don't want to fiddle with.

wh201906 · 2023-11-17T03:54:51Z

I replaced declarations like this
uint8_t bits[XXX] -> malloc()
uint8_t bits[XXX] = {0} -> calloc()

Plus, all of the memory allocated by malloc() is actually initialized immediately by getFromGraphBuf() in commit 8b6a274, so I think it's not necessary to zero-initialize them first.
Anyway, it's up to you. Would you prefer to still use calloc() for them?

iceman1001 · 2023-11-17T04:13:49Z

I found that using calloc has helped us on different platforms, so yes, that is the prefered way.

@iceman1001

Suggested by @iceman1001 Mainly for 8b6a274 Replaced the malloc() in getSamplesFromBufEx() Added memory allocation result check for getSamplesFromBufEx(), lf_read_internal(), and lf_sniff()

wh201906 · 2023-11-17T04:57:01Z

I've replaced them

iceman1001 · 2023-11-19T10:30:28Z

Since I travelling, I will not be able to look at this PR until I get back.

wh201906 · 2023-11-19T14:05:34Z

No problem, have a good time!

iceman1001 · 2023-11-29T12:55:53Z

Got time to play little with this PR today.

looking good, it has some nice potential.
I found that the lf read -r didn't quite make sense. Should be smoother. end user shouldn't need to know, it should just work.

The demodulation code lfdemod.c has 512 bit limits, which now can be increased much more.

I get some freezes, indicating there are some memory issues still.

but I like it.

scrolling in the plot window is confusing when its this large..

wh201906 · 2023-11-29T13:12:04Z

I found that the lf read -r didn't quite make sense. Should be smoother.

But the real-time stuffs has speed requirements. It's not always working, especially for the wireless connections.

I get some freezes, indicating there are some memory issues still.

You mean it's in lfdemod.c?

iceman1001 · 2023-11-29T15:33:18Z

I would say it should be automatic..
Default sample size or less, should go the old way, if a larger sample size, it should go the alternative way.
lf read isn't like real time sniffing..

Hold on with changing demodulation functions. Otherwise this PR will be become too large.
Better to change that after we get this one merged

wh201906 · 2023-11-29T15:41:20Z

OK. Is there anything I need to do for improving this PR?

Default sample size or less, should go the old way, if a larger sample size, it should go the alternative way.

Sounds reasonable

iceman1001 · 2023-11-29T15:53:50Z

Check for memory leaks.

wh201906 · 2023-11-29T16:25:02Z

Any steps to reproduce the memory leak bugs?

iceman1001 · 2023-11-29T19:02:43Z

do different lf / data operations?

wh201906 · 2023-11-30T05:40:36Z

I got a segmentation fault when running data modulation over 132000 samples.
It should be fixed now.
There are not too many changes to the demodulation functions, so I included them in this PR.

iceman1001 · 2023-12-01T07:51:08Z

alright,
doing some more testing, this text should be adapted to match new samples.

"data samples -n 10000"

static int CmdSamples(const char *Cmd) {

    CLIParserContext *ctx;
    CLIParserInit(&ctx, "data samples",
                  "Get raw samples for graph window (GraphBuffer) from device.\n"
                  "If 0, then get whole big buffer from device.",
                  "data samples\n"
                  "data samples -n 10000"
                 );
    void *argtable[] = {
        arg_param_begin,
        arg_int0("n", NULL, "<dec>", "num of samples (512 - 40000)"),
        arg_lit0("v", "verbose", "verbose output"),
        arg_param_end
    };
    ```

iceman1001 · 2023-12-01T07:51:56Z

Or...
data samples is a old style...

iceman1001 · 2023-12-01T08:06:13Z

The lf search only uses the old one too...

lf search -u

wh201906 · 2023-12-01T08:23:55Z

I'm not sure what I should do now...
It looks like the data samples is only for fetching existing data from the device memory, so I guess it has nothing to do with the real-time samples.
As for lf search -u, I just tested lf search -1u to use the samples from the graphbuffer(300000 samples), and it doesn't crash.

iceman1001 · 2023-12-01T09:54:03Z

One of these days, we should refactor lf search to read once and then send through all the demodulators...

Making sure we don't do multiple reads

iceman1001 · 2023-12-01T09:54:40Z

Lets merge,
and see what kind of bugs is coming out.

I have some suggestions for improvements as stated above.

wh201906 · 2023-12-01T10:03:18Z

Thank you @iceman1001 and @henrygab for reviewing this PR and #2167

wh201906 added 7 commits November 14, 2023 01:05

Reduce duplicate payload structure for lf read/sniff

9e8b1ce

Add async_usb_write_XX()

e1dc17d

Real-time LF sampling mode on armsrc

b4cc7c0

Receive raw data on the client

8fdf04f

Receive raw LF samples on the client

42ab3ee

Bigger graph buffer

8b6a274

1280000 samples -> 10.24s for 125kHz 8bit sampling Allocate graph related memory on heap

Some minor fixes

ab9f49f

Use longer timeout in WaitForRawDataTimeout() to handle CMD_WTX Fix a wrong type Apply changes to other similar part Remove unused instructions

wh201906 marked this pull request as ready for review November 14, 2023 02:50

wh201906 mentioned this pull request Nov 14, 2023

Real-time LF sampling #2167

Closed

Add support for trigger threshold

835b7a9

henrygab suggested changes Nov 15, 2023

View reviewed changes

Apply suggestions from @henrygab

3ee13c9

Check if memory allocation fails Fix memory leak Initialize struct in declaration Add/Fix some notes Remove unlikely() in favor of readability Remove a hard-coded magic number

wh201906 mentioned this pull request Nov 17, 2023

No 'hw ping' or 'hw break' stop support for iclass simulation #2180

Closed

wh201906 force-pushed the lf_sniff_clean branch 2 times, most recently from ca771e0 to 706c2d0 Compare November 17, 2023 04:30

Use calloc() instead of malloc()

e82fb8b

Suggested by @iceman1001 Mainly for 8b6a274 Replaced the malloc() in getSamplesFromBufEx() Added memory allocation result check for getSamplesFromBufEx(), lf_read_internal(), and lf_sniff()

wh201906 force-pushed the lf_sniff_clean branch from 706c2d0 to e82fb8b Compare November 17, 2023 04:41

Fix segmentation fault

45df618

wh201906 force-pushed the lf_sniff_clean branch from 9c21f12 to 45df618 Compare November 30, 2023 10:42

iceman1001 merged commit 17a93a3 into RfidResearchGroup:master Dec 1, 2023

wh201906 deleted the lf_sniff_clean branch December 1, 2023 10:02

wh201906 mentioned this pull request Dec 1, 2023

Use real-time sampling with high sample size #2190

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Real-time LF sampling #2173

Real-time LF sampling #2173

wh201906 commented Nov 14, 2023 •

edited

Loading

github-actions bot commented Nov 14, 2023

wh201906 commented Nov 14, 2023

henrygab left a comment

henrygab Nov 14, 2023

wh201906 Nov 15, 2023

iceman1001 commented Nov 17, 2023

wh201906 commented Nov 17, 2023 •

edited

Loading

iceman1001 commented Nov 17, 2023

wh201906 commented Nov 17, 2023

iceman1001 commented Nov 19, 2023

wh201906 commented Nov 19, 2023

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 29, 2023

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 29, 2023 •

edited

Loading

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 29, 2023

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 30, 2023 •

edited

Loading

iceman1001 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

wh201906 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

wh201906 commented Dec 1, 2023

Real-time LF sampling #2173

Real-time LF sampling #2173

Conversation

wh201906 commented Nov 14, 2023 • edited Loading

github-actions bot commented Nov 14, 2023

wh201906 commented Nov 14, 2023

henrygab left a comment

Choose a reason for hiding this comment

henrygab Nov 14, 2023

Choose a reason for hiding this comment

wh201906 Nov 15, 2023

Choose a reason for hiding this comment

iceman1001 commented Nov 17, 2023

wh201906 commented Nov 17, 2023 • edited Loading

iceman1001 commented Nov 17, 2023

wh201906 commented Nov 17, 2023

iceman1001 commented Nov 19, 2023

wh201906 commented Nov 19, 2023

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 29, 2023

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 29, 2023 • edited Loading

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 29, 2023

iceman1001 commented Nov 29, 2023

wh201906 commented Nov 30, 2023 • edited Loading

iceman1001 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

wh201906 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

iceman1001 commented Dec 1, 2023

wh201906 commented Dec 1, 2023

wh201906 commented Nov 14, 2023 •

edited

Loading

wh201906 commented Nov 17, 2023 •

edited

Loading

wh201906 commented Nov 29, 2023 •

edited

Loading

wh201906 commented Nov 30, 2023 •

edited

Loading