Loading sample data and creating eod snapshot

[szverev]

Hello, thank you very much for the repo!
Downloaded https://datasets.tardis.dev/v1/binance-futures/incremental_book_L2/2020/02/01/BTCUSDT.csv.gz had to be cleaned for me as contained 4 rows with column names instead of the data, so code in Data Preparation.ipynb did not work (FYI).

Then having a problem with creating eod snapshot from that data - it occurs empty. Is that because all the events are of type 2 and an order book in create_last_snapshot turns empty?

[nkaz001]

It seems they removed the sample data. Yes, without incremetal_book_L2 data, there is no order book, so nothing is returned from create_last_snapshot. I recommend you to try out using https://github.com/nkaz001/collect-binancefutures

[nkaz001]

could you check https://docs.tardis.dev/historical-data-details/binance-futures ? it's still valid.

[Corny321]

Good job!

[szverev]

Thank you for your reply.
The file in your example (https://datasets.tardis.dev/v1/binance-futures/incremental_book_L2/2020/02/01/BTCUSDT.csv.gz) happened to work for me, but I had to split and clean it.
By splitting I mean it starts as a file_type = TRADE, and then it switches to file_type = DEPTH - so I had to modify the convert algorithm (see below)
As of cleaning - I debugged the data, and it turned out just to repeat the same data for three times. I splitted it to three files and they occured to be equal.

=== APPENDIX ===
The beginning of the incremental book file:

['exchange', 'symbol', 'timestamp', 'local_timestamp', 'id', 'side', 'price', 'amount']
['binance-futures', 'BTCUSDT', '1580515202342000', '1580515202497052', '35530566', 'buy', '9364.51', '1.197']
['binance-futures', 'BTCUSDT', '1580515202342000', '1580515202497346', '35530567', 'buy', '9365.67', '0.02']
['binance-futures', 'BTCUSDT', '1580515202342000', '1580515202497352', '35530568', 'buy', '9365.86', '0.01']

... then out of nowhere:

['exchange', 'symbol', 'timestamp', 'local_timestamp', 'is_snapshot', 'side', 'price', 'amount']
['binance-futures', 'BTCUSDT', '1580515203429000', '1580515203896061', 'true', 'ask', '9366.32', '0.01']
['binance-futures', 'BTCUSDT', '1580515203429000', '1580515203896061', 'true', 'ask', '9366.36', '0.002']
['binance-futures', 'BTCUSDT', '1580515203429000', '1580515203896061', 'true', 'ask', '9366.37', '0.799']

After the split it provided timestamp order error on merge, so I splitted again by timestamps and got those 3 equal files

My version of a piece of convert.py - did not PR as does not seem like a production-ready case, and like a prominent piece of code. Maybe I am to just notify Tardis on their data.

 print('Reading %s' % file)
        with gzip.open(file, 'r') as f, gzip.open('trade.csv.gz', 'w') as ft, \
                gzip.open('depth.csv.gz', 'w') as fd, open('convert.log', 'w') as fl:
            while True:
                line = f.readline()
                try:
                    if line is None or line == b'':
                        break
                    cols = line.decode().strip().split(',')
                    if len(cols) < 8:
                        print('Warning: Invalid Data Row', cols, line)
                        fl.write(line)
                        continue
                    if cols == [
                            'exchange',
                            'symbol',
                            'timestamp',
                            'local_timestamp',
                            'id',
                            'side',
                            'price',
                            'amount'
                        ]:
                            file_type = TRADE
                            ft.write(line)
                    elif cols == [
                            'exchange',
                            'symbol',
                            'timestamp',
                            'local_timestamp',
                            'is_snapshot',
                            'side',
                            'price',
                            'amount'
                        ]:
                            file_type = DEPTH
                            fd.write(line)
                    elif file_type == TRADE:
                        # Insert TRADE_EVENT
                        tmp[row_num] = [
                            2,
                            int(cols[2]),
                            int(cols[3]),
                            1 if cols[5] == 'buy' else -1,
                            float(cols[6]),
                            float(cols[7])
                        ]
                        row_num += 1
                        ft.write(line)
                    elif file_type == DEPTH:
                        if cols[4] == 'true':
                            # Prepare to insert DEPTH_SNAPSHOT_EVENT
                            if not is_snapshot:
                                is_snapshot = True
                                ss_bid = np.empty((ss_buffer_size, 6), np.float64)
                                ss_ask = np.empty((ss_buffer_size, 6), np.float64)
                                ss_bid_rn = 0
                                ss_ask_rn = 0
                            if cols[5] == 'bid':
                                ss_bid[ss_bid_rn] = [
                                    4,
                                    int(cols[2]),
                                    int(cols[3]),
                                    1,
                                    float(cols[6]),
                                    float(cols[7])
                                ]
                                ss_bid_rn += 1
                            else:
                                ss_ask[ss_ask_rn] = [
                                    4,
                                    int(cols[2]),
                                    int(cols[3]),
                                    -1,
                                    float(cols[6]),
                                    float(cols[7])
                                ]
                                ss_ask_rn += 1
                        else:
                            if is_snapshot:
                                # End of the snapshot.
                                is_snapshot = False
                                # Add DEPTH_CLEAR_EVENT before refreshing the market depth by the snapshot.
                                # Clear the bid market depth within the snapshot bid range.
                                tmp[row_num] = [
                                    3,
                                    ss_bid[0, 1],
                                    ss_bid[0, 2],
                                    1,
                                    ss_bid[-1, 4],
                                    0
                                ]
                                row_num += 1
                                # Add DEPTH_SNAPSHOT_EVENT for the bid snapshot
                                tmp[row_num:row_num + len(ss_bid)] = ss_bid[:]
                                # Clear the ask market depth within the snapshot ask range.
                                tmp[row_num] = [
                                    3,
                                    ss_ask[0, 1],
                                    ss_ask[0, 2],
                                    -1,
                                    ss_ask[-1, 4],
                                    0
                                ]
                                row_num += 1
                                # Add DEPTH_SNAPSHOT_EVENT for the ask snapshot
                                tmp[row_num:row_num + len(ss_ask)] = ss_ask[:]
                            # Insert DEPTH_EVENT
                            tmp[row_num] = [
                                1,
                                int(cols[2]),
                                int(cols[3]),
                                1 if cols[5] == 'bid' else -1,
                                float(cols[6]),
                                float(cols[7])
                            ]
                            row_num += 1
                        fd.write(line)
                except Exception as e:
                    print("String not expected, skipping:" + str(cols) + " " + str(e)) 
                    fl.write('Error line: ' + str(line))

Cheers!

[nkaz001]

It seems weird. The newly downloaded Tardis sample data is okay. Not sure but file corruption might happen in your system.

[szverev]

Might be, thank you!

I also noticed that convert() function clears out DEPTH_SNAPHOT_EVENTs by further data - is it intentional?

Otherwise the code would require increasing row_num after inserting ss_bid and ss_ask, like this:

if is_snapshot:
                            # End of the snapshot.
                            is_snapshot = False
                            # Add DEPTH_CLEAR_EVENT before refreshing the market depth by the snapshot.
                            # Clear the bid market depth within the snapshot bid range.
                            tmp[row_num] = [
                                3,
                                ss_bid[0, 1],
                                ss_bid[0, 2],
                                1,
                                ss_bid[-1, 4],
                                0
                            ]
                            row_num += 1
                            # Add DEPTH_SNAPSHOT_EVENT for the bid snapshot
                            tmp[row_num:row_num + len(ss_bid)] = ss_bid[:]
                            # Increase the tmp array index, so we don't erase the bid snapshot with further lines
                            row_num += ss_bid_rn
                            # Clear the ask market depth within the snapshot ask range.
                            tmp[row_num] = [
                                3,
                                ss_ask[0, 1],
                                ss_ask[0, 2],
                                -1,
                                ss_ask[-1, 4],
                                0
                            ]
                            row_num += 1
                            # Add DEPTH_SNAPSHOT_EVENT for the ask snapshot
                            tmp[row_num:row_num + len(ss_ask)] = ss_ask[:]
                            # Increase the tmp array index, so we don't erase the ask snapshot with further lines
                            row_num += ss_ask_rn
                        # Insert DEPTH_EVENT ...

[nkaz001]

Yes, it's intentional. It clears the depth within the range that the snapshot possesses. richwomanbtc made PR about the issue. Please refer to #22 for more details.

[szverev]

Thank you, got it!

[Lorrine99609993]

good