Dynamic loading DataTables full history of paginations on first page

[M-Shahbaz]

Currently first page is showing only no. of entries set by the config settings "last_txs" or full if not specified.

Is there a way to implement DataTables server side processing: pagination of all blockchain data till start for fast loading.

[uaktags]

Hey @M-Shahbaz, I guess I'm not too sure what the ultimate outcome would look like. The datatables is currently reliant on /ext/getlasttxs/:min. This command is simply just a db.get_last_txes and then it sends that out as json.
So i guess the only thing different to do would be to try creating a new api in app.js, move the logic formatting logic from the jade to the api, and try it out.

?

[uaktags]

I'm thinking you're after something like this:

app.js

app.use('/ext/getlasttxsajax', function(req,res){
  db.get_last_txs_ajax(req.query.start, req.query.length,function(txs, count){
    var data = [];
    for(i=0; i<txs.length; i++){
      var row = [];
      row.push(txs[i].blockindex);
      row.push(txs[i].txid);
      row.push(txs[i].vout.length);
      row.push((txs[i].total / 100000000).toFixed(8));
      row.push(new Date((txs[i].timestamp) * 1000).toUTCString());
      data.push(row);
    }
    res.json({"data":data, "draw": req.query.draw, "recordsTotal": count, "recordsFiltered": count});
  });
});

database.js

  get_last_txs_ajax: function(start, length, cb) {
      Tx.count({'total': {$gt: 1}}, function(err, count){
        Tx.find({'total': {$gt: 1}}).sort({_id: 'desc'}).skip(Number(start)).limit(Number(length)).exec(function(err, txs){
          if (err) {
            return cb(err);
          } else {
            return cb(txs, count);
          }
        });
    });
  },

index.jade

var rtable = $('#recent-table').dataTable( {  
        autoWidth: true,      
        searching: false,
        ordering: false,
        responsive: true,
        lengthChange: true,
        processing: true,
        serverSide: true,
        ajax: '/ext/getlasttxsajax',
          
      });

[uaktags]

@M-Shahbaz here's an example of the server-side I detailed above:
http://vulc-test.aioxperts.com/
compare this to the old way:
http://vulc.aioxperts.com/

note this isn't an apples to apples comparison, as vulc-test is running a modified version following what I'm trying to do for #252 , whereas vulc is running ciquidus. So neither of them are vanilla iquidus-explorer.

[M-Shahbaz]

@uaktags Thanks, that's exactly I was looking for.
Also updated for block and tx hyperlinks and limit to max 100 results to prevent potential DOS:

app.js

app.use('/ext/getlasttxsajax', function(req,res){
    if(req.query.length > 100){
        req.query.length = 100;
    }
    db.get_last_txs_ajax(req.query.start, req.query.length,function(txs, count){
        var data = [];
        for(i=0; i<txs.length; i++){
            var row = [];
            row.push('<a href="block/' + txs[i].blockhash + '">' + txs[i].blockindex + '</a>');
            row.push('<a href="tx/' + txs[i].txid + '">' + txs[i].txid + '</a>');
            row.push(txs[i].vout.length);
            row.push((txs[i].total / 100000000).toFixed(8));
            row.push(new Date((txs[i].timestamp) * 1000).toUTCString());
            data.push(row);
        }
        res.json({"data":data, "draw": req.query.draw, "recordsTotal": count, "recordsFiltered": count});
    });
});

[uaktags]

Great catch on the query.length!
Would it be any better/worse to move the html code out of the server side piece itself and back to the template?

var rtable = $('#recent-table').dataTable( {  
        autoWidth: true,      
        searching: false,
        ordering: false,
        responsive: true,
        lengthChange: true,
        processing: true,
        serverSide: true,
        ajax: '/ext/getlasttxsajax',
        rowCallback: function(row, data, index) {
          var blockindex = data[0]; //variables for better readability
          var blockhash = data[1]; //variables for better readability
          var txhash = data[2]; //variables for better readability
          var outputs = data[3]; //variables for better readability
          var amount = data[4]; //variables for better readability
          var timestamp = data[5]; //variables for better readability
          $("td:eq(0)", row).html('<a href="/block/' + blockhash + '">' + blockindex + '</a>');
          $("td:eq(1)", row).html('<a href="/tx/' + txhash + '">' + txhash + '</a>');
          $("td:eq(2)", row).html(outputs);
          $("td:eq(3)", row).html(amount);
          $("td:eq(4)", row).html(timestamp);          
        },
      });```

[uaktags]

Also, to tie in the settings parameter:

app.use('/ext/getlasttxsajax', function(req,res){
  if(req.query.length > settings.index.last_txs){
    req.query.length = settings.index.last_txs;
}

[M-Shahbaz]

That's even better!

[M-Shahbaz]

@uaktags we can implement similar dynamic loading for address transactions history table.

Currently txcount latest txs are updated with sync.js in database.js and ignoring previous history than txcount

We can store complete transaction history of an address and dynamically load address transactions history, which can improve:

1. Performance
2. List all history of address

e.g database.js:

  get_address_txs_ajax: function(hash, start, length, cb) {
    Address.findOne({a_id: hash} , { txs: { $slice: [ Number(start), Number(length) ] } }).exec(function(err, address){
      if(address) {
        return cb(address, address.txs.length);
      } else {
        return cb(err);
      }
    });
  },

[M-Shahbaz]

By removing the txcount limit in database.js to store all transactions history of an address. And then reindex update sync becomes slow. Is there anyway to optimize for fast storing in mongod?

[M-Shahbaz]

While storing all transactions history of an address, BSON document size is limited to 16MB only and ignoring all latest transactions of the address after 16MB BSON document size limitation and would require an alternate approach to store data for address transactions e.g: GridFS

[M-Shahbaz]

@iquidus I updated to store and show all transaction history of an address in pull request 270

Looking for your thoughts.

[uaktags]

Hey @M-Shahbaz , sorry I haven't been working on this in a few weeks as I took on another client that's finally finishing up. I do like the idea of GridFS to overcome the BSON, as this probably affects the extendability of options as well as affects larger coins like BTC, more so than any of the use cases I've used it for.
The PR looks, gotta love Github though, most of the changes are superficial of just being on different linenumbers, with no actual code changes, but GH doesn't care it just paints it red until you hide whitespace changes. -_-.

So AddressTX is the storage of all of these transactions. Looks good. Hows the testing been playing out in terms of before and after results with your used coin?

[M-Shahbaz]

Hey @uaktags , thanks for your response.
AddressTX, yes is the storage of complete history of the address transactions and fast.
The results so far are good e.g: 300K+ history of the address

I also tried with BSON, it was very slow process and was ignoring transactions after 16MB limit and storing only around 150K transactions history of the above same address, that's how I found out the limit.

The white space added by my IDE I think and hiding white spaces with GH feature is showing the code update PR#270 mainly:

1-AddressTX - new collection to store full address history
2-Ajax DataTables to display address history that we did similar for transactions history earlier.

An example of the server-side I detailed above (including address full history and full transactions):
http://www.abbcblockchainexplorer.com/
vs old way:
http://www.abbcexplorer.com/

[TheHolyRoger]

This is included in #319 now, nearly ready to merge.

Thank you for all your work!

[TheHolyRoger]

Merged in 8d44d4a thanks @M-Shahbaz !