Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Database indexing: fix mismatching balances/sent/received, negative balances etc #316

Closed
wants to merge 2 commits into from
Closed

Database indexing: fix mismatching balances/sent/received, negative balances etc #316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
56a36a6
Database indexing: fix mismatching balances and add db file locks
TheHolyRoger Dec 18, 2019
84cfbea
Update CHANGELOG & UPGRADE
TheHolyRoger Dec 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
1.6.2
* Fix mismatching balances/sent/received, negative balances etc (requires a reindex if balances are incorrect)
* Added new file lock during database indexing: tmp/db_index.pid
* New setting to lock during indexing: lock_during_index

1.6.1
* fixed last_txs setting
* added hashrate_units setting
Expand Down
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,10 @@ Ensure mongodb is not exposed to the outside world via your mongo config or a fi

**script is already running.**

If you receive this message when launching the sync script either a) a sync is currently in progress, or b) a previous sync was killed before it completed. If you are certian a sync is not in progress remove the index.pid from the tmp folder in the explorer root directory.
If you receive this message when launching the sync script either a) a sync is currently in progress, or b) a previous sync was killed before it completed. If you are certian a sync is not in progress remove the index.pid and db_index.pid from the tmp folder in the explorer root directory.

rm tmp/index.pid
rm tmp/db_index.pid

**exceeding stack size**

Expand Down
6 changes: 6 additions & 0 deletions UPGRADE
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
Note: All updates require the explorer to be restarted

1.6.1 -> 1.6.2
* remove tmp/db_index.pid (if it exists)
* Add new settings to settings.json (see settings.json.template)
* lock_during_index
* Reindex explorerdb if you have mismatching balances (node --stack-size=15000 scripts/sync.js index reindex)

1.6.0 -> 1.6.1
* Add new cryptsy_id and hashrate_units settings (see settings.json.template)
* remove tmp/market.pid (if it exists)
Expand Down
1 change: 1 addition & 0 deletions app.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ app.set('youtube', settings.youtube);
app.set('genesis_block', settings.genesis_block);
app.set('index', settings.index);
app.set('heavy', settings.heavy);
app.set('lock_during_index', settings.lock_during_index);
app.set('txcount', settings.txcount);
app.set('nethash', settings.nethash);
app.set('nethash_units', settings.nethash_units);
Expand Down
244 changes: 158 additions & 86 deletions lib/database.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ var mongoose = require('mongoose')
, Heavy = require('../models/heavy')
, lib = require('./explorer')
, settings = require('./settings')
, fs = require('fs')
, poloniex = require('./markets/poloniex')
, bittrex = require('./markets/bittrex')
, bleutrade = require('./markets/bleutrade')
Expand Down Expand Up @@ -59,12 +60,12 @@ function update_address(hash, txid, amount, type, cb) {
var tx_array = address.txs;
var received = address.received;
var sent = address.sent;
if (type == 'vin') {
sent = sent + amount;
} else {
received = received + amount;
}
if (unique == true) {
if ((unique == true) || (type != tx_array[index].type)) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was some duplicated code here, have stripped it down as it wasn't catching tx's sent to self properly

if (type == 'vin') {
sent = sent + amount;
} else {
received = received + amount;
}
tx_array.push({addresses: txid, type: type});
if ( tx_array.length > settings.txcount ) {
tx_array.shift();
Expand All @@ -78,18 +79,7 @@ function update_address(hash, txid, amount, type, cb) {
return cb();
});
} else {
if (type == tx_array[index].type) {
return cb(); //duplicate
} else {
Address.update({a_id:hash}, {
txs: tx_array,
received: received,
sent: sent,
balance: received - sent
}, function() {
return cb();
});
}
return cb(); //duplicate
}
});
}
Expand Down Expand Up @@ -253,6 +243,53 @@ function get_market_data(market, cb) {
}
}

function create_lock(lockfile, cb) {
if (settings.lock_during_index == true) {
var fname = './tmp/' + lockfile + '.pid';
fs.appendFile(fname, process.pid, function (err) {
if (err) {
console.log("Error: unable to create %s", fname);
process.exit(1);
} else {
return cb();
}
});
} else {
return cb();
}
}

function remove_lock(lockfile, cb) {
if (settings.lock_during_index == true) {
var fname = './tmp/' + lockfile + '.pid';
fs.unlink(fname, function (err){
if(err) {
console.log("unable to remove lock: %s", fname);
process.exit(1);
} else {
return cb();
}
});
} else {
return cb();
}
}

function is_locked(lockfile, cb) {
if (settings.lock_during_index == true) {
var fname = './tmp/' + lockfile + '.pid';
fs.exists(fname, function (exists){
if(exists) {
return cb(true);
} else {
return cb(false);
}
});
} else {
return cb(false);
}
}

module.exports = {
// initialize DB
connect: function(database, cb) {
Expand All @@ -268,6 +305,16 @@ module.exports = {
});
},

is_locked: function(cb) {
is_locked("db_index", function (exists) {
if (exists) {
return cb(true);
} else {
return cb(false);
}
});
},

check_stats: function(coin, cb) {
Stats.findOne({coin: coin}, function(err, stats) {
if(stats) {
Expand Down Expand Up @@ -361,29 +408,43 @@ module.exports = {
},

create_tx: function(txid, cb) {
save_tx(txid, function(err){
if (err) {
return cb(err);
} else {
//console.log('tx stored: %s', txid);
is_locked("db_index", function (exists) {
if (exists) {
console.log("db_index lock file exists...");
return cb();
} else {
save_tx(txid, function(err){
if (err) {
return cb(err);
} else {
//console.log('tx stored: %s', txid);
return cb();
}
});
}
});
},

create_txs: function(block, cb) {
lib.syncLoop(block.tx.length, function (loop) {
var i = loop.iteration();
save_tx(block.tx[i], function(err){
if (err) {
loop.next();
} else {
//console.log('tx stored: %s', block.tx[i]);
loop.next();
}
});
}, function(){
return cb();
is_locked("db_index", function (exists) {
if (exists) {
console.log("db_index lock file exists...");
return cb();
} else {
lib.syncLoop(block.tx.length, function (loop) {
var i = loop.iteration();
save_tx(block.tx[i], function(err){
if (err) {
loop.next();
} else {
//console.log('tx stored: %s', block.tx[i]);
loop.next();
}
});
}, function(){
return cb();
});
}
});
},

Expand Down Expand Up @@ -642,64 +703,75 @@ module.exports = {

// updates tx, address & richlist db's; called by sync.js
update_tx_db: function(coin, start, end, timeout, cb) {
var complete = false;
lib.syncLoop((end - start) + 1, function (loop) {
var x = loop.iteration();
if (x % 5000 === 0) {
Tx.find({}).where('blockindex').lt(start + x).sort({timestamp: 'desc'}).limit(settings.index.last_txs).exec(function(err, txs){
Stats.update({coin: coin}, {
last: start + x - 1,
last_txs: '' //not used anymore left to clear out existing objects
}, function() {});
});
}
lib.get_blockhash(start + x, function(blockhash){
if (blockhash) {
lib.get_block(blockhash, function(block) {
if (block) {
lib.syncLoop(block.tx.length, function (subloop) {
var i = subloop.iteration();
Tx.findOne({txid: block.tx[i]}, function(err, tx) {
if(tx) {
tx = null;
subloop.next();
} else {
save_tx(block.tx[i], function(err){
if (err) {
console.log(err);
} else {
console.log('%s: %s', block.height, block.tx[i]);
}
setTimeout( function(){
tx = null;
subloop.next();
}, timeout);
is_locked("db_index", function (exists) {
if (exists) {
console.log("db_index lock file exists...");
return cb();
} else {
create_lock("db_index", function (){
var complete = false;
lib.syncLoop((end - start) + 1, function (loop) {
var x = loop.iteration();
if (x % 5000 === 0) {
Tx.find({}).where('blockindex').lt(start + x).sort({timestamp: 'desc'}).limit(settings.index.last_txs).exec(function(err, txs){
Stats.update({coin: coin}, {
last: start + x - 1,
last_txs: '' //not used anymore left to clear out existing objects
}, function() {});
});
}
lib.get_blockhash(start + x, function(blockhash){
if (blockhash) {
lib.get_block(blockhash, function(block) {
if (block) {
lib.syncLoop(block.tx.length, function (subloop) {
var i = subloop.iteration();
Tx.findOne({txid: block.tx[i]}, function(err, tx) {
if(tx) {
tx = null;
subloop.next();
} else {
save_tx(block.tx[i], function(err){
if (err) {
console.log(err);
} else {
console.log('%s: %s', block.height, block.tx[i]);
}
setTimeout( function(){
tx = null;
subloop.next();
}, timeout);
});
}
});
}, function(){
blockhash = null;
block = null;
loop.next();
});
} else {
console.log('block not found: %s', blockhash);
loop.next();
}
});
}, function(){
blockhash = null;
block = null;
} else {
loop.next();
}
});
}, function(){
Tx.find({}).sort({timestamp: 'desc'}).limit(settings.index.last_txs).exec(function(err, txs){
Stats.update({coin: coin}, {
last: end,
last_txs: '' //not used anymore left to clear out existing objects
}, function() {
remove_lock("db_index", function(){
return cb();
});
});
} else {
console.log('block not found: %s', blockhash);
loop.next();
}
});
});
} else {
loop.next();
}
});
}, function(){
Tx.find({}).sort({timestamp: 'desc'}).limit(settings.index.last_txs).exec(function(err, txs){
Stats.update({coin: coin}, {
last: end,
last_txs: '' //not used anymore left to clear out existing objects
}, function() {
return cb();
});
});
}
});
},

Expand Down
1 change: 1 addition & 0 deletions lib/settings.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ exports.genesis_tx = "65f705d2f385dc85763a317b3ec000063003d6b039546af5d8195a5ec2
exports.genesis_block = "b2926a56ca64e0cd2430347e383f63ad7092f406088b9b86d6d68c2a34baef51";

exports.heavy = false;
exports.lock_during_index = false;
exports.txcount = 100;
exports.show_sent_received = true;
exports.supply = "COINBASE";
Expand Down
8 changes: 7 additions & 1 deletion routes/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,13 @@ function route_get_tx(res, txid) {
}

function route_get_index(res, error) {
res.render('index', { active: 'home', error: error, warning: null});
db.is_locked(function(locked) {
if (locked) {
res.render('index', { active: 'home', error: error, warning: locale.initial_index_alert});
} else {
res.render('index', { active: 'home', error: error, warning: null});
}
});
}

function route_get_address(res, hash, count) {
Expand Down
3 changes: 3 additions & 0 deletions settings.json.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,9 @@
//heavy (enable/disable additional heavy features)
"heavy": false,

//disable saving blocks & TXs via API during indexing.
"lock_during_index": false,

//amount of txs to index per address (stores latest n txs)
"txcount": 100,

Expand Down