Centralize cryptocurrency pairs in config.js

Improve webpage layout

Improve general performance by eliminating unnecessary writes to the database

.gitignore

@@ -7,6 +7,12 @@ db/*.db
 ### Linux ###
 *~
 
+# ToDo Task
+.todo
+
+# Backup content files
+*.content
+
 # temporary files which can be created if a process still has a handle open of a deleted file
 .fuse_hidden*
 

CONTRIBUTING.md

@@ -0,0 +1,16 @@
+# Contribution Guidelines
+Contributions are pleasantly welcomed, but following few guideline can go a long way in a hassle free in integration of your code. 
+
+* Forking the main repository is always the best way to contribute and keep track of changes.
+* Document your code, and try to follow the code style of the the base project.
+* Add adequate descriptions with your pull requests. The following are few questions you could answer while writing the description:
+  + What were you trying to do, and were you successful in doing so?
+  + Are there any known issues that you want to mention before the code is integrated?
+* Your pull request must pass the placed build tests, unless a reasonable explanation is provided. 
+* Always double check that your changes do not break the main project. 
+
+In case your pull-requests are not accepted, I will make sure to add a reason.
+
+Issues and feature requests should be posted to [Issues][] tab. If you have any other questions, you can contact me via email or Telegram.
+
+[Issues]: https://github.com/uzillion/crypto-whale-watcher/issues

README.md

@@ -1,7 +1,7 @@
 # Crypto Whale Watcher
 [![Build Status](https://travis-ci.org/uzillion/crypto-whale-watcher.svg?branch=master)](https://travis-ci.org/uzillion/crypto-whale-watcher)
 
-Constantly looking at the order book and depth charts of different crypto-currencies on different exchanges can be painstakingly tedious. I decided to create this app to simultaneously monitor different exchanges and currencies without being bothered by insignificant trades and orders. With this app, a person can get real time trade and volume alerts, currently on Telegram, but easily switchable with other services.
+Constantly looking at the order book and depth charts of different crypto-currencies on different exchanges can be painstakingly tedious. I decided to create this app to simultaneously monitor different exchanges and currencies without being bothered by insignificant trades and orders. With this app, a person can get real time trade and volume alerts from GDAX, Binance, and Bitfinex. These alerts currently occur on Telegram, but are easily switchable with other services.
 
 **Note** : This README mostly deals with the setup and execution of the app. If you want to understand the working of the project and code, please refer to the [wiki][] section.
 
@@ -11,6 +11,7 @@ You might notice I haven't used any of the exchanges' respective node modules in
 ## Table of Contents
 * [Basic Setup](#basic-setup)
 * [Customizing](#customizing)
+  + [Currency Pairs](#currency-pairs)
   + [Limits](#limits)
   + [Alerts](#alerts)
 * [Screenshots](#screenshots)
@@ -58,6 +59,9 @@ You might notice I haven't used any of the exchanges' respective node modules in
 ## Customizing
 The app is made to use certain limits and services that may or may not be suitable for others. Therefore it is possible to make changes and customize the app to better suit the developers requirements. Please do not send pull requests to the main repository with these changes.
 
+### Currency Pairs
+Currency pairs are defined in the [config.js][] file, and can easily be added and removed as needed. One needs to be mindful to try and add pairs that are supported by both Binance and Bitfinex. Not doing so can cause unexpected behavior. 
+
 ### Limits
 The alerts are triggered by checking the various limits for the crypto-currency. You can learn about each limit in the [wiki][] section of this project.
 
@@ -78,21 +82,14 @@ The alerts are managed in [message.js](./lib/message.js).
 <img src="./screenshots/web.png" alt="drawing" width="800px"/>
 
 ## Contributing
-I really appreciate all the help that I can get, but following a few guidelines can go a long way in a hassle-free transition of your contributions into the project.
-
-* Forking the main repository is always the best way to contribute and keep track of changes.
-* Document your code, and try to follow the code style of the the base project.
-* Auth Tokens, Access Codes, etc. should be loaded from environment variables, and should not appear anywhere in the code, nor should they be uploaded to the repository in any form.
-* Always try and add as much description as possible with your pull requests. The following are few questions you could answer while writing the description:
-  + What were you trying to do?
-  + Were you successful in doing so?
-  + If yes, briefly describe what you did.
-  + Else, what do you think is the problem, and what all did you try to solve the problem?
-* Your pull request should at least pass the placed build tests. Always double check that your changes do not break the main project. 
-
-In case your pull-requests are not accepted, I will make sure to add a reason.
+If you are a developer trying to contribute to this project, please follow these steps:
+1. Fork and Clone the repository.
+2. Run `npm install`.
+3. Export the DATABASE_URL to the environment using `export` or adding it to a ".env" file.
+4. Run `npm start` or `npm run start:dev` to see if it runs without errors.
+5. Tests can be performed by running `npm test`
 
-Please feel free to post issues and feature requests to the issue tab. If you have any other questions, you can contact me via email or Telegram.
+Please refer [Contribution Guidelines][] for more information.
 
 ## Contact
 **Telegram** : @uzair_inamdar
@@ -121,3 +118,5 @@ If you like the project and would like to keep seeing future improvements, pleas
 [Tutorial]: https://tutorials.botsfloor.com/creating-a-bot-using-the-telegram-bot-api-5d3caed3266d
 [get the group/channel's chat_id]: https://stackoverflow.com/a/32572159
 [wiki]: https://github.com/uzillion/crypto-whale-watcher/wiki
+[config.js]: https://github.com/uzillion/crypto-whale-watcher/blob/master/config.js
+[Contribution Guidelines]: https://github.com/uzillion/pg-accessor/blob/master/CONTRIBUTING.md

bin/www

@@ -11,28 +11,43 @@ var http = require('http');
 const WebSocket = require('ws');
 const trade = require('../lib/trades');
 const wall = require('../lib/orders');
+const {getPairs} = require('../lib/pairs');
 
 
 // ========================================== WebSockets ========================================
 let sockets = {
-  binance: new WebSocket('wss://stream.binance.com:9443/stream?streams=btcusdt@aggTrade/eosusdt@aggTrade/ethusdt@aggTrade/btcusdt@depth/eosusdt@depth/ethusdt@depth'),
   bitfinex: new WebSocket('wss://api.bitfinex.com/ws/2'),
   gdax: new WebSocket('wss://ws-feed.gdax.com')
 }
 
+const buildBinanceSocket = () => {
+  let stream = "";
+  getPairs("binance").forEach((pair, index) => {
+    pair = pair.toLowerCase();
+    if(index != 0)
+      stream += "/";
+
+    stream += `${pair}@aggTrade/${pair}@depth`;
+  });
+  return stream;
+}
+sockets.binance = new WebSocket('wss://stream.binance.com:9443/stream?streams='+ buildBinanceSocket());
+
+
+
 // sockets.binance.close();
 // sockets.bitfinex.close();
 // soclets.gdax.close();
 
 const refreshStream = () => {
-  sockets.binance = new WebSocket('wss://stream.binance.com:9443/stream?streams=btcusdt@aggTrade/eosusdt@aggTrade/ethusdt@aggTrade/btcusdt@depth/eosusdt@depth/ethusdt@depth');
+  sockets.binance = new WebSocket('wss://stream.binance.com:9443/stream?streams='+ buildBinanceSocket());  
 }
 
 // Binance API documentation says the streams expire after 24 hours.
 // Re-subscribing to the stream after 20 hours
 setInterval(refreshStream, 72000000);
 
-sockets.binance.on('message', function incoming(data) {
+sockets.binance.on('message', function(data) {
   let obj = JSON.parse(data).data;
   if(obj.e == "aggTrade")
     trade.binance(obj);
@@ -41,7 +56,7 @@ sockets.binance.on('message', function incoming(data) {
 });
 
 sockets.bitfinex.on('open', () => {
-  const symbols = ["BTCUSD", "EOSUSD", "LTCUSD", "ETHUSD"]
+  const symbols = getPairs();
 
   let request_object = { 
     event: "subscribe", 
@@ -79,15 +94,11 @@ sockets.bitfinex.on('message', (data) => {
   }
 })
 
-
 sockets.gdax.on('open', () => {
+
   let request_object = {     
     "type": "subscribe",     
-    "product_ids": [         
-        "BTC-USD",
-        "LTC-USD",
-        "ETH-USD"
-     ],     
+    "product_ids": getPairs("gdax"),     
      "channels": ["matches"] 
   }
 

config.js

@@ -3,12 +3,30 @@ module.exports = {
   // PARSE_EMODE: "",
   CHAT_ID: process.env.CHAT_ID,
   TESTING: (process.env.TESTING === 'true'),  // Ignore, unless you want to send alerts to a seperate channel while testing, set this to true
-  TEST_CHAT_ID: process.env.TEST_CHAT_ID, // Used when "TESTING" is set to true.
+  TEST_CHAT_ID: (process.env.TEST_CHAT_ID?process.env.TEST_CHAT_ID:""), // Used when "TESTING" is set to true.
+
+  currencies: [
+    "BTCUSD", "ETHUSD", "EOSUSD", "LTCUSD", // USD Comparative (use USD even for USDT)
+    "ETHBTC", "EOSBTC", "LTCBTC",  // BTC Comparative
+  ],
+
   trade: {
-    alerts: true
+    alerts: true,
+    min_worth: {  // Used while migrating alert limits to the database
+      default: 70000, // Default value for when specific value is not specified below
+      BTC: 100000,
+      LTC: 45000 ,
+      ETH: 65000,
+      EOS: 60000
+    }
   },
-  volume: {
-    alerts: true
+
+  order: {
+    alerts: true,
+    min_worth: {  // Used while migrating alert limits to the database
+      default: 7000000, // Default value for when specific value is not specified below
+      BTC: 1000000,
+      LTC: 500000
+    }
   }
-
 }

db/migration.js

@@ -4,6 +4,8 @@
 */
 
 const db = require('./index');
+const config = require('../config');
+const {getCurrencies, interval_id} = require('../lib/pairs');
 
 const createTables = async () => {
 
@@ -34,19 +36,31 @@ const createTables = async () => {
 }
 
 const insertData = async () => {
-  let p1 = db.query(`INSERT INTO MinTradeWorth (symbol, worth) VALUES
-  ($1, $2),
-  ($3, $4),
-  ($5, $6),
-  ($7, $8)
-  `,['BTC', 70000, 'ETH', 50000, 'LTC', 40000, 'EOS', 40000]);
+
+  // let p1 = db.query(`INSERT INTO MinTradeWorth (symbol, worth) VALUES
+  // ($1, $2),
+  // ($3, $4),
+  // ($5, $6),
+  // ($7, $8)
+  // `,['BTC', 70000, 'ETH', 50000, 'LTC', 40000, 'EOS', 40000]);
 
-  let p2 = db.query(`INSERT INTO MinOrderWorth (symbol, worth) VALUES
-  ($1, $2),
-  ($3, $4),
-  ($5, $6),
-  ($7, $8)
-  `,['BTC', 1000000, 'ETH', 600000, 'LTC', 500000, 'EOS', 500000]);
+  getCurrencies(true).forEach(async (currency) => {
+    await db.query(`INSERT INTO MinTradeWorth (symbol, worth) VALUES
+      ($1, $2)`,[currency, config.trade.min_worth[currency]?config.trade.min_worth[currency]:config.trade.min_worth.default]);
+  });
+
+
+  getCurrencies(true).forEach(async (currency) => {
+    await db.query(`INSERT INTO MinOrderWorth (symbol, worth) VALUES
+      ($1, $2)`,[currency, config.order.min_worth[currency]?config.order.min_worth[currency]:config.order.min_worth.default]);
+  });
+
+  // let p2 = db.query(`INSERT INTO MinOrderWorth (symbol, worth) VALUES
+  // ($1, $2),
+  // ($3, $4),
+  // ($5, $6),
+  // ($7, $8)
+  // `,['BTC', 1000000, 'ETH', 600000, 'LTC', 500000, 'EOS', 500000]);
 
   let p3 = db.query(`INSERT INTO VolumeFilter (type, percent) VALUES
   ($1, $2),
@@ -57,16 +71,21 @@ const insertData = async () => {
   ($1)
   `,[2.5]);
 
-  await Promise.all([p1, p2, p3, p4]);
+  await Promise.all([p3, p4]);
 }
 
 
 db.query('DROP SCHEMA public CASCADE')
   .then(() => {
+    console.log("--> Deleting all Tables")
     db.query('CREATE SCHEMA public')
       .then(() => {
+        console.log("--> Re/Creating Tables")
         createTables().then(() => {
-          insertData();
+          insertData().then(() => {
+            console.log("--> Updating Tables");
+            clearInterval(interval_id);
+          });
         }).catch((err) => {
           console.error("Migration failed");
           throw err;

db/orders/getFilter.js

@@ -1,6 +1,6 @@
 const db = require('../index');
 
-let QUERY = `SELECT percent FROM VolumeFilter WHERE type='order'`;
+let QUERY = `SELECT percent FROM VolumeFilter WHERE type='trade'`;
 
 const getVolFilter = () => {
   return db.one(QUERY)

db/orders/setFilter.js

@@ -1,10 +1,11 @@
 const db = require('../index');
 
-let QUERY = `UPDATE VolumeFilter SET percent=$1 WHERE type='order'`;
+let QUERY = `UPDATE VolumeFilter SET percent=$1 WHERE type='trade'`;
 
 const setVolFilter = (percent) => {
   return db.query(QUERY, [percent])
     .catch((err) => {
+      console.log("SetVolFilter:");
       throw err;
     });
 }

db/orders/setMinRatio.js

@@ -5,6 +5,7 @@ let QUERY = 'UPDATE MinVolumeRatio SET ratio=$1';
 const setMinRatio = (ratio) => {
   return db.query(QUERY, ratio)
     .catch((err) => {
+      console.log("setMinRatio:");
       throw err;
     });
 }

lib/orders/binance.js

@@ -1,8 +1,9 @@
 const message = require('../message');
 const request = require('request-promise-native');
 const sd = require('../sd');
-const alerts = require('../../config').volume.alerts;
+const alerts = require('../../config').order.alerts;
 const orders = require('../../db/orders');
+const {getPairs, usd_values} = require('../pairs');
 
 let sensitivity = 100;
 orders.getMinRatio().then((data) => {
@@ -50,7 +51,7 @@ const updateLimits = () => {
 }
 
 const saveBook = () => {
-  ["BTCUSDT", "EOSUSDT", "ETHUSDT"].forEach((symbol) => {
+  getPairs("binance").forEach((symbol) => {
     first_run[symbol] = true;
     requestOptions.uri = `https://www.binance.com/api/v1/depth?symbol=${symbol}&limit=20`;
 
@@ -96,7 +97,8 @@ const compare = (a, b) => {
 
 const binance = (order) => {
   let symbol = order.s;
-  let currency = symbol.substring(0, 3);
+  let base = symbol.substr((symbol.substr(-4) == "USDT"?-4:-3)); // Base Exchange currency
+  let currency = symbol.replace(base, ""); // Actual Traded Currency
   let U = order.U;
   let u = order.u;
   let bids = order.b;
@@ -188,24 +190,25 @@ const binance = (order) => {
 
     let s_total = sell_total[symbol];
     let b_total = buy_total[symbol];
-    let bPrice = 0;
     let askPrice = 0;
-    // try {
-    //   bPrice = parseFloat(book[symbol].bids[book[symbol].bids.length-1][0]);
-    // } catch(e) {
-    //   bPrice = parseFloat(book[symbol].asks[0][0]);
-    // }
+
     try {
       askPrice = parseFloat(book[symbol].asks[0][0]);
     } catch(e) {
       askPrice = parseFloat(book[symbol].bids[book[symbol].bids.length-1][0]);
     }
-    // console.log(symbol, s_total, b_total, askPrice, bPrice);
-    // console.log(book[symbol]);
+
+    let usdExp = /^USD(T)?$/;
+
+    let s_worth = s_total*askPrice*
+      (usdExp.test(base)?1:usd_values[base]);
+
+    let b_worth = b_total*askPrice*
+      (usdExp.test(base)?1:usd_values[base]);
+
     let book_length_check = Math.abs(book[symbol].bids.length - book[symbol].asks.length) < 4;
-    if(book_length_check && ((s_total*askPrice > min_worth[currency]) || (b_total*askPrice > min_worth[currency]))) {
+    if(book_length_check && ((s_worth > min_worth[currency]) || (b_worth > min_worth[currency]))) {
       let sb_ratio = s_total/b_total;
-      // console.log(symbol, sb_ratio, s_total/b_total);
 
       let messageObj = {
         event: "VOLUME",