Migration script for old LNDhub Redis data

[bumi]

Can we somehow migrate data from the lndhub.js Redis database to our DB?
This would make migration easier.

Maybe one start could be to just create one initial incoming transaction that the user has the correct balance. This way the user would loose the transaction history, but it is already a first step.

I am not sure how doable this is, or how complicated this is as the data structure in Redis seems quite tricky to me.

(potentially writing this in JS might be easier as we can reuse code from the lndhub.js version?)

[skosito]

Hey @bumi , I took a look into lndhub.js and redis data can be retrieved with something like:

redis.keys(keys_prefix*)

Then navigate all keys and values and somehow transform them to lndhub.go db structure. I think key_prefix would be a relational db table, and the rest of the key and value should be entries in that table.
For example users, ids (logins) can be migrated, and then for example balances should be possible etc.

But, maybe I am missing the purpose of the task here, is this is what you meant? How do you get access to this data from redis db?
Also could you please clarify Maybe one start could be to just create one initial incoming transaction that the user has the correct balance. This way the user would loose the transaction history, but it is already a first step.

[bumi]

yeah, the idea is to allow installations of the node.js lndhub to migrate to lndhub.go. for this we have to migrate users and transaction data.

IF it is complicated to migrate the full transaction history then a first step could be to just get the user's balance and create one incoming transaction in lndhub.go. - the the user does not have the full transaction history in lndhub.go but they have the correct balance and one "migration" incoming transaction.

So I think the first goal should be to migrate the users, then the balances and then if possible the transaction history..

[skosito]

@bumi keys for users are stored in lndhub.js in following format:

'user_31afe5c130314a0b07c0_e35ce2cd04d055ab713689db8b4c76ee8437e2129595ecac1ee83852cb4abcd7'

Prefix is user_, login is 31afe5c130314a0b07c0 and last part e35ce2cd04d055ab713689db8b4c76ee8437e2129595ecac1ee83852cb4abcd7 is sha256 password.

Script to read this from redis can be done, but users can not be migrated like this, because lndhub.go is using bcrypt hash for passwords, so it can not work 1:1 to just migrate password hash. Maybe lndhub.go auth code can be updated to support login with sha256 and when user login just swap underlying password hash to bcrypt, or something like that. If that is acceptable to you, migrating users should be straightforward. After that, balances can be moved as well.

EDIT: quick googling around actually gave me same solution:

Typically, sites do a staged conversion where you convert users as they perform successful logins. For example:

1. create a field in your database for password has type, sha256 or bcrypt
2. upon login, verify the password using the type in the database
3. if sha256 and successful, create a new bcrypt entry using the entered password, store that and update the password type to bcrypt. On the next login, bcrypt will now be used for verification.

[skosito]

Very simple script to migrate users (need redis and pg packages). Similar can be done for balances then, balance is saved in kv format: balance_for_{user_id}: balance.

const Redis = require('redis');
const redis = Redis.createClient(); // default is localhost:6379 which you get with running redis container locally

const { Client } = require('pg');

(async () => {
  const pg = new Client({
    user: 'user',
    host: 'localhost',
    database: 'lndhub',
    password: 'password',
    port: 5432,
  });

  // connect to both postgres db and redis
  await pg.connect();
  await redis.connect();
  
  // get all user keys from redis
  const userKeys = await redis.keys("user_*")

  // insert users to postgres db
  for (u of userKeys) {
    const splitted = u.split('_');
    const insertQuery = "INSERT INTO public.users(email, login, password, created_at, updated_at) VALUES ($1, $2, $3, $4, $5)";
    const insertValues = [null, splitted[1], splitted[2], new Date(), null]
    await pg.query(insertQuery, insertValues)
  }
})();

[bumi]

thanks @skosito for documenting this!!!