-
Notifications
You must be signed in to change notification settings - Fork 104
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #491 from opentaps/main
Got the OrbitDB replication to work
- Loading branch information
Showing
16 changed files
with
1,525 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| { | ||
| "env": { | ||
| "browser": true, | ||
| "es2021": true | ||
| }, | ||
| "extends": [ | ||
| "eslint:recommended", | ||
| "plugin:@typescript-eslint/recommended" | ||
| ], | ||
| "parser": "@typescript-eslint/parser", | ||
| "parserOptions": { | ||
| "ecmaVersion": 13, | ||
| "sourceType": "module" | ||
| }, | ||
| "plugins": [ | ||
| "@typescript-eslint" | ||
| ], | ||
| "rules": { | ||
| }, | ||
| "root": true | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # Orbit DB | ||
|
|
||
| This directory contains the common data stored in OrbitDB for emissions calculations. You can use it to replicate the data from our OrbitDB database, or set up your own. | ||
|
|
||
| ## Setup | ||
|
|
||
| Run `npm install` | ||
|
|
||
| ## Replicating Data | ||
|
|
||
| Script: `npx ts-node src/replicateDb.ts` | ||
|
|
||
| By default, this script starts an IPFS node. You can also use another IPFS node, such as a local node started with `ipfs daemon --enable-pubsub-experiment`. In that case, make sure you enable `pubsub`. Then you can use `--ipfsapi http://127.0.0.1:5001/api/v0` to point to your local node. | ||
|
|
||
| By default, this script replicates against the upstream server (`52.204.157.187`). To use another IPFS node as bootstrap, run `ipfs id` to obtain the correct node ID. This is different from which server may run Orbit DB. Then you can use `--bootstrap /ip4/IPADDRESS/tcp/4001/p2p/NODEID` to add your bootstrap node. | ||
|
|
||
| To keep the Orbit DB process running use the `--serve` flag, this allows using the node for replicating other nodes (see examples below). | ||
|
|
||
| This script stores data in `./orbitIpfs` or another directory set by `--ipfsdir` and the orbit DB metadata in `./orbitdb` or another directory set by `--orbitdir`. | ||
|
|
||
| Run `npx ts-node src/replicateDb.ts --ipfsapi http://127.0.0.1:5001/api/v0 --serve` to replicate into the local IPFS daemon and keep it running. | ||
|
|
||
| When this works, you can run | ||
|
|
||
| ``` | ||
| $ ts-node src/getData.ts | ||
| { emission: { value: 17720, uom: 'kg' }, year: 2021 } | ||
| ``` | ||
|
|
||
| Then you can try to replicate from your local instance: Run `ipfs id` to get the local IPFS node ID. For example: | ||
| ``` | ||
| ipfs id | ||
| { | ||
| "ID": "12D3KooWDGUDi3vMhdp3gSq2DM3hJoXBQmmkVPEPcRHZGGPGqit3", | ||
| ... | ||
| } | ||
| ``` | ||
|
|
||
| Then you can run `npx ts-node src/replicateDb.ts --orbitdir orbitdb2 --bootstrap /ip4/127.0.0.1/tcp/4001/p2p/12D3KooWDGUDi3vMhdp3gSq2DM3hJoXBQmmkVPEPcRHZGGPGqit3` to replicate against the local node instead of the upstream server. | ||
|
|
||
| ## Loading Data | ||
|
|
||
| If you want to load your own data, you will need to get your own emissions factors. For example, you can download the [UK Government Greenhouse gas reporting: conversion factors 2019 | ||
| ](https://www.gov.uk/government/publications/greenhouse-gas-reporting-conversion-factors-2019). | ||
|
|
||
| Once you download them, use | ||
|
|
||
| ``` | ||
| $ npx ts-node src/dataLoader.ts load_utility_emissions conversion-factors-2021-flat-file-automatic-processing.xls | ||
| ``` | ||
|
|
||
| It will create an orbitdb address, then parse the worksheet, then load into orbitdb. Then follow the steps from Replicating Data above to get your node id for replicating it. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| { | ||
| "name": "orbitdb", | ||
| "version": "1.0.0", | ||
| "description": "Testing out OrbitDB for use in blockchain-carbon-accounting project", | ||
| "main": "src/dataLoader.ts", | ||
| "engines": { | ||
| "node": ">=16" | ||
| }, | ||
| "scripts": { | ||
| "ipfs": "ts-node src/serveIpfs.ts", | ||
| "serve": "ts-node src/serve.ts", | ||
| "replicateDb": "ts-node src/replicateDb.ts", | ||
| "getData": "ts-node src/getData.ts", | ||
| "build": "tsc", | ||
| "lint": "eslint 'src/**/*.ts'", | ||
| "lint:fix": "eslint 'src/**/*.ts' --fix" | ||
| }, | ||
| "author": "", | ||
| "license": "ISC", | ||
| "dependencies": { | ||
| "cli-progress": "^3.10.0", | ||
| "go-ipfs": "^0.12.0", | ||
| "ipfs": "^0.62.1", | ||
| "ipfs-http-client": "^56.0.1", | ||
| "ipfsd-ctl": "^10.0.6", | ||
| "node-fetch": "^3.2.1", | ||
| "orbit-db": "^0.28.2", | ||
| "orbit-db-docstore": "^1.12.0", | ||
| "orbit-db-io": "^1.0.2", | ||
| "orbit-db-store": "^4.3.3", | ||
| "xlsx": "^0.18.3", | ||
| "yargs": "^17.3.1" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/cli-progress": "^3.9.2", | ||
| "@types/node": "^17.0.21", | ||
| "@types/orbit-db": "github:orbitdb/orbit-db-types", | ||
| "@types/yargs": "^17.0.9", | ||
| "@typescript-eslint/eslint-plugin": "^5.13.0", | ||
| "@typescript-eslint/parser": "^5.13.0", | ||
| "eslint": "^8.10.0", | ||
| "ts-node": "^10.6.0", | ||
| "typescript": "^4.6.2" | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| export const STATE_NAME_MAPPING = { | ||
| AL: 'Alabama', | ||
| AK: 'Alaska', | ||
| AS: 'American_Samoa', | ||
| AZ: 'Arizona', | ||
| AR: 'Arkansas', | ||
| CA: 'California', | ||
| CO: 'Colorado', | ||
| CT: 'Connecticut', | ||
| DE: 'Delaware', | ||
| DC: 'District_Of_Columbia', | ||
| FM: 'Federated_States_Of_Micronesia', | ||
| FL: 'Florida', | ||
| GA: 'Georgia', | ||
| GU: 'Guam', | ||
| HI: 'Hawaii', | ||
| ID: 'Idaho', | ||
| IL: 'Illinois', | ||
| IN: 'Indiana', | ||
| IA: 'Iowa', | ||
| KS: 'Kansas', | ||
| KY: 'Kentucky', | ||
| LA: 'Louisiana', | ||
| ME: 'Maine', | ||
| MH: 'Marshall_Islands', | ||
| MD: 'Maryland', | ||
| MA: 'Massachusetts', | ||
| MI: 'Michigan', | ||
| MN: 'Minnesota', | ||
| MS: 'Mississippi', | ||
| MO: 'Missouri', | ||
| MT: 'Montana', | ||
| NE: 'Nebraska', | ||
| NV: 'Nevada', | ||
| NH: 'New_Hampshire', | ||
| NJ: 'New_Jersey', | ||
| NM: 'New_Mexico', | ||
| NY: 'New_York', | ||
| NC: 'North_Carolina', | ||
| ND: 'North_Dakota', | ||
| MP: 'Northern_Mariana_Islands', | ||
| OH: 'Ohio', | ||
| OK: 'Oklahoma', | ||
| OR: 'Oregon', | ||
| PW: 'Palau', | ||
| PA: 'Pennsylvania', | ||
| PR: 'Puerto_Rico', | ||
| RI: 'Rhode_Island', | ||
| SC: 'South_Carolina', | ||
| SD: 'South_Dakota', | ||
| TN: 'Tennessee', | ||
| TX: 'Texas', | ||
| UT: 'Utah', | ||
| VT: 'Vermont', | ||
| VI: 'Virgin_Islands', | ||
| VA: 'Virginia', | ||
| WA: 'Washington', | ||
| WV: 'West_Virginia', | ||
| WI: 'Wisconsin', | ||
| WY: 'Wyoming', | ||
| }; | ||
|
|
||
| export const COUNTRY_MAPPINGS = { | ||
| BE: 'Belgium', | ||
| BG: 'Bulgaria', | ||
| CZ: 'Czechia', | ||
| DK: 'Denmark', | ||
| DE: 'Germany', | ||
| EE: 'Estonia', | ||
| IE: 'Ireland', | ||
| EL: 'Greece', | ||
| ES: 'Spain', | ||
| FR: 'France', | ||
| HR: 'Croatia', | ||
| IT: 'Italy', | ||
| CY: 'Cyprus', | ||
| LV: 'Latvia', | ||
| LT: 'Lithuania', | ||
| LU: 'Luxembourg', | ||
| HU: 'Hungary', | ||
| MT: 'Malta', | ||
| NL: 'Netherlands', | ||
| AT: 'Austria', | ||
| PL: 'Poland', | ||
| PT: 'Portugal', | ||
| RO: 'Romania', | ||
| SI: 'Slovenia', | ||
| SK: 'Slovakia', | ||
| FI: 'Finland', | ||
| SE: 'Sweden', | ||
| UK: 'United_Kingdom', | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| export const ipfsDirectory = './orbitIpfs' | ||
| export const ipfsRemoteNode = '/ip4/52.204.157.187/tcp/4001/p2p/12D3KooWPKZ2NrS2wGxCQLV2DsEtdZDNRsPhTzdx18PHNvJDeWrQ' | ||
| // export const ipfsRemoteNode = '/ip4/127.0.0.1/tcp/4001/p2p/12D3KooWDGUDi3vMhdp3gSq2DM3hJoXBQmmkVPEPcRHZGGPGqit3' | ||
| export const ipfsOptions = { | ||
| repo: ipfsDirectory, | ||
| config: { | ||
| "Addresses": { | ||
| "Swarm": [ | ||
| "/ip4/127.0.0.1/tcp/4001", | ||
| ], | ||
| "Announce": [], | ||
| "NoAnnounce": [], | ||
| "API": "/ip4/127.0.0.1/tcp/5001", | ||
| "Gateway": "/ip4/127.0.0.1/tcp/8082" | ||
| }, | ||
| "Bootstrap": [ | ||
| ipfsRemoteNode | ||
| ], | ||
| }, | ||
| EXPERIMENTAL: { | ||
| pubsub: true | ||
| } | ||
| } | ||
| // export const ipfsClientUrl = 'http://127.0.0.1:5001/api/v0' | ||
| export const ipfsClientUrl = 'http://52.204.157.187:5001/api/v0' | ||
|
|
||
| export const orbitDbDirectory = './orbitdb' | ||
| export const orbitDbName = 'org.hyperledger.blockchain-carbon-accounting' | ||
| export const orbitDbAddress = 'zdpuAwsVJEKAebXSPJj75UimncxdD11RJ1BbbRPYYVrVdCjed' | ||
| export const orbitDbFullPath = `/orbitdb/${orbitDbAddress}/${orbitDbName}` |
Oops, something went wrong.