Data Point and API specification #8
Comments
DzhideX
changed the title
Introduction (updated spec)There was a change in the scope of the dashboard API implementation. A decision was made to move on from the more complex time-series + database approach to a simpler current data snapshot approach. The design of the dashboard (including the data needed to support it) has changed as well which means there was a change in the scope. Data points which were not possible to acquire are tagged with Tokenconst COINGECKO_API_KEY = "";
const endDate = new Date();
const startDate = new Date();
endDate.setMonth(endDate.getDate() + 1);
startDate.setMonth(startDate.getMonth() - 6);
const end = endDate.toISOString().split("T")[0];
const start = startDate.toISOString().split("T")[0];
const data = await axios.post(
`https://joystream.api.subscan.io/api/scan/price/history`,
{
currency: "string",
end,
format: "hour",
start,
},
{
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
}
);
const tokenPrices = data.data.data.list;
const currentValue = Number(tokenPrices[tokenPrices.length - 1].price);
const lastHourValue = Number(tokenPrices[tokenPrices.length - 2].price);
const lastDayValue = Number(tokenPrices[tokenPrices.length - 24].price);
const last3DaysValue = Number(tokenPrices[tokenPrices.length - 24 * 3].price);
const lastWeekValue = Number(tokenPrices[tokenPrices.length - 24 * 7].price);
const lastMonthValue = Number(tokenPrices[tokenPrices.length - 24 * 30].price);
const last3MonthsValue = Number(tokenPrices[tokenPrices.length - 24 * 90].price);
const last6MonthsValue = Number(tokenPrices[tokenPrices.length - 24 * 180].price);
const roi1hour = ((currentValue - lastHourValue) / lastWeekValue) * 100;
const roi24hours = ((currentValue - lastDayValue) / lastDayValue) * 100;
const roi3days = ((currentValue - last3DaysValue) / last3DaysValue) * 100;
const lastWeekChange = ((currentValue - lastWeekValue) / lastWeekValue) * 100;
const roi1month = ((currentValue - lastMonthValue) / lastMonthValue) * 100;
const roi3months = ((currentValue - last3MonthsValue) / last3MonthsValue) * 100;
const roi6months = ((currentValue - last6MonthsValue) / last6MonthsValue) * 100;
console.log(`Current price: ${tokenPrices[tokenPrices.length - 1].price} USD`);
console.log(`Week over week price change: ${lastWeekChange}%`);
console.log(`6 months of price data: ${tokenPrices}`);
// ROI
console.log(`ROI 1 hour: ${roi1hour}%`);
console.log(`ROI 24 hours: ${roi24hours}%`);
console.log(`ROI 3 days: ${roi3days}%`);
console.log(`ROI 1 week: ${lastWeekChange}%`);
console.log(`ROI 1 month: ${roi1month}%`);
console.log(`ROI 3 months: ${roi3months}%`);
console.log(`ROI 6 months: ${roi6months}%`);
const circulatingSupply = JoyApi.calculateCirculatingSupply();
const totalSupply = JoyApi.totalIssuanceInJOY();
console.log(`Circulating supply: ${circulatingSupply}`);
console.log(`Total supply: ${totalSupply}`);
console.log(`FDV: ${totalSupply * currentValue}}`);
const now = Math.floor(Date.now() / 1000);
const oneWeekAgo = now - 7 * 24 * 60 * 60;
const sixMonthsAgo = now - 6 * 30 * 24 * 60 * 60;
const {
data: { total_volumes: volumesPastSixMonths },
} = await axios.get(
`https://api.coingecko.com/api/v3/coins/joystream/market_chart/range?vs_currency=usd&from=${sixMonthsAgo}&to=${now}&x-cg-pro-api-key=${COINGECKO_API_KEY}`
);
console.log(`6 months of volume data: ${volumesPastSixMonths}`);
const {
data: { total_volumes: totalVolumesThisWeek, market_caps: marketCapsThisWeek },
} = await axios.get(
`https://api.coingecko.com/api/v3/coins/joystream/market_chart/range?vs_currency=usd&from=${oneWeekAgo}&to=${now}&x-cg-pro-api-key=${COINGECKO_API_KEY}`
);
const currentMarketCap = marketCapsThisWeek[marketCapsThisWeek.length - 1][1];
const lastWeekMarketCap = marketCapsThisWeek[0][1];
const lastWeekMarketCapChange = ((currentMarketCap - lastWeekMarketCap) / lastWeekMarketCap) * 100;
console.log(`Current market cap: ${marketCapsThisWeek[marketCapsThisWeek.length - 1][1]}`);
console.log(`Week over week market cap change: ${lastWeekMarketCapChange}%`);
const currentVolume = totalVolumesThisWeek[totalVolumesThisWeek.length - 1][1];
const lastWeekVolume = totalVolumesThisWeek[0][1];
const lastWeekVolumeChange = ((currentVolume - lastWeekVolume) / lastWeekVolume) * 100;
console.log(`Current volume (24h): ${totalVolumesThisWeek[totalVolumesThisWeek.length - 1][1]}`);
console.log(`Week over week volume change: ${lastWeekVolumeChange}%`);
const {
data: {
data: {
detail: { JOY },
},
},
} = await axios.get(`https://joystream.api.subscan.io/api/scan/token/unique_id`, {
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
});
console.log(`Annual JOY inflation rate: ${JOY.inflation}`);
console.log(
`Percent of supply staked for validation: ${
(Number(JOY.bonded_locked_balance) / Number(JOY.total_issuance)) * 100
}%`
);
APR Calculation (using chain data):
const apr =
rewardHistory.length && !stakingInfo.total.toBn().isZero()
? last(rewardHistory)
.eraReward.toBn()
.muln(ERAS_PER_YEAR)
.mul(validatorInfo.commission.toBn())
.div(stakingInfo.total.toBn())
.divn(10 ** 7) // Convert from Perbill to Percent
.toNumber()
: 0;
console.log(`APR: ${apr}%`);
const {
data: { data: accountData },
} = await axios({
method: "post",
url: "https://joystream.api.subscan.io/api/v2/scan/accounts",
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
data: JSON.stringify({ order_field: "balance", order: "desc", page: 0, row: 100, filter: "" }),
});
const CURRENT_JOY_PRICE = 0.05;
const numberOfAddresses = accountData.count;
const onePercentOfAddressesCount = numberOfAddresses * 0.01;
let currentPageCount = 1;
let addresses: any[] = accountData.list;
while (true) {
const {
data: { data: accountData },
} = await axios({
method: "post",
url: "https://joystream.api.subscan.io/api/v2/scan/accounts",
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
data: JSON.stringify({
order_field: "balance",
order: "desc",
page: currentPageCount,
row: 100,
filter: "",
}),
});
const MINIMUM_DOLLAR_ADDRESS_AMOUNT = 100;
const MINIMUM_JOY_ADDRESS_AMOUNT = MINIMUM_DOLLAR_ADDRESS_AMOUNT / CURRENT_JOY_PRICE;
addresses = [...addresses, ...accountData.list];
currentPageCount++;
if (accountData.list[accountData.list.length - 1].balance < MINIMUM_JOY_ADDRESS_AMOUNT) break;
}
const top100Addresses = addresses.slice(0, 100);
const top1PercentAddresses = addresses.slice(0, onePercentOfAddressesCount);
const DISTRIBUTION_INTEREST_POINTS_IN_JOY = [
10_000_000 / CURRENT_JOY_PRICE,
100_000 / CURRENT_JOY_PRICE,
10_000 / CURRENT_JOY_PRICE,
1000 / CURRENT_JOY_PRICE,
100 / CURRENT_JOY_PRICE,
1_000_000,
];
const addressesWith10MillionUSDOrMore = addresses.filter(
(address) => Number(address.balance) >= DISTRIBUTION_INTEREST_POINTS_IN_JOY[0]
);
const addressesWith100ThousandUSDOrMore = addresses.filter(
(address) => Number(address.balance) >= DISTRIBUTION_INTEREST_POINTS_IN_JOY[1]
);
const addressesWith10ThousandUSDOrMore = addresses.filter(
(address) => Number(address.balance) >= DISTRIBUTION_INTEREST_POINTS_IN_JOY[2]
);
const addressesWith1000USDOrMore = addresses.filter(
(address) => Number(address.balance) >= DISTRIBUTION_INTEREST_POINTS_IN_JOY[3]
);
const addressesWith100USDOrMore = addresses.filter(
(address) => Number(address.balance) >= DISTRIBUTION_INTEREST_POINTS_IN_JOY[4]
);
const addressesWith1MJOYOrMore = addresses.filter(
(address) => Number(address.balance) >= DISTRIBUTION_INTEREST_POINTS_IN_JOY[5]
);
const supplyTop100 = top100Addresses.reduce((acc, curr) => acc + Number(curr.balance), 0);
const supplyTop1Percent = top1PercentAddresses.reduce((acc, curr) => acc + Number(curr.balance), 0);
const supply10MillionUSDOrMore = addressesWith10MillionUSDOrMore.reduce(
(acc, curr) => acc + Number(curr.balance),
0
);
const supply100ThousandUSDOrMore = addressesWith100ThousandUSDOrMore.reduce(
(acc, curr) => acc + Number(curr.balance),
0
);
const supply10ThousandUSDOrMore = addressesWith10ThousandUSDOrMore.reduce(
(acc, curr) => acc + Number(curr.balance),
0
);
const supply1000USDOrMore = addressesWith1000USDOrMore.reduce(
(acc, curr) => acc + Number(curr.balance),
0
);
const supply100USDOrMore = addressesWith100USDOrMore.reduce(
(acc, curr) => acc + Number(curr.balance),
0
);
const supply1MJOYOrMore = addressesWith1MJOYOrMore.reduce(
(acc, curr) => acc + Number(curr.balance),
0
);
const CIRCULATING_SUPPLY = 801_457_204;
console.log(`Supply and % of circulating supply:`);
console.log(`Top 100 Addresses: ${supplyTop100} ${(supplyTop100 / CIRCULATING_SUPPLY) * 100}%`);
console.log(
`Top 1% Addresses: ${supplyTop1Percent} ${(supplyTop1Percent / CIRCULATING_SUPPLY) * 100}%`
);
console.log(
`Addresses with 10M USD or more: ${supply10MillionUSDOrMore} ${
(supply10MillionUSDOrMore / CIRCULATING_SUPPLY) * 100
}%`
);
console.log(
`Addresses with 100K USD or more: ${supply100ThousandUSDOrMore} ${
(supply100ThousandUSDOrMore / CIRCULATING_SUPPLY) * 100
}%`
);
console.log(
`Addresses with 10K USD or more: ${supply10ThousandUSDOrMore} ${
(supply10ThousandUSDOrMore / CIRCULATING_SUPPLY) * 100
}%`
);
console.log(
`Addresses with 1K USD or more: ${supply1000USDOrMore} ${
(supply1000USDOrMore / CIRCULATING_SUPPLY) * 100
}%`
);
console.log(
`Addresses with 100 USD or more: ${supply100USDOrMore} ${
(supply100USDOrMore / CIRCULATING_SUPPLY) * 100
}%`
);
console.log(
`Addresses with 1M JOY or more: ${supply1MJOYOrMore} ${
(supply1MJOYOrMore / CIRCULATING_SUPPLY) * 100
}%`
);
import { GraphQLClient, gql } from "graphql-request";
const oneYearAgo = new Date();
oneYearAgo.setFullYear(oneYearAgo.getFullYear() - 1);
const client = new GraphQLClient("https://query.joystream.org/graphql");
const mintingData = (await client.request(`
{
channelRewardClaimedEvents(
limit:1000000,
where: { createdAt_gte: "${oneYearAgo.toISOString()}" }
) {
amount
}
requestFundedEvents(
limit: 1000000,
where: { createdAt_gte: "${oneYearAgo.toISOString()}" }
) {
amount
}
workers (limit: 1000000) {
payouts {
amount
createdAt
}
}
councilMembers(limit: 1000000) {
rewardpaymenteventcouncilMember {
paidBalance
createdAt
}
}
budgetSpendingEvents(limit: 1000000, where: { createdAt_gte: "${oneYearAgo.toISOString()}" }) {
createdAt
amount
}
}
`)) as any;
getValidatorReward(
api: ApiPromise,
startBlockHash: HexString,
endBlockHash: HexString,
) {
let totalReward = 0;
const startEra = Number(
(await (await api.at(startBlockHash)).query.staking.activeEra()).unwrap()
.index,
);
const endEra = Number(
(await (await api.at(endBlockHash)).query.staking.activeEra()).unwrap()
.index,
);
for (let i = startEra; i <= endEra; i++) {
const reward = await (
await api.at(endBlockHash)
).query.staking.erasValidatorReward(i);
if (!reward.isNone) {
totalReward += Number(reward.unwrap()) / Math.pow(10, 10); // this should use BNs instead
}
}
return totalReward;
}
const cumulativeCreatorPayoutsAmountInHAPI = mintingData.channelRewardClaimedEvents.reduce(
(acc: number, event: any) => acc + Number(event.amount),
0
);
const cumulativeSpendingProposalsAmountInHAPI = mintingData.requestFundedEvents.reduce(
(acc: number, event: any) => acc + Number(event.amount),
0
);
const cumulativeWorkersRewardsAmountInHAPI = mintingData.workers.reduce(
(acc: number, worker: any) =>
acc +
worker.payouts.reduce((acc: number, payout: any) => {
if (new Date(payout.createdAt) > oneYearAgo) {
return acc + Number(payout.amount);
}
return acc;
}, 0),
0
);
const cumulativeCouncilorRewardsAmountInHAPI = mintingData.councilMembers.reduce(
(acc: number, councilMember: any) =>
acc +
councilMember.rewardpaymenteventcouncilMember.reduce((acc: number, reward: any) => {
if (new Date(reward.createdAt) > oneYearAgo) {
return acc + Number(reward.paidBalance);
}
return acc;
}, 0),
0
);
const cumulativeDiscretionaryPaymentAmountInHAPI = mintingData.budgetSpendingEvents.reduce(
(acc: number, event: any) => acc + Number(event.amount),
0
);
const cumulativeWorkerRewardsAmountInHAPI =
cumulativeWorkersRewardsAmountInHAPI +
cumulativeCouncilorRewardsAmountInHAPI +
cumulativeDiscretionaryPaymentAmountInHAPI;
console.log(`Cumulative creator payouts amount (in HAPI): ${cumulativeCreatorPayoutsAmountInHAPI}`);
console.log(
`Cumulative spending proposals amount (in HAPI): ${cumulativeSpendingProposalsAmountInHAPI}`
);
console.log(`Cumulative workers rewards amount (in HAPI): ${cumulativeWorkerRewardsAmountInHAPI}`);
console.log(`Cumulative validator rewards amount (in HAPI): ${getValidatorReward()}`);Tractionconst hapiToJoy = (hapi: number) => {
return hapi / 10_000_000_000;
};
const paginatedFetch = async (queryFunction) => {
let resultItems: any[] = [];
let offset = 0;
const NUMBER_OF_ITEMS_TO_FETCH = 100_000;
while (true) {
try {
const {
data: { data },
} = await axios({
url: "https://query.joystream.org/graphql",
method: "post",
data: {
query: `
query MyQuery {
${queryFunction(offset, NUMBER_OF_ITEMS_TO_FETCH)}
}
`,
},
});
const key = Object.keys(data)[0];
const items = data[key];
resultItems = [...resultItems, ...items];
if (items.length < NUMBER_OF_ITEMS_TO_FETCH) {
break;
}
offset += NUMBER_OF_ITEMS_TO_FETCH;
} catch (e) {
return resultItems;
}
}
return resultItems;
};
const separateDataByWeekAndAmount = (data: any[]) => {
let weekIndex = 0;
let weeks = [];
let firstDate = new Date(data[0].createdAt);
let secondDate = new Date(firstDate.getTime());
secondDate = new Date(firstDate.getTime() + 7 * 24 * 60 * 60 * 1000);
for (let item of data) {
if (new Date(item.createdAt) >= firstDate && new Date(item.createdAt) < secondDate) {
if (!weeks[weekIndex]) {
weeks[weekIndex] = { from: firstDate, to: secondDate, numberOfItems: 0, amount: 0 };
if (secondDate > new Date()) {
weeks[weekIndex].to = new Date();
}
}
weeks[weekIndex].numberOfItems++;
weeks[weekIndex].amount += hapiToJoy(Number(item.price));
} else {
weekIndex++;
firstDate = new Date(secondDate.getTime());
secondDate = new Date(firstDate.getTime() + 7 * 24 * 60 * 60 * 1000);
if (!weeks[weekIndex]) {
weeks[weekIndex] = { from: firstDate, to: secondDate, numberOfItems: 0, amount: 0 };
if (secondDate > new Date()) {
weeks[weekIndex].to = new Date();
}
}
weeks[weekIndex].numberOfItems++;
weeks[weekIndex].amount += hapiToJoy(Number(item.price));
}
}
return weeks;
};
const separateDataByWeek = (data: any[]) => {
let weekIndex = 0;
let weeks = [];
let firstDate = new Date(data[0].createdAt);
let secondDate = new Date(firstDate.getTime());
secondDate = new Date(firstDate.getTime() + 7 * 24 * 60 * 60 * 1000);
for (let item of data) {
if (new Date(item.createdAt) >= firstDate && new Date(item.createdAt) < secondDate) {
if (!weeks[weekIndex]) {
weeks[weekIndex] = { from: firstDate, to: secondDate, numberOfItems: 0 };
if (secondDate > new Date()) {
weeks[weekIndex].to = new Date();
}
}
weeks[weekIndex].numberOfItems++;
} else {
weekIndex++;
firstDate = new Date(secondDate.getTime());
secondDate = new Date(firstDate.getTime() + 7 * 24 * 60 * 60 * 1000);
if (!weeks[weekIndex]) {
weeks[weekIndex] = { from: firstDate, to: secondDate, numberOfItems: 0 };
if (secondDate > new Date()) {
weeks[weekIndex].to = new Date();
}
}
weeks[weekIndex].numberOfItems++;
}
}
return weeks;
};
const date = new Date();
date.setMonth(date.getMonth() - 6);
const CHANNELS_QUERY = `
channelsConnection {
totalCount
}
channels(limit: 1000000, where: { createdAt_gt: "${date.toISOString()}" }, orderBy: createdAt_ASC) {
createdAt
}
`;
const VIDEOS_CONNECTION_QUERY = `
videosConnection {
totalCount
}
`;
const VIDEOS_QUERY = (offset: number, limit: number) => `
videos(offset: ${offset}, limit: ${limit}, where: { createdAt_gt: "${date.toISOString()}" }, orderBy: createdAt_ASC) {
createdAt
}
`;
const COMMENTS_AND_REACTIONS_QUERY = `
commentsConnection {
totalCount
}
commentReactionsConnection {
totalCount
}
videoReactionsConnection {
totalCount
}
comments (limit:1000000, where: { createdAt_gt: "${date.toISOString()}" }, orderBy: createdAt_ASC) {
createdAt
}
commentReactions (limit:1000000, where: { createdAt_gt: "${date.toISOString()}" }, orderBy: createdAt_ASC) {
createdAt
}
videoReactions (limit:1000000, where: { createdAt_gt: "${date.toISOString()}" }, orderBy: createdAt_ASC) {
createdAt
}
`;
const NFT_BOUGHT_EVENTS_QUERY = `
nftBoughtEvents (limit: 1000000) {
price
createdAt
video {
title
}
member {
handle
}
}
`;
const { data: channelsData } = await axios({
url: "https://query.joystream.org/graphql",
method: "post",
data: {
query: `
query MyQuery {
${CHANNELS_QUERY}
}
`,
},
});
const {
data: {
channels,
channelsConnection: { totalCount: allTimeNumberOfChannels },
},
} = channelsData;
console.log(allTimeNumberOfChannels);
console.log(separateDataByWeek(channels));
const videos = await paginatedFetch(VIDEOS_QUERY);
const { data: videosData } = await axios({
url: "https://query.joystream.org/graphql",
method: "post",
data: {
query: `
query MyQuery {
${VIDEOS_CONNECTION_QUERY}
}
`,
},
});
const {
data: {
videosConnection: { totalCount: allTimeNumberOfVideos },
},
} = videosData;
console.log(allTimeNumberOfVideos);
console.log(separateDataByWeek(videos));
const { data: commentAndReactionData } = await axios({
url: "https://query.joystream.org/graphql",
method: "post",
data: {
query: `
query MyQuery {
${COMMENTS_AND_REACTIONS_QUERY}
}
`,
},
});
const {
data: {
commentsConnection: { totalCount: allTimeNumberOfComments },
commentReactionsConnection: { totalCount: allTimeNumberOfCommentReactions },
videoReactionsConnection: { totalCount: allTimeNumberOfVideoReactions },
comments,
commentReactions,
videoReactions,
},
} = commentAndReactionData;
console.log(
`All time number of comments and reactions: ${
allTimeNumberOfComments + allTimeNumberOfCommentReactions + allTimeNumberOfVideoReactions
}`
);
console.log(
separateDataByWeek(
[...comments, ...commentReactions, ...videoReactions].sort(
(a, b) => new Date(a.createdAt).getTime() - new Date(b.createdAt).getTime()
)
)
);
const { data: nftBoughtEventsData } = await axios({
url: "https://query.joystream.org/graphql",
method: "post",
data: {
query: `
query MyQuery {
${NFT_BOUGHT_EVENTS_QUERY}
}
`,
},
});
const {
data: { nftBoughtEvents },
} = nftBoughtEventsData;
console.log(
`Volume of sold NFTs (in JOY): ${hapiToJoy(
nftBoughtEvents.reduce((acc: number, curr: any) => acc + Number(curr.price), 0)
)}`
);
console.log(separateDataByWeekAndAmount(nftBoughtEvents));
const {
data: { data: generalBlockchainData },
} = await axios.post("https://joystream.api.subscan.io/api/scan/metadata", {
Headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
});
const {
data: {
data: { list: dailyActiveAccountData },
},
} = await axios.post(
"https://joystream.api.subscan.io/api/scan/daily",
{
category: "ActiveAccount",
start: "2023-12-26",
format: "day",
end: "2024-01-02",
},
{
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
}
);
const {
data: {
data: { list: dailyAccountHolderData },
},
} = await axios.post(
"https://joystream.api.subscan.io/api/scan/daily",
{
category: "AccountHolderTotal",
start: "2023-12-26",
format: "day",
end: "2024-01-02",
},
{
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
}
);
const {
data: {
data: { list: extrinsicData },
},
} = await axios.post(
"https://joystream.api.subscan.io/api/scan/daily",
{
category: "extrinsic",
start: "2022-01-01",
format: "day",
end: "2024-01-03",
},
{
headers: {
"Content-Type": "application/json",
"X-API-Key": "",
},
}
);
// Average block time
console.log(`Average block time: ${generalBlockchainData.avgBlockTime}`);
// Number of transactions
const totalNumberOfTransactions = extrinsicData.reduce(
(acc: number, curr: any) => acc + curr.total,
0
);
const totalNumberOfTransactionsAWeekAgo = extrinsicData
.slice(0, extrinsicData.length - 7)
.reduce((acc: number, curr: any) => acc + curr.total, 0);
console.log(`Number of transactions: ${totalNumberOfTransactions}`);
console.log(
`Number of transactions (WoW): ${
((totalNumberOfTransactions - totalNumberOfTransactionsAWeekAgo) /
totalNumberOfTransactionsAWeekAgo) *
100
}%`
);
// Number of holders
const currentDailyAccountHolders = dailyAccountHolderData[dailyAccountHolderData.length - 1].total;
const dailyAccountHoldersAWeekAgo = dailyAccountHolderData[0].total;
console.log(`Number of holders: ${currentDailyAccountHolders}`);
console.log(
`Holder Accounts Change (WoW): ${
((currentDailyAccountHolders - dailyAccountHoldersAWeekAgo) / dailyAccountHoldersAWeekAgo) * 100
}%`
);
// Daily Active Accounts
const currentDailyActiveAccounts = dailyActiveAccountData[dailyActiveAccountData.length - 1].total;
const dailyActiveAccountsAWeekAgo = dailyActiveAccountData[0].total;
console.log(`Daily Active Accounts: ${currentDailyActiveAccounts}`);
console.log(
`Daily Active Accounts Change (WoW): ${
((currentDailyActiveAccounts - dailyActiveAccountsAWeekAgo) / dailyActiveAccountsAWeekAgo) * 100
}%`
);Engineeringimport { Octokit } from "octokit";
const JOYSTREAM_ORG_NAME = "Joystream";
const getNumberOfItemsFromPageNumbers = (linkString: string | undefined) => {
const result = linkString
?.split(",")[1]
?.match(/&page=(\d+)/g)?.[0]
.replace(/&page=(\d+)/g, "$1");
return result ? parseInt(result) : 0;
};
const fetchPaginatedData = async (
url: string,
options: { per_page: number; [key: string]: any }
) => {
const data: any[] = [];
let page = 1;
while (true) {
const { data: pageData } = await octokit.request(url, {
page,
...options,
});
data.push(...pageData);
if (pageData.length < options.per_page) {
break;
}
page++;
}
return data;
};
const octokit = new Octokit({
auth: "",
});
const {
data: { public_repos, followers },
headers,
} = await octokit.request("GET /orgs/{org}", {
org: JOYSTREAM_ORG_NAME,
});
const { data: repoData } = await octokit.request("GET /orgs/{org}/repos", {
org: JOYSTREAM_ORG_NAME,
per_page: 1000,
});
const repos: string[] = repoData.map((repo: any) => repo.name);
const finalRepoInformation: any[] = [];
const finalcommitData: any = {};
let numberOfCommitsThisWeek = 0;
const allContributors: any = {};
for (const repo of repos) {
const { data } = await octokit.request("GET /repos/{username}/{repo}", {
username: JOYSTREAM_ORG_NAME,
repo,
});
const { headers: pullRequestHeaders } = await octokit.request("GET /repos/{owner}/{repo}/pulls", {
owner: JOYSTREAM_ORG_NAME,
repo,
per_page: 1,
page: 1,
});
const { headers: commitHeaders } = await octokit.request("GET /repos/{username}/{repo}/commits", {
username: JOYSTREAM_ORG_NAME,
repo,
per_page: 1,
page: 1,
});
const twoMonthsAgoDate = new Date();
const weekAgoDate = new Date();
twoMonthsAgoDate.setMonth(twoMonthsAgoDate.getMonth() - 2);
weekAgoDate.setDate(weekAgoDate.getDate() - 7);
// Fetch commits from the last 2 months
const since = twoMonthsAgoDate.toISOString();
const commitData = await fetchPaginatedData("GET /repos/{username}/{repo}/commits", {
username: JOYSTREAM_ORG_NAME,
repo,
per_page: 100,
since,
});
commitData.forEach((commit: any) => {
if (new Date(commit.commit.author.date) > weekAgoDate) {
numberOfCommitsThisWeek++;
}
const [year, month, day] = commit.commit.author.date.split("T")[0].split("-");
const currentYear = new Date().getFullYear().toString();
if (year !== currentYear) {
return;
}
if (!finalcommitData[month]) {
finalcommitData[month] = {};
}
if (!finalcommitData[month][day]) {
finalcommitData[month][day] = 0;
}
finalcommitData[month][day]++;
});
const { data: contributorData } = await octokit.request(
"GET /repos/{owner}/{repo}/contributors",
{
owner: JOYSTREAM_ORG_NAME,
repo,
per_page: 5000,
}
);
const numberOfPullRequests = getNumberOfItemsFromPageNumbers(pullRequestHeaders.link);
finalRepoInformation.push({
name: repo,
numberOfStars: data.stargazers_count,
numberOfCommits: getNumberOfItemsFromPageNumbers(commitHeaders.link),
numberOfOpenIssues: data.open_issues_count - numberOfPullRequests,
numberOfPullRequests,
});
contributorData.forEach((contributor) => {
if (contributor.login) {
if (allContributors[contributor.login]) {
allContributors[contributor.login].numberOfCommits += contributor.contributions;
} else {
allContributors[contributor.login] = {
numberOfCommits: contributor.contributions,
id: contributor.login,
avatar: contributor.avatar_url,
};
}
}
});
}
console.log(
`Total number of stars: ${finalRepoInformation.reduce(
(acc, curr) => acc + curr.numberOfStars,
0
)}`
);
console.log(
`Total number of commits: ${finalRepoInformation.reduce(
(acc, curr) => acc + curr.numberOfCommits,
0
)}`
);
console.log(`Number of commits this week: ${numberOfCommitsThisWeek}`);
console.log(
`Total number of open PRs: ${finalRepoInformation.reduce(
(acc, curr) => acc + curr.numberOfPullRequests,
0
)}`
);
console.log(
`Total number of open issues: ${finalRepoInformation.reduce(
(acc, curr) => acc + curr.numberOfOpenIssues,
0
)}`
);
console.log(`Number of public repos: ${public_repos}`);
console.log(`Number of followers: ${followers}`);
console.log(JSON.stringify(finalcommitData, null, 2));
const topContributors: any[] = Object.values(allContributors)
.sort((a: any, b: any) => b.numberOfCommits - a.numberOfCommits)
.slice(0, 21)
.filter((contributor: any) => contributor.id !== "actions-user");
for (let topContributor of topContributors) {
const { data: user } = await octokit.request("GET /users/{username}", {
username: topContributor.id,
});
topContributor.name = user.name;
}
console.log(JSON.stringify(topContributors, null, 2));Communityimport { REST, Routes } from "discord.js";
const API_KEY = "";
const BASE_URL = "";
const getTweetscoutLevel = (tweetscoutScore: number) => {
if (tweetscoutScore < 100) {
return 1;
} else if (tweetscoutScore < 500) {
return 2;
} else if (tweetscoutScore < 1000) {
return 3;
} else if (tweetscoutScore < 2000) {
return 4;
}
return 5;
};
const {
data: { score: tweetScoutScore },
} = await axios({
method: "get",
url: `${BASE_URL}/score/joystreamdao`,
headers: {
ApiKey: API_KEY,
},
});
console.log(`Tweetscout score is: ${tweetScoutScore}`);
console.log(`Tweetscout level is: ${getTweetscoutLevel(tweetScoutScore)}`);
const { data: topFollowers } = await axios({
method: "get",
url: `${BASE_URL}/top-followers/joystreamdao`,
headers: {
ApiKey: API_KEY,
},
});
console.log(`Top @JoystreamDao followers are: ${topFollowers}`);
const { data: twitterInfo } = await axios({
method: "get",
url: `${BASE_URL}/info/joystreamdao`,
headers: {
ApiKey: API_KEY,
},
});
console.log(twitterInfo);
const BOT_TOKEN = "";
const TEST_SERVER_GUILD_ID = "";
const rest = new REST({ version: "10" }).setToken(BOT_TOKEN);
// Users in a server
const oneMonthAgo = new Date();
oneMonthAgo.setMonth(oneMonthAgo.getMonth() - 1);
const users = (await rest.get(Routes.guildMembers(TEST_SERVER_GUILD_ID), {
query: new URLSearchParams([["limit", "1000"]]),
})) as any[];
const usersJoinedInLastMonth = users.filter((user) => new Date(user.joined_at) > oneMonthAgo);
const usersInLastMonth = users.length - usersJoinedInLastMonth.length;
const percentIncrease = (users.length - usersInLastMonth) / usersInLastMonth;
console.log(`Number of users in the server: ${users.length}`);
console.log(`Change in users in the last month: ${percentIncrease}%`);
// Scheduled events
const events = (await rest.get(Routes.guildScheduledEvents(TEST_SERVER_GUILD_ID))) as any[];
for (const event of events) {
console.log(`Event ${event.id} with name ${event.name} starts at ${event.scheduled_start_time}`);
}
const TELEGRAM_BOT_ID="";
const {
data: { result },
} = await axios.get(
`https://api.telegram.org/bot${TELEGRAM_BOT_ID}/getChatMembersCount?chat_id=@JoystreamOfficial`
);
console.log(`Number of users in JoystreamDAO channel: ${result}`);Teamimport { GraphQLClient, gql } from "graphql-request";
const COUNCIL_QUERY = `
electionRounds(limit: 2, orderBy: cycleId_DESC) {
cycleId
endedAtTime
}
councilMembers(limit: 3, orderBy: updatedAt_DESC) {
member {
handle
metadata {
avatar {
...on AvatarUri {
avatarUri
}
}
externalResources {
type
value
}
}
councilMembers {
id
}
}
}
`;
const WORKERS_QUERY = `
workingGroups {
id
budget
workers {
isActive
isLead
membership {
handle
metadata {
avatar {
...on AvatarUri {
avatarUri
}
}
}
}
}
}
`;
const { data: councilData } = await axios({
url: "https://query.joystream.org/graphql",
method: "post",
data: {
query: `
query MyQuery {
${COUNCIL_QUERY}
}
`,
},
});
const {
data: { councilMembers, electionRounds },
} = councilData;
const currentCouncilMembers = councilMembers.map((cm: any) => ({
avatar: cm.member.metadata.avatar?.avatarUri,
handle: cm.member.handle,
socials: cm.member.metadata.externalResources,
timesServed: cm.member.councilMembers.length,
}));
// const councilIdlePeriodDuration = await api.const.council.idlePeriodDuration();
// const voteStageDuration = await api.const.referendum.voteStageDuration();
// const revealStageDuration = await api.const.referendum.revealStageDuration();
// const announcingPeriodDuration = await api.const.council.announcingPeriodDuration();
const combinedLengthOfCouncilStages = 1 + 43200 + 43200 + 129600;
const approximatedLengthInSeconds = combinedLengthOfCouncilStages * 6;
const startOfElectionRound = new Date(electionRounds[1].endedAtTime);
const endOfElectionRound = new Date(electionRounds[1].endedAtTime);
endOfElectionRound.setSeconds(startOfElectionRound.getSeconds() + approximatedLengthInSeconds);
console.log("Current term: ", electionRounds[1].cycleId);
console.log("Council term length (in days): ", approximatedLengthInSeconds / (60 * 60 * 24));
console.log("Start of election round:", startOfElectionRound.toLocaleString());
console.log("End of election round:", endOfElectionRound.toLocaleString());
const councilorReward = await api.query.council.councilorReward();
const councilorWeeklyRewardInJOY = api.toJOY(new BN(councilorReward)) * BLOCKS_IN_A_WEEK;
console.log("Weekly councilor salary in JOY:", councilorWeeklyRewardInJOY);
console.log(currentCouncilMembers);
const query = gql`{
${WORKERS_QUERY}
}`;
const client = new GraphQLClient("https://query.joystream.org/graphql");
const { workingGroups } = await client.request(query);
const workingGroupsData = workingGroups.reduce((acc: any, wg: any) => {
acc[wg.id] = {
workers: wg.workers
.filter((w: any) => w.isActive)
.map((w: any) => ({
handle: w.membership.handle,
isLead: w.isLead,
avatar: w.membership.metadata.avatar?.avatarUri,
})),
budget: hapiToJoy(Number(wg.budget)),
};
return acc;
}, {});
console.log(workingGroupsData);API Rate LimitingThe following list explains the amount of rate limiting for each of the APIs above. Some have rate limiting for specific endpoints and some are global. The list:
ConclusionImplementation: Since the decision was made to go with the simpler approach, the implementation will fall very much in line with what is already on the status server. This time around there is more APIs to keep track off and more data to fetch so it will necessitate a bit of a cleaner approach in terms of code architecture. That being said, the implementation won't steer far away from previous ones. Some things to note still:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Introduction
This issue is the first step of the process for the implementation of the dashboard-api, specified here: #1
The scope and necessary APIs have been extracted and inferred from the original issue but also, more details has been given to the APIs specified in the designs and comments that can be found here: Joystream/joystream-org#650
Based on the previous information, I have arrived at the following scope for the initial version of the dashboard-api:
In front of all of the potential data points, I've gone ahead and classified them into dynamic and static data points. The dynamic data points are ones that will find themselves in the API and the static ones will be statically integrated into the final dashboard page.
Code
Community
Twitter
Discord
Telegram
Tweetscout
GitHub
Token
Traction
Team
API Rate Limiting
The following list explains the amount of rate limiting for each of the APIs above. Some have rate limiting for specific endpoints and some are global.
The list:
GET /2/users/me-25req/24hours50req/sec30req/sec30req/sec.10,000req5000req/hour30req/minutebut there is a hard monthly limit of10,000req.20req/sec20req/sec5req/secQuestions
These are the questions that arose during research which I wasn't sure about how to resolve:
Notes
The text was updated successfully, but these errors were encountered: