Skip to content

Commit

Permalink
nodeinfo, mit license, list only misskey
Browse files Browse the repository at this point in the history
  • Loading branch information
@tamaina
tamaina committed Dec 25, 2022
1 parent 0f8e2d3 commit 6c75efc
Show file tree
Hide file tree
Showing 4 changed files with 184 additions and 58 deletions.
7 changes: 7 additions & 0 deletions LICENSE
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Copyright 2022 aqz/tamaina

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
64 changes: 33 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,63 +3,65 @@ joinmisskey instances' information api

https://instanceapp.misskey.page/instances.json

**This API doesn't follow forks that say `nodeinfo.software.name !== 'misskey'`.**

## Build Environment
2つの環境変数を設定してください。
You must set following two envs.

- `LB_TOKEN`: GitHubのトークン（GitHub情報取得用）
- `MK_TOKEN`: Misskeyのトークン（Misskey投稿用）
- `LB_TOKEN`= GitHub Token (to get versions)
- `MK_TOKEN`= Misskey Token（to post to misskey）

## Endpoints
nginxおよびCloudflareで静的ファイルを配信しているだけですので、アクセス制限は設けていません。
We are only serving static files via nginx and Cloudflare, so we have no access restrictions.

https://instanceapp.misskey.page 下で以下の情報を取得できます。
You can get the following information under https://instanceapp.misskey.page

### /instances.json
インスタンス情報一覧のjsonです。

```
{
date: Date // instances.json発行日時
stats: { // 統計
notesCount: Number, // 総ノート数
usersCount: Number, // 総ユーザー数
instancesCount: Number, // 稼働インスタンス数
date: Date // The date instances.json was published at.
stats: { // statistics
notesCount: Number, // Total notes
usersCount: Number, // Total Users
mau: Number, // Total MAUs
instancesCount: Number, // Instances counter
},
instancesInfos: [ // インスタンス一覧（※稼働中のみ）
instancesInfos: [ // Instances Infos (only alives)
{
url: String, // ホスト名 e.g. misskey.io
langs: String[], // インスタンスリストでaqzが登録した言語 e.g. ["ja"], ["zh"]
"description": String | Null, // meta.description、なければaqzが設定した説明が入るかもしれない
"isAlive": true, // 稼働中のみ掲載なので、つねにtrue
value: Number, // バージョン等から算定したインスタンスバリュー
meta: Object, // api/metaの結果 ※announcementsは削除されています
stats: Object, // api/statsの結果
banner: Bool, // バナーが存在するかどうか
background: Bool,// バックグラウンドイメージがあるかどうか
icon: Bool, // アイコンがあるかどうか
url: String, // Hostname e.g. misskey.io
name: String, // Name e.g. すしすきー
langs: String[], // Language the API author aqz set manually e.g. ["ja"], ["zh"]
description: String | Null, // meta.description or the the API author aqz set manually
isAlive: true, // must true
value: Number, // The Instance Value calculated from the version, etc.
banner: Bool, // Banner existance
background: Bool,// Background Image existance
icon: Bool, // Icon Image existance
nodeinfo: Object | null, // nodeinfo
meta: Object | null, // result of api/meta
stats: Object, // deprecated (result of api/stats)
}, ...
]
}
```

### /instance-banners/instance.host.{jpeg|webp}
軽量化されたインスタンスのバナーが格納されています。
存在するかどうかはinstancesInfosのbannerで確認できます。
Banner of each instances (lightweighted)

### /instance-backgrounds/instance.host.{jpeg|webp}
軽量化されたインスタンスのバックグラウンドイメージ（ウェルカムページに表示される画像）が格納されています。
存在するかどうかはinstancesInfosのbackgroundで確認できます。
Background image (displayed behind the welcome page) (lightweighted)

### /instance-icons/instance.host.{png|webp}
軽量化されたインスタンスのアイコン（faviconではありません）が格納されています。
存在するかどうかはinstancesInfosのiconで確認できます。
Icon (not favicon) (lightweighted)

### /alives.txt
疎通できたインスタンスのホストのリスト（\n区切り）
List of hosts (separated by `\n`) for instances that were able to communicate

### /deads.txt
疎通不能だったインスタンスのホストのリスト（\n区切り）
List of hosts (separated by `\n`) for instances that were unable to communicate

### versions.json
GitHubから取得した各リポジトリのバージョンリスト
Version list obtained from GitHub
157 changes: 135 additions & 22 deletions getInstancesInfos.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,17 @@ const instances = loadyaml("./data/instances.yml")

const pqueue = new Queue(32)

function safePost(url, options)/*: Promise<Response | null | false | undefined>*/ {
const ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0";

function safeFetch(method, url, options)/*: Promise<Response | null | false | undefined>*/ {
const controller = new AbortController()
const timeout = setTimeout(
() => { controller.abort() },
30000
)
const start = performance.now();
// glog("POST start", url)
return fetch(url, extend(true, options, { method: "POST", signal: controller.signal })).then(
return fetch(url, extend(true, options, { method, signal: controller.signal })).then(
res => {
if (res && res.ok) {
const end = performance.now();
Expand All @@ -42,7 +44,7 @@ function safePost(url, options)/*: Promise<Response | null | false | undefined>*
})
}

async function postJson(url, json) {
async function fetchJson(method, url, json) {
const option = {
body: JSON.stringify(json ? json : {}),
headers: {
Expand All @@ -54,10 +56,10 @@ async function postJson(url, json) {

let retryCount = 0;

while (retryCount < 3) {
while (retryCount < 2) {
if (retryCount > 0) glog('retry', url, retryCount);
await new Promise(resolve => retryCount > 0 ? setTimeout(resolve, 60000) : resolve());
const res = await safePost(url, option)
await new Promise(resolve => retryCount > 0 ? setTimeout(resolve, 20000) : resolve());
const res = await safeFetch(method, url, option)
.then(res => {
if (res === null) return null;
if (!res) return false;
Expand All @@ -75,17 +77,105 @@ async function postJson(url, json) {
return false;
}

async function getNodeinfo(base)/*: Promise<Response | null | false | undefined>*/ {
const controller = new AbortController()
const timeout = setTimeout(
() => { controller.abort() },
30000
)

const wellnownUrl = `https://${base}/.well-known/nodeinfo`;

const wellknown = await fetch(wellnownUrl, {
method: "GET",
headers: {
"User-Agent": ua,
},
redirect: "error",
signal: controller.signal
}).then(res => {
if (res && res.ok) {
glog("Get WellKnown Nodeinfo finish", wellnownUrl, res.status, res.ok)
return res.json();
}
return;
}).catch(async e => {
glog("Get WellKnown Nodeinfo failed...", wellnownUrl, e.errno, e.type)
return;
}).finally(() => {
clearTimeout(timeout);
});

if (wellknown.links == null || !Array.isArray(wellknown.links)) {
glog("WellKnown Nodeinfo was Not Array", wellnownUrl, wellknown);
return null;
}

const links = wellknown.links;

const lnik1_0 = links.find(link => link.rel === 'http://nodeinfo.diaspora.software/ns/schema/1.0');
const lnik2_0 = links.find(link => link.rel === 'http://nodeinfo.diaspora.software/ns/schema/2.0');
const lnik2_1 = links.find(link => link.rel === 'http://nodeinfo.diaspora.software/ns/schema/2.1');
const link = lnik2_1 ?? lnik2_0 ?? lnik1_0;

if (link == null || typeof link !== 'object') {
glog("Nodeinfo Link was Null", wellnownUrl);
return null;
}

const controller2 = new AbortController()
const timeout2 = setTimeout(
() => { controller2.abort() },
30000
)

const info = await fetch(link.href, {
method: "GET",
headers: {
"User-Agent": ua,
},
redirect: "error",
signal: controller2.signal
}).then(res => {
if (res && res.ok) {
glog("Get Nodeinfo finish", link.href, res.status, res.ok)
return res.json();
}
return;
}).catch(async e => {
glog("Get Nodeinfo failed...", link.href, e.errno, e.type)
if (e.errno?.toLowerCase().includes('timeout') || e.type === 'aborted') return null;
return;
}).finally(() => {
clearTimeout(timeout2);
})

return info;
}

async function safeGetNodeInfo(base) {
const retry = (timeout) => new Promise((res, rej) => {
setTimeout(() => {
getNodeinfo(base).then(res, rej)
}, timeout)
});
return getNodeinfo(base)
.then(res => res === undefined ? retry(10000) : res)
.catch(e => retry(10000))
.catch(() => null);
}

// misskey-dev/misskeyを最後に持っていくべし
const ghRepos = [
"mei23/misskey",
"mei23/misskey-v11",
//"mei23/misskey",
//"mei23/misskey-v11",
//"kokonect-link/cherrypick",
"misskey-dev/misskey"
];

const gtRepos = [
"codeberg.org/thatonecalculator/calckey",
"akkoma.dev/FoundKeyGang/FoundKey",
//"codeberg.org/thatonecalculator/calckey",
//"akkoma.dev/FoundKeyGang/FoundKey",
]

module.exports.ghRepos = ghRepos;
Expand All @@ -100,6 +190,7 @@ function hasVulnerability(repo, version) {
//semver.satisfies(version, '< 12.51.0') ||
semver.satisfies(version, '>= 10.46.0 < 10.102.4 || >= 11.0.0-alpha.1 < 11.20.2')
);
/*
case 'mei23/misskey':
return (
semver.satisfies(version, '< 10.102.608-m544') ||
Expand All @@ -114,6 +205,7 @@ function hasVulnerability(repo, version) {
return (
semver.satisfies(version, '< v13.0.0-preview3')
);
*/
default:
return false;
}
Expand All @@ -124,7 +216,6 @@ async function getVersions() {
const maxRegExp = /<https:\/\/.*?>; rel="next", <https:\/\/.*?\?page=(\d+)>; rel="last"/;
const versions = new Map();
const versionOutput = {};
const ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0";

const vqueue = new Queue(3)

Expand Down Expand Up @@ -207,26 +298,38 @@ module.exports.getInstancesInfos = async function() {
glog("Getting Instances' Infos")

const promises = [];
const alives = [], deads = [];
const alives = [], deads = [], notMisskey = [], outdated = [];

const { versions, versionOutput } = await getVersions()

// eslint-disable-next-line no-restricted-syntax
for (let t = 0; t < instances.length; t += 1) {
const instance = instances[t]
promises.push(pqueue.add(async () => {
const meta = (await postJson(`https://${instance.url}/api/meta`)) || false;
const stat = (await postJson(`https://${instance.url}/api/stats`)) || false;
const NoteChart = (await postJson(`https://${instance.url}/api/charts/notes`, { span: "day" })) || false;
const nodeinfo = (await safeGetNodeInfo(instance.url)) || null;

if (meta && stat && NoteChart) {
delete meta.emojis;
delete meta.announcements;
if (nodeinfo && nodeinfo.software.name !== "misskey") {
notMisskey.push({
nodeinfo,
...instance
});
return;
}

const meta = (await fetchJson('POST', `https://${instance.url}/api/meta`)) || null;
const stat = (await fetchJson('POST', `https://${instance.url}/api/stats`)) || null;
const NoteChart = (await fetchJson('POST', `https://${instance.url}/api/charts/notes`, { span: "day" })) || null;

if (nodeinfo && meta && stat && NoteChart) {
if (meta) {
delete meta.emojis;
delete meta.announcements;
}

const versionInfo = (() => {
const sem1 = semver.clean(meta.version, { loose: true })
const sem1 = semver.clean(nodeinfo.software.version, { loose: true })
if (versions.has(sem1)) return { just: true, ...versions.get(sem1) };
const sem2 = semver.valid(semver.coerce(meta.version))
const sem2 = semver.valid(semver.coerce(nodeinfo.software.version))
let current = { repo: 'misskey-dev/misskey', count: 1500 };
for (const [key, value] of versions.entries()) {
if (sem1 && sem1.startsWith(key)) {
Expand All @@ -240,7 +343,13 @@ module.exports.getInstancesInfos = async function() {
return current
})()

if (versionInfo.just && versionInfo.hasVulnerability) return;
if (versionInfo.just && versionInfo.hasVulnerability) {
outdated.push({
nodeinfo,
...instance,
});
return;
};

/* インスタンスバリューの算出 */
let value = 0
Expand All @@ -258,8 +367,10 @@ module.exports.getInstancesInfos = async function() {
alives.push(extend(true, instance, {
value,
meta,
nodeinfo,
stats: stat,
description: meta.description || (instance.description || null),
name: instance.name || nodeinfo.metadata.nodeName || meta.name || instance.url,
description: nodeinfo.metadata.nodeDescription || meta.description || (instance.description || null),
langs: instance.langs || ['ja', 'en', 'de', 'fr', 'zh', 'ko', 'ru', 'th', 'es'],
isAlive: true,
repo: versionInfo?.repo
Expand All @@ -283,6 +394,8 @@ module.exports.getInstancesInfos = async function() {
return {
alives: alives.sort((a, b) => (b.value || 0) - (a.value || 0)),
deads,
notMisskey,
outdated,
versions,
versionOutput,
}
Expand Down
Loading

0 comments on commit 6c75efc

Please sign in to comment.