-
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bringing Crossref, Semantic Scholar, Open Citations and Open Alex lookup + auto-import to Cita for Zotero 7 #300
Open
thebluepotato
wants to merge
30
commits into
diegodlh:zotero7
Choose a base branch
from
thebluepotato:zotero7
base: zotero7
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 28 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
806682a
Bringing Crossref lookup and auto-import to Cita for Zotero 7
thebluepotato bceb456
Updated duplicate detection logic
thebluepotato 185caa8
General cleanup and fixing type errors
thebluepotato 92dde74
Added support for Semantic Scholar and Open Alex as well
thebluepotato a35bf29
Fixes to translations
thebluepotato 0e621a1
Expand Crossref types
thebluepotato 2aba06d
Fixed item submenu labels
thebluepotato 043b859
Commit patch to openalex-sdk (avoiding fs dependency)
thebluepotato 6c1c361
Various fixes
thebluepotato 24b6c48
Merge branch 'zotero7' into zotero7
thebluepotato cfa211c
Update package-lock.json
thebluepotato ee5212a
Improve prompts with citation counts
thebluepotato f9bdb16
Added support for OpenCitatations and further refactored
thebluepotato 70d20aa
Fix menu naming
thebluepotato 3fca65f
Merge branch 'zotero7' into zotero7
thebluepotato 7bc48c8
Preliminary work to expand PIDType
thebluepotato 11016e2
Expand PIDType, simplify Indexer logic and cleanup
thebluepotato f97bffa
Fix submenu enabled/disabled
thebluepotato 1c420b5
Add support for fetching OMID
thebluepotato 612f737
Add ability to wetch OpenAlex work ID
thebluepotato 9de4d25
Remove `declare const Services` because we have it defined in zotero-…
Dominic-DallOsto b4d3294
Make PID rows fit new labels
Dominic-DallOsto dfabd20
Disable fetch PID button if fetching isn't implemented for that PID
Dominic-DallOsto 5882332
Preliminary support for showing fetch progress
thebluepotato 62d390a
Slightly reduce width of PID row labels now that they're not uppercase
Dominic-DallOsto 9ccce0f
Don't show PMID or PMCID PID rows because we can't fetch citations fr…
Dominic-DallOsto 450610f
Implemented DOI fetching (Crossref only)
thebluepotato 692e6e7
Localise fetch button tooltip in pid rows
Dominic-DallOsto 7866b30
Make a DOI type
Dominic-DallOsto a61595e
Update DOI type in sourceItemWrapper
Dominic-DallOsto File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
diff --git a/node_modules/openalex-sdk/dist/src/index.js b/node_modules/openalex-sdk/dist/src/index.js | ||
index 70b2df5..518d455 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/index.js | ||
+++ b/node_modules/openalex-sdk/dist/src/index.js | ||
@@ -3,7 +3,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
return (mod && mod.__esModule) ? mod : { "default": mod }; | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
const authors_1 = require("./utils/authors"); | ||
const exportCSV_1 = require("./utils/exportCSV"); | ||
const helpers_1 = require("./utils/helpers"); | ||
diff --git a/node_modules/openalex-sdk/dist/src/utils/authors.js b/node_modules/openalex-sdk/dist/src/utils/authors.js | ||
index 0b1cbc5..f824766 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/utils/authors.js | ||
+++ b/node_modules/openalex-sdk/dist/src/utils/authors.js | ||
@@ -4,7 +4,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
exports.handleAllAuthorsPages = exports.handleMultipleAuthorsPages = exports.validateAuthorParameters = void 0; | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
const exportCSV_1 = require("./exportCSV"); | ||
const helpers_1 = require("./helpers"); | ||
const http_1 = require("./http"); | ||
diff --git a/node_modules/openalex-sdk/dist/src/utils/exportCSV.js b/node_modules/openalex-sdk/dist/src/utils/exportCSV.js | ||
index 7f4d748..9c01df5 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/utils/exportCSV.js | ||
+++ b/node_modules/openalex-sdk/dist/src/utils/exportCSV.js | ||
@@ -4,7 +4,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
exports.convertToCSV = void 0; | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
function flattenObject(obj, parentKey = '', depth = 0) { | ||
const flattened = {}; | ||
for (const [key, value] of Object.entries(obj)) { | ||
diff --git a/node_modules/openalex-sdk/dist/src/utils/institutions.js b/node_modules/openalex-sdk/dist/src/utils/institutions.js | ||
index 1082435..099275a 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/utils/institutions.js | ||
+++ b/node_modules/openalex-sdk/dist/src/utils/institutions.js | ||
@@ -4,7 +4,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
exports.handleAllInstitutionsPages = exports.handleMultipleInstitutionsPages = exports.validateInstitutionsParameters = void 0; | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
const exportCSV_1 = require("./exportCSV"); | ||
const helpers_1 = require("./helpers"); | ||
const http_1 = require("./http"); | ||
diff --git a/node_modules/openalex-sdk/dist/src/utils/sources.js b/node_modules/openalex-sdk/dist/src/utils/sources.js | ||
index 699da8d..38c68b8 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/utils/sources.js | ||
+++ b/node_modules/openalex-sdk/dist/src/utils/sources.js | ||
@@ -4,7 +4,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
exports.handleAllSourcesPages = exports.handleMultipleSourcesPages = exports.validateSourcesParameters = void 0; | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
const exportCSV_1 = require("./exportCSV"); | ||
const helpers_1 = require("./helpers"); | ||
const http_1 = require("./http"); | ||
diff --git a/node_modules/openalex-sdk/dist/src/utils/topics.js b/node_modules/openalex-sdk/dist/src/utils/topics.js | ||
index b2946cc..b9fd0ed 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/utils/topics.js | ||
+++ b/node_modules/openalex-sdk/dist/src/utils/topics.js | ||
@@ -4,7 +4,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
exports.handleAllTopicsPages = exports.handleMultipleTopicsPages = exports.validateTopicsParameters = void 0; | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
const exportCSV_1 = require("./exportCSV"); | ||
const helpers_1 = require("./helpers"); | ||
const http_1 = require("./http"); | ||
diff --git a/node_modules/openalex-sdk/dist/src/utils/works.js b/node_modules/openalex-sdk/dist/src/utils/works.js | ||
index 46d5d38..2a5591c 100644 | ||
--- a/node_modules/openalex-sdk/dist/src/utils/works.js | ||
+++ b/node_modules/openalex-sdk/dist/src/utils/works.js | ||
@@ -4,7 +4,7 @@ var __importDefault = (this && this.__importDefault) || function (mod) { | ||
}; | ||
Object.defineProperty(exports, "__esModule", { value: true }); | ||
exports.formatNumber = exports.handleAllPagesInChunks = exports.handleMultiplePages = exports.handleAllPages = exports.validateParameters = void 0; | ||
-const fs_1 = __importDefault(require("fs")); | ||
+//const fs_1 = __importDefault(require("fs")); | ||
const exportCSV_1 = require("./exportCSV"); | ||
const helpers_1 = require("./helpers"); | ||
const http_1 = require("./http"); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,17 +1,229 @@ | ||
/* eslint-disable @typescript-eslint/no-unsafe-member-access */ | ||
/* eslint-disable @typescript-eslint/no-unsafe-call */ | ||
import Wikicite from "./wikicite"; | ||
|
||
declare const Services: any; | ||
|
||
export default class Crossref { | ||
static getCitations() { | ||
Services.prompt.alert( | ||
window, | ||
Wikicite.getString("wikicite.global.unsupported"), | ||
Wikicite.getString("wikicite.crossref.get-citations.unsupported"), | ||
import { IndexerBase, IndexedWork, LookupIdentifier } from "./indexer"; | ||
import ItemWrapper from "./itemWrapper"; | ||
import Wikicite, { debug } from "./wikicite"; | ||
import Lookup from "./zotLookup"; | ||
|
||
interface CrossrefResponse { | ||
status: string; | ||
"message-type": string; | ||
"message-version": string; | ||
message: CrossrefWork; | ||
} | ||
|
||
interface CrossrefWork { | ||
"reference-count": number; | ||
reference: Reference[]; | ||
"references-count": number; | ||
} | ||
|
||
interface Reference { | ||
key: string; | ||
issn?: string; | ||
"standards-body"?: string; | ||
"series-title"?: string; | ||
"isbn-type"?: string; | ||
"doi-asserted-by"?: string; | ||
DOI?: string; | ||
ISBN?: string; | ||
component?: string; | ||
"article-title"?: string; | ||
"volume-title"?: string; | ||
author?: string; | ||
year?: string; | ||
unstructured?: string; | ||
issue?: string; | ||
"first-page"?: string; | ||
volume?: string; | ||
"journal-title"?: string; | ||
edition?: string; | ||
"standard-designator"?: string; | ||
"issn-type"?: string; | ||
} | ||
|
||
function mapCrossrefWorkToIndexedWork( | ||
work: CrossrefWork, | ||
): IndexedWork<Reference> { | ||
return { | ||
referenceCount: work["reference-count"], // Map Crossref's `reference-count` to `IndexedWork`'s `referenceCount` | ||
referencedWorks: work.reference, // Map `reference` to `referencedWorks` | ||
}; | ||
} | ||
|
||
export default class Crossref extends IndexerBase<Reference> { | ||
indexerName = "Crossref"; | ||
|
||
supportedPIDs: PIDType[] = ["DOI"]; | ||
|
||
async fetchDOI(item: ItemWrapper): Promise<string | null> { | ||
const crossrefOpenURL = | ||
"https://doi.crossref.org/[email protected]&"; | ||
const ctx = Zotero.OpenURL.createContextObject(item, "1.0"); | ||
|
||
if (ctx) { | ||
const url = crossrefOpenURL + ctx + "&multihit=true"; | ||
const response = await Zotero.HTTP.request("GET", url).catch( | ||
(e) => { | ||
debug( | ||
`Couldn't access URL: ${url}. Got status ${e.status}.`, | ||
); | ||
}, | ||
); | ||
|
||
const xml = response?.responseXML; | ||
|
||
if (xml) { | ||
const status = xml | ||
.getElementsByTagName("query")[0] | ||
.getAttribute("status"); | ||
switch (status) { | ||
case "resolved": | ||
case "multiresolved": { | ||
// We just take the first one | ||
const doi = | ||
xml.getElementsByTagName("doi")[0].textContent; | ||
return doi; | ||
} | ||
case "unresolved": | ||
return null; | ||
default: | ||
throw new Error(`Unexpected status: ${status}`); | ||
} | ||
} | ||
} | ||
return null; | ||
} | ||
|
||
/** | ||
* Get a list of references from Crossref for an item with a certain DOI. | ||
* Returned in JSON Crossref format. | ||
* @param {string[]} identifiers - DOI for the item for which to get references. | ||
* @returns {Promise<IndexedWork<Reference>[]>} list of references, or [] if none. | ||
*/ | ||
async getReferences( | ||
identifiers: LookupIdentifier[], | ||
): Promise<IndexedWork<Reference>[]> { | ||
// Crossref-specific logic for fetching references | ||
const requests = identifiers.map(async (doi) => { | ||
const url = `https://api.crossref.org/works/${Zotero.Utilities.cleanDOI(doi.id)}`; | ||
const options = { | ||
headers: { | ||
"User-Agent": `${Wikicite.getUserAgent()} mailto:[email protected]`, | ||
}, | ||
responseType: "json", | ||
}; | ||
const response = await Zotero.HTTP.request( | ||
"GET", | ||
url, | ||
options, | ||
).catch((e) => { | ||
debug(`Couldn't access URL: ${url}. Got status ${e.status}.`); | ||
if (e.status == 429) { | ||
// Extract rate limit headers | ||
const rateLimitLimit = | ||
e.xmlhttp.getResponseHeader("X-Rate-Limit-Limit"); | ||
const rateLimitInterval = e.xmlhttp.getResponseHeader( | ||
"X-Rate-Limit-Interval", | ||
); | ||
|
||
throw new Error( | ||
"Received a 429 rate limit response from Crossref (https://github.com/CrossRef/rest-api-doc#rate-limits). Try getting references for fewer items at a time. Rate limits in action: Limit: ${rateLimitLimit}, Interval: ${rateLimitInterval}", | ||
); | ||
} | ||
}); | ||
|
||
const crossrefWork = (response?.response as CrossrefResponse) | ||
.message; | ||
return mapCrossrefWorkToIndexedWork(crossrefWork); // Map to IndexedWork<Reference> | ||
}); | ||
return Promise.all(requests); | ||
} | ||
|
||
/** | ||
* Parse a list of references in JSON Crossref format. | ||
* @param {Reference[]} references - Array of Crossref references to parse to Zotero items. | ||
* @returns {Promise<Zotero.Item[]>} Zotero items parsed from references (where parsing is possible). | ||
*/ | ||
async parseReferences(references: Reference[]): Promise<Zotero.Item[]> { | ||
// Crossref-specific parsing logic | ||
// Extract one identifier per reference (prioritising DOI) and filter out those without identifiers | ||
const _identifiers = references | ||
.map((ref) => ref.DOI ?? ref.ISBN ?? null) | ||
.filter((e) => e !== null); | ||
// Remove duplicates and extract identifiers | ||
const identifiers = [...new Set(_identifiers)].flatMap((e) => | ||
Zotero.Utilities.extractIdentifiers(e!), | ||
); | ||
const crossrefReferencesWithoutIdentifier = references.filter( | ||
(item) => !item.DOI && !item.ISBN, | ||
); | ||
|
||
// Use Lookup to get items for all identifiers | ||
const result = await Lookup.lookupItemsByIdentifiers(identifiers); | ||
const parsedReferences = result ? result : []; | ||
|
||
// Manually create items for references without identifiers | ||
const manualResult = await Promise.allSettled( | ||
crossrefReferencesWithoutIdentifier.map((item) => | ||
this.parseItemFromCrossrefReference(item), | ||
), | ||
); | ||
const parsedReferencesWithoutIdentifier = manualResult | ||
.filter( | ||
(ref): ref is PromiseFulfilledResult<Zotero.Item> => | ||
ref.status === "fulfilled", | ||
) // Only keep fulfilled promises | ||
.map((ref) => ref.value); // Extract the `value` from fulfilled promises; | ||
parsedReferences.push(...parsedReferencesWithoutIdentifier); | ||
|
||
return parsedReferences; | ||
} | ||
|
||
static getDOI() {} | ||
/** | ||
* Create a Zotero Item from a Crossref reference item that doesn't include an identifier. | ||
* @param {Reference} crossrefItem - A reference item in JSON Crossref format. | ||
* @returns {Promise<Zotero.Item>} Zotero item parsed from the identifier, or null if parsing failed. | ||
*/ | ||
async parseItemFromCrossrefReference( | ||
crossrefItem: Reference, | ||
): Promise<Zotero.Item> { | ||
//Zotero.log(`Parsing ${crossrefItem.unstructured}`); | ||
const jsonItem: any = {}; | ||
if (crossrefItem["journal-title"]) { | ||
jsonItem.itemType = "journalArticle"; | ||
jsonItem.title = | ||
crossrefItem["article-title"] || crossrefItem["volume-title"]; | ||
jsonItem.publicationTitle = crossrefItem["journal-title"]; | ||
} else if (crossrefItem["volume-title"]) { | ||
jsonItem.itemType = "book"; | ||
jsonItem.title = crossrefItem["volume-title"]; | ||
} else if (crossrefItem.unstructured) { | ||
// todo: Implement reference text parsing here | ||
throw new Error( | ||
"Couldn't parse Crossref reference - unstructured references are not yet supported. " + | ||
JSON.stringify(crossrefItem), | ||
); | ||
} else { | ||
throw new Error( | ||
"Couldn't determine type of Crossref reference - doesn't contain `journal-title` or `volume-title` field. " + | ||
JSON.stringify(crossrefItem), | ||
); | ||
} | ||
jsonItem.date = crossrefItem.year; | ||
jsonItem.pages = crossrefItem["first-page"]; | ||
jsonItem.volume = crossrefItem.volume; | ||
jsonItem.issue = crossrefItem.issue; | ||
jsonItem.creators = crossrefItem.author | ||
? [Zotero.Utilities.cleanAuthor(crossrefItem.author, "author")] | ||
: []; | ||
// remove undefined properties | ||
for (const key in jsonItem) { | ||
if (jsonItem[key] === undefined) { | ||
delete jsonItem[key]; | ||
} | ||
} | ||
|
||
const newItem = new Zotero.Item(jsonItem.itemType); | ||
newItem.fromJSON(jsonItem); | ||
return newItem; | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These could maybe be capitalised too?