TEI translator full rewrite #3245

glorieux-f · 2024-01-30T13:38:26Z

The present TEI exporter in Zotero is old, 15 years old, without improvements. Even if the new exporter is not yet perfect, it is a lot better than the old one. All comments of reviewers have been handled.

New features

Rich html text in notes exported as TEI
Rich text in titles, more Zotero conformant
Extra field parsed for extra CSL properties
XML/TEI indented
More conformant TEI

Best is to see how is handled a quite complex item.

{
	"version": 3038
	"itemType": "journalArticle"
	"title": "Not closing <b> or<strong>unknown tag</strong>, kept as text nodes; <i>italic</i>, is an element"
	"date": "1929"
	"language": "fr"
	"callNumber": "piaget1929a02"
	"extra": "Start of text note.\nGenre: Review of\nReviewed Title: Rich text >> <i>italic</i>\nMedium: Paris, <Payot>, 1927\nUnknown CSL property: CLA1929_5\nNote continuing.\nEditorial Director: One || John\nEditorial Director: Two || John"
	"volume": "22"
	"pages": "117-118"
	"publicationTitle": "Archives de psychologie"
	"issue": "85"
	"creators": [
		"0": {
			"firstName": "Jean"
			"lastName": "Piaget"
			"creatorType": "author"
		}
		"1": {
			"name": "Eugène Minkowski"
			"creatorType": "reviewedAuthor"
		}
		"2": {
			"firstName": "J."
			"lastName": "Contributor"
			"creatorType": "contributor"
		}
		"3": {
			"firstName": "M."
			"lastName": "Editor"
			"creatorType": "editor"
		}
	]
	"tags": [
		"0": {
			"tag": "AddATag"
		}
	]
	"collections": [
		"0": "U8JSTJZ5"
	]
	"relations": {
		"owl:sameAs": [
			"0": "http://zotero.org/groups/5048422/items/G99GTU5D"
		]
	}
	"dateAdded": "2024-01-29T19:58:13Z"
	"dateModified": "2024-01-30T12:55:46Z"
	"uri": "http://zotero.org/users/8989645/items/RLRXRWYM"
	"attachments": []
	"notes": [
		"0": {
			"key": "6SE87GSB"
			"version": 3020
			"itemType": "note"
			"parentItem": "RLRXRWYM"
			"note": "<div data-schema-version=\"8\"><p>A zotero note with some <em>formatting</em>.</p>\n<ol>\n<li>\nHate Numbered List\n</li>\n<li>\nGo to 1.\n</li>\n</ol>\n</div>"
			"tags": []
			"relations": {}
			"dateAdded": "2024-01-30T12:10:49Z"
			"dateModified": "2024-01-30T12:12:30Z"
			"uri": "http://zotero.org/users/8989645/items/6SE87GSB"
		}
	]
}

What Zotero do with this item in APA7.

Piaget, J. (1929). Not closing orunknown tag, kept as text nodes; italic, is an element [Review of Rich text >> italic, par Eugène Minkowski; Paris, <Payot>, 1927]. Archives de psychologie, 22(85), 117‑118.

This TEI export.

<biblStruct type="journalArticle" corresp="http://zotero.org/users/8989645/items/RLRXRWYM">
	<analytic xml:lang="fr">
		<author>
		<forename>Jean</forename>
		<surname>Piaget</surname>
		</author>
		<respStmt>
		<resp>contributor</resp>
		<persName>
			<forename>J.</forename>
			<surname>Contributor</surname>
		</persName>
		</respStmt>
		<respStmt>
		<resp>editorial-director</resp>
		<persName>
			<forename>John</forename>
			<surname>One</surname>
		</persName>
		</respStmt>
		<respStmt>
		<resp>editorial-director</resp>
		<persName>
			<forename>John</forename>
			<surname>Two</surname>
		</persName>
		</respStmt>
		<title level="a">Not closing &lt;b&gt; or&lt;strong&gt;unknown tag&lt;/strong&gt;,
 kept as text nodes; <hi rend="italic">italic</hi>, is an element</title>
		<idno type="callNumber">piaget1929a02</idno>
	</analytic>
	<monogr>
		<editor>
		<forename>M.</forename>
		<surname>Editor</surname>
		</editor>
		<title level="j">Archives de psychologie</title>
		<imprint>
		<date when="1929">1929</date>
		<biblScope unit="volume">22</biblScope>
		<biblScope unit="issue">85</biblScope>
		<biblScope unit="page">117-118</biblScope>
		</imprint>
	</monogr>
	<relatedItem type="reviewed">
		<bibl>
		<author>
			<name>Eugène Minkowski</name>
		</author>
		<title>Rich text &gt;&gt; <hi rend="italic">italic</hi></title>
		<edition>Paris, &lt;Payot&gt;, 1927</edition>
		</bibl>
	</relatedItem>
	<note type="extra">Start of text note.
	
	Unknown CSL property: CLA1929_5
	Note continuing.</note>
	<note corresp="http://zotero.org/users/8989645/items/6SE87GSB"><p>A zotero note with some <emph>formatting</emph>.</p>
	<list rend="numbered">
	<item>
	Hate Numbered List
	</item>
	<item>
	Go to 1.
	</item>
	</list>
	</note>
	<note type="tags">
		<term type="tag">AddATag</term>
	</note>
</biblStruct>

The previous TEI export (original is not well idented)

<biblStruct type="journalArticle">
	<analytic>
		<title level="a">Not closing <hi rend="bold"> or<strong>unknown tag</strong>,
		 kept as text nodes; <hi rend="italics">italic</hi>, is an element</hi></title>
		<author><forename>Jean</forename><surname>Piaget</surname></author>
		<respStmt>
			<resp>reviewedAuthor</resp>
			<persName><name>Eugène Minkowski</name></persName>
		</respStmt>
		<respStmt>
			<resp>contributor</resp>
			<persName><forename>J.</forename><surname>Contributor</surname></persName>
		</respStmt>
	</analytic>
	<monogr>
		<title level="j">Archives de psychologie</title>
		<idno type="callNumber">piaget1929a02</idno>
		<editor><forename>M.</forename><surname>Editor</surname></editor>
		<imprint>
			<biblScope unit="volume">22</biblScope>
			<biblScope unit="issue">85</biblScope>
			<biblScope unit="page">117-118</biblScope>
			<date>1929</date>
		</imprint>
	</monogr>
	<note>A zotero note with some formatting.
		
		
		
		
		Hate Numbered List
		
		
		Go to 1.
		
		
	</note>
	<note type="tags"><note type="tag">AddATag</note></note>
</biblStruct>

Tested with malformed html in Zotero fields.

…into tei_export

AbeJellinek · 2024-02-08T18:35:44Z

TEI.js

+
+/* 2024, Frédéric Glorieux.
+
+// item produced by Zotero


I wish we had test cases for export translators, but this is a lot to put at the top of the translator. We can keep it in the commit history but should remove from the translator before merging IMO.

What I tried was a kind of documentation for end users. TEI is always an interpretation, there are more than one way to encode an information. To understand how the old translator was working, I had to test various kind of records to trace the fields in TEI output, I thought a good idea to explain the code a bit more. Comment in source code is not the right place, you are right, test case is a good idea, but there is still something to find for a documentation.

AbeJellinek · 2024-02-08T18:39:04Z

TEI.js

+ * Imitated from zotero source code
+ * https://github.com/zotero/zotero/blob/main/chrome/content/zotero/itemTree.jsx#L2472


FWIW, Zotero is imitating the behavior of Citeproc.js here, so let's link straight to the source too.

Link added. Below the spec, @dstillman #3054 (comment) make me rewrite the parser to handle tricky cases like
Not closing orunknown tag, kept as text nodes; italic, is an element
Reading code helps me to make the right choices on each case.

AbeJellinek · 2024-02-08T18:40:28Z

TEI.js

+		let discardedNode = nodeStack.pop();
+		nodeStack[0].append(discardedMarkup.token, ...discardedNode.childNodes);
+	}
+	// return textContent; // lint see it’s not used


Old code used for debug. Deleted.

AbeJellinek · 2024-02-08T18:40:48Z

TEI.js

+	if (!html) return;
+	// import html as dom
+	let dom = xmlParser.parseFromString(html, "text/html");
+	let body = dom.getElementsByTagName("body").item(0);


Suggested change

let body = dom.getElementsByTagName("body").item(0);

let body = dom.body;

AbeJellinek · 2024-02-08T18:42:32Z

TEI.js

+ * @param {*} item
+ * @returns
+ */
+function parseExtraFields(item) {


Why do we need this? Won't it happen upon saving the item anyway?

I’m sorry to not understand your question. In the context of a translator, do you mean I can get a state of an item where extra field is parsed to CSL? I would be glad if extraToCSL() have been available through ZU, but it is not.

AbeJellinek · 2024-02-08T18:43:55Z

TEI.js

+		// unicode classes seems not supported
+		// xmlid = xmlid.normalize("NFD").replace(/\p{M}+/u, '');


Use XRegExp:

translators/eLibrary.ru.js

Line 138 in edf44a1

var nameFormat1RE = new ZU.XRegExp("^\\p{Letter}+\\s\\p{Letter}+\\s\\p{Letter}+$");

This will be supported natively in Z7+.

Thanks for the tip.

AbeJellinek · 2024-02-08T18:44:50Z

TEI.js

+	let iso = null;
+	let year = Number(date.year);
+	if (isNaN(year)) return iso;
+	iso = String(date.year).padStart(4, '0');
+	let month = Number(date.month);
+	if (isNaN(month)) return iso;
+	// january = 0
+	iso += '-' + String(date.month + 1).padStart(2, '0');
+	if (!date.day) return iso;
+	iso += '-' + String(date.day).padStart(2, '0');
+	return iso;


Why not strToISO?

Sorry, I missed ZU.strToISO(item.date). Deleted.

glorieux-f · 2025-02-09T12:38:10Z

TEI.js

Handle field rights. Ex : item.rights = "© Gallimard";

<biblStruct type="book" corresp="http://zotero.org/groups/5048422/items/RPRC4566"> <monogr xml:lang="fr">  <availability status="restricted"> © Gallimard </availability> </monogr> </biblStruct>```

glorieux-f added 24 commits June 15, 2023 12:52

Import html rich text as TEI elements

f62f537

Tested with malformed html in Zotero fields.

Big refacroring

3104d92

The january surprise ! 0 is not null but first month

6d28b0f

Rewrite for indentation and better control on variable scope.

8173431

Export abstract as a note

f313105

Reviewed author and formating

41686fa

Better tagsoup handling

7ecbe91

Element.appendChild() > append()

9de4d67

APA7, reviewed-title

bc362e3

Reviewed-Title and Call-Number, better TEI place ; some refactoring

0bab265

better TEI tag for extra.format

e74bdec

language, some refactor

c49681f

Merge branch 'zotero:master' into master

5cc1434

Merge branch 'zotero:master' into tei_export

f298eec

Review, more semantic ; indent refactor

e04e3b8

Merge branch 'zotero:master' into master

6a70732

Merge branch 'zotero:master' into tei_export

7e25f37

Rewrite of Extra parsing, inspired by Zotero.Utilities.extraToCSL()

2e056aa

Merge branch 'tei_export' of https://github.com/glorieux-f/translators …

923f905

…into tei_export

eslint is happy

4cca6b0

Merge branch 'tei_export'

290b3bc

Documentation and better handling of creators in Extra field

0006d4d

date update, eslint

bfc2486

Merge branch 'zotero:master' into master

382473f

AbeJellinek requested changes Feb 8, 2024

View reviewed changes

glorieux-f added 5 commits February 10, 2024 17:58

suggestions from @AbeJellinek zotero#3245

fe9efaa

Merge branch 'zotero:master' into master

3b5949b

Merge branch 'zotero:master' into master

a6ee87d

Merge branch 'zotero:master' into master

74f78af

item.rights -> <availability status="restricted">

5a327d5

glorieux-f commented Feb 9, 2025

View reviewed changes

glorieux-f requested a review from AbeJellinek February 9, 2025 12:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEI translator full rewrite #3245

TEI translator full rewrite #3245

glorieux-f commented Jan 30, 2024 •

edited

Loading

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024 •

edited

Loading

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024 •

edited

Loading

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024 •

edited

Loading

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

AbeJellinek Feb 8, 2024

glorieux-f Feb 10, 2024

glorieux-f Feb 9, 2025

		* Imitated from zotero source code
		* https://github.com/zotero/zotero/blob/main/chrome/content/zotero/itemTree.jsx#L2472

	let body = dom.getElementsByTagName("body").item(0);
	let body = dom.body;

		// unicode classes seems not supported
		// xmlid = xmlid.normalize("NFD").replace(/\p{M}+/u, '');

TEI translator full rewrite #3245

Are you sure you want to change the base?

TEI translator full rewrite #3245

Conversation

glorieux-f commented Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

glorieux-f Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorieux-f Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorieux-f Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glorieux-f commented Jan 30, 2024 •

edited

Loading

glorieux-f Feb 10, 2024 •

edited

Loading

glorieux-f Feb 10, 2024 •

edited

Loading

glorieux-f Feb 10, 2024 •

edited

Loading