Encoding subset updates (#1257) (#1258)

* updating the TS1 sub-encoding declarations to match font changes * 2023 -> 2024 * minor doc edits --------- Co-authored-by: David Carlisle <[email protected]>
latex3 · Jan 28, 2024 · e2ddc3b · e2ddc3b
1 parent da981d9
commit e2ddc3b
Show file tree

Hide file tree

Showing 32 changed files with 810 additions and 107 deletions.
diff --git a/base/alltt.ins b/base/alltt.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/build.lua b/base/build.lua
@@ -26,6 +26,7 @@ installfiles   =
     "*.ltx",
     "*.lua",
     "*.sty",
+    "checkencodingsubset.tex",
     "docstrip.tex",
     "idx.tex",
     "lablst.tex",

diff --git a/base/changes.txt b/base/changes.txt
@@ -6,6 +6,17 @@ to completeness or accuracy and it contains some references to files that are
 not part of the distribution.
 ================================================================================
 
+2024-01-27  Frank Mittelbach  <[email protected]>
+
+	* lttextcomp.dtx (section{Font family sub-encodings setup}):
+	Adjusted/corrected TS1 sub-encoding declarations for various families.
+	Some families have changed their names (due to licensing issues),
+	some got new glyphs added, and some were simply incorrectly categorized
+	(gh/1257)
+
+	* lttextcomp.dtx (section{The \texttt{checkencodingsubset.tex} file}):
+	Added check file for encoding subset
+
 2024-01-03  Phelype Oleinik  <[email protected]>
 	* lthooks.dtx:
 	Correct expansion of \@@_print_args:nn argument (gh/1221).
@@ -23,7 +34,7 @@ not part of the distribution.
 	* ltfilehook.dtx
 	Change flag markup in macro env
 
-2023-12-07 Ulrike Fischer <[email protected]> 
+2023-12-07 Ulrike Fischer <[email protected]>
 	* ltproperties.dtx: use \protected@write, (gh1200)
 
 2023-12-02  Frank Mittelbach  <[email protected]>

diff --git a/base/classes.ins b/base/classes.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/cmextra.ins b/base/cmextra.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/cmfonts.fdd b/base/cmfonts.fdd
@@ -1,6 +1,6 @@
 % \iffalse meta-comment
 %
-% Copyright (C) 1993-2023
+% Copyright (C) 1993-2024
 % The LaTeX Project and any individual authors listed elsewhere
 % in this file.
 %

diff --git a/base/cmfonts.ins b/base/cmfonts.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/doc.dtx b/base/doc.dtx
@@ -143,8 +143,8 @@
 % \MakeShortVerb{\|}
 % \setcounter{StandardModuleDepth}{1}
 %
-% {\catcode`\p=12 \catcode`\t=12 ^^A hack used later on to print
-% \gdef\dimenvalue#1pt{$#1$pt}}  ^^A a register value with a - sign
+% {\catcode`\p=12 \catcode`\t=12 ^^A hack used later on to print a
+% \gdef\dimenvalue#1pt{$#1$pt}}  ^^A register value with a - sign
 %
 % \newcommand{\DOC}{\texttt{doc}\xspace}
 %

diff --git a/base/doc/ltnews39.tex b/base/doc/ltnews39.tex
@@ -239,6 +239,108 @@ \subsection{Optimize creation of simple document commands}
 %
 \githubissue{1189}
 
+\subsection{Declaring appropriate sub-encodings for \texttt{TS1} symbol fonts}
+
+In 2020 we incorporated support for the \texttt{TS1} symbol encoding
+directly into the kernel and in this way removed the need to load the
+\pkg{textcomp} package~\cite{39:ltnews31} to make commands such as
+\cs{texteuro} available.
+
+There is, however, a big problem with this \texttt{TS1} symbol
+encoding: only very few fonts can provide every glyph that is supposed
+to be part of \texttt{TS1}. This means that changing font families
+might result in certain symbols becoming unavailable. This can be a
+major disaster if, for example, the \cs{textteuro} (\texteuro) or the
+\cs{textohm} (\textohm) are no longer printed in your document, just
+because you altered your text font family.
+
+To mitigate this problem, we also introduced in 2020 the declaration
+\cs{DeclareEncodingSubset}. This declaration is supposed to be used in
+font definition files for the \texttt{TS1} encoding to specify which
+subset (we have defined 10 common ones) a specific font implements. If
+such a declaration is used then missing symbols are automatically
+taking from a fallback font. While this is not perfect, it is the best
+you can do other than painstrickenly checking that your document only
+uses glyphs that the font supports and if necessary switch to a
+different font or avoid the missing symbols. See also the discussion
+in \cite{39:ltnews33}.
+
+To jumpstart the process we also added declarations to the \LaTeX{}
+kernel for most of the fonts found in \TeX{}Live at the time\Dash with
+the assumption that such declarations would over time be superseded by
+declarations in the \texttt{.fd} files. Unfortunately, this hasn't
+happened yet (or not often) and so many of the initial declarations
+went stale: several fonts got new glyphs added to them (so their
+sub-encoding should have been changed but didn't); others (mainly due
+to license issues) changed the family name and thus our declarations
+became useless and the renamed fonts (now without a declaration) ended
+up in the default sub-encoding which offers only few glyphs; yet
+others such as CharisSIL (which triggered the GitHub issue) were
+simply not around at the time.
+
+We have therefore, again attempted to provide the (currently) correct
+declarations, but it is obvious that this is not a workable
+process. As we do not maintain the fonts we do not have the
+information that something has changed, and to regularly check the
+ever growing font support bundles is simply not possible. It is
+therefore very important that maintainers of font packages do not only
+provide \texttt{.fd} files but also add such a declaration to every
+\texttt{TS1...fd} font definition file that they distribute.
+
+To simplify this process, we now provide a simple \LaTeX{} file
+(\texttt{checkencodingsubset.tex}) for determining the correct (safe)
+sub-encoding. If run, it asks for a font family and then outputs its
+findings, for example, for \texttt{AlgolRevived-TLF} you will get:
+\begin{verbatim}
+-----------------------------------------
+Testing font family AlgolRevived-TLF
+(currently TS1-sub-encoding 9)
+-----------------------------------------
+Some glyphs are missing from sub-encoding 8:
+   ==> \textcelsius (137) is missing
+   ==> \texttwosuperior (178) is missing
+   ==> \textthreesuperior (179) is missing
+   ==> \textonesuperior (185) is missing
+Some glyphs are missing from sub-encoding 7:
+   ==> \texteuro (191) is missing
+All glyphs between sub-encoding 6 and 7 exist
+All glyphs between sub-encoding 5 and 6 exist
+All glyphs between sub-encoding 4 and 5 exist
+Some glyphs are missing from sub-encoding 3:
+   ==> \textwon (142) is missing
+All glyphs between sub-encoding 2 and 3 exist
+Some glyphs are missing from sub-encoding 1:
+   ==> \textmho (77) is missing
+   ==> \textpertenthousand (152) is missing
+All glyphs between sub-encoding 0 and 1 exist
+All glyphs in core exist
+-----------------------------------------
+TS1 encoding subset for AlgolRevived-TLF (ok)
+Use sub-encoding 9
+-----------------------------------------
+\end{verbatim}
+This output is meant for human consumption, e.g., you see which glyphs
+are missing and why a certain sub-encoding is suggested, but it is not
+that hard to use it in a script and extract the suggested sub-encoding
+by grepping for the line starting with \texttt{Use sub-encoding}.
+
+Of course, this check will only work if the missing glyphs are really
+missing: some fonts placed \enquote{tofu}\footnote{Little squares to
+indicate a missing symbol.} into such slots and in this case it looks
+to \TeX{} as if the glyph is provided. For example, for the old
+Palatino fonts (family \texttt{ppl}) it would report
+\begin{verbatim}
+-----------------------------------------
+TS1 encoding subset for ppl (bad)
+Use sub-encoding 0 (not 5)
+-----------------------------------------
+\end{verbatim}
+i.e., all glyphs are provided, while in reality more than twenty are
+missing and sub-encoding 5 as declared in the kernel is correct.
+%
+\githubissue{1257}
+
+
 
 %\section{Bug fixes}
 
@@ -312,8 +414,16 @@ \subsection{Correct definition of \cs{k}}
 %\newblock Reprinted with corrections in 1996.
 %
 %
+\bibitem{39:ltnews31} \LaTeX{} Project Team.
+  \emph{\LaTeXe{} news 31}. February, 2020.
+  \url{https://latex-project.org/news/latex2e-news/ltnews31.pdf}
+
+\bibitem{39:ltnews33} \LaTeX{} Project Team.
+  \emph{\LaTeXe{} news 33}. June 2021.\\
+  \url{https://latex-project.org/news/latex2e-news/ltnews33.pdf}
+
 \bibitem{39:ltnews37} \LaTeX{} Project Team.
-  \emph{\LaTeXe{} news 37}.\\
+  \emph{\LaTeXe{} news 37}. June 2023.\\
   \url{https://latex-project.org/news/latex2e-news/ltnews37.pdf}
 
 %\bibitem{39:blueprint} Frank Mittelbach and Chris Rowley.

diff --git a/base/docstrip.ins b/base/docstrip.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -52,7 +52,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/ec.ins b/base/ec.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/exscale.ins b/base/exscale.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/fix-cm.ins b/base/fix-cm.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 

diff --git a/base/format.ins b/base/format.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 
@@ -88,7 +88,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 2015-2023
+Copyright (C) 2015-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 
@@ -123,7 +123,7 @@ extension .ins) which are part of the distribution.
 
 This is a generated file.
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.
 
@@ -228,6 +228,8 @@ the system are in the document `cfgguide.tex'.
           \from{lttextcomp.dtx}{TS1sty}}
    \file{textcomp-2018-08-11.sty}{%
           \from{lttextcomp.dtx}{TS1oldsty}}
+   \file{checkencodingsubset.tex}{%
+          \from{lttextcomp.dtx}{TS1check}}
   }
 
 \generateFile{oldlfont.sty}{t}{%

diff --git a/base/graphpap.ins b/base/graphpap.ins
@@ -3,7 +3,7 @@
 %% driver files from the doc files in this package when run through
 %% LaTeX or TeX.
 %%
-%% Copyright (C) 1993-2023
+%% Copyright (C) 1993-2024
 %% The LaTeX Project and any individual authors listed elsewhere
 %% in this file.
 %%
@@ -48,7 +48,7 @@ reports for it can be opened at https://latex-project.org/bugs.html
 (but please observe conditions on bug reports sent to that address!)
 
 
-Copyright (C) 1993-2023
+Copyright (C) 1993-2024
 The LaTeX Project and any individual authors listed elsewhere
 in this file.