Peter Krautzberger

(Print) "Algorithm Layout" in HTML and CSS

2022-07-26T00:00:00Z

July 2022. I started this piece well over three years ago (some time before June 2019 in any case). As it ~~came along~~ grew out of proportion, I had tried turning it into three pieces but never got far enough to start publishing it. Then I tried to recombine them (some time in 2020?) in the hopes of finally pushing them out but didn't finish the last few pieces. Then 2021 was worse than 2020 and now it's 2022. So I'm thinking I will make this a new kind of piece around here: a partially complete piece of writing that may (or more likely: may not) get edits and updates. Let's jump in then.

A while ago I started to think about "algorithm layout". Not syntax highlighting of source code (which prism.js handles nicely) but the kind of print traditions captured in the following image.

Part 1. Prior art and the problem of content conversion

If you're familiar with LaTeX packages such as algorithmicx or algorithm2e you'll recognize this particular style of layout. As the names of those packages imply, this type of content is more about the abstract algorithms and less about implementation details in a specific language.

Realizing this specific style on the web is an interesting challenge. On the one hand, it's a problem of realizing print layout traditions on the web, an area I deal with professionally on a daily basis. On the other hand, there's an authoring problem behind this: realizing print layout is one thing, realizing it in a way that allows for a good (web!) reading and authoring experience is another.

When I started this side project, I had only had a quick look at some tools that solve this problem and before I wanted to write about what I did, I thought I should take another, closer look at them to provide context (and check my own work). I'm glad I didn't look too closely earlier since that may have sent me off in a completely different direction. It also reminded me of the pitfalls of authoring, in particular print content and conversion.

Prior art 1: algorithmicx

To pick a starting point, let's use a classic in every sense of the word: Euclid's algorithm.

Here's what you'd write using the algorithmicx LaTeX package.

\begin{algorithm}
    \caption{Euclid's algorithm}\label{euclid}
    \begin{algorithmic}[1]
        \Procedure{Euclid}{$a,b$}\Comment{The g.c.d. of a and b}
            \State $r\gets a\bmod b$
            \While{$r\not=0$}\Comment{We have the answer if r is 0}
                \State $a\gets b$
                \State $b\gets r$
                \State $r\gets a\bmod b$
            \EndWhile\label{euclidendwhile}
            \State \textbf{return} $b$\Comment{The gcd is b}
        \EndProcedure
    \end{algorithmic}
\end{algorithm}

and you'd get the image from earlier. It's fun to note that this source is straight from the package documentation, including inconsistencies (such as a mismatch of math mode in states and comments).

Let's naively HTML-ify this:

<algorithm>
    <caption id="euclid">Euclid's algorithm</caption>
    <algorithmic numberLineSkip="1">
        <procedure>
            <label>Euclid</label>
            <start>a,b</start><comment>The g.c.d. of a and b</comment>
            <state>r ← a mod b</state>
            <while>
                <start>r ≠ 0</start> <comment>We have the answer if r is 0</comment>
                <state>a ← b</state>
                <state>b ← r</state>
                <state>r ← a mod b</state>
            </while>
            <state> <strong>return</strong> b</state> <comment>The gcd is b</comment>
        </procedure>
    </algorithmic>
</algorithm>

That seems fairly straight forward even if the label "euclidendwhile" has been lost.

I'd like to focus on two problems

how to get from here to something we can style easily (for the web)
while keeping things relatively easy to style (and customize/extend)
while keeping things relatively easy to author
while making this accessible

Or put differently: how to move from content conversion to content authoring.

Pior art 2: latexml

While I personally wouldn't recommend it, latexml is an interesting option to consider because it is so heavily focuses on extracting semantic structure from print documents.

Here's a codepen embedding of what you'd get from it:

See the Pen Algorithms via LaTeXML by Peter Krautzberger (@pkra) on CodePen.

So to me this is a terrible mess of markup and layout but it's not surprising given what latexml focuses on. When you get into the business of generating HTML and CSS from things that are fairly far removed from HTML and CSS (here: long form print documents) you are bound to create such a mess of a markup. That's not a criticism; I think it's simply inevitable.

After all, an author who uses latexml was not interested in authoring web content. Instead, they authored print content using LaTeX and then used a second piece of software (filled to the brim with heuristics) to generate a close approximation of the print product in a custom XML. In a third step, they created HTML and CSS from that.

Nevertheless, there is a lot of noise in this which may make it hard to explore the example. Here's another codepen embedding with a more minimal version of the latexml output that has (hopefully) less noise and still exhibits the inherent issues I see with this approach towards authoring of web content.

See the Pen Algorithms via LaTeXML, reduced by Peter Krautzberger (@pkra) on CodePen.

A quick list of things that jump out when going through this.

functional/atomic CSS (in the tailwind or tachyon sense), e.g., a border-top class, a small-caps class, a bold class - a natural concept for content conversion from an only potentially structured format like TeX.
inline styles - usually necessary when print layout is mimicked in detail (though somewhat surprising in this particular case which looks like it could be replaced by a functional class if you think about it from a TeX point of view)
the weird case of line 3 - it looks like a bug that the comment is wrapped in the bold part of the while...do construct and thus needs a font weight override (but maybe it's a weird thing in the LaTeX package)
right-floating comments can overlap on small viewports
line counters and comment markers are yucky (much worse in the original)
divs and spans galore. Each "line" is a div but the line counter and state are wrapped together
and the "highlight": repeated nbsp's to create indentation. Yikes, the 90s want their spacer GIFs back. (But again, this might be a faithful conversion from how the LaTeX package realizes the print layout.)

I'm not trying to complain too much here. Again, this is mostly what you'd expect when you build a tool like latexml. Print is just a crappy format to generate web content from and even good LaTeX can't save you from that fact.

Prior art 2: pseudocode

The next example is a probably less known JS library called pseudocode.js.

Here's again a codepen embedding of what we might get.

See the Pen pseudocode.js by Peter Krautzberger (@pkra) on CodePen.

Cleaning it up a bit seems harder given. Maybe that's because it is closer to thinking in terms of web design (as opposed to print-reproduction). But we can delete the equation layout mess (which is weirdly inconsistent in the algorithmicx source anyway).

See the Pen pseudocode.js, reduced by Peter Krautzberger (@pkra) on CodePen.

Again some random observations.

moar divs. Not even a figure/figcaption (but a class with-caption); that's a bit disappointing.
decent line-wrapping on small screens is nice
class names are less functional and more semantic (in the sense of layout), e.g., ps-line. That kind of intent seems appropriate.
odd inline styles
- canceling things out: style="text-indent:-1.2em;padding-left:1.2em;" is a funny extreme but probably just a side effect of the overall strategy to combine margin-right with negative text-indent.
- rounding issues style="margin-left:1.7999999999999998em;" - just a bug but makes me wonder about code quality (ok, not really).
relative positioning. Feels weird to me for this kind of layout (of course it's at the heart of how this particular approach works)
nested substructures - that is right up my alley (more on that later hopefully). Stacking margin-left is neat but comes at the price of negative text-indent shenanigans for the line counters; I don't get why the margin is inline CSS rather than a class but there's possibly a decent reason for that (e.g., authoring).

Coda Part 1.

In both cases, the main reason why I find these approaches lacking lies, I think, in the fact that they are the result of (print) content conversion. The disconnect between the authoring and the layout is too large to get one in touch with the other. This is a loss for both sides: the layout gets messy and the author comes no closer to understanding the medium they author for.

A typical side effect of this shows when you want to modify the output: neither output can be re-arranged efficiently. Usually, all you can do is go to the source, change it, and convert it again. We'llcome back to that later.

To be clear, I'm not saying that some of the "bad" design decisions could be avoided; I definitely haven't found a way around every pain point. But my experience has been that the law of the instrument is at the heart of the problem. Content conversion inevitably leads to certain types of hacks (such as meaningless DOM structures, class names and inline styles) to allow authors customizations without them having to think about the actual medium. If it's content conversion from a format meant for print, this is problem exacerbated because the media and their traditions often match poorly. But once those hacky instruments are in the tool belt, it's hard not to use them and you end up with hacks all the way down.

In the next part, I'll explore a different approach, also thinking about it from the perspective of the source material but hopefully with some level of detachment. After that, perhaps I'll find time to jot down some ideas on how to move towards authoring. But this piece has been more fun than I thought it would be; it's been interesting to dive a little deeper into these two after trying things on my own; good to see similarities and differences.

Part 2. A first draft

Let's go back to the beginning.

In Part 1 I started at the wrong end, so to speak, and I looked at solutions that try to be keep the spirit of the print source, both in authoring "convenience" and layout design. I also outlined what I think of as issues and limitations of such (print) content conversion.

Still, it is painfully fascinating. I wanted to find another way of dealing with this problem and, ideally, learning (with detachement) from the print authoring experience to enable web authoring.

Back to the source: algorithmicx (again)

Again, we keep to the classics and Euclid's algorithm.

Here's what you'd write in algorithmicx.

\begin{algorithm}
    \caption{Euclid's algorithm}\label{euclid}
    \begin{algorithmic}[1]
        \Procedure{Euclid}{$a,b$}\Comment{The g.c.d. of a and b}
            \State $r\gets a\bmod b$
            \While{$r\not=0$}\Comment{We have the answer if r is 0}
                \State $a\gets b$
                \State $b\gets r$
                \State $r\gets a\bmod b$
            \EndWhile\label{euclidendwhile}
            \State \textbf{return} $b$\Comment{The gcd is b}
        \EndProcedure
    \end{algorithmic}
\end{algorithm}

and you'd get the image from earlier. And here's our naive HTML-ification again:

<algorithm>
    <caption id="euclid">Euclid's algorithm</caption>
    <algorithmic numberLineSkip="1">
        <procedure>
            <label>Euclid</label>
            <start>a,b</start><comment>The g.c.d. of a and b</comment>
            <state>r ← a mod b</state>
            <while>
                <start>r ≠ 0</start> <comment>We have the answer if r is 0</comment>
                <state>a ← b</state>
                <state>b ← r</state>
                <state>r ← a mod b</state>
            </while>
            <state> <strong>return</strong> b</state> <comment>The gcd is b</comment>
        </procedure>
    </algorithmic>
</algorithm>

Some random thoughts when looking at this:

we should obviously use a figure with figcaption element for HTML; maybe sprinkle some aria-roledescription in there.
CSS pseudo-content seems an obvious choice for "Procedure" and "While...do / done".
why don't those comments align? Seems worth changing.
As I mentioned already, I like the nesting (Procedure/EndProcedure, While/EndWhile). We should find a way to keep it. But that also seems hard thanks to those line counters.
This layout is obviously a case for grid

head on

So let's try not to overthink this, let's try not to take more than inspiration from the source, let's try to think about the layout, its meaning, and how it could work with the web's grain.

What do we need at first sight?

multiple lines of 3 columns: counter, content, comment
- get nice alignment
- try to avoid tables (for both layout and accessibility but it's complicated; maybe more on that later)
a way to handle begin/end type of things (subroutines, loops etc)

Again, the obvious modern choice for two dimensional layouts is CSS grid. And CSS counters should help as well:

.algorithm {
  counter-reset: algolinecounter;
  display: grid;
  grid-template-columns: repeat(3, max-content);
  grid-column-gap: 1em;
}

To try it out, here's a dummy set of two "lines", first as part of the page then as code.

Counter

Content

Comment

Counter

Content

Comment

<section class="algorithm">
    <div>Counter</div>
    <div>Content</div>
    <div>Comment</div>
    <div>Counter</div>
    <div>Content</div>
    <div>Comment</div>
</section>

Nothing really to see yet, so let's move on.

To handle our line counters, we use the counters (obviously) but we also get some critical help from grid: since we might not have comments, we don't know whether the line is full. But that's not a problem with grid - we can just force our counter to always start in the first column

.counter {
  grid-column-start: 1;
  counter-increment: algolinecounter;
}

.counter::before {
  content: counter(algolinecounter) ': ';
}

To try it out, here's two lines with only the second on having a comment. Again, first as part of the page, then as source.

Content

Comment

<section class="algorithm">
    <div class="counter"></div>
    <div>Content</div>
    <div class="counter"></div>
    <div>Content</div>
    <div>Comment</div>
</section>

This is a good point to think about accessiblity. CSS generated content is of course ok but we might want to add some ARIA attributes to indicate the visual effect of starting a line; then again, maybe that tends to be clear from context and we shouldn't add noise. It's tempting to aria-hide the counters and use table roles which would gives AT users the advantage of table walkers - a grid role would be more appropriate but then we'd have to write our own walker so definitely out of scope for a first draft. Anyway, we don't have to do this right now, we just have to keep it in mind.

Next up, let's deal with the other fixed property: comments. This is simple anyway:

.comment {
  margin-left: 3em;
}

.comment::before {
  content: '▷ ';
}

This looks nice enough but we should probably hide that from non-visual users. We could indicate that it's a comment but again it might be enough to grok this from context. After all, context is all we have visually. Let's take a look, in the page and source:

Content

Comment, longer

Content

Comment

<section class="algorithm">
    <div class="counter"></div>
    <div>Content</div>
    <div class="comment">Comment, longer</div>
    <div class="counter"></div>
    <div>Content</div>
    <div class="comment">Comment</div>
</section>

Oooh, look at that alignment! Isn't it pretty?

So, 9 lines of CSS and our basic layout is done. Nice.

Since this is a first draft, let's continue not to over think this. The begin/end routines seem to always have, well, a beginning and an end. So let's just make that happen.

.while::before,
.while::after {
  font-weight: bolder;
}

.while__start::before {
  content: 'while ';
}

.while__start::after {
  content: ' do';
}

.while__end::before {
  content: 'end while';
}

In action and as source:

Some condition

Otherwise, we're done.

We do some work

<section class="algorithm">
    <div class="counter"></div>
    <div class="while while__start">
        Some condition
    </div>
    <div class="comment">
        Otherwise, we're done.
    </div>
    <div class="counter"></div>
    <div>We do some work</div>
    <div class="counter"></div>
    <div class="while while__end"></div>
</section>

The final piece doesn't look very nice but neither does the original source.

There's a worse glitch though: if you look at the original, the part "inside" the loop is indented. This is slightly tricky since we don't have a good way of writing a selector for those kinds of elements. Luckily, this is a first draft and we're lazy, so let's just add a helper class.

.ml1 {
  margin-left: 1em;
}

Problem solved - let's have a look:

Some condition

Otherwise, we're done.

We do some work

<section class="algorithm">
    <div class="counter"></div>
    <div class="while while__start">
        Some condition
    </div>
    <div class="comment">
        Otherwise, we're done.
    </div>
    <div class="counter"></div>
    <div class="ml1">We do some work</div>
    <div class="counter"></div>
    <div class="while while__end"></div>
</section>

Moving on to the last issue, nested subroutines, so let's copy and paste our loop.

Some condition

Otherwise, we're done.

Some condition

Otherwise, we're done.

We do some work

<section class="algorithm">
    <div class="counter"></div>
    <div class="while while__start">
        Some condition
    </div>
    <div class="comment">
        Otherwise, we're done.
    </div>
    <div class="counter"></div>
    <div class="while while__start">
        Some condition
    </div>
    <div class="comment">
        Otherwise, we're done.
    </div>
    <div class="counter"></div>
    <div class="ml1">We do some work</div>
    <div class="counter"></div>
    <div class="while while__end"></div>
    <div class="counter"></div>
    <div class="while while__end"></div>
</section>

But of course now we have the same margin problem all over again, so we don't think too hard and add another helper.

.ml2 {
  margin-left: 2em;
}

Then we can make it:

Some condition

Otherwise, we're done.

Some condition

Otherwise, we're done.

We do some work

<section class="algorithm">
    <div class="counter"></div>
    <div class="while while__start">
        Some condition
    </div>
    <div class="comment">
        Otherwise, we're done.
    </div>
    <div class="counter"></div>
    <div class="ml1 while while__start">
        Some condition
    </div>
    <div class="comment">
        Otherwise, we're done.
    </div>
    <div class="counter"></div>
    <div class="ml2">We do some work</div>
    <div class="counter"></div>
    <div class="ml1 while while__end"></div>
    <div class="counter"></div>
    <div class="while while__end"></div>
</section>

At this point, it's clear that we have a lurking problem but luckily this was just a first draft and we can finish our example. Let's just add a procedure class as follows.

.procedure::before {
  font-weight: bolder;
}

.procedure__start::before {
  content: 'procedure ';
}

.procedure__end::before {
  content: 'end procedure';
}

And it looks like we're there.

Euclid(a, b)

The g.c.d. of a and b

r ← a mod b

r ≠ 0

We have the answer if r is 0

a ← b

b ← r

r ← a mod b

return b

The gcd is b

<section class="algorithm">
<div class="counter"></div>
<div class="procedure procedure__start">
    <span class="sc">Euclid</span>(<i>a, b</i>)
</div>
<div class="comment">
    The g.c.d. of <i>a</i> and <i>b</i>
</div>
<div class="counter"></div>
<div class="state ml1"><i>r ← a </i>mod<i> b</i></div>
<div class="counter"></div>
<div class="while while__start ml1">
    <i>r ≠ </i>0
</div>
<div class="comment">
    We have the answer if <i>r</i> is 0
</div>
<div class="counter"></div>
<div class="ml2"><i>a ← b</i></div>
<div class="counter"></div>
<div class="state ml2"><i>b ← r</i></div>
<div class="counter"></div>
<div class="ml2"><i>r ← a </i>mod<i> b</i></div>
<div class="counter"></div>
<div class="while while__end  ml1"></div>
<div class="counter"></div>
<div class="ml1"><strong>return</strong> <i>b</i></div>
<div role="region" aria-label="comment" class="comment">
    The gcd is <i>b</i>
</div>
<div role="region" aria-label="line" class="counter"></div>
<div class="procedure procedure__end"></div>
</section>

Coda Part 2.

Not bad for a first draft. Grid makes so many things just easy, sprinkle some counters.

There are a couple of glaring issues at this point

those helper classes feel like content conversion. Can we do better?
grid soup - could subroutines be wrapped in a single element?
responsivity - scrolling is not awesome.

Inevitably, these will lead us into the weeds. But at least the web's weeds.

Part 3. But what is it

Let's go back to our starting point.

In Part 1, I started at the wrong end, if you will: I looked at a TeX source and then at existing solutions that aim to stay close to print, both in authoring "convenience" and layout design. In Part 2, I went through a rought draft of realizing a simlar layout in CSS "properly".

Now, let's try to let go.

Forget the source, Luke.

What can we observe for this layout?

a title
a bunch of numbered lines
containing three distinct parts per line - counter, main block, comment block
indentation where subgroups of lines have their main blocks indented
some bold key words that match those subgroups
some decorative top/bottom borders around the title and whole algorithm

The first draft was trying to be so minimalistic that it skipped on the two primary semantic pieces: lines and subgroups of lines. Isn't that odd?

So thinking about those two, where would we start?

Authoring with batteries included

There are a couple of conveniences when it comes to authoring that I'd like to see preserved. The core problem is change: removing and injecting content into other content. You want it to magically work and of course it often doesn't. In particular copy&paste is a key usability challenge that good authoring tools try to solve. Obviously, this is is a hard problem in general.

It also leads to other factors such as good (automated) default behavior alongside customizability aka "macros".

The first challenge from the first draft is: how do we solve the indentation problem, i.e., how can we avoid the .ml1, .ml2 etc classes. Because if we cannot solve that, authors will have to a) add those classes themselves and b) update them whenever they move or add something to an existing structure.

Coda

This coda marks where the "original" draft ended, i.e, the draft from which I started in June 2022 (though many edits were done in June / July 2022). Handling the indentation is really the only "hard" problem for this layout so I'm not sure why I stopped here.

I have a couple of codepens from that time that might be worth writing about. And then there's the solution I adopted at work to talk about (which, inevitably, is different). Let's stop here and see if I get around to those updates.

Thoughts on Print Equation Layout

2022-09-17T00:00:00Z

This will possibly be another piece that will get updates in the future. Actually, I hope not but we'll see. This, above all, is for me. For me to gather my thoughts one (hopefully) last time. For me to deal with stupid garbage which has accumulated over the years. As for you, dear reader, I don't know if this is for you. I didn't write it for the handful of experts in this area; maybe for someone like my former self, back when I started writing mathematics on the web. Maybe for someone who knows the web and wonders what all that fuss with equations is always about. In any case here's a fair warning: I'm not going into every detail so if I lose you along the way, remember these first few lines and go do something more fun, it's fine. Just don't be that guy (and it's invariably a guy) complaining about free stuff on the internet.

This is also a lamentation. Nothing I write will change anything of what's to come. It is also (hopefully) a bit of mental cleansing. A few years ago I had strong burn-out symptoms due to working in this (as it turned out toxic) field. It takes a lot of effort for me to not be dragged down again. But one last time I want to try to write up where I've arrived after thinking about equation layout on the web for over 10 years now.

If I make a claim without substantiating links, it's most likely to avoid more burn out; you can ping me privately though. If I'm giving just one or two examples, it's not because that's it but because this piece is very long already. [If you want deeply detailed research results, feel free to contact my company - we do that kind of thing.]

Finally, I thank the kind people who read this piece before I pushed it live - your feedback was invaluable; I owe you one.

Introduction

10 years ago I joined the MathJax project which forced me to think a lot about how to put equations on web pages. Actually, I had already spent quite a bit of time during my PhD and postdoc years thinking (*cough* procrastinating) about the same thing, albeit from a different perspective.

When I started writing on the web all 13 years ago, I quickly went through a phase I've seen repeated by many people since:

"oh, there's something called MathML, I should use that"

followed by

"oh this does not work at all, let's not use that",

usually followed by choosing jsMath (if you're old enough) and later MathJax because they ultimately remain - by far - the best tools for the job.

After joining MathJax, my job was to profess the prowess of (read:blindly advertise for) MathML, so I did. That was largely because MathJax was formally part of Design Science at the time and they made money with MathML. Luckily, the project moved to the AMS where we could think about things more freely.

When the whole Chrome 24 debacle happened, there was this one comment from a Googler (on Google+, naturally you might say). I can't find it anymore but what I recall them saying is: "maybe it's a good format but still a bad format for the web". I kept coming back to that thought for a few years and, eventually, came to agree with it. (This piece is basically summarizing that.)

In short, I realized that MathML is fundamentally a useful XML print publishing tool but a bad technology for the web. So I tried to pivot MathJax away from MathML and its toxic community towards actual web standards and their underlying principles. We wrote a whitepaper, got our sponsors on board, and started work on MathJax v3. (This is not what v3 became but that's ok. I left my position at MathJax around the first beta which was a thrill and I'm grateful that Davide and Volker enabled me to leave with that big leap forward having begun.)

Towards the end of that period, I wrote a number of pieces here, outlining MathML's many flaws and failures. But as I kept writing: it's not MathML's fault. MathML does what it's supposed to; but what it's supposed to do is a bad fit for the web. (Again, this piece is basically re-iterating those pieces. What do you expect from a recovering academic.)

In the past 5 years, I've worked primarily on the issue of converting full mathematical (LaTeX) documents to web formats. If you follow me on the fediverse, you may have seen me use the CW "TeX in real world production" sharing some of the "gems" from this experience.

All of this has led me to realize that there's a "deeper" problem and I feel like it needs a (sort of) new name: print equation layout.

Framing

Framing is important. What I call "print equation layout" is more frequently called "math layout", "formula layout", or just "math" (I find that last one somewhat insulting). The phrase "print equation layout" may seem unnecessary or convoluted. To some degree, I agree. For example, I don't believe we have seen "web equation layout" come into existence during these first 30 years of the web. Nor, sadly, do I expect to see it in the next 10.

But as a framing device "print equation layout" serves to ground the conversation and keep aspects in focus that I find critical to many discussions.

In my opinion, print equation layout is, ultimately, a flawed print tradition that web technology should not have to make extra efforts to support (e.g., as proposed by MathML's addition to HTML5 way back when). The layout features are largely archaic with very few actual challenges remaining in the context of contemporary (2022) web layout.

Furthermore, print equation layout is fundamentally inaccessible in the sense of web accessibility. However - and this is where this part gets very tricky - print accessibility traditions for print equation layout do exist (at least in the US as well as a few other countries and of course the US perspective always dominates these conversations). This may seem like a contradiction - how can there be an accessibility tradition that is bad in the context of the web - but it is really no different from the visual layout traditions.

This is a key complication in the discussions (and myths) around this topic: because print layout and print accessibility traditions exist, people assume that they must be realized on the web - and in the same way.

I want to argue that neither aspect of print equation layout - visual layout traditions and accessibility traditions - is particularly good or useful on the web. As a result, it is a particularly problematic set of techniques because the contemporary web already has plenty of good and useful features that solve print equation layout.

Adding an additional, separate feature layer for a problematic print technology tradition does not make any sense, in fact it has a (by now long) history of hurting the community.

Interlude

In TeX, we can put punctuation after a table. In HTML, we cannot. Should HTML change because this print tradition exists?

What's in a word

Let's try to be a bit more specific.

Print equation layout is...

what people think of when you say "math/formula/equation" // what MathML partially captures // what math mode TeX usually produces // what MS Word formulas usually look like
an elaborate, two-dimensional form of abbreviated notation for compressed, highly complex, context dependent, abstract thought.

That's a bit of a mouthful, I know. But, in the spirit of recursion, this compressed, complex description captures the essential problems well and much follows easily from it.

This is also not an observation of edge cases, it holds at an elementary level, e.g., in the shape of long addition/multiplication/division or "explanatory" annotated layout in school-level material. At the research level, things are much (much) worse, because research by its nature invents new concepts but more so because especially compression was exacerbated by the content explosion of the 20th century, combined with print page limits.

A first consequence of this description is that print equation layout is highly ambiguous because the compression is very lossy. What's worse is that print equation layout not just needs context but is frequently integrated into other complex notation (e.g., text annotation or graphical documents).

A consequence of this is that it is "non-text content" (in the sense of WCAG SC 1.1.1). It is much closer to "ascii art" than HTML traditions of "semantic markup" with concepts such as heading, landmark, or widget.

Naturally, this does not apply to all instances of print equation layout. It crucially applies where the actual problems occur - both in visual and non-visual rendering. The problem is not to handle 1+1=2 (well, actually we'll get back to this), the problem is more like:

\det(A)=\sum _{\begin{array}{c}k_{1},k_{2},\ldots ,k_{n}\geq 0\\ k_{1}+2k_{2}+\cdots +nk_{n}=n\end{array}}\prod _{l=1}^{n}{\frac {(-1)^{k_{l}+1}}{l^{k_{l}}k_{l}!}}\operatorname {tr} \left(A^{l}\right)^{k_{l}}

This is a simple piece of print equation layout, something from the first few months at university. And it is packed.

interlude

Cars are a bad technology.

They need massive resources to build, most use fossil fuel to run, all need massive infrastructure, destroying our habitat; they are vastly inefficient, they stress people out, they fill our streets and sidewalks with megatons of metallic monstrosities.

And yet most people own one or want to own one. They have their uses, some at which they excel. So they're here and we need to think about them.

But that doesn't change the fact that we should think about a world without cars because it could really be a much better world for everyone. And we shouldn't insist to keep building our world around cars or make cars a core assumption in our "cultural stack".

The same holds true for print equation layout.

Visual print equation layout on the web

practically

The problem of realizing print equation layout on the web has been solved for almost two decades now. Well, 18 years to be exact.

Let that sink in for a bit. 18! years!

With CSS, this solution started as (client-side) JS only and span soup; we have had that since 2004 (via jsmath, MathJax's precursor).

With SVG, we've had this since 2011. MathJax v2, at first client-side but quickly also server-side (via phantomjs and later via nodejs). [To some degree, various tools could produce SVG from, e.g., TeX, even earlier that but they didn't generally care about the web as a use case, e.g., they wouldn't give you good vertical alignment to meet a baseline.]

It has been possible to realize print equation layout with just CSS (e.g., server-side generated span soup and CSS) since IE8 - IE8 dammit, released in 2009 - and with just SVG since IE9 (2011). [To be fair, MathJax was not the first to realize such a CSS approach, KaTeX was, but one of my favorite successes while at MathJax was to prod Davide long enough about a "fast Preview without JS" that he built the "CommonHTML" output for v2.7 which added this. I'm also still fond of the previewHTML output which preceded it and traded beauty for CSS simplicity.]

Visual print equation layout has been solved thrice over with a myriad of smaller and larger tools doing it in varying ways, balancing code simplicity with quality of layout. (mathquill, jqmath, math-ml etc)

[Aside. This is why a certain blog post from a few years ago about "alternate timelines" was so very much off base - CSS driven print equation layout was first, SVG a close second, not some XML dialect nobody was using for anything but print production. (And no, IE plugins do not count in my book.) In the similar situations the same people would (indeed did famously) argue that working solutions should be considered the prime candidates to move standardization forward, not some contrived academic proposal. But I digress.]

So print equation layout solutions for the web have existed for a long time.

Are they perfect? No.

Are they still getting better? Emphatically yes - all the time. The ever evolving CSS continues to make things easier (grids, container queries, variable fonts, layers!). SVG is just rock solid (for these purposes; and it improves, too).

The most significant CSS challenge at this point are stretchy character constructions - not a massive problem and one that could (and should) be solved independently of any "mathematical" considerations since stretchy constructions are frequently used in other design scenarios (e.g., stretchy braces around lists are pretty common design patterns).

In the nice-to-have category, I'd place better control over glyph bounding boxes. Right now, high quality print equation layout realized with CSS depends on webfonts. This is not a particularly critical consideration in my book - much high quality text layout on the web relies on webfonts - and the bounding box problem also needs a more general solution (and in fact there are proposals for better access to font internals).

Some people argue that the CSS and SVG layout solutions like MathJax generate "convoluted" markup output. This is a red herring.

On the on hand, convoluted XML is not "better" if it says "math" in it. On the other hand, nobody from CSS or SVG (spec) land has ever tried to help so tools in this space make do with what they can find. My favorite comparison for web people is: the output looks a lot like Bootstrap's grid layout in its early days (somewhat convoluted) or an accessible datepicker markup today (extremely convoluted). In other words: it looks the way it looks to get the job done with the tools available. And it does. And it gets simpler every year.

If print equation layout tools for the web were heard the same way grid frameworks were heard way back when, we might get simpler output even quicker and in a way that benefits the entire web stack. In a world of div-soup SPA's I think the "ugly output" argument is entirely a red herring.

interlude

From Oral History of Butler Lampson (Interviewed by:
Alan Kay) (via Robin Sloan's newsletter)

Alan Kay: But I wish that you had been at CERN on a sabbatical when [the Web was being born].

Butler Lampson: I probably would have been a disaster.

Kay: I don’t know. But I think you would have made a slightly better …

Lampson: No. No. No. No. No. No. What Tim [Berners-Lee] did was perfect. My view about the web is that it’s the great failure of computer systems research. Why did computer systems researchers not invent the web? And I can tell you the answer. It’s because it’s too simple.

Kay: It is too simple.

Lampson: If I had been there I would have mucked it up. I swear to God. The idea that you’re going to make a new TCP connection for every mouse click on a link? Madness! The idea that you’re going to have this crusty universal data type called HTML with all those stupid angle brackets? We never would have done that! But those were the things that allowed it to succeed.

theoretically

Ultimately, the problem is: is this layout tradition a good fit for the web platform? Take table layout. At first people thought it was a good fit. Now we don't.

I think print equation layout is a thoroughly bad fit, building it into the web actually damages the platform, just like tables have taken two decades to become unscrewed (oh hi Google, still using tables for layout in some products).

Even for the simplest examples we run into "fun" print equation layout traditions that are problematic. The easiest to understand are the many text-dependent behaviors, i.e., layout behavior that changes depending on the specific text nodes present (beyond the obvious layout changes from having different text nodes).

The simplest example is probably: single letter variables are italic, multi-letter ones are upright. This is pure tradition, i.e., something a lot of people like to do. Obviously people sometimes don't do this. Obviously, there's no technical difficulty to make this happen as an author / authoring tool on the web today. But as a feature for the web it is weirdly different from any CSS/text rendering features we have; that's just not how styling text works.

A more complex example are moveable limits. For example the TeX expression \sum_{i=0}^\infty would, at plain view, indicate we get a summation symbol with a subscript of "i=0" and a superscript of infinty.

However, this depends. As an inline expression, TeX will lay this out as I just described.

\sum_{i=0}^\infty

However, as a display/block expression, the sub- and superscripts will become under- and overscripts.

\sum_{i=0}^\infty

Most print equation layout systems have similar mechanics. This is of course convenient from an authoring perspective - you can switch modes as you like, thus macros or copy&paste are much easier to do.

However, from a web design perspective it is a wild thing. Not the change as such (of course children change when a container switches from inline to block). But when you replace \sum with something else (say an "S" for "sum"), you will suddenly not get this behavior anymore. It's driven by (an allow list of) text nodes.

From a layout perspective, this automatism wouldn't have to be builtin - you have to have tooling to specify both layouts anyway - yet MathML, as a print equation layout system, builds it in. This means, we have intentionally ambiguous layout instructions that change rendering depending on text content. This very weird from the perspective text layout on today's web.

From this we can take a step up and talk about the operator dictionary. Remember when I spoke about 1+1=2 not being the hard problem. Guess again, we made it hard!

[I mean, what do you expect? There's a reason why movies will visually indicate something is difficult beyond comprehension by showing print equations layouts; even memes about confusion get more popular when you add equations to!]

Naturally, there are traditions around how you space things. 1+1=2? 1+1 = 2? 1 + 1 = 2? Or something more subtle? There are many traditions like this, the most well known being Donald Knuth's as captured in the TeX book.

But that's just it - they are traditions. Traditions, based on Knuth's review of Western (and mostly English language) publications from the late 19th and 20th century.

And people can, do, and should deviate from traditions. If there's one thing I've learned working in mathematical publishing production, it's that authors are full of ideas for how these traditions should change (most are bad, some are genuinely good, sometimes the worst ones are the best).

I mean, look at this weird theorem layout from 1940:

Thank goodness that style didn't catch on.

Instead of giving them the tools that allow adjustments, most print equation layout systems (and MathML in particular), only hand them spacing hacks. This may be fine for print where, in the end, you just want to put some ink on a canvas and no one can see your hackery. On the web, the structural damage is apparent and causes numerous issues (e.g., performance, accessibility). Compare this to CSS where we have benefited from a system that allows resets and customization clearly and cleanly; in comparison, a paradise of tools for designers, authors, and developers.

None of this is to say that there aren't interesting aspects of print equation layout that are worth a look for the web; something as basic as multiscripts or something as complex as stretchy glyphs. But bringing them in a way that drags the baggage of 100 years of print equation layout along is an obviously stupid idea (that can be explained but hardly forgiven).

Stretchy characters are also a weirdly terrible tradition: to stitch together glyph pieces to form something else. Of course it makes sense in a print setting - you work with what you got (just like you re-use typefaces left right and center). But what do people use stretchy character for? It is actually a very simple concept: a border. Yes, the specifics of the border style may carry some (contextual) meaning, but that's the case with most borders in general. The main layout purpose is very specifically to separate a chunk of layout that benefits from a border on some sides of its box. That's all there is to it. It's even in the name: these constructions are commonly called "fences" (mathml 3 even has an mfenced element, may it rest in pieces). A fence is a separator (ARIA pun intended), something to mark a border.

The MathML people want the ability to build stretchy characters to be limited to internal magic of MathML implementations in browsers, relying on low-level access to Open Type MATH tables without doing the hard work of surfacing this complex technology to userland. This technology has a somewhat problematic history in itself. The extension was developed by Microsoft exclusively for Word and only years later proposed to Open Type properly to help, really, the MathML people to make the case that "it's a standard". There are only a handful of fonts with MATH tables around because they are quite complex (read: expensive) to create.

Even ignoring the troublesome history, this approach limits a rather lovely concept used throughout design (just search dribble or codepen for curly braces) to supposedly "mathematical" markup and layout. It is quite silly especially since even basic uses case won't actually be possible this way (e.g., a stretchy brace around a list would still be invalid, even ignoring "semantics"). Any which way, we still need an alternative.

So why not rethink this in the context of the web? Why carry over a silly tradition (which is actually even worse: fonts will have multiple sizes of fences and those will be used if the box they wrap around is small enough). We have so many useful ways to indicate that something is grouped together on the web, we have traditions around those too (e.g., hover, focus, target effects, easily shaded or animated backgrounds to indicate depth), some of which are problematic for other reasons. We have a dynamic medium that allows you to only expose the gnarly information like this when a user actually requires it instead of on-the-nose all the time. We can also leverage the web's infinite canvas to build new traditions that serve users better. For example, stretchy braces do not increase by just nesting. They stretch only to match the contents, thus in a double parentheses situation (like ((...)+(...))) they will not grow, thus not actually represent the full complexity. That's understandable in the limited, fixed page sizes of print products. It also forces authors to do silly hacks to get additional dimensions should they want to communicate differently than tradition allows for. Maybe authors on the web want to do things differently from the web?

And of course it's funny how even if you insist on a fonts-based alternative there's already a solid alternative to build upon today: variable fonts can do wonderful stretchy things and can obviously solve this problem. Just without magic and with control in the hands of authors.

If you move away from the silliness that you need to reproduce print tradition, you'll find that you can solve more problems. For example, asymmetrical or partially colored stretchy constructions. It's perfectly reasonable to have a \begin{cases} environment but want the part before it on the same line as the first case. You can't do that with Open Type MATH tables. You can hardly do it in TeX.

The operator dictionary is another good example for this. Doing layout based on text content is an interesting topic - we've had years of debates around aligning table cells at decimals. For trivial spacing changes around elements - which authors want to customize anyway - it seems just a bit silly.

As related and similarly bad example are math alphabets. They are pretty mad to begin with; they are fontface traditions, captured as Unicode codepoints. It's understandable that mathematical writing and print traditions have abused fontfaces - it was a nice hack when all you have is moveable type. Enshrining them in Unicode does make some sense for print because (in a twist that should not be a surprise) it turns out that these glyphs have grown their own requirements. For example, Greek character glyphs tend to need special shapes for mathematical context than for Greek texts. In the mash up of technology limitations that is print equation layout these glyphs serve contradicting roles: be identifiably different but fit nicely with the Latin glyphs. This leads to really weird design quirks, see for example From Unicode to Typography, a Case Study: the Greek Script" by Yannis Haralambous (tip of the hat to Christian's wonderful Interesting Esoterica).

The recent addition of math calligraphic scripts to unicode (while unsurprising given the people involved) is weird to me because authors will not and do not stop. The other day, we had a paper with runes come through production. Should they add "math runes" to Unicode next? And what about David Griffith's script r, does he get a codepoint all for himself? What about math emoji? Also, what about math alphabets when they're in text mode - do they have special meaning, too? (Other than making horribly inaccessible social media profiles, that is.) More seriously, I'm not aware of any discussion of how the math alphabets are purely focused on supporting authors using the Latin alphabet. For lack of expertise, I have not written about non-European print equation layout traditions because I know next to nothing of them but from the little I know (e.g. the ancient W3C note on Arabic mathematical notation) I suspect they might make for an interesting post-colonialism study. [Aside, there is a direction where I could see Unicode actually help, just like it does with emoji: codepoints for actual mathematical concepts that are usually represented as a single grapheme. Indeed, the history of some math alphabets hints at this, e.g., the few blackboard bold symbols that are in the Basic Multilingual Plane. Just like we have a heart and a heart as card deck suit.]

It is worthwhile to note at this point that in recent years more and more TeX users (including the AMS) have been leveraging a re-interpretation of Knuth's rulesets by adopting xelatex which realizes equation layout using Open Type MATH Table fonts which, again, was designed for Microsoft Word when they built their print equation layout facilities alongside a first font from Tiro. While Microsoft and Tiro followed the TeX book there are noticeable differences, making it hard to know if the bug is in the font, in MS Word, or in xelatex.

On a slightly higher level, we have traditions such as labeled equations. They are surprisingly complicated in themselves. For example, most systems will provide authors flexibility in aligning or auto-numbering them, they may be mixed with other constructs (such as stretchy characters, e.g., in the LaTeX package cases or empheq). In some print equation layout systems they'll be realized via table-like constructs, in some via lower level primitives (horizontal fills, floats etc).

It also gets complex when you look to integrate equations into surrounding content. Print (equation) layout systems tend to treat equation layout as starting an entirely different context; the break is fairly absolute, both for inline and block layout (albeit in different ways). This doesn't fit well with the web's DOM and CSS's box model however. For example, equations tend to have break out of all context and align to some global property, in particular their labels. So a block equation in a list will simply break out of everything and potentially place a label at a far end of the line. This fits very badly into the web's layout model(s).

Next step up you'll find alignment across equations. A common feature in print layout engines with equation support (e.g., amsmath's intertext). Should we enable these kinds of traditions? Maybe; I do think this last one is fascinating. Should we do it by replicating print (equation) layout traditions? That sounds rather ludicrous to me.

Another problematic print tradition lies in inheritance. Traditionally, print equation layout will work as a reset. E.g., in TeX math mode will not inherit surrounding text styles or weight. Why? Well, because that would change the "meaning" because you're abusing style and weight: if you use a normal V and a bold V for two different concepts, then naturally in a bold heading you'll run into trouble. On the web, we see other traditions to surface; for example, people find it very irritating if a fraction is not bold inside a heading. A very real problem happens if a link starts or ends with print equation layout which is suddenly no longer styled like a link (e.g., neither blue nor underlined). That's a real problem for users (and possibly failing one or more WCAG SCs). Purists will insist - and they are wrong, at the expense of users. [Fun fact: that's why I personally prefer SVG with paths instead of characters/glyphs when reproducing print equation layout on the web. It's more honest to the print content tradition and happens to avoid many of these problems with ease.]

Of course there's yet another exception. In TeX for example, when you nest text mode inside math mode, that "inner" text mode can inherit. This means text in a piece of equation layout ends up, say, italic if the text surrounding the equation layout is italic (e.g., in a definition). Does this lead to confusion? You bet.

To recap

The problem of creating visual layout that matches print equation layout has been solved for well over a decade. It's somewhat ridiculous to suggest the web has a significant technology gap here that only a completely new set of markup and layout features can fix. Additionally, the theoretical structure of print equation layout enshrines problematic traditions that fit very badly, going against the web's grain.

This leaves non-visual-layout considerations regarding print equation layout, in particular (web and print) accessibility considerations.

Ambiguity

Let's talk about ambiguity.

A different example

To start, consider a different example, musical notation. In modern notation, a note is about as ambiguous on its own as a grapheme in text. For example, without a staff and other context the note itself doesn't tell you much. You might know the value by looking at just the note but even then this might not be enough (e.g., if you happen to miss a subsequent dot or preceding accidentals). Even with a staff, you might still have trouble, e.g., if all you have is a single measure you might lack the key. Similarly for dynamics. Also, it might be part of music that ranges from Opera to Jazz and you'll want to know that to adjust your understanding.

It's hard to know when you have all the relevant information. A piece may appear complete but lack information on, e.g., which instrument this is for, what other voices there might be.

And this is for a fairly limited set of notational elements.

Now expand this to the ever changing mathematical notation, then expand to engineering and sciences where there is no limit in generalization and specialization. [Of course, musicians can easily discuss a single note as passionately and intricately as mathematicians discuss a single number. But while this is much more interesting, it is unfortunately not the topic here.]

Some examples

a superscript 2 (e.g., x^2, f^2, \sin^2) - is probably some sort of "squared" thingie? But is that always true? How about A_1^2 or f|^2?
a binomial coefficient cannot easily be distinguished from a 2-dim vector notation - n \choose k vs \left(\begin{smallmatrix}{} n \\ k \end{smallmatrix}\right); similarly for Stirling numbers.
frequent use of space or punctuation to imbue layout with extra meaning or to avoid having to add clarifying notation (e.g., +a + -c + +b vs, say, ++b https://whystartat.xyz/wiki/Space_is_significant)- "(a,b)" - open interval? point in the plane? bilinear form in a Hilbert space?
punctuation in general. If it's in the middle, is it just a separator as in i, j \in \mathbb{N} or an operator as in 1-2 or 1.2? If it's at the end, is it part of the equation or actually text (a common sight in block expressions)
[a] might be the sign of a // [c, d] might be a matrix // A[b] might be a ring
|...| - absolute value, norm, determinant?
| - evaluation of definite integral (alternatively [ ... ] but that could also be a closed interval or a commutator)
what is the difference between \begin{array}{|c|}x \\ y\end{array} and \left|\begin{array}{c}x \\ y\end{array}\right| and how would you find out?
superscript -1. Is it an inverse? A "1 over something" ? A reverse operation? Some kind of transposition? An index for something enumerated by integers?
"d" in (a + bx^3 + cx^2 + dx = 0) vs "d" in \int f(x) dx
uppercase delta \Delta and the triangle symbol \triangle can look surprisingly similar depending on the font.
invisible operators (U+2062, U+2061) - where to even start?

For more "inspiration", check out https://whystartat.xyz/wiki/Category:Ambiguities

An absolute favorite of mine is the Legendre symbol \left({\frac {a}{p}}\right) which is obviously easy to confuse with a fraction that happens to have parenthesis (e.g., from reducing a more complex expression). It introduced before the year 1800 and yet, apparently, there's no dominant way to speak it, see this MO discussions.

We can also go the other way around. We can have the same voicing for different notations. |A| and det(A) may read "determinant of A"; both exp(x) ande^x may read as "exponential function at x". How would a non-visual user know which notation was used when someone said this?

A separate problem are the math alphabets, i.e., the abuse of using typefaces to identify different, context-dependent concepts. It seems so easy to - visually - discern italic from normal, serif from sans-serif, gothic from Sütterlin - well, in truth it is not that easy but here we are anyway. In addition, foreign alphabets are abused so much that mathematics has become reliant on particular typeface designs as mentioned earlier.

For non-visual users, these then have to be differentiated (otherwise you're no better than a "read more" link) and the noise ("Upper Gothic X") is immense - or more often: the silence is immense.

And then we haven't even opened the box of notation that's specific to engineering, physics and chemistry but realized using print equation layout.

Semantics or layout?

There's a moving goalpost (or perhaps gaslighting) issue that I've encountered many times: should assistive technologies (AT) provide information about layout or semantics when it comes to print equation layout?

A common stance in the general accessibility community is: it's not about special treatment, it's about equal treatment. What does that mean for print equation layout? Equal treatment would provide precise information about layout.

But that's not what users actually expect (or are accustomed to). Instead, "semantics" are desired. Take this simple example:

f: X → Y

Should it read

"lower italic f colon upper italic X right arrow upper italic Y"

"f maps from upper X to upper Y".

The latter is certainly easier and happens to be how humans speak.

However, print equation layout does not offer such semantic information. Therefore, solutions turn to guesswork aka heuristics.

These heuristics work barely for school-level print equation layout and some, more "stable" parts of college education (e.g., engineering). But they easily and frequently fail even in these areas.

To come back to our examples: yes, that superscript 2 is very likely "squared" but then again it is, inevitably, far from always "squared". As usual, xkcd has you covered on the subject of "2" (cc-by-nc 2.5):

Beyond the evident lack of semantics in print layout per se, the second problem is the lack of contextual information. While MathJax and its sister project Speech Rule Engine did some Simons-funded research into the problem a few years back, no existing solution takes even other equations into account (e.g., using the guess for a variable that was used earlier).

Heuristics definitely do not try to analyze the whole document context to trigger specialized heuristics despite the many obvious advantages. Just by field, you can differentiate a lot - X\perp Y in statistics vs geometry, Bra-Ket notation in quantum mechanics, arrows in category theory, or (again) the Legendre symbol in number theory. Even outside subject-specific traditions, this could help, e.g., if you can identify a definition, it helps quite a bit later on. You know, like a human reader.

The simplest examples can show how tricky this is. Think about f(g+h)(x). You might agree that there's a good chance that there's a function application (at x) happening but it could also just be a multiplication with x after some calculations within the parentheses, who knows. Fairly limited contextual checks can probably help out here but without them we're lost.

What's worse: today's heuristics also often do not work within an equation, e.g.,

\operatorname{det}(A) = |A| = {\begin{vmatrix} a&b \\ c&d \end{vmatrix}} = ad-bc.

Speech Rule Engine's heuristics (the best around) will identify the middle part as a matrix determinant but still treats |A| as absolute value. And in fairness, if we didn't have the first part, that A might still be some poorly named scalar.

A key feature of the heuristics is surprisingly boring and complex: grouping things. Where do subexpressions start and end? Since print equation layout is inherently non-semantic and built for author convenience, most authoring does not (have to) care about grouping the structure well. Who cares, you'll work it out visually and from context (thanks, PEMDAS!).

Here's a pretty normal, real world example:

\begin{aligned} & u_2(t,x) \\ & {}= \frac{2}{t^{1/2}}. \operatorname {Re} \left( \left( \tilde{\Psi }(y) ( \log t + \log \sqrt {1-|y|^2}) +\tilde{\Phi }_2 (y) \right)\right. \\ & \quad \left. \left. \times \exp \left(it\sqrt {1-|y|^2} +i\Psi (y) \log t \right) \right)\right|_{y=\frac{x}{t+2B}}\\ & \quad + O(t^{-\frac{3}{2}+c\delta }) \end{aligned}

Note how the parenthetical expression starting after Re ends in another table row. This is perfectly understandable authoring from a print author perspective - your line is fixed and limited, you'll need to break things up. Since you're in print (and you want alignments to work), you push pieces into a table.

How should heuristics work out this grouping? The answer is: probably not.

Another interesting example is the treatment of horizontal and vertical space. "Clearly", if you find $(a \quad b)$ and $(a\ b)$ near each other, they're different, right? But what can heuristics realistically do here? Especially, when they only work on a single expression? [Aside: I give the source here - SRE would indeed give the same descriptions for these expressions.]

Print equation layout uses positive and negative space abundantly and often confusingly. Some authors like things spaced out where others like them tightened up. This makes it impossible to leverage space in heuristics. As a consequence, space is not to be messed with. [This also creates problems if you're mistakingly thinking print equation layout is text - because WCAG 1.4.12 would then force you to allow users to change word and letter spacing. In the end, space is significant.]

Similarly, one of the most important visual tools is completely absent from non-visual presentation: the size of stretchy glyphs. This is sad because the whole reason we use stretchy constructions in the first place is to make layout more accessible for users. Stretchy constructions indicate both grouping and the complexity of subexpressions. Authors frequently use it to explicitly highlight subexpressions because they realize their complexity. And yet despite their focus on layout, non-visual presentations provide no access to this information; there is no representation of stretchiness let alone the relative dimensions. The most you can hope for is something like Speech Rule Engine's ability to summarize a subexpression (along the lines of "a sum of 16 summands"). Good grouping informs users but there's a significant benefit to a stretchy character: the larger it is, the more crap is going to be inside whatever it groups - to the point where they can be downright scary. They are critical information.

Frequently, people (especially in the MathML community) promise magical semantics from print equation layout. Yet when faced with the obvious, plentiful, and unsolvable problems, the goal post is casually moved to "you need to give layout information" only so that when you provide layout information, people go back to "oh, that's not the idea, you should provide semantics".

Unfortunately, many people believe this empty promise.

Problems from print layout traditions

There are both theoretical and practical problems arising from print layout tradition.

Theoretical

Besides the fundamental issues outlined above, print equation layout has some traditions that add another layer of problems.

For example, we go back to the concept of moveable limits. The expression

\sum_{i=0}^\infty

would, at plain view, indicate we get Σ with a subscript of i=0 and a superscript of ∞.

However, as you might recall, this depends on the mode. In inline math mode, TeX will lay this out as described. In display math mode, the sub- and superscripts will become under- and overscripts. This is, again, convenient from an authoring perspective - you can switch modes as you like, thus macros / copy&paste are much easier.

This automatism wouldn't have to be built into a system from a purely functional/layout point of view; it's authoring convenience. Of course, MathML builds it in to be compatible with print equation layout traditions for which it was designed. In fact, it adds the (ever growing) operator dictionary to describe various layout rules like this instead of focusing on just the necessary parts of layout.

This means, we have intentionally ambiguous layout instructions and that's tricky for heuristics to work - yet another context to check. [For once, that's not terribly weird for the web but media and container queries separate this much more cleanly - and they are separate from any semantic interpretation.]

So what do heuristics with such ambiguous layout? They cheat to make this ambiguity go away. When voicing, simply say "sum from ... to" - tada, problem solved! Of course now you have 3 new problems: a) what if it isn't a sum? b) what if it is a sum but using a different character for the summation operator?c) what if the authors want to use side- and overscripts to mean different things?

Practical

Given the nature of print products, authors are prone to abuse layout instructions. In the end, it has to look good, who cares how they got there.

For example, authors frequently make their content work purely visually, placing elements side-by-side, relying on the fixed positions of print layout.

They will use table layout to split content apart so that it can fit page dimensions (e.g., complex breaks across lines), thereby making it impossible to reflow in a dynamic environment and group correctly in anon-visual semantic analysis.

Reversely, they will space content out to fill a page, impeding reflow or resizing in dynamic environments.

They may "accidentally" rely on TeX's defaults, e.g., consecutive display math blocks may align if those blocks are generally centered (rather than explicitly grouped by aligned blocks); yet with different alignment defaults, the alignment may fail unexpectedly. Similarly with multi-line environments or other manual and/or automatic line breaks.

All of this makes it even harder for heuristics.

I want to stress that these are not bad practices - many of them have long and detailed traditions (e.g., commutative diagrams). They are simply print traditions with no consideration for the web, let alone accessibility.

interlude

On the web (and in many publishing content / production pipelines), abusing HTML markup is frequently favored (i.e., text superscripts, tables, simple vertical alignments) alongside low quality (usually GIF) rendering for inline math and high-quality (usually PNG) for display content. Frequently, only the 'problematic' parts are rendered as images leading to poor layout (e.g., tables with some equational content and a text label in lieu of properly labeled equational content, HTML scripts with graphics in either base or script position). Accordingly, content providers have a long history of "optimizing" their content for these approaches (e.g., STEM publishers have really terrible XML workflows stripping out as much complex markup as possible). That content is in some ways lost forever (because it would be cost prohibitive to re-key it).

Non-visual print equation layout

In several countries, print equation layout has some form of non-visual accessibility tradition.

The US traditions are dominating in this context since MathML was developed largely by US companies for a US audience.

For print traditions in the US, the most well-known methods for non-visual presentation for equation layout were developed by Abraham Nemeth - nowadays known as the MathSpeak ruleset for voicing and the Nemeth Braille notation for tactile presentation. A critical point is that both sets were designed for human creation. MathSpeak, to enable a non-expert reader to read mathematical publications to blind users; Nemeth Braille for a specialist braille transcriber to transcribe (and a trained reader to read).

If you can catch him, Volker Sorge (who implemented support for both as part of Speech Rule Engine and MathJax) is happy to go into much detail telling you how neither MathSpeak nor Nemeth Braille can be produced from input without extensive heuristics. The rulesets are in fact not properly specified and can be contradictory.

My impression is that this is by design - these rulesets could assume a human orator/transcriber who can adjust rules on the fly to enhance the simplistic heuristics. Even without understanding mathematics, it's fairly easy to make a change on the fly, e.g., after initial feedback from the user that in this particular pattern the superscripted 2 is not "squared" but, say, a row or column vector.

It's probably worth noting that Nemeth Braille has not been revised since the 1970s (though a revision has finally started a few years back). It is a specialized Braille notation that describes visual layout and only partially linearizes the output (e.g., fractions and scripts are linearized while tabular structures are not), leading to many problems as contemporary refreshable braille displays for computers are single-line devices. And of course, if you're not a Braille user, then none of it helps you.

heuristics are bad, non-recursive ones worse

At this point let me point out that heuristics are something that web accessibility eschews; heuristics are only encouraged to recover from errors (e.g., turning invalid documents into a valid ones, making sense of authoring or connection errors, dealing with legacy code like table layout).

Accessibility on the web revolves around the so-called "accessible name" which is calculated for each element in the document. This calculation is rather complicated but at the heart of it is Step F: the accessible name of a node is accumulated from the accessible name of its children. From the end users point of view, you tend to get some additional information (such as the role of the element) added by AT. And that's about it.

Print equation layout frequently does not allow for recursive naming. The heuristics necessary to guess a non-visual presentation have to frequently analyze large subtrees to make sense of things.

A simple example is binomial notation, n \choose k. Without the parentheses {n \atop k} is not binomial notation and yet when a binomial is voiced, you wil not hear the parentheses. If you were to calculate the description recursively, you would have to decide on the description of what's within the parentheses before you get to them. How can you decide what to do at that point?

To make things worse, in many print equation layout systems this notation is realized via fraction primitives by removing the fraction bar (e.g., TeX, MathML). If you were to calculate the description you now have to decide what to do with a fraction that lacks a fraction bar. Now add to this mix the Legendre symbol \left({\frac {a}{p}}\right).

We can also go back to our script examples from earlier - for 2 and -1. While 7^2 = 7\times 7 and 7^{-1} = {1 \over 7}, we have \sin^2(x) = \sin x \times \sin x yet \sin^{-1} x is actually the inverse function at x, \arcsin x. If you tried to capture this, you'd have to ignore whatever you're creating for superscript -1 but still know about it to interpret the other part.

MathML is even worse for this, as it is a stratified format for rendering purposes. With TeX you can at least have author macros to try to make sense.

Authors, especially on a research level, invent layout that requires new heuristics - both large and small. For example, they create custom glyphs (such as blackboard bold Greek characters) or they invent notations as grapheme-like tikz diagrams which then get used like any other element of a larger expression.

Historically, such inventions coincide with naming. Jeff Miller collects several historic examples (such as nabla). Yet, naming is not always invented alongside notation if you recall the Legendre symbol once again. [Authors also regularly just hack layout badly (e.g., \mathop{l\overline{og}}) but that's technically speaking a different problem.]

Authors also intentionally hack things to work around traditions. A while back at work we ran into someone apparently abusing amsmath's sideset macro in something like \sideset{}{^*}{\sum}_{a\mid A_y}. We only noticed that because MathJax had improved sideset to be more accessible which changed the rendering (and introduced regressions). Upon investigation, it turned out we had a lot of similar papers. Even better, all of these authors used sideset consistently to get a superscripted sum that would still have a "movable" subscript.

In other words, authors clearly want a certain type of layout that goes against traditions; they have good reasons to do so. Heuristics trying to make sense based on those traditions suddenly break as well.

This is a perfect example how traditions hold us back. They are too rigid - you can't have "partially moveable limits" without resorting to hackery. And once you have that kind of a hammer, everything looks like a nail.

MathML

MathML does not provide sufficient quality, neither in theory nor in practice.

visually

Practically: visual rendering in browsers is poor, buggy, hard to work around. It is not maintained by browser companies and never has been.

Bugs range from mildly annoying to problematic to terrifyingly bizarre. I stopped noting them publicly; it's too much fun to check every few years that even basic bugs do not get fixed or even reported.

Theoretically: visual rendering is not actually specified (thought here is a new effort to change that, making all existing MathML renderers invalid and breaking content without care).

This is where they are after 25 years of MathML, 15 years of MathML in HTML5, ~10 years of MathML "support" in Firefox, ~8 years of MathML "support" in Safari, neither of which are due to actually Mozilla or Apple spending resources but co-opted volunteer efforts. No amount of Igalia newspeak will change that fact. [Apologies to all the wonderful people I've met from Igalia - the MathML people are not a good look for you all.]

Optimistically, I'd wager it will take a good decade to get reliable visual rendering quality. Certainly, CSS, SVG, HTML and ARIA will all move faster and help tools like MathJax even when they're not trying.

In any case, the handful of people behind the implementations seem unable to grasp what the real life problems are so they live in their little bubble insisting that repeating "but it's a standard" does anything for users.

non-visually

MathML captures abstracted, shared concepts of print equation layout traditions. It suffers the consequences in terms of non-visual rendition.

Practically, MathML support is poor among screenreaders and is non-existing in"simpler" assistive technologies (with way more users) such as "read aloud" features in browsers, voice assistants etc.

Only three screenreaders formally claim support and their support is limited, buggy, and of poor quality - and the pieces are minimally maintained by AT companies. Brian Richwine's talk at ATHENs documented many cases, e.g., how some ATs don't voice "end fraction/script" causing confusion (my favorite part was the anecdata form Apple support saying there's just not enough users to warrant fixes.) [Though admittedly, I have yet to see someone exhaustively testing them. Nobody is paying to test that, not even publishers who have a vested interest because the "experts" say "just use MathML".]

In addition, the user experience of what little support screenreaders provide is often questionable, e.g., JAWS will not read larger MathML fragments but just announce the presence of a math fragment; users have to then open a separate viewer application to actually read and explore the expression. [Fun fact: this is called "Math Viewer" and actually uses MathJax for visual rendering, leading to significant differences from the browser.]

Similarly, VoiceOver on iOS will open pop-ups for exploration of MathML fragments, creating a separate context and confusion. NVDA needs third party plugins to do anything and has no visual component (i.e., highlighting while exploring)leaving non-blind to guess what is going on.

I don't think it's enough accessibility support in the sense of WCAG conformance.

Theoretically, MathML cannot be accessible in the web's sense. Heuristics must be applied and now you (frequently) have 2 problems - your MathML must be hacked to work visually and then you might still have to hack around heuristics gone hawywire.

Back when I led the W3C Math on the Web Community Group, we had fairly productive discussions even with the MathML people and we had strong consensus (with them) that MathML inevitably requires extensive heuristics to provide non-visual (voice, tactile) renditions. Nevertheless, the same people later turned around again in wider W3C discussions, going back to the spurious claims that MathML is the only viable accessible solution.

The sad part is that the W3C's TAG fell for it (and will fall for it again). My guess is because they just don't care to dig deeper - and why should they? Nobody wants to and anyone sensible will just use MathJax.

Other aspects

localization

Another argument sometimes mentioned by MathML people is that MathML can be localized automatically. While that's true to some degree (e.g., Speech Rule Engine can create MathSpeak in a few languages), the benefit seems a questionable if the locale does not match the contextual language, say, coming to an article in Arabic and getting an equation localized in Korean. At the very least, I'm unaware of research substantiating a positive effect.

In practice, the available localizations in AT is quite limited and pretty undocumented. This is even worse for Braille.

There are obvious obstacles for localizations. Many equations contain some text elements which would not be localized. Similarly, common terminology has different localizations/abbreviations. For example,

sine as "sen" (ES, IT) or "sin" (EN, DE)
in geometry SAS/ASA/SSS (EN) vs SWS/WSW/SSS (DE)
sin^-1 (EN) vs arcsin (EU) (etc)
repeating decimal notation (above, below / arc vs line)
open intervals as (...) or ]...[
any operator names the authors may introduce.

While layout-relevant terms might have suitable automatic translations (e.g., start/end fraction), layout traditions can also differ across languages (long division, matrix notation, integral notation).

Nobody stops to ask: should we do this? As print equation layout is fairly meaningless without lots of context, it's difficult to imagine a reader that would read content in a language they don't understand enough to make sense of generic information like "begin/end fraction".

The "Don't worry about it" argument

The MathML community likes to say "don't worry about the complicated markup, MathML is something that should be generated, not handwritten"

This is a strange statement full of falsehoods.

For example, MathML is also advertised as "something you can edit without understanding it", e.g., while copy checking content, if an editor spots a symbol that does not match the (author) PDF, then - thanks to the structured XML markup - they can fix it without understanding.

This marketing example also indicates two problems: conversion is not "easy" but riddled with errors, and having an "agnostic" markup is actually a boon here - to any production person trying to fix a mistyped macro in TeX sources: I see you.

But of course the same would be true for span soup since the markup really doesn't matter much here. Though actually, semantics tend get in the way while layout information would be useful. (Again, it makes actually sense for XML production to have a layout focus.)

Similarly, as described before, MathML supports "moveable limits" (including magical dictionary-based behavior). These features show how important it was for the MathML community to be compatible with authoring traditions instead of focusing on the "pure" layout. If you generate it anyway, why not make it precise and avoid this?

Finally, this line of argument is self-defeating. If the whole point is to convert from some kind of "real source" to MathML, then it's just as natural to convert from that "real source" to HTML or SVG.

Why have an extra markup system that requires an extra rendering process when you can just use what's already much better supported?

Limited support for MathML

Among screenreader, only Apple VoiceOver and Vispero's JAWS claim to support for MathML. In reality, the implementations are full of bugs and gaps. Bug reports are rarely responded to, far less fixed. Apple has essentially communicated "it is what it is".

Nemeth Braille support has relied on the liblouis library - which recently discontinued its support of it; liblouis only ever converted MathML to the Nemeth/UEB and German/Dutch/Flamish families braille notations - and it did so quite poorly. [No, the new "cat" on the block doesn't count either. Old wine, new bottles, just as rusty.]

More importantly, no "light" assistive technology supports MathML,e.g., "read aloud" features in smaller assistive helpers, in epub reading applications or in voice assistants (Siri et al, e.g., when reading out web content). This lack of support impacts the largest portion of users, since these tools are much more widely used than the more advanced, full-fledged screenreaders like VoiceOver, JAWS, and NVDA.

What do we do to address this inherent lack of accessibility?

Print equation layout is inherently not accessible in the sense of the web. MathML is a red herring as it (at most) aims to reproduce accessibility traditions from the print era but inherently cannot reproduce it well since those traditions relied on human operators(orators or transcribers). It also only works for English and, it seems, really only for Nemeth users.

I approach such problems by looking for guidance regarding the web in general. The web's grain, if you like.

The web's core principles

"Users before author before implementors before theory"

From the W3C design principles:

User needs come before the needs of web page authors, which come before the needs of user agent implementors, which come before the needs of specification writers, which come before theoretical purity.

What do users want from print equation layout? And what do they actually need?

We simply don't know.

Yes, most countries do something at the school level. But actual research is barely existent; what little there is, is usually limited to small groups and all the biases of simplistic educational studies.

My impression has been that there are too few AT users among the mathematical community - support is so poor, few make it to college let alone research-level. Accordingly, few have both the skills and the luxury to think freely about the problem. (T.V. Raman is one of the few exceptions and I'm still grateful he gave Davide and me the opportunity to work with the ChromeVox team for a bit - and to meet Volker Sorge who was on sabbatical at Google Research back then.)

Anecdotally, the most consistent feedback I've seen in educational studies comes down to "I want it to be read like my teacher/prof reads it in class". That sounds an awful lot like users want to have what authors want to say, and matches the core principles pretty well. It's one of those "did you need a study for that?" moments (yes, we do) and yet it shows how flawed the current approaches are. Yes, it's a common paradigm that authors should enable AT users to customize their output. But if you take away the author's ability to express themselves, are you really helping users?

Assuming I'm wrong in my assessment for lower level equation layout: for mathematically heavy research material? There's absolutely nothing to base an opinion on.

Those users probably want TeX annotations because research-level users very likely know TeX syntax. But is that what they need, e.g., when exploring a complex expression? [Fun fact, some European communities favor presenting TeX strings in Braille - starting from first grade.]

interlude

This discussion is usually skewed by two major factors: US users and blind users. As with most of the web, the US dominance has led to people ignoring other factors. If it works for US users, it's fine, amiright. So the limited MathSpeak and Nemeth implementations in MathPlayer were enough to woo the US blind community and its influence wooed the rest.

To take an example. Blind users (especially students) need to write assessments using the tools their classes require. If they need to use MS Word, they'll need something to paste into Word. MathML was originally conceived as just such an exchange format for print equation layout - abstracting the minimal common ground so that each engine could reasonably present it.

However, the authoring problem requires conversion. This easily goes haywire. For example, Libre office's "Math editor" digests MathML but converts it into its own linearized (~Asciimath like) syntax. And while it includes MathML in its file, that MathML will be generated from the linearized format. This is a process full of bugs.

MS Word does the same thing but has a custom XML format for print equation layout. Same approach, same problems. It doesn't help to add more formats (such as TeX-like input).

MathML-based wysiwyg editors exacerbate the problem (as wysiwyg is wont to do) since they cannot easily discern logical structure of input // do not make it apparent to users that they're doing something really terrible (e.g., inserting random whitespace, flip-flopping font choices).

However, if you are a blind user who knows Nemeth well, then that conversion process might just work sufficiently well for you because things were designed just for this use case. Ergo, you will say it's great and your influential community will prevent any discussion about whether this is actually a good trade off for other user needs.

At ATHENs earlier this year, all speakers I heard included at least one statement along the lines of: "our successful students are those who know Nemeth". Clearly, it's good that Nemeth helps students succeed but in the words of a spaceship captain: "Not good enough, dammit, not good enough".

The reality of all users is: the web is fairly bad at being copy&pasted elsewhere. And that's an interesting, complex, and very broad problem. I'd love to see more movement in this direction but I don't think it makes sense to make copy&paste to MS Word a requirement for realizing print equation layout on the web when nothing else on the web can do this. (Pasting SVG into Word anyone?)

[Repeated conversion tends to be lossy. From my limited yet painful experience of editing conversion results, I doubt it will be great if you went back and forth across print equation systems for a bit; if I cared enough, I'd test that hypothesis.]

Ultimately, Braille is a print technology, and in the context of a refreshable display there isn't even a page, let alone a multidimensional presentation that a dynamic webpage can provide. Again, this is a broad and interesting problem, again print equation layout is not a great case for special treatment.

Similarly, Unicode math alphabets are a print technology. It makes sense in a print context to separate out codepoints that have very special fontface requirements. They work fairly terribly in terms of non-visual rendering. For example, I once tested the string ϕ(x)=∑ₖⁿ xᵏ and it came out as "open paren x close paren equals n-ary summation n x" (JAWS) or "x equals n-ary summation superscript n x" (NVDA) and "phi x equals to the power of n x" (Chrome speech synthesis). Yay us. Now you might argue that in the right markup, voicing could be different, e.g., automatically more verbose. And that's an excellent point - but I would couner that math alphabets are far from the only Unicode characters with these kinds of problems - emoji for one are even more prevalent than math alphabets to fill account names or social media postings - and a better solution would be a general one, not "math markup".

Another quirk of the accessibility traditions around print equation layout is what layout information is exposed. Font variants, upper/lower, under/overlines are all exposed - yet color is not. Why? Likely because print traditionally doesn't use (a lot of) color out of the pure necessity that color print is not readily available. But that is not stopping authors; color is a useful and beautiful tool. Of course, it's "obvious" that non-visual media won't present color well but neither will it present different typefaces well - and somehow that was ok with everyone. This is particularly strange, when you find dedicated syntax for "mathcolor" with no effect.

What to want for authors

I want to give authors the ability to express themselves well.

And authors want to do that.

Just look at any "list of symbols" in a monograph, e.g., from this combinatorics book

Despite using very ambiguous terms, the authors clearly have a strong intententions.

I want authors to bring their intententions into the process.

We can (barely) do that with Speech Rule Engine. We cannot do that with MathML.

What do you want for your users?

I want to take responsibility for my content.

I want to provide high-quality consistent, robust rendering for all users.

I want to provide visual layout that is robust against font issues (especially onepub).

I want to provide speech and braille alternatives.

I want to provide consistent, reliable, robust exploration.

I want to be able to fix issues.

I want to be able to provide additional features (e.g., convenient access to TeX sources, customization options).

What do you do?

I usually provide high-quality visual rendering using MathJax's SVG output, alongside the high-quality heuristics in Speech Rule Engine to generate non-visual rendering.

I also generate fully explorable layout and descriptions that are the same for everyone everywhere. They are robust enhancements with solid fallbacks. If there is a problem, it is my fault, not the user's. And if need be I can fix it in post.

I tend to use the MathSpeak ruleset for descriptions meant for voicing because MathSpeak doesn't attempt to be as "smart" as other available heuristics (which target school level mathematics).

I use Nemeth Braille because content is primarily in English. Arguably, Speech Rule Engine doesn't have any support for other Braille output so I accept that I need to use Nemeth for other languages. I'm not beyond using TeX as quick alt text. If I want more, I'll try to help Speech Rule Engine get funding.

I make sure that speech descriptions work with all tools - from full-fledged screenreaders and simple read-aloud tools. If there are issues (e.g., a recent VoiceOver on iOS regression I ran into), I can work around it.

Yes, the ARIA technique for exposing Braille alongside speech-optimized text is cutting edge and not widely supported yet. But they went from editor's draft to full browser support in 2 years and they help much more than just print equation layout.

I often embed TeX sources that automatically copy&paste when you select things; it's a nice bit of magic under my control.

I set things up so as to allow for additional options to expose various formats (e.g., populating everything with TeX or loading full-on client-side MathJax).

I do so with intent and knowledge of the problem space, the capabilities of the solution I built and the flexibility to help users.

interlude

Some good old whataboutism:

What about electrical engineering diagrams? Same garbage print stuff but at least nobody pretends to cook up markup. Take diodes - their symbols for their types are basically letter like. Shouldn't we have those in Unicode?

What about chemical markup? ChemML is another wonderful XML abomination.

What about, again, music notation?

Should all of these be first class citizens on the web, with separate rendering systems to adhere to their particular (print) histories?

Tangents

copy & paste

The MathML community will frequently point out how "easy" it supposedly is to copy&paste MathML around.

The reality is that extremely few applications accept MathML, it's a cumbersome process, and MathML is dropped in the end, converted to whatever internal format is used by the target application.

The primary reason why this myth holds is that MS Word digests MathML. However nowadays, MS Word also accepts (reasonable math mode) TeX when pasting into "equation fields" which shows both the limited usefulness of MathML as well as the pervasiveness of (math mode) TeX. And of course Word uses its custom markup syntax internally (and its home-grown technology for braille and voice rendering instead of whatever ATs provide).

More importantly, while copy&paste is a worthwhile challenge, it is also a general challenge with web content. Copy&pasting website content into any kind of text editor (word processor, email application, TeX sources) is very poorly supported in general. While headings might survive, even something as simple as a list can break, tables usually break and "true"web design (e.g., CSS grids) does not survive. Even (inline) SVG content is not supported widely (despite being a well-supported, self-contained graphics format).

This is not at all surprising given how most of the targets are meant to create print layout - and struggle with web layout. Why then should equation layout on the web be less complicated than the rest? Instead of insisting on broken technology (like print equation layout and specifically MathML), it would be better to think about the more general problem that affects everyone. We have to anyway and a niche solution does not move us forward (yes, HTML v3 did have it wrong).

authoring and recreating

Related to copy&paste is the area of authoring and recreating print equation layout.

Let's go back to the example of matrix determinants. Both det(A) vs|A| would be voiced as "determinant of A" yet their authoring is quite different.

Sometimes, readers have to (re)create content to match someone else's example (e.g., in any educational setting). Again, this is a general problem (color vs colour). For text, AT usually provides character-by-character navigation (incidentally, I'm not sure about graphemes more generally). For print equation layout this is naturally trickier: on the one hand, we come back to the idea to "just" give layout information, on the other hand layout may be insufficient (e.g., table-like).

Voice input/output

Voice input is becoming an important aspect of how users interact with web content, both using AT (e.g., in combination with Dragon NaturallySpeaking) as well as using general purpose tools such as smart assistants (Siri, Alexa etc).

Despite funded efforts, there has been not even light progress for enabling print equation layout by voice input.

Audio and Video Captioning

Providing accessible audio and video media requires captions, in particular transcripts, subtitles, and audio descriptions. For content rich with print equation layout, automatic conversion requires voice recognition, image analysis, and conversion to speech (e.g., a mathematical lecture with equations on a blackboard that the speaker discusses). Since the standard formats (e.g., for subtitle functionality in a video player) require plain text, MathML is of no use here even if it were an adequate fit for how people speak. In addition, there are no tools nor traditions around "speaking equation layout" in such settings.

Reversely, there's audification. Take Google's AI-driven audio book narration. I doubt it'll do print equation layout anytime soon (by which I mean: ever).

Search

Despite decades of research, search around MathML has never provided tangible benefits despite clear user needs to discover existing results both to avoid and identify duplicate research. This is true for all of the ever expanding STEM fields.

Springer's discontinued LaTeX search or zbmath's MathML based system are good examples of this failure. A sizeable research corpus revolves around analyzing MathML fragments, yet no actually useful system is available.

Progress

Here's a very personal opinion: the MathML people have actively worked against progress for equation layout on the web, and STEM content as a larger issue.

Why do I say this? Because anyone who has followed this area as closely as I have in the past decade or so (which, alas, is barely a handful of people) know that it's the MathML people who insist that MathML must be implemented at all costs, actively damaging other approaches, even approaches with better results for users.

To the purists, MathML is more important than users, authors, and browsers.

As a result, projects that made actual progress (like MathJax, mathlive, hell - even KaTeX) are in danger of losing support from their communities because people are being told "you must use MathML". [Don't get me wrong, I can gripe about these projects, too, but at least they move.]

All the while, there's no indication that MathML development will speedup at all, even if Chrome ships the new, cut-down, incompatible spec.

So if you ever again complain about the lack of improvements in this space, blame the MathML people. They are to blame. We could have made many small improvements to the platform that would have benefited everyone if the MathML people hadn't killed everything that didn't suit them and drove away anyone who didn't agree with them.

Reversely, MathML is being held back where it actually shines - XML print production. There's so much useful print equation content to consider for XML workflows (tikz alone!); extending MathML would be natural here. Instead, even the XML people get nothing (or worse - a cut down spec they can't use until JATS adopts it).

I wrote earlier that "web equation layout" is not something that exists. So you might say, what's the problem? I think the problem is that we might get stuck, in a bad way. Here's quote from On Mathematical Symbols in China (again, thanks to Christian's Interesting Esoterica).

Ancient Europeans struggled with fractions because of the use of tedious Roman numerals, so ancient Europeans were weakest in algebra. However, when Indian-Arabic numerals were introduced into Europe, four operations in Europe were simplified. [... As] a result, algebra developed rapidly. Similarly, China's ancient mathematics failed to form a perfect symbol system and were stagnant because of the limits of the counting rod, resulting in not only a lack of the axiomatic system, but also the absence of symbolic algebra, analytic geometry, calculus and modern variable mathematics in China. Naturally, to move forward it was necessary to break barriers and to get in touch with international standards and to forward to symbolic mathematics. This shows that the extension and development of any prosperous culture cannot be confined to the boundaries of that nation, and it is hard to create a complete and rich symbol system independently and separately from the influence of other nations and the outside world.

I think the web should not be confined to print equation layout. We should not want to get stuck replicating a limited medium in a much more versatile one.

MathML is the zombie of web standards. Each decade brought 1 additional, differently broken implementation (first Firefox, then Safari, soon Chrome). The new spec for Chrome will render a lot of existing content unexpectedly or not at all, but it will be the author's fault - how clever of them. XML people will have even more "fun" getting their content to work on the web. (Or, more likely, just continue to use MathJax which just keeps getting better instead of worse.)

Bugs in those three disjoint implementations are, of course, aplenty; I find a new one every time something forces me to look at native MathML implementations, some are ridiculously bad (wiping out HTML content is one of my favorites) and most bugs go for years without bugs even being filed (remember when Safari didn't draw fraction lines reliably for over a year? No, you don't because you don't use it and nobody even filed a bug).

For the web, the best case scenario is: nobody cares and nobody uses it. It will become the new font/center/marquee or whatever.

Even if the MathML people were interested in moving the layout facilities into CSS, we'll get, at best, something like CSS tables. If you don't know the "joy" of working with CSS tables: to this day they are not able to suitably realize HTML table layout, they were super buggy for the longest time and are still buggy in many edge cases (despite years of active improvements). But most of all: people realized they actually needed something completely different, i.e., CSS grid, flexbox, and container queries.

Accessibility wise, the shoddy implementations in JAWS and VoiceOver will continue to barely help English speaking blind users. Soiffer will try to push his weird MathPlayer clone into every AT so that he can continue to claim patently false things. No "intent" in the world will make print equation layout accessible in the sense of the web. Nemeth users might eventually get what they want - the rest? Not so much.

In other words, users will continue to find themselves out of luck. But hey, "it's a standard". They made you beg for it, then they begged you for money (instead of Apple and Google); now you have something you don't want, something that doesn't work and something that cannot work. It's what you wanted, a "standard".

I think MathML (this of course means: its community) is fundamentally not willing and not able to be a positive force. It's "MathML or die" all the way. The zombie standard continues its shambling walk.

interlude

The future will be different from what you expect. The other day/month/year, our oldest started learning the piano. Of course, this means learning musical notation; it's one of the benefits, I'd say. But it's (inevitably) hard to motivate.

So being a stupid internet person, I searched around for fun apps that teach the piano. I found apps that, shockingly, didn't really use traditional musical notation. Instead, they had some barebones resemblance (I'm guessing to enable later learning) but they took things in another direction by leveraging the medium, i.e., modern mobile applications.

These apps would listen via microphone, give immediate feedback on correctness in a range of properties (dynamically increasing tolerance for errors). They would use their infinite canvas to provide just-in-time context in large visuals to keep silly younglings focused and unconcerned about length. They would play along simple training material providing more complex arrangements so that even an early learner would not be facing dull pieces of repetitive notes. They would immediately get you to jump into contemporary pop songs, sync highlighting along the way, bringing enormous satisfaction. They were just generally being impressively awesome. None of it with traditional musical print notation.

Quo vadis?

So where do we go from here?

I'm not saying you shouldn't put print equation layout on the web. I do it for a living; knock yourself out.

I am saying that you should consider two things when you do.

even mildly complex print equation layout is simply a bad type of content.

Find better ways, communicate better. You don't need to go all "Kill math" about it. But for crying out loud experiment, be creative. Or at the very least copy creative people.

There are soooo many good people out there experimenting -- just check math teachers on twitter and you'll find wondersome and wonderful experimentation - shout out to Fawn's visual patterns!

Think about responsive arrangements of your equation layout. Think about animations of equation content. Think about mixing imagery with complex text content and only a touch of equation layout. Think about scrollytelling techniques. Think about using something else instead. And fail! Fail again! Fail well! Fail at failing until you fail to fail.

The web is a wonderous medium. Its grain is fantastic to work with, read its direction, feel it out, mess with it.

story telling

Way back when I disliked Bret Victor's Kill math for actively ignoring the power of compression of equation layout.
But nowadays I don't like it for its lack of imagination regarding the presentation of equation layout. It just gives up and uses a completely different medium. Before you ask, I didn't like the scrubbing calculator that much either - even when you ignore that it's not accesible. But at least the idea Explorable Explanations inspired more people. I readily admit I might be jaded and just can't find anything good in mathematical examples while I enjoy lovely examples like why is the sky blue.

Apps like soulver showed a while ago how some kinds of complex equation layout might become unnecessary. Tandis is mesmerizing and I suspect accessibility questions around these could teach us an awful lot more than any MathSpeak/Nemeth debate.

Where are the animated equations? What would that even look like?
Where are the scrollytelling explanations of deep concepts in new ways?
Where are at least typesetting animations for equation layout?
Why capture a complex diagram statically when you can send something that dynamically or interactively draws it?

if you do it, do it accessibly and that means do accessibility yourself

If you put print equation layout on the web, be aware that you're bringing inaccessible content to the web. I'm happy to point anyone to WCAG 1.1.1 because every print equation layout tradition clearly separates its use from text. TeX literally separates "math mode" from "text mode" - you can't be much clearer. Other systems do the same thing: in MS Word and similar tools, the equation editing will be found in an entirely separate UI area from text editing. The two are disjoint. Call me non-intuitionistic but "not text" is "non-text".

Essentially, you're putting up a structured image. SVG is the best mental model here: if you generate a diagram with mermaidjs, you could use some of your input to enable accessibility. But you probably have to do a whole lot more.

You don't have to do everything yourself. Speech Rule Engine provides you the best heuristics on the market and a direct way to bring your own - whether you bring a plain label (that still offers SRE's heuristics and layout information) or you write your own heuristics, it's there to build upon.

Above all: be mindful what you want to do. If you want to provide TeX as non-visual presentation, do that; you are the author and it's your prerogative. Just tell users and be open to criticism.

If you want to provide more standardized descriptions, use Speech Rule Engine to generate them. You can provide braille-specific descriptions alongside speech-oriented ones. MathJax and SRE are the best there is for that kind of thing. If you don't use JS on the client, you can still do things server-side. If equation layout gets complex, use an ARIA tree or tree grid with a modicum of client-side JS (check this site's source to see one way of doing that); several ideas are being discussed in the standards world which might make exploration work without client-side JS in the future.

If any of that sounds interesting but you don't know where to go next, feel free to get in touch, I'd love to help.

The end

You've made it this far. Incredible! Time to move on. Here, read this transcript of a talk by the wonderful Audrey Watters about hope. I wished I could write as well as her but alas.

Granularity walkers for textual descriptions

2023-08-20T00:00:00Z

I started volunteering at an elementary school this year, helping kids learning to read. I'm not an elementary school teacher (and no amount of university teaching would save me) but luckily anyone who can read can help; it's all about patience and the simple reality of being present for a child. One of the textbooks typesets syllables in different colors to help the inexperienced readers. This simple typograhic tool reminded me of a problem I've been thinking about for a while now: enabling users on the web to adjust the granularity of textual descriptions.

Before we get started, let me give you a quick overview over what to expect in this awfully long bit of rambling on my part. First, I'll try to give you a rough outline of the thinking behind this idea, followed by a couple of code examples and the reasoning behind their evolution. Finally, there'll be a more fully fledge example, including demos on the authoring experience and the user experience (across major screenreaders).

When I have to provide a textual description, it often helps me to strike up a conversation with someone without access to the object or context (e.g., over the phone). Rubber-ducking such a conversation works, too, e.g., think about it as a form of conversational chatbot (well, if such a thing actually existed without irreperable damage to society). Reversely, you could try and coax a description out of a child. Anyways.

What happens naturally in my experience is that you'll start from a coarse description (say, "it's a photo of a cat") and then naturally progress towards finer and finer details. For example, "it's a black and white photo, a cat appears in close-up, with black fur but white paws, prowling right-to-left through a grass field; the background is out of focus but appears to show a clear sky above a winter landscape of a snowy field with snow-laden conifers far off to the right").

Now such results may not be good advice for when you're asked to create a bespoke textual description because, e.g., in the context of WCAG, textual alternatives are there to serve the same purpose as what they describe. However, often you can tweak this approach to fit the relevant context. So if it's, say, a picture shown to exemplify a particular cat breed, then you'd focus on more and more details relevant to differentiating cat breeds such as coat length and pattern.

Sometimes you are also simply in luck and you have extremely detailed descriptions for the exact purpose at hand. This happens, for example, with content that is generated from rich, heavily annotated (or annotatable) data such as archival images, data visualization, diagrammatic content, or print equation layout.

In such situations (or, you know, when the kid you're talking to is very chatty) you can easily have an overwhelming amount of information that could be useful but is probably confusing for users if you use all of it as textual description. That's why we have an abundance of techniques around long form descriptions.

While it's great to have textual descriptions available to everyone, I think there is a specific boon to annotating in situ, so to speak. That means, not just having concise descriptions at the location of non-textual content but have overly verbose ones there, too. The former enables users to efficiently access the context as whole, the latter enables them to deeply dive into the particulars of the content that is being described, immerse themselves, relish in the details.

Ideally the levels of detail are malleable so that you enable the user to decide which level of detail they need in the flow.

web accessibility for complex content isn't great

This might be a bit controversial but I think web accessibility doesn't really do much for complex content. HTML deals with "documents", whatever that means, but provides only a fairly minimal set of content structure - sectioning, headings, lists, figures, tables - before you grab a generic element. Even in contemporary HTML there isn't much for complex, non-interactive content. Sure, tables are kind of neat from an accessibility point of view - but they are for data. Figures have some nice logic with grouping and captioning at least, but are very specific in scope. When I think of contemporary HTML improvements, I think of dialog, popup, select menu. All are important, none of them are helping with complex "static" content. I always feel that SVG is overlooked in this context but it does provide a notion of graphic document and it does cover basic geometric shapes.

ARIA doesn't go much beyond HTML when it comes to content. Yes, there's the dpub-aria module which in my eyes suffers too much from its epub history to be useful to users. You might say it's simply in the name: ARIA stands for accessible rich internet applications, after all. If you've followed the recent discussion arount aria-flowto, you'll have noticed that a fair number of experts feel that ARIA provides too little for complex diagrammatic content such as flow charts. The same is true for many data visualizations or simply complex drawings, collages, annotated archival images etc. All of these are "static" (even if easily responsive these days) but their exploration is usually very complex.

Just take the classic meme with a pin board detailing a conspiracy theory, full of photos, newspaper clips, and notes, chaotically connected by strings across the board. How would you make that into an accessible piece of content on the web?

One good answer is indeed: applications. Maps are a good example for that because they've embraced the dynamic abilities of the web. Digital maps have moved so far away from their print origins - it's hard to actually get a decent print product out of them again (not the least of which are the many many options you have nowadays).

Another good answer is: use the infinite canvas of the web to simplify through deconstruction. What I mean by that is that much complex content comes from print traditions where you have to cram a lot of information into the limited dimensions of print formats. On the web, you can often untangle content into smaller pieces that are more usable (and accessible). Perhaps you do this by breaking the complexity down into several steps. For example, a school-level geometric construction in a print publication (or, sadly, on the web) is often overlaying many steps at once rather than creating many diagrams for the individual steps. Such a series of diagrams would be far less complex to convey non-visually as the individual step's description can focus on a small amount of changes, despite the overall complexity. You can even still combine these back to a single graphic if you need.

So, at the very least in theory, I believe we could often use the infinite size of a web page or even a series of web pages to achieve an accessible presentation where the print equivalent would be nigh impossible to make accessible.

And yet, when push comes to shove, it's not easy. We lack traditions. As a result, it's easier to build an accessible, vastly complex autocomplete searchbox today than to build an accessible version of Gray's Anatomy or Byrne's version of Euclid's elements.

Users before authors before implementors

Call me oldfashioned but I put my trust in authors. Maybe it's out of desperation; without authors, there's nothing for users and implementors can implement all they like, it doesn't matter without authors. [Shout out to all the web developers still building with floats and jquery today!]

So if we lack traditions, we must build them. No amount of wishing for magical solutions from the print era will help us. Yes, a braille embossable version of a complex diagram is a fantastic achievement to pull off from the same source as your print content. But does that help us move everyone forward?

For me this also means that those who build tools should build them so that they're easy to be adapted by authors. Education, it's a thing.

Obligatory Baldur Bjarnason quote:

Idle thought: most Open Source Software, certainly most OSS on npm, shouldn’t be OSS but should instead be educational resources
https://notes.baldurbjarnason.com/2021/05/03/idle-thought-most.html

trees, may be too complex

But I digress. If you have followed my previous attempts at exploring accessibility patterns for complex content, then you might ask: Peter, didn't you say the ARIA tree pattern is the great solution? (If you don't, here's a quick recap: way back when, I got David Tseng, Volker Sorge, and Davide Cervone to create a minimal version of the ChromeVox/Speech-Rule-Engine/MathJax's walker that would work on server-side rendered MathJax output. Later, Volker helped me stabilize that into a small library and I continued to "dismantle" it so that it became more of a multi-purpose tool.)

Anyway, I'm pretty sure I never said or wrote that but, clearly, I found many benefits from re-purposing this traditional ARIA pattern to complex content fragments. The tree pattern is well supported and comes with many nice conveniences for users. Hoever, I struggled for quite some time now to get where I really wanted to get with this pattern.

Firstly, mobile support has proven elusive. Right now, I'm at a point where I think I understand things well enough to say that mobile AT is simply too limited to allow for how I re-purposed the tree pattern.

Secondly, it seems it's still too complex. Even MathJax v4's alpha failed to properly adopt it (despite my help) and I've seen few people trying to build on, say, the tikz examples in the code base, let alone any interesting different kinds of content (e.g., musical staff or chord notation, physics content such as Feynman diagrams).

Thirdly, the walker's focus management is a bit heavy handed - only one tree item is ever accessible - which feels like going against the grain of tree patterns (pardon the pun). Tree patterns should expose a more complex structure (even though I'd also say it doesn't do a great job in best real life scenarios either).

So, on and off over the past year or so - or 2 years since the first draft of this line is from 2023-02-03 - I've been thinking about starting fresh. Is there perhaps a better pattern out there? Maybe not an ARIA pattern but perhaps one that fits better nevertheless.

granularity

There is a core feature of most assistive technologies that I kept returning to: changing the granularity of AT output. Any screenreader or read-aloud tool worth their salt will allow users to change the granularity at which they can explore the content. Usually, this goes from "everything" to paragraphs (or some form of chunking) and on to words and ultimately to characters.

This is obviously a good idea and a key feature to make web pages more understandable - form the big picture problem down to the smallest ambiguities such as homophones, heteronyms, or just plain spelling.

In other words, this is a good strategy for dealing with regular but complex text fragments - and if it's good enough for web pages globally, then it seems a decent assumption that a similar approach could help users locally, i.e., with highly structured, non-textual, content fragments.

Users need to dive into some details but not others while at the same time we want to preserve a consistent structure across the content fragment. In other words, they want to change the granularity of the description. While some content fragments might indeed match other existing patterns (such as a trees), I think this might be one of the most basic constructs:

finer and finer granularity when users want it
with minimal changes, i.e., while keeping a good integration into the overall document flow
without exploration ending up in isolation

I must admit to a feeling of retconning things when I originally added this section. There's a decent chance that I'm rationalizing post-hoc. At the same time, the affordances offered by AT to navigate plain text was very much a driving paradigm as soon as I realized the parallel.

Obligatory abstruse goose comic #230 on writing proofs (warning: no textual description there).

naive attempts

How hard can it be? Well, really not hard at all. Especially if you don't care about accessibility right away.

The requisites

a DOM fragment (e.g., an inline SVG, some HTML overlapping a graphic, or just some HTML), focusable so as to allow keyboard accessibility [I wished we had a way to create "light" interactivity without adding to the taborder but that's just not how things are right now.]
annotated with some kind of consistent "level" information, i.e., some nodes have "levels", some of their descendants might have lower "levels".
each element with a level has a good accessible name. [We'll see how this gets tricky but again that's just not how things are right now.]

I realize that "level" is a confusing word here (what with aria-level being a thing). You can call it whatever you like - granularity or chunk.

If we think about basic keyboard operations, then we could start with just up and down arrows. If our fragment is focused, hitting the down key should "unwrap" it to expose the level below it. If we hit the up arrow key, we should wrap the closest ancestor.

That's doable in a few dozen lines of code, depending on how much error handling you want to do. Here's an example - again, please be warned that this is not actually very accessible yet. To summarize, you can focus the DOM node, then use up and down arrows to change granularity - but the tab key will always move you around all chunks of the fragment (which might be at varying degress of refinement).

There are obviously more clever ways of doing this, depending on your situation. For example, you might find it neat to avoid level values, opting to cleverly filter out lower descendants instead; or you could prefer to use just 2 alternating types of levels; or or or.

For my other goal, a key part is that ~30 lines of code seem fairly manageable to any user with a bit of JS knowledge.

function move(event) {
  if (![38, 40].includes(event.keyCode)) return; // Only move on up/down arrow
  if (event.target !== document.activeElement ||!event.target.hasAttribute("data-level")) return; // Only move if the currently focused element is the target and part of the game
  event.preventDefault(); // Avoid scrolling away
  changeLevel(event.keyCode, event.target); // Do the thing
}
function changeLevel(keyCode, activeNode) {
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  if (keyCode === 40){ //down 
    const newLevel = activeNodeLevel + 1; //increase granularity / switch to descendants
    if (!activeNode.querySelector(`[data-level="${newLevel}"]`)) return;
    activeNode.setAttribute("tabindex", "-1"); // take current node out of taborder
    activeNode
      .querySelectorAll(`[data-level="${newLevel}"]`)
      .forEach((currentLevelChild) =>
        currentLevelChild.setAttribute("tabindex", "0")
      );// add next level descendants to taborder
    activeNode.querySelector(`[data-level="${newLevel}"]`).focus(); // move focus to first descendant
  } else if (keyCode === 38) {  // up 
    if (activeNodeLevel < 1) return;
    const newLevel = Math.max(activeNodeLevel - 1, 0); // decrease granularity / switch to ancestor
    const ancestor = activeNode.closest(`[data-level="${newLevel}"]`);
    ancestor.setAttribute("tabindex", "0"); // add ancestor to taborder
    ancestor
      .querySelectorAll('[tabindex="0"]')
      .forEach((descendant) => descendant.setAttribute("tabindex", "-1")); // take "siblings" out of taborder
    ancestor.focus(); // focus ancestor
  }
}

document.addEventListener("keydown", move); // add keydown listener

And of course the user experience relies on very simple primitives: up/down and tabbing.

less naive attempts

What's missing here? On the one hand, usability, on the other accessibility (which are, in the end, one and the same).

In terms of usability, adding right/left to accompany tabbing seems rather straight forward.

With a bit of a refactoring, the additional functionality moves us up to ~60 lines of code.

function move(event) {
  if (![37, 38, 39, 40].includes(event.keyCode)) return; // only move on arrow keys
  if (event.target !== document.activeElement ||!event.target.hasAttribute("[data-level]"))
    return;
  event.preventDefault();
  switch (event.keyCode) {
    case 37: //left
      moveHorizontally(event.target, true);
      break;
    case 38: //up
      moveUp(event.target);
      break;
    case 39: //right
      moveHorizontally(event.target, false);
      break;
    case 40: //down
      moveDown(event.target);
      break;
    default:
      break;
  }
}

function moveDown(activeNode) { // refactored
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  const newLevel = activeNodeLevel + 1;
  if (!activeNode.querySelector(`[data-level="${newLevel}"]`)) return;
  activeNode.setAttribute("tabindex", "-1");
  activeNode
    .querySelectorAll(`[data-level="${newLevel}"]`)
    .forEach((currentLevelChild) =>
      currentLevelChild.setAttribute("tabindex", "0")
    );
  activeNode.querySelector(`[data-level="${newLevel}"]`).focus();
}

function moveUp(activeNode) { // refactored
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  if (activeNodeLevel < 1) return;
  const newLevel = Math.max(activeNodeLevel - 1, 0);
  const ancestor = activeNode.closest(`[data-level="${newLevel}"]`);
  ancestor.setAttribute("tabindex", "0");
  ancestor
    .querySelectorAll('[tabindex="0"]')
    .forEach((descendant) => descendant.setAttribute("tabindex", "-1"));
  ancestor.focus();
}
// the new thing: moving left/right
function moveHorizontally(activeNode, isReversed) {
  const increment = isReversed ? -1 : 1; // forwards or backwards?
  const currentNodes = [
    ...activeNode.closest('[data-level="0"]').querySelectorAll('[tabindex="0"]')
  ];// grab all active siblings
  currentNodes[currentNodes.indexOf(activeNode) + increment]?.focus();//move focus forwards/backwards
}

document.addEventListener("keydown", move); // add keydown listener

As a next step, let's go down a dead end. In some way, all of this originated with the idea to support click (or more specficially: tap) as an interaction method. Initially, I had hoped to get away with single click for down, double-click for up:

Using click here is so poor, it's hard to call it functional. You have to hit the target you want to dive in or out of and that's hard. But at least it wasn't a huge increase in code to test it out.

// the new thing: click
let isSingleClick = false; // quick&dirty global flag to identify single clicks
function handleClick(event) {
  const realTarget = event.target.closest("[data-level]"); // Spoiler: what might be a problem with this?
  if (!realTarget) return;
  const activeAncestor = realTarget.closest('[tabindex="0"]');
  console.log(isSingleClick);
  if (event.detail == 1) {
    // it's a singleclick
    isSingleClick = true;
    setTimeout(() => {
      // wait for potential double click
      if (isSingleClick) {
        //if the flag is still set after the delay
        moveDown(activeAncestor);
      }
    }, 200); // common delay advice from the intertubes
  } else if (event.detail == 2) {
    isSingleClick = false;
    moveUp2(activeAncestor);
  }
}

function move(event) {
  // snip! same as before
}

function moveDown(activeNode) {
  // snip! same as before
}

function moveUp(activeNode) {
  // snip! same as before
}

function moveHorizontally(activeNode, isReversed) {
  // snip! same as before
}

document.addEventListener("click", handleClick);// add click listener
document.addEventListener("keydown", move); // add keydown listener

My driving motivation here was actually mobile AT users. The assumption being that they're focused on an element and triggering a synthetic click to explore.

But as far as I know, I can't really access more than single-tap on mobile AT (in a browser). Sure, talkback can do a bit more if you expect two-finger input (so you could theoretically pick up on gestures). And sure, VoiceOver has its weird double-tap-and-hold mechanism for direct input (if you're able to pull that off as a user).

I really wished we could have the equivalent of "focus mode" on mobile, too.

well actually, accessibility

Anyway, let's leave click input for now and turn to non-visual accessiblity. As it turns out, there are a couple of problems we have to deal with. Namely, if you, say, use a screenreader in browse mode you might be annoyed that every single label is actually presented to you. Oops.

A particular goal for me here is to create a good user experience no matter the state of the fragment. The idea behind this is that users will know best what state is good for them. And whichever state of granularity they prefer should integrate smoothly into the surrounding content.

Another goal stems form one of the reasons I went with an ARIA tree pattern before: the AT experience of easy access to focus-mode while transitioning through text in browse mode is really good, even for inline content (some weird Firefox bugs notwithstanding). Users can just hit enter while the content label is voiced or braille-displayed to jump to focus mode and then they can hit ESC whenever to get back into browse mode.

Now with trees, there's a certain level of focus management expected of developers, whether roving tabindex or active descendant management, and since you control the position so tightly, you might not have to worry much about the rest of the tree items. In the case of the tree walker, I opted keep users restricted to their current branch so all I had to do was reset the tree to its root when the user leaves the fragment. That way all content is present in the accessibility tree while also avoiding having a mess of tree items around for users in browse mode.

In this approach I'd like to keep the state when leaving, which is slightly more complex. Because we leave multiple chunks in the accessibility tree, we need to clean up better.

But the first step is similar to the tree walker: we have to hide and reveal our content on the accessibility tree. This is pretty straight forward: ahead of time, we add aria-hidden to everything we want to keep hidden (i.e. the lower levels); then we un-hide what we want when we want it to be accessible:

The modifications are adding a handful of lines:

// we'll fix you later
function handleClick(event) {
// snip
}

function move(event) {
  // snip! same as before
}

function moveDown(activeNode) {
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  const newLevel = activeNodeLevel + 1;
  if (!activeNode.querySelector(`[data-level="${newLevel}"]`)) return;
  activeNode.removeAttribute("tabindex"); // NEW: a bit stronger than before
  activeNode
    .querySelectorAll(`[data-level="${newLevel}"]`)
    .forEach((currentLevelChild) => {
      currentLevelChild.setAttribute("tabindex", "0");
      currentLevelChild.removeAttribute("aria-hidden"); // NEW: un-hide descendants
    });
  activeNode.querySelector(`[data-level="${newLevel}"]`).focus();
}
function moveUp(activeNode) {
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  if (activeNodeLevel < 1) return;
  const newLevel = Math.max(activeNodeLevel - 1, 0);
  const ancestor = activeNode.closest(`[data-level="${newLevel}"]`);
  ancestor.setAttribute("tabindex", "0");
  ancestor
    .querySelectorAll('[tabindex="0"]')
    .forEach((descendant) => {
        descendant.removeAttribute('tabindex'); // NEW: a bit stronger than before
        descendant.setAttribute("aria-hidden", "true"); // NEW: hide descendants again
    });
  ancestor.focus();
}

function moveHorizontally(activeNode, isReversed) {
  // snip! same as before
}

document.addEventListener("click", handleClick);// add click listener
document.addEventListener("keydown", move); // add keydown listener

However, we now have a different problem: the upper layers are still around. Depending on the roles you use - and you can check the raw experiments to see which ones I'd tried - you can get anything from losing content to triggering noisy heuristics in AT tryng to fix what they (not incorrectly) perceive as broken markup.

So we have to be more precise and hide the "upper" layers more actively. One way to approach this is to stash labels (and possibly roles) of "upper" layers. For this, we start without labels (and roles), i.e., move them into a neutral position.

The modifications are yet another handful of lines (for now, we don't change the roles):

// we'll fix you later
function handleClick(event) {
// snip
}

function move(event) {
  // snip! same as before
}

function moveDown(activeNode) {
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  const newLevel = activeNodeLevel + 1;
  if (!activeNode.querySelector(`[data-level="${newLevel}"]`)) return;
  activeNode.removeAttribute("tabindex");
  activeNode.removeAttribute("aria-label"); // NEW: remove label
  activeNode
    .querySelectorAll(`[data-level="${newLevel}"]`)
    .forEach((currentLevelChild) => {
      currentLevelChild.setAttribute("tabindex", "0");
      currentLevelChild.removeAttribute("aria-hidden");
      currentLevelChild.setAttribute("aria-label", currentLevelChild.getAttribute("data-label")); // NEW: add label
    });
  activeNode.querySelector(`[data-level="${newLevel}"]`).focus();
}
function moveUp(activeNode) { // refactored
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  if (activeNodeLevel < 1) return;
  const newLevel = Math.max(activeNodeLevel - 1, 0);
  const ancestor = activeNode.closest(`[data-level="${newLevel}"]`);
  ancestor.setAttribute("tabindex", "0");
  ancestor.setAttribute("aria-label", ancestor.getAttribute("data-label")); // NEW: add label
  ancestor
    .querySelectorAll('[tabindex="0"]')
    .forEach((descendant) => {
        descendant.removeAttribute('tabindex');
        descendant.setAttribute("aria-hidden", "true");
        descendant.removeAttribute("aria-label"); // NEW: remove label
    });
  ancestor.focus();
}
function moveHorizontally(activeNode, isReversed) {
  // snip! same as before
}

document.addEventListener("click", handleClick);// add click listener
document.addEventListener("keydown", move); // add keydown listener

With this we've almost reached 100 lines of code; then again, we've come a long way.

We no longer have duplicate content being announced but instead the level of granularity decided by the user is honored in browse mode. Relatively speaking it's significant work but in the end we still have a fairly simple piece of code.

pick a role, why don't you?!

As I mentioned, I tried a few different roles in my experiments. When coming up with a pattern like this, I inevitably face the trouble that there's no obvious ARIA role for me to use.

Ideally, a role would fit in both the semantics as well as AT behavior.

With tree patterns, you have a good balance, I think. In the end, any DOM is a tree (even if tree patterns are derived from file trees), and the UX (as mentioned earlier) is quite nice. Most importantly, it tested well with users, even when it's used for content like this.

But I wanted to test out some alternatives here. On the one hand, I'm not sure how disorienting the unwrapping would be in the context of tree patterns. On the other hand, it's good to try new things.

The most straight forward role might be image. Especially when dealing with actual images, it's obviously perfect semantically. The UX is ok, too: once you explore, you get a bunch of images. You'd have to modify the code to switch the role on and off because role=image sets children to be presentational, i.e., ignored. Something like

    ...
    activeNode.setAttribute("role", "none");
    ...
    currentLevelChild.setAttribute("aria-label", currentLevelChild.getAttribute("data-label"));
    currentLevelChild.setAttribute("role", "img");
    // and later...
    ancestor.setAttribute("role", "img"); 
    ...
    descendant.removeAttribute("aria-label"); 
    descendant.setAttribute("role", "none");

The only trouble is that images are historically static so the discoverability is problematic - why would you try to interact with an image? I guess role descriptions can help but it's always best to avoid them.

Thinking about such content fragments more broadly, I once again tested with using the group role - and of course the abstract SVG is just full of groups. Again, semantically this seems very apt. It offers decent UX and is highly discoverable as AT users expect to explore groups. I really like that many ATs announce entering and leaving groups but unfortunately this gets very noisy when you're deep in the content where AT announces every single higher-up group that you're leaving or entering.

Even a mix of the two - an outermost group, say, with a role-description, and then just image roles inside - comes out too noisily for my taste when going through a granular state in browse mode.

And of course the tree role remains a good option. In the end, user testing should decide where to go.

so about that click (and other nasty details)

As I mentioned, click-based exploration in experiment #5 was not good. In particular, mobile ATs actually don't provide double clicking in browsers (naturally).

Somebody kindly pointed out on mastodon, a virtual dpad is one possible way to solve this. That's a super robust solution and I was grateful for the discussion. But I wanted to try something minimalistic.

The best idea I've come up with so far is stolen from good old physical device UX. If you have only a single button to set some kind of level in discrete steps (e.g., volume, speed, temperature), then one common UX pattern is as follows: pressing the button will start at one end of the scale, step through the levels and upon reaching the other end of the scale, the button's behavior will reverse, so we can step in reverse until we're back at the starting point.

I think there's a good chance this can work for users because realistically content that's so complex that this becomes cumbersome, is likely too complex (read: flawed) to begin with. But as always, user testing will tell you if it works with your content for your users. (For example, another UX patterns from: jump immediately to the starting level instead of reversing direction.)

This change doesn't do much in terms of our code length, since we traded double click handling against the reversing logic.

...
function handleClick(event) {
  if (!event.target.closest("[data-level]")) return;
  const activeAncestor = event.target.closest('[tabindex="0"]');
  if (!activeAncestor) return;
  // If we're in reverse, move up
  if (
    activeAncestor.getAttribute("data-reverse") === "true" ||
    activeAncestor.querySelector('[data-reverse="true"]')
  ) {
    let newActiveNode = moveUp(activeAncestor);
    if (!newActiveNode)
      activeAncestor
        .querySelectorAll('[data-reverse="true"]')
        .forEach((node) => node.removeAttribute("data-reverse"));
    return;
  }
  // Otherwise, try moving down, if it fails, set to reverse
  let newActiveNode = moveDown(activeAncestor);
  if (!newActiveNode) activeAncestor.setAttribute("data-reverse", "true");
}

function move(event) {
  // snip! same as before
}

function moveDown(activeNode) {
  // snip!
  return activeNode; //NEW: helps with click handling
}
function moveUp(activeNode) { // refactored
  // snip! same as before
}
function moveHorizontally(activeNode, isReversed) {
  // snip! same as before
}

document.addEventListener("click", handleClick);// add click listener
document.addEventListener("keydown", move); // add keydown listener

While the UX for AT users seems reasonable, it comes at the price of rather opaque behavior for non-AT users. Clicking around the SVG does not seem very intuitive to me. There's also the question on making it a bit more clever, e.g., you might want to drop out of reverse mode if the user moves away horizontally as they might not remember the state if they come back. Alternatively you might want to provide a cue which mode the user is in (e.g., an earcon, pre/suffix).

To some degree this relates to a nastly little snag I eventually hit on (mobile) AT: the synthetic click event that's triggered by AT when double tapping can miss the target (or more specifically: appear to miss the target).

Namely, if you have an SVG, then the synthetic click seems to land on the first pixel of the element's bounding box. If the node you're on does not have any paint at that position, this won't register with the child but with whatever ancestor is considered to be painting that pixel. Thus, when an AT user triggers a click on the focused element, that click might register for simply the SVG root - and our users will never be able to explore.

Of course you get the same problem with a mouse click - you'll have to make sure you hit a path to get the expected result. But I guess visual users are more likely to automatically for a painted path.

This is of course not a new problem - clicking SVGs being iffy is just the way it is. One workaround is to add a transparent rectangle that covers the relevant nodes (and remove them again when levels change). As annoying as that is, it's not a huge lot of code to maintain - and comes with the benefit of much simpler highlighting.

This adds a bit of complexity, of course:

// adding and removing rectangles
function addRect(node) {
  if (node.tagName !== "g") return;
  const bbox = node.getBBox();
  node.insertAdjacentHTML(
    "afterbegin",
    `<rect x="${bbox.x}" y="${bbox.y}" width="${bbox.width}" height="${bbox.height}" data-rect="true" fill="transparent"/>`
  );
}
function clearRect(node) {
  if (node.tagName !== "g") return;
  node.querySelectorAll("rect[data-rect]").forEach((node) => node.remove());
}
function move(event) {
//snip
}
function moveDown(activeNode) {
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  const newLevel = activeNodeLevel + 1;
  if (!activeNode.querySelector(`[data-level="${newLevel}"]`)) return;
  activeNode.removeAttribute("tabindex");
  activeNode.removeAttribute("aria-label");
  activeNode.setAttribute("role", "none");
  clearRect(activeNode);
  activeNode
    .querySelectorAll(`[data-level="${newLevel}"]`)
    .forEach((currentLevelChild) => {
      currentLevelChild.setAttribute("tabindex", "0");
      currentLevelChild.removeAttribute("aria-hidden");
      currentLevelChild.setAttribute("aria-label", currentLevelChild.getAttribute("data-label"));
      currentLevelChild.setAttribute("role", "img");
      addRect(currentLevelChild);
    });
  activeNode.querySelector(`[data-level="${newLevel}"]`).focus();
  return activeNode; //NEW: help with click
}
function moveUp(activeNode) {
  const activeNodeLevel = parseInt(activeNode.getAttribute("data-level"));
  if (activeNodeLevel < 1) return;
  const newLevel = Math.max(activeNodeLevel - 1, 0);
  const ancestor = activeNode.closest(`[data-level="${newLevel}"]`);
  ancestor.setAttribute("tabindex", "0");
  ancestor.setAttribute("aria-label", ancestor.getAttribute("data-label"));
  ancestor.setAttribute("role", "img");
  addRect(ancestor);
  ancestor
    .querySelectorAll('[tabindex="0"]')
    .forEach((descendant) => {
        descendant.removeAttribute('tabindex');
        descendant.setAttribute("aria-hidden", "true");
        descendant.removeAttribute("aria-label");
        descendant.setAttribute("role", "none");
        clearRect(descendant);
    });
  ancestor.focus();
}
...

And it does not add a whole lot to our code: now at ~120 lines of code.

Of course, you might have more ideas. For example, creating a subtitle from the active node or using speech synthesis adds maybe another 10 lines each (see the aria-tree-walker demos for an example), whatever helps your users.

Where to go from here

Overall, I'm quite satisfied with these experiments. Despite the growth in complexity, the approach still feels appealingly simple at heart. I can add and remove complexity as use cases require/permit. I actually see a production use case for something like it - which might pave the way to more (user) testing.

I leave you with a real world example where I tested out the authoring experience form yself. This works with a diagram from Wikipedia's article on the lever principle. Attribution for the referenced raster is: Jjw, CC BY-SA 3.0, via Wikimedia Commons.

To start out, I include a screencapture below. It was captured while I took that raster graphic into SVGEdit, then quickly drew some rectangles on top of of the various parts, followed by manipulating the SVG source in a text editor to add groups and the necessary attributes for our purposes. Apologies for the lack of audio or subtitles - I added this more as a decorative item so I hope this description suffices.

While it wasn't my first try working with the graphic, I think 10-15mins is a reasonable expectation for this kind of work, especially if you're the author of the graphic.

Here's the result integrated in to our final code.

Of course, "really" you want to automate this kind of thing. That might work if you're using a tool to generate such a diagram.

In case you can't use SVG but HTML with CSS: when I tried it the first time around, I started with divs in a CSS grid - here's a slightly incomplete codepen. This works the same from an accessibility perspective so you always have that option, too.

test test test

To wrap things up, here are some rough screenreader demos of a simplified interaction.

First, the standard DPAD/arrow navigation on the desktops. I've tried to be consistent in the recordings but these are in no way polished. Ideally, you'll notice the following:

starting at the paragraph just above the lever diagram
use browse mode to read past it
go back and explore a bit in focus mode
back to browse mode, move back to the paragraph before the diagram and read the explored version

Again, apologies for lack of proper video subtitles. The active speech text display of each screenreader is hopefully sufficient to get a first impression. (I should really have found the time to add some subtitling to the code...).

First off, NVDA with Chrome which has some verbosity due to the role description being repeated alongside the proper role and some virtual buffer repetition when re-entering browse mode; as an aside, Firefox is similar but doesn't suffer from repeating role and outer role description:

Next is JAWS with Chrome which has less verbosity but role description is voiced; again, Firefox is basically the same.

Now ORCA on Chrome does not add extra verbosity but triggers a click / exploration when exiting focus mode (which looks like a bug).

Finally, VoiceOver on Mac with Safari. I hope the German Macbook VO reading English with a funky German accent doesn't throw you off too much. Functionally, the result on VO is much the same as the others. Notably, VO doesn't voice role descriptions (which is IIRC generally the case) so depending on your needs, you could try to replace role description with aria-description to see if that yields better results for your users.

That wraps up desktop screenreaders. On mobile I've also tried to be consistent in what I'm doing in the demos. Here's hopefully what you'll notice:

starting at the paragraph just above the lever diagram
flick manually to read past it
go back and explore a bit using double-tap (i.e., synthetic single click)
move back to the paragraph before the diagram and let AT read continuously
then go back to diagram and collapse it again, i.e., navigate down and up again

First off, VoiceOver on iOS. Again, role description won't work (and again aria-description is worth a test). Additionally, when flicking manually, VO adds its magic OCR output. To me, it adds noise but I'm guessing users either like it or disable it. On the bright side, the OCR magic doesn't show up during continuous reading.

Lastly, Talkback. It behaves basically the same as all the others. Again, my German devices lead to funny results - the role description is read in a German voice (and of course instructional voicing and menus are in German); still, I hope you get the ist. Visually, it's a bit weird how it jumps around but it could be worse.

Note that I didn't test the hardest use case: inline content. That is the trickiest part in my experience. It's often feasible to find patterns where AT can enter easily into a complex inline content fragments while in browse mode (again, trees are pretty great here as is this pattern). The tough part is exiting and continuing inline; AT tends to not do great here, i.e., loses its position, at minimum switching back to the starting of the browse mode buffer, but often losing its position quite badly.

Again, I hope this pattern might make that easier because at least users can control what state the fragment is in when they next come across it, ideally enabling them to make it more usable. Do I think that sounds great? Not at all. But better is sometimes all I can aim for.

Finally, I cleaned up the code a bit and left a copy on GitHub. This brings it back down to ~100 lines. Hopefully I've added enough warnings to the repository to make it clear that you should only use the idea, not really the code.

If you have a chance to try out this pattern, I'd love to hear about it.

... and in the long run?

I can't shake the feeling that there's something more fundamental lurking in all of this, something that's worth considering on the platform layer. More experiments might help surface the relevant parts.

For example fairly late in writing this, I looked at the graphics-aria specification again. I admit I'd somewhat forgotten about it - it doesn't see a lot of activity. Coming from this exploration, the examples in the spec seemed pretty wild to me. Besides things like overly clever use of aria-labelledBy/describedBy it seems very odd to me how AT is working with these. I didn't have time to dig into what is really supposed to happen or where perhaps there are bugs or incomplete implementations stirring trouble. Perhaps this will be for another post. I'll just have to see.

Web components for (mathematical) writing, a first problem

2023-11-14T00:00:00Z

So a while back - checks notes - ahem, 6 years ago Sam gifted me with some of his precious time to do a fun little experiment around writing mathematical long form using the web. With "using the web", I don't mean writing markdown or LaTeX (or whatever) and converting that to HTML. Instead, I had this weird idea that we could actually use the web platform - HTML, CSS, JS - to write content. Sounds crazy, right?

Well, it kind of was. We gave up after a few months (due to life in general but also after getting stuck on paragraphs of all things) and I have been meaning to come back to it ever since. At the end of October I was finally of a mind to do just that.

Back then I had already wanted to restart this as an experiment in web components, getting a chance to do more than write a few dummy components. I'm not very far along but there have already been a few lessons for me. Instead of spending years on a blog post, I thought I'd try to jot down a first problem I ran into. If only because then I'd document my thoughts a little.

One of the many reasons that Sam was relentelessly helpful back then is that he is a working mathematician (as opposed to myself). So when I think about a possible user experience for writing long form mathematical documents he is the kind of user I'd turn to - knowledgeable and open to exploring new ideas. Obviously, we both grew up mathematically with LaTeX so that was a base line in terms of UX. You might think LaTeX would not be a high bar but of course muscle memory for any system develops a UX of its own. I mean, you'd think just showing someone emmet and modern linting&debugging tools would be amazing for people stuck in TeX's Chomsky-Type-1-maybe-0-hell.

Perhaps needless to say but just because we started out from the point of view of a working mathematician doesn't mean that there's not something to be found that benefits a wider audience.

Anyway. Let me focus on my first steps starting fresh. So. The idea is to write some web components that magically solve particular challenges or pain points. Let's collect a few.

A central item for mathematical writing is first-class support for enunciations (as the AMS style guide lovingly calls them), that is definitions, theorems, propositions, lemmata, remarks etc. - and the ever problematic proof environment.

Another item that makes TeX great is automation around counters. A particular key to magically populating labels of various things - sectioning elements, figures, enunciations, and equations - with (sometimes too much) control for resetting and adjusting them.

Finally, a central item is magic around referencing - just specify the ID and you get a lot of magic to fill those references. Even more so if you use fancier packages like cleverref.

As a first test, those felt like a good set of items: enunciations, automatic labeling and magical cross-references.

Generating content with web components

We want our HTML simple. In emmet terms: type thm->p, then hit TAB, finally write the first paragraph.

<thm->
    <p>This is amazing</p>
</thm->

That's nice. [Fun fact: custom elements need a dash and thm- is valid - though -thm is not.]

But where's our label? Well, we can generate it automatically. Since I know I'll have plenty of variation, I start with a simple base element:

// enunciation base
class enunciation extends HTMLElement {
  // no boilerplate to keep this short
  connectedCallback(name = 'Enunciation') {
    // this.setAttribute('role', '???'); // let's get to that another time
    this.insertAdjacentHTML('afterbegin', `<enunciation-label>${name}</enunciation-label`)
  }
}
customElements.define('enunciation-', enunciation);

Long names are ok, I think, since this element is not meant to be used directly; verbosity is not the problem here but hopefully helps with understanding the output. [Or not, let's not tie ourselves down this early.]

When this element is enters the DOM (or more likely: gets upgraded), it will generate a label, containing a customizable name. (Should I use templates and slots? Should I look into mixins? Way too early to answer, I think. Let's stay naive as long as we can.)

Anyway, the key thing is that we can pass an argument to connectedCallback which we do in our actual theorem element:

// theorem
class theorem extends enunciation {
  connectedCallback() {
    super.connectedCallback('Theorem');
  }
}
customElements.define('thm-', theorem);

Fair enough, we have generated our label - and a prototype to base other enunciations on. Let's move on.

CSS counters are sooo lovely (until they aren't)

For automatic counters, CSS goes very well with our components.

Just add

thm- {
  counter-increment: thmCounter;
}
thm- > enunciation-label::after {
  content: " " counter(thmCounter) ". ";
}

Sweet. This seems super simple.

Resetting CSS counters is also trivial; typical variations in mathematical writing such as adding a title alongside the label or an option to surpress the label would be straight forward next steps.

However, let's ignore these tempting side quests and move on again so we can run into trouble.

Call me by my label

At first, the third item seem equally easy: magic cross references with web components.

Obviously, we need a point of reference and a reference to it. Let's adjust our theorem example; in emmet terms: thm#thm>p. Also, add a paragraph with a cross-reference to that ID, maybe (in emmet) ref[target=thm].

<thm- id="thm">
    <p>This is amazing</p>
</thm->

<p>Have you heard about <ref- target="thm"></ref->?</p>

Could be that target is too long but I like readable strings. In any case: what should ref- do? Well, just find the label of the theorem and use that, obvs.

class crossreference extends HTMLElement {
  connectedCallback() {
    const targetId = this.getAttribute('target');
    this.innerHTML = `<a href="#${targetId}">${document.getElementById(targetId)?.querySelector('enunciation-label')?.innerHTML || targetId}</a>`; // maybe prepend so authors can add parenthetical notes?
  }
}
customElements.define('ref-', crossreference);

Finally, I run into trouble. The resulting link just says "Theorem", not "Theorem 1". That's not super helpful (and clearly fails WCAG 2.4.4). But of course that's my mistake - the innerHTML is "Theorem", after all.

So how do you get the pseudo-element's content? As you might do, I hit the usual search engines to figure out if it can be done. In the process, I found some ancient stackoverflow posts - but it's not too helpful to know this didn't work in 2008.

I eventually remembered that getComputedStyle can look up pseudo elements. But unfortunately it gives us " " counter(thmCounter) ". "- very much correct and very much not what I want. (Reminds me of an old joke about mathematicians, fog and a hot air balloon, actually.)

So I asked my fedi bubble and kind people helped - only to confirm that it's not doable today, despite having been discussed by relevant people (approvingly) 9 years ago.

Shucks.

This is why we can't have nice things

I guess we'll have to realize the wonders of CSS counters in our custom elements instead. Hold my beer?

Ok, maybe just a counter for what we need? Either way, then we can populate the label with the actual counter directly, and references will start to work. A quick & dirty solution might be something like this

class enunciation extends HTMLElement {
  connectedCallback(name = 'Enunciation', counter) {
    this.insertAdjacentHTML('afterbegin', `<enunciation-label>${name}${(counter ? ` ${counter}.` : '')}</enunciation-label`)
  }
}

class theorem extends enunciation {
  connectedCallback() {
    this.counter = ++theorem.counter;
    super.connectedCallback('Theorem', this.counter);
  }
}
theorem.counter = 0;

Well, that feels quick and dirty. But it also raises some questions.

As I already mentioned, CSS counters offer a lovely resetting mechanism - just something like

section {
    counter-reset: thmCounter;
}

to reset the theorem counter at the start of each section. I need something for that as well.

I guess if I have a custom sectioning element, say chapter-, I could try resetting this:

class chapter extends HTMLElement {
  connectedCallback() {
    this.setAttribute('role', 'doc-chapter');
    theorem.counter = 0; // (don't do this)
  }
}

Even if I make that work, what would I do for a regular HTML sectioning element like article or section? Well, I can't. Instead, I'll have to exclusively use custom elements for sectioning (sec- maybe?), so I can do this kind of a reset. [Though maybe that's not a real problem since I probably want ref-s pointing to sections.]

It was funny how dirty this reset felt to me. I mean, it is how CSS does it, right? Then again, perhaps counter-reset is an excellent footgun on the CSS end? I admit I've never used it outside of lists.

Anyway.

To keep the counter logic in the theorem element, I need to find out how many theorems precede the current one. Maybe a quick&dirty while-loop along previousElementSibling is enough - if you're a super clean author and theorems with related counter values are surely siblings, good for you, I'm probably not. I could also go full on DOM treewalker to walk up the DOM to find the previous theorem, read its counter, increment it (with some break condition to match the counter reset desires).

As with everything I've written here, I have no answer, but at least a few ideas. For now I'm taking what feels like the middle ground - find a closest ancestor of my (reset-)liking, look at the reset's descendants, use the current element's index therein.

const siblings = [...this.closest('section').querySelectorAll(this.tagName)];
super.connectedCallback('Theorem', siblings.indexOf(this)+1);

The end result seems good enough for now; easy to muck about, learn, try something.

See the Pen Theorems with counters, part 1 by Peter Krautzberger (@pkra) on CodePen.

The code is so trivial that I can throw it away once I've learned more (e.g., how about "Theorem 2.1"? should that label be inside the first paragraph? What role to use?).

As an aside, the links earlier point to target-counter from the CSS Generated Content Level 3 spec where you'll also find target-text. Besides not having gained any traction I admit I find those three oddly unappealing. It seems more natural to me to have "getComputedText" to access the actual pseudo-element value, e.g., when attr() is involved. But perhaps I'm too JS-oriented here; it just seems to me like people would need this anyway to cover more ground. And maybe it's lower hanging fruit.

In any case, surely there's more trouble ahead. On the bright side, I know this will continue to work for a very (very) long time thanks to the backward compatibility of the web platform. (Definitely longer than the LaTeX stylesheets from my PhD thesis.) And once counters catch up, I can simplify.

In the end, this first set of experiments has worked out as I'd hoped. It got me thinking (and learning) about web components again, about limitations of the web platform, and about writing itself. More than I expected to get to in a few evenings.

In the mean time, I've continued this journey a bit further. But that's another story for another time perhaps.

A first look at MathJax v4

2023-12-31T00:00:00Z

So here on the blog (and more relevantly at work) I've been rolling out MathJax v4 after it hit its third beta in July. MathJax v4 is chock full of fixes, features and even a tiny bit of finally-getting-rid-of-stuff. To close off the year, I thought I'd sum up my personal highlights.

Here's my (obviously totally personal and biased) short list.

STIX Two (and fonts in general)

I've been using the experimental STIX2 branch in production for a few years but now that Davide had time to completely rewrite the fonts support, the generalization leads to a lot of small (and large) improvements in STIX Two again. It's lovely to see the best font keeps getting better. (And yes, a better default CM-derived font is cool too, as are the other options!)

Working together on a small project to build a custom font earlier this year also gave me a first look into the fantastic capabilities (and slightly scary machinery) behind the new font tooling. I can't wait to see what people might do when it's opened up to the public.

line breaking

While I don't care much about line breaking inside display elements these days, inline reflow has been on my wish list for a long time. But what I hadn't expected at all was Davide adding support for MathJax's SVG output.

That's a bit unbelievable, really, but MathJax v4 will split up the SVG output into reflowable chunks which is just lovely. I wished the webfont hack could be avoided but I'm looking forward to hacking a decent compromise there so that this becomes usable in the horrors of epub, too (where I need it the most).

ESM

You might say "it's about damn time!" but it's super sweet to see this move, even if it broke everything during the beta for me. I don't think I'll ever like MathJax's custom build system for its "components" but with Davide's patient help I could move my third party extensions over to the new system.

TeX input

Soo many improvements! A lot of subtle changes matching LaTeX behavior more accurately, a few more extensions.

If I had to pick just one thing, HTML-in-TeX support would be it. The potential seems boundless as MathJax is not limited by obnoxious theoretical-purity nonsense but instead embraces HTML. Which is as it should be.

non-visual rendering

SRE also saw a massive set of enhancements in its recent releases. Better performance, more languages, 2D nemeth, plenty of bug fixes. Amazing work by Volker - nothing gets even close to SRE's range of non-visual capabilities.

Even more importantly, SRE will (finally!) be active by default in v4 - the most accessible output as the default is good in my book.

coda

There'd be much more to write but the year is over. For balance, any complaints? I think the only one I found would be testing which was painful at times - but that's what you get for adopting a beta release. Still, I don't know how I'd feel if I didn't know how good Davide and Volker are (and if I didn't know how much testing happens at the AMS).

Of course I still wished that MathJax was not so focused on being client-side first. It is without competition when it comes to server-side-rendering, too - the most robust and most accessible option available. More people need to hear that! But of course I remember why that is: MathJax users love the ease of client-side rendering, they have never cared about the downsides (e.g., page load, performance). Still, I'll keep spreading the word.

As it stands MathJax v4 is a massive step forward, proving again that MathJax is the only reliable project to make progress in this space.

Web components for (mathematical) writing, continued

2024-01-27T00:00:00Z

After sharing the first interesting problem I had run into, Sam and I spent a few more days working on this experiment. I had hoped to finish this write up before attending JMM 2024 but life was more important. In any case, time for an update.

If you want to skip to the end, here's a link to the (now public) repository. You can just browse through the git history if you prefer - I stopped at this commit when finishing up.

unexpected pains

An overly large chunk of work went into fixing up the example document. That was probably inevitable. I wanted to start with a pre-existing source to avoid the trap of writing content to fit the tools. As a consequence, I had to hunt down markup inconsistencies from 6 years ago including times when I had to just shake my head at my former self (role="bla blub" - I don't even).

Along similar lines: Sam at some point decided to rename all custom element tags. That was an easy challenge and a good reminder of real world editing with co-authors.

that counter problem

As I wrote last time, counters turned out to be a slightly curious problem so I adjusted the code with the knowledge I had gained around CSS counter issues. Another "fun" fact I learned is that display: contents breaks counter increments - which is one of those "makes sense and yet I'm annoyed" moments. Since I was using contents to fake sub-grids, that particular case would resolve itself easily in 2024.

Anyway, for theorems, I ended up going with the simplest approach of using this.closest(bla).querySelectorAll(this.tagName) to find the index and magically populated cross references with that. Performance wasn't an issue even on this reasonably large (~80pp) document - and even less of an issue for reasons given below.

wasting time

I also wasted a modicum of time customizing the enunciation element to allow proofs to re-use it, only to realize that I don't like this at all. A proof and its label are different to me (in more than one way) so now they have their own, simpler custom element inserting <proof-label>Proof.</proof-label> if no such element exists.

Of course there were plenty of bugs to fix in the new code. You might say it's because I was just fooling around instead of doing it "properly", say via test driven development (which you should totally go an learn from Baldur). But that was kind of the point of this. I wanted many "oops, this was stupid"-moments. In the end, a lot of that work came from re-using our source of our 6-year-old experiment instead of starting with a clean slate. However, I would like to think that it gave some grounding - and serendipity.

a little bit design

After I had done the first bit of work on elements, it was time to think a little bit about design. Just something small, to get things going. And somehow I started to think about print (of all things) because I was doing something similar at work. So I dragged out pagedjs and started tweaking print styles. While it started out fun, I fairly quickly hit a mental wall. What was I doing looking up LaTeX class files (of all things)? The idea was to do something for&on the web!

So I stepped away for a while. I dreamed up a few cool things (sequential diagrams! scroll linked animations! interactive highlighting!) and then faced the reality of the effort that would take, especially during a normal messy daycare/school fall (one germ hitting the family after the other).

I finally settled on something simple - a few flashy colors, a lot of grid, vertical writing, sticky positioning, silly borders for various enunciations. And yes, some traditional print styles are in there. All this should eventually hint at how different the presentation can be.

At some point pagedjs starting crashing on me so there currently isn't a flashy pagedjs print demo. I think I know the problem so I hope I find time to add this back (stay tuned for another update?). What I really want is time to explore print CSS more seriously.

Is the design well thought out? Not in the slightest. Are there bugs in the design? Yeah! Plenty. But, again, still not the point right now. Polish can come later (if ever).

that sweet SSR magic

Here's the weird thing. When people think of web components, they think of something interactive. Because, after all, isn't HTML enough for "just content"? And really, should you be relying on JavaScript when you can just ship static HTML?

I agree. But the point of this experiment is not (just) the final content, it is also the creation thereof. There's plenty of writing about the perils of favoring developer experience over user experience. And using web components for writing content seems like just such a DX trap.

I hope to avoid it by thinking about server-side-rendering (SSR) from the start. In the end, a custom tag is valid whether or not it is ever registered as a custom element.

So the ideas is to use SSR to magically render components out of existence, or rather: compile them out so that the document no longer needs JavaScript just for the static content. (I definitely don't want to get rid of the custom tags.)

That's easy enough. I love linkedom for this kind of thing - it's small, it's fast, it's simple to use - and we can just dynamically import our web component code:

import { parseHTML } from 'linkedom';
import fs from 'node:fs';
const {
    document, customElements,
    HTMLElement
} = parseHTML(fs.readFileSync('./m522.html').toString());
// make some globals so customelements.js can find it
globalThis.HTMLElement = HTMLElement;
globalThis.customElements = customElements;
globalThis.document =  document;
// upgrade/render our elements
await import('./customelements.js');
// save
fs.writeFileSync('ssr.html', document.toString());

Done.

Naturally, I can't stop there. I love MathJax most when I use it in an SSR setting so I added that as well. Well actually. It turned out to be slightly harder. Because I was lazy and the way MathJax detaches nodes, my components were connecting twice, causing duplication. That took a second to work around lazily. I also found a bug which Andrea rejected a while back (I like his opinionated OSS style!) so I added a polyfill and let the MathJax folks know. Another second gone.

Next on my list was building a dedicated pagedjs output but as mentioned, pagedjs started crashing on me so I haven't returned to it. It's just as simple, obviously.

bits and pieces

While wrapping things up at the end of last year, I did a few other tweaks. I DRYed up the code to generate enunciation variations. I renamed name- elements to h- because at this point they were only serving as headings. I renamed Sam's lovely blame- tag to e-title (which probably should become enunciation-title for clarity) because I've grown accustomed to this use for the words label and title.

Finally, Sam asked for a table of contents so I add simplistic toc- element that populates itself. Speaking of conversations, Sam was surprised that I turned almost all cases of *...* markdown syntax into dfn elements. That's somewhat unusual since you'd usually choose i or em for this. However, I noticed that almost always these were used for "inline definitions", e.g., in proofs. We have a tag exactly for that; we should use it. And maybe one day Volker will do what we wanted to do years ago and teach speech-rule-engine to pick up definitions to leverage in its heuristics.

what's next

So there you have it. A first somewhat longer experiment. And always more to do.

For example, I need to rewrite the cross-references code to emit from the target element. The reason is twofold. First, the target may not yet be in the DOM when the reference connects, thus things get complicated; that's only half a reason because the same argument applies the other way around. Second, the code for cross references would otherwise become weird - so far it's only been enunciations but when sectioning, figures, equations come into play, the code would become this convoluted list of separate cases. I think I'd much rather keep the code for links with the code for targets. As a result, we won't need a custom element anymore - an a tag will do just fine.

Next there are some accessibility considerations to take care of. For example, our generic heading element should label its sectioning element. I also prefer enunciations as figures with captions; it fits well semantically and AT provides nice affordances around them. I'll probably just use ARIA directly here but maybe I'd really like to have time to see how extends holds up. Maybe more ideas will stop by. And I think I already spotted some unfortunate limitations in the current state of the AOM so that will be fun to dive into.

Then there's the bibliography. The sample lecture notes didn't have one so I disabled the old code for now. We need to try things out in a document that has them (or add some for testing).

At this point, I still feel pretty good about this. It developed the way I expected and new ideas are still becoming apparent. Complexity is still low (~100 lines of JS, ~500 lines of CSS, both pretty simple stuff). Pop over to the code or just the live demo if you like and let me know what you think.

I hope I have more time and inspiration to work on this. Why, Sam even threatened to write something with these components. Now wouldn't that be something.

On texml and texml-to-html

2025-07-26T00:00:00Z

For the past, uhm, decade? (wut?) I've been helping the AMS to build its tooling for TeX to HTML conversion. The first two parts were made open-source in 2022 (texml) and 2023 (texml-to-html). And yet, I've never written about them once. Let's change that.

There are plenty of tool for TeX-to-HTML conversion, so why another? The reason is fairly trivial: it already existed. The AMS had done some of this for a very long time (e.g. to populate abstract pages and MathSciNet reviews). Since their tooling wasn't built by some script kiddie trying to hack around TeX but by the AMS's in-house TeX expert, this turned out to be a great basis.

When I joined the AMS for a bit in 2013 (while running MathJax), David was more than ready to expand this tooling to full-text publications. I was knowledgeable enough in MathJax to help "outsourcing" math mode layout to MathJax. So we started to experiment.

David made texml a TeX-to-XML converter (with the output roughly following jats/bits). After a few experiments with with O'Reilly's htmlbook and eLife's(/substance's) Lens, I pitched the AMS that I could build the HTML side of things from scratch. All of this went well enough and we scaled steadily from 2 tiny journals and a few books to all journals and (hopefully soon) all books.

Now a curious thing happened a while back. Somebody became interested enough to try to get those two open pieces to run. This was honestly surprising given that there's virtually no documentation alongside two quick youtube videos I had recorded for Sam a while back.

In a recent conversation, the inescapable question came up: why use texml/texml-to-html rather than all the other tools out there? I generally don't like "elevator pitch" questions. But I sometimes teach using TeX-to-HTML tools (and how to get something accessible out them).

When I teach these courses, I tend to expose people to pandoc and make4ht. It's not that they're great - honestly, no tooling in this space is particularly great. However, they're easy to install and they represent the two broad tendencies in this space: staying close to TeX and reaching close to the web.

I've found pandoc to be decent when your priority is the quality of HTML. It's generally clean, well structured, malleable. But I found its consideration for TeX to be lacking when TeX is the input. Since pandoc seems uninterested in deeply re-implementing TeX's input (whoever would?), it can struggle with fairly basic TeX constructs.

This is a decent tradeoff. And given its large community, there is a great variety of additional tooling you can hook up (e.g. custom filters) to get yourself further along. Combined with a strong support for HTML, you can then get really nice side-effects (e.g., customization). It becomes a steep learning curve however.

On the other end there's make4ht (the modern tex4ht) which stays very close to TeX, leveraging dvi to get around some of the messiness of TeX as an input. The trade off here is that the resulting HTML is often capturing (and getting stuck reproducing) print layout rather than helping you build good web content (and design). Again, that's a good trade off in many situation. And if you can hack TeX, you can learn to hack a make4ht config file.

Since I cheered for it way back when, I feel like I should mention latexml. I haven't looked at it in great detail in a while but my last evaluation left me thinking that it still suffers from its historic twists. Going from building the DLMF with custom XML semantics to converting all of the arXiv to "vain" HTML leaves a mark. While latxml's approach appears to be more like pandoc's on the surface, I've always found its output to be more like make4ht - a lot of TeX-derived layout, noisy markup and some core decisions I strongly dislike.

I also want to mention to plastex. I liked it whenever I gave it a spin but I don't feel comfortable enough in python to enjoy it (or to suggest in workshops, except for people who are comfortable with python). Still, it has one huge plus in my book: Christian's Chirun builds on it.

Anyway, so what about texml and texml-to-html?

I would say the main thing about texml, texml-to-html, and the rest of the AMS toolchain is that it's opinionated: texml does not want to support everything under the sun, it supports what AMS publications need - and some things are supported but maybe in intentionally limited or unexpected ways (e.g. to improve HTML output or layout on the web).

At first sight this may seem no different than the other tools I just described the difference is control.

You can get away with this when you control the source. The AMS runs an essentially self-sufficient publishing operation. This gives them great control over the production process and helps enforce reasonable standards during production.

This is a massive shortcut for any tools having to deal with the resulting TeX sources. The amounts of maddening hacks these tools need to support is very limited. You want to hack \relax? No thank you. Your minipage-infested layout hackery? Goes out the window (or into a graphic). You want tikz to draw across your page? Nope.

The second cornerstone is that the toolchain was devised for HTML production. Because TeX continues to be in charge of producing the PDF output, you don't worry about XML-to-print workflows. HTML is where it's at. This gives you the freedom to re-interpret TeX packages in a way that makes better HTML.

The third cornerstone is: MathJax-first. Texml handles math mode in a way that optimizes the result for MathJax's TeX input. Macros get resolved far enough to make sense to MathJax (and ideally no further). IDs are generated and integrated to match text and math mode (e.g., links to tagged expressions), generated graphics (e.g. tikz) are extracted but integrated into TeX input for MathJax to re-combine. Additionally, the work has led to half a dozen MathJax extensions, some of which made their way back into MathJax proper.

I think that's the main gist. I suppose the other question is: should you use it?

I don't know. Like all tools in this space, it's easy to point out its many flaws and failings. Even more so, given how specialized it is. I also don't write much LaTeX myself these days. Nor do I have to deal with students (who are probably using LLMs to generate questionable TeX sources). And I would always tell you: to create web content, learn HTML and CSS (and maybe web components).

Still, I would say this: if you've tried "all the tools" and were never quite satisfied, give these a try. If you think "oh, this drives AMS publications, interesting!", give them a try. Here's a demo on codeberg you can poke. Let me know how it goes.

Probably don't put equation layout in links

2025-10-02T00:00:00Z

Links with complex content are a long standing accessibility challenge, essentially ever since HTML opened up the a tag to contain virtually anything else. Equation layout is not accessible and there is no single ARIA pattern that will help you make it so. So you need to adjust your approach as the situation requires - and, above all, as your users require.

One of my old sturdy helpers (to make mathjax and sre actually useful server-side) was having a bit of a problem recently. (You may think this is about links inside equation layout but it's actually about links wrapping around equation layout; it doesn't matter if the link is in or outside the equation layout wrapper.)

It took me forever to decide how I wanted to resolve this. Luckily, real world data came to the rescue: this simply doesn't seem to happen a lot in practical content. Or rather, this mostly happens when the equation layout is really just flat text (e.g., a number or letter or word). That's the trivial case - because that's not actually equation layout - and easy to resolve as it presents none of the actual complications commonly found in equation layout (i.e. highly compressed, ambiguous notation in 2-dimensional layout). However, every once in a while (in this data: 0.1% of the time) authors will put something non-trivial inside such links.

And that's a problem. Elementary accessibility advice will tell you: don't nest interactive elements. One reason why equation layout is not accessible is that it usually requires complex interactive exploration to reach a minimal level of accessibility. When that's in a link, we have a problem. You don't have to hit ACT rule 307n5z but there's generally a very good chance that the interaction models will interfere with each other, causing user confusion and developer headaches.

Now you could go the (client-side) mathjax/sre-way and just nuke the whole thing and build every interaction from scratch; who needs a platform when you have endless spans and javascript? Or you could join the fanatical purists and expect ATs to magically resolve this for you (because they are not part of the platform so you won't have to prove it actually works - win/win!).

I'm a fan of the platform. Yes, it sometimes (often?) sucks because it's still so young and limited, especially when it comes to accessibility. You may have to push it to move things forward but going against the grain is unwise. ARIA patterns in particular are fairly limited - but at least they are part of the platform. On my end, user testing has always shown a clear preference among users for standard ARIA patterns. Even if no pattern fits, giving users a pattern they know how to operate removes a significant barrier for them; that's what it's about. Users don't have a problem seeing a pattern used out of its typical context - they've probably seen so much worse - as long as it operates like that pattern should.

Even if it's not going to be a perfect or even a great fit at least you're not forcing them to first figure out your magical ideas about UI or the gazillion shortcuts you put in. Seriously, who has time for that when they just want to read something that's difficult enough to grok as it is? (And yes, of course it makes it worse that accessibility traditions for equation layout remain are still stuck in the print era.)

So what to do with my little problem? Honestly, I got lucky. The real world data I'm dealing with showed that the non-trivial equation layout inside links was still pretty simple equations, e.g. a simple fraction or a simple base with short superscript. So adapting the simplest of patterns - a flat label - was sufficient. Don't overthink it, make it usable.

Of course that leaves the reverse case - links inside equation layout. Whaddayaknow, that's a hot mess, too, for effectively the same reasons. Let's see what the real world usage tells me when I get to improving that part.

All I'm saying is, maybe don't put equation layout in links. And when you have to, keep them simple so they can be flattened. Don't mess up your readers for no good reason. Texts with equation layout are usually hard enough. Bonus points: your underlined link (because they should be) won't run into the trouble of (most likely) losing the underline under that equation layout (giving you a WCAG failure).

Being unhappy with accessible SVGs: text elements

2026-01-24T00:00:00Z

This started out last fall and is mostly a quick observation on screenreader behaviors and turned into some meandering thoughts about "linearizable" SVGs.

The other day (month/season/year), I found myself thinking about a text role in ARIA. Not exactly original but at least I was thinking about it in the context of SVG. So technically I was thinking about a role for the (woefully unmaintained) Graphics ARIA Module where I have a hunch a role for text (say, role=graphics-text) could be useful. The simplest reason is probably handwriting in its various forms, whether it's signatures, calligraphy, graffiti, or other heavily stylized writing. Generally speaking, this kind of content is not (and often cannot be) represented as SVG text elements; instead it usually appears as SVG paths, rendering a reproduction of text (handwritten or otherwise). But many other forms of stylized text tend to be realized as paths in SVG as well.

It's still text though.

It's used as text, it looks like text, its text alternative will match. So, duck test passed?

One obvious objection is: it's only text if you use the SVG text element. I think that's a weak argument. SVG's text element is so limited in terms of design that even text that could be using the text element is often turned into paths. There's a reason why every vector graphics editor has a function to replace text with paths. While the SVG spec is finally seeing some activity again, its current charter is focused on maintenance and interop, not new features, so this is not going to improve anytime soon. Besides, ARIA is about expanding developer options to allow solutions that host languages cannot otherwise provide.

A slightly stronger argument might be that text is somehow special. It provides certain fundamental affordances that nothing else can match. I don't disagree but, again, ARIA is for when the correct™ way is not available. And ARIA roles are only ever a (weak) signal, a promise from developer to user; it's the developer's responsibility to make good on that promise.

The graphics-aria module only has 3 roles to offer: graphics-document, graphics-object, and graphics-symbol. I'd say none of these fit so I find myself thinking graphics-text might be reasonable in SVG.

Testing text

Inevitably this made me wonder whether it's actually a good idea (for accessibility) to use text elements in SVGs. What actually happens when you use the text element, and what happens more when things turn a little more complex?

To start off, I took the MDN example for SVG text for a spin, testing different screenreader in different modes (with default settings). A rough list of results:

ORCA with Chrome: reads the text content in every mode, no additional cues that this is (inside) an svg. browse mode stops after each text element; no indication that it's a text element.
Narrator with Edge: reads the text content in read all, paragraph/browse, and word mode; adds "image" once upon entering (in any mode).
NVDA with Chrome/Edge: adds "image" to every text element's content in read-all and browse/paragraph mode; in word navigation adds "image" to first (or last) text element when entering svg, then "leave graphic" when leaving. With Firefox, read-all is similar but browse mode gives only a single combined reading of all text contents (no "stepping through") when visiting the image (no "image" added); same if navigating by word.
JAWS with Chrome/Edge: adds "graphic" to every text element in read-all and browse/virtualCursor. Using word navigation, there's no "graphic" at all. With Firefox, similar to NVDA.
VoiceOver Mac with Safari 26. Read-all (VO+A) reads the text content, with no additional cues that anything is (inside) an svg but secondary information about text elements (selectable text). VO+right/left arrow exploration is similar and stops at every text element, navigation by word doesn't do much since each text element is treated separately (and contains 1 word). VO with Chrome behaved the same.
VoiceOver iOS 26: read all includes the text content, no indication that there's an SVG around. Swipe left/right walks stops at each element; no indication of SVG. Reading by line or word: same result.
Talkback (Android 16). With Chrome, read all and manual exploration both read all text elements in order; no additional cues that this is (inside) an inline svg.

So despite the possibly questionable SVG-AAM mappings, things aren't terrible but also not great. Firefox sticks out negatively in that I couldn't find a way to step through the text elements individually (i.e. basic exploration).

What bugs me is that the SVG context is never surfaced. The addition of "image" and "graphic" is identical to role/element img; that seems incorrect here. Adding "image" to text element content seems plain wrong (whether once or multiple times). Treating the SVG like a group of some sort and indicating entering/leaving is relevant, but keeping its internal semantics intact is equally so.

Only VoiceOver on Mac surfaces the nature of the text elements by default. While it probably shows my limitations as a screenreader users that I'm not sure other screenreader surface this information some other way, I doubt few people will consider looking when it's announced as image.

I suppose not announcing text element roles is consistent with the view that "real" text is special (or at least it matches the SVG-AAM's mapping to paragraphs). It's the default so it needs no additional information/noise. Still, in the context of an SVG, I wonder if this assumption holds up.

VO announcing text selectability points to the core interaction model of text: you can select it (e.g. for copy&paste). If that's what text is about, then couldn't a graphics-text role be used for similar functionality (e.g. selectability of the accessible name)? Probably a dangerous idea. Move along, nothing to see.

Rambling a bit

When I attended my first W3C event (too) many moons ago, I was amazed by a demo exploring the classic Ghostscript tiger as SVG, deeply annotated with non-visual information. (I want to say it was done by Janina Sajka using Presto-era Opera but I'm not sure I would've known at the time). This stuck with me. It helped me when we incorporated ChromeVox's equation support into MathJax later on and more when Volker further expanded MathJax's non-visual rendering. You can also see that influence in my abuse of trees for content and my more recent experiment with a granularity walker.

If you stay as academic as those, you can find a lot of interesting stuff like this one from the MIT Visualization Group. But in recent years I've lost faith in this kind of thing. It's too academic to ever make a significant difference in the real world. There never seems to be enough understanding of the web's grain to help move things forward to the benefit of the wider (accessibility) community. Neither are these academic tools becoming robust solutions for data journalism (or whatever), nor are they helpful in identifying cow paths (and gaps) in the accessibility infrastructure of the web.

In the general web development world we instead get regular (yearly) posts where someone explains how title and desc elements work to set the accessible name nd description. And how to use role=img to make an SVG behave like img. That makes me a bit sad. At most, you get a post that linearizes something like a simple graph chart. Like, there's this 10 year old talk by Léonie Watson and Chaals McCathie Nevile when that was still fairly new.

That's basically still where we are - unless you build role=application-style solutions like the MIT stuff (or client-side MathJax). Take this recent article from a11y-collective last year. Don't get me wrong: it's good stuff! But just. so. basic. (Obligatory shout-out to one of my old favorites: Heather Migliorisi's 2016 post on CSS Tricks.)

I feel there's a massive divide and we're not working to bridge it. Complex data visualization is one sliver of graphical documents, no matter how important it may be. Nothing good will come from focusing on just its use cases. Take something like this odd cloth-a-person SVG on wikicommons or this quadrant chart from the mermaidjs docs - I doubt any data viz ARIA module will help those much simpler (yet not simple) use cases.

A slightly more real world example

Here's a common pattern I see at work. Many graphics in scientific writing contain diagrammatic content and that content is often annotated by text (in particular, labeled). For example, Figure 2 in this paper,

Usually, these graphics come in a form that allow for fairly simple linearization of their text alternatives. These are not flowcharts with complex chains and loops, these are not complex data visualizations.

Here's a simplified example. (The embedded raster image is by Mamoru800, CC BY-SA 4.0, Wikimedia source).

The SVG structure in this contrived example is as follows: first, an image tag linking to a raster graphic; the raster graphic shows a size comparison of giant squids and humans. Next, 3 well positioned text elements annotating 3 areas of the graphic, namely the diagrammatic depiction of a human being as well as the 2 diagrammatic depictions of giant squids (one average and one large specimen).

It's contrived insofar as text is rarely done as text elements and I've chosen an image tag to reduce complexity (and also because I have an old issue open on axe-core about a false positive for image tags with aria-label). Nevertheless this is representative of a significant chunk of content I come across daily and, biased as I am by my own experience, I feel the world would be better if this had a better solution than sticking a long-description somewhere.

Coming back to the example, here's another round of notes from testing this.

ORCA with Chrome: in read all, it reads the image's aria-label (followed by "image") and then the text content, again no cues this is an svg. In browse mode similarly, it stops after image and each text element (though sometimes it just moved across all text elements at once). By word, orca reads the entire aria-label (plus "image") but is able to step through words in the text elements.
NVDA with Chrome: read all gives the aria-label followed by the text content in DOM order; the image element's aria-label gets "graphic graphic" added (while text elements still get one "graphic"); browse mode stops at the image element and each text element. Moving by word gives additionally gives cues when exiting the image element as well as exiting the SVG element ("out of graphic"). In Firefox, read all opens with "graphic", provides aria-label and text, no indication of the end of the graphic or the image and text elements. In browse mode, it reads the same but stops at the usual character limit, i.e. somewhere in the aria-label and in the second text element's content.
Narrator with Edge: read-all announces "graphic" when entering the SVG and also after aria-label is announced, otherwise no indications. Browse mode stops with "graphic", then steps through image (with another "graphic" before the label), then steps through each text element as before. No announcement on exiting the SVG. When going back, visual outline is thrown off a bit. By word, announces graphic when entering from either direction, skips the image tag's aria-label, steps through the text elements properly (though the visual outline gets tripped up a lot in text elements, possibly due to overflow in my testing viewport); no exit announcement of course.
JAWS Chrome: read all reads the aria-label and all text content in the DOM order, adding "graphic" after each element's name. Browse mode steps through each of these, adding "graphic" after aria-label / text contents. As before, no entry or exit announcement for the SVG. Moving by word steps through aria-label and text content in the right order with no additions (i.e., no "graphic" for the image element).
VoiceOver Safari 26: read all: reads image aria-label and text contents in expected/DOM order, adds "image" after aria-label of image element; no indication of the SVG, secondary information on text elements (selectable text). VO+left/right: steps through image (adding "image") and text elements, no indication of SVG, secondary information on text. By word: again steps through everything by word; no indication of SVG, secondary information on text. Same with Chrome.
VoiceOver iOS 26: read all: reads image aria-label and text contents in DOM order, adding "image" after aria-label; no indication of the SVG. swipe left/right: steps through image (adding "image") and text elements, no indicatio of SVG. By word: again steps through everything by word (funky highlighting of the text elements while stepping through the image elements aria-label); no indication of SVG. By line reads every text element as a whole.
Talkback (Android 16). With Chrome, read all and manual exploration both read the aria-label (with "image") and all text elements in order. I also got an announcement ("page") before the aria-label was read which may again have been due to scrolling; similarly a sound cue after the last text fragment.

Overall, these results matched my expectations after seeing the behavior for text elements.

It's not terribly bad and yet full of gaps in AT behavior, I'd say. While using text elements is rarely an option, I suspect it's not too relevant here (but I don't want to sit on this another season). The SVG-AAM mappings seem to work and it looks like the ATs are falling short here. Very likely because this pattern is not encountered much. Possibly also because graphics-aria offers too little. Chicken and egg (and resources).

Obligatory quote from Adrian

So please test on your own, using this post as a template — not the final word.

Basic

This is just one sliver of SVG content and yet improving these "linearizable" SVG structure seems like a fairly small improvement that would provide building blocks (e.g. improving inline svg announcements, text element announcements, and maybe even adding a role) that help here as well as in other scenarios. And they also don't strike me as high risk for hindering further improvements.

Can this go? On role presentation inheritance.

2026-09-16T00:00:00Z

As co-editor I often feel like the janitor of the ARIA specification. I tend to spend time on things that feel like going into the basement, full of stuff nobody remembers putting there, much of it questionable, and very little that can just be thrown out.

As small dent

A few weeks months, I stumbled upon this thread on axe's tracker where they were discussing test cases for ARIA markup in invalid HTML. I was fairly certain that no ARIA spec says much about invalid HTML markup but off I went to double check the basement.

This brought me to the spec's section discussing ARIA's concept of Presentational Role Inheritance. If you don't know this concept, here's the gist: if you have complex markup that turns out to not deserve semantics, then it can be cumbersome to remove its semantics (by adding role none). Take a big HTML table that turns out to be for layout, not data. Adding role none to not just the table element but to its caption, body, each row and each cell - that's just a drag (and error prone). So ARIA wants to make things easier for authors: slap the role on the table root and the rest is inherited.

Sounds nice - and it is: disable semantics of a complex HTML snippet with a single use of role. What's not to like?

Now what caught my eyes was the following language (in the current ED):

When an explicit or inherited role of none/presentation is applied to an element with the implicit semantic of a WAI-ARIA role that has Allowed Accessibility Child Roles, in addition to the element with the explicit role of none/presentation, the user agent MUST apply an inherited role of none to any accessibility descendants that do not have an explicit role defined.

The last bit about any accessibility descendants struck me as too broad and - as it turned out - this had only referred to owned elements until a recent search&replace to introduce this new terminology so it should probably read accessibility children.

The janitor in me was happy - just a small ~~dent~~ typo to fix. Issue filed, ARIA WG triage agreed, add it to the list.

What's behind this wall?

When I looked again a few weeks ago, however, I became confused. Here's how the spec currently continues:

Also, when an explicit or inherited role of none/presentation is applied to a host language element which has specifically allowed children as defined by the host language specification, in addition to the element with the explicit role of none/presentation, the user agent MUST apply an inherited role of none to any specifically allowed children that do not have an explicit role defined.

On the one hand, this confirms that we didn't need to talk about descendants since the concept is applied recursively to specifically allowed children.

Unfortunately, it's unclear what the phrase specifically allowed children means here (just like Allowed Accessibility Child Roles doesn't really make sense in the earlier quote).

This stems in part from another wider terminology change in ARIA 1.3 replacing "required children" with "allowed accessibility child roles".

Going back to ARIA 1.2, here's the equivalent part (within ARIA 1.2 role presentation):

Also, when an explicit or inherited role of presentation is applied to a host language element which has required children as defined by the host language specification, in addition to the element with the explicit role of presentation, the user agent MUST apply an inherited role of presentation to any required children that do not have an explicit role defined.

This seems better at first. Plain "children" make more sense in a host language context. However, the phrase "required children as described by the host language" does not.

Intuitively, you know what it means. As the spec puts it

Some elements are only complete when additional descendant elements are provided.

A table needs rows and cells to be complete. A list needs list items. Obvs.

Unfortunately, that's not actually true. The closest matching concept is probably the content model. But HTML does not require any children or descendants for, say, a table element. Nor for virtually any other element. Sure, it might make no sense and cause accessibility issues but it's perfectly valid. So how could we decide when something is required?

Does this old projector even work?

One idea might be to invert this then. Some elements do require particular parents. Maybe those work? At this point I was too suspicious and a reality check seemed in order.

So I tested some simple snippets outside the given examples of table and list. For example, figcaption can only occur withing figure and summary only in details. So what happens when you add role none to figure and summary?

While HTML-ARIA doesn't allow role none, it seems browser happily make these elements presentational - but neither figcaption nor summary then inherit this role. (Thankfully neither would any other content alongside them such as an image in a figure or the contents of a details).

Perhaps a better example is hgroup. HTML-ARIA allows any role and its content model is fairly restrictive - essentially only one heading and surrounding paragraphs are allowed. Lo and behold, if you set role none to an hgroup, its children's roles remain untouched.

And that seems like a good thing! I'd hate to lose a perfectly good heading this way. More generally it makes little sense to inherit role none when the content model allows all flow or phrasing content.

Is there room in the other building?

So I went through all HTML elements to look at their content models and there really isn't much left, especially once you take into account what HTML-ARIA already prevents (by not allowing role none on some elements). Maybe canvas will need an update as part the html-in-canvas proposal. Maybe audio and video need a look because they are not fully spec'ed on the ARIA end. But beyond that?

Looking at SVG only muddies the water further. Very few SVG elements allow any roles and their content models are in most cases extremely limited (e.g. circle, ellipse, line) and otherwise extremely broad (e.g. svg, g, symbol, use). Given how SVG is in a poor state regarding ARIA, it wouldn't really mean much anyway.

Given all that, tables and lists still make sense to me. Those are very old HTML elements and we need a workaround for legacy table/list layouts. Thankfully, for these elements inheritance also seems to work. (Well, mostly anyway - Safari does not seem to inherit onto list items.)

But keeping a complicated general rule for inheritance because two elements have been used poorly in the past? Not a great idea.

Instead, this fits much better with host-language integration. HTML-ARIA and HTML-AAM already handle orphaned elements. It seems straight forward to me that we can extend this language to role none inheritance where needed (i.e. tables and lists ). Any other host language is free to do the same. If we ever come to a point where this is so abundant that a simplification is needed, then a new mechanism for this is probably the wiser choice. (I admit I lack the imagination for this.)

Trash day

So I think this part of the ARIA spec could just g?. Would be nice to clear that room in the basement. We'll have to see what the ~~teachers~~ WG will say.