Backend interface and text rendering

[kamrann]

What is the approach, or plan, regarding text rendering? Presumably the backend interface draw_text() function is intended to pass strings eventually.

Does the library have any existing logic for determining space requirements for titles/axis tick labels and adjusting plot/axis layouts accordingly, or is this something handled only within gnuplot itself? Realistically I suppose the text extent calculation is something that has to be done by the backend, so the interface would need a way for matplot++ to query string extents, on top of a function to actually draw it.

In the short term, I think being able to draw text, even without any layout calculations, is probably the one thing missing for additional backends to be minimally usable.

[alandefreitas]

Yes. Although including helper functions might be a good idea, text rendering is delegated to the backend in all libraries I've seen. The way you might want to treat that is too specific to the context. Many backends won't need to render their own text. For instance, an SVG backend needs to write text directly to the file. A Gnuplot backend needs to embed the text in the commands. A markdown backend probably wants to literally just put the text in there. Skia,Qt,... have their own functions for rendering text. A backend I've writing uses freetype.

There are some functions that need to and do consider text extent, but I've never seen a library that automatically reduces the font size or does something about text that goes beyond the plot. The backend interfaces usually just draw the text at a given position, but we can reconsider anything we might need to improve that.

I agree drawing text would improve that backend a lot. The best way to do that depends on the backend we want to implement.

[kamrann]

Yeah, the gnuplot backend seems so different that it barely seems worthwhile attempting to pull it into the same abstraction as the others. Though I hadn't considered the idea of a markdown backend, I guess that would also deviate from the norm.

I wrote a plugin for UE4 a long while back that had some logic for determining axis dimensions based on text extent of the tick labels being drawn, along with some automatic rotation of labels on the horizontal axes to reduce height requirements. But I agree it's not important in 99% of cases. I'll look into making a PR to update the text drawing in the backend interface to get some minimal functionality. I was thinking I could implement it in the OpenGL example too, but may pass on that for now since it looks like it would need quite some work plus additional dependencies to get text rendering in OpenGL.

I'm curious, was there an explicit decision made to use per-coordinate arrays (rather than an array of point structures) when passing lists of points?

[alandefreitas]

Yeah, the gnuplot backend seems so different that it barely seems worthwhile attempting to pull it into the same abstraction as the others.

You're correct. This project started attempting to use the same abstraction for gnuplot. The rough abstraction looked good and gnuplot seemed like the best choice for C++ at the time because it was what most people used for plots. Although gnuplot allows you to draw individual objects to the canvas, the problem was 1) it was very slow to do that through pipes, and 2) all the interactivity was gone. So we put bool consumes_gnuplot_commands() in the interface and all plot categories were eventually using that instead.

Though I hadn't considered the idea of a markdown backend, I guess that would also deviate from the norm.

There are all kinds of "weird" backends. Text-based backends are quite common. SVG might be the best example (needs to select font but not render). Latex backends are quite common too (needs to ignore the font to match the latex document). Markdown might be useful for documentation and quickly transmitting the plot in a small text message. Markdown is also useful to print the plot to the console. It might actually look quite beautiful in small messages if we allow unicode in it. In this context, the resolution is so low (usually 20x80 chars) that rendering text with some font is also a bad idea.

I'll look into making a PR to update the text drawing in the backend interface to get some minimal functionality. I was thinking I could implement it in the OpenGL example too, but may pass on that for now since it looks like it would need quite some work plus additional dependencies to get text rendering in OpenGL.

That's OK. It's probably best to have a more high-level backend before OpenGL, where things are more manual.

I'm curious, was there an explicit decision made to use per-coordinate arrays (rather than an array of point structures) when passing lists of points?

We usually use three separate arrays to represent per-coordinate arrays. When the plot is 2D, the third array is left empty. It seems like both matplotlib and matlab do that. This might look weird if you're used to graphics applications in OpenGL, but it makes a lot of sense for plots because we only support 2d and 3d, and this avoids compiling separate objects for 3D. It allows us to support "4D" colors as a separate array. But this is for the axes objects only, the backend may use whatever makes the most sense in the context.

[kamrann]

We usually use three separate arrays to represent per-coordinate arrays. When the plot is 2D, the third array is left empty. It seems like both matplotlib and matlab do that.

From what I can see, matplotlib uses a templated 'Path Iterator' abstraction. I think eventually something similar would be a good choice. Although for most use cases it's probably irrelevant, using de-interleaved coordinates could be a performance issue for realtime plotting of heavy data. Since you will nearly always operate on the data per-vertex, you really want the components of a vertex to be adjacent in memory. Interleaved also simplifies the iterating code. It's of no immediate concern of course, was just surprised since I haven't seen that done outside of audio processing, where there are many cases where you want to operate on channels independently.

That's OK. It's probably best to have a more high-level backend before OpenGL, where things are more manual.

Right that makes sense. My use case is Unreal Engine, and I got a basic backend for it up and running with the existing interface. But due to UE4's constrained build and execution environment, it's very definitely not a good candidate for a reference backend. Did you have anything in mind for a higher level backend that could be a good one to use as a reference, and to drive the development of the backend interface? Perhaps imGUI would be worth considering? I've not used it but have heard lots of good things about it.

Thanks for taking the time to respond in such depth, by the way!

[alandefreitas]

From what I can see, matplotlib uses a templated 'Path Iterator' abstraction

They use the Agg PathIterator, if I'm not wrong. This could add a lot of complexity for 3d rendering though, unless we redo everything with templates, variants or type erase some stuff. If we really go for performance, there are also "4d"/"5d"/... points as each data point usually has a color and a size, and these could also go to the shader. Another solution is to replicate the data in the canvas object, with its obvious drawbacks. But many if not most canvas objects might need to replicate this data (in another format) anyway.

Although for most use cases it's probably irrelevant, using de-interleaved coordinates could be a performance issue for realtime plotting of heavy data

That's correct. Most applications for plots don't require so much performance. Unlike other C++ applications, it's usually safer to worry about making it work first and then worry about optimizations. Premature optimization in plots seems like a very low return on investment.

Since you will nearly always operate on the data per-vertex, you really want the components of a vertex to be adjacent in memory. Interleaved also simplifies the iterating code. It's of no immediate concern of course, was just surprised since I haven't seen that done outside of audio processing, where there are many cases where you want to operate on channels independently.

You're 100% right. This is definitely not how I would structure any other graphics application by default. They are just very particular reasons we had for this library. I imagine it would be best to optimize that class by class, looking at particular demands, once we have a good idea of what the points of friction with backends will be. We can come up with some kind of view object to make the transition smoother.

Right that makes sense. My use case is Unreal Engine, and I got a basic backend for it up and running with the existing interface. But due to UE4's constrained build and execution environment, it's very definitely not a good candidate for a reference backend

Can't UE integrate with some other technologies and libraries?

Did you have anything in mind for a higher level backend that could be a good one to use as a reference, and to drive the development of the backend interface?

Some people have been suggesting Skia lately. There might be some similar options. These are 2D libraries, like Agg, but they are very productive and already take care of text rendering. The 3d transforms we need are very simple, so they could even go into the interface as a placeholder in case the user only wants to define the 2d functions. Supporting 2D-only libraries actually has the potential to simplify things a lot in regards to new backends. Maybe UE can directly integrate with one of these libraries like Skia.

Perhaps imGUI would be worth considering? I've not used it but have heard lots of good things about it.

I did consider ImGUI once. I do use imGui a lot for quick applications with GUIs, but I wouldn't consider it as a matplot++ backend. ImGui is structured in a very specific way with an object stack. It also needs to run on top of another backend, and some of the low-level complexity is also back. My impression is that people who don't use ImGUI already wouldn't like the integration at all. And people who use ImGui would hate it because the programming paradigm is completely different.

[kamrann]

Unlike other C++ applications, it's usually safer to worry about making it work first and then worry about optimizations. Premature optimization in plots seems like a very low return on investment.

Yep understood. I'm coming from a realtime use case so that shapes my take on things, but I absolutely understand it's not the main focus of this library.

Can't UE integrate with some other technologies and libraries?

Ah, there is no issue integrating third party libraries into Unreal, it's more that Unreal has its own very specialized environment and build system, so it can't be integrated into other projects - you can't, for instance, write a regular app that uses Unreal's renderer as a library. So my point was really just that it's not a practical candidate for a backend that could be integrated into the matplot++ repo and used as an example/reference.

Some people have been suggesting Skia lately. There might be some similar options. These are 2D libraries, like Agg, but they are very productive and already take care of text rendering. The 3d transforms we need are very simple, so they could even go into the interface as a placeholder in case the user only wants to define the 2d functions. Supporting 2D-only libraries actually has the potential to simplify things a lot in regards to new backends. Maybe UE can directly integrate with one of these libraries like Skia.

Honestly, I hadn't even heard of Skia before seeing that other thread, so I can't give any input there. It had occurred to me though, that there may actually be an argument for an extension interface to support 3D plotting, rather than trying to encapsulate both 2D and 3D into the same functions. In practice I can imagine it would become a great deal more than just adding a third coordinate, to properly support all the potential and nuances of 3D plotting.

Does the library have any kind of Python interface/bindings currently? I read in an old discussion someone mentioning building with Python, but I'm not clear on what they were referring to. Is there any existing Python module that builds against matplot++, on either the frontend or backend side? If not, has that been considered?

[alandefreitas]

Yep understood. I'm coming from a realtime use case so that shapes my take on things, but I absolutely understand it's not the main focus of this library.

Yes. It's also OK to optimize things when we find a bottleneck. It's not like we are making things inefficient on purpose either. :D

Ah, there is no issue integrating third party libraries into Unreal, it's more that Unreal has its own very specialized environment and build system, so it can't be integrated into other projects - you can't, for instance, write a regular app that uses Unreal's renderer as a library. So my point was really just that it's not a practical candidate for a backend that could be integrated into the matplot++ repo and used as an example/reference.

Agreed. 100%.

Honestly, I hadn't even heard of Skia before seeing that other thread, so I can't give any input there. It had occurred to me though, that there may actually be an argument for an extension interface to support 3D plotting, rather than trying to encapsulate both 2D and 3D into the same functions. In practice I can imagine it would become a great deal more than just adding a third coordinate, to properly support all the potential and nuances of 3D plotting.

If I'm not wrong, I think matplotlib just converts 2D to 3D with simple transforms to use Agg, while matlab seems to have real 3D plots. Both should work.

For performance, the plot objects in the frontend that support 2d and 3d could be split into two different objects, so that we can layout the points differently without having to move objects. The only problem I can think of is that plots might also have colors, etc, and these things would usually go in the same vertex in OpenGL, Vulkan, DirectX, etc... but we also don't want to create an exponential number of objects. I've been also avoiding too many template meta-programming tricks because it often creates an exponential number of cases to test that we will never instantiate. A third option is just replacing the three vectors with a std::variant that might hold 2d or 3d points, or any required layout for efficiency. It would actually spend less memory than the current vectors.

On the backend, I imagine the 2D functions could be pure virtual while the 3D functions can be virtual with placeholders that transform the coordinates on the CPU if no other implementation is provided. That could serve as the 3D extension interface you mentioned so that the 2D functions are mandatory and the 3D functions are optional, for performance or specific behavior for 3D.

So something like

matplotplusplus/source/matplot/backend/backend_interface.h

Line 158 in 68185b7

virtual void draw_path(const std::vector<double> &x,

could become something like

virtual void draw_path(const std::vector<2d_point> &v, const std::array<float, 4> &color) = 0;
virtual void draw_path(const std::vector<3d_point> &v, const std::array<float, 4> &color) {
    // transform with glm if not overridden and call the 2d_point
}

We might even need to aggregate the color as well. In any case, the interface that facilitates the most performant implementation might vary according to the backend. We just have to find a reasonable trade-off. A good starting point is probably structs which are easy to vectorize for GPUs, and let other backends handle other cases from these structs.

Does the library have any kind of Python interface/bindings currently? I read in an old discussion someone mentioning building with Python, but I'm not clear on what they were referring to. Is there any existing Python module that builds against matplot++, on either the frontend or backend side? If not, has that been considered?

I'm not aware of any Python interface at the moment. It shouldn't be too complicated to create one with pybind11, as we just need the figure_type for most things. It's just that matplotlib is already very convenient in Python. The only use case I can imagine for Python bindings is if we, at some point, come up with very efficient backends, so that matplot++ could be used for something more performant than matplotlib's Agg.

[kamrann]

I'd say it's probably not really worth worrying too much about exactly how to structure data in the interface, so long as it's reasonably flexible and not a total performance killer. As you said previously, it will vary a lot between backends and most will probably need to transform and copy the data received. I think a more immediate performance blocker for realtime use cases is that currently I don't see any way to update a plot with additional data points, without pushing everything again to the backend. By far the most common realtime use case will be plotting live incoming data, where small amounts of additional data need to be added at high frequency. So I think the interface would need some support for incremental updates.

[alandefreitas]

I'd say it's probably not really worth worrying too much about exactly how to structure data in the interface, so long as it's reasonably flexible and not a total performance killer

2d and 3d points in structs seem fine for that. The data points are double currently, but they should probably become float for compatibility with more backends.

As you said previously, it will vary a lot between backends and most will probably need to transform and copy the data received

Yes. It seems that the main difference is that some will transform the data with shaders, so structs with points are more likely to attend both demands because we can send these points directly to vertex shaders.

I think a more immediate performance blocker for realtime use cases is that currently I don't see any way to update a plot with additional data points, without pushing everything again to the backend.

On the frontend, you can push new points directly to a line plot, as usual.

The backend represents a canvas. Something like an "OpenGL context" for plots. So on the backend, in the general case, I'm not sure it's possible to update the data points because once the points are drawn, the backends don't even need to store these points, like in an OpenGL context or a data structure representing a raster image.

The image would only be updated if there's some user interaction (if the plot is interactive) in which case the frontend would transform the data again and it would probably need to be sent it to the backend again anyway in most cases.

Maybe some kind of extensions could provide that update functionality for backends that store pre-processed data, given some alternatives to fallback, but I'm not sure this is possible for the general case. One possibility would be to have to indicator or id to tell the backend whether this is new data or data being updated. These possibilities and the best alternative for this extension will probably only become clear once we have a simpler version of the backend. They are definitely possible though.

By far the most common realtime use case will be plotting live incoming data, where small amounts of additional data need to be added at high frequency. So I think the interface would need some support for incremental updates.

Agreed.

On the frontend, at least, we usually think of line plots or scatter plots. These are relatively easy to update incrementally. The frontend has an interface for that and the backend can implement some kind of extension for that.

Other plot types (histograms, contours, maps, ...) are not as easy to update incrementally though. We can pre-process some data but we cannot update histograms in constant time because the bins depend on the distribution of the data points, which might change with new data. Both identifying the distribution and dividing the bins take linear time in the general case. Contours and maps are even more complicated.

There are some heuristics to update histograms with an approximation in constant time. But there are lots of possible algorithms and parameters, and this approximation is not what users are expecting by default, so this would require some interface that would make it clear that this is an approximation.

But of course we can start with line/scatter plots and incrementally improve other plot types as we need. I would probably start with simpler solution and then identify the best ways to improve that.

It's just important to remember the other plot types exist, because without considering their existence we might end up trying to reimplement the frontend in the backend just to find out this doesn't generalize well in the future.

[kamrann]

It's just important to remember the other plot types exist, because without considering their existence we might end up trying to reimplement the frontend in the backend just to find out this doesn't generalize well in the future.

Yep, that makes perfect sense. And to be honest, I was forgetting - again, my thinking is shaped by my own intended use case, which is pretty much exclusively realtime line/scatter plots.
Can you point me to an example, or the relevant code, for updating such a plot incrementally in the frontend? I think I must have missed it, unless you're talking about adding additional axes objects when you have new points? I was assuming it would be possible to augment an existing one.

[alandefreitas]

Can you point me to an example, or the relevant code, for updating such a plot incrementally in the frontend?

For every plot (axes_object) you include in the axes (axes_type), you get a handle to the object. For instance,

matplotplusplus/source/matplot/core/axes_type.h

Line 647 in 68185b7

line_handle plot(const std::vector<double> &x,

You can use this handle to manipulate the storage, properties, etc, without recreating the plot. For instance, you could use

matplotplusplus/source/matplot/axes_objects/line.h

Line 58 in 68185b7

const std::vector<double> &y_data() const;

to just push new points with emplace_back and so on. Each plot handle has its own appropriate methods.

Actually this would be the wrong overload for emplacing points because this is const, but you get the point. You can use the handle to update the plot in the frontend.

[kamrann]

Actually this would be the wrong overload for emplacing points because this is const, but you get the point. You can use the handle to update the plot in the frontend.

Right, that was the issue I ran into. It's const, and the only other option is to replace the entire vector. Having said that, I guess this distinction won't really matter until there is dedicated support for handling incremental updates right through to the backend.

I guess we've gone off on a tangent from text rendering! I think I'll start a branch on a fork to prototype some things; I'll just see where it goes and how feasible it might be for me to use this library for my purposes.

[alandefreitas]

Right, that was the issue I ran into. It's const, and the only other option is to replace the entire vector.

Yes. Another overload or some extra functions should improve this. Even const_cast would work in that case (not that I'm recommending it).

Ideally, we should prevent the user from pushing new values on one coordinate without the corresponding values for the other coordinates. 2d_points structs should solve that problem.

However, in general and bugs aside, things can be updated through the handles.

I guess we've gone off on a tangent from text rendering

I just assumed this discussion was about backends in general. 😆

I think I'll start a branch on a fork to prototype some things; I'll just see where it goes and how feasible it might be for me to use this library for my purposes

Cool. 😎 Let me know if you need any help.