Language reconsideration - Go maybe?

[Beanow]

I'm having serious reconsiderations about using C as the language here.
My original thought was, C has the most compatibility. The library should neither need nor want OOP in it's architecture, hence not C++. So, done deal! Right?

Except for a few things:

• You will force memory management onto other applications.
• It will always be used for concurrency, yet is very painful to do in C.
• It's almost impossible to avoid mutable data structures.

And because of that, I've been racking my brain over what the API should look like in C to do this properly. There just doesn't seem to be a good solution.

---

Some things I did come up with.

Protobuf for any binary format would make a swell envelope. The support is great and will let you use something other than this library by just taking the schema. Of course you will still encounter byte arrays, especially in patch data, that need interpretation. But for all the versioning, metadata and smaller structures, it's a good standard to use.

As established in #1 there are currently no strong requirements happening. With the only exception being microcontrollers as they are resource constrained. They seem incompatible with the libraries goals and shouldn't be targeted.

Regarding concurrency. I do think it will always be involved. Honestly I don't see use-cases for lib-katana that don't benefit from designs ready for, or using concurrency.

• If you send anything to the Katana over MIDI. That's I/O. Concurrency.
• If you compress or convert TSL, sysex or binary files. That's I/O. Concurrency.
• If you're using some algorithm in the library to make diffs, or sysex upload strategies.
  You will use the result for faster I/O. Concurrency.
• If you're using the library to track state locally. You will have race conditions. Concurrency.

The whole library will need to consider concurrency as one of it's primary concerns. And C absolutely sucks at this.

---

For that reason Go should be considered as an alternative.

• Concurrency is part of the language.
• It has garbage collection.
• Elegant solution to distinguish between mutable state (structs and methods) and pure functions.
• Compiles to machine code, meaning it will work with most execution environments with very few dependencies or overhead.
• Everything is statically typed, including function signatures, leading to a type safe and testable library.
• Documentation and formatting tools are built in.

[Beanow]

Digging deeper into Go, I found a few things.

Pros

• You can create shared libraries with a C-style ABI. Giving the same portability as C.
• Build and test tools are part of the standard tools and workflow. (Saved me half a day of automake hassle to port some code to Go)
• gomobile is available to create Android and iOS projects including Go code.
• Everything from the previous comment.

Cons

• Creating the C-style ABI requires writing cgo code to convert your Go API to a C API.
  gomobile can auto generate some of this using gobind, but only for Android and iOS.
• Performance and complexity suffers from these multiple conversions.
  For example python, py -> ctypes -> cgo -> go.
• You have to think about converting your native types to C types, unless your app is in Go or C.
• The Go runtime environment and standard library will be dependencies and add some MBs to disk and RAM space used.

Conclusion

I found that Go lets me be far more productive than C does. And most of the downsides are not a big deal. The performance hit of the library bindings through C is very unlikely to be a bottleneck compared to the I/O bottlenecks. Storage and RAM on a desktop/laptop, Raspberry or Android can definitely handle a few MBs of runtime and libstd for Go.

The real thing to keep in mind is the complexity on lib-katana's side to maintain the bindings and the application's side to interface with the bindings. Therefore I will distinguish between what I will expose in the C API vs the Go API.

The Go API will simply expose everything that I would expose internally as well.
The C API will be limited to higher-level functionality. Patch management, serialization, strategies, compression, etc. And will not include low-level functionality like direct calls to the checksum algorithm, 7bit sysex <-> int conversion functions, scaling math, etc. Because frankly the time you spend thinking about type conversions going through a C-style API for basic functionality like that could have been spent on taking the Go code as reference and porting it to your language. This will take about as long, be better for performance and is far less error prone.

Should you have a scenario where high throughput is required and the C bindings give you bottlenecks, but still want to use lib-katana. You should write the application in Go and will get great performance.

Not a lot of people seem to be using Go in this fashion, so it's hard to tell if this is the right approach. All the more reason to find out and build lib-katana this way.