Lisplog

Nicolas Hafner: Harmony 2 - Confession 90

via Planet Lisp by on Sat, 05 Sep 2020 18:07:16 GMT

It turns out that sound processing is pretty difficult. I've been hacking away at an almost from-scratch rewrite of Shirakumo's sound systems for the past few months and it's caused a lot of anguish.

Now, Harmony is not a new system. It dates back quite a few years, and I had even written another article on it previously. However, this previous version had several serious flaws and problems, some of which penetrating all the way through the audio stack. With the rewrite I'm however now much more confident in the quality and usability of the system.

First though, a bit of terminology: in digital audio processing, audio is represented as a sequence of samples; regularly recorded amplitudes of the audio signal. These samples are recorded at a constant rate, the "sample rate," which often is either 44.1kHz or 48kHz. Often each sample is represented as a float going from -1 to +1, and multiple such sample sequences are combined to form the signal for as many channels as you need (stereo, surround, etc.) When processing audio data, a limited sequence of samples is kept in a buffer, which processors can then operate on.

The initial problem with the system was one of resampling: the system was written with the assumption that one could keep a constant samplerate throughout the entire audio processing pipeline. This, however, turned out to not be suitable. The issue manifested itself on Windows, where the output backend could require a different samplerate to the one the system was initially configured for. Thus, at least at the end-points, resampling would be required.

This immediately lead to another problem though: the system was also written with the assumption that every part of the pipeline could consume and produce a full audio buffer every time it was run. However, with resampling, border issues appear and it's not always possible to consume the full input buffer. This issue permeates throughout the processing pipeline, as now the final processor cannot consume all data, and so when the system is run next, the processor before the last cannot produce a full buffer as it would overwrite data.

Ultimately though, the fixed samplerate and fixed buffer size design lead to a restriction that made it impossible to represent certain effects like a speed change, which would produce and consume samples at different rates. And so, pretty much everything had to be rewritten to work with this in mind. To spare you the troublesome process of figuring out a design, let's just jump to what the system is like now:

At the most basic level resides the bip-buffer interface, which implements a lockless bipartite buffer. It's lockless so that one thread can write, and another can read from it simultaneously. It's bipartite so that the regions it hands out are always consecutive regions of memory, rather than wrapping around like in a ring buffer. This interface is implemented by buffers and packs. buffers represent internal audio samples of one channel in float format, whereas packs represent external audio samples in any format, with any number of channels.

Then there's the parts that actually perform audio processing. These are called segments, and follow a generic interface that allows them to do their work, and also allows them to be introspected. Namely they each have a number of input fields, a number of output fields, and a number of parameter fields. To the input and output fields you can attach a buffer, which will cause the segments to exchange data. Assembling a network is then just a matter of creating the segments, creating a buffer for each connection, and then setting them at the appropriate in/out fields.

At the endpoints, where you need to exchange data with other systems such as file decoders or device drivers, you'll probably want to make use of the unpacker and packer segments, which perform the necessary encoding to translate between the float buffers and the compact packs. These segments will also perform sample rate conversion as necessary.

Since we have proper bip buffers connecting everything, a segment can now consume and produce at a variable rate without needing to be aware of the rates going on in the rest of the system. The rates will automatically propagate through the system as the buffers are updated.

Now, all of this behaviour, including many practical standard segments are implemented in a C library called libmixed. Audio has some pretty severe latency restrictions, and that's why, with great pain, I decided to implement the bulk of the audio processing in C, rather than Lisp. This has cost me a lot of time, but I still think the performance gains are worth it, or I would have had to spend similar, if not more time, trying to match the performance with Lisp code. I hope that this kind of thing will no longer be necessary at some point in the future, but for now this is where we are.

Anyway, being implemented in C also means it can be useful for people outside of Lisp, and I really do hope that others will take advantage of libmixed, as I think it has a lot of useful work behind it. To my knowledge there's currently no free (as in BSD) and capable audio processing system out there. The library also offers a plugin and reflection/introspection API so that one could build a GUI that can represent segments and buffers in a very generic fashion, allowing users to easily plug together processing networks.

Now, one level above libmixed sits cl-mixed, the Lisp bindings library that takes care of the low level stuff and wraps it all in a nice Lisp interface. It also takes care of offering some support structures where needed, such as managing the input locations when dealing with variable input segments such as mixers. It also offers a ton of extension systems for interacting with various file formats and playback backends:

• ALSA Linux playback
• CoreAudio macOS playback
• FLAC FLAC file decoding
• Jack JackAudio playback
• OSS OSS playback (BSD)
• PulseAudio Linux desktop playback
• SDL2 SDL2 integration if you're already using SDL2
• WASAPI Windows Vista+ playback
• WAV WAV file decoding
• WinMM Windows 3.0+ playback
• XAudio2 Windows 8+ playback
• mpg123 MP3 decoding
• out123 Cross-platform playback (C blob)

I'd like to add more decoders, and at some point also input for the various operating system backends, but for now this is more than plenty. Some of the backends still have issues (WinMM, XAudio2, CoreAudio), which I have spent a long time trying to figure out already, so far unsuccessful. I'm not too bothered about WinMM and XAudio2, but CoreAudio definitely needs to be made to work properly soon.

The reason these backends are implemented in Lisp is so that there's no additional dependencies on shared libraries that might be versioned and interact poorly when deployed. Since the actual work performed in their respective segment amounts to requesting a buffer region and performing one call, the performance impact from it should also be entirely negligible.

cl-mixed also offers a virtual segment that allows you to implement a segment in Lisp and integrate it into a standard pipeline. This is possible thanks to the standardised architecture in libmixed, and can be very useful to experiment with effects very quickly. If I ever intend on developing a new effects segment, I'll definitely implement it in Lisp first to take advantage of rapid prototyping, before lowering it down to C if performance should become an issue.

On that note, cl-mixed actually uses static-vectors to implement the backing storage of packs and buffers, as well as all of the bip-buffer protocol. This means that you can interact with packs and buffers from Lisp as if they were normal Lisp arrays, without ever having to worry about FFI.

That said, cl-mixed will not do buffer management or resource management in general for you. You'll still have to manually create and free segments and buffers and make sure they're connected. You'll also have to run the mixing loop yourself and make sure you do that often enough to not cause stuttering.

This is where Harmony steps in. Being the high-level component, it imposes a bit of architecture on you, but in turn takes care of a lot of lower level plumbing. In effect, with Harmony you can perform playback as easily as:

(harmony:start (harmony:make-simple-server))
(harmony:play "music.mp3" :mixer :music :loop T)
(harmony:play "effect.wav" :mixer :effect :location '(10 0 0))

It'll take care of detecting the appropriate backend for your platform, setting up channel conversion and basic mixing infrastructure, allocating and re-using buffers, automatically cleaning up when a sound source ends, and performing low-latency audio processing in the background.

It can also do fun stuff like automatically creating a network to apply effects to a source.

(harmony:play "music.wav" :mixer :music :effects
  '((mixed:speed-change :speed-factor 2.0)
    (mixed:pitch :pitch 0.5)))

Which would play the music at double the speed, but with a pitch correction applied so that the notes should still be the correct frequency.

Hopefully this will make it easy enough to use for games without having to worry about all the low level detail aspects. I'm going to find out how well this all works soon, as it's now at a stable enough state that I can start working it into Kandria.

If you're interested in using these systems or contributing to them, let me know! I'd be happy to provide assistance.

If you like my work in general and want to donate, you can do that too, either on GitHub Sponsors for recurring donations, or on Ko-Fi for one-time donations.

Thanks for reading!

How to conditonally include elements in a list?

via Elm - Latest posts by @surferjeff Jeffrey Rennie on Sat, 05 Sep 2020 14:43:50 GMT

Thank you Hans-Helmut! That worked!

Extracting type metadata from Elm code?

via Elm - Latest posts by @sebbes Sébastien Besnier on Sat, 05 Sep 2020 13:58:47 GMT

Maybe https://package.elm-lang.org/packages/MartinSStewart/elm-serialize/1.2.1/ can help, but I’ve never used it. @MartinS could maybe give more details.

Elm2node: transform an elm module to a synchronous node module

via Elm - Latest posts by @sebbes Sébastien Besnier on Sat, 05 Sep 2020 13:47:16 GMT

Ok, my bad! Thank you @luke for correcting me.

I don’t know why I thought those functions was used during port process. I’ve checked what I’ve done and I don’t use these functions neither.

So my claim was wrong.

In the case we would want to use typescript to get rid of the encoding part, I wonder if we should care about “copy” the function argument in order to prevent mutation by the caller. If we do:

arg = { b: 5};
res = elmModule.f(arg);
arg.b = 4;

this should not bother f . But is it possible for arg to be modified in another place (say it is a global variable), during the execution of f?

What happened since elm-review v2?

via Elm - Latest posts by @system system on Sat, 05 Sep 2020 11:11:37 GMT

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.

How to conditonally include elements in a list?

via Elm - Latest posts by @surferjeff Jeffrey Rennie on Sat, 05 Sep 2020 04:11:29 GMT

I’m building a simple hangman game to learn elm, translating it from some old javascript code.

To draw the hangman with a canvas, I found https://package.elm-lang.org/packages/joakin/elm-canvas/latest/ which is working out pretty well.

However, I need to conditionally draw parts of the body depending on how many turns the player has left.

Here’s my old Javacript code:

        let ctx = canvas.getContext('2d');
        ctx.beginPath();
        ctx.fillStyle = "white";
        ctx.fillRect(0, 0, 300, 300);
        ctx.strokeStyle = "black";
        ctx.lineWidth = 5;
        ctx.lineCap = "round";

        // Draw the stand.
        ctx.moveTo(3, 297);
        ctx.lineTo(297, 297);
        ctx.moveTo(13, 297);
        ctx.lineTo(13, 3);
        ctx.lineTo(150, 3);
        ctx.lineTo(150, 25);

        let turns:number = game.getTurnsLeft();

        if (turns < 8) {
            // Draw the head.
            ctx.moveTo(150, 25);
            ctx.arc(150, 50, 25, -Math.PI / 2, 2 * Math.PI);
        }
        if (turns < 7) {
            // Draw the body.
            ctx.moveTo(150, 75);
            ctx.lineTo(150, 180);
        }
        if (turns < 6) {
            // Draw left arm.
            ctx.moveTo(150, 90);
            ctx.lineTo(90, 150);
        }
        if (turns < 5) {
            // Draw right arm.
            ctx.moveTo(150, 90);
            ctx.lineTo(210, 150);
        }
        if (turns < 4) {
            // Draw left leg.
            ctx.moveTo(150, 180);
            ctx.lineTo(90, 270);
        }
        if (turns < 3) {
            // Draw right leg.
            ctx.moveTo(150, 180);
            ctx.lineTo(210, 270);
        }
        if (turns < 2) {
            // Draw left foot.
            ctx.moveTo(90, 270);
            ctx.lineTo(70, 260);
        }
        if (turns < 1) {
            // Draw right foot.
            ctx.moveTo(210, 270);
            ctx.lineTo(230, 260);
        }

        ctx.stroke();

And here’s my elm code. How do I add something like an if {} statement to only draw parts of the man depending on how many turns the player has left?

    Canvas.shapes
        [ -- Settings
          Canvas.Settings.Line.lineWidth 5
        , Canvas.Settings.Line.lineCap Canvas.Settings.Line.RoundCap
        , Canvas.Settings.stroke Color.black
        ]
        [ -- Shapes
          path ( 2, 297 )
            -- Draw the stand.
            [ lineTo ( 297, 297 )
            , moveTo ( 13, 297 )
            , lineTo ( 13, 3 )
            , lineTo ( 150, 3 )
            , lineTo ( 150, 25 )
            ]

        -- Draw the head
        , circle ( 150, 50 ) 25
        , path ( 150, 75 )
            -- Draw the body
            [ lineTo ( 150, 180 )

            -- Draw left arm
            , moveTo ( 150, 90 )
            , lineTo ( 90, 150 )

            -- Draw right arm
            , moveTo ( 150, 90 )
            , lineTo ( 210, 150 )

            -- Draw left leg.
            , moveTo ( 150, 180 )
            , lineTo ( 90, 270 )

            -- Draw right leg.
            , moveTo ( 150, 180 )
            , lineTo ( 210, 270 )

            -- Draw left foot.
            , moveTo ( 90, 270 )
            , lineTo ( 70, 260 )

            -- Draw right foot.
            , moveTo ( 210, 270 )
            , lineTo ( 230, 260 )
            ]
        ]

Alexander Artemenko: conduit-packages

via Planet Lisp by on Fri, 04 Sep 2020 17:52:13 GMT

This system is pretty old. It does not have its own revision control and is hosted here. It provides the replacement for the defpackage macro. This replacement makes easy to "inherit" your package from another and to replace some symbols with your own.

For example, @stylewarning's cl-generic-arithmetic uses it to redefine some functions from cl package. It defines a new package which uses cl and reexports all symbols except some which are defined in by the cl-generic-arithmetic as generic functions.

Let's repeat this to make + a function generic!

POFTHEDAY> (org.tfeb.clc:defpackage cl-generic
             (:extends/excluding #:cl
                                 #:+)
             (:export #:+))

;; For simplicity, I'll define this operation as
;; binary. But for real implementation it should
;; support variadic arguments.
POFTHEDAY> (defgeneric cl-generic:+ (left right)
             (:method ((left number) (right number))
               (cl:+ left right))
             (:method ((left string) (right string))
               (concatenate 'string left right))
             (:method ((left string) (right number))
               (format nil "~A~A" left right)))

Now we can define another package which will use this generic function. Note, I'll just :use this new package instead of standard :cl package:

POFTHEDAY> (defpackage foo
             (:use :cl-generic))

POFTHEDAY> (in-package foo)

FOO> (+ 1 2)
3

FOO> (+ "Hello " "World!")
"Hello World!"

;; Other function are standard, becase they are inherited
;; from the standard package:
FOO> (- 5 3)
2

FOO> (- "Hello " "World!")
; Debugger entered on #<TYPE-ERROR expected-type: NUMBER datum: "Hello ">

By the way, you can get the same effect by using uiop:define-package. But it will be a little bit wordy:

POFTHEDAY> (uiop:define-package cl-generic2
             (:use #:cl)
             (:shadow #:+)
             (:reexport #:cl))
#<PACKAGE "CL-GENERIC2">

POFTHEDAY> (defgeneric cl-generic2:+ (left right)
             (:method ((left number) (right number))
               (cl:+ left right))
             (:method ((left string) (right string))
               (concatenate 'string left right))
             (:method ((left string) (right number))
               (format nil "~A~A" left right)))

POFTHEDAY> (defpackage foo2
             (:use :cl-generic2))

POFTHEDAY> (in-package foo)

POFTHEDAY> (in-package foo2)

FOO2> (+ 1 2)
3

FOO2> (+ "Hello " "World!")
"Hello World!"

FOO2> (- 5 3)
2

FOO2> (- "Hello " "World!")
; Debugger entered on #<TYPE-ERROR expected-type: NUMBER datum: "Hello ">

Probably I missed some of the conduit-packages features. Please, read its sources and tell me if you will find something interesting!

Elm2node: transform an elm module to a synchronous node module

via Elm - Latest posts by @luke Luke Westby on Fri, 04 Sep 2020 18:56:28 GMT

Neither of these lines are invoked when passing a value through a port. The first is called only when trying to parse a String, like from the body of an HTTP response. There is also Json.Decode.decodeValue which deals only with Values. The second line similarly is only invoked if you need to render a Value to a String by explicitly calling Json.Encode.encode.

When the compiler generates code for ports it checks the type and sets up the appropriate converter from JS to Elm for that type. Lists and tuples, for example, get special treatment by getting converted to and from JS Arrays, but in the case of a Value, the converter is identity or Json.Decode.value (the Decoder analogue to identity). No stringifying.

Generated port code for a Value looks like this:

var $author$project$Module$portName = _Platform_outgoingPort('portName', $elm$core$Basics$identity);
var $author$project$Module$portName = _Platform_incomingPort('portName', $elm$json$Json$Decode$value);

Those generated arguments are used here https://github.com/elm/core/blob/master/src/Elm/Kernel/Platform.js#L363-L471

Liikennematto devlog #2: build your own roundabout!

via Elm - Latest posts by @system system on Fri, 04 Sep 2020 07:46:32 GMT

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.

Elm2node: transform an elm module to a synchronous node module

via Elm - Latest posts by @pd-andy Andy Thompson on Fri, 04 Sep 2020 05:58:01 GMT

Elm definitely doesn’t do this by default for ports. You can pass in any javascript value you like (stringifiable or not) and elm can handle it fine.