Hacking on the PipeWire GStreamer elements
Last week I attended the GStreamer spring hackfest in Thessaloniki. It was very nice to meet all the usual people again, as it’s been a while (I last attended a GStreamer event in 2022), and we had a great time!
What did I do there? Well, it was a great opportunity to deal with something I don’t normally do but it is quite important that someone eventually does… work on the PipeWire GStreamer elements! (yes, see what I did there? 😉)
Pretty much anyone who has used pipewiresrc or pipewiresink in GStreamer can tell you the experience is not great. Of course you can get things done one way or another, but you can easily find yourself facing issues. The solution is often not to use those elements, at least for audio. pulsesrc and pulsesink do a great job and they are still the recommended way to access audio devices. For video, however, you need pipewiresrc to capture, and that specific scenario works relatively well, but don’t try to share a stream from pipewiresink to pipewiresrc unless you are looking for trouble.
How is it that PipeWire, of all things, does not have good GStreamer support? The truth is, these elements originate all the way back in PulseVideo… PulseVideo was the original project that was then turned into PipeWire after it was rewritten with audio and MIDI support added. The first and most important goal of this project was to allow applications to capture video from a camera, so all the focus was on that use case. And this is the use case that indeed works reasonably well until today. Everything else, including audio support, was later added without paying too much attention to detail, as things were evolving quickly and more attention was needed in the PipeWire core, plugins and session management.
Today, the codebase is a bit of a mess, with unused code paths and variables, sometimes commented out and sometimes not. Some ugly hacks appear here and there, sometimes making you wonder why this actually works (or, does it?). This is what I decided to unravel, clean up and improve during this hackfest.
Thankfully, I had help. One of the first people I met when I arrived at the event was Arun Raghavan, PulseAudio maintainer and GStreamer / PipeWire contributor. Arun had a similar agenda and we had some discussions on this topic, which helped us both to focus on the problem and identify areas of work.
One thing I did fix while being there is the problem of pipewiresink completely hanging when a pipewiresrc that was previously linked to it disconnects. It is a reasonable expectation with these elements to be able to share video from one GStreamer pipeline, using pipewiresink, to another GStreamer pipeline that uses pipewiresrc. Each of those elements creates a node in the PipeWire graph, and assuming the session manager is correctly instructed, these nodes are linked together. When the link is created, a set of buffers is negotiated between them and in the simplest case, PipeWire allocates a set of buffers that are then used to populate the buffer pools of both GStreamer elements. But when one of those pipelines stops, the equivalent node is destroyed from the PipeWire graph and the link inevitably breaks. PipeWire then forcibly removes the allocated buffers from the other node and deallocates them. That is a big issue for that remaining GStreamer pipeline, as the buffers that it was using are suddenly gone.
Focusing on the case where this remaining GStreamer pipeline is the one that contains pipewiresink, what I did was that I added a mechanism to detect when the buffers are removed and post an element error on the GStreamer bus. This makes the application aware and stops the pipeline, which is the only reasonable thing that can be done at this point.
Unfortunately, the case where the remaining pipeline is the one that contains pipewiresrc was not as simple, because the code there weirdly bypasses the buffer pool using a hack that I did not want to touch, in fear of breaking. For that reason, the next thing I started investigating was how to improve the buffer pooling, making the code easier to understand and work with. I made a bit of progress there, but it soon became a big refactoring task.
And this is where all good things come to an end… The hackfest was only 3 days long, and that is never enough time to do complex changes, but at least it was a good excuse to start! I still have a bunch of local refactoring changes that I will soon push upstream, and hopefully there will be more improvements contributed soon.
I want to thank the organizers of the event for all the effort they put in to organize such an enjoyable and productive event as well as Collabora for sponsoring my trip there.
