gstreamer gstreamer-vaapi gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-plugins-base gst-libav 1.16.0

[Bruce Dubbs]

New minor version.

[Douglas R. Reno]

Heads up, the rust bindings are now part of this.

[Bruce Dubbs]

The GStreamer team is proud to announce a new major feature release in the stable 1.x API series of your favourite cross-platform multimedia framework!

Highlights

• GStreamer WebRTC stack gained support for data channels for

peer-to-peer communication based on SCTP, BUNDLE support, as well as support for multiple TURN servers.

• AV1 video codec support for Matroska and QuickTime/MP4 containers

and more configuration options and supported input formats for the AOMedia AV1 encoder

• Support for Closed Captions and other Ancillary Data in video

• Support for planar (non-interleaved) raw audio

• GstVideoAggregator, compositor and OpenGL mixer elements are now in

-base

• New alternate fields interlace mode where each buffer carries a

single field

• WebM and Matroska ContentEncryption support in the Matroska demuxer

• new WebKit WPE-based web browser source element

• Video4Linux: HEVC encoding and decoding, JPEG encoding, and improved

dmabuf import/export

• Hardware-accelerated Nvidia video decoder gained support for VP8/VP9

decoding, whilst the encoder gained support for H.265/HEVC encoding.

• Many improvements to the Intel Media SDK based hardware-accelerated

video decoder and encoder plugin (msdk): dmabuf import/export for zero-copy integration with other components; VP9 decoding; 10-bit HEVC encoding; video post-processing (vpp) support including deinterlacing; and the video decoder now handles dynamic resolution changes.

• The ASS/SSA subtitle overlay renderer can now handle multiple

subtitles that overlap in time and will show them on screen simultaneously

• The Meson build is now feature-complete (*) and it is now the

recommended build system on all platforms. The Autotools build is scheduled to be removed in the next cycle.

• The GStreamer Rust bindings and Rust plugins module are now

officially part of upstream GStreamer.

• The GStreamer Editing Services gained a gesdemux element that allows

directly playing back serialized edit list with playbin or (uri)decodebin

• Many performance improvements

Noteworthy new API

• GstAggregator has a new "min-upstream-latency" property that forces

a minimum aggregate latency for the input branches of an aggregator. This is useful for dynamic pipelines where branches with a higher latency might be added later after the pipeline is already up and running and where a change in the latency would be disruptive. This only applies to the case where at least one of the input branches is live though, it won’t force the aggregator into live mode in the absence of any live inputs.

• GstBaseSink gained a "processing-deadline" property and

setter/getter API to configure a processing deadline for live pipelines. The processing deadline is the acceptable amount of time to process the media in a live pipeline before it reaches the sink. This is on top of the systemic latency that is normally reported by the latency query. This defaults to 20ms and should make pipelines such as v4l2src ! xvimagesink not claim that all frames are late in the QoS events. Ideally, this should replace the "max-lateness" property for most applications.

• RTCP Extended Reports (XR) parsing according to RFC 3611:

Loss/Duplicate RLE, Packet Receipt Times, Receiver Reference Time, Delay since the last Receiver (DLRR), Statistics Summary, and VoIP Metrics reports. This only provides the ability to parse such packets, generation of XR packets is not supported yet and XR packets are not automatically parsed by rtpbin / rtpsession but must be actively handled by the application.

• a new mode for interlaced video was added where each buffer carries

a single field of interlaced video, with buffer flags indicating whether the field is the top field or bottom field. Top and bottom fields are expected to alternate in this mode. Caps for this interlace mode must also carry a format:Interlaced caps feature to ensure backwards compatibility.

• The video library has gained support for three new raw pixel

formats:

• Y410: packed 4:4:4 YUV, 10 bits per channel
• Y210: packed 4:2:2 YUV, 10 bits per channel
• NV12_10LE40: fully-packed 10-bit variant of NV12_10LE32,

i.e. without the padding bits

• GstRTPSourceMeta is a new meta that can be used to transport

information about the origin of depayloaded or decoded RTP buffers, e.g. when mixing audio from multiple sources into a single stream. A new "source-info" property on the RTP depayloader base class determines whether depayloaders should put this meta on outgoing buffers. Similarly, the same property on RTP payloaders determines whether they should use the information from this meta to construct the CSRCs list on outgoing RTP buffers.

• gst_sdp_message_from_text() is a convenience constructor to parse

SDPs from a string which is particularly useful for language bindings.

Support for Planar (Non-Interleaved) Raw Audio

Raw audio samples are usually passed around in interleaved form in GStreamer, which means that if there are multiple audio channels the samples for each channel are interleaved in memory, e.g. |LEFT|RIGHT|LEFT|RIGHT|LEFT|RIGHT| for stereo audio. A non-interleaved or planar arrangement in memory would look like |LEFT|LEFT|LEFT|RIGHT|RIGHT|RIGHT| instead, possibly with |LEFT|LEFT|LEFT| and |RIGHT|RIGHT|RIGHT| residing in separate memory chunks or separated by some padding.

GStreamer has always had signalling for non-interleaved audio since version 1.0, but it was never actually properly implemented in any elements. audioconvert would advertise support for it, but wasn’t actually able to handle it correctly.

With this release we now have full support for non-interleaved audio as well, which means more efficient integration with external APIs that handle audio this way, but also more efficient processing of certain operations like interleaving multiple 1-channel streams into a multi-channel stream which can be done without memory copies now.

New API to support this has been added to the GStreamer Audio support library: There is now a new GstAudioMeta which describes how data is laid out inside the buffer, and buffers with non-interleaved audio must always carry this meta. To access the non-interleaved audio samples you must map such buffers with gst_audio_buffer_map() which works much like gst_buffer_map() or gst_video_frame_map() in that it will populate a little GstAudioBuffer helper structure passed to it with the number of samples, the number of planes and pointers to the start of each plane in memory. This function can also be used to map interleaved audio buffers in which case there will be only one plane of interleaved samples.

Of course support for this has also been implemented in the various audio helper and conversion APIs, base classes, and in elements such as audioconvert, audioresample, audiotestsrc, audiorate.

Support for Closed Captions and Other Ancillary Data in Video

The video support library has gained support for detecting and extracting Ancillary Data from videos as per the SMPTE S291M specification, including:

• a VBI (Vertical Blanking Interval) parser that can detect and

extract Ancillary Data from Vertical Blanking Interval lines of component signals. This is currently supported for videos in v210 and UYVY format.

• a new GstMeta for closed captions: GstVideoCaptionMeta. This

supports the two types of closed captions, CEA-608 and CEA-708, along with the four different ways they can be transported (other systems are a superset of those).

• a VBI (Vertical Blanking Interval) encoder for writing ancillary

data to the Vertical Blanking Interval lines of component signals.

The new closedcaption plugin in gst-plugins-bad then makes use of all this new infrastructure and provides the following elements:

• cccombiner: a closed caption combiner that takes a closed captions

stream and another stream and adds the closed captions as GstVideoCaptionMeta to the buffers of the other stream.

• ccextractor: a closed caption extractor which will take

GstVideoCaptionMeta from input buffers and output them as a separate closed captions stream.

• ccconverter: a closed caption converter that can convert between

different formats

• line21encoder, line21decoder: inject/extract line21 closed captions

to/from SD video streams

• cc708overlay: decodes CEA 608/708 captions and overlays them on

video

Additionally, the following elements have also gained Closed Caption support:

• qtdemux and qtmux support CEA 608/708 Closed Caption tracks

• mpegvideoparse, h264parse extracts Closed Captions from MPEG-2/H.264

video streams

• avviddec, avvidenc, x264enc got support for extracting/injecting

Closed Captions

• decklinkvideosink can output closed captions and decklinkvideosrc

can extract closed captions

• playbin and playbin3 learned how to autoplug CEA 608/708 CC overlay

elements

• the externally maintained ajavideosrc element for AJA capture cards

has support for extracting closed captions

The rsclosedcaption plugin in the Rust plugins collection includes a MacCaption (MCC) file parser and encoder.

New Elements

• overlaycomposition: New element that allows applications to draw

GstVideoOverlayCompositions on a stream. The element will emit the "draw" signal for each video buffer, and the application then generates an overlay for that frame (or not). This is much more performant than e.g. cairooverlay for many use cases, e.g. because pixel format conversions can be avoided or the blitting of the overlay can be delegated to downstream elements (such as gloverlaycompositor). It’s particularly useful for cases where only a small section of the video frame should be drawn on.

• gloverlaycompositor: New OpenGL-based compositor element that

flattens any overlays from GstVideoOverlayCompositionMetas into the video stream. This element is also always part of glimagesink.

• glalpha: New element that adds an alpha channel to a video stream.

The values of the alpha channel can either be set to a constant or can be dynamically calculated via chroma keying. It is similar to the existing alpha element but based on OpenGL. Calculations are done in floating point so results may not be identical to the output of the existing alpha element.

• rtpfunnel funnels together RTP streams into a single session. Use

cases include multiplexing and bundle. webrtcbin uses it to implement BUNDLE support.

• testsrcbin is a source element that provides an audio and/or video

stream and also announces them using the recently-introduced GstStream API. This is useful for testing elements such as playbin3 or uridecodebin3 etc.

• New closed caption elements: cccombiner, ccextractor, ccconverter,

line21encoder, line21decoder and cc708overlay (see above)

• wpesrc: new source element acting as a Web Browser based on WebKit

WPE

• Two new OpenCV-based elements: cameracalibrate and cameraundistort

that can communicate to figure out distortion correction parameters for a camera and correct for the distortion.

• New sctp plugin based on usrsctp with sctpenc and sctpdec elements.

These elements are used inside webrtcbin for implementing data channels.

New element features and additions

• playbin3, playbin and playsink have gained a new "text-offset"

property to adjust the positioning of the selected subtitle stream vis-a-vis the audio and video streams. This uses subtitleoverlay’s new "subtitle-ts-offset" property. GstPlayer has gained matching API for this, namely gst_player_get_text_video_offset().

• playbin3 buffering improvements: in network playback scenarios there

may be multiple inputs to decodebin3, and buffering will be done before decodebin3 using queue2 or downloadbuffer elements inside urisourcebin. Since this is before any parsers or demuxers there may not be any bitrate information available for the various streams, so it was difficult to configure the buffering there smartly within global constraints. This was improved now: The queue2 elements inside urisourcebin will now use the new bitrate query to figure out a bitrate estimate for the stream if no bitrate was provided by upstream, and urisourcebin will use the bitrates of the individual queues to distribute the globally-set "buffer-size" budget in bytes to the various queues. urisourcebin also gained "low-watermark" and "high-watermark" properties which will be proxied to the internal queues, as well as a read-only "statistics" property which allows querying of the minimum/maximum/average byte and time levels of the queues inside the urisourcebin in question.

• splitmuxsink has gained a couple of new features:

• new "async-finalize" mode: This mode is useful for muxers or

outputs that can take a long time to finalize a file. Instead of blocking the whole upstream pipeline while the muxer is doing its stuff, we can unlink it and spawn a new muxer + sink combination to continue running normally. This requires us to receive the muxer and sink (if needed) as factories via the new "muxer-factory" and "sink-factory" properties, optionally accompanied by their respective properties structures (set via the new "muxer-properties" and "sink-properties" properties). There are also new "muxer-added" and "sink-added" signals in case custom code has to be called for them to configure them.

• "split-at-running-time" action signal: When called by the user,

this action signal ends the current file (and starts a new one) as soon as the given running time is reached. If called multiple times, running times are queued up and processed in the order they were given.

• "split-after" action signal to finish outputting the current GOP

to the current file and then start a new file as soon as the GOP is finished and a new GOP is opened (unlike the existing "split-now" which immediately finishes the current file and writes the current GOP into the next newly-started file).

• "reset-muxer" property: when unset, the muxer is reset using

flush events instead of setting its state to NULL and back. This means the muxer can keep state across resets, e.g. mpegtsmux will keep the continuity counter continuous across segments as required by hlssink2.

• qtdemux gained PIFF track encryption box support in addition to the

already-existing PIFF sample encryption support, and also allows applications to select which encryption system to use via a "drm-preferred-decryption-system-id" context in case there are multiple options.

• qtmux: the "start-gap-threshold" property determines now whether an

edit list will be created to account for small gaps or offsets at the beginning of a stream in case the start timestamps of tracks don’t line up perfectly. Previously the threshold was hard-coded to 1% of the (video) frame duration, now it is 0 by default (so edit list will be created even for small differences), but fully configurable.

• rtpjitterbuffer has improved end-of-stream handling

• rtpmp4vpay will be prefered over rtpmp4gpay for MPEG-4 video in

autoplugging scenarios now

• rtspsrc now allows applications to send RTSP SET_PARAMETER and

GET_PARAMETER requests using action signals.

• rtspsrc has a small (100ms) configurable teardown delay by default

to try and make sure an RTSP TEARDOWN request gets sent out when the source element shuts down. This will block the downward PAUSED to READY state change for a short time, but can be disabled where it’s a problem. Some servers only allow a limited number of concurrent clients, so if no proper TEARDOWN is sent new clients may have problems connecting to the server for a while.

• souphttpsrc behaves better with low bitrate streams now. Before it

would increase the read block size too quickly which could lead to it not reading any data from the socket for a very long time with low bitrate streams that are output live downstream. This could lead to servers kicking off the client.

• filesink: do internal buffering to avoid performance regression with

small writes since we bypass libc buffering by using writev() instead of fwrite()

• identity: add "eos-after" property and fix "error-after" property

when the element is reused

• input-selector: lets context queries pass through, so that

e.g. upstream OpenGL elements can use contexts and displays advertised by downstream elements

• queue2: avoid ping-pong between 0% and 100% buffering messages if

upstream is pushing buffers larger than one of its limits, plus performance optimisations

• opusdec: new "phase-inversion" property to control phase inversion.

When enabled, this will slightly increase stereo quality, but produces a stream that when downmixed to mono will suffer audio distortions.

• The x265enc HEVC encoder also exposes a "key-int-max" property to

configure the maximum allowed GOP size now.

• decklinkvideosink has seen stability improvements for long-running

pipelines (potential crash due to overflow of leaked clock refcount) and clock-slaving improvements when performing flushing seeks (causing stalls in the output timeline), pausing and/or buffering.

• srtpdec, srtpenc: add support for MKIs which allow multiple keys to

be used with a single SRTP stream

• srtpdec, srtpenc: add support for AES-GCM and also add support for

it in gst-rtsp-server and rtspsrc.

• The srt Secure Reliable Transport plugin has integrated server and

client elements srt{client,server}{src,sink} into one (srtsrc and srtsink), since SRT connection mode can be changed by uri parameters.

• h264parse and h265parse will handle SEI recovery point messages and

mark recovery points as keyframes as well (in addition to IDR frames)

• webrtcbin: "add-turn-server" action signal to pass multiple ICE

relays (TURN servers).

• The removesilence element has received various new features and

properties, such as a "threshold" property, detecting silence only after minimum silence time/buffers, a "silent" property to control bus message notifications as well as a "squash" property.

• AOMedia AV1 decoder gained support for 10/12bit decoding whilst the

AV1 encoder supports more image formats and subsamplings now and acquired support for rate control and profile related configuration.

• The Fraunhofer fdkaac plugin can now be built against the 2.0.0

version API and has improved multichannel support

• kmssink now supports unpadded 24-bit RGB and can configure mode

setting from video info, which enables display of multi-planar formats such as I420 or NV12 with modesetting. It has also gained a number of new properties: The "restore-crtc" property does what it says on the tin and is enabled by default. "plane-properties" and "connector-properties" can be used to pass custom properties to the DRM.

• waylandsink has a "fullscreen" property now and supports the

XDG-Shell protocol.

• decklinkvideosink, decklinkvideosrc support selecting between

half/full duplex

• The vulkan plugin gained support for macOS and iOS via MoltenVK in

addition to the existing support for X11 and Wayland

• imagefreeze has a new num-buffers property to limit the number of

buffers that are produced and to send an EOS event afterwards

• webrtcbin has a new, introspectable get-transceiver signal in

addition to the old get-transceivers signal that couldn’t be used from bindings

• Support for per-element latency information was added to the latency

tracer

Plugin and library moves

• The stereo element was moved from -bad into the existing audiofx

plugin in -good. If you get duplicate type registration warnings when upgrading, check that you don’t have a stale stereoplugin lying about somewhere.

GstVideoAggregator, compositor, and OpenGL mixer elements moved from -bad to -base

GstVideoAggregator is a new base class for raw video mixers and muxers and is based on GstAggregator. It provides defined-latency mixing of raw video inputs and ensures that the pipeline won’t stall even if one of the input streams stops producing data.

As part of the move to stabilise the API there were some last-minute API changes and clean-ups, but those should mostly affect internal elements. Most notably, the "ignore-eos" pad property was renamed to "repeat-after-eos" and the conversion code was moved to a GstVideoAggregatorConvertPad subclass to avoid code duplication, make things less awkward for subclasses like the OpenGL-based video mixer, and make the API more consistent with the audio aggregator API.

It is used by the compositor element, which is a replacement for ‘videomixer’ which did not handle live inputs very well. compositor should behave much better in that respect and generally behave as one would expected in most scenarios.

The compositor element has gained support for per-pad blending mode operators (SOURCE, OVER, ADD) which determines what operator to use for blending this pad over the previous ones. This can be used to implement crossfading and the available operators can be extended in the future as needed.

A number of OpenGL-based video mixer elements (glvideomixer, glmixerbin, glvideomixerelement, glstereomix, glmosaic) which are built on top of GstVideoAggregator have also been moved from -bad to -base now. These elements have been merged into the existing OpenGL plugin, so if you get duplicate type registration warnings when upgrading, check that you don’t have a stale openglmixers plugin lying about somewhere.

Plugin removals

The following plugins have been removed from gst-plugins-bad:

• The experimental daala plugin has been removed, since it’s not so

useful now that all effort is focused on AV1 instead, and it had to be enabled explicitly with --enable-experimental anyway.

• The spc plugin has been removed. It has been replaced by the gme

plugin.

• The acmmp3dec and acmenc plugins for Windows have been removed. ACM

is an ancient legacy API and there was no point in keeping the plugins around for a licensed MP3 decoder now that the MP3 patents have expired and we have a decoder in -good. We also didn’t ship these in our cerbero-built Windows packages, so it’s unlikely that they’ll be missed.

Miscellaneous API additions

• GstBitwriter: new generic bit writer API to complement the existing

bit reader

• gst_buffer_new_wrapped_bytes() creates a wrap buffer from a GBytes

• gst_caps_set_features_simple() sets a caps feature on all the

structures of a GstCaps

• New GST_QUERY_BITRATE query: This allows determining from downstream

what the expected bitrate of a stream may be which is useful in queue2 for setting time based limits when upstream does not provide timing information. tsdemux, qtdemux and matroskademux have basic support for this query on their sink pads.

• elements: there is a new “Hardware” class specifier. Elements

interacting with hardware devices should specify this classifier in their element factory class metadata. This is useful to advertise as one might need to put such elements into READY state to test if the hardware is present in the system for example.

• protection: Add a new definition for unspecified system protection,

GST_PROTECTION_UNSPECIFIED_SYSTEM_ID

• take functions for various mini objects that didn’t have them yet:

gst_query_take(), gst_message_take(), gst_tag_list_take(), gst_buffer_list_take(). Unlike the various _replace() functions _take() does not increase the reference count but takes ownership of the mini object passed.

• clear functions for various mini object types and GstObject which

unrefs the object or mini object (if non-NULL) and sets the variable pointed to to NULL: gst_clear_structure(), gst_clear_tag_list(), gst_clear_query(), gst_clear_message(), gst_clear_event(), gst_clear_caps(), gst_clear_buffer_list(), gst_clear_buffer(), gst_clear_mini_object(), gst_clear_object()

• miniobject: new API gst_mini_object_add_parent() and

gst_mini_object_remove_parent() to set parent pointers on mini objects to ensure correct writability: Every container of miniobjects now needs to store itself as parent in the child object, and remove itself again later. A mini object is then only writable if there is at most one parent, that parent is writable itself, and the reference count of the mini object is 1. GstBuffer (for memories), GstBufferList (for buffers), GstSample (for caps, buffer, bufferlist), and GstVideoOverlayComposition were updated accordingly. Without this it was possible to have e.g. a buffer list with a refcount of 2 used in two places at once that both modify the same buffer with refcount 1 at the same time wrongly thinking it is writable even though it’s really not.

• poll: add API to watch for POLLPRI and stop treating POLLPRI as a

read. This is useful to wait for video4linux events which are signalled via POLLPRI.

• sample: new API to update the contents of a GstSample and make it

writable: gst_sample_set_buffer(), gst_sample_set_caps(), gst_sample_set_segment(), gst_sample_set_info(), plus gst_sample_is_writable() and gst_sample_make_writable(). This makes it possible to reuse a sample object and avoid unnecessary memory allocations, for example in appsink.

• ClockIDs now keep a weak reference to underlying clock to avoid

crashes in basesink in corner cases where a clock goes away while the ClockID is still in use, plus some new API (gst_clock_id_get_clock(), gst_clock_id_uses_clock()) to check the clock a ClockID is linked to.

• The GstCheck unit test library gained a

fail_unless_equals_clocktime() convenience macro as well as some new GstHarness API for for proposing meta APIs from the allocation query: gst_harness_add_propose_allocation_meta(). ASSERT_CRITICAL() checks in unit tests are now skipped if GStreamer was compiled with GST_DISABLE_GLIB_CHECKS.

• gst_audio_buffer_truncate() convenience function to truncate a raw

audio buffer

• GstDiscoverer has support for caching the results of discovery in

the default cache directory. This can be enabled with the use-cache property and is disabled by default.

• GstMeta that are attached to GstBuffers are now always stored in the

order in which they were added.

• Additional support for signalling ONVIF specific features were

added: the SEEK event can store a trickmode-interval now and support for the Rate-Control and Frames RTSP headers was added to the RTSP library.

Much more -- seet the NEWS file in the tarball

[Bruce Dubbs]

Fixed at revision 21512.

[Bruce Dubbs]

Milestone renamed