Btw, setting av1 on channels which don't have any AV1 streams available results in just regular h264 streams being returned by Twitch.

This seems to work. I've been able to watch the stream of r0dn3y with any of the selected streams working, including the 1440p120 hevc stream, but the behavior of the parameter is kind of weird. The current stream of r0dn3y offered 480p30 (avc), 720p60 (av1), 1080p60 (av1) and 1440p120 (hevc). Now when I pass av1 as the parameter, all of the streams will show up and when I pass av1,h264, the 1440p120 will disappear, but other then this quirkiness of the parameter, any of the streams that show up, can be selected and will also be watchable in mpv.

This is not "parameter specific" and has nothing to do with the plugin implementation. It's just the data which is returned by the multivariant playlist when setting the supported_codecs HTTP GET parameter on the usher.ttvnw.net endpoint (see the actual code changes). This is exactly what their site is doing, hence the same stream/quality results.

See the examples in the OP. I have no idea why they don't return all the streams when av1,h264 is set and why av1 does contain additional streams (which aren't even av1). Remember that this is still beta on their end, so this may change.

I don't know why but I have h265 in supported_codecs.

I also get av1,h265,h264 in supported codecs when using Google Chrome but only av1,h264 in Firefox.

Edit: Actually the playlists for just av1 or h265 seem to be identical to av1,h265,h264. Maybe twitch doesn't anticipate users who only support (one of) the new codecs without also supporting h264?

I also get av1,h265,h264 in supported codecs when using Google Chrome but only av1,h264 in Firefox.

You can enable HEVC support on firefox by setting media.wmf.hevc.enabled to 1 in about:config (Windows only and Firefox 120+). After that, HEVC should show up as supported under about:support. See Bug 1853448.

h265 is irrelevant, because it's a patent encumbered codec, hence the development of AV1 by the AOM. As said in the OP, I'm surprised that Twitch even offers streams using this codec in their beta program, because this is all about AV1 support.

If you request the multivariant playlists on Twitch and set h265 as the supported_codecs parameter, then you'll get the same list of streams as when setting av1, namely a mixture of both av1 streams and h265 streams (and regular h264 streams), at least on the r0dn3y channel (I didn't check any other channels). Considering that the stream names are stream0, stream1, stream2, etc., this suggests that this is the naming scheme for the input streams which are uploaded simultaneously, which they have added support for in this beta program. The fact that h265 is included here is an indicator to me that they are just testing stuff with the user inputs. These are not re-encoded streams by Twitch.

There is no reason to add h265 here, because Twitch won't use it.

$ streamlink twitch.tv/r0dn3y --stream-url --twitch-supported-codecs av1 | xargs curl -s | sed -E -e 's/https:.+/.../' -e '/EXT-X-TWITCH-INFO/d'
#EXTM3U
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="2160p60 (source)",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=12583349,RESOLUTION=3840x2160,CODECS="av01.0.13M.08,mp4a.40.2",VIDEO="chunked",FRAME-RATE=60.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="stream0",NAME="1080p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=6675849,RESOLUTION=1920x1080,CODECS="av01.0.09M.08,mp4a.40.2",VIDEO="stream0",FRAME-RATE=60.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="stream1",NAME="720p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=4054318,RESOLUTION=1280x720,CODECS="hvc1.1.2.L120.90.0.0.0.0.0,mp4a.40.2",VIDEO="stream1",FRAME-RATE=60.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="stream2",NAME="480p30",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1383458,RESOLUTION=854x480,CODECS="avc1.64001F,mp4a.40.2",VIDEO="stream2",FRAME-RATE=30.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="audio_only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=160000,CODECS="mp4a.40.2",VIDEO="audio_only"
...

$ streamlink twitch.tv/esl_dota2 --stream-url --twitch-supported-codecs av1 | xargs curl -s | sed -E -e 's/https:.+/.../' -e '/EXT-X-TWITCH-INFO/d'
#EXTM3U
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="chunked",NAME="1080p60 (source)",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=6930562,RESOLUTION=1920x1080,CODECS="avc1.64002A,mp4a.40.2",VIDEO="chunked",FRAME-RATE=60.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p60",NAME="720p60",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=3422999,RESOLUTION=1280x720,CODECS="avc1.4D401F,mp4a.40.2",VIDEO="720p60",FRAME-RATE=60.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="720p30",NAME="720p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=2373000,RESOLUTION=1280x720,CODECS="avc1.4D401F,mp4a.40.2",VIDEO="720p30",FRAME-RATE=30.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="480p30",NAME="480p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=1427999,RESOLUTION=852x480,CODECS="avc1.4D401F,mp4a.40.2",VIDEO="480p30",FRAME-RATE=30.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="360p30",NAME="360p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=630000,RESOLUTION=640x360,CODECS="avc1.4D401F,mp4a.40.2",VIDEO="360p30",FRAME-RATE=30.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="160p30",NAME="160p",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-STREAM-INF:BANDWIDTH=230000,RESOLUTION=284x160,CODECS="avc1.4D401F,mp4a.40.2",VIDEO="160p30",FRAME-RATE=30.000
...
#EXT-X-MEDIA:TYPE=VIDEO,GROUP-ID="audio_only",NAME="audio_only",AUTOSELECT=NO,DEFAULT=NO
#EXT-X-STREAM-INF:BANDWIDTH=160000,CODECS="mp4a.40.2",VIDEO="audio_only"
...

While I do understand you reasoning for preferring AV1 Twitch mentions both AV1 and HEVC on their beta page and obviously allows both AV1 and HEVC streams being served.

Why not just add the value av1,h265,h264 to the possible values so users can get all the streams available? (at least in this "temporary" solution?).
As the streams on r0dn3y channel are configured right now you wouldn't be able to watch the 720p streams.

Twitch mentions both AV1 and HEVC

That is surprising, tbh... Let me update the plugin arg real quick, with support for all possible name combinations.

Rebased to master and made some changes to the new plugin argument setup, so the allowed argument values are defined there and not in the UsherService class.

Simply sideloading the plugin from the PR branch won't work anymore. To anyone who wants to try out this PR, you'll have to install from the PR branch, because it relies on new code on the master branch.

h265 is irrelevant, because it's a patent encumbered codec, hence the development of AV1 by the AOM. As said in the OP, I'm surprised that Twitch even offers streams using this codec in their beta program, because this is all about AV1 support.

Free public streams (even with ad) don't require patent royalties for hevc/h.265
There are royalties per device and cloud services, but hevc (and av1) encoding is supposed to be client-side, not from Twitch servers, so that basically excludes any royalties
And hw decoders (and encoders) are much more common, unlike av1, so that makes sense, also some highlights about Enhanced Broadcasting Beta from a recent podcast:
https://www.twitch.tv/videos/2035515657?t=00h54m54s

The new AV1 and H265 streams are backed by MP4 containers instead of MPEG-TS, which have slightly different streaming capabilities. MP4 is a lot more susceptible to the stream becoming corrupt if packets are dropped or partially written, MPEG-TS inherently allows this and can just re-sync the stream on errors. I'm not sure how much of a problem this is in regular use, I'm not sure if streamlink buffers whole segments before writing them or not, but I imagine at least when using hls-segment-stream-data, if there is any kind of error in the download, the MP4 stream will likely become unreadable. I think one way to fix this would be to buffer the output stream and only write once a complete MP4 box is available, though this would have to be done on an individual segment level.

The new AV1 and H265 streams are backed by MP4 containers instead of MPEG-TS
MP4 is a lot more susceptible to the stream becoming corrupt if packets are dropped or partially written

I'm aware of them having switched the container format, but that shouldn't matter. Streamlink does support HLS maps (initialization sections), which is requirement for fMP4, so we're perfectly fine here. The error resilience of an MPEG transport stream should only be relevant for non-bi-directional data transmissions like DVB, and not for HTTP(S) connections over TCP (with TLS). Twitch is not the only platform which (now) uses fMP4 HLS streams. I've never seen such errors with other fMP4 streams which you worry about.

In regards to stream discontinuities, this also shouldn't matter, because this is based on full HLS segments and not on partial segment data, and (most) decoders should be able to recover from a gap of data here as well.

If hls-segment-stream-data is True, then Streamlink writes 8 KiB data chunks to the output buffer while the data arrives. If it's False, then the full segment gets buffered first before data gets written, which is the default behavior. Encrypted segments never get streamed because of the data-padding at the end of the decrypted AES block sequence.

The problem would be if something happened to the connection mid-segment download, either on the user or the server end. MP4 (ISOBMF) boxes use length prefixing, MPEG-TS uses a fixed packet size and sync bytes. If a write of an MP4 segment is interrupted without writing the specified box length, the stream will get out of sync with no way to re-sync. For MPEG-TS, the demuxer will notice that the sync byte isn't correct and it will re-sync to the next sync byte.

So the problem situation is for example if there was some error in the middle of downloading segment 1000 and it was only partially written to the output, when it starts downloading and writing 1001, the demuxer would still be assuming it's reading the partially written box from segment 1000, and when it "finishes" reading the box and tries to parse the next one it will just see garbage data and be unable to continue decoding the stream.

I consider this issue far too rare for it being relevant. A segment download would need to stop mid-way for some reason while the next segment or another one after that (#5603) would need to continue regularly, so parts of the incomplete bitstream of the previous segment get skipped. As said, this would only work when data gets streamed to the output buffer.

This is not worth making sacrifices in terms of output delay or implementing bitstream inspection in Python before writing data to the output, which would be slow and also rather complex (see https://github.com/beardypig/pymp4).

If a user has an unstable connection and the bitstream-continuation of the HLS segments can't be guaranteed due to that, then they shouldn't set the stream option (which also gets enabled when --twitch-low-latency is set).

I agree it's probably a rare issue, so not sure if it's worth pursuing, but I also don't really think it would be that complex. You don't have to implement full MP4 parsing, you just need to parse the first 32 bits of the top-level box to get the length, buffer that amount, and then write it out, and repeat. If the segment EOFs before the length amount is read, discard the buffer. The buffer would only have to hold the single largest box, which in an fMP4 stream should be fairly small (kilobytes range).

It's March now... Have there been any developments on the side of Twitch?

@bastimeyer I have not seen anything from them regarding this.

It's March now... Have there been any developments on the side of Twitch?

A timeline has not been announced. Though AV1/HEVC is intended to be experimented with more widely as part of the Enhanced Broadcasting beta. So it'll be more than just my channel soon™

OBS Studio 30.2 Beta 1 is out which adds general support for multitrack video encoding ("Enhanced Broadcasting").

It seems Twitch is still rolling out support for this on a per-account basis, there's also this notice on their announcement page:

Update as of Friday, March 15th, 2024. The closed beta is primarily focused on testing the new technologies that enable multiple client-side encodes and automatic stream configuration. Currently, the beta supports streaming up to 5 automatically configured video qualities (H.264/AVC only, 1080p60 max resolution). We are not currently testing HEVC/AV1 or bitrates beyond the current broadcasting guidelines. In the future, we plan to use automatic stream configuration to experiment with new codecs (e.g. HEVC/AV1) and test other new video features.

OBS Studio 30.2 Beta 1 is out which adds general support for multitrack video encoding ("Enhanced Broadcasting").

It seems Twitch is still rolling out support for this on a per-account basis, there's also this notice on their announcement page:

Multitrack streaming with just AVC/H.264 is now available as a public beta, HEVC/AV1 are still later and will be limited to the closed beta group first.

As a little update: There are now a few bigger streamers that have been given the opportunity to trial 4K HEVC streaming, such as maximilian_dood, burkeblack, and EposVox. So this would be nice to have.

I tested the changes for recording twitch streams locally. The CLI works and can record the best quality HEVC stream to an output file. However, the recording seems to have like incorrect metadata. Playing the recording in mpv shows a start time that is the duration of the stream when the recording began + 1 minute. It's strangely very close if not exactly 1 minute ahead of the stream duration. For example, if a stream has gone on for 3:43:00 and you start recording, the recording will start at 3:44:00 even though it should start at 0:00:00. The video otherwise plays correctly what was recorded. Playing the recording in browser shows the correct length of like 10 seconds, but struggles to play it by stopping and waiting for some time before resuming repeatedly as if it was buffering. I ran ffprobe on the file and the duration lines up with what mpv is seeing. I tested recording a normal h264 livestream and it still works as normal with the starting timestamp being 0.

Also, trying to extract the stream urls through the Python API does not return any h265 m3u8 links. I only tested on an h265 stream so I am unsure if it would find AV1 links if available. Seems like only CLI is working.

@vchuynh

However, the recording seems to have like incorrect metadata. Playing the recording in mpv shows a start time that is the duration of the stream when the recording began + 1 minute

This is a side effect of the container format being different on Twitch for H264 vs H265/AV1 streams. H264 uses MPEGTS and H265/AV1 uses MP4. Both of the containers have the same timestamp information inside of them, but mpv/ffmpeg chooses to display them differently between MPEGTS and MP4. You should be able to "fix" this by remuxing the stream with ffmpeg, which by default shifts the start timestamp back to 0.

ffmpeg -i stream.mp4 -map 0 -c copy stream_fixed.mp4

@vchuynh

However, the recording seems to have like incorrect metadata. Playing the recording in mpv shows a start time that is the duration of the stream when the recording began + 1 minute

This is a side effect of the container format being different on Twitch for H264 vs H265/AV1 streams. H264 uses MPEGTS and H265/AV1 uses MP4. Both of the containers have the same timestamp information inside of them, but mpv/ffmpeg chooses to display them differently between MPEGTS and MP4. You should be able to "fix" this by remuxing the stream with ffmpeg, which by default shifts the start timestamp back to 0.

ffmpeg -i stream.mp4 -map 0 -c copy stream_fixed.mp4

Thanks for the info. Didn't realize that's what it was. Was hoping I didn't have to remux, but oh well.

use the options -fflags +genpts and -avoid_negative_ts make_zero for ts files.

We can wrap all instances of TwitchHLSStream where the video codec is not avc1 in a MuxedHLSStream, so the timestamp will be reset to 0 when remuxing using the default configuration.

The solution however is a really ugly hack, on top of this non-ideal plugin-argument implementation, until the entire stream selection logic has been rewritten (#4902 / #5764). It should still work fine though. Let me rebase this, so you can check...

Current:

$ streamlink --stdout -l none --hls-duration=1 --twitch-supported-codecs=h264,h265,av1 twitch.tv/happybombs best \
  | ffprobe -v error -of json -show_packets pipe:0 \
  | jq -r '[.packets[] | select(.codec_type == "video")] | first | .pts'
3342065000

Updated:

$ streamlink --stdout -l none --hls-duration=1 --twitch-supported-codecs=h264,h265,av1 twitch.tv/happybombs best \
  | ffprobe -v error -of json -show_packets pipe:0 \
  | jq -r '[.packets[] | select(.codec_type == "video")] | first | .pts'
131940

Seems to work perfect for me. The recording starts at 0 in mpv and it plays perfectly. Definitely a nicer solution for me. Many thanks for adding this.

Edit: Actually, there is one issue that wasn't there before.

If you jump around an HEVC recording while using mpv, it will look pixelated as if there was a drop in bitrate or the streamer had connection issues. After a moment to a few seconds, it looks perfectly fine. The video itself is fine and if you don't seek or jump around, it looks perfect with no issues. PotPlayer has no issue with seeking and looks perfect. It intuitively seems to me that mpv needs time to process before it starts displaying the video correctly and potplayer is doing something more advanced to not have this problem.

Edit 2: Actually, I suspect the seeking issues is an ffmpeg bug with how it handles mpeg ts files that use hevc based on ffmpeg having an open issue on that topic. Remuxing the recorded produces files that seek perfectly fine.

2k HEVC streaming was released to all affiliates and partners last week. I suspect a lot of streamers are going to use this from now on. While its still in beta, there is 10000 viewers watching high res streams right now, maybe its time to merge to main?
https://help.twitch.tv/s/article/stream-quality?language=en_US

But HEVC isn't H.265 codec? That's a licensing nightmare...

I've just rebased the PR branch onto master and fixed a few minor things. We should be able to merge this soon if there are no issues with the streams.

But as said in #5769 (comment), I don't like this solution at all.

It should probably be noted that Twitch has both geolocked and login-gated AV1/HEVC streams, so auth data (and maybe a proxy if you're not in an approved country) is need to get these streams in the playlist response.

Regarding the MuxedHLSStream hack, is it actually necessary? The offset timestamps don't seem to affect playback in mpv as far as I can tell, though it might vary depending on the player I guess.

It might be worth considering forcing --twitch-disable-ads on HEVC/AV1 streams. Stream discontinuities are much less forgiving on fragmented MP4 streams than they are with MPEG-TS. I don't know if Twitch injects ads into these fMP4 streams yet, but I imagine they eventually will, which would probably completely break playback.

Any ids that are geolocked to test?

It's been reported that any countries outside the following won't receive HEVC/AV1 streams, even when logged in:
Argentina, Australia, Austria, Belgium, Brazil, Canada, Chile, Colombia, Denmark, Finland, France, Germany, Ireland, Italy, Japan, Kuwait, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Qatar, Spain, Sweden, Switzerland, UAE, UK, and USA.
The tweet says "1440p" streams, but in my testing it's any enhanced broadcast encodings (HEVC/AV1). You can find active HEVC streams for testing here: https://eb.rodney.io/

@Hakkin Any examples of vods geo restricted? But I mean the entire vod not some renditions or qualities.

@Hakkin these are all good points

It should probably be noted that Twitch has both geolocked and login-gated AV1/HEVC streams, so auth data (and maybe a proxy if you're not in an approved country) is need to get these streams in the playlist response.

We can add a note to the plugin-argument's help text about region restrictions.

Regarding the MuxedHLSStream hack, is it actually necessary? The offset timestamps don't seem to affect playback in mpv as far as I can tell, though it might vary depending on the player I guess.

Actually true. FFmpeg apparently doesn't set the timestamps to 0 and the timestamps are still anchored to the beginning of the broadcast. -copyts and -start_at_zero don't do anything.

This wasn't the case when I added this commit to the PR initially IIRC. No idea what happened... While it doesn't cause any problems in mpv at least, I haven't tested any other players yet. The playback timeline being anchored to the start of the broadcast is also a bit annoying if you have set a backbuffer, because the visuals for this will be affected by the timestamps.

I'll need to have another look at this.

It might be worth considering forcing --twitch-disable-ads on HEVC/AV1 streams. Stream discontinuities are much less forgiving on fragmented MP4 streams than they are with MPEG-TS. I don't know if Twitch injects ads into these fMP4 streams yet, but I imagine they eventually will, which would probably completely break playback.

Yes, this might actually be necessary because there are ads (of course). This also makes me question whether we can even remove the MuxedHLSStream wrapper, because certain players that don't rely on FFmpeg could probably have an issue with discontinuities. mpv at least (with and without the muxedhlsstream wrapper) continues its playback with garbage image data and no sound, and then it recovers when the correct stream's data arrives again. With the ads being filtered out, garbage data is only visible for a fraction of a second once the playback continues.

It would probably even make sense renaming --twitch-disable-ads to --twitch-enable-ads and making the plugin filter out ad segments by default. Since we request the embedded playerType anyway, there are no ads and just the purple placeholders.

We should probably isolate 9ed6496 from this PR. That would resolve #6564

There's the CHANNEL-METADATA="enhanced_broadcast" attribute on the custom #EXT-X-TWITCH-INFO tag in the multivariant playlist that can be used to figure out that MPEG-4 is being used in the media playlists instead of MPEG-TS.

Apparently, this is not correct.

I've checked a couple of streams, and CHANNEL-METADATA="enhanced_broadcast" is sometimes also set when it's not a higher quality stream. This is really annoying, because this makes figuring out a solution far more difficult.

The channel wirtual for example has the enhanced_broadcast property set, while it's a normal stream with an MPEG2 transport stream. Remuxing this is therefore unnecessary.

I have a local branch with a refactor and cleanup of 9ed6496 that I wanted to push as a separate PR, but now I'm not so sure about this anymore.

Another solution could be in combination with this PR and always requesting av1,h265,h264 and then filtering out unwanted media playlists and checking whether there's a non-h264 stream in the multivariant playlist (which would mean that all stream qualities are using an MPEG-4 container), but this would lead to another problem, because when requesting higher quality streams, some of the re-encoded lower stream qualities are not included in the multivariant playlist that would be there otherwise when only requesting h264. That's obviously not a solution unless we make two separate multivariant playlist requests, one with av1,h265,h264 and one with only h264.

The "proper" solution would be requesting all the media playlists and checking whether there's a segment whose URL ends with .mp4. However, since requests doesn't allow us to make concurrent/async HTTP requests, we'd have to do this synchronously (slow) or spawn worker threads (annoying).

So I guess the CHANNEL-METADATA="enhanced_broadcast" check is "good enough" and some cases of unnecessary remuxing can't be avoided...

I'm still not really convinced such a hack is necessary, in all my testing there's 0 issues with streaming the un-remuxed fMP4 streams from Twitch, besides "the number doesn't start from 0". Playback and seeking both work correctly in mpv and VLC (obviously attempting to seek to a period before the stream was actually downloaded doesn't work, but that's expected). Maybe it would be easier to just include a --ffmpeg-force-remux flag for people who want the stream to be passed through ffmpeg?

Maybe it would be easier to just include a --ffmpeg-force-remux flag for people who want the stream to be passed through ffmpeg?

No. That would require lots and lots of code refactorings and reimplementations if done generically (not plugin-specific) which would then be totally unnecessary duplicate work considering that #4902 will have to be resolved eventually anyway. This is also the reason why the codec selection is done here locally in the plugin.

such a hack is necessary

To be honest, I'm also thinking that remuxing is unnecessary. Sure, the timestamp offset is annoying, but it doesn't affect playback. For local recordings, people can always remux themselves afterwards, or even on-the-fly.