Published 30 Oct, 2022

Java - How to extract timed ID3 metadata from HLS stream in ExoPlayer?

Category Java
Modified : Nov 30, 2022
100

I have an M3U8 file located here: https://vcloud.blueframetech.com/file/hls/13836.m3u8

This video contains timed metadata every single second. My goal is to read this metadata from ExoPlayer. I currently have the following in my MainActivity.java:

package com.test.exoplayermetadatatest;

import androidx.appcompat.app.AppCompatActivity;

import android.content.Context;
import android.net.Uri;
import android.os.Bundle;
import android.util.Log;

import com.google.android.exoplayer2.ExoPlayerFactory;
import com.google.android.exoplayer2.Player;
import com.google.android.exoplayer2.SimpleExoPlayer;
import com.google.android.exoplayer2.metadata.Metadata;
import com.google.android.exoplayer2.metadata.MetadataOutput;
import com.google.android.exoplayer2.source.MediaSource;
import com.google.android.exoplayer2.source.ProgressiveMediaSource;
import com.google.android.exoplayer2.source.TrackGroup;
import com.google.android.exoplayer2.source.TrackGroupArray;
import com.google.android.exoplayer2.source.hls.HlsMediaSource;
import com.google.android.exoplayer2.trackselection.TrackSelectionArray;
import com.google.android.exoplayer2.ui.PlayerView;
import com.google.android.exoplayer2.upstream.DataSource;
import com.google.android.exoplayer2.upstream.DefaultDataSourceFactory;
import com.google.android.exoplayer2.upstream.DefaultHttpDataSourceFactory;
import com.google.android.exoplayer2.util.Util;

public class MainActivity extends AppCompatActivity implements MetadataOutput, Player.EventListener
{

    @Override
    protected void onCreate ( Bundle savedInstanceState )
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        Context context = getApplicationContext();

        SimpleExoPlayer player = ExoPlayerFactory.newSimpleInstance(context);

        PlayerView view = findViewById(R.id.player);

        view.setPlayer(player);

        DataSource.Factory dataSourceFactory =
            new DefaultHttpDataSourceFactory(Util.getUserAgent(context, "app-name"));

        HlsMediaSource hlsMediaSource =
            new HlsMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse("https://vcloud.blueframetech.com/file/hls/13836.m3u8"));

        player.addMetadataOutput(this);
        player.addListener(this);

        player.prepare(hlsMediaSource);

        player.setPlayWhenReady(true);
    }

    @Override
    public void onTracksChanged( TrackGroupArray trackGroups, TrackSelectionArray trackSelections)
    {
        for ( int i = 0; i < trackGroups.length; i++ )
        {
            TrackGroup trackGroup = trackGroups.get(i);
            for ( int j = 0; j < trackGroup.length; j++ )
            {
                Metadata trackMetadata = trackGroup.getFormat(j).metadata;
                if ( trackMetadata != null )
                {
                    Log.d("METADATA TRACK", trackMetadata.toString());
                }
            }
        }
    }

    @Override
    public void onMetadata ( Metadata metadata )
    {
        Log.d("METADATA", metadata.toString());
    }

}

When the app loads I see the METADATA TRACK log appear a single time, but the METADATA log never appears once. What am I missing or doing wrong?

Answers

There are 1 suggested solutions here and each one has been listed below with a detailed description. The following topics have been covered briefly such as Java, Android, Exoplayer. These have been categorized in sections for a clear and precise explanation.

50

I've got a bit of a long answer here...

The Issue

So first, I noticed that my exact solution worked in ExoPlayer 2.1.1 but not in 2.10.1. This led me to think there had been a regression with ID3 metadata, so I reached out to Google about this via GitHub. They were quick to respond and noticed that there's actually an issue with the metadata in my video. The data_alignment_indicator bit is supposed to be 1 for every packet which is the start of an ID3 tag, and 0 for every packet which is a continuation of a previous ID3 tag (in case an ID3 tag is too large to fit in the 64 kilobyte limit of a single tag). For our content, this bit is always getting set to 0 - meaning that there's no "start of an ID3 tag" anywhere.

The older version of ExoPlayer did not check for this, and therefore did not properly support metadata over 64 kilobytes. The new version does check for this, but therefore can't read our busted video


The Solution

The obviously correct answer is to fix our content, but we have over 100,000 videos with malformed metadata so fixing them all would take a lot of time and money. Instead, we wanted to find a player-side solution. Here's what I was able to do:

1. Pass a custom HlsExtractorFactory to the HlsMediaSource.Factory instance:

HlsMediaSource hlsMediaSource = new HlsMediaSource.Factory(dataSourceFactory)
    .setExtractorFactory(new HlsExtractorFactoryProxy())
    .createMediaSource(Uri.parse("https://vcloud.blueframetech.com/file/hls/13836.m3u8"));

2. Create a custom HlsExtractorFactory

I could not extend DefaultHlsExtractorFactory and did not want to implement my own extractor factory from scratch, so instead I went with a Proxy Pattern

    public class HlsExtractorFactoryProxy implements HlsExtractorFactory
    {

        private DefaultHlsExtractorFactory internal = new DefaultHlsExtractorFactory();

        @Override
        public HlsExtractorFactory.Result createExtractor (
            Extractor previousExtractor,
            Uri uri,
            Format format,
            List<Format> muxedCaptionFormats,
            DrmInitData drmInitData,
            TimestampAdjuster timestampAdjuster,
            Map<String, List<String>> responseHeaders,
            ExtractorInput extractorInput
        )
            throws InterruptedException, IOException
        {
            HlsExtractorFactory.Result result = internal.createExtractor(
                previousExtractor,
                uri,
                format,
                muxedCaptionFormats,
                drmInitData,
                timestampAdjuster,
                responseHeaders,
                extractorInput
            );

            if ( result.extractor instanceof TsExtractor )
            {
                return createNewTsExtractor(
                    0,
                    true,
                    format,
                    muxedCaptionFormats,
                    timestampAdjuster
                );
            }

            return result;
        }

        private HlsExtractorFactory.Result createNewTsExtractor (
            @DefaultTsPayloadReaderFactory.Flags int userProvidedPayloadReaderFactoryFlags,
            boolean exposeCea608WhenMissingDeclarations,
            Format format,
            List<Format> muxedCaptionFormats,
            TimestampAdjuster timestampAdjuster
        )
        {
            @DefaultTsPayloadReaderFactory.Flags
            int payloadReaderFactoryFlags =
                DefaultTsPayloadReaderFactory.FLAG_IGNORE_SPLICE_INFO_STREAM
                | userProvidedPayloadReaderFactoryFlags;
            if ( muxedCaptionFormats != null )
            {
                // The playlist declares closed caption renditions, we should ignore descriptors.
                payloadReaderFactoryFlags |= DefaultTsPayloadReaderFactory.FLAG_OVERRIDE_CAPTION_DESCRIPTORS;
            }
            else if ( exposeCea608WhenMissingDeclarations )
            {
                // The playlist does not provide any closed caption information. We preemptively declare a
                // closed caption track on channel 0.
                muxedCaptionFormats =
                    Collections.singletonList(
                        Format.createTextSampleFormat(
                            null,
                            MimeTypes.APPLICATION_CEA608,
                            0,
                            null
                        ));
            }
            else
            {
                muxedCaptionFormats = Collections.emptyList();
            }
            String codecs = format.codecs;
            if ( !TextUtils.isEmpty(codecs) )
            {
                // Sometimes AAC and H264 streams are declared in TS chunks even though they don't really
                // exist. If we know from the codec attribute that they don't exist, then we can
                // explicitly ignore them even if they're declared.
                if ( !MimeTypes.AUDIO_AAC.equals(MimeTypes.getAudioMediaMimeType(codecs)) )
                {
                    payloadReaderFactoryFlags |= DefaultTsPayloadReaderFactory.FLAG_IGNORE_AAC_STREAM;
                }
                if ( !MimeTypes.VIDEO_H264.equals(MimeTypes.getVideoMediaMimeType(codecs)) )
                {
                    payloadReaderFactoryFlags |= DefaultTsPayloadReaderFactory.FLAG_IGNORE_H264_STREAM;
                }
            }

            TsExtractor extractor = new TsExtractor(
                TsExtractor.MODE_HLS,
                timestampAdjuster,
                new TsPayloadReaderFactoryProxy(payloadReaderFactoryFlags, muxedCaptionFormats)
            );

            return new HlsExtractorFactory.Result(
                extractor,
                false,
                true
            );
        }

    }

This class only exposes one public method, per the HlsExtractorFactory interface: createExtractor. This method runs the DefaultHlsExtractorFactory's createExtractor method and, if it produces a TsExtractor, replaces it with its own custom version of TsExtractor (TsExtractorProxy).

To create this custom TsExtractorProxy I copied the entire contents of the createTsExtractor method from the DefaultHlsExtractorFactory class and changed a single statement:

new TsExtractor(
        TsExtractor.MODE_HLS,
        timestampAdjuster,
new DefaultTsPayloadReaderFactory(payloadReaderFactoryFlags, muxedCaptionFormats));
new TsExtractor(
        TsExtractor.MODE_HLS,
        timestampAdjuster,
new TsPayloadReaderFactoryProxy(payloadReaderFactoryFlags, muxedCaptionFormats));

3. Create the TsPayloadReaderFactory Proxy

As above, I needed to create a proxy here. This one exposed two public methods: createInitialPayloadReaders and createPayloadReader. I only needed to adjust the implementation of createPayloadReader

    public class TsPayloadReaderFactoryProxy implements TsPayloadReader.Factory
    {

        private DefaultTsPayloadReaderFactory internal;

        public TsPayloadReaderFactoryProxy(int payloadReaderFactoryFlags, List<Format>  muxedCaptionFormats)
        {
            internal = new DefaultTsPayloadReaderFactory(payloadReaderFactoryFlags, muxedCaptionFormats);
        }

        @Override
        public SparseArray<TsPayloadReader> createInitialPayloadReaders ()
        {
            return internal.createInitialPayloadReaders();
        }

        @Override
        public TsPayloadReader createPayloadReader (
            int streamType, TsPayloadReader.EsInfo esInfo
        )
        {
            if ( streamType == TsExtractor.TS_STREAM_TYPE_ID3)
            {
                return new PesReader(new Id3ReaderProxy());
            }
            else
            {
                return internal.createPayloadReader(streamType, esInfo);
            }
        }

    }

As you can see more plainly here, when dealing with stream of type TsExtractor.TS_STREAM_TYPE_ID3 instead of instantiating an Id3Reader, I instead instantiate an Id3ReaderProxy

4. Create the Id3Reader Proxy

This class has five public methods, but only one needs to be adjusted: packetStarted. Instead of passing along the flags parameters, I overwrite it with TsPayloadReader.FLAG_DATA_ALIGNMENT_INDICATOR

    class Id3ReaderProxy implements ElementaryStreamReader
    {

        private Id3Reader internal = new Id3Reader();

        @Override
        public void seek ()
        {
            internal.seek();
        }

        @Override
        public void createTracks (
            ExtractorOutput extractorOutput, TsPayloadReader.TrackIdGenerator idGenerator
        )
        {
            internal.createTracks(extractorOutput, idGenerator);
        }

        @Override
        public void packetStarted ( long pesTimeUs, int flags )
        {
            internal.packetStarted(pesTimeUs, TsPayloadReader.FLAG_DATA_ALIGNMENT_INDICATOR);
        }

        @Override
        public void consume ( ParsableByteArray data ) throws ParserException
        {
            internal.consume(data);
        }

        @Override
        public void packetFinished ()
        {
            internal.packetFinished();
        }

    }

With all of that hard work done, I can now get metadata events despite my busted ID3 tags