Streaming video annotation

This page describes how to retrieve analyses of a video stream from a discrete file or from archived data. See live video streaming to learn how to annotate a live video stream.

You can use the streaming API (StreamingVideoIntelligenceService) for real-time video detection with the following features:

You can use Vertex AI AutoML for the following features:

Using the Streaming API

You can split a video file into chunks and stream each chunk by using gRPC to send multiple messages to the Video Intelligence API streaming service. The first message can only contain a StreamingVideoConfig message. All subsequent messages must only contain video data.

The Video Intelligence API streaming service returns a response that contains results based on the feature that you specified in your video config.

In a gRPC request, you write the binary data directly to the stream. You do not need to convert the video content to base64.

We recommend that you keep the size of your video chunks to less than 10 MB.

The Video Intelligence also allows you to specify a Google Cloud Storage bucket for your results.

Supported Video Codecs

Video Intelligence supports the following common video formats:

Format Notes
MTS, FLV All MTS, FLV video formats are supported.
MP4 (MPEG4), MOV, 3GP

The MOOV atom must be at the beginning of the file.

You can use the ffmpeg tool to put the MOOV atom at the beginning of your video file, as shown in the following example:

$ ffmpeg -i input_file.mp4 -c copy -map 0 -movflags faststart output_file.mp4
MKV, WebM

Cues must be at the beginning of the file.

You can use the mkclean tool to move the cue to the start of the file, as shown in the following example:

$ mkclean --keep-cues --optimize in.mkv out.mkv
$ mkclean --keep-cues --optimize --doctype 4 in.webm out.webm

For video formats that are not listed as supported, you can determine if your video file supports streaming by using a named pipe and the ffmpeg tool. The following sample commands decode a video into PNG images at a rate of 0.2 fps. If your file can be successfully decoded using these commands, then you can stream it using the Video Intelligence streaming service.

$ mkfifo /tmp/inpipe
$ ffmpeg -i /tmp/inpipe -r 0.2 output_%04d.png
$ cat [FILE NAME] > /tmp/inpipe