sleap.io.video¶
Video reading and writing interfaces for different formats.
-
class
sleap.io.video.HDF5Video(filename: str = None, dataset: str = None, input_format: str = 'channels_last', convert_range: bool = True)[source]¶ Video data stored as 4D datasets in HDF5 files.
- Parameters
filename – The name of the HDF5 file where the dataset with video data is stored.
dataset – The name of the HDF5 dataset where the video data is stored.
file_h5 – The h5.File object that the underlying dataset is stored.
dataset_h5 – The h5.Dataset object that the underlying data is stored.
input_format –
A string value equal to either “channels_last” or “channels_first”. This specifies whether the underlying video data is stored as:
”channels_first”: shape = (frames, channels, width, height)
”channels_last”: shape = (frames, width, height, channels)
convert_range – Whether we should convert data to [0, 255]-range
-
get_frame(idx) → numpy.ndarray[source]¶ Get a frame from the underlying HDF5 video data.
- Parameters
idx – The index of the frame to get.
- Returns
The numpy.ndarray representing the video frame data.
-
property
last_frame_idx¶ The idx number of the last frame.
Overrides method of base
Videoclass for videos with select frames indexed by number from original video, since the last frame index here will not match the number of frames in video.
-
class
sleap.io.video.ImgStoreVideo(filename: str = None, index_by_original: bool = True)[source]¶ Video data stored as an ImgStore dataset.
See: https://github.com/loopbio/imgstore This class is just a lightweight wrapper for reading such datasets as video sources for SLEAP.
- Parameters
filename – The name of the file or directory to the imgstore.
index_by_original – ImgStores are great for storing a collection of selected frames from an larger video. If the index_by_original is set to True then the get_frame function will accept the original frame numbers of from original video. If False, then it will accept the frame index from the store directly. Default to True so that we can use an ImgStoreVideo in a dataset to replace another video without having to update all the frame indices on
LabeledFrameobjects in the dataset.
-
get_frame(frame_number: int) → numpy.ndarray[source]¶ Get a frame from the underlying ImgStore video data.
- Parameters
frame_number – The number of the frame to get. If index_by_original is set to True, then this number should actually be a frame index within the imgstore. That is, if there are 4 frames in the imgstore, this number should be be from 0 to 3.
- Returns
The numpy.ndarray representing the video frame data.
-
property
imgstore¶ Get the underlying ImgStore object for this Video.
- Returns
The imgstore that is backing this video object.
-
property
last_frame_idx¶ The idx number of the last frame.
Overrides method of base
Videoclass for videos with select frames indexed by number from original video, since the last frame index here will not match the number of frames in video.
-
class
sleap.io.video.MediaVideo(filename: str, grayscale: bool = NOTHING, bgr: bool = True, dataset: str = '', input_format: str = '')[source]¶ Video data stored in traditional media formats readable by FFMPEG
This class provides bare minimum read only interface on top of OpenCV’s VideoCapture class.
- Parameters
filename – The name of the file (.mp4, .avi, etc)
grayscale – Whether the video is grayscale or not. “auto” means detect based on first frame.
bgr – Whether color channels ordered as (blue, green, red).
-
property
fps¶ Returns frames per second of video.
-
class
sleap.io.video.NumpyVideo(filename: _CountingAttr(counter=31, _default=NOTHING, repr=True, eq=True, order=True, hash=None, init=True, metadata={}))[source]¶ Video data stored as Numpy array.
- Parameters
filename – Either a file to load or a numpy array of the data.
numpy data shape (*) – (frames, width, height, channels)
-
class
sleap.io.video.SingleImageVideo(filename: Optional[str] = None, filenames: Optional[List[str]] = NOTHING, height_: Optional[int] = None, width_: Optional[int] = None, channels_: Optional[int] = None)[source]¶ Video wrapper for individual image files.
- Parameters
filenames – Files to load as video.
-
class
sleap.io.video.Video(backend: Union[sleap.io.video.HDF5Video, sleap.io.video.NumpyVideo, sleap.io.video.MediaVideo, sleap.io.video.ImgStoreVideo, sleap.io.video.SingleImageVideo])[source]¶ The top-level interface to any Video data used by SLEAP.
This class provides a common interface for various supported video data backends. It provides the bare minimum of properties and methods that any video data needs to support in order to function with other SLEAP components. This interface currently only supports reading of video data, there is no write support. Unless one is creating a new video backend, this class should be instantiated from its various class methods for different formats. For example:
>>> video = Video.from_hdf5(filename="test.h5", dataset="box") >>> video = Video.from_media(filename="test.mp4")
Or we can use auto-detection based on filename:
>>> video = Video.from_filename(filename="test.mp4")
- Parameters
backend – A backend is an object that implements the following basic required methods and properties
Properties (*) –
frames: The number of frames in the videochannels: The number of channels in the video (e.g. 1 for grayscale, 3 for RGB)width: The width of each frame in pixelsheight: The height of each frame in pixels
Methods (*) –
get_frame(frame_index: int) -> np.ndarray: Get a single frame from the underlying video data with output shape=(width, height, channels).
-
static
cattr()[source]¶ Returns a cattr converter for serialiazing/deserializing Video objects.
- Returns
A cattr converter.
-
static
fixup_path(path: str, raise_error: bool = False) → str[source]¶ Tries to locate video if the given path doesn’t work.
Given a path to a video try to find it. This is attempt to make the paths serialized for different video objects portable across multiple computers. The default behavior is to store whatever path is stored on the backend object. If this is an absolute path it is almost certainly wrong when transferred when the object is created on another computer. We try to find the video by looking in the current working directory as well.
Note that when loading videos during the process of deserializing a saved
Labelsdataset, it’s usually preferable to fix video paths using a video_callback.- Parameters
path – The path the video asset.
raise_error – Whether to raise error if we cannot find video.
- Raises
FileNotFoundError – If file still cannot be found and raise_error is True.
- Returns
The fixed up path
-
classmethod
from_filename(filename: str, *args, **kwargs) → sleap.io.video.Video[source]¶ Create an instance of a video object, auto-detecting the backend.
- Parameters
filename –
The path to the video filename. Currently supported types are:
Media Videos - AVI, MP4, etc. handled by OpenCV directly
HDF5 Datasets - .h5 files
Numpy Arrays - npy files
- imgstore datasets - produced by loopbio’s Motif recording
system. See: https://github.com/loopbio/imgstore.
args – Arguments to pass to
NumpyVideokwargs – Arguments to pass to
NumpyVideo
- Returns
A Video object with the detected backend.
-
classmethod
from_hdf5(dataset: Union[str, h5py._hl.dataset.Dataset], filename: Union[str, h5py._hl.files.File] = None, input_format: str = 'channels_last', convert_range: bool = True) → sleap.io.video.Video[source]¶ Create an instance of a video object from an HDF5 file and dataset.
This is a helper method that invokes the HDF5Video backend.
- Parameters
dataset – The name of the dataset or and h5.Dataset object. If filename is h5.File, dataset must be a str of the dataset name.
filename – The name of the HDF5 file or and open h5.File object.
input_format – Whether the data is oriented with “channels_first” or “channels_last”
convert_range – Whether we should convert data to [0, 255]-range
- Returns
A Video object with HDF5Video backend.
-
classmethod
from_image_filenames(filenames: List[str], height: Optional[int] = None, width: Optional[int] = None, *args, **kwargs) → sleap.io.video.Video[source]¶ Create an instance of a SingleImageVideo from individual image file(s).
-
classmethod
from_media(filename: str, *args, **kwargs) → sleap.io.video.Video[source]¶ Create an instance of a video object from a typical media file.
For example, mp4, avi, or other types readable by FFMPEG.
- Parameters
filename – The name of the file
args – Arguments to pass to
MediaVideokwargs – Arguments to pass to
MediaVideo
- Returns
A Video object with a MediaVideo backend
-
classmethod
from_numpy(filename: str, *args, **kwargs) → sleap.io.video.Video[source]¶ Create an instance of a video object from a numpy array.
- Parameters
filename – The numpy array or the name of the file
args – Arguments to pass to
NumpyVideokwargs – Arguments to pass to
NumpyVideo
- Returns
A Video object with a NumpyVideo backend
-
get_frame(idx: int) → numpy.ndarray[source]¶ Return a single frame of video from the underlying video data.
- Parameters
idx – The index of the video frame
- Returns
The video frame with shape (width, height, channels)
-
get_frames(idxs: Union[int, Iterable[int]]) → numpy.ndarray[source]¶ Return a collection of video frames from the underlying video data.
- Parameters
idxs – An iterable object that contains the indices of frames.
- Returns
The requested video frames with shape (len(idxs), width, height, channels)
-
classmethod
imgstore_from_filenames(filenames: list, output_filename: str, *args, **kwargs) → sleap.io.video.Video[source]¶ Create an imgstore from a list of image files.
- Parameters
filenames – List of filenames for the image files.
output_filename – Filename for the imgstore to create.
- Returns
A Video object for the new imgstore.
-
property
last_frame_idx¶ The idx number of the last frame. Usually numframes - 1.
-
property
num_frames¶ The number of frames in the video. Just an alias for frames property.
-
property
shape¶ Returns (frame count, height, width, channels).
-
to_hdf5(path: str, dataset: str, frame_numbers: List[int] = None, format: str = '', index_by_original: bool = True)[source]¶ Converts frames from arbitrary video backend to HDF5Video.
Used for building an HDF5 that holds all data needed for training.
- Parameters
path – Filename to HDF5 (which could already exist).
dataset – The HDF5 dataset in which to store video frames.
frame_numbers – A list of frame numbers from the video to save. If None save the entire video.
format – If non-empty, then encode images in format before saving. Otherwise, save numpy matrix of frames.
index_by_original – If the index_by_original is set to True then the get_frame function will accept the original frame numbers of from original video. If False, then it will accept the frame index directly. Default to True so that we can use resulting video in a dataset to replace another video without having to update all the frame indices in the dataset.
- Returns
A new Video object that references the HDF5 dataset.
-
to_imgstore(path: str, frame_numbers: List[int] = None, format: str = 'png', index_by_original: bool = True) → sleap.io.video.Video[source]¶ Converts frames from arbitrary video backend to ImgStoreVideo.
This should facilitate conversion of any video to a loopbio imgstore.
- Parameters
path – Filename or directory name to store imgstore.
frame_numbers – A list of frame numbers from the video to save. If None save the entire video.
format – By default it will create a DirectoryImgStore with lossless PNG format unless the frame_indices = None, in which case, it will default to ‘mjpeg/avi’ format for video.
index_by_original – ImgStores are great for storing a collection of selected frames from an larger video. If the index_by_original is set to True then the get_frame function will accept the original frame numbers of from original video. If False, then it will accept the frame index from the store directly. Default to True so that we can use an ImgStoreVideo in a dataset to replace another video without having to update all the frame indices on
LabeledFrameobjects in the dataset.
- Returns
A new Video object that references the imgstore.