sleap.io.dataset#

A SLEAP dataset collects labeled video frames, together with required metadata.

This contains labeled frame data (user annotations and/or predictions), together with all the other data that is saved for a SLEAP project (videos, skeletons, etc.).

The most convenient way to load SLEAP labels files is to use the high level loader:

> import sleap
> labels = sleap.load_file(filename)

The Labels class provides additional functionality for loading SLEAP labels files. To load a labels dataset file from disk:

> labels = Labels.load_file(filename)

If you’re opening a dataset file created on a different computer (or if you’ve moved the video files), it’s likely that the paths to the original videos will not work. We automatically check for the videos in the same directory as the labels file, but if the videos aren’t there, you can tell load_file where to search for the videos. There are various ways to do this:

> Labels.load_file(filename, single_path_to_search)
> Labels.load_file(filename, [path_a, path_b])
> Labels.load_file(filename, callback_function)
> Labels.load_file(filename, video_search=...)

The callback_function can be created via make_video_callback() and has the option to make a callback with a GUI window so the user can locate the videos.

To save a labels dataset file, run:

> Labels.save_file(labels, filename)

If the filename has a supported extension (e.g., “.slp”, “.h5”, “.json”) then the file will be saved in the corresponding format. You can also specify the default extension to use if none is provided in the filename.

class sleap.io.dataset.Labels(labeled_frames: List[sleap.instance.LabeledFrame] = NOTHING, videos: List[sleap.io.video.Video] = NOTHING, skeletons: List[sleap.skeleton.Skeleton] = NOTHING, nodes: List[sleap.skeleton.Node] = NOTHING, tracks: List[sleap.instance.Track] = NOTHING, suggestions: List[sleap.gui.suggestions.SuggestionFrame] = NOTHING, negative_anchors: Dict[sleap.io.video.Video, list] = NOTHING, provenance: Dict[str, Union[str, int, float, bool]] = NOTHING)[source]#

The Labels class collects the data for a SLEAP project.

This class is front-end for all interactions with loading, writing, and modifying these labels. The actual storage backend for the data is mostly abstracted away from the main interface.

labeled_frames#

A list of LabeledFrame objects

Type: List[sleap.instance.LabeledFrame]

videos#

A list of Video objects that these labels may or may not reference. The video for every LabeledFrame will be stored in videos attribute, but some videos in this list may not have any associated labeled frames.

Type: List[sleap.io.video.Video]

skeletons#

A list of Skeleton objects (again, that may or may not be referenced by an Instance in labeled frame).

Type: List[sleap.skeleton.Skeleton]

tracks#

A list of Track that instances can belong to.

Type: List[sleap.instance.Track]

suggestions#

List that stores “suggested” frames for videos in project. These can be suggested frames for user to label or suggested frames for user to review.

Type: List[sleap.gui.suggestions.SuggestionFrame]

negative_anchors#

Dictionary that stores center-points around which to crop as negative samples when training. Dictionary key is Video, value is list of (frame index, x, y) tuples.

Type: Dict[sleap.io.video.Video, list]

provenance#

Dictionary that denotes the origin of the Labels.

Type: Dict[str, Union[str, int, float, bool]]

add_instance(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)[source]#: Add instance to frame, updating track occupancy.

add_suggestion(video: sleap.io.video.Video, frame_idx: int)[source]#

Add a suggested frame to the labels.

Parameters

video – sleap.Video instance of the suggestion.
frame_idx – Index of the frame of the suggestion.

add_track(video: sleap.io.video.Video, track: sleap.instance.Track)[source]#: Add track to labels, updating occupancy.

add_video(video: sleap.io.video.Video)[source]#

Add a video to the labels if it is not already in it.

Video instances are added automatically when adding labeled frames, but this function allows for adding videos to the labels before any labeled frames are added.

Parameters: video – Video instance

property all_instances: List[sleap.instance.Instance]#: Return list of all instances.

append(value: sleap.instance.LabeledFrame)[source]#: Add labeled frame to list of labeled frames.

append_suggestions(suggestions: List[sleap.gui.suggestions.SuggestionFrame])[source]#: Append the suggested frames.

clear_suggestions()[source]#: Delete all suggestions.

classmethod complex_merge_between(base_labels: sleap.io.dataset.Labels, new_labels: sleap.io.dataset.Labels, unify: bool = True) → tuple[source]#

Merge frames and other data from one dataset into another.

Anything that can be merged cleanly is merged into base_labels.

Frames conflict just in case each labels object has a matching frame (same video and frame idx) with instances not in other.

Frames can be merged cleanly if:

the frame is in only one of the labels, or
the frame is in both labels, but all instances perfectly match (which means they are redundant), or
the frame is in both labels, maybe there are some redundant instances, but only one version of the frame has additional instances not in the other.

Parameters

base_labels – the Labels that we’re merging into
new_labels – the Labels that we’re merging from
unify – whether to replace objects (e.g., Video) in new_labels with matching objects from base

Returns

Dictionary, keys are Video, values are
dictionary in which keys are frame index (int) and value is list of Instance objects
list of conflicting Instance objects from base
list of conflicting Instance objects from new

Return type

tuple of three items

copy() → sleap.io.dataset.Labels[source]#: Return a full deep copy of the labels. .. admonition:: Notes

All objects will be re-created by serializing and then deserializing the labels. This may be slow and will create new instances of all data structures.

delete_suggestions(video)[source]#: Delete suggestions for specified video.

describe()[source]#: Print basic statistics about the labels dataset.

export(filename: str)[source]#

Export labels to analysis HDF5 format.

This expects the labels to contain data for a single video (e.g., predictions).

Parameters: filename – Path to output HDF5 file.

Notes

This will write the contents of the labels out as a HDF5 file without complete metadata.

The resulting file will have datasets:

/node_names: List of skeleton node names.
/track_names: List of track names.
/tracks: All coordinates of the instances in the labels.
/track_occupancy: Mask denoting which instances are present in each
frame.

export_nwb(filename: str, overwrite: bool = False, session_description: str = 'Processed SLEAP pose data', identifier: Optional[str] = None, session_start_time: Optional[datetime.datetime] = None)[source]#

Export all PredictedInstance objects in a Labels object to an NWB file.

Use Labels.numpy to create a pynwb.NWBFile with a separate pynwb.ProcessingModule for each Video in the Labels object.

To access the pynwb.ProcessingModule for a specific Video, use the key ‘SLEAP_VIDEO_{video_idx:03}_{video_fn.stem}’ where isinstance(video_fn, pathlib.PurePath). Ex:

video: ‘path_to_video/my_video.mp4’ video index: 3/5 key: ‘003_my_video’

Within each pynwb.ProcessingModule is a ndx_pose.PoseEstimation for each unique track in the Video.

The ndx_pose.PoseEstimation for each unique Track is stored under the key ‘track{track_idx:03}’ if tracks are set or ‘untrack{track_idx:03}’ if untracked where track_idx ranges from 0 to (number of tracks) - 1. Ex:

track_idx: 1 key: ‘track001’

Each ndx_pose.PoseEstimation has a ndx_pose.PoseEstimationSeries for every Node in the Skeleton.

The ndx_pose.PoseEstimationSeries for a specific Node is stored under the key ‘Node.name’. Ex:

node name: ‘head’ key: ‘head’

Parameters

filename – Output path for the NWB format file.
labels – The Labels object to covert to a NWB format file.
overwrite – Boolean that overwrites existing NWB file if True. If False, data will be appended to existing NWB file.
session_description – Description for entire project. Stored under NWBFile “session_description” key. If appending data to a preexisting file, then the session_description will not be used.
identifier – Unique identifier for project. If no identifier is specified, then will generate a GUID. If appending data to a preexisting file, then the identifier will not be used.
session_start_time – THe datetime associated with the project. If no session_start_time is given, then the current datetime will be used. If appending data to a preexisting file, then the session_start_time will not be used.

Returns

A pynwb.NWBFile with a separate pynwb.ProcessingModule for each Video in the Labels object.

extend_from(new_frames: Union[sleap.io.dataset.Labels, List[sleap.instance.LabeledFrame]], unify: bool = False)[source]#

Merge data from another Labels object or LabeledFrame list.

Arg:: new_frames: the object from which to copy data unify: whether to replace objects in new frames with

corresponding objects from current Labels data

Returns: bool, True if we added frames, False otherwise

extract(inds, copy: bool = False) → sleap.io.dataset.Labels[source]#

Extract labeled frames from indices and return a new Labels object. :param inds: Any valid indexing keys, e.g., a range, slice, list of label indices,

numpy array, Video, etc. See __getitem__ for full list.

Parameters

copy – If True, create a new copy of all of the extracted labeled frames and associated labels. If False (the default), a shallow copy with references to the original labeled frames and other objects will be returned.

Returns

A new Labels object with the specified labeled frames. This will preserve the other data structures even if they are not found in the extracted labels, including:

Labels.videos

Labels.skeletons

Labels.tracks

Labels.suggestions

Labels.provenance

find(video: sleap.io.video.Video, frame_idx: Optional[Union[int, Iterable[int]]] = None, return_new: bool = False) → List[sleap.instance.LabeledFrame][source]#

Search for labeled frames given video and/or frame index.

Parameters

video – A Video that is associated with the project.
frame_idx – The frame index (or indices) which we want to find in the video. If a range is specified, we’ll return all frames with indices in that range. If not specific, then we’ll return all labeled frames for video.
return_new – Whether to return singleton of new and empty LabeledFrame if none is found in project.

Returns

List of LabeledFrame objects that match the criteria. Empty if no matches found, unless return_new is True, in which case it contains a new LabeledFrame with video and frame_index set.

find_first(video: sleap.io.video.Video, frame_idx: Optional[int] = None, use_cache: bool = False) → Optional[sleap.instance.LabeledFrame][source]#

Find the first occurrence of a matching labeled frame.

Matches on frames for the given video and/or frame index.

Parameters

video – A Video instance that is associated with the labeled frames
frame_idx – An integer specifying the frame index within the video
use_cache – Boolean that determines whether Labels.find_first() should instead instead call Labels.find() which uses the labels data cache. If True, use the labels data cache, else loop through all labels to search.

Returns

First LabeledFrame that match the criteria or None if none were found.

find_last(video: sleap.io.video.Video, frame_idx: Optional[int] = None) → Optional[sleap.instance.LabeledFrame][source]#

Find the last occurrence of a matching labeled frame.

Matches on frames for the given video and/or frame index.

Parameters

video – a Video instance that is associated with the labeled frames
frame_idx – an integer specifying the frame index within the video

Returns

Last LabeledFrame that match the criteria or None if none were found.

find_suggestion(video, frame_idx)[source]#: Find SuggestionFrame by video and frame index.

find_track_occupancy(video: sleap.io.video.Video, track: Union[sleap.instance.Track, int], frame_range=None) → List[sleap.instance.Instance][source]#

Get instances for a given video, track, and range of frames.

Parameters

video – the Video
track – the Track or int (“pseudo-track” index to instance list)
frame_range (optional) – If specified, only return instances on frames in range. If None, return all instances for given track.

Returns

List of Instance objects.

static finish_complex_merge(base_labels: sleap.io.dataset.Labels, resolved_frames: List[sleap.instance.LabeledFrame])[source]#

Finish conflicted merge from complex_merge_between.

Parameters

base_labels – the Labels that we’re merging into
resolved_frames – the list of frames to add into base_labels

frames(video: sleap.io.video.Video, from_frame_idx: int = - 1, reverse=False)[source]#

Return an iterator over all labeled frames in a video.

Parameters

video – A Video that is associated with the project.
from_frame_idx – The frame index from which we want to start. Defaults to the first frame of video.
reverse – Whether to iterate over frames in reverse order.

Yields

LabeledFrame

get(key: Union[int, slice, numpy.integer, numpy.ndarray, list, range, sleap.io.video.Video, Tuple[sleap.io.video.Video, Union[numpy.integer, numpy.ndarray, int, list, range]]], *secondary_key: Union[int, slice, numpy.integer, numpy.ndarray, list, range], use_cache: bool = False, raise_errors: bool = False) → Union[sleap.instance.LabeledFrame, List[sleap.instance.LabeledFrame]][source]#

Return labeled frames matching key or return None if not found.

This is a safe version of labels[...] that will not raise an exception if the item is not found.

Parameters

key – Indexing argument to match against. If key is a Video or tuple of (Video, frame_index), frames that match the criteria will be searched for. If a scalar, list, range or array of integers are provided, the labels with those linear indices will be returned.
secondary_key – Numerical indexing argument(s) which supplement key. Only used when key is of type Video.
use_cache – Boolean that determines whether Labels.find_first() should instead instead call Labels.find() which uses the labels data cache. If True, use the labels data cache, else loop through all labels to search.
raise_errors – Boolean that determines whether KeyErrors should be raised. If True, raises KeyErrors, else catches KeyErrors and returns None instead of raising KeyError.

Raises

KeyError – If the specified key could not be found.

Returns

A list with the matching LabeledFrame`s, or a single `LabeledFrame if a scalar key was provided, or None if not found.

get_next_suggestion(video, frame_idx, seek_direction=1)[source]#: Return a (video, frame_idx) tuple seeking from given frame.

get_suggestions() → List[sleap.gui.suggestions.SuggestionFrame][source]#: Return all suggestions as a list of SuggestionFrame items.

get_track_count(video: sleap.io.video.Video) → int[source]#: Return the number of occupied tracks for a given video.

get_track_occupancy(video: sleap.io.video.Video) → List[source]#: Return track occupancy list for given video.

get_unlabeled_suggestion_inds() → List[int][source]#

Find labeled frames for unlabeled suggestions and return their indices.

This is useful for generating a list of example indices for inference on unlabeled suggestions.

Returns

List of indices of the labeled frames that correspond to the suggestions that do not have user instances.

If a labeled frame corresponding to a suggestion does not exist, an empty one will be created.