sleap.io.dataset

A SLEAP dataset collects labeled video frames.

This contains labeled frame data (user annotations and/or predictions), together with all the other data that is saved for a SLEAP project (videos, skeletons, negative training sample anchors, etc.).

class sleap.io.dataset.Labels(labeled_frames: List[sleap.instance.LabeledFrame] = NOTHING, videos: List[sleap.io.video.Video] = NOTHING, skeletons: List[sleap.skeleton.Skeleton] = NOTHING, nodes: List[sleap.skeleton.Node] = NOTHING, tracks: List[sleap.instance.Track] = NOTHING, suggestions: List[SuggestionFrame] = NOTHING, negative_anchors: Dict[sleap.io.video.Video, list] = NOTHING)

The Labels class collects the data for a SLEAP project.

This class is front-end for all interactions with loading, writing, and modifying these labels. The actual storage backend for the data is mostly abstracted away from the main interface.

labeled_frames

A list of LabeledFrame objects

videos

A list of Video objects that these labels may or may not reference. The video for every LabeledFrame will be stored in videos attribute, but some videos in this list may not have any associated labeled frames.

skeletons

A list of Skeleton objects (again, that may or may not be referenced by an Instance in labeled frame).

tracks

A list of Track that instances can belong to.

suggestions

List that stores “suggested” frames for videos in project. These can be suggested frames for user to label or suggested frames for user to review.

negative_anchors

Dictionary that stores center-points around which to crop as negative samples when training. Dictionary key is Video, value is list of (frame index, x, y) tuples.

add_instance(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)

Adds instance to frame, updating track occupancy.

add_track(video: sleap.io.video.Video, track: sleap.instance.Track)

Adds track to labels, updating occupancy.

add_video(video: sleap.io.video.Video)

Add a video to the labels if it is not already in it.

Video instances are added automatically when adding labeled frames, but this function allows for adding videos to the labels before any labeled frames are added.

Parameters

video – Video instance

property all_instances

Returns list of all instances.

append(value: sleap.instance.LabeledFrame)

Adds labeled frame to list of labeled frames.

classmethod complex_merge_between(base_labels: sleap.io.dataset.Labels, new_labels: sleap.io.dataset.Labels, unify: bool = True) → tuple

Merge frames and other data from one dataset into another.

Anything that can be merged cleanly is merged into base_labels.

Frames conflict just in case each labels object has a matching frame (same video and frame idx) with instances not in other.

Frames can be merged cleanly if:

• the frame is in only one of the labels, or

• the frame is in both labels, but all instances perfectly match (which means they are redundant), or

• the frame is in both labels, maybe there are some redundant instances, but only one version of the frame has additional instances not in the other.

Parameters

• base_labels – the Labels that we’re merging into

• new_labels – the Labels that we’re merging from

• unify – whether to replace objects (e.g., Video) in new_labels with matching objects from base

Returns

• Dictionary, keys are Video, values are

  dictionary in which keys are frame index (int) and value is list of Instance objects

• list of conflicting Instance objects from base

• list of conflicting Instance objects from new

Return type

tuple of three items

delete_suggestions(video)

Deletes suggestions for specified video.

export_training_data(save_path: str)

Exports a set of images and points for training with minimal metadata.

Parameters

save_path – Path to HDF5 that training data will be saved to.

Notes

The exported HDF5 file will contain no SLEAP-specific metadata or dependencies for serialization. These files cannot be read back in for labeling, but are useful when training on environments where it is hard to install complex dependencies.

extend_from(new_frames: Union[Labels, List[sleap.instance.LabeledFrame]], unify: bool = False)

Merge data from another Labels object or LabeledFrame list.

Arg:

new_frames: the object from which to copy data unify: whether to replace objects in new frames with

corresponding objects from current Labels data

Returns

bool, True if we added frames, False otherwise

find(video: sleap.io.video.Video, frame_idx: Union[int, range, None] = None, return_new: bool = False) → List[sleap.instance.LabeledFrame]

Search for labeled frames given video and/or frame index.

Parameters

• video – A Video that is associated with the project.

• frame_idx – The frame index (or indices) which we want to find in the video. If a range is specified, we’ll return all frames with indices in that range. If not specific, then we’ll return all labeled frames for video.

• return_new – Whether to return singleton of new and empty LabeledFrame if none is found in project.

Returns

List of LabeledFrame objects that match the criteria. Empty if no matches found, unless return_new is True, in which case it contains a new LabeledFrame with video and frame_index set.

find_first(video: sleap.io.video.Video, frame_idx: Optional[int] = None) → Optional[sleap.instance.LabeledFrame]

Finds the first occurrence of a matching labeled frame.

Matches on frames for the given video and/or frame index.

Parameters

• video – a Video instance that is associated with the labeled frames

• frame_idx – an integer specifying the frame index within the video

Returns

First LabeledFrame that match the criteria or None if none were found.

find_last(video: sleap.io.video.Video, frame_idx: Optional[int] = None) → Optional[sleap.instance.LabeledFrame]

Finds the last occurrence of a matching labeled frame.

Matches on frames for the given video and/or frame index.

Parameters

• video – a Video instance that is associated with the labeled frames

• frame_idx – an integer specifying the frame index within the video

Returns

Last LabeledFrame that match the criteria or None if none were found.

find_suggestion(video, frame_idx)

Find SuggestionFrame by video and frame index.

find_track_occupancy(video: sleap.io.video.Video, track: Union[sleap.instance.Track, int], frame_range=None) → List[sleap.instance.Instance]

Get instances for a given video, track, and range of frames.

Parameters

• video – the Video

• track – the Track or int (“pseudo-track” index to instance list)

• frame_range (optional) – If specified, only return instances on frames in range. If None, return all instances for given track.

Returns

List of Instance objects.

static finish_complex_merge(base_labels: sleap.io.dataset.Labels, resolved_frames: List[sleap.instance.LabeledFrame])

Finish conflicted merge from complex_merge_between.

Parameters

• base_labels – the Labels that we’re merging into

• resolved_frames – the list of frames to add into base_labels

Returns

None.

frames(video: sleap.io.video.Video, from_frame_idx: int = -1, reverse=False)

Iterator over all labeled frames in a video.

Parameters

• video – A Video that is associated with the project.

• from_frame_idx – The frame index from which we want to start. Defaults to the first frame of video.

• reverse – Whether to iterate over frames in reverse order.

Yields

LabeledFrame

generate_training_data() → Tuple[Union[numpy.ndarray, List[numpy.ndarray]], List[numpy.ndarray]]

Generates images and points for training.

Returns

A tuple of (imgs, points).

imgs: Array of shape (n_samples, height, width, channels) containing the image data for all frames with user labels. If frames are of variable size, imgs is a list of length n_samples with elements of shape (height, width, channels).

points: List of length n_samples with elements of shape (n_instances, n_nodes, 2), containing all user labeled instances in the frame, with NaN-padded xy coordinates for each visible body part.

get_next_suggestion(video, frame_idx, seek_direction=1) → list

Returns a (video, frame_idx) tuple seeking from given frame.

get_suggestions() → list

Return all suggestions as a list of SuggestionFrame items.

get_track_count(video: sleap.io.video.Video) → int

Returns the number of occupied tracks for a given video.

get_track_occupancy(video: sleap.io.video.Video) → List

Returns track occupancy list for given video

get_video_suggestions(video: sleap.io.video.Video) → list

Returns the list of suggested frames for the specified video or suggestions for all videos (if no video specified).

index(value) → int

Returns index of labeled frame in list of labeled frames.

insert(index, value: sleap.instance.LabeledFrame)

Inserts labeled frame at given index.

instance_count(video: sleap.io.video.Video, frame_idx: int) → int

Returns number of instances matching video/frame index.

instances(video: sleap.io.video.Video = None, skeleton: sleap.skeleton.Skeleton = None)

Iterate over instances in the labels, optionally with filters.

Parameters

• video – Only iterate through instances in this video

• skeleton – Only iterate through instances with this skeleton

Yields

Instance – The next labeled instance

property labels

Alias for labeled_frames.

classmethod load_file(filename: str, *args, **kwargs)

Load file, detecting format from filename.

classmethod make_video_callback(search_paths: Optional[List] = None, use_gui: bool = False) → Callable

Create a callback for finding missing videos.

The callback can be used while loading a saved project and allows the user to find videos which have been moved (or have paths from a different system).

The callback function returns True to signal “abort”.

Parameters

search_paths – If specified, this is a list of paths where we’ll automatically try to find the missing videos.

Returns

The callback function.

static merge_container_dicts(dict_a: Dict, dict_b: Dict) → Dict

Merge data from dict_b into dict_a.

merge_matching_frames(video: Optional[sleap.io.video.Video] = None)

Merge LabeledFrame objects that are for the same video frame.

Parameters

video – combine for this video; if None, do all videos

Returns

None

property predicted_instances

Returns list of all user (non-predicted) instances.

remove(value: sleap.instance.LabeledFrame)

Removes given labeled frame.

remove_instance(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance, in_transaction: bool = False)

Removes instance from frame, updating track occupancy.

remove_video(video: sleap.io.video.Video)

Removes a video from the labels and ALL associated labeled frames.

Parameters

video – Video instance to be removed

classmethod save_file(labels: sleap.io.dataset.Labels, filename: str, default_suffix: str = '', *args, **kwargs)

Save file, detecting format from filename.

Parameters

• labels – The dataset to save.

• filename – Path where we’ll save it. We attempt to detect format from the suffix (e.g., “.json”).

• default_suffix – If we can’t detect valid suffix on filename, we can add default suffix to filename (and use corresponding format). Doesn’t need to have “.” before file extension.

Raises

ValueError – If cannot detect valid filetype.

Returns

None.

save_frame_data_hdf5(output_path: str, format: str = 'png', all_labels: bool = False)

Write images for labeled frames from all videos to hdf5 file.

Note that this will make an HDF5 video, not an HDF5 labels dataset.

Parameters

• output_path – Path to HDF5 file.

• format – The image format to use for the data. Defaults to png.

• all_labels – Include any labeled frames, not just the frames we’ll use for training (i.e., those with Instances).

Returns

A list of HDF5Video objects with the stored frames.

save_frame_data_imgstore(output_dir: str = './', format: str = 'png', all_labels: bool = False)

Write images for labeled frames from all videos to imgstore datasets.

This only writes frames that have been labeled. Videos without any labeled frames will be included as empty imgstores.

Parameters

• output_dir – Path to directory which will contain imgstores.

• format – The image format to use for the data. Use “png” for lossless, “jpg” for lossy. Other imgstore formats will probably work as well but have not been tested.

• all_labels – Include any labeled frames, not just the frames we’ll use for training (i.e., those with Instance objects ).

Returns

A list of ImgStoreVideo objects with the stored frames.

set_suggestions(suggestions: List[SuggestionFrame])

Sets the suggested frames.

to_dict(skip_labels: bool = False)

Serialize all labels in the underling list of LabeledFrames to a dict structure. This function returns a nested dict structure composed entirely of primitive python types. It is used to create JSON and HDF5 serialized datasets.

Parameters

• skip_labels – If True, skip labels serialization and just do the

• metadata. –

Returns

• version - The version of the dict/json serialization format.

• skeletons - The skeletons associated with these underlying instances.

• nodes - The nodes that the skeletons represent.

• videos - The videos that that the instances occur on.

• labels - The labeled frames

• tracks - The tracks associated with each instance.

• suggestions - The suggested frames.

• negative_anchors - The negative training sample anchors.

Return type

A dict containing the followings top level keys

to_json()

Serialize all labels in the underling list of LabeledFrame(s) to a JSON structured string.

Returns

The JSON representation of the string.

track_set_instance(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance, new_track: sleap.instance.Track)

Sets track on given instance, updating occupancy.

track_swap(video: sleap.io.video.Video, new_track: sleap.instance.Track, old_track: Optional[sleap.instance.Track], frame_range: tuple)

Swaps track assignment for instances in two tracks.

If you need to change the track to or from None, you’ll need to use track_set_instance() for each specific instance you want to modify.

Parameters

• video – The Video for which we want to swap tracks.

• new_track – A Track for which we want to swap instances with another track.

• old_track – The other Track for swapping.

• frame_range – Tuple of (start, end) frame indexes. If you want to swap tracks on a single frame, use (frame index, frame index + 1).

Returns

None.

property user_instances

Returns list of all user (non-predicted) instances.

property user_labeled_frames

Returns all labeled frames with user (non-predicted) instances.

class sleap.io.dataset.LabelsDataCache(labels: Labels)

Class for maintaining cache of data in labels dataset.

add_instance(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)

Updates cache as needed.

add_track(video: sleap.io.video.Video, track: sleap.instance.Track)

Updates cache as needed.

find_fancy_frame_idxs(video, from_frame_idx, reverse)

Returns a list of frame idxs, with optional start position/order.

find_frames(video: sleap.io.video.Video, frame_idx: Union[int, range, None] = None) → Optional[List[sleap.instance.LabeledFrame]]

Returns list of LabeledFrames matching video/frame_idx, or None.

get_filtered_frame_idxs(video: Optional[sleap.io.video.Video] = None, filter: str = '')

Returns list of (video_idx, frame_idx) tuples matching video/filter.

get_frame_count(video: Optional[sleap.io.video.Video] = None, filter: str = '')

Returns (possibly cached) count of frames matching video/filter.

get_track_occupancy(video: sleap.io.video.Video, track: sleap.instance.Track) → sleap.rangelist.RangeList

Accessor for track occupancy cache that adds video/track as needed.

get_video_track_occupancy(video: sleap.io.video.Video) → Dict[sleap.instance.Track, sleap.rangelist.RangeList]

Returns track occupancy information for specified video.

remove_frame(frame: sleap.instance.LabeledFrame)

Updates cache as needed.

remove_instance(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)

Updates cache as needed.

remove_video(video: sleap.io.video.Video)

Updates cache as needed.

track_swap(video: sleap.io.video.Video, new_track: sleap.instance.Track, old_track: Optional[sleap.instance.Track], frame_range: tuple)

Updates cache as needed.

update(new_frame: Optional[sleap.instance.LabeledFrame] = None)

Builds (or rebuilds) various caches.

update_counts_for_frame(frame: sleap.instance.LabeledFrame)

Updated the cached count. Should be called after frame is modified.