sleap.io.dataset¶
A SLEAP dataset collects labeled video frames, together with required metadata.
This contains labeled frame data (user annotations and/or predictions), together with all the other data that is saved for a SLEAP project (videos, skeletons, etc.).
The most convenient way to load SLEAP labels files is to use the high level loader:
> import sleap > labels = sleap.load_file(filename)
The Labels class provides additional functionality for loading SLEAP labels files. To load a labels dataset file from disk:
> labels = Labels.load_file(filename)
If you’re opening a dataset file created on a different computer (or if you’ve moved the video files), it’s likely that the paths to the original videos will not work. We automatically check for the videos in the same directory as the labels file, but if the videos aren’t there, you can tell load_file where to search for the videos. There are various ways to do this:
> Labels.load_file(filename, single_path_to_search) > Labels.load_file(filename, [path_a, path_b]) > Labels.load_file(filename, callback_function) > Labels.load_file(filename, video_search=…)
The callback_function can be created via make_video_callback() and has the option to make a callback with a GUI window so the user can locate the videos.
To save a labels dataset file, run:
> Labels.save_file(labels, filename)
If the filename has a supported extension (e.g., “.slp”, “.h5”, “.json”) then the file will be saved in the corresponding format. You can also specify the default extension to use if none is provided in the filename.
-
class
sleap.io.dataset.
Labels
(labeled_frames: List[sleap.instance.LabeledFrame] = NOTHING, videos: List[sleap.io.video.Video] = NOTHING, skeletons: List[sleap.skeleton.Skeleton] = NOTHING, nodes: List[sleap.skeleton.Node] = NOTHING, tracks: List[sleap.instance.Track] = NOTHING, suggestions: List[SuggestionFrame] = NOTHING, negative_anchors: Dict[sleap.io.video.Video, list] = NOTHING, provenance: Dict[str, Union[str, int, float]] = NOTHING)[source]¶ The
Labels
class collects the data for a SLEAP project.This class is front-end for all interactions with loading, writing, and modifying these labels. The actual storage backend for the data is mostly abstracted away from the main interface.
-
labeled_frames
¶ A list of
LabeledFrame
objects
-
videos
¶ A list of
Video
objects that these labels may or may not reference. The video for every LabeledFrame will be stored in videos attribute, but some videos in this list may not have any associated labeled frames.
-
skeletons
¶ A list of
Skeleton
objects (again, that may or may not be referenced by anInstance
in labeled frame).
-
tracks
¶ A list of
Track
that instances can belong to.
-
suggestions
¶ List that stores “suggested” frames for videos in project. These can be suggested frames for user to label or suggested frames for user to review.
-
negative_anchors
¶ Dictionary that stores center-points around which to crop as negative samples when training. Dictionary key is
Video
, value is list of (frame index, x, y) tuples.
-
add_instance
(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)[source]¶ Add instance to frame, updating track occupancy.
-
add_track
(video: sleap.io.video.Video, track: sleap.instance.Track)[source]¶ Add track to labels, updating occupancy.
-
add_video
(video: sleap.io.video.Video)[source]¶ Add a video to the labels if it is not already in it.
Video instances are added automatically when adding labeled frames, but this function allows for adding videos to the labels before any labeled frames are added.
- Parameters
video – Video instance
-
property
all_instances
¶ Return list of all instances.
-
append
(value: sleap.instance.LabeledFrame)[source]¶ Add labeled frame to list of labeled frames.
-
classmethod
complex_merge_between
(base_labels: sleap.io.dataset.Labels, new_labels: sleap.io.dataset.Labels, unify: bool = True) → tuple[source]¶ Merge frames and other data from one dataset into another.
Anything that can be merged cleanly is merged into base_labels.
Frames conflict just in case each labels object has a matching frame (same video and frame idx) with instances not in other.
Frames can be merged cleanly if:
the frame is in only one of the labels, or
the frame is in both labels, but all instances perfectly match (which means they are redundant), or
the frame is in both labels, maybe there are some redundant instances, but only one version of the frame has additional instances not in the other.
- Parameters
base_labels – the Labels that we’re merging into
new_labels – the Labels that we’re merging from
unify – whether to replace objects (e.g., Video) in new_labels with matching objects from base
- Returns
- Dictionary, keys are
Video
, values are dictionary in which keys are frame index (int) and value is list of
Instance
objects
- Dictionary, keys are
list of conflicting
Instance
objects from baselist of conflicting
Instance
objects from new
- Return type
tuple of three items
-
extend_from
(new_frames: Union[Labels, List[sleap.instance.LabeledFrame]], unify: bool = False)[source]¶ Merge data from another Labels object or LabeledFrame list.
- Arg:
new_frames: the object from which to copy data unify: whether to replace objects in new frames with
corresponding objects from current Labels data
- Returns
bool, True if we added frames, False otherwise
-
find
(video: sleap.io.video.Video, frame_idx: Optional[Union[int, Iterable[int]]] = None, return_new: bool = False) → List[sleap.instance.LabeledFrame][source]¶ Search for labeled frames given video and/or frame index.
- Parameters
video – A
Video
that is associated with the project.frame_idx – The frame index (or indices) which we want to find in the video. If a range is specified, we’ll return all frames with indices in that range. If not specific, then we’ll return all labeled frames for video.
return_new – Whether to return singleton of new and empty
LabeledFrame
if none is found in project.
- Returns
List of LabeledFrame objects that match the criteria. Empty if no matches found, unless return_new is True, in which case it contains a new LabeledFrame with video and frame_index set.
-
find_first
(video: sleap.io.video.Video, frame_idx: Optional[int] = None) → Optional[sleap.instance.LabeledFrame][source]¶ Find the first occurrence of a matching labeled frame.
Matches on frames for the given video and/or frame index.
- Parameters
video – a Video instance that is associated with the labeled frames
frame_idx – an integer specifying the frame index within the video
- Returns
First LabeledFrame that match the criteria or None if none were found.
-
find_last
(video: sleap.io.video.Video, frame_idx: Optional[int] = None) → Optional[sleap.instance.LabeledFrame][source]¶ Find the last occurrence of a matching labeled frame.
Matches on frames for the given video and/or frame index.
- Parameters
video – a Video instance that is associated with the labeled frames
frame_idx – an integer specifying the frame index within the video
- Returns
Last LabeledFrame that match the criteria or None if none were found.
-
find_track_occupancy
(video: sleap.io.video.Video, track: Union[sleap.instance.Track, int], frame_range=None) → List[sleap.instance.Instance][source]¶ Get instances for a given video, track, and range of frames.
- Parameters
video – the Video
track – the Track or int (“pseudo-track” index to instance list)
frame_range (optional) – If specified, only return instances on frames in range. If None, return all instances for given track.
- Returns
List of
Instance
objects.
-
static
finish_complex_merge
(base_labels: sleap.io.dataset.Labels, resolved_frames: List[sleap.instance.LabeledFrame])[source]¶ Finish conflicted merge from complex_merge_between.
- Parameters
base_labels – the Labels that we’re merging into
resolved_frames – the list of frames to add into base_labels
-
frames
(video: sleap.io.video.Video, from_frame_idx: int = - 1, reverse=False)[source]¶ Return an iterator over all labeled frames in a video.
- Parameters
video – A
Video
that is associated with the project.from_frame_idx – The frame index from which we want to start. Defaults to the first frame of video.
reverse – Whether to iterate over frames in reverse order.
- Yields
LabeledFrame
-
get_next_suggestion
(video, frame_idx, seek_direction=1)[source]¶ Return a (video, frame_idx) tuple seeking from given frame.
-
get_suggestions
() → List[sleap.gui.suggestions.SuggestionFrame][source]¶ Return all suggestions as a list of SuggestionFrame items.
-
get_track_count
(video: sleap.io.video.Video) → int[source]¶ Return the number of occupied tracks for a given video.
-
get_track_occupancy
(video: sleap.io.video.Video) → List[source]¶ Return track occupancy list for given video.
-
get_video_suggestions
(video: sleap.io.video.Video) → List[int][source]¶ Return a list of suggested frame indices.
- Parameters
video – Video to get suggestions for.
- Returns
Indices of the labeled frames for for the specified video.
-
has_frame
(lf: Optional[sleap.instance.LabeledFrame] = None, video: Optional[sleap.io.video.Video] = None, frame_idx: Optional[int] = None, use_cache: bool = True) → bool[source]¶ Check if the labels contain a specified frame.
- Parameters
lf – LabeledFrame to search for. If not provided, the video and frame_idx must not be None.
video – Video of the frame. Not necessary if lf is given.
frame_idx – Integer frame index of the frame. Not necessary if lf is given.
use_cache – If True (the default), use label lookup cache for faster searching. If False, check every frame without the cache.
- Returns
A bool indicating whether the specified LabeledFrame is contained in the labels.
This will return True if there is a matching frame with the same video and frame index, even if they contain different instances.
Notes
The Video instance must be the same as the ones in these labels, so if comparing to Video`s loaded from another file, be sure to load those labels with matching, i.e.: `sleap.Labels.load_file(…, match_to=labels).
-
property
has_missing_videos
¶ Return True if any of the video files in the labels are missing.
-
insert
(index, value: sleap.instance.LabeledFrame)[source]¶ Insert labeled frame at given index.
-
instance_count
(video: sleap.io.video.Video, frame_idx: int) → int[source]¶ Return number of instances matching video/frame index.
-
instances
(video: sleap.io.video.Video = None, skeleton: sleap.skeleton.Skeleton = None)[source]¶ Iterate over instances in the labels, optionally with filters.
- Parameters
video – Only iterate through instances in this video
skeleton – Only iterate through instances with this skeleton
- Yields
Instance – The next labeled instance
-
property
labels
¶ Alias for labeled_frames.
-
classmethod
load_file
(filename: str, video_search: Optional[Union[Callable, List[str]]] = None, *args, **kwargs)[source]¶ Load file, detecting format from filename.
-
classmethod
make_video_callback
(search_paths: Optional[List] = None, use_gui: bool = False) → Callable[source]¶ Create a callback for finding missing videos.
The callback can be used while loading a saved project and allows the user to find videos which have been moved (or have paths from a different system).
The callback function returns True to signal “abort”.
- Parameters
search_paths – If specified, this is a list of paths where we’ll automatically try to find the missing videos.
- Returns
The callback function.
-
static
merge_container_dicts
(dict_a: Dict, dict_b: Dict) → Dict[source]¶ Merge data from dict_b into dict_a.
-
merge_matching_frames
(video: Optional[sleap.io.video.Video] = None)[source]¶ Merge LabeledFrame objects that are for the same video frame.
- Parameters
video – combine for this video; if None, do all videos
-
property
predicted_instances
¶ Return list of all predicted instances.
-
remove
(value: sleap.instance.LabeledFrame)[source]¶ Remove given labeled frame.
-
remove_frame
(lf: sleap.instance.LabeledFrame, update_cache: bool = True)[source]¶ Remove a given labeled frame.
- Parameters
lf – Labeled frame instance to remove.
update_cache – If True, update the internal frame cache. If False, cache update can be postponed (useful when removing many frames).
-
remove_frames
(lfs: List[sleap.instance.LabeledFrame])[source]¶ Remove a list of frames from the labels.
- Parameters
lfs – A sequence of labeled frames to remove.
-
remove_instance
(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance, in_transaction: bool = False)[source]¶ Remove instance from frame, updating track occupancy.
-
remove_predictions
(new_labels: Optional[Labels] = None)[source]¶ Clear predicted instances from the labels.
Useful prior to merging operations to prevent overlapping instances from new predictions.
- Parameters
new_labels – If not None, only predicted instances in frames that also contain predictions in the new labels will be removed. If not provided (the default), all predicted instances will be removed.
Notes
If providing new_labels, it must have been loaded using sleap.Labels.load_file(…, match_to=labels) to ensure that conflicting frames can be detected.
Labeled frames without any instances after clearing will also be removed from the dataset.
-
remove_user_instances
(new_labels: Optional[Labels] = None)[source]¶ Clear user instances from the labels.
Useful prior to merging operations to prevent overlapping instances from new labels.
- Parameters
new_labels – If not None, only user instances in frames that also contain user instances in the new labels will be removed. If not provided (the default), all user instances will be removed.
Notes
If providing new_labels, it must have been loaded using sleap.Labels.load_file(…, match_to=labels) to ensure that conflicting frames can be detected.
Labeled frames without any instances after clearing will also be removed from the dataset.
-
remove_video
(video: sleap.io.video.Video)[source]¶ Remove a video from the labels and all associated labeled frames.
- Parameters
video – Video instance to be removed.
-
save
(filename: str, with_images: bool = False, embed_all_labeled: bool = False, embed_suggested: bool = False)[source]¶ Save the labels to a file.
- Parameters
filename – Path to save the labels to ending in .slp. If the filename does not end in .slp, the extension will be automatically appended.
with_images – If True, the image data for frames with labels will be embedded in the saved labels. This is useful for generating a single file to be used when training remotely. Defaults to False.
embed_all_labeled – If True, save image data for labeled frames without user-labeled instances (defaults to False). This is useful for selecting arbitrary frames to save by adding empty `LabeledFrame`s to the dataset. Labeled frame metadata will be saved regardless.
embed_suggested – If True, save image data for frames in the suggestions (defaults to False). Useful for predicting on remaining suggestions after training. Suggestions metadata will be saved regardless.
Notes
This is an instance-level wrapper for the Labels.save_file class method.
-
classmethod
save_file
(labels: sleap.io.dataset.Labels, filename: str, default_suffix: str = '', *args, **kwargs)[source]¶ Save file, detecting format from filename.
- Parameters
labels – The dataset to save.
filename – Path where we’ll save it. We attempt to detect format from the suffix (e.g., “.json”).
default_suffix – If we can’t detect valid suffix on filename, we can add default suffix to filename (and use corresponding format). Doesn’t need to have “.” before file extension.
- Raises
ValueError – If cannot detect valid filetype.
-
save_frame_data_hdf5
(output_path: str, format: str = 'png', user_labeled: bool = True, all_labeled: bool = False, suggested: bool = False) → List[sleap.io.video.HDF5Video][source]¶ Write images for labeled frames from all videos to hdf5 file.
Note that this will make an HDF5 video, not an HDF5 labels dataset.
- Parameters
output_path – Path to HDF5 file.
format – The image format to use for the data. Defaults to png.
user_labeled – Include labeled frames with user instances. Defaults to True.
all_labeled – Include all labeled frames, including those with user-labeled instances, predicted instances or labeled frames with no instances. Defaults to False.
suggested – Include suggested frames even if they do not have instances. Useful for inference after training. Defaults to False.
- Returns
A list of
HDF5Video
objects with the stored frames.
-
save_frame_data_imgstore
(output_dir: str = './', format: str = 'png', all_labels: bool = False) → List[sleap.io.video.ImgStoreVideo][source]¶ Write images for labeled frames from all videos to imgstore datasets.
This only writes frames that have been labeled. Videos without any labeled frames will be included as empty imgstores.
- Parameters
output_dir – Path to directory which will contain imgstores.
format – The image format to use for the data. Use “png” for lossless, “jpg” for lossy. Other imgstore formats will probably work as well but have not been tested.
all_labels – Include any labeled frames, not just the frames we’ll use for training (i.e., those with Instance objects ).
- Returns
A list of
ImgStoreVideo
objects with the stored frames.
-
set_suggestions
(suggestions: List[sleap.gui.suggestions.SuggestionFrame])[source]¶ Set the suggested frames.
-
property
skeleton
¶ Return the skeleton if there is only a single skeleton in the labels.
-
to_dict
(skip_labels: bool = False) → Dict[str, Any][source]¶ Serialize all labels to dicts.
Serializes the labels in the underling list of LabeledFrames to a dict structure. This function returns a nested dict structure composed entirely of primitive python types. It is used to create JSON and HDF5 serialized datasets.
- Parameters
skip_labels – If True, skip labels serialization and just do the metadata.
- Returns
version - The version of the dict/json serialization format.
skeletons - The skeletons associated with these underlying instances.
nodes - The nodes that the skeletons represent.
videos - The videos that that the instances occur on.
labels - The labeled frames
tracks - The tracks associated with each instance.
suggestions - The suggested frames.
negative_anchors - The negative training sample anchors.
- Return type
A dict containing the followings top level keys
-
to_json
()[source]¶ Serialize all labels in the underling list of LabeledFrame(s) to JSON.
- Returns
The JSON representation of the string.
-
track_set_instance
(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance, new_track: sleap.instance.Track)[source]¶ Set track on given instance, updating occupancy.
-
track_swap
(video: sleap.io.video.Video, new_track: sleap.instance.Track, old_track: Optional[sleap.instance.Track], frame_range: tuple)[source]¶ Swap track assignment for instances in two tracks.
If you need to change the track to or from None, you’ll need to use
track_set_instance()
for each specific instance you want to modify.- Parameters
video – The
Video
for which we want to swap tracks.new_track – A
Track
for which we want to swap instances with another track.old_track – The other
Track
for swapping.frame_range – Tuple of (start, end) frame indexes. If you want to swap tracks on a single frame, use (frame index, frame index + 1).
-
property
user_instances
¶ Return list of all user (non-predicted) instances.
-
property
user_labeled_frames
¶ Return all labeled frames with user (non-predicted) instances.
-
property
video
¶ Return the video if there is only a single video in the labels.
-
-
class
sleap.io.dataset.
LabelsDataCache
(labels: Labels)[source]¶ Class for maintaining cache of data in labels dataset.
-
add_instance
(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)[source]¶ Add an instance to the labels.
-
add_track
(video: sleap.io.video.Video, track: sleap.instance.Track)[source]¶ Add track a track to the labels.
-
find_fancy_frame_idxs
(video, from_frame_idx, reverse)[source]¶ Return a list of frame idxs, with optional start position/order.
-
find_frames
(video: sleap.io.video.Video, frame_idx: Optional[Union[int, Iterable[int]]] = None) → Optional[List[sleap.instance.LabeledFrame]][source]¶ Return list of LabeledFrames matching video/frame_idx, or None.
-
get_filtered_frame_idxs
(video: Optional[sleap.io.video.Video] = None, filter: str = '') → Set[Tuple[int, int]][source]¶ Return list of (video_idx, frame_idx) tuples matching video/filter.
-
get_frame_count
(video: Optional[sleap.io.video.Video] = None, filter: str = '') → int[source]¶ Return (possibly cached) count of frames matching video/filter.
-
get_track_occupancy
(video: sleap.io.video.Video, track: sleap.instance.Track) → sleap.rangelist.RangeList[source]¶ Access track occupancy cache that adds video/track as needed.
-
get_video_track_occupancy
(video: sleap.io.video.Video) → Dict[sleap.instance.Track, sleap.rangelist.RangeList][source]¶ Return track occupancy information for specified video.
-
remove_frame
(frame: sleap.instance.LabeledFrame)[source]¶ Remvoe frame and update cache as needed.
-
remove_instance
(frame: sleap.instance.LabeledFrame, instance: sleap.instance.Instance)[source]¶ Remove an instance and update the cache as needed.
-
remove_video
(video: sleap.io.video.Video)[source]¶ Remove video and update cache as needed.
-
track_swap
(video: sleap.io.video.Video, new_track: sleap.instance.Track, old_track: Optional[sleap.instance.Track], frame_range: tuple)[source]¶ Swap tracks and update cache as needed.
-
update
(new_frame: Optional[sleap.instance.LabeledFrame] = None)[source]¶ Build (or rebuilds) various caches.
-
update_counts_for_frame
(frame: sleap.instance.LabeledFrame)[source]¶ Updated the cached count. Should be called after frame is modified.
-
-
sleap.io.dataset.
find_path_using_paths
(missing_path: str, search_paths: List[str]) → str[source]¶ Find a path to a missing file given a set of paths to search in.
- Parameters
missing_path – Path to the missing filename.
search_paths – List of paths to search in.
- Returns
The corrected path if it was found, or the original missing path if it was not.
-
sleap.io.dataset.
load_file
(filename: str, detect_videos: bool = True, search_paths: Optional[Union[List[str], str]] = None) → sleap.io.dataset.Labels[source]¶ Load a SLEAP labels file.
- Parameters
filename – Path to a SLEAP labels (.slp) file.
detect_videos – If True, will attempt to detect missing videos by searching for their filenames in the search paths. This is useful when loading SLEAP labels files that were generated on another computer with different paths.
search_paths – A path or list of paths to search for the missing videos. This can be the direct path to the video file or its containing folder. If not specified, defaults to searching for the videos in the same folder as the labels.
- Returns
The loaded Labels instance.
Notes
This is a convenience method to call sleap.Labels.load_file. See that class method for more functionality in the loading process.
The video files do not need to be accessible in order to load the labels, for example, when only the predicted instances or user labels are required.