Open In Colab

Data structures#

In this notebook, we will explore some of the major data structures used in SLEAP and how they can be manipulated when generating predictions from trained models.

A quick overview of the data structures before we start:

  • Point/PredictedPoint → Contains the x and y coordinates (and score for predictions) of a landmark.

  • Instance/PredictedInstance → Contains a set of Point/PredictedPoints. This represent a single individual within a frame and may also contain an associated Track.

  • Skeleton → Defines the nodes and edges that define the set of unique landmark types that each point represents, e.g., “head”, “tail”, etc. This does not contain positions – those are stored in individual Points.

  • LabeledFrame → Contains a set of Instance/PredictedInstances for a single frame.

  • Labels → Contains a set of LabeledFrames and the associated metadata for the videos and other information related to the project or predictions.

1. Setup SLEAP and data#

We’ll start by installing SLEAP and downloading some data and models to play around with.

If you get a dependency error in subsequent cells, just click RuntimeRestart runtime to reload the packages.

# This should take care of all the dependencies on colab:
!pip uninstall -y opencv-python opencv-contrib-python && pip install sleap

# But to do it locally, we'd recommend the conda package (available on Windows + Linux):
# conda create -n sleap -c sleap -c conda-forge -c nvidia sleap
Found existing installation: opencv-python 4.1.2.30
Uninstalling opencv-python-4.1.2.30:
  Successfully uninstalled opencv-python-4.1.2.30
Found existing installation: opencv-contrib-python 4.1.2.30
Uninstalling opencv-contrib-python-4.1.2.30:
  Successfully uninstalled opencv-contrib-python-4.1.2.30
Collecting sleap
  Downloading sleap-1.2.2-py3-none-any.whl (62.0 MB)
     |████████████████████████████████| 62.0 MB 1.1 MB/s 
?25hCollecting python-rapidjson
  Downloading python_rapidjson-1.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 28.0 MB/s 
?25hCollecting opencv-python-headless<=4.5.5.62,>=4.2.0.34
  Downloading opencv_python_headless-4.5.5.62-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.7 MB)
     |████████████████████████████████| 47.7 MB 82 kB/s 
?25hRequirement already satisfied: h5py<=3.6.0,>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from sleap) (3.1.0)
Collecting pykalman==0.9.5
  Downloading pykalman-0.9.5.tar.gz (228 kB)
     |████████████████████████████████| 228 kB 61.2 MB/s 
?25hRequirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from sleap) (0.11.2)
Collecting attrs==21.2.0
  Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 2.3 MB/s 
?25hCollecting imgstore==0.2.9
  Downloading imgstore-0.2.9-py2.py3-none-any.whl (904 kB)
     |████████████████████████████████| 904 kB 47.6 MB/s 
?25hRequirement already satisfied: pyzmq in /usr/local/lib/python3.7/dist-packages (from sleap) (22.3.0)
Collecting qimage2ndarray<=1.8.3,>=1.8.2
  Downloading qimage2ndarray-1.8.3-py3-none-any.whl (11 kB)
Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from sleap) (2.6.3)
Collecting scikit-video
  Downloading scikit_video-1.1.11-py2.py3-none-any.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 51.0 MB/s 
?25hRequirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from sleap) (0.18.3)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from sleap) (3.13)
Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from sleap) (5.4.8)
Requirement already satisfied: numpy<=1.21.5,>=1.19.5 in /usr/local/lib/python3.7/dist-packages (from sleap) (1.21.5)
Requirement already satisfied: scipy<=1.7.3,>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from sleap) (1.4.1)
Collecting rich==10.16.1
  Downloading rich-10.16.1-py3-none-any.whl (214 kB)
     |████████████████████████████████| 214 kB 63.7 MB/s 
?25hCollecting segmentation-models==1.0.1
  Downloading segmentation_models-1.0.1-py3-none-any.whl (33 kB)
Collecting cattrs==1.1.1
  Downloading cattrs-1.1.1-py3-none-any.whl (16 kB)
Requirement already satisfied: scikit-learn==1.0.* in /usr/local/lib/python3.7/dist-packages (from sleap) (1.0.2)
Requirement already satisfied: imageio<=2.15.0 in /usr/local/lib/python3.7/dist-packages (from sleap) (2.4.1)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from sleap) (1.3.5)
Requirement already satisfied: certifi<=2021.10.8,>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from sleap) (2021.10.8)
Collecting jsonpickle==1.2
  Downloading jsonpickle-1.2-py2.py3-none-any.whl (32 kB)
Collecting PySide2<=5.14.1,>=5.13.2
  Downloading PySide2-5.14.1-5.14.1-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (165.5 MB)
     |████████████████████████████████| 165.5 MB 79 kB/s 
?25hCollecting imgaug==0.4.0
  Downloading imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
     |████████████████████████████████| 948 kB 54.8 MB/s 
?25hCollecting jsmin
  Downloading jsmin-3.0.1.tar.gz (13 kB)
Requirement already satisfied: tensorflow<2.9.0,>=2.6.3 in /usr/local/lib/python3.7/dist-packages (from sleap) (2.8.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (1.15.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (3.2.2)
Collecting opencv-python
  Downloading opencv_python-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.5 MB)
     |████████████████████████████████| 60.5 MB 1.3 MB/s 
?25hRequirement already satisfied: Shapely in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (1.8.1.post1)
Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (7.1.2)
Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (2018.9)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (2.8.2)
Requirement already satisfied: tzlocal in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (1.5.1)
Collecting commonmark<0.10.0,>=0.9.0
  Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
     |████████████████████████████████| 51 kB 8.0 MB/s 
?25hRequirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.7/dist-packages (from rich==10.16.1->sleap) (2.6.1)
Collecting colorama<0.5.0,>=0.4.0
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: typing-extensions<5.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from rich==10.16.1->sleap) (3.10.0.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==1.0.*->sleap) (3.1.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==1.0.*->sleap) (1.1.0)
Collecting image-classifiers==1.0.0
  Downloading image_classifiers-1.0.0-py3-none-any.whl (19 kB)
Collecting keras-applications<=1.0.8,>=1.0.7
  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
     |████████████████████████████████| 50 kB 6.9 MB/s 
?25hCollecting efficientnet==1.0.0
  Downloading efficientnet-1.0.0-py3-none-any.whl (17 kB)
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py<=3.6.0,>=3.1.0->sleap) (1.5.2)
Collecting shiboken2==5.14.1
  Downloading shiboken2-5.14.1-5.14.1-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (847 kB)
     |████████████████████████████████| 847 kB 52.6 MB/s 
?25hRequirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image->sleap) (2021.11.2)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image->sleap) (1.3.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (1.4.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (0.11.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (3.0.7)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.6.3)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.44.0)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.24.0)
Requirement already satisfied: keras<2.9,>=2.8.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.8.0)
Requirement already satisfied: absl-py>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.0.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (57.4.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (3.3.0)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (3.17.3)
Requirement already satisfied: flatbuffers>=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.0)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.2.0)
Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.14.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.1.0)
Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (13.0.0)
Collecting tf-estimator-nightly==2.8.0.dev2021122109
  Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
     |████████████████████████████████| 462 kB 57.3 MB/s 
?25hRequirement already satisfied: tensorboard<2.9,>=2.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.8.0)
Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.1.2)
Requirement already satisfied: gast>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.5.3)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow<2.9.0,>=2.6.3->sleap) (0.37.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.35.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.6.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.0.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.8.1)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (2.23.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.3.6)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.8)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.2.4)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.11.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.7.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.4.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (2.10)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.2.0)
Building wheels for collected packages: pykalman, jsmin
  Building wheel for pykalman (setup.py) ... ?25l?25hdone
  Created wheel for pykalman: filename=pykalman-0.9.5-py3-none-any.whl size=48462 sha256=a06494160ef192a795ebcc248474d9c759e93594f237a46d572d71045302de71
  Stored in directory: /root/.cache/pip/wheels/6a/04/02/2dda6ea59c66d9e685affc8af3a31ad3a5d87b7311689efce6
  Building wheel for jsmin (setup.py) ... ?25l?25hdone
  Created wheel for jsmin: filename=jsmin-3.0.1-py3-none-any.whl size=13782 sha256=11175f12c4cdb3583f65125aa1f875e232ab437f5d9bdf1a6a73fbdb3d9ba69a
  Stored in directory: /root/.cache/pip/wheels/a4/0b/64/fb4f87526ecbdf7921769a39d91dcfe4860e621cf15b8250d6
Successfully built pykalman jsmin
Installing collected packages: keras-applications, tf-estimator-nightly, shiboken2, opencv-python, image-classifiers, efficientnet, commonmark, colorama, attrs, segmentation-models, scikit-video, rich, qimage2ndarray, python-rapidjson, PySide2, pykalman, opencv-python-headless, jsonpickle, jsmin, imgstore, imgaug, cattrs, sleap
  Attempting uninstall: attrs
    Found existing installation: attrs 21.4.0
    Uninstalling attrs-21.4.0:
      Successfully uninstalled attrs-21.4.0
  Attempting uninstall: imgaug
    Found existing installation: imgaug 0.2.9
    Uninstalling imgaug-0.2.9:
      Successfully uninstalled imgaug-0.2.9
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.4.0 which is incompatible.
Successfully installed PySide2-5.14.1 attrs-21.2.0 cattrs-1.1.1 colorama-0.4.4 commonmark-0.9.1 efficientnet-1.0.0 image-classifiers-1.0.0 imgaug-0.4.0 imgstore-0.2.9 jsmin-3.0.1 jsonpickle-1.2 keras-applications-1.0.8 opencv-python-4.5.5.64 opencv-python-headless-4.5.5.62 pykalman-0.9.5 python-rapidjson-1.6 qimage2ndarray-1.8.3 rich-10.16.1 scikit-video-1.1.11 segmentation-models-1.0.1 shiboken2-5.14.1 sleap-1.2.2 tf-estimator-nightly-2.8.0.dev2021122109
# Test video:
!wget https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.mp4

# Test video labels (from predictions/not necessary for inference benchmarking):
!wget https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.slp

# Bottom-up model:
# !wget https://storage.googleapis.com/sleap-data/reference/flies13/bu.210506_230852.multi_instance.n%3D1800.zip

# Top-down model (two-stage):
!wget https://storage.googleapis.com/sleap-data/reference/flies13/centroid.fast.210504_182918.centroid.n%3D1800.zip
!wget https://storage.googleapis.com/sleap-data/reference/flies13/td_fast.210505_012601.centered_instance.n%3D1800.zip
--2022-04-04 00:19:01--  https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.mp4
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.97.128, 142.251.107.128, 173.194.214.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.97.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85343812 (81M) [video/mp4]
Saving to: ‘190719_090330_wt_18159206_rig1.2@15000-17560.mp4’

190719_090330_wt_18 100%[===================>]  81.39M   142MB/s    in 0.6s    

2022-04-04 00:19:02 (142 MB/s) - ‘190719_090330_wt_18159206_rig1.2@15000-17560.mp4’ saved [85343812/85343812]

--2022-04-04 00:19:02--  https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.slp
Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.214.128, 173.194.215.128, 173.194.216.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.214.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1581400 (1.5M) [application/octet-stream]
Saving to: ‘190719_090330_wt_18159206_rig1.2@15000-17560.slp’

190719_090330_wt_18 100%[===================>]   1.51M  --.-KB/s    in 0.01s   

2022-04-04 00:19:02 (151 MB/s) - ‘190719_090330_wt_18159206_rig1.2@15000-17560.slp’ saved [1581400/1581400]

--2022-04-04 00:19:02--  https://storage.googleapis.com/sleap-data/reference/flies13/centroid.fast.210504_182918.centroid.n%3D1800.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.214.128, 173.194.215.128, 173.194.216.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.214.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6372537 (6.1M) [application/zip]
Saving to: ‘centroid.fast.210504_182918.centroid.n=1800.zip’

centroid.fast.21050 100%[===================>]   6.08M  --.-KB/s    in 0.05s   

2022-04-04 00:19:02 (134 MB/s) - ‘centroid.fast.210504_182918.centroid.n=1800.zip’ saved [6372537/6372537]

--2022-04-04 00:19:02--  https://storage.googleapis.com/sleap-data/reference/flies13/td_fast.210505_012601.centered_instance.n%3D1800.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.216.128, 173.194.217.128, 173.194.218.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.216.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30775963 (29M) [application/zip]
Saving to: ‘td_fast.210505_012601.centered_instance.n=1800.zip’

td_fast.210505_0126 100%[===================>]  29.35M   190MB/s    in 0.2s    

2022-04-04 00:19:03 (190 MB/s) - ‘td_fast.210505_012601.centered_instance.n=1800.zip’ saved [30775963/30775963]
!ls -lah
total 119M
drwxr-xr-x 1 root root 4.0K Apr  4 00:19  .
drwxr-xr-x 1 root root 4.0K Apr  4 00:15  ..
-rw-r--r-- 1 root root  82M May 20  2021  190719_090330_wt_18159206_rig1.2@15000-17560.mp4
-rw-r--r-- 1 root root 1.6M May 20  2021  190719_090330_wt_18159206_rig1.2@15000-17560.slp
-rw-r--r-- 1 root root 6.1M May 20  2021 'centroid.fast.210504_182918.centroid.n=1800.zip'
drwxr-xr-x 4 root root 4.0K Mar 23 14:21  .config
drwxr-xr-x 1 root root 4.0K Mar 23 14:22  sample_data
-rw-r--r-- 1 root root  30M May 20  2021 'td_fast.210505_012601.centered_instance.n=1800.zip'
import sleap

# This prevents TensorFlow from allocating all the GPU memory, which leads to issues on
# some GPUs/platforms:
sleap.disable_preallocation()

# This would hide GPUs from the TensorFlow altogether:
# sleap.use_cpu_only()

# Print some info:
sleap.versions()
sleap.system_summary()
INFO:numexpr.utils:NumExpr defaulting to 2 threads.
SLEAP: 1.2.2
TensorFlow: 2.8.0
Numpy: 1.21.5
Python: 3.7.13
OS: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True

2. Data structures and inference#

SLEAP can read videos in a variety of different formats through the sleap.load_video high level API. Once loaded, the sleap.Video object allows you to access individual frames as if the it were a standard numpy array.

Note: The actual frames are not loaded until you access them so we don’t blow up our memory when using long videos.

# Videos can be represented agnostic to the backend format
video = sleap.load_video("190719_090330_wt_18159206_rig1.2@15000-17560.mp4")

# sleap.Video objects have a numpy-like interface:
print(video.shape)

# And we can load images in the video using array indexing:
imgs = video[:4]
print(imgs.shape, imgs.dtype)
(2560, 1024, 1024, 1)
(4, 1024, 1024, 1) uint8

The high level interface for loading models (sleap.load_model()) takes model folders or zipped folders as input. These are outputs from our training procedure and need to contain a "best_model.h5" and "training_config.json".

best_model.h5 is an HDF5-serialized tf.keras.Model that was checkpointed during training. It includes the architecture as well as the weights, so they’re standalone and don’t need SLEAP – BUT they do not contain the inference methods.

training_config.json is a serialized sleap.TrainingJobConfig that contains metadata like what channels of the model correspond to what landmarks and etc.

Top-down models have two stages: centroid and centered instance confidence maps, which we train and save out separately, so loading them together links them up into a single inference model.

# Top-down
predictor = sleap.load_model([
    "centroid.fast.210504_182918.centroid.n=1800.zip",
    "td_fast.210505_012601.centered_instance.n=1800.zip"
    ])

# Bottom-up
# predictor = sleap.load_model("bu.210506_230852.multi_instance.n=1800.zip")

The high level predictor creates all the SLEAP data structures after doing inference. For example:

labels = predictor.predict(video)
labels


Labels(labeled_frames=2560, videos=1, skeletons=1, tracks=0)

Labels contain not just the predicted data, but all the other associated data structures and metadata:

labels.videos
[Video(backend=MediaVideo(filename='190719_090330_wt_18159206_rig1.2@15000-17560.mp4', grayscale=True, bgr=True, dataset='', input_format='channels_last'))]
labels.skeletons
[Skeleton(name='Skeleton-0', nodes=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], edges=[('thorax', 'head'), ('thorax', 'abdomen'), ('thorax', 'wingL'), ('thorax', 'wingR'), ('thorax', 'forelegL4'), ('thorax', 'forelegR4'), ('thorax', 'midlegL4'), ('thorax', 'midlegR4'), ('thorax', 'hindlegL4'), ('thorax', 'hindlegR4'), ('head', 'eyeL'), ('head', 'eyeR')], symmetries=[('wingL', 'wingR'), ('forelegL4', 'forelegR4'), ('hindlegL4', 'hindlegR4'), ('eyeL', 'eyeR'), ('midlegL4', 'midlegR4')])]

Individual labeled frames are accessible through a list-like interface:

labeled_frame = labels[0]  # shortcut for labels.labeled_frames[0]
labeled_frame
LabeledFrame(video=MediaVideo('190719_090330_wt_18159206_rig1.2@15000-17560.mp4'), frame_idx=0, instances=2)

Convenient methods allow for easy inspection:

labels[0].plot(scale=0.5)
../_images/Data_structures_20_0.png

The labeled frame is itself a container for instances:

labeled_frame.instances
[PredictedInstance(video=Video(filename=190719_090330_wt_18159206_rig1.2@15000-17560.mp4, shape=(2560, 1024, 1024, 1), backend=MediaVideo), frame_idx=0, points=[head: (234.2, 430.5, 0.98), thorax: (271.6, 436.1, 0.94), abdomen: (308.0, 438.6, 0.59), wingL: (321.8, 440.1, 0.39), wingR: (322.0, 436.8, 0.49), forelegL4: (246.1, 450.6, 0.92), forelegR4: (242.3, 413.9, 0.78), midlegL4: (285.8, 459.9, 0.47), midlegR4: (272.3, 406.7, 0.77), hindlegR4: (317.6, 430.6, 0.30), eyeL: (242.1, 441.9, 0.89), eyeR: (245.3, 420.9, 0.92)], score=0.95, track=None, tracking_score=0.00),
 PredictedInstance(video=Video(filename=190719_090330_wt_18159206_rig1.2@15000-17560.mp4, shape=(2560, 1024, 1024, 1), backend=MediaVideo), frame_idx=0, points=[head: (319.4, 435.9, 0.83), thorax: (354.4, 435.2, 0.80), abdomen: (368.3, 433.8, 0.71), wingL: (393.9, 480.3, 0.83), wingR: (398.4, 430.0, 0.81), forelegL4: (307.8, 445.7, 0.26), forelegR4: (305.6, 421.4, 0.69), midlegL4: (325.7, 475.0, 0.94), midlegR4: (331.8, 385.1, 0.88), hindlegL4: (363.7, 474.1, 0.88), hindlegR4: (376.0, 398.4, 0.52), eyeL: (329.3, 445.6, 0.90), eyeR: (327.9, 425.1, 0.84)], score=0.84, track=None, tracking_score=0.00)]
instance = labeled_frame[0]  # shortcut for labeled_frame.instances[0]
instance
PredictedInstance(video=Video(filename=190719_090330_wt_18159206_rig1.2@15000-17560.mp4, shape=(2560, 1024, 1024, 1), backend=MediaVideo), frame_idx=0, points=[head: (234.2, 430.5, 0.98), thorax: (271.6, 436.1, 0.94), abdomen: (308.0, 438.6, 0.59), wingL: (321.8, 440.1, 0.39), wingR: (322.0, 436.8, 0.49), forelegL4: (246.1, 450.6, 0.92), forelegR4: (242.3, 413.9, 0.78), midlegL4: (285.8, 459.9, 0.47), midlegR4: (272.3, 406.7, 0.77), hindlegR4: (317.6, 430.6, 0.30), eyeL: (242.1, 441.9, 0.89), eyeR: (245.3, 420.9, 0.92)], score=0.95, track=None, tracking_score=0.00)

Finally, instances are containers for points:

instance.points
(PredictedPoint(x=234.244384765625, y=430.52001953125, visible=True, complete=False, score=0.9790461659431458),
 PredictedPoint(x=271.5894470214844, y=436.1461181640625, visible=True, complete=False, score=0.9357967376708984),
 PredictedPoint(x=308.02899169921875, y=438.5711975097656, visible=True, complete=False, score=0.5859644412994385),
 PredictedPoint(x=321.8167419433594, y=440.0872802734375, visible=True, complete=False, score=0.3912011682987213),
 PredictedPoint(x=322.0196533203125, y=436.77008056640625, visible=True, complete=False, score=0.48613619804382324),
 PredictedPoint(x=246.1430206298828, y=450.56182861328125, visible=True, complete=False, score=0.9176540970802307),
 PredictedPoint(x=242.2632293701172, y=413.94976806640625, visible=True, complete=False, score=0.7807964086532593),
 PredictedPoint(x=285.78167724609375, y=459.9156494140625, visible=True, complete=False, score=0.4739593267440796),
 PredictedPoint(x=272.27996826171875, y=406.71759033203125, visible=True, complete=False, score=0.7721188068389893),
 PredictedPoint(x=317.5997619628906, y=430.6052551269531, visible=True, complete=False, score=0.2960105538368225),
 PredictedPoint(x=242.1038055419922, y=441.94561767578125, visible=True, complete=False, score=0.8855815529823303),
 PredictedPoint(x=245.3200225830078, y=420.93609619140625, visible=True, complete=False, score=0.9199579954147339))

These can be converted into concrete arrays:

pts = instance.numpy()
print(pts)
[[234.24438477 430.52001953]
 [271.58944702 436.14611816]
 [308.0289917  438.57119751]
 [321.81674194 440.08728027]
 [322.01965332 436.77008057]
 [246.14302063 450.56182861]
 [242.26322937 413.94976807]
 [285.78167725 459.91564941]
 [272.27996826 406.71759033]
 [         nan          nan]
 [317.59976196 430.60525513]
 [242.10380554 441.94561768]
 [245.32002258 420.93609619]]

Images can be embedded together with the predictions in the same format:

labels = sleap.Labels(labels.labeled_frames[:4])  # crop to the first few labels for this example
labels.save("labels_with_images.pkg.slp", with_images=True, embed_all_labeled=True)

Let’s delete the source data:

!rm "190719_090330_wt_18159206_rig1.2@15000-17560.mp4"

And check out what happens when we load in some labels with embedded images:

labels = sleap.load_file("labels_with_images.pkg.slp")
labels
Labels(labeled_frames=4, videos=1, skeletons=1, tracks=0)
labels[0].plot(scale=0.5)
../_images/Data_structures_34_0.png