Data structures#
In this notebook, we will explore some of the major data structures used in SLEAP and how they can be manipulated when generating predictions from trained models.
A quick overview of the data structures before we start:
Point/PredictedPoint→ Contains thexandycoordinates (andscorefor predictions) of a landmark.Instance/PredictedInstance→ Contains a set ofPoint/PredictedPoints. This represent a single individual within a frame and may also contain an associatedTrack.Skeleton→ Defines the nodes and edges that define the set of unique landmark types that each point represents, e.g., “head”, “tail”, etc. This does not contain positions – those are stored in individualPoints.LabeledFrame→ Contains a set ofInstance/PredictedInstances for a single frame.Labels→ Contains a set ofLabeledFrames and the associated metadata for the videos and other information related to the project or predictions.
1. Setup SLEAP and data#
We’ll start by installing SLEAP and downloading some data and models to play around with.
If you get a dependency error in subsequent cells, just click Runtime → Restart runtime to reload the packages.
# This should take care of all the dependencies on colab:
!pip uninstall -y opencv-python opencv-contrib-python && pip install sleap
# But to do it locally, we'd recommend the conda package (available on Windows + Linux):
# conda create -n sleap -c sleap -c conda-forge -c nvidia sleap
Found existing installation: opencv-python 4.1.2.30
Uninstalling opencv-python-4.1.2.30:
Successfully uninstalled opencv-python-4.1.2.30
Found existing installation: opencv-contrib-python 4.1.2.30
Uninstalling opencv-contrib-python-4.1.2.30:
Successfully uninstalled opencv-contrib-python-4.1.2.30
Collecting sleap
Downloading sleap-1.2.2-py3-none-any.whl (62.0 MB)
|████████████████████████████████| 62.0 MB 1.1 MB/s
?25hCollecting python-rapidjson
Downloading python_rapidjson-1.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
|████████████████████████████████| 1.6 MB 28.0 MB/s
?25hCollecting opencv-python-headless<=4.5.5.62,>=4.2.0.34
Downloading opencv_python_headless-4.5.5.62-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.7 MB)
|████████████████████████████████| 47.7 MB 82 kB/s
?25hRequirement already satisfied: h5py<=3.6.0,>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from sleap) (3.1.0)
Collecting pykalman==0.9.5
Downloading pykalman-0.9.5.tar.gz (228 kB)
|████████████████████████████████| 228 kB 61.2 MB/s
?25hRequirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from sleap) (0.11.2)
Collecting attrs==21.2.0
Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
|████████████████████████████████| 53 kB 2.3 MB/s
?25hCollecting imgstore==0.2.9
Downloading imgstore-0.2.9-py2.py3-none-any.whl (904 kB)
|████████████████████████████████| 904 kB 47.6 MB/s
?25hRequirement already satisfied: pyzmq in /usr/local/lib/python3.7/dist-packages (from sleap) (22.3.0)
Collecting qimage2ndarray<=1.8.3,>=1.8.2
Downloading qimage2ndarray-1.8.3-py3-none-any.whl (11 kB)
Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from sleap) (2.6.3)
Collecting scikit-video
Downloading scikit_video-1.1.11-py2.py3-none-any.whl (2.3 MB)
|████████████████████████████████| 2.3 MB 51.0 MB/s
?25hRequirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from sleap) (0.18.3)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from sleap) (3.13)
Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from sleap) (5.4.8)
Requirement already satisfied: numpy<=1.21.5,>=1.19.5 in /usr/local/lib/python3.7/dist-packages (from sleap) (1.21.5)
Requirement already satisfied: scipy<=1.7.3,>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from sleap) (1.4.1)
Collecting rich==10.16.1
Downloading rich-10.16.1-py3-none-any.whl (214 kB)
|████████████████████████████████| 214 kB 63.7 MB/s
?25hCollecting segmentation-models==1.0.1
Downloading segmentation_models-1.0.1-py3-none-any.whl (33 kB)
Collecting cattrs==1.1.1
Downloading cattrs-1.1.1-py3-none-any.whl (16 kB)
Requirement already satisfied: scikit-learn==1.0.* in /usr/local/lib/python3.7/dist-packages (from sleap) (1.0.2)
Requirement already satisfied: imageio<=2.15.0 in /usr/local/lib/python3.7/dist-packages (from sleap) (2.4.1)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from sleap) (1.3.5)
Requirement already satisfied: certifi<=2021.10.8,>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from sleap) (2021.10.8)
Collecting jsonpickle==1.2
Downloading jsonpickle-1.2-py2.py3-none-any.whl (32 kB)
Collecting PySide2<=5.14.1,>=5.13.2
Downloading PySide2-5.14.1-5.14.1-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (165.5 MB)
|████████████████████████████████| 165.5 MB 79 kB/s
?25hCollecting imgaug==0.4.0
Downloading imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
|████████████████████████████████| 948 kB 54.8 MB/s
?25hCollecting jsmin
Downloading jsmin-3.0.1.tar.gz (13 kB)
Requirement already satisfied: tensorflow<2.9.0,>=2.6.3 in /usr/local/lib/python3.7/dist-packages (from sleap) (2.8.0)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (1.15.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (3.2.2)
Collecting opencv-python
Downloading opencv_python-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.5 MB)
|████████████████████████████████| 60.5 MB 1.3 MB/s
?25hRequirement already satisfied: Shapely in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (1.8.1.post1)
Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (7.1.2)
Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (2018.9)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (2.8.2)
Requirement already satisfied: tzlocal in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (1.5.1)
Collecting commonmark<0.10.0,>=0.9.0
Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
|████████████████████████████████| 51 kB 8.0 MB/s
?25hRequirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.7/dist-packages (from rich==10.16.1->sleap) (2.6.1)
Collecting colorama<0.5.0,>=0.4.0
Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: typing-extensions<5.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from rich==10.16.1->sleap) (3.10.0.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==1.0.*->sleap) (3.1.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==1.0.*->sleap) (1.1.0)
Collecting image-classifiers==1.0.0
Downloading image_classifiers-1.0.0-py3-none-any.whl (19 kB)
Collecting keras-applications<=1.0.8,>=1.0.7
Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
|████████████████████████████████| 50 kB 6.9 MB/s
?25hCollecting efficientnet==1.0.0
Downloading efficientnet-1.0.0-py3-none-any.whl (17 kB)
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py<=3.6.0,>=3.1.0->sleap) (1.5.2)
Collecting shiboken2==5.14.1
Downloading shiboken2-5.14.1-5.14.1-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (847 kB)
|████████████████████████████████| 847 kB 52.6 MB/s
?25hRequirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image->sleap) (2021.11.2)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image->sleap) (1.3.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (1.4.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (0.11.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (3.0.7)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.6.3)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.44.0)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.24.0)
Requirement already satisfied: keras<2.9,>=2.8.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.8.0)
Requirement already satisfied: absl-py>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.0.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (57.4.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (3.3.0)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (3.17.3)
Requirement already satisfied: flatbuffers>=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.0)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.2.0)
Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.14.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.1.0)
Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (13.0.0)
Collecting tf-estimator-nightly==2.8.0.dev2021122109
Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
|████████████████████████████████| 462 kB 57.3 MB/s
?25hRequirement already satisfied: tensorboard<2.9,>=2.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.8.0)
Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.1.2)
Requirement already satisfied: gast>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.5.3)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow<2.9.0,>=2.6.3->sleap) (0.37.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.35.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.6.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.0.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.8.1)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (2.23.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.3.6)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.8)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.2.4)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.11.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.7.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.4.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (2.10)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.2.0)
Building wheels for collected packages: pykalman, jsmin
Building wheel for pykalman (setup.py) ... ?25l?25hdone
Created wheel for pykalman: filename=pykalman-0.9.5-py3-none-any.whl size=48462 sha256=a06494160ef192a795ebcc248474d9c759e93594f237a46d572d71045302de71
Stored in directory: /root/.cache/pip/wheels/6a/04/02/2dda6ea59c66d9e685affc8af3a31ad3a5d87b7311689efce6
Building wheel for jsmin (setup.py) ... ?25l?25hdone
Created wheel for jsmin: filename=jsmin-3.0.1-py3-none-any.whl size=13782 sha256=11175f12c4cdb3583f65125aa1f875e232ab437f5d9bdf1a6a73fbdb3d9ba69a
Stored in directory: /root/.cache/pip/wheels/a4/0b/64/fb4f87526ecbdf7921769a39d91dcfe4860e621cf15b8250d6
Successfully built pykalman jsmin
Installing collected packages: keras-applications, tf-estimator-nightly, shiboken2, opencv-python, image-classifiers, efficientnet, commonmark, colorama, attrs, segmentation-models, scikit-video, rich, qimage2ndarray, python-rapidjson, PySide2, pykalman, opencv-python-headless, jsonpickle, jsmin, imgstore, imgaug, cattrs, sleap
Attempting uninstall: attrs
Found existing installation: attrs 21.4.0
Uninstalling attrs-21.4.0:
Successfully uninstalled attrs-21.4.0
Attempting uninstall: imgaug
Found existing installation: imgaug 0.2.9
Uninstalling imgaug-0.2.9:
Successfully uninstalled imgaug-0.2.9
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.4.0 which is incompatible.
Successfully installed PySide2-5.14.1 attrs-21.2.0 cattrs-1.1.1 colorama-0.4.4 commonmark-0.9.1 efficientnet-1.0.0 image-classifiers-1.0.0 imgaug-0.4.0 imgstore-0.2.9 jsmin-3.0.1 jsonpickle-1.2 keras-applications-1.0.8 opencv-python-4.5.5.64 opencv-python-headless-4.5.5.62 pykalman-0.9.5 python-rapidjson-1.6 qimage2ndarray-1.8.3 rich-10.16.1 scikit-video-1.1.11 segmentation-models-1.0.1 shiboken2-5.14.1 sleap-1.2.2 tf-estimator-nightly-2.8.0.dev2021122109
# Test video:
!wget https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.mp4
# Test video labels (from predictions/not necessary for inference benchmarking):
!wget https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.slp
# Bottom-up model:
# !wget https://storage.googleapis.com/sleap-data/reference/flies13/bu.210506_230852.multi_instance.n%3D1800.zip
# Top-down model (two-stage):
!wget https://storage.googleapis.com/sleap-data/reference/flies13/centroid.fast.210504_182918.centroid.n%3D1800.zip
!wget https://storage.googleapis.com/sleap-data/reference/flies13/td_fast.210505_012601.centered_instance.n%3D1800.zip
--2022-04-04 00:19:01-- https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.mp4
Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.97.128, 142.251.107.128, 173.194.214.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.97.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85343812 (81M) [video/mp4]
Saving to: ‘190719_090330_wt_18159206_rig1.2@15000-17560.mp4’
190719_090330_wt_18 100%[===================>] 81.39M 142MB/s in 0.6s
2022-04-04 00:19:02 (142 MB/s) - ‘190719_090330_wt_18159206_rig1.2@15000-17560.mp4’ saved [85343812/85343812]
--2022-04-04 00:19:02-- https://storage.googleapis.com/sleap-data/reference/flies13/190719_090330_wt_18159206_rig1.2%4015000-17560.slp
Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.214.128, 173.194.215.128, 173.194.216.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.214.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1581400 (1.5M) [application/octet-stream]
Saving to: ‘190719_090330_wt_18159206_rig1.2@15000-17560.slp’
190719_090330_wt_18 100%[===================>] 1.51M --.-KB/s in 0.01s
2022-04-04 00:19:02 (151 MB/s) - ‘190719_090330_wt_18159206_rig1.2@15000-17560.slp’ saved [1581400/1581400]
--2022-04-04 00:19:02-- https://storage.googleapis.com/sleap-data/reference/flies13/centroid.fast.210504_182918.centroid.n%3D1800.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.214.128, 173.194.215.128, 173.194.216.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.214.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6372537 (6.1M) [application/zip]
Saving to: ‘centroid.fast.210504_182918.centroid.n=1800.zip’
centroid.fast.21050 100%[===================>] 6.08M --.-KB/s in 0.05s
2022-04-04 00:19:02 (134 MB/s) - ‘centroid.fast.210504_182918.centroid.n=1800.zip’ saved [6372537/6372537]
--2022-04-04 00:19:02-- https://storage.googleapis.com/sleap-data/reference/flies13/td_fast.210505_012601.centered_instance.n%3D1800.zip
Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.216.128, 173.194.217.128, 173.194.218.128, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.216.128|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30775963 (29M) [application/zip]
Saving to: ‘td_fast.210505_012601.centered_instance.n=1800.zip’
td_fast.210505_0126 100%[===================>] 29.35M 190MB/s in 0.2s
2022-04-04 00:19:03 (190 MB/s) - ‘td_fast.210505_012601.centered_instance.n=1800.zip’ saved [30775963/30775963]
!ls -lah
total 119M
drwxr-xr-x 1 root root 4.0K Apr 4 00:19 .
drwxr-xr-x 1 root root 4.0K Apr 4 00:15 ..
-rw-r--r-- 1 root root 82M May 20 2021 190719_090330_wt_18159206_rig1.2@15000-17560.mp4
-rw-r--r-- 1 root root 1.6M May 20 2021 190719_090330_wt_18159206_rig1.2@15000-17560.slp
-rw-r--r-- 1 root root 6.1M May 20 2021 'centroid.fast.210504_182918.centroid.n=1800.zip'
drwxr-xr-x 4 root root 4.0K Mar 23 14:21 .config
drwxr-xr-x 1 root root 4.0K Mar 23 14:22 sample_data
-rw-r--r-- 1 root root 30M May 20 2021 'td_fast.210505_012601.centered_instance.n=1800.zip'
import sleap
# This prevents TensorFlow from allocating all the GPU memory, which leads to issues on
# some GPUs/platforms:
sleap.disable_preallocation()
# This would hide GPUs from the TensorFlow altogether:
# sleap.use_cpu_only()
# Print some info:
sleap.versions()
sleap.system_summary()
INFO:numexpr.utils:NumExpr defaulting to 2 threads.
SLEAP: 1.2.2
TensorFlow: 2.8.0
Numpy: 1.21.5
Python: 3.7.13
OS: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
2. Data structures and inference#
SLEAP can read videos in a variety of different formats through the sleap.load_video high level API. Once loaded, the sleap.Video object allows you to access individual frames as if the it were a standard numpy array.
Note: The actual frames are not loaded until you access them so we don’t blow up our memory when using long videos.
# Videos can be represented agnostic to the backend format
video = sleap.load_video("190719_090330_wt_18159206_rig1.2@15000-17560.mp4")
# sleap.Video objects have a numpy-like interface:
print(video.shape)
# And we can load images in the video using array indexing:
imgs = video[:4]
print(imgs.shape, imgs.dtype)
(2560, 1024, 1024, 1)
(4, 1024, 1024, 1) uint8
The high level interface for loading models (sleap.load_model()) takes model folders or zipped folders as input. These are outputs from our training procedure and need to contain a "best_model.h5" and "training_config.json".
best_model.h5 is an HDF5-serialized tf.keras.Model that was checkpointed during
training. It includes the architecture as well as the weights, so they’re standalone
and don’t need SLEAP – BUT they do not contain the inference methods.
training_config.json is a serialized sleap.TrainingJobConfig that contains metadata
like what channels of the model correspond to what landmarks and etc.
Top-down models have two stages: centroid and centered instance confidence maps, which we train and save out separately, so loading them together links them up into a single inference model.
# Top-down
predictor = sleap.load_model([
"centroid.fast.210504_182918.centroid.n=1800.zip",
"td_fast.210505_012601.centered_instance.n=1800.zip"
])
# Bottom-up
# predictor = sleap.load_model("bu.210506_230852.multi_instance.n=1800.zip")
The high level predictor creates all the SLEAP data structures after doing inference. For example:
labels = predictor.predict(video)
labels
Labels(labeled_frames=2560, videos=1, skeletons=1, tracks=0)
Labels contain not just the predicted data, but all the other associated data structures and metadata:
labels.videos
[Video(backend=MediaVideo(filename='190719_090330_wt_18159206_rig1.2@15000-17560.mp4', grayscale=True, bgr=True, dataset='', input_format='channels_last'))]
labels.skeletons
[Skeleton(name='Skeleton-0', nodes=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], edges=[('thorax', 'head'), ('thorax', 'abdomen'), ('thorax', 'wingL'), ('thorax', 'wingR'), ('thorax', 'forelegL4'), ('thorax', 'forelegR4'), ('thorax', 'midlegL4'), ('thorax', 'midlegR4'), ('thorax', 'hindlegL4'), ('thorax', 'hindlegR4'), ('head', 'eyeL'), ('head', 'eyeR')], symmetries=[('wingL', 'wingR'), ('forelegL4', 'forelegR4'), ('hindlegL4', 'hindlegR4'), ('eyeL', 'eyeR'), ('midlegL4', 'midlegR4')])]
Individual labeled frames are accessible through a list-like interface:
labeled_frame = labels[0] # shortcut for labels.labeled_frames[0]
labeled_frame
LabeledFrame(video=MediaVideo('190719_090330_wt_18159206_rig1.2@15000-17560.mp4'), frame_idx=0, instances=2)
Convenient methods allow for easy inspection:
labels[0].plot(scale=0.5)
The labeled frame is itself a container for instances:
labeled_frame.instances
[PredictedInstance(video=Video(filename=190719_090330_wt_18159206_rig1.2@15000-17560.mp4, shape=(2560, 1024, 1024, 1), backend=MediaVideo), frame_idx=0, points=[head: (234.2, 430.5, 0.98), thorax: (271.6, 436.1, 0.94), abdomen: (308.0, 438.6, 0.59), wingL: (321.8, 440.1, 0.39), wingR: (322.0, 436.8, 0.49), forelegL4: (246.1, 450.6, 0.92), forelegR4: (242.3, 413.9, 0.78), midlegL4: (285.8, 459.9, 0.47), midlegR4: (272.3, 406.7, 0.77), hindlegR4: (317.6, 430.6, 0.30), eyeL: (242.1, 441.9, 0.89), eyeR: (245.3, 420.9, 0.92)], score=0.95, track=None, tracking_score=0.00),
PredictedInstance(video=Video(filename=190719_090330_wt_18159206_rig1.2@15000-17560.mp4, shape=(2560, 1024, 1024, 1), backend=MediaVideo), frame_idx=0, points=[head: (319.4, 435.9, 0.83), thorax: (354.4, 435.2, 0.80), abdomen: (368.3, 433.8, 0.71), wingL: (393.9, 480.3, 0.83), wingR: (398.4, 430.0, 0.81), forelegL4: (307.8, 445.7, 0.26), forelegR4: (305.6, 421.4, 0.69), midlegL4: (325.7, 475.0, 0.94), midlegR4: (331.8, 385.1, 0.88), hindlegL4: (363.7, 474.1, 0.88), hindlegR4: (376.0, 398.4, 0.52), eyeL: (329.3, 445.6, 0.90), eyeR: (327.9, 425.1, 0.84)], score=0.84, track=None, tracking_score=0.00)]
instance = labeled_frame[0] # shortcut for labeled_frame.instances[0]
instance
PredictedInstance(video=Video(filename=190719_090330_wt_18159206_rig1.2@15000-17560.mp4, shape=(2560, 1024, 1024, 1), backend=MediaVideo), frame_idx=0, points=[head: (234.2, 430.5, 0.98), thorax: (271.6, 436.1, 0.94), abdomen: (308.0, 438.6, 0.59), wingL: (321.8, 440.1, 0.39), wingR: (322.0, 436.8, 0.49), forelegL4: (246.1, 450.6, 0.92), forelegR4: (242.3, 413.9, 0.78), midlegL4: (285.8, 459.9, 0.47), midlegR4: (272.3, 406.7, 0.77), hindlegR4: (317.6, 430.6, 0.30), eyeL: (242.1, 441.9, 0.89), eyeR: (245.3, 420.9, 0.92)], score=0.95, track=None, tracking_score=0.00)
Finally, instances are containers for points:
instance.points
(PredictedPoint(x=234.244384765625, y=430.52001953125, visible=True, complete=False, score=0.9790461659431458),
PredictedPoint(x=271.5894470214844, y=436.1461181640625, visible=True, complete=False, score=0.9357967376708984),
PredictedPoint(x=308.02899169921875, y=438.5711975097656, visible=True, complete=False, score=0.5859644412994385),
PredictedPoint(x=321.8167419433594, y=440.0872802734375, visible=True, complete=False, score=0.3912011682987213),
PredictedPoint(x=322.0196533203125, y=436.77008056640625, visible=True, complete=False, score=0.48613619804382324),
PredictedPoint(x=246.1430206298828, y=450.56182861328125, visible=True, complete=False, score=0.9176540970802307),
PredictedPoint(x=242.2632293701172, y=413.94976806640625, visible=True, complete=False, score=0.7807964086532593),
PredictedPoint(x=285.78167724609375, y=459.9156494140625, visible=True, complete=False, score=0.4739593267440796),
PredictedPoint(x=272.27996826171875, y=406.71759033203125, visible=True, complete=False, score=0.7721188068389893),
PredictedPoint(x=317.5997619628906, y=430.6052551269531, visible=True, complete=False, score=0.2960105538368225),
PredictedPoint(x=242.1038055419922, y=441.94561767578125, visible=True, complete=False, score=0.8855815529823303),
PredictedPoint(x=245.3200225830078, y=420.93609619140625, visible=True, complete=False, score=0.9199579954147339))
These can be converted into concrete arrays:
pts = instance.numpy()
print(pts)
[[234.24438477 430.52001953]
[271.58944702 436.14611816]
[308.0289917 438.57119751]
[321.81674194 440.08728027]
[322.01965332 436.77008057]
[246.14302063 450.56182861]
[242.26322937 413.94976807]
[285.78167725 459.91564941]
[272.27996826 406.71759033]
[ nan nan]
[317.59976196 430.60525513]
[242.10380554 441.94561768]
[245.32002258 420.93609619]]
Images can be embedded together with the predictions in the same format:
labels = sleap.Labels(labels.labeled_frames[:4]) # crop to the first few labels for this example
labels.save("labels_with_images.pkg.slp", with_images=True, embed_all_labeled=True)
Let’s delete the source data:
!rm "190719_090330_wt_18159206_rig1.2@15000-17560.mp4"
And check out what happens when we load in some labels with embedded images:
labels = sleap.load_file("labels_with_images.pkg.slp")
labels
Labels(labeled_frames=4, videos=1, skeletons=1, tracks=0)
labels[0].plot(scale=0.5)
