Interactive and resumable training#
Most of the time, you will be training models through the GUI or using the sleap-train CLI.
If you’d like to customize the training process, however, you can use SLEAP’s low-level training functionality interactively. This allows you to define scripts that train models according to your own workflow, for example, to resume training on an already trained model. Another possible application would be to train a model using transfer learning, where a pretrained model can be used to initialize the weights of the new model.
In this notebook we will explore how to set up a training job and train a model for multiple rounds without the GUI or CLI.
1. Setup SLEAP#
Run this cell first to install SLEAP. If you get a dependency error in subsequent cells, just click Runtime → Restart runtime to reload the packages.
Don’t forget to set Runtime → Change runtime type → GPU as the accelerator.
# This should take care of all the dependencies on colab:
!pip uninstall -y opencv-python opencv-contrib-python && pip install sleap
# But to do it locally, we'd recommend the conda package (available on Windows + Linux):
# conda create -n sleap -c sleap -c conda-forge -c nvidia sleap
Found existing installation: opencv-python 4.1.2.30
Uninstalling opencv-python-4.1.2.30:
Successfully uninstalled opencv-python-4.1.2.30
Found existing installation: opencv-contrib-python 4.1.2.30
Uninstalling opencv-contrib-python-4.1.2.30:
Successfully uninstalled opencv-contrib-python-4.1.2.30
Collecting sleap
Downloading sleap-1.2.2-py3-none-any.whl (62.0 MB)
|████████████████████████████████| 62.0 MB 1.3 MB/s
?25hRequirement already satisfied: certifi<=2021.10.8,>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from sleap) (2021.10.8)
Requirement already satisfied: tensorflow<2.9.0,>=2.6.3 in /usr/local/lib/python3.7/dist-packages (from sleap) (2.8.0)
Requirement already satisfied: pyzmq in /usr/local/lib/python3.7/dist-packages (from sleap) (22.3.0)
Collecting jsonpickle==1.2
Downloading jsonpickle-1.2-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: scikit-learn==1.0.* in /usr/local/lib/python3.7/dist-packages (from sleap) (1.0.2)
Collecting opencv-python-headless<=4.5.5.62,>=4.2.0.34
Downloading opencv_python_headless-4.5.5.62-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (47.7 MB)
|████████████████████████████████| 47.7 MB 85 kB/s
?25hCollecting rich==10.16.1
Downloading rich-10.16.1-py3-none-any.whl (214 kB)
|████████████████████████████████| 214 kB 49.1 MB/s
?25hRequirement already satisfied: numpy<=1.21.5,>=1.19.5 in /usr/local/lib/python3.7/dist-packages (from sleap) (1.21.5)
Requirement already satisfied: imageio<=2.15.0 in /usr/local/lib/python3.7/dist-packages (from sleap) (2.4.1)
Collecting scikit-video
Downloading scikit_video-1.1.11-py2.py3-none-any.whl (2.3 MB)
|████████████████████████████████| 2.3 MB 38.3 MB/s
?25hRequirement already satisfied: h5py<=3.6.0,>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from sleap) (3.1.0)
Collecting cattrs==1.1.1
Downloading cattrs-1.1.1-py3-none-any.whl (16 kB)
Collecting python-rapidjson
Downloading python_rapidjson-1.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
|████████████████████████████████| 1.6 MB 39.3 MB/s
?25hCollecting imgaug==0.4.0
Downloading imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
|████████████████████████████████| 948 kB 40.4 MB/s
?25hRequirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from sleap) (2.6.3)
Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from sleap) (0.11.2)
Collecting pykalman==0.9.5
Downloading pykalman-0.9.5.tar.gz (228 kB)
|████████████████████████████████| 228 kB 52.8 MB/s
?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from sleap) (3.13)
Collecting PySide2<=5.14.1,>=5.13.2
Downloading PySide2-5.14.1-5.14.1-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (165.5 MB)
|████████████████████████████████| 165.5 MB 76 kB/s
?25hCollecting attrs==21.2.0
Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
|████████████████████████████████| 53 kB 2.3 MB/s
?25hRequirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from sleap) (5.4.8)
Collecting imgstore==0.2.9
Downloading imgstore-0.2.9-py2.py3-none-any.whl (904 kB)
|████████████████████████████████| 904 kB 39.4 MB/s
?25hCollecting jsmin
Downloading jsmin-3.0.1.tar.gz (13 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from sleap) (1.3.5)
Collecting segmentation-models==1.0.1
Downloading segmentation_models-1.0.1-py3-none-any.whl (33 kB)
Requirement already satisfied: scipy<=1.7.3,>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from sleap) (1.4.1)
Requirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from sleap) (0.18.3)
Collecting qimage2ndarray<=1.8.3,>=1.8.2
Downloading qimage2ndarray-1.8.3-py3-none-any.whl (11 kB)
Requirement already satisfied: Shapely in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (1.8.1.post1)
Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (7.1.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (3.2.2)
Collecting opencv-python
Downloading opencv_python-4.5.5.64-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.5 MB)
|████████████████████████████████| 60.5 MB 1.4 MB/s
?25hRequirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from imgaug==0.4.0->sleap) (1.15.0)
Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (2018.9)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (2.8.2)
Requirement already satisfied: tzlocal in /usr/local/lib/python3.7/dist-packages (from imgstore==0.2.9->sleap) (1.5.1)
Collecting colorama<0.5.0,>=0.4.0
Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: typing-extensions<5.0,>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from rich==10.16.1->sleap) (3.10.0.2)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.7/dist-packages (from rich==10.16.1->sleap) (2.6.1)
Collecting commonmark<0.10.0,>=0.9.0
Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
|████████████████████████████████| 51 kB 6.5 MB/s
?25hRequirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==1.0.*->sleap) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==1.0.*->sleap) (3.1.0)
Collecting keras-applications<=1.0.8,>=1.0.7
Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
|████████████████████████████████| 50 kB 6.0 MB/s
?25hCollecting image-classifiers==1.0.0
Downloading image_classifiers-1.0.0-py3-none-any.whl (19 kB)
Collecting efficientnet==1.0.0
Downloading efficientnet-1.0.0-py3-none-any.whl (17 kB)
Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py<=3.6.0,>=3.1.0->sleap) (1.5.2)
Collecting shiboken2==5.14.1
Downloading shiboken2-5.14.1-5.14.1-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (847 kB)
|████████████████████████████████| 847 kB 39.4 MB/s
?25hRequirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image->sleap) (2021.11.2)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from scikit-image->sleap) (1.3.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (1.4.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->imgaug==0.4.0->sleap) (3.0.7)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.24.0)
Requirement already satisfied: flatbuffers>=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.6.3)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (3.3.0)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (3.17.3)
Collecting tf-estimator-nightly==2.8.0.dev2021122109
Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
|████████████████████████████████| 462 kB 48.3 MB/s
?25hRequirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.2.0)
Requirement already satisfied: tensorboard<2.9,>=2.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.8.0)
Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.14.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (57.4.0)
Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.1.2)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.1.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.44.0)
Requirement already satisfied: gast>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (0.5.3)
Requirement already satisfied: libclang>=9.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (13.0.0)
Requirement already satisfied: absl-py>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (1.0.0)
Requirement already satisfied: keras<2.9,>=2.8.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2.9.0,>=2.6.3->sleap) (2.8.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow<2.9.0,>=2.6.3->sleap) (0.37.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.3.6)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.8.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.0.1)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (2.23.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.35.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.4.6)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.2.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (4.11.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.7.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (0.4.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (2.10)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.9,>=2.8->tensorflow<2.9.0,>=2.6.3->sleap) (3.2.0)
Building wheels for collected packages: pykalman, jsmin
Building wheel for pykalman (setup.py) ... ?25l?25hdone
Created wheel for pykalman: filename=pykalman-0.9.5-py3-none-any.whl size=48462 sha256=dde739150408cee5e4cb98680575a79e9cf2574d606fea22d81dac69689e1b5f
Stored in directory: /root/.cache/pip/wheels/6a/04/02/2dda6ea59c66d9e685affc8af3a31ad3a5d87b7311689efce6
Building wheel for jsmin (setup.py) ... ?25l?25hdone
Created wheel for jsmin: filename=jsmin-3.0.1-py3-none-any.whl size=13782 sha256=28e30a78deeb41cb8a5a2a452ecd4209438e26a6f74af8de2e29a7da35b6fe93
Stored in directory: /root/.cache/pip/wheels/a4/0b/64/fb4f87526ecbdf7921769a39d91dcfe4860e621cf15b8250d6
Successfully built pykalman jsmin
Installing collected packages: keras-applications, tf-estimator-nightly, shiboken2, opencv-python, image-classifiers, efficientnet, commonmark, colorama, attrs, segmentation-models, scikit-video, rich, qimage2ndarray, python-rapidjson, PySide2, pykalman, opencv-python-headless, jsonpickle, jsmin, imgstore, imgaug, cattrs, sleap
Attempting uninstall: attrs
Found existing installation: attrs 21.4.0
Uninstalling attrs-21.4.0:
Successfully uninstalled attrs-21.4.0
Attempting uninstall: imgaug
Found existing installation: imgaug 0.2.9
Uninstalling imgaug-0.2.9:
Successfully uninstalled imgaug-0.2.9
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.4.0 which is incompatible.
Successfully installed PySide2-5.14.1 attrs-21.2.0 cattrs-1.1.1 colorama-0.4.4 commonmark-0.9.1 efficientnet-1.0.0 image-classifiers-1.0.0 imgaug-0.4.0 imgstore-0.2.9 jsmin-3.0.1 jsonpickle-1.2 keras-applications-1.0.8 opencv-python-4.5.5.64 opencv-python-headless-4.5.5.62 pykalman-0.9.5 python-rapidjson-1.6 qimage2ndarray-1.8.3 rich-10.16.1 scikit-video-1.1.11 segmentation-models-1.0.1 shiboken2-5.14.1 sleap-1.2.2 tf-estimator-nightly-2.8.0.dev2021122109
Import SLEAP to make sure it installed correctly and print out some information about the system:
import sleap
sleap.versions()
sleap.system_summary()
INFO:numexpr.utils:NumExpr defaulting to 2 threads.
SLEAP: 1.2.2
TensorFlow: 2.8.0
Numpy: 1.21.5
Python: 3.7.13
OS: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: None
2. Setup training data#
Here we will download an existing training dataset package. This is an .slp file that contains both the labeled poses, as well as the image data for labeled frames.
If running on Google Colab, you’ll want to replace this with mounting your Google Drive folder containing your own data, or if running locally, simply change the path to your labels below in TRAINING_SLP_FILE.
# !curl -L --output labels.pkg.slp https://www.dropbox.com/s/b990gxjt3d3j3jh/210205.sleap_wt_gold.13pt.pkg.slp?dl=1
!curl -L --output labels.pkg.slp https://storage.googleapis.com/sleap-data/datasets/wt_gold.13pt/tracking_split2/train.pkg.slp
!ls -lah
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 619M 100 619M 0 0 106M 0 0:00:05 0:00:05 --:--:-- 110M
total 620M
drwxr-xr-x 1 root root 4.0K Apr 3 23:48 .
drwxr-xr-x 1 root root 4.0K Apr 3 23:40 ..
drwxr-xr-x 4 root root 4.0K Mar 23 14:21 .config
-rw-r--r-- 1 root root 620M Apr 3 23:48 labels.pkg.slp
drwxr-xr-x 1 root root 4.0K Mar 23 14:22 sample_data
TRAINING_SLP_FILE = "labels.pkg.slp"
3. Setup training job#
A SLEAP TrainingJobConfig is a structure that contains all of the hyperparameters needed to train a SLEAP model. This is typically saved out to initial_config.json and training_config.json in the model folder so that training runs can be reproduced if needed, as well as to store metadata necessary for inference.
Normally, these are generated interactively by the GUI, or manually by editing an existing JSON file in a text editor. Here, we will define a configuration interactively entirely in Python.
from sleap.nn.config import *
# Initialize the default training job configuration.
cfg = TrainingJobConfig()
# Update path to training data we just downloaded.
cfg.data.labels.training_labels = TRAINING_SLP_FILE
cfg.data.labels.validation_fraction = 0.1
# Preprocesssing and training parameters.
cfg.data.instance_cropping.center_on_part = "thorax"
cfg.optimization.augmentation_config.rotate = True
cfg.optimization.epochs = 10 # This is the maximum number of training rounds.
# These configures the actual neural network and the model type:
cfg.model.backbone.unet = UNetConfig(
filters=16,
output_stride=4
)
cfg.model.heads.centered_instance = CenteredInstanceConfmapsHeadConfig(
anchor_part="thorax",
sigma=1.5,
output_stride=4
)
# Setup how we want to save the trained model.
cfg.outputs.run_name = "baseline_model.topdown"
Existing configs can also be loaded from a .json file with:
cfg = sleap.load_config("training_config.json")
4. Training#
Next we will create a SLEAP Trainer from the configuration we just specified. This handles all the nitty gritty mechanics necessary to setup training in the backend.
trainer = sleap.nn.training.Trainer.from_config(cfg)
INFO:sleap.nn.training:Loading training labels from: labels.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training: Splits: Training = 1440 / Validation = 160.
Great, now we’re ready to do the first round of training. This is when the model will actually start to improve over time:
trainer.train()
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
INFO:sleap.nn.training:Loaded test example. [6.047s]
INFO:sleap.nn.training: Input shape: (160, 160, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training: Backbone: UNet(stacks=1, filters=16, filters_rate=2, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=False, block_contraction=False)
INFO:sleap.nn.training: Max stride: 16
INFO:sleap.nn.training: Parameters: 2,101,501
INFO:sleap.nn.training: Heads:
INFO:sleap.nn.training: [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part='thorax', sigma=1.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training: Outputs:
INFO:sleap.nn.training: [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 40, 40, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 1440
INFO:sleap.nn.training:Validation set: n = 160
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training: Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training: Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-06, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.training:Created run path: models/baseline_model.topdown
INFO:sleap.nn.training:Setting up visualization...
Unable to use Qt backend for matplotlib. This probably means Qt is running headless.
INFO:sleap.nn.training:Finished trainer set up. [10.4s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [29.5s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/10
360/360 - 70s - loss: 0.0037 - head: 0.0029 - thorax: 0.0030 - abdomen: 0.0037 - wingL: 0.0041 - wingR: 0.0041 - forelegL4: 0.0037 - forelegR4: 0.0038 - midlegL4: 0.0041 - midlegR4: 0.0041 - hindlegL4: 0.0039 - hindlegR4: 0.0040 - eyeL: 0.0033 - eyeR: 0.0034 - val_loss: 0.0033 - val_head: 0.0017 - val_thorax: 0.0025 - val_abdomen: 0.0035 - val_wingL: 0.0039 - val_wingR: 0.0039 - val_forelegL4: 0.0033 - val_forelegR4: 0.0036 - val_midlegL4: 0.0040 - val_midlegR4: 0.0040 - val_hindlegL4: 0.0040 - val_hindlegR4: 0.0040 - val_eyeL: 0.0022 - val_eyeR: 0.0023 - lr: 1.0000e-04 - 70s/epoch - 194ms/step
Epoch 2/10
360/360 - 53s - loss: 0.0028 - head: 0.0013 - thorax: 0.0020 - abdomen: 0.0028 - wingL: 0.0031 - wingR: 0.0031 - forelegL4: 0.0032 - forelegR4: 0.0033 - midlegL4: 0.0039 - midlegR4: 0.0039 - hindlegL4: 0.0037 - hindlegR4: 0.0038 - eyeL: 0.0013 - eyeR: 0.0014 - val_loss: 0.0025 - val_head: 9.5906e-04 - val_thorax: 0.0013 - val_abdomen: 0.0023 - val_wingL: 0.0025 - val_wingR: 0.0025 - val_forelegL4: 0.0029 - val_forelegR4: 0.0030 - val_midlegL4: 0.0037 - val_midlegR4: 0.0038 - val_hindlegL4: 0.0037 - val_hindlegR4: 0.0038 - val_eyeL: 8.8668e-04 - val_eyeR: 9.7728e-04 - lr: 1.0000e-04 - 53s/epoch - 148ms/step
Epoch 3/10
360/360 - 55s - loss: 0.0023 - head: 8.0222e-04 - thorax: 9.4507e-04 - abdomen: 0.0022 - wingL: 0.0022 - wingR: 0.0022 - forelegL4: 0.0027 - forelegR4: 0.0028 - midlegL4: 0.0035 - midlegR4: 0.0036 - hindlegL4: 0.0034 - hindlegR4: 0.0036 - eyeL: 8.5909e-04 - eyeR: 8.8003e-04 - val_loss: 0.0021 - val_head: 7.4704e-04 - val_thorax: 6.8354e-04 - val_abdomen: 0.0020 - val_wingL: 0.0018 - val_wingR: 0.0019 - val_forelegL4: 0.0024 - val_forelegR4: 0.0025 - val_midlegL4: 0.0031 - val_midlegR4: 0.0034 - val_hindlegL4: 0.0032 - val_hindlegR4: 0.0035 - val_eyeL: 7.6220e-04 - val_eyeR: 7.1808e-04 - lr: 1.0000e-04 - 55s/epoch - 154ms/step
Epoch 4/10
360/360 - 61s - loss: 0.0019 - head: 6.5537e-04 - thorax: 5.3996e-04 - abdomen: 0.0019 - wingL: 0.0018 - wingR: 0.0018 - forelegL4: 0.0023 - forelegR4: 0.0024 - midlegL4: 0.0027 - midlegR4: 0.0029 - hindlegL4: 0.0029 - hindlegR4: 0.0032 - eyeL: 7.4337e-04 - eyeR: 7.2396e-04 - val_loss: 0.0017 - val_head: 5.5193e-04 - val_thorax: 3.6303e-04 - val_abdomen: 0.0018 - val_wingL: 0.0016 - val_wingR: 0.0016 - val_forelegL4: 0.0020 - val_forelegR4: 0.0020 - val_midlegL4: 0.0023 - val_midlegR4: 0.0026 - val_hindlegL4: 0.0027 - val_hindlegR4: 0.0031 - val_eyeL: 6.5068e-04 - val_eyeR: 6.0169e-04 - lr: 1.0000e-04 - 61s/epoch - 169ms/step
Epoch 5/10
360/360 - 57s - loss: 0.0016 - head: 5.6982e-04 - thorax: 4.1064e-04 - abdomen: 0.0017 - wingL: 0.0016 - wingR: 0.0016 - forelegL4: 0.0020 - forelegR4: 0.0020 - midlegL4: 0.0021 - midlegR4: 0.0022 - hindlegL4: 0.0024 - hindlegR4: 0.0028 - eyeL: 6.5447e-04 - eyeR: 6.3768e-04 - val_loss: 0.0014 - val_head: 4.9811e-04 - val_thorax: 3.0411e-04 - val_abdomen: 0.0015 - val_wingL: 0.0014 - val_wingR: 0.0014 - val_forelegL4: 0.0017 - val_forelegR4: 0.0019 - val_midlegL4: 0.0018 - val_midlegR4: 0.0020 - val_hindlegL4: 0.0023 - val_hindlegR4: 0.0026 - val_eyeL: 5.9634e-04 - val_eyeR: 5.8405e-04 - lr: 1.0000e-04 - 57s/epoch - 157ms/step
Epoch 6/10
360/360 - 54s - loss: 0.0014 - head: 5.1206e-04 - thorax: 3.4952e-04 - abdomen: 0.0015 - wingL: 0.0014 - wingR: 0.0014 - forelegL4: 0.0017 - forelegR4: 0.0018 - midlegL4: 0.0017 - midlegR4: 0.0018 - hindlegL4: 0.0020 - hindlegR4: 0.0023 - eyeL: 6.0045e-04 - eyeR: 5.7847e-04 - val_loss: 0.0012 - val_head: 4.3860e-04 - val_thorax: 2.5352e-04 - val_abdomen: 0.0014 - val_wingL: 0.0013 - val_wingR: 0.0012 - val_forelegL4: 0.0015 - val_forelegR4: 0.0016 - val_midlegL4: 0.0014 - val_midlegR4: 0.0017 - val_hindlegL4: 0.0020 - val_hindlegR4: 0.0022 - val_eyeL: 5.1261e-04 - val_eyeR: 5.5203e-04 - lr: 1.0000e-04 - 54s/epoch - 151ms/step
Epoch 7/10
360/360 - 54s - loss: 0.0012 - head: 4.7131e-04 - thorax: 3.1231e-04 - abdomen: 0.0014 - wingL: 0.0012 - wingR: 0.0012 - forelegL4: 0.0016 - forelegR4: 0.0016 - midlegL4: 0.0015 - midlegR4: 0.0016 - hindlegL4: 0.0018 - hindlegR4: 0.0020 - eyeL: 5.7016e-04 - eyeR: 5.4539e-04 - val_loss: 0.0011 - val_head: 4.3133e-04 - val_thorax: 2.2694e-04 - val_abdomen: 0.0013 - val_wingL: 0.0011 - val_wingR: 0.0011 - val_forelegL4: 0.0014 - val_forelegR4: 0.0015 - val_midlegL4: 0.0013 - val_midlegR4: 0.0015 - val_hindlegL4: 0.0018 - val_hindlegR4: 0.0020 - val_eyeL: 5.5373e-04 - val_eyeR: 5.0355e-04 - lr: 1.0000e-04 - 54s/epoch - 149ms/step
Epoch 8/10
360/360 - 53s - loss: 0.0011 - head: 4.3369e-04 - thorax: 2.6750e-04 - abdomen: 0.0013 - wingL: 0.0011 - wingR: 0.0011 - forelegL4: 0.0015 - forelegR4: 0.0015 - midlegL4: 0.0014 - midlegR4: 0.0014 - hindlegL4: 0.0017 - hindlegR4: 0.0018 - eyeL: 5.2745e-04 - eyeR: 5.0480e-04 - val_loss: 0.0011 - val_head: 4.1774e-04 - val_thorax: 2.4407e-04 - val_abdomen: 0.0013 - val_wingL: 0.0011 - val_wingR: 0.0010 - val_forelegL4: 0.0013 - val_forelegR4: 0.0014 - val_midlegL4: 0.0012 - val_midlegR4: 0.0014 - val_hindlegL4: 0.0017 - val_hindlegR4: 0.0018 - val_eyeL: 6.2877e-04 - val_eyeR: 5.7243e-04 - lr: 1.0000e-04 - 53s/epoch - 148ms/step
Epoch 9/10
360/360 - 53s - loss: 0.0010 - head: 4.0425e-04 - thorax: 2.3597e-04 - abdomen: 0.0012 - wingL: 0.0010 - wingR: 0.0011 - forelegL4: 0.0014 - forelegR4: 0.0014 - midlegL4: 0.0013 - midlegR4: 0.0013 - hindlegL4: 0.0016 - hindlegR4: 0.0017 - eyeL: 5.0906e-04 - eyeR: 4.9227e-04 - val_loss: 0.0010 - val_head: 3.9088e-04 - val_thorax: 2.1458e-04 - val_abdomen: 0.0012 - val_wingL: 0.0010 - val_wingR: 9.4879e-04 - val_forelegL4: 0.0012 - val_forelegR4: 0.0013 - val_midlegL4: 0.0011 - val_midlegR4: 0.0014 - val_hindlegL4: 0.0016 - val_hindlegR4: 0.0017 - val_eyeL: 4.6829e-04 - val_eyeR: 4.7323e-04 - lr: 1.0000e-04 - 53s/epoch - 147ms/step
Epoch 10/10
360/360 - 55s - loss: 9.7632e-04 - head: 3.7896e-04 - thorax: 2.1828e-04 - abdomen: 0.0011 - wingL: 9.9185e-04 - wingR: 9.9033e-04 - forelegL4: 0.0014 - forelegR4: 0.0013 - midlegL4: 0.0012 - midlegR4: 0.0012 - hindlegL4: 0.0015 - hindlegR4: 0.0016 - eyeL: 4.7323e-04 - eyeR: 4.5868e-04 - val_loss: 9.2870e-04 - val_head: 3.3704e-04 - val_thorax: 1.5806e-04 - val_abdomen: 0.0010 - val_wingL: 9.5121e-04 - val_wingR: 9.2122e-04 - val_forelegL4: 0.0012 - val_forelegR4: 0.0014 - val_midlegL4: 0.0010 - val_midlegR4: 0.0012 - val_hindlegL4: 0.0015 - val_hindlegR4: 0.0016 - val_eyeL: 4.2130e-04 - val_eyeR: 4.1479e-04 - lr: 1.0000e-04 - 55s/epoch - 154ms/step
INFO:sleap.nn.training:Finished training loop. [9.4 min]
INFO:sleap.nn.training:Deleting visualization directory: models/baseline_model.topdown/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
INFO:sleap.nn.evals:Saved predictions: models/baseline_model.topdown/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: models/baseline_model.topdown/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.518988
INFO:sleap.nn.evals:Saved predictions: models/baseline_model.topdown/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: models/baseline_model.topdown/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.520377
5. Continuing training#
If we still have the trainer in memory, we can continue training by simply calling trainer.train() again with a potentially different number of epochs:
trainer.config.optimization.epochs = 3
trainer.train()
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [29.4s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/3
360/360 - 57s - loss: 9.1732e-04 - head: 3.5629e-04 - thorax: 1.9609e-04 - abdomen: 0.0010 - wingL: 9.1318e-04 - wingR: 9.1330e-04 - forelegL4: 0.0013 - forelegR4: 0.0013 - midlegL4: 0.0011 - midlegR4: 0.0011 - hindlegL4: 0.0014 - hindlegR4: 0.0015 - eyeL: 4.4475e-04 - eyeR: 4.3944e-04 - val_loss: 9.2727e-04 - val_head: 3.8719e-04 - val_thorax: 1.5200e-04 - val_abdomen: 0.0011 - val_wingL: 9.3115e-04 - val_wingR: 8.9376e-04 - val_forelegL4: 0.0012 - val_forelegR4: 0.0012 - val_midlegL4: 9.9703e-04 - val_midlegR4: 0.0012 - val_hindlegL4: 0.0015 - val_hindlegR4: 0.0016 - val_eyeL: 4.5374e-04 - val_eyeR: 5.1839e-04 - lr: 1.0000e-04 - 57s/epoch - 158ms/step
Epoch 2/3
360/360 - 56s - loss: 8.7900e-04 - head: 3.4532e-04 - thorax: 1.7895e-04 - abdomen: 0.0010 - wingL: 8.7539e-04 - wingR: 8.8524e-04 - forelegL4: 0.0012 - forelegR4: 0.0012 - midlegL4: 0.0010 - midlegR4: 0.0010 - hindlegL4: 0.0014 - hindlegR4: 0.0014 - eyeL: 4.3484e-04 - eyeR: 4.2888e-04 - val_loss: 8.5310e-04 - val_head: 3.0429e-04 - val_thorax: 1.4837e-04 - val_abdomen: 0.0010 - val_wingL: 8.2237e-04 - val_wingR: 8.3093e-04 - val_forelegL4: 0.0011 - val_forelegR4: 0.0012 - val_midlegL4: 8.5634e-04 - val_midlegR4: 0.0011 - val_hindlegL4: 0.0014 - val_hindlegR4: 0.0015 - val_eyeL: 4.0362e-04 - val_eyeR: 3.8104e-04 - lr: 1.0000e-04 - 56s/epoch - 156ms/step
Epoch 3/3
360/360 - 56s - loss: 8.4466e-04 - head: 3.4540e-04 - thorax: 1.6180e-04 - abdomen: 9.6890e-04 - wingL: 8.4974e-04 - wingR: 8.5187e-04 - forelegL4: 0.0012 - forelegR4: 0.0012 - midlegL4: 9.5015e-04 - midlegR4: 9.8870e-04 - hindlegL4: 0.0013 - hindlegR4: 0.0014 - eyeL: 4.2245e-04 - eyeR: 4.0856e-04 - val_loss: 8.2153e-04 - val_head: 3.1832e-04 - val_thorax: 1.4803e-04 - val_abdomen: 9.4013e-04 - val_wingL: 8.4738e-04 - val_wingR: 8.4686e-04 - val_forelegL4: 0.0010 - val_forelegR4: 0.0011 - val_midlegL4: 8.5740e-04 - val_midlegR4: 0.0010 - val_hindlegL4: 0.0014 - val_hindlegR4: 0.0015 - val_eyeL: 3.7928e-04 - val_eyeR: 3.8285e-04 - lr: 1.0000e-04 - 56s/epoch - 156ms/step
INFO:sleap.nn.training:Finished training loop. [2.8 min]
INFO:sleap.nn.training:Deleting visualization directory: models/baseline_model.topdown/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
INFO:sleap.nn.evals:Saved predictions: models/baseline_model.topdown/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: models/baseline_model.topdown/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.551905
INFO:sleap.nn.evals:Saved predictions: models/baseline_model.topdown/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: models/baseline_model.topdown/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.551469
As you can see, the loss and accuracy pick up from where it left off in the previous training.
Usually, however, if you’re continuing training it’s likely because you’re starting off from an already trained model.
In this case, all you need to do to continue training is to create a new Trainer from the existing model configuration and load up the weights before continuing training:
# Load config.
cfg = sleap.load_config("models/baseline_model.topdown")
# cfg.outputs.run_name = "new_folder" # Set the run_name to a new value if you want the model to be saved to a different folder.
# Create and initialize the trainer.
trainer = sleap.nn.training.Trainer.from_config(cfg)
trainer.setup()
# Replace the randomly initialized weights with the saved weights.
trainer.keras_model.load_weights("models/baseline_model.topdown/best_model.h5")
INFO:sleap.nn.training:Loading training labels from: labels.pkg.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training: Splits: Training = 1440 / Validation = 160.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
INFO:sleap.nn.training:Loaded test example. [1.909s]
INFO:sleap.nn.training: Input shape: (160, 160, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training: Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=False, block_contraction=False)
INFO:sleap.nn.training: Max stride: 16
INFO:sleap.nn.training: Parameters: 2,101,501
INFO:sleap.nn.training: Heads:
INFO:sleap.nn.training: [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part='thorax', sigma=1.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training: Outputs:
INFO:sleap.nn.training: [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 40, 40, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 1440
INFO:sleap.nn.training:Validation set: n = 160
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training: Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training: Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-06, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.training:Created run path: models/baseline_model.topdown
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [6.0s]
trainer.config.optimization.epochs = 3
trainer.train()
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [28.9s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/3
360/360 - 63s - loss: 8.2769e-04 - head: 3.4427e-04 - thorax: 1.6900e-04 - abdomen: 9.4941e-04 - wingL: 8.1514e-04 - wingR: 8.1826e-04 - forelegL4: 0.0012 - forelegR4: 0.0012 - midlegL4: 9.2980e-04 - midlegR4: 9.6439e-04 - hindlegL4: 0.0013 - hindlegR4: 0.0013 - eyeL: 4.2129e-04 - eyeR: 4.0767e-04 - val_loss: 7.8855e-04 - val_head: 3.2701e-04 - val_thorax: 1.8405e-04 - val_abdomen: 0.0010 - val_wingL: 7.3709e-04 - val_wingR: 7.1027e-04 - val_forelegL4: 0.0010 - val_forelegR4: 0.0011 - val_midlegL4: 9.3918e-04 - val_midlegR4: 9.0288e-04 - val_hindlegL4: 0.0012 - val_hindlegR4: 0.0013 - val_eyeL: 3.8746e-04 - val_eyeR: 3.3939e-04 - lr: 1.0000e-04 - 63s/epoch - 174ms/step
Epoch 2/3
360/360 - 58s - loss: 7.9662e-04 - head: 3.2407e-04 - thorax: 1.5127e-04 - abdomen: 9.1911e-04 - wingL: 7.6866e-04 - wingR: 7.8884e-04 - forelegL4: 0.0011 - forelegR4: 0.0011 - midlegL4: 8.8560e-04 - midlegR4: 9.3151e-04 - hindlegL4: 0.0012 - hindlegR4: 0.0013 - eyeL: 4.1677e-04 - eyeR: 3.9983e-04 - val_loss: 7.3673e-04 - val_head: 2.8314e-04 - val_thorax: 1.1026e-04 - val_abdomen: 9.4263e-04 - val_wingL: 6.7871e-04 - val_wingR: 6.4992e-04 - val_forelegL4: 0.0011 - val_forelegR4: 0.0011 - val_midlegL4: 8.0315e-04 - val_midlegR4: 8.3331e-04 - val_hindlegL4: 0.0012 - val_hindlegR4: 0.0012 - val_eyeL: 3.4531e-04 - val_eyeR: 3.5707e-04 - lr: 1.0000e-04 - 58s/epoch - 162ms/step
Epoch 3/3
360/360 - 58s - loss: 7.6463e-04 - head: 3.0854e-04 - thorax: 1.3497e-04 - abdomen: 8.9188e-04 - wingL: 7.4921e-04 - wingR: 7.5430e-04 - forelegL4: 0.0011 - forelegR4: 0.0011 - midlegL4: 8.3320e-04 - midlegR4: 8.7736e-04 - hindlegL4: 0.0012 - hindlegR4: 0.0013 - eyeL: 3.9640e-04 - eyeR: 3.7940e-04 - val_loss: 7.0126e-04 - val_head: 2.8905e-04 - val_thorax: 1.1305e-04 - val_abdomen: 9.0676e-04 - val_wingL: 6.4827e-04 - val_wingR: 6.2576e-04 - val_forelegL4: 0.0010 - val_forelegR4: 9.8253e-04 - val_midlegL4: 8.0471e-04 - val_midlegR4: 7.3788e-04 - val_hindlegL4: 0.0011 - val_hindlegR4: 0.0012 - val_eyeL: 3.1543e-04 - val_eyeR: 3.4044e-04 - lr: 1.0000e-04 - 58s/epoch - 161ms/step
INFO:sleap.nn.training:Finished training loop. [3.0 min]
INFO:sleap.nn.training:Deleting visualization directory: models/baseline_model.topdown/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
INFO:sleap.nn.evals:Saved predictions: models/baseline_model.topdown/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: models/baseline_model.topdown/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.597609
INFO:sleap.nn.evals:Saved predictions: models/baseline_model.topdown/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: models/baseline_model.topdown/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.621393
Again, the loss and accuracy pick up from where they left off prior to this round of training.
The resulting model can be used as usual for inference on new data.