Training and inference on an example dataset#
In this notebook we’ll install SLEAP, download a sample dataset, run training and inference on that dataset using the SLEAP command-line interface, and then download the predictions.
Install SLEAP#
Note: Before installing SLEAP check SLEAP releases page for the latest version.
!pip uninstall -qqq -y opencv-python opencv-contrib-python
!pip install -qqq "sleap[pypi]>=1.3.3"
ERROR: Cannot uninstall opencv-python 4.6.0, RECORD file not found. Hint: The package was installed by conda.
ERROR: Cannot uninstall shiboken2 5.15.6, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps shiboken2==5.15.6'.
Download sample training data into Colab#
Let’s download a sample dataset from the SLEAP sample datasets repository into Colab.
!apt-get install tree
!wget -O dataset.zip https://github.com/talmolab/sleap-datasets/releases/download/dm-courtship-v1/drosophila-melanogaster-courtship.zip
!mkdir dataset
!unzip dataset.zip -d dataset
!rm dataset.zip
!tree dataset
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
--2023-09-01 13:30:33-- https://github.com/talmolab/sleap-datasets/releases/download/dm-courtship-v1/drosophila-melanogaster-courtship.zip
Resolving github.com (github.com)... 192.30.255.113
Connecting to github.com (github.com)|192.30.255.113|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/263375180/16df8d00-94f1-11ea-98d1-6c03a2f89e1c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230901%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230901T203033Z&X-Amz-Expires=300&X-Amz-Signature=b9b0638744af3144affdc46668c749128bd6c4f23ca2a1313821c7bbcd54ccdd&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=263375180&response-content-disposition=attachment%3B%20filename%3Ddrosophila-melanogaster-courtship.zip&response-content-type=application%2Foctet-stream [following]
--2023-09-01 13:30:33-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/263375180/16df8d00-94f1-11ea-98d1-6c03a2f89e1c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230901%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230901T203033Z&X-Amz-Expires=300&X-Amz-Signature=b9b0638744af3144affdc46668c749128bd6c4f23ca2a1313821c7bbcd54ccdd&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=263375180&response-content-disposition=attachment%3B%20filename%3Ddrosophila-melanogaster-courtship.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 111973079 (107M) [application/octet-stream]
Saving to: ‘dataset.zip’
dataset.zip 100%[===================>] 106.79M 63.0MB/s in 1.7s
2023-09-01 13:30:35 (63.0 MB/s) - ‘dataset.zip’ saved [111973079/111973079]
Archive: dataset.zip
creating: dataset/drosophila-melanogaster-courtship/
inflating: dataset/drosophila-melanogaster-courtship/.DS_Store
creating: dataset/__MACOSX/
creating: dataset/__MACOSX/drosophila-melanogaster-courtship/
inflating: dataset/__MACOSX/drosophila-melanogaster-courtship/._.DS_Store
inflating: dataset/drosophila-melanogaster-courtship/20190128_113421.mp4
inflating: dataset/__MACOSX/drosophila-melanogaster-courtship/._20190128_113421.mp4
inflating: dataset/drosophila-melanogaster-courtship/courtship_labels.slp
inflating: dataset/__MACOSX/drosophila-melanogaster-courtship/._courtship_labels.slp
inflating: dataset/drosophila-melanogaster-courtship/example.jpg
inflating: dataset/__MACOSX/drosophila-melanogaster-courtship/._example.jpg
dataset
├── drosophila-melanogaster-courtship
│ ├── 20190128_113421.mp4
│ ├── courtship_labels.slp
│ └── example.jpg
└── __MACOSX
└── drosophila-melanogaster-courtship
3 directories, 3 files
Train models#
For the top-down pipeline, we’ll need train two models: a centroid model and a centered-instance model.
Using the command-line interface, we’ll first train a model for centroids using the default training profile. The training profile determines the model architecture, the learning rate, and other parameters.
When you start training, you’ll first see the training parameters and then the training and validation loss for each training epoch.
As soon as you’re satisfied with the validation loss you see for an epoch during training, you’re welcome to stop training by clicking the stop button. The version of the model with the lowest validation loss is saved during training, and that’s what will be used for inference.
If you don’t stop training, it will run for 200 epochs or until validation loss fails to improve for some number of epochs (controlled by the early_stopping fields in the training profile).
!sleap-train baseline.centroid.json "dataset/drosophila-melanogaster-courtship/courtship_labels.slp" --run_name "courtship.centroid" --video-paths "dataset/drosophila-melanogaster-courtship/20190128_113421.mp4"
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.2
TensorFlow: 2.7.0
Numpy: 1.21.5
Python: 3.7.12
OS: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid
INFO:sleap.nn.training:Training labels file: dataset/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /home/talmolab/sleap-estimates-animal-poses/pull-requests/sleap/sleap/training_profiles/baseline.centroid.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
"training_job_path": "baseline.centroid.json",
"labels_path": "dataset/drosophila-melanogaster-courtship/courtship_labels.slp",
"video_paths": [
"dataset/drosophila-melanogaster-courtship/20190128_113421.mp4"
],
"val_labels": null,
"test_labels": null,
"base_checkpoint": null,
"tensorboard": false,
"save_viz": false,
"zmq": false,
"run_name": "courtship.centroid",
"prefix": "",
"suffix": "",
"cpu": false,
"first_gpu": false,
"last_gpu": false,
"gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
"data": {
"labels": {
"training_labels": null,
"validation_labels": null,
"validation_fraction": 0.1,
"test_labels": null,
"split_by_inds": false,
"training_inds": null,
"validation_inds": null,
"test_inds": null,
"search_path_hints": [],
"skeletons": []
},
"preprocessing": {
"ensure_rgb": false,
"ensure_grayscale": false,
"imagenet_mode": null,
"input_scaling": 0.5,
"pad_to_stride": null,
"resize_and_pad_to_target": true,
"target_height": null,
"target_width": null
},
"instance_cropping": {
"center_on_part": null,
"crop_size": null,
"crop_size_detection_padding": 16
}
},
"model": {
"backbone": {
"leap": null,
"unet": {
"stem_stride": null,
"max_stride": 16,
"output_stride": 2,
"filters": 16,
"filters_rate": 2.0,
"middle_block": true,
"up_interpolate": true,
"stacks": 1
},
"hourglass": null,
"resnet": null,
"pretrained_encoder": null
},
"heads": {
"single_instance": null,
"centroid": {
"anchor_part": null,
"sigma": 2.5,
"output_stride": 2,
"loss_weight": 1.0,
"offset_refinement": false
},
"centered_instance": null,
"multi_instance": null,
"multi_class_bottomup": null,
"multi_class_topdown": null
},
"base_checkpoint": null
},
"optimization": {
"preload_data": true,
"augmentation_config": {
"rotate": true,
"rotation_min_angle": -15.0,
"rotation_max_angle": 15.0,
"translate": false,
"translate_min": -5,
"translate_max": 5,
"scale": false,
"scale_min": 0.9,
"scale_max": 1.1,
"uniform_noise": false,
"uniform_noise_min_val": 0.0,
"uniform_noise_max_val": 10.0,
"gaussian_noise": false,
"gaussian_noise_mean": 5.0,
"gaussian_noise_stddev": 1.0,
"contrast": false,
"contrast_min_gamma": 0.5,
"contrast_max_gamma": 2.0,
"brightness": false,
"brightness_min_val": 0.0,
"brightness_max_val": 10.0,
"random_crop": false,
"random_crop_height": 256,
"random_crop_width": 256,
"random_flip": false,
"flip_horizontal": true
},
"online_shuffling": true,
"shuffle_buffer_size": 128,
"prefetch": true,
"batch_size": 4,
"batches_per_epoch": null,
"min_batches_per_epoch": 200,
"val_batches_per_epoch": null,
"min_val_batches_per_epoch": 10,
"epochs": 200,
"optimizer": "adam",
"initial_learning_rate": 0.0001,
"learning_rate_schedule": {
"reduce_on_plateau": true,
"reduction_factor": 0.5,
"plateau_min_delta": 1e-08,
"plateau_patience": 5,
"plateau_cooldown": 3,
"min_learning_rate": 1e-08
},
"hard_keypoint_mining": {
"online_mining": false,
"hard_to_easy_ratio": 2.0,
"min_hard_keypoints": 2,
"max_hard_keypoints": null,
"loss_scale": 5.0
},
"early_stopping": {
"stop_training_on_plateau": true,
"plateau_min_delta": 1e-08,
"plateau_patience": 20
}
},
"outputs": {
"save_outputs": true,
"run_name": "courtship.centroid",
"run_name_prefix": "",
"run_name_suffix": null,
"runs_folder": "models",
"tags": [],
"save_visualizations": true,
"delete_viz_images": true,
"zip_outputs": false,
"log_to_csv": true,
"checkpointing": {
"initial_model": false,
"best_model": true,
"every_epoch": false,
"latest_model": false,
"final_model": false
},
"tensorboard": {
"write_logs": false,
"loss_frequency": "epoch",
"architecture_graph": false,
"profile_graph": false,
"visualizations": true
},
"zmq": {
"subscribe_to_controller": false,
"controller_address": "tcp://127.0.0.1:9000",
"controller_polling_timeout": 10,
"publish_updates": false,
"publish_address": "tcp://127.0.0.1:9001"
}
},
"name": "",
"description": "",
"sleap_version": "1.3.2",
"filename": "/home/talmolab/sleap-estimates-animal-poses/pull-requests/sleap/sleap/training_profiles/baseline.centroid.json"
}
INFO:sleap.nn.training:
2023-09-01 13:30:38.827290: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:38.831845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:38.832633: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.training:Auto-selected GPU 0 with 22980 MiB of free memory.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: dataset/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training: Splits: Training = 134 / Validation = 15.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2023-09-01 13:30:39.755154: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-01 13:30:39.756024: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:39.757213: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:39.758315: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:40.089801: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:40.090652: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:40.091464: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:30:40.092164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21084 MB memory: -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:sleap.nn.training:Loaded test example. [1.326s]
INFO:sleap.nn.training: Input shape: (512, 512, 3)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training: Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training: Max stride: 16
INFO:sleap.nn.training: Parameters: 1,953,393
INFO:sleap.nn.training: Heads:
INFO:sleap.nn.training: [0] = CentroidConfmapsHead(anchor_part=None, sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training: Outputs:
INFO:sleap.nn.training: [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 1), dtype=tf.float32, name=None), name='CentroidConfmapsHead/BiasAdd:0', description="created by layer 'CentroidConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 134
INFO:sleap.nn.training:Validation set: n = 15
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training: Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-08, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training: Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=20)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.training:Created run path: models/courtship.centroid
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [3.5s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [5.4s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/200
2023-09-01 13:30:49.814560: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2023-09-01 13:31:07.940585: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
200/200 - 20s - loss: 2.5945e-04 - val_loss: 1.5190e-04 - lr: 1.0000e-04 - 20s/epoch - 99ms/step
Epoch 2/200
200/200 - 11s - loss: 1.2513e-04 - val_loss: 9.5694e-05 - lr: 1.0000e-04 - 11s/epoch - 57ms/step
Epoch 3/200
200/200 - 11s - loss: 9.6987e-05 - val_loss: 6.8224e-05 - lr: 1.0000e-04 - 11s/epoch - 57ms/step
Epoch 4/200
200/200 - 12s - loss: 8.1486e-05 - val_loss: 5.0657e-05 - lr: 1.0000e-04 - 12s/epoch - 58ms/step
Epoch 5/200
200/200 - 11s - loss: 7.2174e-05 - val_loss: 5.3859e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 6/200
200/200 - 11s - loss: 5.9181e-05 - val_loss: 7.0259e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 7/200
200/200 - 11s - loss: 4.9353e-05 - val_loss: 4.9832e-05 - lr: 1.0000e-04 - 11s/epoch - 57ms/step
Epoch 8/200
200/200 - 11s - loss: 3.8997e-05 - val_loss: 4.4787e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 9/200
200/200 - 11s - loss: 3.5596e-05 - val_loss: 6.5150e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 10/200
200/200 - 12s - loss: 2.9256e-05 - val_loss: 3.8968e-05 - lr: 1.0000e-04 - 12s/epoch - 58ms/step
Epoch 11/200
200/200 - 11s - loss: 2.8572e-05 - val_loss: 3.5451e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 12/200
200/200 - 11s - loss: 2.2156e-05 - val_loss: 4.8602e-05 - lr: 1.0000e-04 - 11s/epoch - 53ms/step
Epoch 13/200
200/200 - 11s - loss: 1.7656e-05 - val_loss: 4.1905e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 14/200
200/200 - 11s - loss: 1.6440e-05 - val_loss: 3.6607e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 15/200
200/200 - 11s - loss: 1.4415e-05 - val_loss: 4.1699e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 16/200
200/200 - 11s - loss: 1.3589e-05 - val_loss: 3.5362e-05 - lr: 1.0000e-04 - 11s/epoch - 56ms/step
Epoch 17/200
200/200 - 11s - loss: 1.0888e-05 - val_loss: 2.1600e-05 - lr: 1.0000e-04 - 11s/epoch - 56ms/step
Epoch 18/200
200/200 - 11s - loss: 1.0426e-05 - val_loss: 3.6782e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 19/200
200/200 - 11s - loss: 9.9092e-06 - val_loss: 3.8284e-05 - lr: 1.0000e-04 - 11s/epoch - 56ms/step
Epoch 20/200
200/200 - 11s - loss: 8.0018e-06 - val_loss: 2.9439e-05 - lr: 1.0000e-04 - 11s/epoch - 57ms/step
Epoch 21/200
200/200 - 11s - loss: 7.7977e-06 - val_loss: 2.8703e-05 - lr: 1.0000e-04 - 11s/epoch - 56ms/step
Epoch 22/200
Epoch 00022: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.
200/200 - 11s - loss: 6.5981e-06 - val_loss: 3.6030e-05 - lr: 1.0000e-04 - 11s/epoch - 55ms/step
Epoch 23/200
200/200 - 11s - loss: 4.6479e-06 - val_loss: 2.8081e-05 - lr: 5.0000e-05 - 11s/epoch - 55ms/step
Epoch 24/200
200/200 - 11s - loss: 4.2579e-06 - val_loss: 3.7954e-05 - lr: 5.0000e-05 - 11s/epoch - 55ms/step
Epoch 25/200
200/200 - 11s - loss: 3.9628e-06 - val_loss: 2.6399e-05 - lr: 5.0000e-05 - 11s/epoch - 56ms/step
Epoch 26/200
200/200 - 11s - loss: 3.6915e-06 - val_loss: 1.9973e-05 - lr: 5.0000e-05 - 11s/epoch - 56ms/step
Epoch 27/200
200/200 - 11s - loss: 3.4726e-06 - val_loss: 3.5831e-05 - lr: 5.0000e-05 - 11s/epoch - 55ms/step
Epoch 28/200
200/200 - 11s - loss: 3.2110e-06 - val_loss: 2.7290e-05 - lr: 5.0000e-05 - 11s/epoch - 56ms/step
Epoch 29/200
200/200 - 11s - loss: 3.3421e-06 - val_loss: 3.1827e-05 - lr: 5.0000e-05 - 11s/epoch - 56ms/step
Epoch 30/200
200/200 - 11s - loss: 3.3472e-06 - val_loss: 3.4653e-05 - lr: 5.0000e-05 - 11s/epoch - 56ms/step
Epoch 31/200
Epoch 00031: ReduceLROnPlateau reducing learning rate to 2.499999936844688e-05.
200/200 - 11s - loss: 3.1221e-06 - val_loss: 2.7741e-05 - lr: 5.0000e-05 - 11s/epoch - 56ms/step
Epoch 32/200
200/200 - 11s - loss: 2.5739e-06 - val_loss: 3.2486e-05 - lr: 2.5000e-05 - 11s/epoch - 55ms/step
Epoch 33/200
200/200 - 11s - loss: 2.5589e-06 - val_loss: 3.3135e-05 - lr: 2.5000e-05 - 11s/epoch - 56ms/step
Epoch 34/200
200/200 - 11s - loss: 2.4215e-06 - val_loss: 2.8923e-05 - lr: 2.5000e-05 - 11s/epoch - 56ms/step
Epoch 35/200
200/200 - 11s - loss: 2.4033e-06 - val_loss: 2.8776e-05 - lr: 2.5000e-05 - 11s/epoch - 56ms/step
Epoch 36/200
200/200 - 11s - loss: 2.3358e-06 - val_loss: 2.5874e-05 - lr: 2.5000e-05 - 11s/epoch - 56ms/step
Epoch 37/200
200/200 - 11s - loss: 2.2922e-06 - val_loss: 3.6051e-05 - lr: 2.5000e-05 - 11s/epoch - 55ms/step
Epoch 38/200
Epoch 00038: ReduceLROnPlateau reducing learning rate to 1.249999968422344e-05.
200/200 - 11s - loss: 2.1278e-06 - val_loss: 2.4898e-05 - lr: 2.5000e-05 - 11s/epoch - 55ms/step
Epoch 39/200
200/200 - 11s - loss: 2.0474e-06 - val_loss: 2.8901e-05 - lr: 1.2500e-05 - 11s/epoch - 56ms/step
Epoch 40/200
200/200 - 11s - loss: 2.0612e-06 - val_loss: 3.7469e-05 - lr: 1.2500e-05 - 11s/epoch - 56ms/step
Epoch 41/200
200/200 - 11s - loss: 1.8414e-06 - val_loss: 2.8496e-05 - lr: 1.2500e-05 - 11s/epoch - 56ms/step
Epoch 42/200
200/200 - 11s - loss: 2.0196e-06 - val_loss: 3.5206e-05 - lr: 1.2500e-05 - 11s/epoch - 56ms/step
Epoch 43/200
200/200 - 11s - loss: 1.8551e-06 - val_loss: 2.6483e-05 - lr: 1.2500e-05 - 11s/epoch - 56ms/step
Epoch 44/200
200/200 - 11s - loss: 1.9705e-06 - val_loss: 2.4643e-05 - lr: 1.2500e-05 - 11s/epoch - 55ms/step
Epoch 45/200
Epoch 00045: ReduceLROnPlateau reducing learning rate to 6.24999984211172e-06.
200/200 - 11s - loss: 1.9136e-06 - val_loss: 2.8379e-05 - lr: 1.2500e-05 - 11s/epoch - 56ms/step
Epoch 46/200
200/200 - 11s - loss: 1.7911e-06 - val_loss: 4.0055e-05 - lr: 6.2500e-06 - 11s/epoch - 56ms/step
Epoch 00046: early stopping
INFO:sleap.nn.training:Finished training loop. [8.7 min]
INFO:sleap.nn.training:Deleting visualization directory: models/courtship.centroid/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 33.7 FPS31m51.9 FPS31m52.6 FPSFPS
?25hINFO:sleap.nn.evals:Saved predictions: models/courtship.centroid/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: models/courtship.centroid/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.725241
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 7.3 FPS0:00:01 184.6 FPSm
?25hINFO:sleap.nn.evals:Saved predictions: models/courtship.centroid/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: models/courtship.centroid/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.870526
Let’s now train a centered-instance model.
!sleap-train baseline_medium_rf.topdown.json "dataset/drosophila-melanogaster-courtship/courtship_labels.slp" --run_name "courtship.topdown_confmaps" --video-paths "dataset/drosophila-melanogaster-courtship/20190128_113421.mp4"
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.2
TensorFlow: 2.7.0
Numpy: 1.21.5
Python: 3.7.12
OS: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid
INFO:sleap.nn.training:Training labels file: dataset/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /home/talmolab/sleap-estimates-animal-poses/pull-requests/sleap/sleap/training_profiles/baseline_medium_rf.topdown.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
"training_job_path": "baseline_medium_rf.topdown.json",
"labels_path": "dataset/drosophila-melanogaster-courtship/courtship_labels.slp",
"video_paths": [
"dataset/drosophila-melanogaster-courtship/20190128_113421.mp4"
],
"val_labels": null,
"test_labels": null,
"base_checkpoint": null,
"tensorboard": false,
"save_viz": false,
"zmq": false,
"run_name": "courtship.topdown_confmaps",
"prefix": "",
"suffix": "",
"cpu": false,
"first_gpu": false,
"last_gpu": false,
"gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
"data": {
"labels": {
"training_labels": null,
"validation_labels": null,
"validation_fraction": 0.1,
"test_labels": null,
"split_by_inds": false,
"training_inds": null,
"validation_inds": null,
"test_inds": null,
"search_path_hints": [],
"skeletons": []
},
"preprocessing": {
"ensure_rgb": false,
"ensure_grayscale": false,
"imagenet_mode": null,
"input_scaling": 1.0,
"pad_to_stride": null,
"resize_and_pad_to_target": true,
"target_height": null,
"target_width": null
},
"instance_cropping": {
"center_on_part": null,
"crop_size": null,
"crop_size_detection_padding": 16
}
},
"model": {
"backbone": {
"leap": null,
"unet": {
"stem_stride": null,
"max_stride": 16,
"output_stride": 4,
"filters": 24,
"filters_rate": 2.0,
"middle_block": true,
"up_interpolate": true,
"stacks": 1
},
"hourglass": null,
"resnet": null,
"pretrained_encoder": null
},
"heads": {
"single_instance": null,
"centroid": null,
"centered_instance": {
"anchor_part": null,
"part_names": null,
"sigma": 2.5,
"output_stride": 4,
"loss_weight": 1.0,
"offset_refinement": false
},
"multi_instance": null,
"multi_class_bottomup": null,
"multi_class_topdown": null
},
"base_checkpoint": null
},
"optimization": {
"preload_data": true,
"augmentation_config": {
"rotate": true,
"rotation_min_angle": -15.0,
"rotation_max_angle": 15.0,
"translate": false,
"translate_min": -5,
"translate_max": 5,
"scale": false,
"scale_min": 0.9,
"scale_max": 1.1,
"uniform_noise": false,
"uniform_noise_min_val": 0.0,
"uniform_noise_max_val": 10.0,
"gaussian_noise": false,
"gaussian_noise_mean": 5.0,
"gaussian_noise_stddev": 1.0,
"contrast": false,
"contrast_min_gamma": 0.5,
"contrast_max_gamma": 2.0,
"brightness": false,
"brightness_min_val": 0.0,
"brightness_max_val": 10.0,
"random_crop": false,
"random_crop_height": 256,
"random_crop_width": 256,
"random_flip": false,
"flip_horizontal": true
},
"online_shuffling": true,
"shuffle_buffer_size": 128,
"prefetch": true,
"batch_size": 4,
"batches_per_epoch": null,
"min_batches_per_epoch": 200,
"val_batches_per_epoch": null,
"min_val_batches_per_epoch": 10,
"epochs": 200,
"optimizer": "adam",
"initial_learning_rate": 0.0001,
"learning_rate_schedule": {
"reduce_on_plateau": true,
"reduction_factor": 0.5,
"plateau_min_delta": 1e-08,
"plateau_patience": 5,
"plateau_cooldown": 3,
"min_learning_rate": 1e-08
},
"hard_keypoint_mining": {
"online_mining": false,
"hard_to_easy_ratio": 2.0,
"min_hard_keypoints": 2,
"max_hard_keypoints": null,
"loss_scale": 5.0
},
"early_stopping": {
"stop_training_on_plateau": true,
"plateau_min_delta": 1e-08,
"plateau_patience": 10
}
},
"outputs": {
"save_outputs": true,
"run_name": "courtship.topdown_confmaps",
"run_name_prefix": "",
"run_name_suffix": null,
"runs_folder": "models",
"tags": [],
"save_visualizations": true,
"delete_viz_images": true,
"zip_outputs": false,
"log_to_csv": true,
"checkpointing": {
"initial_model": false,
"best_model": true,
"every_epoch": false,
"latest_model": false,
"final_model": false
},
"tensorboard": {
"write_logs": false,
"loss_frequency": "epoch",
"architecture_graph": true,
"profile_graph": false,
"visualizations": true
},
"zmq": {
"subscribe_to_controller": false,
"controller_address": "tcp://127.0.0.1:9000",
"controller_polling_timeout": 10,
"publish_updates": false,
"publish_address": "tcp://127.0.0.1:9001"
}
},
"name": "",
"description": "",
"sleap_version": "1.3.2",
"filename": "/home/talmolab/sleap-estimates-animal-poses/pull-requests/sleap/sleap/training_profiles/baseline_medium_rf.topdown.json"
}
INFO:sleap.nn.training:
2023-09-01 13:39:43.324520: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:43.329181: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:43.329961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.training:Auto-selected GPU 0 with 23056 MiB of free memory.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: dataset/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training: Splits: Training = 134 / Validation = 15.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2023-09-01 13:39:44.254912: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-01 13:39:44.255468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:44.256291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:44.257158: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:44.546117: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:44.546866: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:44.547533: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:39:44.548184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21151 MB memory: -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:sleap.nn.training:Loaded test example. [1.684s]
INFO:sleap.nn.training: Input shape: (144, 144, 3)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training: Backbone: UNet(stacks=1, filters=24, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training: Max stride: 16
INFO:sleap.nn.training: Parameters: 4,311,877
INFO:sleap.nn.training: Heads:
INFO:sleap.nn.training: [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part=None, sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training: Outputs:
INFO:sleap.nn.training: [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 36, 36, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 134
INFO:sleap.nn.training:Validation set: n = 15
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training: Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-08, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training: Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.training:Created run path: models/courtship.topdown_confmaps
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [3.2s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [5.9s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/200
2023-09-01 13:39:54.940083: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2023-09-01 13:40:00.337645: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
200/200 - 8s - loss: 0.0108 - head: 0.0073 - thorax: 0.0067 - abdomen: 0.0111 - wingL: 0.0125 - wingR: 0.0126 - forelegL4: 0.0111 - forelegR4: 0.0108 - midlegL4: 0.0127 - midlegR4: 0.0128 - hindlegL4: 0.0131 - hindlegR4: 0.0131 - eyeL: 0.0082 - eyeR: 0.0083 - val_loss: 0.0087 - val_head: 0.0033 - val_thorax: 0.0039 - val_abdomen: 0.0089 - val_wingL: 0.0105 - val_wingR: 0.0106 - val_forelegL4: 0.0091 - val_forelegR4: 0.0091 - val_midlegL4: 0.0123 - val_midlegR4: 0.0116 - val_hindlegL4: 0.0128 - val_hindlegR4: 0.0116 - val_eyeL: 0.0045 - val_eyeR: 0.0045 - lr: 1.0000e-04 - 8s/epoch - 38ms/step
Epoch 2/200
200/200 - 4s - loss: 0.0064 - head: 0.0019 - thorax: 0.0029 - abdomen: 0.0057 - wingL: 0.0061 - wingR: 0.0073 - forelegL4: 0.0075 - forelegR4: 0.0078 - midlegL4: 0.0092 - midlegR4: 0.0092 - hindlegL4: 0.0099 - hindlegR4: 0.0102 - eyeL: 0.0025 - eyeR: 0.0025 - val_loss: 0.0061 - val_head: 0.0015 - val_thorax: 0.0024 - val_abdomen: 0.0049 - val_wingL: 0.0056 - val_wingR: 0.0078 - val_forelegL4: 0.0079 - val_forelegR4: 0.0067 - val_midlegL4: 0.0086 - val_midlegR4: 0.0089 - val_hindlegL4: 0.0093 - val_hindlegR4: 0.0081 - val_eyeL: 0.0037 - val_eyeR: 0.0032 - lr: 1.0000e-04 - 4s/epoch - 19ms/step
Epoch 3/200
200/200 - 3s - loss: 0.0048 - head: 8.9048e-04 - thorax: 0.0019 - abdomen: 0.0036 - wingL: 0.0041 - wingR: 0.0051 - forelegL4: 0.0063 - forelegR4: 0.0066 - midlegL4: 0.0076 - midlegR4: 0.0076 - hindlegL4: 0.0076 - hindlegR4: 0.0080 - eyeL: 0.0015 - eyeR: 0.0015 - val_loss: 0.0058 - val_head: 0.0014 - val_thorax: 0.0021 - val_abdomen: 0.0044 - val_wingL: 0.0051 - val_wingR: 0.0070 - val_forelegL4: 0.0072 - val_forelegR4: 0.0063 - val_midlegL4: 0.0088 - val_midlegR4: 0.0085 - val_hindlegL4: 0.0097 - val_hindlegR4: 0.0079 - val_eyeL: 0.0038 - val_eyeR: 0.0032 - lr: 1.0000e-04 - 3s/epoch - 16ms/step
Epoch 4/200
200/200 - 3s - loss: 0.0041 - head: 7.6417e-04 - thorax: 0.0015 - abdomen: 0.0028 - wingL: 0.0035 - wingR: 0.0041 - forelegL4: 0.0058 - forelegR4: 0.0060 - midlegL4: 0.0066 - midlegR4: 0.0064 - hindlegL4: 0.0066 - hindlegR4: 0.0070 - eyeL: 0.0013 - eyeR: 0.0012 - val_loss: 0.0048 - val_head: 7.6555e-04 - val_thorax: 0.0013 - val_abdomen: 0.0034 - val_wingL: 0.0042 - val_wingR: 0.0065 - val_forelegL4: 0.0063 - val_forelegR4: 0.0064 - val_midlegL4: 0.0069 - val_midlegR4: 0.0071 - val_hindlegL4: 0.0080 - val_hindlegR4: 0.0062 - val_eyeL: 0.0028 - val_eyeR: 0.0026 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 5/200
200/200 - 3s - loss: 0.0034 - head: 6.1233e-04 - thorax: 0.0012 - abdomen: 0.0023 - wingL: 0.0028 - wingR: 0.0032 - forelegL4: 0.0052 - forelegR4: 0.0054 - midlegL4: 0.0052 - midlegR4: 0.0051 - hindlegL4: 0.0057 - hindlegR4: 0.0058 - eyeL: 0.0011 - eyeR: 0.0011 - val_loss: 0.0044 - val_head: 9.3809e-04 - val_thorax: 0.0012 - val_abdomen: 0.0027 - val_wingL: 0.0032 - val_wingR: 0.0048 - val_forelegL4: 0.0062 - val_forelegR4: 0.0053 - val_midlegL4: 0.0068 - val_midlegR4: 0.0063 - val_hindlegL4: 0.0067 - val_hindlegR4: 0.0065 - val_eyeL: 0.0035 - val_eyeR: 0.0032 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 6/200
200/200 - 3s - loss: 0.0028 - head: 5.5957e-04 - thorax: 9.3519e-04 - abdomen: 0.0019 - wingL: 0.0023 - wingR: 0.0025 - forelegL4: 0.0045 - forelegR4: 0.0045 - midlegL4: 0.0040 - midlegR4: 0.0040 - hindlegL4: 0.0047 - hindlegR4: 0.0048 - eyeL: 0.0010 - eyeR: 9.7287e-04 - val_loss: 0.0038 - val_head: 7.6837e-04 - val_thorax: 9.9723e-04 - val_abdomen: 0.0027 - val_wingL: 0.0025 - val_wingR: 0.0046 - val_forelegL4: 0.0058 - val_forelegR4: 0.0049 - val_midlegL4: 0.0054 - val_midlegR4: 0.0058 - val_hindlegL4: 0.0057 - val_hindlegR4: 0.0065 - val_eyeL: 0.0023 - val_eyeR: 0.0022 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 7/200
200/200 - 3s - loss: 0.0024 - head: 4.7941e-04 - thorax: 7.5772e-04 - abdomen: 0.0017 - wingL: 0.0020 - wingR: 0.0022 - forelegL4: 0.0039 - forelegR4: 0.0041 - midlegL4: 0.0033 - midlegR4: 0.0033 - hindlegL4: 0.0039 - hindlegR4: 0.0040 - eyeL: 9.3055e-04 - eyeR: 8.9191e-04 - val_loss: 0.0036 - val_head: 6.1078e-04 - val_thorax: 0.0010 - val_abdomen: 0.0023 - val_wingL: 0.0025 - val_wingR: 0.0039 - val_forelegL4: 0.0053 - val_forelegR4: 0.0058 - val_midlegL4: 0.0049 - val_midlegR4: 0.0056 - val_hindlegL4: 0.0054 - val_hindlegR4: 0.0049 - val_eyeL: 0.0026 - val_eyeR: 0.0024 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 8/200
200/200 - 3s - loss: 0.0020 - head: 4.4425e-04 - thorax: 6.8283e-04 - abdomen: 0.0014 - wingL: 0.0015 - wingR: 0.0017 - forelegL4: 0.0035 - forelegR4: 0.0035 - midlegL4: 0.0027 - midlegR4: 0.0026 - hindlegL4: 0.0033 - hindlegR4: 0.0033 - eyeL: 7.7111e-04 - eyeR: 7.2022e-04 - val_loss: 0.0035 - val_head: 7.1555e-04 - val_thorax: 9.1508e-04 - val_abdomen: 0.0022 - val_wingL: 0.0023 - val_wingR: 0.0033 - val_forelegL4: 0.0054 - val_forelegR4: 0.0049 - val_midlegL4: 0.0049 - val_midlegR4: 0.0052 - val_hindlegL4: 0.0052 - val_hindlegR4: 0.0051 - val_eyeL: 0.0025 - val_eyeR: 0.0025 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 9/200
200/200 - 3s - loss: 0.0017 - head: 3.8990e-04 - thorax: 5.4963e-04 - abdomen: 0.0012 - wingL: 0.0012 - wingR: 0.0014 - forelegL4: 0.0030 - forelegR4: 0.0031 - midlegL4: 0.0022 - midlegR4: 0.0022 - hindlegL4: 0.0027 - hindlegR4: 0.0027 - eyeL: 6.9041e-04 - eyeR: 6.7679e-04 - val_loss: 0.0034 - val_head: 5.6666e-04 - val_thorax: 7.9156e-04 - val_abdomen: 0.0023 - val_wingL: 0.0020 - val_wingR: 0.0041 - val_forelegL4: 0.0043 - val_forelegR4: 0.0048 - val_midlegL4: 0.0041 - val_midlegR4: 0.0051 - val_hindlegL4: 0.0053 - val_hindlegR4: 0.0052 - val_eyeL: 0.0024 - val_eyeR: 0.0026 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 10/200
200/200 - 3s - loss: 0.0015 - head: 3.6281e-04 - thorax: 5.2471e-04 - abdomen: 0.0010 - wingL: 0.0011 - wingR: 0.0012 - forelegL4: 0.0027 - forelegR4: 0.0028 - midlegL4: 0.0019 - midlegR4: 0.0019 - hindlegL4: 0.0023 - hindlegR4: 0.0024 - eyeL: 7.0986e-04 - eyeR: 6.9581e-04 - val_loss: 0.0024 - val_head: 4.8376e-04 - val_thorax: 6.2502e-04 - val_abdomen: 0.0016 - val_wingL: 0.0014 - val_wingR: 0.0027 - val_forelegL4: 0.0035 - val_forelegR4: 0.0033 - val_midlegL4: 0.0028 - val_midlegR4: 0.0041 - val_hindlegL4: 0.0036 - val_hindlegR4: 0.0038 - val_eyeL: 0.0015 - val_eyeR: 0.0016 - lr: 1.0000e-04 - 3s/epoch - 16ms/step
Epoch 11/200
200/200 - 3s - loss: 0.0013 - head: 3.1183e-04 - thorax: 4.7891e-04 - abdomen: 9.4567e-04 - wingL: 9.6811e-04 - wingR: 0.0011 - forelegL4: 0.0023 - forelegR4: 0.0025 - midlegL4: 0.0016 - midlegR4: 0.0016 - hindlegL4: 0.0020 - hindlegR4: 0.0021 - eyeL: 5.7635e-04 - eyeR: 5.3648e-04 - val_loss: 0.0028 - val_head: 5.2940e-04 - val_thorax: 6.6554e-04 - val_abdomen: 0.0020 - val_wingL: 0.0013 - val_wingR: 0.0024 - val_forelegL4: 0.0041 - val_forelegR4: 0.0041 - val_midlegL4: 0.0034 - val_midlegR4: 0.0042 - val_hindlegL4: 0.0047 - val_hindlegR4: 0.0040 - val_eyeL: 0.0025 - val_eyeR: 0.0022 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 12/200
200/200 - 3s - loss: 0.0011 - head: 2.8863e-04 - thorax: 4.2604e-04 - abdomen: 8.0488e-04 - wingL: 8.1238e-04 - wingR: 8.5798e-04 - forelegL4: 0.0021 - forelegR4: 0.0021 - midlegL4: 0.0014 - midlegR4: 0.0014 - hindlegL4: 0.0017 - hindlegR4: 0.0018 - eyeL: 5.1007e-04 - eyeR: 4.5654e-04 - val_loss: 0.0031 - val_head: 8.1802e-04 - val_thorax: 7.9789e-04 - val_abdomen: 0.0018 - val_wingL: 0.0014 - val_wingR: 0.0028 - val_forelegL4: 0.0040 - val_forelegR4: 0.0048 - val_midlegL4: 0.0057 - val_midlegR4: 0.0037 - val_hindlegL4: 0.0053 - val_hindlegR4: 0.0050 - val_eyeL: 0.0020 - val_eyeR: 0.0018 - lr: 1.0000e-04 - 3s/epoch - 14ms/step
Epoch 13/200
200/200 - 3s - loss: 0.0010 - head: 2.8818e-04 - thorax: 4.1018e-04 - abdomen: 7.8027e-04 - wingL: 7.8017e-04 - wingR: 8.4529e-04 - forelegL4: 0.0019 - forelegR4: 0.0019 - midlegL4: 0.0013 - midlegR4: 0.0013 - hindlegL4: 0.0015 - hindlegR4: 0.0016 - eyeL: 4.6272e-04 - eyeR: 4.3265e-04 - val_loss: 0.0026 - val_head: 3.5806e-04 - val_thorax: 6.6352e-04 - val_abdomen: 0.0017 - val_wingL: 0.0015 - val_wingR: 0.0037 - val_forelegL4: 0.0036 - val_forelegR4: 0.0042 - val_midlegL4: 0.0034 - val_midlegR4: 0.0032 - val_hindlegL4: 0.0041 - val_hindlegR4: 0.0047 - val_eyeL: 0.0013 - val_eyeR: 0.0013 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 14/200
200/200 - 3s - loss: 9.4029e-04 - head: 2.8339e-04 - thorax: 3.6739e-04 - abdomen: 7.0118e-04 - wingL: 7.4831e-04 - wingR: 7.1158e-04 - forelegL4: 0.0017 - forelegR4: 0.0017 - midlegL4: 0.0012 - midlegR4: 0.0011 - hindlegL4: 0.0014 - hindlegR4: 0.0015 - eyeL: 4.2793e-04 - eyeR: 4.1400e-04 - val_loss: 0.0024 - val_head: 3.4292e-04 - val_thorax: 7.1119e-04 - val_abdomen: 0.0014 - val_wingL: 0.0013 - val_wingR: 0.0028 - val_forelegL4: 0.0030 - val_forelegR4: 0.0043 - val_midlegL4: 0.0031 - val_midlegR4: 0.0030 - val_hindlegL4: 0.0039 - val_hindlegR4: 0.0038 - val_eyeL: 0.0017 - val_eyeR: 0.0015 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 15/200
200/200 - 3s - loss: 7.8295e-04 - head: 2.3028e-04 - thorax: 3.3006e-04 - abdomen: 5.9391e-04 - wingL: 5.8825e-04 - wingR: 6.0989e-04 - forelegL4: 0.0015 - forelegR4: 0.0015 - midlegL4: 9.6945e-04 - midlegR4: 9.3611e-04 - hindlegL4: 0.0011 - hindlegR4: 0.0012 - eyeL: 3.4493e-04 - eyeR: 3.1164e-04 - val_loss: 0.0019 - val_head: 4.4152e-04 - val_thorax: 5.4500e-04 - val_abdomen: 0.0013 - val_wingL: 0.0012 - val_wingR: 0.0026 - val_forelegL4: 0.0024 - val_forelegR4: 0.0037 - val_midlegL4: 0.0024 - val_midlegR4: 0.0024 - val_hindlegL4: 0.0030 - val_hindlegR4: 0.0030 - val_eyeL: 0.0011 - val_eyeR: 0.0011 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 16/200
200/200 - 3s - loss: 7.3208e-04 - head: 2.3573e-04 - thorax: 3.0631e-04 - abdomen: 5.5007e-04 - wingL: 5.3431e-04 - wingR: 5.9773e-04 - forelegL4: 0.0013 - forelegR4: 0.0014 - midlegL4: 9.1004e-04 - midlegR4: 8.7803e-04 - hindlegL4: 0.0010 - hindlegR4: 0.0011 - eyeL: 3.3279e-04 - eyeR: 2.9841e-04 - val_loss: 0.0023 - val_head: 3.5381e-04 - val_thorax: 7.0128e-04 - val_abdomen: 0.0015 - val_wingL: 0.0013 - val_wingR: 0.0022 - val_forelegL4: 0.0031 - val_forelegR4: 0.0041 - val_midlegL4: 0.0033 - val_midlegR4: 0.0028 - val_hindlegL4: 0.0036 - val_hindlegR4: 0.0033 - val_eyeL: 0.0017 - val_eyeR: 0.0014 - lr: 1.0000e-04 - 3s/epoch - 14ms/step
Epoch 17/200
200/200 - 3s - loss: 6.3161e-04 - head: 2.0100e-04 - thorax: 2.8088e-04 - abdomen: 4.9153e-04 - wingL: 4.7586e-04 - wingR: 4.9866e-04 - forelegL4: 0.0011 - forelegR4: 0.0012 - midlegL4: 7.6100e-04 - midlegR4: 8.0266e-04 - hindlegL4: 8.9697e-04 - hindlegR4: 8.9149e-04 - eyeL: 2.8189e-04 - eyeR: 2.7208e-04 - val_loss: 0.0018 - val_head: 2.8070e-04 - val_thorax: 5.1903e-04 - val_abdomen: 0.0011 - val_wingL: 9.8509e-04 - val_wingR: 0.0025 - val_forelegL4: 0.0022 - val_forelegR4: 0.0026 - val_midlegL4: 0.0025 - val_midlegR4: 0.0021 - val_hindlegL4: 0.0031 - val_hindlegR4: 0.0031 - val_eyeL: 0.0011 - val_eyeR: 9.7838e-04 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 18/200
200/200 - 3s - loss: 5.7844e-04 - head: 1.9896e-04 - thorax: 2.9112e-04 - abdomen: 4.7495e-04 - wingL: 4.5591e-04 - wingR: 4.5877e-04 - forelegL4: 0.0011 - forelegR4: 0.0012 - midlegL4: 6.9042e-04 - midlegR4: 6.6195e-04 - hindlegL4: 7.9452e-04 - hindlegR4: 7.6819e-04 - eyeL: 2.5989e-04 - eyeR: 2.4763e-04 - val_loss: 0.0018 - val_head: 3.1925e-04 - val_thorax: 6.0394e-04 - val_abdomen: 0.0012 - val_wingL: 9.0835e-04 - val_wingR: 0.0019 - val_forelegL4: 0.0022 - val_forelegR4: 0.0029 - val_midlegL4: 0.0026 - val_midlegR4: 0.0024 - val_hindlegL4: 0.0033 - val_hindlegR4: 0.0022 - val_eyeL: 0.0015 - val_eyeR: 0.0011 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 19/200
200/200 - 3s - loss: 5.1323e-04 - head: 1.8346e-04 - thorax: 2.5475e-04 - abdomen: 4.2159e-04 - wingL: 4.3027e-04 - wingR: 3.9814e-04 - forelegL4: 9.5814e-04 - forelegR4: 9.9765e-04 - midlegL4: 5.9968e-04 - midlegR4: 5.8423e-04 - hindlegL4: 6.7869e-04 - hindlegR4: 6.9121e-04 - eyeL: 2.4343e-04 - eyeR: 2.3077e-04 - val_loss: 0.0021 - val_head: 3.3346e-04 - val_thorax: 5.9007e-04 - val_abdomen: 0.0014 - val_wingL: 0.0013 - val_wingR: 0.0031 - val_forelegL4: 0.0026 - val_forelegR4: 0.0036 - val_midlegL4: 0.0029 - val_midlegR4: 0.0021 - val_hindlegL4: 0.0037 - val_hindlegR4: 0.0036 - val_eyeL: 0.0011 - val_eyeR: 9.4254e-04 - lr: 1.0000e-04 - 3s/epoch - 14ms/step
Epoch 20/200
200/200 - 3s - loss: 4.7991e-04 - head: 1.7328e-04 - thorax: 2.2397e-04 - abdomen: 4.2417e-04 - wingL: 3.9313e-04 - wingR: 3.9871e-04 - forelegL4: 8.8547e-04 - forelegR4: 8.9704e-04 - midlegL4: 5.3515e-04 - midlegR4: 5.8294e-04 - hindlegL4: 6.5212e-04 - hindlegR4: 6.2828e-04 - eyeL: 2.2438e-04 - eyeR: 2.2012e-04 - val_loss: 0.0014 - val_head: 2.7034e-04 - val_thorax: 4.7978e-04 - val_abdomen: 9.7903e-04 - val_wingL: 8.6477e-04 - val_wingR: 0.0020 - val_forelegL4: 0.0018 - val_forelegR4: 0.0024 - val_midlegL4: 0.0019 - val_midlegR4: 0.0018 - val_hindlegL4: 0.0024 - val_hindlegR4: 0.0022 - val_eyeL: 9.9423e-04 - val_eyeR: 8.4541e-04 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 21/200
200/200 - 3s - loss: 4.4100e-04 - head: 1.6076e-04 - thorax: 2.4080e-04 - abdomen: 3.8343e-04 - wingL: 3.6759e-04 - wingR: 3.7489e-04 - forelegL4: 8.1060e-04 - forelegR4: 8.1600e-04 - midlegL4: 4.7288e-04 - midlegR4: 5.2695e-04 - hindlegL4: 5.6401e-04 - hindlegR4: 6.3519e-04 - eyeL: 1.9033e-04 - eyeR: 1.8954e-04 - val_loss: 0.0018 - val_head: 2.5764e-04 - val_thorax: 5.8718e-04 - val_abdomen: 0.0011 - val_wingL: 9.6939e-04 - val_wingR: 0.0019 - val_forelegL4: 0.0022 - val_forelegR4: 0.0026 - val_midlegL4: 0.0025 - val_midlegR4: 0.0026 - val_hindlegL4: 0.0032 - val_hindlegR4: 0.0028 - val_eyeL: 0.0014 - val_eyeR: 0.0011 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 22/200
200/200 - 3s - loss: 3.7738e-04 - head: 1.4725e-04 - thorax: 2.0905e-04 - abdomen: 3.2447e-04 - wingL: 3.2224e-04 - wingR: 3.0585e-04 - forelegL4: 6.2169e-04 - forelegR4: 6.7379e-04 - midlegL4: 4.5061e-04 - midlegR4: 4.3931e-04 - hindlegL4: 5.1129e-04 - hindlegR4: 5.2449e-04 - eyeL: 1.9372e-04 - eyeR: 1.8213e-04 - val_loss: 0.0015 - val_head: 2.2947e-04 - val_thorax: 5.4640e-04 - val_abdomen: 9.8293e-04 - val_wingL: 8.6663e-04 - val_wingR: 0.0013 - val_forelegL4: 0.0018 - val_forelegR4: 0.0027 - val_midlegL4: 0.0021 - val_midlegR4: 0.0019 - val_hindlegL4: 0.0027 - val_hindlegR4: 0.0022 - val_eyeL: 0.0013 - val_eyeR: 0.0010 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 23/200
200/200 - 3s - loss: 3.6084e-04 - head: 1.4440e-04 - thorax: 2.0277e-04 - abdomen: 3.0561e-04 - wingL: 3.0192e-04 - wingR: 2.8845e-04 - forelegL4: 6.3221e-04 - forelegR4: 6.7722e-04 - midlegL4: 3.9143e-04 - midlegR4: 4.3545e-04 - hindlegL4: 5.1985e-04 - hindlegR4: 4.5058e-04 - eyeL: 1.7636e-04 - eyeR: 1.6468e-04 - val_loss: 0.0015 - val_head: 2.9639e-04 - val_thorax: 4.6412e-04 - val_abdomen: 0.0011 - val_wingL: 9.0466e-04 - val_wingR: 0.0021 - val_forelegL4: 0.0015 - val_forelegR4: 0.0025 - val_midlegL4: 0.0018 - val_midlegR4: 0.0016 - val_hindlegL4: 0.0029 - val_hindlegR4: 0.0022 - val_eyeL: 8.7357e-04 - val_eyeR: 7.0067e-04 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 24/200
200/200 - 3s - loss: 3.4886e-04 - head: 1.4382e-04 - thorax: 1.9157e-04 - abdomen: 3.2551e-04 - wingL: 3.0634e-04 - wingR: 3.0727e-04 - forelegL4: 6.3863e-04 - forelegR4: 6.0904e-04 - midlegL4: 3.5949e-04 - midlegR4: 4.1201e-04 - hindlegL4: 4.2893e-04 - hindlegR4: 4.8121e-04 - eyeL: 1.6669e-04 - eyeR: 1.6464e-04 - val_loss: 0.0022 - val_head: 3.2159e-04 - val_thorax: 7.2743e-04 - val_abdomen: 0.0014 - val_wingL: 0.0011 - val_wingR: 0.0027 - val_forelegL4: 0.0025 - val_forelegR4: 0.0037 - val_midlegL4: 0.0033 - val_midlegR4: 0.0020 - val_hindlegL4: 0.0043 - val_hindlegR4: 0.0031 - val_eyeL: 0.0017 - val_eyeR: 0.0012 - lr: 1.0000e-04 - 3s/epoch - 14ms/step
Epoch 25/200
Epoch 00025: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05.
200/200 - 3s - loss: 3.0444e-04 - head: 1.2563e-04 - thorax: 1.7247e-04 - abdomen: 2.6934e-04 - wingL: 2.5754e-04 - wingR: 2.4728e-04 - forelegL4: 5.8390e-04 - forelegR4: 5.3959e-04 - midlegL4: 3.3003e-04 - midlegR4: 3.6432e-04 - hindlegL4: 4.0270e-04 - hindlegR4: 3.5518e-04 - eyeL: 1.5609e-04 - eyeR: 1.5365e-04 - val_loss: 0.0017 - val_head: 2.5420e-04 - val_thorax: 5.5809e-04 - val_abdomen: 0.0011 - val_wingL: 9.6708e-04 - val_wingR: 0.0022 - val_forelegL4: 0.0018 - val_forelegR4: 0.0033 - val_midlegL4: 0.0025 - val_midlegR4: 0.0017 - val_hindlegL4: 0.0031 - val_hindlegR4: 0.0031 - val_eyeL: 9.8718e-04 - val_eyeR: 8.0263e-04 - lr: 1.0000e-04 - 3s/epoch - 15ms/step
Epoch 26/200
200/200 - 3s - loss: 2.3368e-04 - head: 1.1149e-04 - thorax: 1.5177e-04 - abdomen: 2.1763e-04 - wingL: 2.2159e-04 - wingR: 1.9396e-04 - forelegL4: 3.8234e-04 - forelegR4: 3.8248e-04 - midlegL4: 2.7555e-04 - midlegR4: 2.8653e-04 - hindlegL4: 2.7842e-04 - hindlegR4: 2.8074e-04 - eyeL: 1.3157e-04 - eyeR: 1.2374e-04 - val_loss: 0.0017 - val_head: 2.1815e-04 - val_thorax: 5.0063e-04 - val_abdomen: 0.0011 - val_wingL: 8.2248e-04 - val_wingR: 0.0020 - val_forelegL4: 0.0019 - val_forelegR4: 0.0035 - val_midlegL4: 0.0022 - val_midlegR4: 0.0016 - val_hindlegL4: 0.0031 - val_hindlegR4: 0.0022 - val_eyeL: 0.0013 - val_eyeR: 9.8071e-04 - lr: 5.0000e-05 - 3s/epoch - 14ms/step
Epoch 27/200
200/200 - 3s - loss: 2.0711e-04 - head: 9.7513e-05 - thorax: 1.4018e-04 - abdomen: 2.0210e-04 - wingL: 1.8693e-04 - wingR: 1.7399e-04 - forelegL4: 3.1753e-04 - forelegR4: 3.7613e-04 - midlegL4: 2.2838e-04 - midlegR4: 2.4643e-04 - hindlegL4: 2.4471e-04 - hindlegR4: 2.4706e-04 - eyeL: 1.1696e-04 - eyeR: 1.1452e-04 - val_loss: 0.0011 - val_head: 1.7855e-04 - val_thorax: 3.7885e-04 - val_abdomen: 7.0074e-04 - val_wingL: 6.4821e-04 - val_wingR: 0.0012 - val_forelegL4: 0.0012 - val_forelegR4: 0.0017 - val_midlegL4: 0.0014 - val_midlegR4: 0.0013 - val_hindlegL4: 0.0019 - val_hindlegR4: 0.0018 - val_eyeL: 8.8941e-04 - val_eyeR: 7.0606e-04 - lr: 5.0000e-05 - 3s/epoch - 15ms/step
Epoch 28/200
200/200 - 3s - loss: 1.9539e-04 - head: 9.4716e-05 - thorax: 1.3617e-04 - abdomen: 1.8547e-04 - wingL: 1.8173e-04 - wingR: 1.6716e-04 - forelegL4: 3.2783e-04 - forelegR4: 3.1060e-04 - midlegL4: 2.2172e-04 - midlegR4: 2.2648e-04 - hindlegL4: 2.3846e-04 - hindlegR4: 2.2823e-04 - eyeL: 1.1204e-04 - eyeR: 1.0944e-04 - val_loss: 0.0012 - val_head: 1.9505e-04 - val_thorax: 3.8105e-04 - val_abdomen: 7.7888e-04 - val_wingL: 6.8985e-04 - val_wingR: 0.0016 - val_forelegL4: 0.0015 - val_forelegR4: 0.0020 - val_midlegL4: 0.0017 - val_midlegR4: 0.0011 - val_hindlegL4: 0.0022 - val_hindlegR4: 0.0019 - val_eyeL: 9.1223e-04 - val_eyeR: 7.0778e-04 - lr: 5.0000e-05 - 3s/epoch - 15ms/step
Epoch 29/200
200/200 - 3s - loss: 1.8262e-04 - head: 9.2364e-05 - thorax: 1.3126e-04 - abdomen: 1.7625e-04 - wingL: 1.7494e-04 - wingR: 1.5998e-04 - forelegL4: 3.0159e-04 - forelegR4: 2.9470e-04 - midlegL4: 1.9773e-04 - midlegR4: 2.0446e-04 - hindlegL4: 2.0576e-04 - hindlegR4: 2.1560e-04 - eyeL: 1.1218e-04 - eyeR: 1.0720e-04 - val_loss: 0.0015 - val_head: 2.2535e-04 - val_thorax: 4.8031e-04 - val_abdomen: 9.5428e-04 - val_wingL: 7.7468e-04 - val_wingR: 0.0016 - val_forelegL4: 0.0017 - val_forelegR4: 0.0025 - val_midlegL4: 0.0021 - val_midlegR4: 0.0018 - val_hindlegL4: 0.0029 - val_hindlegR4: 0.0019 - val_eyeL: 0.0013 - val_eyeR: 9.6936e-04 - lr: 5.0000e-05 - 3s/epoch - 15ms/step
Epoch 30/200
200/200 - 3s - loss: 1.7461e-04 - head: 8.9617e-05 - thorax: 1.2428e-04 - abdomen: 1.7234e-04 - wingL: 1.6780e-04 - wingR: 1.5580e-04 - forelegL4: 2.7324e-04 - forelegR4: 2.8042e-04 - midlegL4: 1.9090e-04 - midlegR4: 2.0420e-04 - hindlegL4: 1.9914e-04 - hindlegR4: 2.0318e-04 - eyeL: 1.0518e-04 - eyeR: 1.0386e-04 - val_loss: 0.0015 - val_head: 1.9058e-04 - val_thorax: 4.9603e-04 - val_abdomen: 0.0011 - val_wingL: 9.7566e-04 - val_wingR: 0.0018 - val_forelegL4: 0.0016 - val_forelegR4: 0.0028 - val_midlegL4: 0.0022 - val_midlegR4: 0.0015 - val_hindlegL4: 0.0028 - val_hindlegR4: 0.0028 - val_eyeL: 9.9699e-04 - val_eyeR: 8.3721e-04 - lr: 5.0000e-05 - 3s/epoch - 15ms/step
Epoch 31/200
200/200 - 3s - loss: 1.7064e-04 - head: 8.7373e-05 - thorax: 1.2365e-04 - abdomen: 1.6765e-04 - wingL: 1.5656e-04 - wingR: 1.4505e-04 - forelegL4: 2.7352e-04 - forelegR4: 2.6274e-04 - midlegL4: 1.9639e-04 - midlegR4: 1.9628e-04 - hindlegL4: 2.0323e-04 - hindlegR4: 1.9917e-04 - eyeL: 1.0639e-04 - eyeR: 1.0032e-04 - val_loss: 0.0011 - val_head: 1.7938e-04 - val_thorax: 3.6727e-04 - val_abdomen: 7.7820e-04 - val_wingL: 6.4437e-04 - val_wingR: 0.0014 - val_forelegL4: 0.0014 - val_forelegR4: 0.0020 - val_midlegL4: 0.0016 - val_midlegR4: 0.0010 - val_hindlegL4: 0.0021 - val_hindlegR4: 0.0016 - val_eyeL: 8.0607e-04 - val_eyeR: 6.6172e-04 - lr: 5.0000e-05 - 3s/epoch - 16ms/step
Epoch 32/200
Epoch 00032: ReduceLROnPlateau reducing learning rate to 2.499999936844688e-05.
200/200 - 4s - loss: 1.6547e-04 - head: 8.6407e-05 - thorax: 1.1578e-04 - abdomen: 1.6160e-04 - wingL: 1.5752e-04 - wingR: 1.4326e-04 - forelegL4: 2.5855e-04 - forelegR4: 2.8317e-04 - midlegL4: 1.7880e-04 - midlegR4: 1.8021e-04 - hindlegL4: 1.9743e-04 - hindlegR4: 1.8831e-04 - eyeL: 1.0074e-04 - eyeR: 9.9381e-05 - val_loss: 0.0012 - val_head: 1.9257e-04 - val_thorax: 3.7361e-04 - val_abdomen: 7.0451e-04 - val_wingL: 7.8240e-04 - val_wingR: 0.0015 - val_forelegL4: 0.0014 - val_forelegR4: 0.0020 - val_midlegL4: 0.0016 - val_midlegR4: 0.0011 - val_hindlegL4: 0.0020 - val_hindlegR4: 0.0019 - val_eyeL: 8.9328e-04 - val_eyeR: 7.3886e-04 - lr: 5.0000e-05 - 4s/epoch - 18ms/step
Epoch 33/200
200/200 - 3s - loss: 1.4767e-04 - head: 8.0575e-05 - thorax: 1.1097e-04 - abdomen: 1.4927e-04 - wingL: 1.4112e-04 - wingR: 1.3113e-04 - forelegL4: 2.1913e-04 - forelegR4: 2.1998e-04 - midlegL4: 1.6045e-04 - midlegR4: 1.6535e-04 - hindlegL4: 1.8091e-04 - hindlegR4: 1.7343e-04 - eyeL: 9.5387e-05 - eyeR: 9.2035e-05 - val_loss: 0.0014 - val_head: 1.9046e-04 - val_thorax: 4.6921e-04 - val_abdomen: 9.4087e-04 - val_wingL: 7.5647e-04 - val_wingR: 0.0015 - val_forelegL4: 0.0015 - val_forelegR4: 0.0025 - val_midlegL4: 0.0020 - val_midlegR4: 0.0015 - val_hindlegL4: 0.0026 - val_hindlegR4: 0.0021 - val_eyeL: 0.0013 - val_eyeR: 0.0010 - lr: 2.5000e-05 - 3s/epoch - 16ms/step
Epoch 34/200
200/200 - 3s - loss: 1.4506e-04 - head: 7.9790e-05 - thorax: 1.0771e-04 - abdomen: 1.5052e-04 - wingL: 1.4143e-04 - wingR: 1.2485e-04 - forelegL4: 2.2486e-04 - forelegR4: 2.1619e-04 - midlegL4: 1.6584e-04 - midlegR4: 1.6250e-04 - hindlegL4: 1.6521e-04 - hindlegR4: 1.6717e-04 - eyeL: 9.1550e-05 - eyeR: 8.8112e-05 - val_loss: 0.0013 - val_head: 1.8689e-04 - val_thorax: 3.7203e-04 - val_abdomen: 9.3770e-04 - val_wingL: 7.0190e-04 - val_wingR: 0.0019 - val_forelegL4: 0.0015 - val_forelegR4: 0.0023 - val_midlegL4: 0.0016 - val_midlegR4: 0.0012 - val_hindlegL4: 0.0025 - val_hindlegR4: 0.0022 - val_eyeL: 8.0213e-04 - val_eyeR: 6.5036e-04 - lr: 2.5000e-05 - 3s/epoch - 15ms/step
Epoch 35/200
200/200 - 3s - loss: 1.3911e-04 - head: 7.9674e-05 - thorax: 1.0668e-04 - abdomen: 1.4330e-04 - wingL: 1.3906e-04 - wingR: 1.2752e-04 - forelegL4: 1.9657e-04 - forelegR4: 1.9577e-04 - midlegL4: 1.5228e-04 - midlegR4: 1.5642e-04 - hindlegL4: 1.6610e-04 - hindlegR4: 1.6394e-04 - eyeL: 9.1523e-05 - eyeR: 8.9620e-05 - val_loss: 0.0013 - val_head: 1.7511e-04 - val_thorax: 4.2162e-04 - val_abdomen: 9.5009e-04 - val_wingL: 6.7908e-04 - val_wingR: 0.0013 - val_forelegL4: 0.0015 - val_forelegR4: 0.0023 - val_midlegL4: 0.0018 - val_midlegR4: 0.0014 - val_hindlegL4: 0.0027 - val_hindlegR4: 0.0019 - val_eyeL: 0.0012 - val_eyeR: 9.8818e-04 - lr: 2.5000e-05 - 3s/epoch - 16ms/step
Epoch 36/200
200/200 - 3s - loss: 1.3697e-04 - head: 7.5207e-05 - thorax: 1.0507e-04 - abdomen: 1.3913e-04 - wingL: 1.3497e-04 - wingR: 1.2511e-04 - forelegL4: 1.9152e-04 - forelegR4: 2.0264e-04 - midlegL4: 1.5207e-04 - midlegR4: 1.5519e-04 - hindlegL4: 1.6368e-04 - hindlegR4: 1.5869e-04 - eyeL: 9.0233e-05 - eyeR: 8.7055e-05 - val_loss: 0.0013 - val_head: 1.8066e-04 - val_thorax: 4.6591e-04 - val_abdomen: 9.9582e-04 - val_wingL: 7.2600e-04 - val_wingR: 0.0012 - val_forelegL4: 0.0015 - val_forelegR4: 0.0022 - val_midlegL4: 0.0019 - val_midlegR4: 0.0015 - val_hindlegL4: 0.0028 - val_hindlegR4: 0.0018 - val_eyeL: 0.0012 - val_eyeR: 9.6224e-04 - lr: 2.5000e-05 - 3s/epoch - 15ms/step
Epoch 37/200
200/200 - 3s - loss: 1.3638e-04 - head: 7.6822e-05 - thorax: 1.0531e-04 - abdomen: 1.4107e-04 - wingL: 1.4047e-04 - wingR: 1.2177e-04 - forelegL4: 1.9564e-04 - forelegR4: 1.7970e-04 - midlegL4: 1.5364e-04 - midlegR4: 1.5089e-04 - hindlegL4: 1.6647e-04 - hindlegR4: 1.6322e-04 - eyeL: 9.0198e-05 - eyeR: 8.7722e-05 - val_loss: 0.0017 - val_head: 2.3218e-04 - val_thorax: 5.3881e-04 - val_abdomen: 0.0011 - val_wingL: 0.0010 - val_wingR: 0.0019 - val_forelegL4: 0.0021 - val_forelegR4: 0.0028 - val_midlegL4: 0.0025 - val_midlegR4: 0.0016 - val_hindlegL4: 0.0033 - val_hindlegR4: 0.0029 - val_eyeL: 0.0015 - val_eyeR: 0.0012 - lr: 2.5000e-05 - 3s/epoch - 16ms/step
Epoch 00037: early stopping
INFO:sleap.nn.training:Finished training loop. [2.0 min]
INFO:sleap.nn.training:Deleting visualization directory: models/courtship.topdown_confmaps/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 39.3 FPS31m48.8 FPS31m49.5 FPSFPS
?25hINFO:sleap.nn.evals:Saved predictions: models/courtship.topdown_confmaps/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: models/courtship.topdown_confmaps/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.899237
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 14.2 FPS0:00:01 270.2 FPSm
?25hINFO:sleap.nn.evals:Saved predictions: models/courtship.topdown_confmaps/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: models/courtship.topdown_confmaps/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.691378
The models (along with the profiles and ground truth data used to train and validate the model) are saved in the models/ directory:
!tree models/
models/
├── courtship.centroid
│ ├── best_model.h5
│ ├── initial_config.json
│ ├── labels_gt.train.slp
│ ├── labels_gt.val.slp
│ ├── labels_pr.train.slp
│ ├── labels_pr.val.slp
│ ├── metrics.train.npz
│ ├── metrics.val.npz
│ ├── training_config.json
│ └── training_log.csv
└── courtship.topdown_confmaps
├── best_model.h5
├── initial_config.json
├── labels_gt.train.slp
├── labels_gt.val.slp
├── labels_pr.train.slp
├── labels_pr.val.slp
├── metrics.train.npz
├── metrics.val.npz
├── training_config.json
└── training_log.csv
2 directories, 20 files
Inference#
Let’s run inference with our trained models for centroids and centered instances.
!sleap-track "dataset/drosophila-melanogaster-courtship/20190128_113421.mp4" --frames 0-100 -m "models/courtship.centroid" -m "models/courtship.topdown_confmaps"
Started inference at: 2023-09-01 13:42:03.066840
Args:
{
│ 'data_path': 'dataset/drosophila-melanogaster-courtship/20190128_113421.mp4',
│ 'models': [
│ │ 'models/courtship.centroid',
│ │ 'models/courtship.topdown_confmaps'
│ ],
│ 'frames': '0-100',
│ 'only_labeled_frames': False,
│ 'only_suggested_frames': False,
│ 'output': None,
│ 'no_empty_frames': False,
│ 'verbosity': 'rich',
│ 'video.dataset': None,
│ 'video.input_format': 'channels_last',
│ 'video.index': '',
│ 'cpu': False,
│ 'first_gpu': False,
│ 'last_gpu': False,
│ 'gpu': 'auto',
│ 'max_edge_length_ratio': 0.25,
│ 'dist_penalty_weight': 1.0,
│ 'batch_size': 4,
│ 'open_in_gui': False,
│ 'peak_threshold': 0.2,
│ 'max_instances': None,
│ 'tracking.tracker': None,
│ 'tracking.target_instance_count': None,
│ 'tracking.pre_cull_to_target': None,
│ 'tracking.pre_cull_iou_threshold': None,
│ 'tracking.post_connect_single_breaks': None,
│ 'tracking.clean_instance_count': None,
│ 'tracking.clean_iou_threshold': None,
│ 'tracking.similarity': None,
│ 'tracking.match': None,
│ 'tracking.robust': None,
│ 'tracking.track_window': None,
│ 'tracking.min_new_track_points': None,
│ 'tracking.min_match_points': None,
│ 'tracking.img_scale': None,
│ 'tracking.of_window_size': None,
│ 'tracking.of_max_levels': None,
│ 'tracking.save_shifted_instances': None,
│ 'tracking.kf_node_indices': None,
│ 'tracking.kf_init_frame_count': None
}
2023-09-01 13:42:03.098811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.103255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.103982: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.inference:Auto-selected GPU 0 with 23050 MiB of free memory.
Versions:
SLEAP: 1.3.2
TensorFlow: 2.7.0
Numpy: 1.21.5
Python: 3.7.12
OS: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid
System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
Video: dataset/drosophila-melanogaster-courtship/20190128_113421.mp4
2023-09-01 13:42:03.157392: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-01 13:42:03.158019: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.158864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.159656: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.455402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.456138: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.456803: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-09-01 13:42:03.457464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21145 MB memory: -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% ETA: -:--:-- ?2023-09-01 13:42:07.038687: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 51.9 FPS[0m 126.4 FPS FPSFPS
?25hFinished inference at: 2023-09-01 13:42:10.842469
Total runtime: 7.775644779205322 secs
Predicted frames: 101/101
Provenance:
{
│ 'model_paths': [
│ │ 'models/courtship.centroid/training_config.json',
│ │ 'models/courtship.topdown_confmaps/training_config.json'
│ ],
│ 'predictor': 'TopDownPredictor',
│ 'sleap_version': '1.3.2',
│ 'platform': 'Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid',
│ 'command': '/home/talmolab/micromamba/envs/s0/bin/sleap-track dataset/drosophila-melanogaster-courtship/20190128_113421.mp4 --frames 0-100 -m models/courtship.centroid -m models/courtship.topdown_confmaps',
│ 'data_path': 'dataset/drosophila-melanogaster-courtship/20190128_113421.mp4',
│ 'output_path': 'dataset/drosophila-melanogaster-courtship/20190128_113421.mp4.predictions.slp',
│ 'total_elapsed': 7.775644779205322,
│ 'start_timestamp': '2023-09-01 13:42:03.066840',
│ 'finish_timestamp': '2023-09-01 13:42:10.842469'
}
Saved output: dataset/drosophila-melanogaster-courtship/20190128_113421.mp4.predictions.slp
When inference is finished, predictions are saved in a file. Since we didn’t specify a path, it will be saved as <video filename>.predictions.slp in the same directory as the video:
!tree dataset/drosophila-melanogaster-courtship
dataset/drosophila-melanogaster-courtship
├── 20190128_113421.mp4
├── 20190128_113421.mp4.predictions.slp
├── courtship_labels.slp
└── example.jpg
0 directories, 4 files
You can inspect your predictions file using sleap-inspect:
!sleap-inspect dataset/drosophila-melanogaster-courtship/20190128_113421.mp4.predictions.slp
Labeled frames: 101
Tracks: 0
Video files:
dataset/drosophila-melanogaster-courtship/20190128_113421.mp4
labeled frames: 101
labeled frames from 0 to 100
user labeled frames: 0
tracks: 1
max instances in frame: 2
Total user labeled frames: 0
Provenance:
model_paths: ['models/courtship.centroid/training_config.json', 'models/courtship.topdown_confmaps/training_config.json']
predictor: TopDownPredictor
sleap_version: 1.3.2
platform: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid
command: /home/talmolab/micromamba/envs/s0/bin/sleap-track dataset/drosophila-melanogaster-courtship/20190128_113421.mp4 --frames 0-100 -m models/courtship.centroid -m models/courtship.topdown_confmaps
data_path: dataset/drosophila-melanogaster-courtship/20190128_113421.mp4
output_path: dataset/drosophila-melanogaster-courtship/20190128_113421.mp4.predictions.slp
total_elapsed: 7.775644779205322
start_timestamp: 2023-09-01 13:42:03.066840
finish_timestamp: 2023-09-01 13:42:10.842469
args: {'data_path': 'dataset/drosophila-melanogaster-courtship/20190128_113421.mp4', 'models': ['models/courtship.centroid', 'models/courtship.topdown_confmaps'], 'frames': '0-100', 'only_labeled_frames': False, 'only_suggested_frames': False, 'output': None, 'no_empty_frames': False, 'verbosity': 'rich', 'video.dataset': None, 'video.input_format': 'channels_last', 'video.index': '', 'cpu': False, 'first_gpu': False, 'last_gpu': False, 'gpu': 'auto', 'max_edge_length_ratio': 0.25, 'dist_penalty_weight': 1.0, 'batch_size': 4, 'open_in_gui': False, 'peak_threshold': 0.2, 'max_instances': None, 'tracking.tracker': None, 'tracking.target_instance_count': None, 'tracking.pre_cull_to_target': None, 'tracking.pre_cull_iou_threshold': None, 'tracking.post_connect_single_breaks': None, 'tracking.clean_instance_count': None, 'tracking.clean_iou_threshold': None, 'tracking.similarity': None, 'tracking.match': None, 'tracking.robust': None, 'tracking.track_window': None, 'tracking.min_new_track_points': None, 'tracking.min_match_points': None, 'tracking.img_scale': None, 'tracking.of_window_size': None, 'tracking.of_max_levels': None, 'tracking.save_shifted_instances': None, 'tracking.kf_node_indices': None, 'tracking.kf_init_frame_count': None}
If you’re using Chrome you can download your trained models like so:
# Zip up the models directory
!zip -r trained_models.zip models/
# Download.
from google.colab import files
files.download("/content/trained_models.zip")
adding: models/ (stored 0%)
adding: models/courtship.topdown_confmaps/ (stored 0%)
adding: models/courtship.topdown_confmaps/labels_pr.val.slp (deflated 74%)
adding: models/courtship.topdown_confmaps/metrics.val.npz (deflated 0%)
adding: models/courtship.topdown_confmaps/labels_pr.train.slp (deflated 67%)
adding: models/courtship.topdown_confmaps/labels_gt.val.slp (deflated 72%)
adding: models/courtship.topdown_confmaps/initial_config.json (deflated 73%)
adding: models/courtship.topdown_confmaps/training_log.csv (deflated 55%)
adding: models/courtship.topdown_confmaps/metrics.train.npz (deflated 0%)
adding: models/courtship.topdown_confmaps/labels_gt.train.slp (deflated 61%)
adding: models/courtship.topdown_confmaps/best_model.h5 (deflated 8%)
adding: models/courtship.topdown_confmaps/training_config.json (deflated 88%)
adding: models/courtship.centroid/ (stored 0%)
adding: models/courtship.centroid/labels_pr.val.slp (deflated 82%)
adding: models/courtship.centroid/metrics.val.npz (deflated 1%)
adding: models/courtship.centroid/labels_pr.train.slp (deflated 79%)
adding: models/courtship.centroid/labels_gt.val.slp (deflated 73%)
adding: models/courtship.centroid/initial_config.json (deflated 74%)
adding: models/courtship.centroid/training_log.csv (deflated 57%)
adding: models/courtship.centroid/metrics.train.npz (deflated 0%)
adding: models/courtship.centroid/labels_gt.train.slp (deflated 61%)
adding: models/courtship.centroid/best_model.h5 (deflated 7%)
adding: models/courtship.centroid/training_config.json (deflated 88%)
And you can likewise download your predictions:
from google.colab import files
files.download('dataset/drosophila-melanogaster-courtship/20190128_113421.mp4.predictions.slp')
In some other browsers (Safari) you might get an error and you can instead download using the “Files” tab in the side panel (it has a folder icon). Select “Show table of contents” in the “View” menu if you don’t see the side panel.