Training help

[kidconcept]

Hello,

I'm attempting to train the model using corals I've tagged from scans taken in Bonaire. So far, I've been unable to get any results as TagLab crashes before the training is complete. I'm writing here to find out if the problem is with my method, and am asking for help to ensure I'm doing this correctly. I've switched to training with the included sample projects to reduce the chance of problems.

1. I begin by starting a new, empty project and add a new map, via Project > Add New map. I fill out the name, point it to the PNG, and leave the depth map empty. Acquisition date and pixel size here I just use filler values.
2. I open the Labels Dictionary Editor and create a new dictionary to have the labels I'm planning to categorize. Here I'll use the labels healthy and dead.
3. I set about delinitating the corals using 4-clicks segmentation tool and the Positive/Negative clicks segmentation tool. As I go I label the corals I delinated as either Healthy or Dead.
4. Now I export the training data via File > Export New Training Dataset, and give it a name and location. I'll leave all other settings on the default values.
5. I check the training folder and it looks like the training data has exported correctly.
6. Now I'll attempt to train the network via File > Train your network. I give a network name and point it at the training folder, leaving all other settings as the defaults. I click Train and then:
7. TagLab crashes.

So that's everything I'm doing. Can someone tell me if I'm doing something wrong or have missunderstood something fundemental? Is it ok to train custom label names? Is it wrong to train within the same file I labeled? Any advice here would be appricated.

Project Files:
https://www.dropbox.com/scl/fo/763rxcem334sqns3aicyo/AGRoKXgmRHEgI3tp0rpfms4?rlkey=ui6y9furkampyyf55ebifqaru&dl=0

[maxcorsini]

We have checked the procedure and the files. First of all, one comment. Taking into account that if you use only this map the data are few, so we expect that the performance of the resulting classifier will be poor. Apart this, one strange thing is that the labels shown in your screenshot (plot18_0003.png) are different from the one saved in the Project files you provided. This may be related to the problem.
You have exported the tiles before to change the dictionary ?

[kidconcept]

one strange thing is that the labels shown in your screenshot (plot18_0003.png) are different from the one saved in the Project files you provided.

I don't see different labels in the Project files on the dropbox link, when I open the project file there I see a Dictionary with the Labels Empty, Healthy, and Diseased. And the export was after setting up the dictionary.

I did reupload the project to this same folder on Dropbox shortly before posting this page. Perhaps you somehow accessed an older verison of the project file.

[maxcorsini]

I am talking about the color of the labels. In the fifth screenshots you provide, the label "plot0018_0003.png" has colors different with respect to the one assigned to the labels in the program. Instead, if I open it the colors are the ones of the dictionary (and this is ok). Have you an explanation of this ?

[maxcorsini]

I am talking about the color of the labels. In the fifth screenshots, the label "plot0018_0003.png" has colors different with respect to the one assigned to the labels in the program. Instead, if I open it the colors are the ones of the dictionary (and this is ok). Have you an explanation of this ?

[maxcorsini]

I am talking about the color of the labels. In the fifth screenshots you provide, the label "plot0018_0003.png" has colors different with respect to the one assigned to the labels in the program. Instead, if I open it the colors are the ones of the dictionary (and this is ok). Have you an explanation of this ?

[kidconcept]

Oh, sorry I see! Yes, that is a mistake of the screenshots. I had the wrong project folder open when I took that screenshot. The Label colors I see in the .pngs found in the training/test|training/validation/labels/ folders are

RGB 131 255 179
and
RGB 255 123 94

Which is what I see in the Labels Dictionary editor of the Plot18.json project file.

[kidconcept]

I updated the screenshot with pngs from the correct project folder, which you can find in the dropbox link.

[maxcorsini]

Ok. It seems a problem related to the data augmentation during the training phase. I will check and let you know.

[maxcorsini]

A new version to solve this issue has been released. Update your TagLab's version and let us know if all work (take care that with high probability the classifier will not work in this case due to the low amount of data).

[kidconcept]

OK, that seems to have fixed the bug and I'm getting really close! However, there seems to be another instance of the deserialize object on a CUDA device bug (I'm on a cpu only machine, m1 macbook pro). Full error below. Any help is greatly appricated. Once I get this classifier running without bugs I'll start over with a larger data set.

Dataset setup.. Computing average... 98.00 Computing frequencies... 98.00{'Healthy': 1, 'Dead': 2, 'Background': 0} [3.0691193 7.69705705 1.83737793] [0.32582637 0.12991979 0.54425384] done. NETWORK USED: DEEPLAB V3+ Downloading: "https://download.pytorch.org/models/resnet101-5d3b4d8f.pth" to /Users/danvallentyne/.cache/torch/hub/checkpoints/resnet101-5d3b4d8f.pth 100.0% /Applications/TagLab-main/models/training.py:299: FutureWarning: You are using torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load("models/deeplab-resnet.pth.tar")
Traceback (most recent call last):
File "/Applications/TagLab-main/TagLab.py", line 5095, in trainNewNetwork
dataset_train_info, train_loss_values, val_loss_values = training.trainingNetwork(images_dir_train, labels_dir_train,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/TagLab-main/models/training.py", line 299, in trainingNetwork
state = torch.load("models/deeplab-resnet.pth.tar")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/danvallentyne/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/serialization.py", line 1384, in load
return _legacy_load(
^^^^^^^^^^^^^
File "/Users/danvallentyne/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/serialization.py", line 1638, in _legacy_load
result = unpickler.load()
^^^^^^^^^^^^^^^^
File "/Users/danvallentyne/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/serialization.py", line 1566, in persistent_load
obj = restore_location(obj, location)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/danvallentyne/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/serialization.py", line 601, in default_restore_location
result = fn(storage, location)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/danvallentyne/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/serialization.py", line 539, in _deserialize
device = _validate_device(location, backend_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/danvallentyne/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/serialization.py", line 508, in _validate_device
raise RuntimeError(
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
zsh: abort python TagLab.py

`

[maxcorsini]

Have you installed TagLab using the "cpu" option ?