Custom object detection ...

[IshinEV]

Thank you for your code!

I run a simple code from heare: https://github.com/OlafenwaMoses/ImageAI/blob/master/imageai/Detection/Custom/CUSTOMDETECTIONTRAINING.md
and:

> cd 'd:\ANGEL'; ${env:PYTHONIOENCODING}='UTF-8'; ${env:PYTHONUNBUFFERED}='1'; & 'C:\Program Files\Python36\python.exe' 'c:\Users\User\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\ptvsd_launcher.py' '--default' '--client' '--host' 'localhost' '--port' '63215' 'd:\ANGEL\train.py'
Using TensorFlow backend.                                                                                                               use `mi
Generating anchor boxes for training images and annotation...                                                                e removed, 
Average IOU for 9 anchors: 0.76
Anchor Boxes generated.
Detection configuration saved in  hololens\json\detection_config.json
Training on:    ['hololens']
Training with Batch Size:  4
Number of Experiments:  100
Training with transfer learning from pretrained Model
C:\Program Files\Python36\lib\site-packages\keras\callbacks.py:1065: UserWarning: `epsilon` argument is deprecated and will be removed, use `min_delta` instead.
  warnings.warn('`epsilon` argument is deprecated and '
Epoch 1/100
>

Nothing. No error. No result. Just end task after "Epoch 1/100"... I try change number of Epoch and number of Batch Size - not work.
Do you have some idea?

Some setting:
pip freeze: absl-py==0.7.1 astor==0.8.0 astroid==2.2.5 colorama==0.4.1 cycler==0.10.0 gast==0.2.2 google-pasta==0.1.7 grpcio==1.22.0 h5py==2.9.0 imageai==2.1.3 isort==4.3.21 Keras==2.2.4 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.0 kiwisolver==1.1.0 lazy-object-proxy==1.4.1 Markdown==3.1.1 matplotlib==3.1.1 mccabe==0.6.1 mock==3.0.5 numpy==1.14.5 opencv-python==4.1.0.25 Pillow==6.1.0 protobuf==3.9.1 pylint==2.3.1 pyparsing==2.4.2 python-dateutil==2.8.0 PyYAML==5.1.2 scipy==1.3.1 six==1.12.0 tensorboard==1.10.0 tensorflow==1.14.0 tensorflow-estimator==1.13.0 tensorflow-gpu==1.10.0 termcolor==1.1.0 typed-ast==1.4.0 Werkzeug==0.15.5 wrapt==1.11.2

Hardware:

CPU: Intel Core i7  - 4790 @ 3.6GHz
RAM: 24Gb
GPU: GeForce GTX 950

[OlafenwaMoses]

Hello, Kindly upgrade your Tensorflow GPU to 1.13.1 . It might solve the problem.
pip3 install tensorflow-gpu==1.13.1

[IshinEV]

I try it. After thet i have error:

> cd 'd:\ANGEL'; ${env:PYTHONIOENCODING}='UTF-8'; ${env:PYTHONUNBUFFERED}='1'; & 'C:\Program Files\Python36\python.exe' 'c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\ptvsd_launcher.py' '--default' '--client' '--host' 'localhost' '--port' '63807' 'd:\ANGEL\train.py'     
Using TensorFlow backend.
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in 
<module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Program Files\Python36\lib\imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "C:\Program Files\Python36\lib\imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\lib\python\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\lib\python\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\Program Files\Python36\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files\Python36\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\ANGEL\train.py", line 1, in <module>
    from imageai.Detection.Custom import DetectionModelTrainer
  File "C:\Program Files\Python36\lib\site-packages\imageai\Detection\__init__.py", line 2, in <module>  
    from imageai.Detection.keras_retinanet.models.resnet import resnet50_retinanet
  File "C:\Program Files\Python36\lib\site-packages\imageai\Detection\keras_retinanet\models\resnet.py", 
line 19, in <module>
    import keras
  File "C:\Program Files\Python36\lib\site-packages\keras\__init__.py", line 3, in <module>
    from . import utils
  File "C:\Program Files\Python36\lib\site-packages\keras\utils\__init__.py", line 6, in <module>        
    from . import conv_utils
  File "C:\Program Files\Python36\lib\site-packages\keras\utils\conv_utils.py", line 9, in <module>      
    from .. import backend as K
  File "C:\Program Files\Python36\lib\site-packages\keras\backend\__init__.py", line 89, in <module>     
    from .tensorflow_backend import *
  File "C:\Program Files\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 5, in <module>
    import tensorflow as tf
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\__init__.py", line 24, in <module>        
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module> 
    from tensorflow.python import pywrap_tensorflow
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in 
<module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in 
<module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "C:\Program Files\Python36\lib\imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "C:\Program Files\Python36\lib\imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: DLL load failed: The specified module could not be found.


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

I found idea to downgrade to 'Tensorflow GPU to 1.10' on thenserflow forum

[OlafenwaMoses]

Please refer to the solutions provided in this thread.

tensorflow/tensorflow#22794

[IshinEV]

I try this configuration :

    Windows 10 
    CUDA Toolkit 10.0 (cuda_10.0.130_win10_network.exe)
    cuDNN v7.6.0 (cudnn-10.0-windows10-x64-v7.6.0.64)
    Python 3.7.3 Windows AMD64
    Tensorflow-gpu 1.13.1

and have new error message:

Using TensorFlow backend.
C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Generating anchor boxes for training images and annotation...
Average IOU for 9 anchors: 0.76
Anchor Boxes generated.
Detection configuration saved in  hololens\json\detection_config.json
Training on:    ['hololens']
Training with Batch Size:  4
Number of Experiments:  100
WARNING:tensorflow:From C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From C:\Program Files\Python37\lib\site-packages\imageai\Detection\Custom\yolo.py:24: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Training with transfer learning from pretrained Model
C:\Program Files\Python37\lib\site-packages\keras\callbacks.py:1065: UserWarning: `epsilon` argument is deprecated and will be removed, use `min_delta` instead.
  warnings.warn('`epsilon` argument is deprecated and '
WARNING:tensorflow:From C:\Program Files\Python37\lib\site-packages\tensorflow\python\ops\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/100
Traceback (most recent call last):
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\lib\python\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\user\.vscode\extensions\ms-python.python-2019.8.29288\pythonFiles\lib\python\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\Program Files\Python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files\Python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\ANGEL\train.py", line 10, in <module>
    trainer.trainModel()
  File "C:\Program Files\Python37\lib\site-packages\imageai\Detection\Custom\__init__.py", line 286, in trainModel
    max_queue_size=8
  File "C:\Program Files\Python37\lib\site-packages\keras\legacy\interfaces.py", line 91, 
in wrapper
    return func(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\keras\engine\training.py", line 1418, 
in fit_generator
    initial_epoch=initial_epoch)
  File "C:\Program Files\Python37\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "C:\Program Files\Python37\lib\site-packages\keras\engine\training.py", line 1217, 
in train_on_batch
    outputs = self.train_function(ins)
  File "C:\Program Files\Python37\lib\site-packages\keras\backend\tensorflow_backend.py", 
line 2715, in __call__
    return self._call(inputs)
  File "C:\Program Files\Python37\lib\site-packages\keras\backend\tensorflow_backend.py", 
line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Program Files\Python37\lib\site-packages\tensorflow\python\client\session.py", 
line 1439, in __call__
    run_metadata_ptr)
  File "C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.     
         [[{{node bnorm_62/beta/read}}]]

[IshinEV]

and one more:

......
  File "C:\Program Files\Python37\lib\site-packages\keras\engine\training.py", line 1418, 
in fit_generator
    initial_epoch=initial_epoch)
  File "C:\Program Files\Python37\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "C:\Program Files\Python37\lib\site-packages\keras\engine\training.py", line 1217, 
in train_on_batch
    outputs = self.train_function(ins)
  File "C:\Program Files\Python37\lib\site-packages\keras\backend\tensorflow_backend.py", 
line 2715, in __call__
    return self._call(inputs)
  File "C:\Program Files\Python37\lib\site-packages\keras\backend\tensorflow_backend.py", 
line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Program Files\Python37\lib\site-packages\tensorflow\python\client\session.py", 
line 1439, in __call__
    run_metadata_ptr)
  File "C:\Program Files\Python37\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.     
         [[{{node conv_59/kernel/read}}]]
         [[{{node training/Adam/gradients/replica_1/model_1/bnorm_20/FusedBatchNorm_grad/FusedBatchNormGrad}}]]

[OlafenwaMoses]

There is a chance your GPU memory cannot handle the batch_size = 4. Try to keep it as 2 and see if the problem persists.

Also see the discussion below.

https://stackoverflow.com/questions/37313818/tensorflow-dst-tensor-is-not-initialized

[IshinEV]

Ok. I try it. It problem same is problem in issue #281 .
You can close this issue.
Thank you!