Cannot allocate tensors in C++

[FabienSailliet]

Hello,

I am trying to run inferences on the TPU of a Coral dev board using TensorFlow Lite with C++, following this indications on the Coral website.

Here is the code I am trying to run:

#include <iostream>
#include <memory>

#include <edgetpu/edgetpu.h>
#include <tensorflow/lite/model.h>
#include <tensorflow/lite/kernels/register.h>

#define MODEL_PATH "models/ssd_mobilenet_v2_face_quant_postprocess_edgetpu.tflite"

using namespace std;

unique_ptr<tflite::Interpreter> BuildEdgeTpuInterpreter(
    const tflite::FlatBufferModel& model,
    edgetpu::EdgeTpuContext* edgetpu_context)
{
    tflite::ops::builtin::BuiltinOpResolver resolver;
    resolver.AddCustom(edgetpu::kCustomOp, edgetpu::RegisterCustomOp());
    std::unique_ptr<tflite::Interpreter> interpreter;
    if (tflite::InterpreterBuilder(model, resolver)(&interpreter) != kTfLiteOk) {
        std::cerr << "Failed to build interpreter." << std::endl;
    }
    // Bind given context with interpreter.
    interpreter->SetExternalContext(kTfLiteEdgeTpuContext, edgetpu_context);
    interpreter->SetNumThreads(1);
    if (interpreter->AllocateTensors() != kTfLiteOk) {
        std::cerr << "Failed to allocate tensors." << std::endl;
    }

    return interpreter;
}

int main(int argc, char* argv[])
{
	unique_ptr<tflite::FlatBufferModel> model = tflite::FlatBufferModel::BuildFromFile(MODEL_PATH);
    shared_ptr<edgetpu::EdgeTpuContext> edgetpu_context =
        edgetpu::EdgeTpuManager::GetSingleton()->OpenDevice();
    unique_ptr<tflite::Interpreter> model_interpreter =
        BuildEdgeTpuInterpreter(*model, edgetpu_context.get());

	return 0;
}

I am compiling on the dev board using the latest libedgetpu.so found there. To build the TensorFlow Lite static library following the indications from the Coral website, I first tried building it on the commit indicated there. I had compilation and linking issues, so I used the commit indicated in the sources of this lib, which allows me to compile TensorFlow Lite and my program without any issue.

When I run it, I get the following error:

ERROR: Internal: Unsupported data type in custom op handler: -1368865872
ERROR: Node number 0 (edgetpu-custom-op) failed to prepare.

Failed to allocate tensors.

The indicated data type value is not the same if I run it again. I tried multiple detection and classification models from the Coral website and I tried a model I compiled from this TensorFlow example model using edgetpu_compiler and it gives me the same error, with a varying number.

The only difference I found is using the mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite model from the Python example, which always gives:

ERROR: Internal: Unsupported data type in custom op handler: -1
ERROR: Node number 0 (edgetpu-custom-op) failed to prepare.

Failed to allocate tensors.

[Namburger]

Hey, you could checkout this answer for a quick solution: #137 (comment)

My suggestions:

I am compiling on the dev board using the latest libedgetpu.so found there. To build the TensorFlow Lite static library following the indications from the Coral website, I first tried building it on the commit indicated there. I had compilation and linking issues

Ahhhh, okay so you're on the right track, but unfortunately, the commit we picked for the current libedgetpu release also had a build bug when building libtensorflow-lite (I know :( ).
But the fix is quite easy: tensorflow/tensorflow@72cd947

so I used the commit indicated in the sources of this lib, which allows me to compile TensorFlow Lite and my program without any issue.

If you use the commit in that lib, you'll also need to build libedgetpu on that repo, the point is that libedgetpu.so and libtensorflow-lite.a are both built from the same commit.

[FabienSailliet]

I already tried compiling TensorFlow Lite by doing a checkout on tensorflow/tensorflow@d855adf and cherry-picking tensorflow/tensorflow@72cd947, but by doing so I'm getting errors from the linker when I try to compile my code:

/usr/bin/ld: CMakeFiles/inference_test.dir/src/inference_test.cpp.o: in function `BuildEdgeTpuInterpreter(tflite::FlatBufferModel const&, edgetpu::EdgeTpuContext*)':
inference_test.cpp:(.text+0x3c): undefined reference to `tflite::MutableOpResolver::AddCustom(char const*, TfLiteRegistration const*, int)'
/usr/bin/ld: inference_test.cpp:(.text+0x54): undefined reference to `tflite::impl::InterpreterBuilder::InterpreterBuilder(tflite::FlatBufferModel const&, tflite::OpResolver const&)'
/usr/bin/ld: inference_test.cpp:(.text+0x60): undefined reference to `tflite::impl::InterpreterBuilder::operator()(std::unique_ptr<tflite::impl::Interpreter, std::default_delete<tflite::impl::Interpreter> >*)'
/usr/bin/ld: inference_test.cpp:(.text+0x74): undefined reference to `tflite::impl::InterpreterBuilder::~InterpreterBuilder()'
/usr/bin/ld: inference_test.cpp:(.text+0xdc): undefined reference to `tflite::impl::Interpreter::SetExternalContext(TfLiteExternalContextType, TfLiteExternalContext*)'
/usr/bin/ld: inference_test.cpp:(.text+0xec): undefined reference to `tflite::impl::Interpreter::SetNumThreads(int)'
/usr/bin/ld: inference_test.cpp:(.text+0xf8): undefined reference to `tflite::impl::Interpreter::AllocateTensors()'
/usr/bin/ld: inference_test.cpp:(.text+0x150): undefined reference to `tflite::impl::InterpreterBuilder::~InterpreterBuilder()'
/usr/bin/ld: CMakeFiles/inference_test.dir/src/inference_test.cpp.o: in function `std::default_delete<tflite::impl::Interpreter>::operator()(tflite::impl::Interpreter*) const':
inference_test.cpp:(.text._ZNKSt14default_deleteIN6tflite4impl11InterpreterEEclEPS2_[_ZNKSt14default_deleteIN6tflite4impl11InterpreterEEclEPS2_]+0x24): undefined reference to `tflite::impl::Interpreter::~Interpreter()'
/usr/bin/ld: ../lib/libedgetpu.so.1.0: undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info const&)@GLIBCXX_3.4.26'
collect2: error: ld returned 1 exit status

[Namburger]

@FabienSailliet humn that's a linker error, are you copying the headers from the same commits and the correct libtensorflow-lite.a?
Here is a supper minimal, no cmake, standard make file build: https://drive.google.com/file/d/1nsbOQtZIid3NCDjdHHob72r4YDSU2Qud/view

I believe this one uses this commit for both libedgetpu and libtensorflow-lite. Be sure to set LD_LIBRARY_PATH when run or else it could load the wrong runtime and breaks with same error.

Cheers!

[FabienSailliet]

Thanks, I forgot to update my includes, it is now working if I use the diploria3 release of libedgetpu. However, if I use the last one (eel2), I am getting this during the compilation:

./bin/inference_test: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/mendel/inference_test/bin/../lib/libedgetpu.so.1)

Maybe the compiler you used to compile it is too recent, I have g++ 8.3.0 on the dev board running Mendel Linux.

[Namburger]

@FabienSailliet
hummn, that's odd, this is due to glibc version, but I'm really confuse because the library definitely works for me on the dev board and that's pointing to libstdc++.so.6 o_0

Can you share:

cat /etc/os-release
strings /usr/lib/aarch64-linux-gnu/libstdc++.so.6 | grep GLIBCXX

[FabienSailliet]

My bad, I took the wrong .so file, with the eel2 release of the lib and TensorFlow Lite compiled from tensorflow/tensorflow@d855adf after aplying the patch tensorflow/tensorflow@72cd947 I can now run it.

Thank you very much!