concatenate CNN features and image features

[mrlihellohorld]

Hello, do you have experiments to concatenate CNN features and image features（haze、sky、contrast、histogram\sharpness）?

[berkgulay]

Hey,
No we did not concatenate CNN features and descriptive features such as haze, sharpness etc. We evaluated them as separate models for image classification. But you can concatenate flattened output of CNN and descriptive features in a single array as description of an image. This is totally possible and can lead different results for sure.

We have a desriptor function describe() in ImageDescriptor.py which returns flattened descriptive array of an image. You can concat its output with output of Flatten layer of your CNN model. This would be a completely different description array for an image and you can evaluate its performance through different experiments.

Please share your results with me, if you try such experiments!

[mrlihellohorld]

hey,
the desriptor function describe() in ImageDescriptor.py returns the description_array of an image. and "image_to_matrix" function provides numpy files to train Neural Network model.There is a problem here.the numpy files to train CNN are randomly disrupted,because used the shuffle function.such as train_data, train_label = shuffle(train_data, train_labe).so,can we concat desriptor's output with output of Flatten layer of your CNN model?These two features cannot correspond one-to-one.I don't know if I express it clearly.This is my email,[email protected], I look forward to your reply.

[berkgulay]

Hey.
Yeah, I got you.

Of course, you can't directly concat all images with their features using our numpy files. Because as I explained, those numpy files of images were not generated in order to concat with any other feature. So; if you want to concat an image and it's feature array in a single numpy array, you should write a new function which processes images one by one. Because you can't make sure of each read image from directory corresponds to its numpy or feature array. In brief, you should not use our image_to_matrix() function as it is. You can change that function to take a single image(not a directory) and return it's numpy version as we do. But do not try to process all directory at once. So, you wont have any shuffle problem as well, if you process each image one by one instead of all at once like us.

You can directly use output of our describe() function, because it processes a single image and returns it description array. So, we are fine there.

Hope it helps :)

[mrlihellohorld]

Hey,
thanks for u help.I have now created two .npy files:train_img_data.npy(extracted by image_to_matrix）, and the hand_feature.npy(extracted ImageDescriptor，descriptive features such as haze, sharpness etc）,It is worth noting that the features in these two files correspond one-to-one according to the pictures.Also, I set up an end-to-end CNN network, after FLatten, concate the files in hand_feature.npy. But the result of concate is very close to the result of not concate. There is no significant improvement in accuracy. I have a few questions to ask you：

1. Do you have any suggestions for my design ideas?
2. Can I improve the accuracy by adding weight to the hand_feature?
3. Do you have a paper on your report, and if so, which journal is it?

[berkgulay]

Hey,

1. Most probably; since CNNs can extract needed features by itself very sufficiently, there is no significant change in accuracy results. Thus, I don't think you can improve accuracy just by adding new features. But you can try Transfer Learning. Transfer Learning from similar tasks (weather prediction) can improve the results significantly.
2. Again, I don't think you can change accuracy just by adding new features or weights to features in hand.
3. If you ask about papers on this issue specifically, no. In our report, there is no paper or research on concatenating different kinds of features and classify them.