Using custom / pres-selected images option

[Limbicnation]

Hello, great notebook! Is it possible to use your own / pres-selected or custom pictures? If so, where would I adapt the code? Thanks very much!

[robgon-art]

Hi Limbicnation, thanks for your interest in the project.

Yes, it would be fairly easy to create a collage from custom images.

You just need to fill up all_files with a list of image files and all_images with a list of images.

But first, you should make a copy of the colab to your google drive to save your work.

Then, if not needed, comment out the code to collect images from Wikimedia and Open Images, from lines 30 to 104 in the Find Source Images block.

Then add code like this after line 104:

import glob
file_path_list = glob.glob("/content/custom_images/*.jpg")
for file_path in file_path_list:
      img = Image.open(file_path)
      img = img.convert(mode="RGB")
      all_images.append(img)
      all_files.append(file_path)
print("num total images", len(all_images))

Note that I didn't test this so there may be some typos/bugs in the code.

If you put some jpg files in /content/custom_images, the system will pick them up and use them for the collage.

Cheer,
-- Rob

[Limbicnation]

Hi Rob,

thank you very much for your reply! I really appreciate your help.
I am trying to run this notebook locally on my Ubuntu 20.04 machine.
Once I got that working, I will follow the steps in your description. I'll let you know if it worked!

Best, Gero

[Limbicnation]

Hi again Rob,

I am trying to run the notebook locally, by converting to python. However, I get the following error:

Traceback (most recent call last): File "/media/**-**/Data-ml1/ML_Images/CLIPandPASTE/Clip_and_Paste.py", line 103, in <module> os.chdir('./content') #<cc-cm> FileNotFoundError: [Errno 2] No such file or directory: './content'

#-O checkpoints/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco_20200604_211715-42eb79e1.pth os.chdir('/content') #<cc-cm>

Is there a way I can fix this?
Thanks!

[robgon-art]

Yeah, the /content folder is effectively the root folder for the project running on a Colab.

To fix this, just replace all instances of “/content” with the name of your local root folder for the project, i.e., “/media/-/Data-ml1/ML_Images/CLIPandPASTE”

[Limbicnation]

Thank you for your help!

[Limbicnation]

Unfortunately now I am getting another error:

raise FileNotFoundError(f'{filename} can not be found.') FileNotFoundError: /media/**-**/Data-ml1/ML_Images/CLIPandPASTE/mmdetection/checkpoints/mask_rcnn_r50_fpn_syncbn-backbone_r4_gcb_c3-c5_groie_1x_coco_20200604_211715-42eb79e1.pth can not be found.

I think this is too high-level for me, to get this code running locally. :)

I changed the path pointing to the file to the Root directory of the project.

[Limbicnation]

Traceback (most recent call last): File "Clip_and_Paste.py", line 382, in <module> image_input = torch.tensor(np.stack(images)).cuda() File "<__array_function__ internals>", line 6, in stack File "/home/**-**/anaconda3/envs/CLIPandPASTE/lib/python3.6/site-packages/numpy/core/shape_base.py", line 423, in stack raise ValueError('need at least one array to stack') ValueError: need at least one array to stack

Unfortunately I am not familiar with this error.

[robgon-art]

Hi, both of these errors seem to be stemming from a problem with the file paths. The np.stack() call is probably failing because it didn't find any images to process.

[Limbicnation]

Hi Rob, thank you for your help. I've now gotten it up and running to the point where the images are being analyzed and cropped accordingly. But I don't know how this can be further automated in the code, I still exclude some code that might be necessary I think. I now have to click away each image so the process can continue. Thanks!

[Limbicnation]

OK. It worked now. :)
However, I had to manually click away every picture so that the process could continue.

1. is it possible to get images with transparency for the background?
2. Is it possible to control the size of the individual elements?

Thanks!

Best regards,
Gero

[robgon-art2]

Hi Gero, Nice work!

Yes, it is possible to do both of the things you are asking about.

Resizing the elements is easier. This can be done by changing the number 512 used in lines 300 to 303 in the Find Source Images block. The code currently resizes any image with a width or height greater than 512 down to 512, preserving the aspect ratio. Depending on how big your images are, changing all of the 512s to be smaller will make everything smaller, and making it bigger will make everything bigger. You could replace all the 512s with an image_size variable and play with it. Or, if that doesn't work for you, could rewrite the block and do something different.

Using transparency would be a bit harder. First, I convert am currently forcing all images to be in mode="RGB". You have to change the call to read the images img = img.convert(mode="RGBA"). But then you would have to decide if you need the image segmentation with GRoIE or not. If your images are already segmented, you could skip using GRoIE and simply pull the RGB and alpha image from the sources at lines 305 to 307. But this would require a bit of work to do, but it is possible.

Hope this helps.

Cheers,

• Rob

[Limbicnation]

Hi Rob, thanks for your help!

I think the alpha images are a bit too difficult for me.
I implemented the 'image_size' variable.

How can I get more control over the background gradient? I need some insight into this script please.

Thanks!

Cheers,
Gero

[robgon-art]

Hi, if you haven’t seen it yet you should read my write up on Medium first.
https://towardsdatascience.com/clip-and-paste-using-ai-to-create-modern-collages-from-text-prompts-38de46652827

This has a section on how creating the background gradient image works.

There are a lot of ways to extend this. What type of additional controls are you looking for?

[Limbicnation]

Hi Rob,

thanks for sharing the Link!

I've already looked at it, but my python skills are not up to standard. I would like to control the colors or create patterns like with Perlin Noise e.g.
Would that be possible?

sorry if I ask again and if that topic was already explained in the article.

[robgon-art]

Hi Gero,

Yes, good idea! Using Perlin noise would work well.

You could code it up to have Adam modify parameters like the number of octaves (for scaling), the y and x offsets (for translation), and even add a z offset (for perturbing the form).

Here's some code to start with.

from perlin_noise import PerlinNoise
import numpy as np
import matplotlib.pyplot as plt
height, width = 224, 224

octaves = 2.0
offsets = [0.0, 0.0, 0.0]

noise_r = PerlinNoise(octaves=octaves)
noise_g = PerlinNoise(octaves=octaves)
noise_b = PerlinNoise(octaves=octaves)

img_r = [[noise_r([i/height+offsets[0], j/width+offsets[1], offsets[2]]) for j in range(height)] for i in range(width)]
img_g = [[noise_g([i/height+offsets[0], j/width+offsets[1], offsets[2]]) for j in range(height)] for i in range(width)]
img_b = [[noise_b([i/height+offsets[0], j/width+offsets[1], offsets[2]]) for j in range(height)] for i in range(width)]

img = np.array([img_r, img_g, img_b])
img = (img + 0.5).clip(0,1)
img = np.transpose(img, (1,2,0))

plt.imshow(img)
plt.axis("off")
plt.show()

There may be a faster way to generate Perlin Noise, but this seems to work ok.

• Rob

[Limbicnation]

Hi Rob, thank you for your message. I'm not sure where to insert this code. Thanks!

[robgon-art]

Hi Gero,

Yeah, you need to remove the spline color generator and add this code. And the Adam optimizer needs to be configured to change the octaves and offsets variables.

If I have time, I’ll add this to the Colab as an option.

Cheers,

• Rob

[Limbicnation]

Hi Rob,

If you have the time, I would be very grateful, if you could help me implement the code.

Thank you!