gocv code memory leak

[478254406]

about gocv code memory leak

Description

Due to business scenarios and company regulations, I cannot make the code public.
Our project's main work is to split an image into multiple 512*512 images, send them to an AI model, and then paste the images returned by the AI model back into the original image. The gocv functions used are listed in the table below.

1. gocv.IMDecode
2. mat.Empty()
3. mat.Rows()
4. mat.Cols()
5. mat.Channels()
6. mat.Channels()
7. mat.Type()
8. gocv.NewMat()
9. gocv.Resize
10. gocv.IMEncode
11. mat.Region
12. mat.CopyTo
13. gocv.NewMatWithSizeFromScalar
14. mat.SetTo
15. gocv.GaussianBlur
16. gocv.NewScalar
17. mat.Clone()
18. mat.ConvertTo
19. gocv.Merge
20. gocv.ConvertScaleAbs
21. gocv.NewMatWithSize
22. gocv.Subtract
23. gocv.Multiply
24. gocv.Add
25. mat.DataPtrFloat32
26. mat.DataPtrUint8
27. gocv.Split

mat is gocv.Mat

Below is the memory usage graph of a single request test environment. It shows that the memory does not completely return to its initial state, indicating a memory leak.

In the production environment, user requests also exhibit noticeable transition phenomena.

I have tried running the following code to detect memory leak issues, and found that the number of leaked mats is 0.

		var b bytes.Buffer
		gocv.MatProfile.WriteTo(&b, 1)
		fmt.Print(b.String())
		ctx.String(http.StatusOK, b.String())

In the production environment, user requests also exhibit noticeable transition phenomena.

How should I continue investigating the subsequent issues?

[478254406]

my gocv version is 0.36 and opencv version is 4.10.0

[swdee]

There isn't enough information provided to be able to help so if you can't show us code then would you create a goroutine which dumps the MatProfile every 5 minutes for at least a 15 minute period and post the output here.

This is untested code, but something like.

func main() {
	go dumpLoop()
}

func dumpLoop() {
	ticker := time.NewTicker(5 * time.Minute)
	defer ticker.Stop()
	
	dumpMat()
	
	for {
		select {
		case <-ticker.C:
			dumpMat()
		}
	}
}

func dumpMat() {
	var b bytes.Buffer
	gocv.MatProfile.WriteTo(&b, 1)
	fmt.Println(b.String())
}

[478254406]

There isn't enough information provided to be able to help so if you can't show us code then would you create a goroutine which dumps the MatProfile every 5 minutes for at least a 15 minute period and post the output here.

This is untested code, but something like.

func main() {
	go dumpLoop()
}

func dumpLoop() {
	ticker := time.NewTicker(5 * time.Minute)
	defer ticker.Stop()
	
	dumpMat()
	
	for {
		select {
		case <-ticker.C:
			dumpMat()
		}
	}
}

func dumpMat() {
	var b bytes.Buffer
	gocv.MatProfile.WriteTo(&b, 1)
	fmt.Println(b.String())
}

hello swdee

Do I need to run this asynchronous code only while the gocv program is running? If so, since gocv is constantly allocating and releasing Mat memory, will the number of unreleased Mat objects always be greater than 0 during the execution of the gocv program?

[swdee]

Yes just run it temporarily to capture the output on your gocv program. For clarity I should note the dumpMat code I wrote above needs to be added to your gocv program.

As for your second question, we need to see the output to see whats happening.

However for efficiency you should not be opening and closing Mats all the time, instead they should be considered preallocated memory spaces that OpenCV manages in the background. For a simple example you should be doing.

	webcam, _ := gocv.VideoCaptureDevice(0)

        // open a single Mat here
	img := gocv.NewMat()
        defer img.Close()

	for {
		webcam.Read(&img)
                // ... do some work ...
	}

When you run the MatProfile against the above code it will only show a single Mat being open.

Whilst the code still works, its inefficient to do the following as it is allocating and deallocating memory all the time;

	webcam, _ := gocv.VideoCaptureDevice(0)
	
	for {
                img := gocv.NewMat()
		webcam.Read(&img)
                // ... do some work ...
                img.Close()
	}

The above MatProfile will probably show zero Mat's and if you forget the img.Close() near the bottom of the for loop you would see thousands of Mats and thus have a memory leak.

[478254406]

Yes, when my service had no requests, I ran the following code every half hour, making a total of four runs.

		var b bytes.Buffer
		gocv.MatProfile.WriteTo(&b, 1)
		fmt.Print(b.String())

But each time the result output is:

gocv.io/x/gocv.Mat profile: total 0 

But the memory still hasn't been released.

[swdee]

If your showing 0 Mats then there is no memory leak from GoCV.

The memory usage graph you posted is probably Go's ordinary memory usage (unrelated to GoCV) which does not get released immediately. Go's pprof will help you confirm that.

[478254406]

I execute the following code every ten minutes in our production environment and found that the memory leak addresses are different each time. Can this indicate that the memory leak might not be occurring in gocv?

		var b bytes.Buffer
		gocv.MatProfile.WriteTo(&b, 1)
		fmt.Print(b.String())

I execute the following code every ten minutes in our production environment and found that the memory leak addresses are different each time. Can this indicate that the memory leak might not be occurring in gocv? Additionally, I noticed that the total number of leaked mats is only 53, but the memory usage increased from a stable 10GB to 18GB.

We also ran Go's pprof memory analysis tool and found that the memory could be perfectly released

[swdee]

The addresses changing is ok as your program could open a Mat, close it, and when the next time that code function is called it would then open a new Mat and close that. Each Mat would have a different address.

Its a bit hard to tell from your screenshot, so can you copy/paste text, this would be more useful. Also can you show the output captured every 10 minutes for multiple entries to see how the Mat profile total count changes over time.

There being 53 Mat's open does not necessarily mean those are leaked Mats, rather those are just ones open when the profile was taken. If that number increases over time and you have 1000's of Mats open then you would expect some part of your code to be leaking memory as it has not closed a Mat.

[478254406]

I ran memory leak detection code in another service and found only 13 Mat memory leaks. Below are the results taken half an hour apart. I executed the code a total of 6 times.

gocv.io/x/gocv.Mat profile: total 13
10 @ 0x1027f2d 0x1027f2e 0x103d0e6 0x103ac94 0x1039d45 0x1032185 0xaa7e36 0x4875e1
#       0x1027f2c       gocv.io/x/gocv.newMat+0x4c                                      /go/src/linuxapp/vendor/gocv.io/x/gocv/mat_profile.go:71
#       0x1027f2d       gocv.io/x/gocv.NewMatWithSize+0x4d                              /go/src/linuxapp/vendor/gocv.io/x/gocv/core.go:196
#       0x103d0e5       linuxapp/logic.(*DenoiseHandelImpl).CallModelWithCrop+0x1005    /go/src/linuxapp/logic/denoiseHandel.go:931
#       0x103ac93       linuxapp/logic.(*DenoiseHandelImpl).SegmentImageCallModel+0x93  /go/src/linuxapp/logic/denoiseHandel.go:723
#       0x1039d44       linuxapp/logic.(*DenoiseHandelImpl).CallCommonDenoise+0x64      /go/src/linuxapp/logic/denoiseHandel.go:657
#       0x1032184       linuxapp/logic.(*DenoiseHandelImpl).PicDenoise.func1+0xa4       /go/src/linuxapp/logic/denoiseHandel.go:165
#       0xaa7e35        golang.org/x/sync/errgroup.(*Group).Go.func1+0x55               /go/src/linuxapp/vendor/golang.org/x/sync/errgroup/errgroup.go:78

2 @ 0x1027ec5 0x1027ec6 0x103ce96 0x103ac94 0x1039d45 0x1032185 0xaa7e36 0x4875e1
#       0x1027ec4       gocv.io/x/gocv.newMat+0x44                                      /go/src/linuxapp/vendor/gocv.io/x/gocv/mat_profile.go:71
#       0x1027ec5       gocv.io/x/gocv.NewMat+0x45                                      /go/src/linuxapp/vendor/gocv.io/x/gocv/core.go:191
#       0x103ce95       linuxapp/logic.(*DenoiseHandelImpl).CallModelWithCrop+0xdb5     /go/src/linuxapp/logic/denoiseHandel.go:907
#       0x103ac93       linuxapp/logic.(*DenoiseHandelImpl).SegmentImageCallModel+0x93  /go/src/linuxapp/logic/denoiseHandel.go:723
#       0x1039d44       linuxapp/logic.(*DenoiseHandelImpl).CallCommonDenoise+0x64      /go/src/linuxapp/logic/denoiseHandel.go:657
#       0x1032184       linuxapp/logic.(*DenoiseHandelImpl).PicDenoise.func1+0xa4       /go/src/linuxapp/logic/denoiseHandel.go:165
#       0xaa7e35        golang.org/x/sync/errgroup.(*Group).Go.func1+0x55               /go/src/linuxapp/vendor/golang.org/x/sync/errgroup/errgroup.go:78

1 @ 0x1027fe5 0x1027fe6 0x103c21a 0x103ac94 0x1039d45 0x1032185 0xaa7e36 0x4875e1
#       0x1027fe4       gocv.io/x/gocv.newMat+0x64                                      /go/src/linuxapp/vendor/gocv.io/x/gocv/mat_profile.go:71
#       0x1027fe5       gocv.io/x/gocv.NewMatWithSizeFromScalar+0x65                    /go/src/linuxapp/vendor/gocv.io/x/gocv/core.go:273
#       0x103c219       linuxapp/logic.(*DenoiseHandelImpl).CallModelWithCrop+0x139     /go/src/linuxapp/logic/denoiseHandel.go:871
#       0x103ac93       linuxapp/logic.(*DenoiseHandelImpl).SegmentImageCallModel+0x93  /go/src/linuxapp/logic/denoiseHandel.go:723
#       0x1039d44       linuxapp/logic.(*DenoiseHandelImpl).CallCommonDenoise+0x64      /go/src/linuxapp/logic/denoiseHandel.go:657
#       0x1032184       linuxapp/logic.(*DenoiseHandelImpl).PicDenoise.func1+0xa4       /go/src/linuxapp/logic/denoiseHandel.go:165
#       0xaa7e35        golang.org/x/sync/errgroup.(*Group).Go.func1+0x55               /go/src/linuxapp/vendor/golang.org/x/sync/errgroup/errgroup.go:78

[478254406]

I noticed that the memory usage is still slowly increasing.

[swdee]

Ok, can you show us the code from file /go/src/linuxapp/logic/denoiseHandel.go particularly the function CallModelWithCrop().

Would need to see the code where it creates the Mat, how it uses it, and where its closed.

[478254406]

			subImg := inputMat
			if bottomPad > 0 || rightPad > 0 {
				subImg = gocv.NewMatWithSize(InputSize, InputSize, inputMat.Type())
				gocv.CopyMakeBorder(inputMat, &subImg, 0, bottomPad, 0, rightPad, gocv.BorderReflect, color.RGBA{R: 0, G: 0, B: 0, A: 0})
				_ = inputMat.Close()
			}

I ran the memory leak detection code and found that the line subImg = gocv.NewMatWithSize(InputSize, InputSize, inputMat.Type()) has a memory leak.

where in use
···
subPicInfo := &SubImgInfo{
InImg: &subImg,
X1: outputStartX,
Y1: outputStartY,
X2: outputEndX,
Y2: outputEndY,
OutX1: outputStartXTile,
OutY1: outputStartYTile,
OutX2: outputEndXTile,
OutY2: outputEndYTile,
Type: picInfo.Type,
}
subImgInfoList = append(subImgInfoList, subPicInfo)
}
}
err := d.CallModelConcurrently(context.Background(), subImgInfoList, url)
···
In the CallModelConcurrently function, I only used subImgInfoList, InImg's Cols, Rows, Channels, and ToBytes() functions.

··
defer d.ReleaseSubResource(ctx, subImgInfoList)

func (d *DenoiseHandelImpl) ReleaseSubResource(ctx context.Context, subPicInfos []*SubImgInfo) {
for _, subPicInfo := range subPicInfos {
if subPicInfo.OutImg != nil {
_ = subPicInfo.OutImg.Close()
}
if subPicInfo.InImg != nil {
subPicInfo.InImg.Close()
}

}

}
···
At the end of the function, I will release the Mat memory.

[478254406]

@swdee thank you very much

[swdee]

On this code

			subImg := inputMat
			if bottomPad > 0 || rightPad > 0 {
				subImg = gocv.NewMatWithSize(InputSize, InputSize, inputMat.Type())
				gocv.CopyMakeBorder(inputMat, &subImg, 0, bottomPad, 0, rightPad, gocv.BorderReflect, color.RGBA{R: 0, G: 0, B: 0, A: 0})
				_ = inputMat.Close()
			}

It looks to me that your copying inputMat to subImg, then later creating a NewMatWithSize to subImg. However inputMat is being closed whilst subImg is left open. Hence the memory leak.

So I gather you have resolved this issue now?

[478254406]

subImg is subsequently assigned to subImgInfoList and will be closed collectively later.