Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix leak in FlatMapPrefix operator. #1622

Merged
merged 1 commit into from
Jan 3, 2025
Merged

Conversation

He-Pin
Copy link
Member

@He-Pin He-Pin commented Dec 22, 2024

extracted from #1621

Motivation:
In #1566 , @queimadus points out a leak

Modification:

  1. change the builder from a buffer to a vector builder.
  2. free that builder to null after the prefix is generated.

Result:
leak fixed.

import org.apache.pekko.actor.ActorSystem
import org.apache.pekko.stream.scaladsl.{Flow, Sink, Source}
import org.apache.pekko.util.ByteString

import scala.concurrent.Await


object PekkoQuickstart extends App {
  private implicit val system: ActorSystem = ActorSystem()

  val s = Source
    .repeat(())
    .map(_ => ByteString('a' * 400000))
    .take(1000000)
    .flatMapPrefix(50000) { _ => Flow[ByteString] }

  val r = Source.empty
    .concatAllLazy(List.tabulate(30000)(_ => s): _*)
    .runWith(Sink.ignore)

  Await.result(r, scala.concurrent.duration.Duration.Inf)
  println(r.value)

//  Source
//    .repeat(s)
//    .take(30000)
//    .flatMapConcat(x => x)
//    .runWith(Sink.ignore)
//    .onComplete(println(_))

//  Source.empty
//    .concatAllLazy(List.tabulate(30000)(_ => Source.lazySource(() => s)): _*)
//    .runWith(Sink.ignore).onComplete(println(_))
}

@He-Pin He-Pin added t:stream Pekko Streams bug Something isn't working backport labels Dec 22, 2024
@He-Pin He-Pin added this to the 1.1.3 milestone Dec 22, 2024
@He-Pin He-Pin changed the title chore: Fix leak in FlatMapPrefix operator. fix: Fix leak in FlatMapPrefix operator. Dec 22, 2024
@He-Pin
Copy link
Member Author

He-Pin commented Dec 28, 2024

@pjfanning Would you like to give this a review?

@pjfanning
Copy link
Contributor

@pjfanning Would you like to give this a review?

Would it be possible to unit test this? We probably don't want anything that takes too much time or memory to run but it might be feasible to add some extra test coverage.

@He-Pin He-Pin removed the backport label Jan 3, 2025
@He-Pin
Copy link
Member Author

He-Pin commented Jan 3, 2025

I think I can't test it because the builder is protected inside the logic, otherwise, I have to extract the code to a dedicated class and expose it to tests.

Another way is using the MBean, but that is too much.

@He-Pin He-Pin force-pushed the fixPrefixMap branch 2 times, most recently from 9373a45 to bc37570 Compare January 3, 2025 15:39
@He-Pin He-Pin requested a review from raboof January 3, 2025 15:47
@He-Pin He-Pin merged commit ce3620f into apache:main Jan 3, 2025
9 checks passed
@He-Pin He-Pin deleted the fixPrefixMap branch January 3, 2025 16:56
@He-Pin
Copy link
Member Author

He-Pin commented Jan 3, 2025

@queimadus Thanks for the reporting, this do spend me a weekend to work out:)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working t:stream Pekko Streams
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants