Large multipart file being read into memory in SpringSingleMultipartFileWriter making OOM issue

[YEPing02]

Hello,

When uploading a multipart file via a feign client, the OOM is raised when the file is larger than 500 Mbs (for example).

Log shows that the exception occurs at:
https://github.com/OpenFeign/feign-form/blob/master/feign-form-spring/src/main/java/feign/form/spring/SpringSingleMultipartFileWriter.java#L45

@Override
  protected void write (Output output, String key, Object value) throws EncodeException {
    val file = (MultipartFile) value;
    writeFileMetadata(output, key, file.getOriginalFilename(), file.getContentType());

    byte[] bytes;
    try {
      bytes = file.getBytes();
    } catch (IOException ex) {
      throw new EncodeException("Getting multipart file's content bytes error", ex);
    }
    output.write(bytes);
  }

the write method reads all the bytes of the file into memory.

The solution to avoid OOM would be just to stream the file into the output, could we get this fix?

Thanks

[bbulgarelli-yseop]

Yes, this has also been opened before here: #88
Honestly, this is really annoying when you have to handle large file transfers...

I'm also waiting for some fix or even any comment by maintainers of this repo.
I can even do a pull request if needed.

I made a workaround by writing a custom writer for MultipartFile that is only based on the getInputStream method, and another one handling MultipartFile[].
Then create a custom SpringFormEncoder and add the writers as the first priority ones to get the correct behavior:
In the constructor

        MultipartFormContentProcessor processor = (MultipartFormContentProcessor) getContentProcessor(MULTIPART);
        processor.addFirstWriter(new CustomStreamManyMultipartFilesWriter(STREAM_BUFFER_SIZE));
        processor.addFirstWriter(new CustomStreamSingleMultipartFileWriter(STREAM_BUFFER_SIZE));

[SanhongWong]

Yes, this has also been opened before here: #88 Honestly, this is really annoying when you have to handle large file transfers...

I'm also waiting for some fix or even any comment by maintainers of this repo. I can even do a pull request if needed.

I made a workaround by writing a custom writer for MultipartFile that is only based on the getInputStream method, and another one handling MultipartFile[]. Then create a custom SpringFormEncoder and add the writers as the first priority ones to get the correct behavior: In the constructor

        MultipartFormContentProcessor processor = (MultipartFormContentProcessor) getContentProcessor(MULTIPART);
        processor.addFirstWriter(new CustomStreamManyMultipartFilesWriter(STREAM_BUFFER_SIZE));
        processor.addFirstWriter(new CustomStreamSingleMultipartFileWriter(STREAM_BUFFER_SIZE));

Hi, bro，I encounter the same problem, but i guess your code can't solve it，because in the feign.form.MultipartFormContentProcessor#process method end，it also read all into the memory，just like this：

    val bytes = output.toByteArray();
    val body = Request.Body.encoded(bytes, null);
    template.body(body);

[bbulgarelli-yseop]

Oh I didn't see this one...
I guess we are stuck now.
This comes from a deep feign-core explicit byte[] into Request.Body: OpenFeign/feign#1243
It looks like Feign core itself doesn't support upload file streaming...

[xxlabaza]

Yep, Feign itself doesn't support streaming and doesn't provide any API (like OutputStream) to resolve it