Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase request limits for POST / PUT datasets #142

Open
jnehring opened this issue Dec 1, 2015 · 6 comments
Open

Increase request limits for POST / PUT datasets #142

jnehring opened this issue Dec 1, 2015 · 6 comments
Labels
enhancement

Comments

@jnehring
Copy link
Member

jnehring commented Dec 1, 2015

  • 200 MB
  • 1 hour
@jnehring jnehring self-assigned this Dec 1, 2015
@jnehring jnehring added this to the FREME 0.5 milestone Dec 1, 2015
@jnehring
Copy link
Member Author

jnehring commented Dec 2, 2015

I wrote some tests and commited them to the "timeout test branch" in the broker repo.

  • There is a new endpoint "/e-test" that takes the POST body and writes it to a file
  • There is a test client in "eu.freme.broker.Test" that reads a local file (in my tests of 100 MB) and sends it to the /e-test endpoint. Every 5 MB it waits for 10 seconds

When I start first the broker and then the test client then it sends the file to the test endpoint. The request takes 200 seconds and submits a 100 MB file.

So I think we do not have to increase the request limits because they are already very high.

@m1ci
Copy link
Contributor

m1ci commented Dec 7, 2015

When I start first the broker and then the test client then it sends the file to the test endpoint. The request takes 200 seconds and submits a 100 MB file.

Yes, you managed to submit 100 MB in 200 seconds since you have relatively fast internet. The internet speed of some clients might be low as 50-100 KB/sec

IMO, we still need to consider increasing the time and size limit for processing requests.

@jnehring
Copy link
Member Author

jnehring commented Dec 7, 2015

I think that right now there a maximum request size of 2000 MB and no time limit. But we should do some big testing, I am sure that in some parts of FREME large requests will break the system.

@jnehring
Copy link
Member Author

jnehring commented Jan 5, 2016

@ArneBinder please try to submit large files to FREME NER. Create textfiles of 50 MB, 100 MB, 500 MB and 1 GB size and try to submit them to FREME NER on freme-dev.

@jnehring jnehring assigned ArneBinder and unassigned jnehring Jan 5, 2016
@ArneBinder
Copy link
Contributor

A request to FREME NER with 500KB of plain text already raises:

{
  "exception": "eu.freme.broker.exception.ExternalServiceFailedException",
  "path": "/e-entity/freme-ner/documents",
  "error": "Bad Gateway",
  "status": 502,
  "timestamp": 1452014214010
}

50MB plain text:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at api-dev.freme-project.eu Port 80</address>
</body></html>

3.3MB of turtle sent to e-Link, freme-ner or the filter controller produces the same.

@jnehring jnehring assigned jnehring and unassigned ArneBinder Feb 2, 2016
@jnehring jnehring removed this from the FREME 0.5 milestone Feb 2, 2016
@jnehring
Copy link
Member Author

jnehring commented Jun 7, 2016

Please try large requests for

  • FREME NER
  • e-Translation
  • NIF converter

The results of this test may influence FREME configuration documentation multipart.maxFileSize
and multipart.maxRequestSize

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@m1ci @ArneBinder @jnehring