-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Web server isn't returning the proper responses #283
Comments
@srudin 👋 that may be coming from the rails server. Do you think you get a shell inside the container and check the contents of |
The complete output in this file is: [2023-01-20T10:44:19.749042 #63] INFO -- : 2023-01-20T10:44:19+0000: [Worker(host:poc-isa-osm-test-osm-seed-web-6d748b69b4-th4f5 pid:63)] Starting job worker |
@srudin thanks for reporting this - this whole build process does tend to be a bit flaky / prone to breakage when versions of some libraries change / upstream makes minor changes, etc. that we try and fix, but it's totally possible you've run into a legitimate issue that we need to fix. Would you be able to provide the exact steps you ran in order to get this error so we can try and reproduce on our end? Unfortunately I can't guarantee us getting to it immediately, but if we're able to reproduce, and it is something broken somewhere in the ruby build process, we'd definitely try and get to it as soon as possible. @Rub21 once we understand the exact steps to reproduce this issue, do you think you might be able to find some time next week to look and see if there's something broken here again? |
I am using the container with tag 1.0.0-dev.hdebc44c. I am doing a helm deployment on k8/Argo. Currently only db and web are enabled. Inside the helm deployment there are a number of configuration changes I had to make:
I understand there are a number of things that could impact the startup here. However there are usually error messages showing what the problem is, e.g. when readonly access is required it says so, if network access is not working I cannot connect etc. In this case it is starting up, there are no error messages but it still doesn't work. From my understanding it is compiling some ruby code and starting the ruby app, then launching apache and redirecting the requests from apache to ruby, correct? Is there some way I can test that locally (I mean in the terminal of the pod)? I can run Some more details: When I run /etc/apache2/ports.conf:
/etc/apache2/sites-enabled/production.conf:
|
@srudin - ha, unfortunately, since this uses Again, it's likely that ideally this would be setup quite differently, to output logs to This does sound frustrating - perhaps I'd recommend just trying to run the container locally, and perhaps seeing it work locally will make it easier to debug remotely? Unfortunately, I think our method of debugging issues is to ssh into the container and look at log-files that are written to various locations as I don't think errors are always reliably printed to If you need a list of all the places where log-files could hold errors, I can poke around our docs / try and refresh my memory, or @Rub21 probably knows a bit more for certain. |
Well I am in the terminal. I am running the commands manually including --trace. No error. But apache fails. |
After a lot of debugging I found the problem(s):
It was really difficult to find these issues as error messages were only displayed in trace mode or log files. Also I have to say that I am not yet 100% sure if ALL of these fixes are really required, I will test it now and keep only the necessary changes. |
As discussed here: #281
I am currently running into the issue that the web container is returning "Internal server error - Application could not be started. Please try again later."
Here comes the weird part: If the /tmp and /var/www/tmp folders are readonly the apache server returns the file system view of /var/www/public. However if I make the above mentioned folders writeable I get the above error message.
I am somewhat clueless what to do about that. I can see that start.sh is calling "bundle exec rake jobs:work & apachectl -k start -DFOREGROUND". I have enabled tracing for these calls but I get no error message and the same output no matter if the temp folders are readonly or not. I can also make the calls manually in the terminal window and they seem to succeed. It feels like something is wrong with the url rewriting and that only happens when the ruby app is running and that only happens when the compiling succeeded and that only happens when the temp folders are writeable... but just guessing here, how to really find out? I can only see that connections to port 8000 are refused and in the cgi config file it says the app should be running on that port but not sure if this is relevant at all (is the ruby app using the cgi config? Or what port should it run on?). I am a .NET/Java guy so ruby/rake/apache doesn't really feel like home... :-(
Any ideas or suggestions how to debug this? Btw the output is:
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
/var/www/config/initializers/config.rb:19: warning: already initialized constant ALLOWED_STATUS
/var/www/config/initializers/config.rb:19: warning: previous definition of ALLOWED_STATUS was here
** Execute db:load_config
** Execute db:migrate
/var/lib/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/postgresql_adapter.rb:78: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/pg-1.3.1/lib/pg.rb:68: warning: The called method "connect" is defined here
** Invoke db:_dump (first_time)
** Execute db:_dump
[Fri Jan 20 09:55:57.719805 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module access_compat_module from /usr/lib/apache2/modules/mod_access_compat.so
[Fri Jan 20 09:55:57.720127 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module alias_module from /usr/lib/apache2/modules/mod_alias.so
[Fri Jan 20 09:55:57.720388 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module auth_basic_module from /usr/lib/apache2/modules/mod_auth_basic.so
[Fri Jan 20 09:55:57.720616 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module authn_core_module from /usr/lib/apache2/modules/mod_authn_core.so
[Fri Jan 20 09:55:57.720811 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module authn_file_module from /usr/lib/apache2/modules/mod_authn_file.so
[Fri Jan 20 09:55:57.721279 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module authz_core_module from /usr/lib/apache2/modules/mod_authz_core.so
[Fri Jan 20 09:55:57.721478 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module authz_host_module from /usr/lib/apache2/modules/mod_authz_host.so
[Fri Jan 20 09:55:57.721872 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module authz_user_module from /usr/lib/apache2/modules/mod_authz_user.so
[Fri Jan 20 09:55:57.726018 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module autoindex_module from /usr/lib/apache2/modules/mod_autoindex.so
[Fri Jan 20 09:55:57.726779 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module deflate_module from /usr/lib/apache2/modules/mod_deflate.so
[Fri Jan 20 09:55:57.727145 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module dir_module from /usr/lib/apache2/modules/mod_dir.so
[Fri Jan 20 09:55:57.727472 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module env_module from /usr/lib/apache2/modules/mod_env.so
[Fri Jan 20 09:55:57.727809 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module filter_module from /usr/lib/apache2/modules/mod_filter.so
[Fri Jan 20 09:55:57.728221 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module mime_module from /usr/lib/apache2/modules/mod_mime.so
[Fri Jan 20 09:55:57.728612 2023] [so:debug] [pid 67] mod_so.c(266): AH01575: loaded module mpm_event_module from /usr/lib/apache2/modules/mod_mpm_event.so
[Fri Jan 20 09:55:57.728971 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module negotiation_module from /usr/lib/apache2/modules/mod_negotiation.so
[Fri Jan 20 09:55:57.733143 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module passenger_module from /usr/lib/apache2/modules/mod_passenger.so
[Fri Jan 20 09:55:57.733637 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module proxy_module from /usr/lib/apache2/modules/mod_proxy.so
[Fri Jan 20 09:55:57.734049 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module proxy_http_module from /usr/lib/apache2/modules/mod_proxy_http.so
[Fri Jan 20 09:55:57.734436 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module reqtimeout_module from /usr/lib/apache2/modules/mod_reqtimeout.so
[Fri Jan 20 09:55:57.734767 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module rewrite_module from /usr/lib/apache2/modules/mod_rewrite.so
[Fri Jan 20 09:55:57.735274 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module setenvif_module from /usr/lib/apache2/modules/mod_setenvif.so
[Fri Jan 20 09:55:57.736035 2023] [so:debug] [pid 67:tid 140394999651392] mod_so.c(266): AH01575: loaded module status_module from /usr/lib/apache2/modules/mod_status.so
** Invoke jobs:work (first_time)
** Invoke jobs:environment_options (first_time)
** Invoke environment (first_time)
** Execute environment
/var/www/config/initializers/config.rb:19: warning: already initialized constant ALLOWED_STATUS
/var/www/config/initializers/config.rb:19: warning: previous definition of ALLOWED_STATUS was here
** Execute jobs:environment_options
** Execute jobs:work
/var/lib/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/connection_adapters/postgresql_adapter.rb:78: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/pg-1.3.1/lib/pg.rb:68: warning: The called method "connect" is defined here
Waiting to start rails ports server...
Waiting to start rails ports server...
The text was updated successfully, but these errors were encountered: