App-NotFound and events missing for some applications

[jameshochadel]

Issue

For some applications in our foundation, users observe two problems:

1. They cannot SSH to the apps using the v6 CLI cf ssh command. The v6 CLI's cf v3-ssh command works fine, and CLI v8 works as well.
2. The affected apps do not emit events of any kind related to SSH, successful or otherwise, regardless of CLI version.

Context

Attempting to SSH prints this error:

FAILED
Error opening SSH connection: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

Running cf -v ssh appname shows the last request before the failure is GET /login?code=[PRIVATE DATA HIDDEN], and CAPI responds with a HTTP/1.1 302 Found.

In the Cloud Controller logs, I find the following error:

Request failed: 404: {
  "description"=>"The app could not be found: GUID", 
  "error_code"=>"CF-AppNotFound", 
  "code"=>100004, 
  "test_mode_info"=>{"description"=>"The app could not be found: GUID", 
  "error_code"=>"CF-AppNotFound", 
  "backtrace"=>[
    "/var/vcap/data/packages/cloud_controller_ng/3a803a3058638bf5336b89efd4df70607dca404b/cloud_controller_ng/app/controllers/base/model_controller.rb:319:in `find_guid'", 
    "/var/vcap/data/packages/cloud_controller_ng/3a803a3058638bf5336b89efd4df70607dca404b/cloud_controller_ng/app/controllers/base/model_controller.rb:312:in `find_guid_and_validate_access'", 
    "/var/vcap/data/packages/cloud_controller_ng/3a803a3058638bf5336b89efd4df70607dca404b/cloud_controller_ng/app/controllers/runtime/apps_ssh_controller.rb:30:in `ssh_access_with_index'", 
    "/var/vcap/data/packages/cloud_c

(GUID is the app GUID, omitted by me.) Our log system truncated the stack trace, but the important part is included: The failure is on app/controllers/base/model_controller.rb:319, which throws an exception if the L318 call to find_model.find(guid:) returns nil. In this context, I presume find_model is meant to be an instance of AppModel, but I'm not sure how exactly that works — my Ruby is a little rusty.

Assuming I have the model correct, I SSH'd into a cloud_controller_ng API node and accessed the console via /var/vcap/jobs/cloud_controller_ng/bin/console. I can call app = AppModel.find(guid: "GUID") and the app object is returned as expected. So my suspicion is that some other model is being mistakenly called here. As an additional data point, I can get app data from cf curl v2/apps/GUID and cf curl v3/apps/GUID just fine.

Steps to Reproduce

Unfortunately I don't know how to reproduce the issue. The apps look identical in the CCDB, and their manifests are nearly identical, with only one line difference:

---
applications:
- name: my-app
  buildpacks:
  - python_buildpack
  stack: cflinuxfs4
  services:
  - my-service
  processes:
  - type: web
    instances: 1
    memory: 256M
    disk_quota: 1024M
    log-rate-limit-per-second: -1
    health-check-type: port
    health-check-http-endpoint: '' # appears on the failing app only
    readiness-health-check-type: process

Expected result

I can SSH to both apps, since they appear identical.

Current result

I cannot SSH to some apps. (Details above.)

Possible Fix

Not sure — I'd welcome tips on additional debugging steps.