IIR - Update VAPRO Contact Lookup to Filter by Valid Phone Numbers

[mjones-oddball]

User Story - Business Need

Currently, VA Notify attempts to deliver SMS to any mobile phone number found in VA Profile. Sometimes these SMS fail because the "mobile" phone recorded in VA Profile is actually a landline or other invalid type for texting. Rather than attempting to text a number that will not succeed, we should use the phone validation data provided by VAPRO to determine if it makes sense to proceed to delivery, or fail and alert the client (via callbacks) that the phone number was invalid - thus making the person unreachable by SMS.

See IIR-900

• Ticket is understood, and QA has been contacted (if the ticket has a QA label).

User Story(ies)

As a Business
I want to know if the Veteran is unreachable by SMS due to an invalid phone number
So that I can ensure they are receiving the information I'm sending them, and determine if alternate communications are required

As VA Notify
I want to determine if a phone number is valid before proceeding to delivery
So that I can efficiently process notifications, and deliver status information back to businesses ASAP if a recipient is unreachable

Additional Info and Resources

• Parent spike 792 with details in the comments
• Profile API Notes
• Metadata about phone numbers may be available from the Profile API, which would allow us to confirm that a number is a cell phone. This lives in the response to a Contact Information request under telephones, then in the TelephoneResponse object we're interested in classification then classificationCode.
• Classifications:

{
  "classificationList": [
    {
      "classificationCode": 0,
      "classificationName": "MOBILE"
    },
    {
      "classificationCode": 1,
      "classificationName": "LANDLINE"
    },
    {
      "classificationCode": 2,
      "classificationName": "VOIP"
    },
    {
      "classificationCode": 3,
      "classificationName": "INVALID"
    },
    {
      "classificationCode": 4,
      "classificationName": "OTHER"
    },
    {
      "classificationCode": 5,
      "classificationName": "PREPAID"
    }
  ]
}

Acceptance Criteria

• Valid phone number proceeds with delivery

GIVEN VA Profile returns a mobile phone number
WHEN the classification is mobile (0), voip (2), prepaid (5)
OR a classification is not provided
THEN proceed to SMS delivery

• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable

GIVEN VA Profile returns a mobile phone number
WHEN the classification is landline (1), invalid (3), or other (4)
THEN raise a VAProfileNonRetryableException with status: permanent-failure and status_reason: Phone number is invalid

QA Considerations

Potential Dependencies

[mjones-oddball]

Noting that we determined other was not a valid type because of AWS's reply:
Occasionally, the response from the phone number validation service includes a PhoneType value of OTHER. The service might return this kind of response in the following situations:

• The phone number is a toll-free (freephone) number.
• The phone number is reserved for use in TV shows and movies, such as North American phone numbers that begin with 555.
• The phone number includes an area code that is not currently in use, such as the 999 area code in North America.
• The phone number is reserved for some other purpose.

VA Profile said there were currently 32,379 flagged as OTHER as of 9/3.

Other breakdowns below:

[cris-oddball]

@EvanParish and @AdamKing0126 to sync on this before the branch is deployed, then Evan will sync with me for testing notes.

[cris-oddball]

Evan touched base with me about testing. We discussed:

• setting the feature flag to False in Perf, Staging, Prod, that way I can turn it on in Perf when able to test and can ensure that with the test flag as false, everything works as expected
• adding a task definition file for prod now
• adding celery task definitions.

[cris-oddball]

PR is approved and merged. Sending this to Perf and holding it there until Monday.

[cris-oddball]

On Perf, with Flag disabled, I have verified that Pinpoint is still handling Phone Validation failures by changing a test user's contact Mobile to a prefix 555 number and observing in the logs a Pinpoint event_type=_SMS.FAILURE and observing the test logs for the feature:

V3 Profile -- VA_PROFILE_V3_IDENTIFY_MOBILE_TELEPHONE_NUMBERS is not enabled. Will not check classification info.

Will deploy this up to Prod (staging and prod feature flags are off), then cut a branch to flip Perf on and test that the feature flag works as expected.

[cris-oddball]

This was sent up to Staging/Prod.
Will find time to cut a branch to turn the FF on in Perf and test, but this ticket may carry until tomorrow.

[cris-oddball]

Deployed branch out to Perf with the Feature Flag enabled for telephone numbers.

When the regression ran, found the following logs showing that the new feature is in use for all sms records sent:

Can begin testing the feature now.

[cris-oddball]

QA FAILED

Happy Path
Expect status: delivered

• Valid phone number proceeds with delivery for a mobile number - Using my own mobile
• Valid phone number proceeds with delivery for a VOIP number "+18042221111"
• Valid phone number proceeds with delivery for a Prepaid number (unable to test as do not have a prepaid number available)

Negative Test Cases

• expect the phone to be validated by VAPRO and not the SMS sender.
• expect a callback with the correct info sent if callback is on the service
• expect for failures: 'status': 'permanent-failure' and 'status_reason': 'Phone number is invalid'
  
  
  
• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for landline - Using test landline "+16317918378"
• This test failed in that it passed the VAPRO telephone lookup and was sent to Pinpoint, who then returned a temporary-failure. This replicates what happened when I sent to this number directly in Postman and it's possible that this is an edge case with the VAPRO validation service. Texting this number from my mobile resulted in a message that if I wanted to pay extra money, I could then send a text to this landline.
  
  
  
• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for other/invalid number (number used in the US for television) "17195551212"
• This test failed in that the status reason was "No contact info found from VA Profile" when I expected something about an invalid number (?) There is also nothing set in the GET response phone_number field.
• The logs show that a lookup was made to VAPRO but then a log is

app.va.va_profile.exceptions.NoContactInfoException: No telephones in response for VA Profile ID <RecipientIdentifier d043bde5-787d-4408-8863-ea3c79a6e045, VAPROFILEID, 1550116> with AuditId DATA SEEDING

• Presently, when Pinpoint identifies a bad number, we get a status reason of Phone number is invalid (the status is also technical-failure, not permanent-failure, but this one came back as a permanent-failure.
  
  
  
• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for other/invalid tollfree "+18002211212" (Delta Airlines)
• Same as above, the lookup to VAPRO returns no contact info, there is no phone_number on the GET request, the response is "status": "permanent-failure", and "status_reason": "No contact info found from VA Profile",
  
  
  
• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for other/invalid tollfree "+19991234567" (999 is not a valid area code)
• Unable to validate because the app itself will not allow a veteran to change their number to a non-valid US area code.
• 

[cris-oddball]

The fix for the above failures is being sent up to Prod right now. Feature Flag is off in Perf, Staging, Prod.

When I have time, I will set the FF to True in Perf and retest. Might be today (Thurs) or tomorrow (Fri).

[cris-oddball]

QA PASSED

Enabled FF in Perf on a branch

Happy Path
Expect status: delivered

• Valid phone number proceeds with delivery for a mobile number - Using my own mobile
• Valid phone number proceeds with delivery for a VOIP number "+18042221111"
• Valid phone number proceeds with delivery for a Prepaid number
• unable to test as do not have a prepaid number available

Negative Test Cases

• expect the phone to be validated by VAPRO and not the SMS sender.
• expect a callback with the correct info sent if callback is on the service
• expect for failures: 'status': 'permanent-failure' and 'status_reason': 'Phone number is invalid'
  
  
  
• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for landline - Using test landline "+16317918378" Turns out that this number is a VOIP number. Instead found a personal real landline number (not posting here)

"529eb881-d3b7-42e5-9cb1-e0dd0216084f",
'status': 'permanent-failure', 'status_reason': 'Phone number is invalid'
[2024-10-03 23:11:50,330: DEBUG/ForkPoolWorker-16] V3 Profile -- Phone classification code of 1 is not a valid SMS recipient (VA Profile ID: 1550116)

• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for other/invalid number (number used in the US for television) "17195551212"

"b11c4e5c-343f-40d7-904c-586c4b922a9b"
'status': 'permanent-failure', 'status_reason': 'Phone number is invalid'
[2024-10-03 23:17:36,463: DEBUG/ForkPoolWorker-16] V3 Profile -- Phone classification code of 4 is not a valid SMS recipient (VA Profile ID: 1550116)

• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for other/invalid tollfree "+18002211212" (Delta Airlines)

"75cc3aad-bfd7-4c51-9aa4-c4ef4bfe0eb3"
'status': 'permanent-failure', 'status_reason': 'Phone number is invalid',
[2024-10-03 23:22:13,437: DEBUG/ForkPoolWorker-16] V3 Profile -- Phone classification code of 4 is not a valid SMS recipient (VA Profile ID: 1550116)

• Invalid phone number fails prior to delivery and status is sent to client via callback if applicable for other/invalid tollfree "+19991234567" (999 is not a valid area code)
• Unable to validate because the app itself will not allow a veteran to change their number to a non-valid US area code.
  
  
  
• No mobile number listed

'status': 'permanent-failure', 'status_reason': 'No contact info found from VA Profile'

Just making sure that we are using some other phone number (like the WORK number this user has, which happens to also be a mobile number).

[cris-oddball]

Closing this ticket as Done, noting that before the next set of tickets can be worked (which remove the feature flags) we probably want to enable the feature flags for both features all the way up to Prod.

@EvanParish @k-macmillan