O+M 05-19-2023

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Tracking Update
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[nickumia-reisys]

Day 1

• FCC Data.json was able to harvest on 5/14. It had errors still, but it got past the connection error from last week.
• DMARC Reports:
  • Google 5/15

    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  pass     spf:  pass ses-513xxxx.ssb.data.gov
    Success Rate 423/423 = 1.00

  • Outlook 5/14

    ...
    x.x.x.x         dkim:  fail     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  fail     spf:  softfail ses-513xxxx.ssb.data.gov ['ses-513xxxx.ssb.data.gov', 'amazonses.com'] mail.ses-513xxxx.ssb.data.gov
    x.x.x.x         dkim:  fail     spf:  pass ses-513xxxx.ssb.data.gov [] epa.gov
    ...
    Success Rate 94/218 = 0.43

• Catalog Tracking Update is back down to a reasonable number. We still don't know the cause of the spike, but it's down to 70k.
• Fixed Catalog Solr Core Locked alert in NR.
  • By default, alerts are closed after 72 hours.
  • Paired with @FuhuXia and determined that the "lost signal" threshold was related to the alert condition and not the overall health of the app. As a result, set up automatic close of this alert if the "signal is lost".
  • References:
    • https://docs.newrelic.com/docs/alerts-applied-intelligence/new-relic-alerts/alert-violations/how-alert-condition-violations-are-closed/#close-automatic
    • https://docs.newrelic.com/docs/new-relic-solutions/get-started/glossary/#recovery-period
    • https://forum.newrelic.com/s/hubtopic/aAX8W0000008ZEzWAM/auto-close-the-alerts-once-condition-is-not-met
    • https://forum.newrelic.com/s/hubtopic/aAX8W0000008YrIWAU/relic-solution-alert-incident-preferences-are-the-key-to-consistent-alert-noti
    • https://docs.newrelic.com/docs/alerts-applied-intelligence/new-relic-alerts/alert-conditions/create-nrql-alert-conditions/#condition-tips

[nickumia-reisys]

Day 2

• Inventory went down from 3:37a to 4:36a... not sure of the cause, but it seems to be back up and healthy.
• Did running the dedupe script help reduce the db-solr-sync number of operations? (the world may never know 🤯)
  

[nickumia-reisys]

Day 3 + 4

• Catalog Solr Follower 1 restarted @ 2:41a 5/18
• Catalog Solr Follower 2 restarted @ 4:23a 5/18
• Catalog Solr Follower 0 restarted @ 4:26a 5/18
• Random flakiness in our github action pipelines.. but nothing that interferred long-term with restarts/actions.
• 5/18: db-solr-sync and tracking-update took twice the time to execute on the same number of record changes...
• Incident with federation.data.gov (record removed from dns by error)
  • Original alert: https://gsa-tts.slack.com/archives/C4RGAM1Q8/p1684430100691159

[nickumia-reisys]

Day 5

• Catalog Solr Follower 0 restarted @ 7:42a
• Catalog Solr Follower 2 restarted @ 9:39a
• Both db-solr-sync and tracking-update are showing persistent slowdowns...
  
  
• No meaningful harvesting issues this week.
• Assessed Upcoming Elastic Load Balancing API changes: https://gsa-tts.slack.com/archives/C2N85536E/p1684526961142519