Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove references to the old datagov-deploy repo #1143

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ You should now be able to visit `https://[ROUTE]`, where `[ROUTE]` is the route
### Secrets

ips on managing
[secrets](https://github.com/GSA/datagov-deploy/wiki/Cloud.gov-Cheat-Sheet#secrets-management).
[secrets](https://github.com/GSA/data.gov/wiki/cloud.gov#secrets-management).
When creating the service for the first time, use `create-user-provided-service`
instead of update.

Expand All @@ -142,12 +142,12 @@ SAML2_PRIVATE_KEY | Base64 encoded SAML2 key matching the certificate configured
## Login.gov integration

We use Login.gov as our
[SAML2](https://github.com/GSA/datagov-deploy/wiki/SAML2-authentication)
[SAML2](https://github.com/GSA/data.gov/wiki/SAML2-authentication)
Identity Provider (IdP). Production apps use the production Login.gov instance
while other apps use the Login.gov identity sandbox.

Each year in March, Login.gov rotates their credentials. See our
[wiki](https://github.com/GSA/datagov-deploy/wiki/SAML2-authentication#working-with-logingov)
[wiki](https://github.com/GSA/data.gov/wiki/SAML2-authentication#working-with-logingov)
for details.

Our Service Provider (SP) certificate and key are provided in through
Expand Down
2 changes: 1 addition & 1 deletion ckan/setup/ckan.ini
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ ckanext.harvest.email = on
ckan.harvest.status_mail.all=True

## SAML2auth Settings
# TODO fetch and verify remote metadata https://github.com/GSA/datagov-deploy/issues/2860
# TODO fetch and verify remote metadata https://github.com/GSA/data.gov/issues/2860
ckanext.saml2auth.idp_metadata.location=local
#ckanext.saml2auth.idp_metadata.local_path = $CKANEXT__SAML2AUTH__IDP_METADATA__LOCAL_PATH
#ckanext.saml2auth.key_file_path = $CKANEXT__SAML2AUTH__KEY_FILE_PATH
Expand Down
5 changes: 2 additions & 3 deletions ckan/setup/gunicorn.conf.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# gunicorn prior to version 20.1.0 discloses "server" header
# https://github.com/GSA/datagov-deploy/issues/2826
import gunicorn
gunicorn.SERVER_SOFTWARE = ''

gunicorn.SERVER_SOFTWARE = ""
74 changes: 37 additions & 37 deletions e2e/cypress/integration/facets.cy.js
Original file line number Diff line number Diff line change
@@ -1,43 +1,43 @@
describe('Facets', { testIsolation: false }, () => {
before(() => {
cy.login();
});
describe("Facets", { testIsolation: false }, () => {
before(() => {
cy.login();
});

after(() => {
cy.delete_organization('org-tags');
cy.delete_group('group-facets');
cy.logout();
});
after(() => {
cy.delete_organization("org-tags");
cy.delete_group("group-facets");
cy.logout();
});

it('Show datagov facet list on dataset page', () => {
cy.visit('/dataset');
cy.get('.filters h2').its('length').should('be.equal', 9);
cy.get('.filters h2').first().contains('Topics');
cy.get('.filters h2').last().contains('Bureaus');
});
it("Show datagov facet list on dataset page", () => {
cy.visit("/dataset");
cy.get(".filters h2").its("length").should("be.equal", 9);
cy.get(".filters h2").first().contains("Topics");
cy.get(".filters h2").last().contains("Bureaus");
});

it('Show datagov facet list on organization page', () => {
cy.visit('/organization');
cy.get('a[class="btn btn-primary"]').click();
cy.create_organization_ui('org-tags', 'tags for org test');
cy.visit('/organization/org-tags');
cy.get('.module-shallow').its('length').should('be.equal', 11);
cy.get('.module-shallow').contains('Topics');
cy.get('.module-shallow').contains('Harvest Source');
cy.get('.module-shallow').last().contains('Bureaus');
});
it("Show datagov facet list on organization page", () => {
cy.visit("/organization");
cy.get('a[class="btn btn-primary"]').click();
cy.create_organization_ui("org-tags", "tags for org test");
cy.visit("/organization/org-tags");
cy.get(".module-shallow").its("length").should("be.equal", 11);
cy.get(".module-shallow").contains("Topics");
cy.get(".module-shallow").contains("Harvest Source");
cy.get(".module-shallow").last().contains("Bureaus");
});

it('Show datagov facet list on group page', () => {
cy.create_group('group-facets');
cy.visit('/group/group-facets');
cy.get('.filters h2').its('length').should('be.equal', 6);
cy.get('.filters h2').first().contains('Categories');
cy.get('.filters h2').last().contains('Organizations');
});
it("Show datagov facet list on group page", () => {
cy.create_group("group-facets");
cy.visit("/group/group-facets");
cy.get(".filters h2").its("length").should("be.equal", 6);
cy.get(".filters h2").first().contains("Categories");
cy.get(".filters h2").last().contains("Organizations");
});

// https://github.com/GSA/datagov-deploy/issues/3672
it('Can visit organization and group facet link', () => {
cy.visit('/organization/org-tags?tags=_');
cy.visit('/group/group-facets?tags=_');
});
// https://github.com/GSA/data.gov/issues/3672
it("Can visit organization and group facet link", () => {
cy.visit("/organization/org-tags?tags=_");
cy.visit("/group/group-facets?tags=_");
});
});
2 changes: 0 additions & 2 deletions postgresql/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,3 @@ ARG DATASTORE_READONLY_PASSWORD

# Include extra setup scripts (eg datastore)
ADD docker-entrypoint-initdb.d /docker-entrypoint-initdb.d
ADD https://github.com/GSA/datagov-deploy/raw/71936f004be1882a506362670b82c710c64ef796/ansible/roles/software/ec2/ansible/files/locations.sql.gz /docker-entrypoint-initdb.d/locations.sql.gz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of updating this link, I think referencing a key piece of DB setup (for dev and prod) at a historical git commit to a file that is no longer being tracked isn't good practice. Can we just add the file in the repo similar to the other files in docker-entrypoint-initdb.d folder?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to that

RUN chmod 666 /docker-entrypoint-initdb.d/locations.sql.gz
Binary file not shown.
2 changes: 0 additions & 2 deletions solr/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ RUN git clone https://github.com/aws/efs-utils && \
# Install hostname resolution dependencies
RUN apt-get install -y dnsutils

# Fix Issue https://github.com/GSA/datagov-deploy/issues/3285
# Update the angularjs library files
ADD https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.8.2/angular.min.js \
https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.8.2/angular.js \
Expand Down Expand Up @@ -63,7 +62,6 @@ RUN sed -i 's/jquery-1.7.2.min.js/jquery-3.6.0.min.js/' \
/opt/solr/server/solr/configsets/sample_techproducts_configs/conf/velocity/head.vm


# Fix Issue https://github.com/GSA/datagov-deploy/issues/3283
# Disable directory listing
RUN perl -0777 -i -pe 's/ \
<param-name>dirAllowed<\/param-name>\n.*<param-value>true<\/param-value>/ \
Expand Down
8 changes: 4 additions & 4 deletions tools/locations-table/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Locations table

The locations table [was defined in 2016](https://github.com/GSA/datagov-deploy/search?q=location+table&type=commits).
This table was deleted from the repo but is still available from old commits: [locations.sql.gz](https://github.com/GSA/datagov-deploy/raw/71936f004be1882a506362670b82c710c64ef796/ansible/roles/software/ec2/ansible/files/locations.sql.gz).
The locations table [was defined in 2016](https://github.com/GSA/data.gov/search?q=location+table&type=commits).
This table was deleted from the repo but is still available from old commits: [locations.sql.gz](https://github.com/GSA/data.gov/raw/71936f004be1882a506362670b82c710c64ef796/ansible/roles/software/ec2/ansible/files/locations.sql.gz).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is outdated now that the table is in the docker-entrypoint dir.


This table could be dd to database with

```
```bash
./install-locations-table.sh HOST DB_NAME DB_USER PASS
```
```
16 changes: 9 additions & 7 deletions tools/locations-table/install-locations-table.sh
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
#!/bin/bash

echo "*** USAGE: $(basename "$0") HOST DB_NAME DB_USER PASS"

DEST_FOLDER=/tmp
# TODO check path on run
SQL_FILE=30_locations.sql.gz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm almost positive the pathing needs to be adjusted

HOST=$1
DB_NAME=$2
DB_USER=$3
PASS=$4

echo "Downloading locations table"
wget https://github.com/GSA/datagov-deploy/raw/71936f004be1882a506362670b82c710c64ef796/ansible/roles/software/ec2/ansible/files/locations.sql.gz -O $DEST_FOLDER/locations.sql.gz
if ! [ -f $SQL_FILE ]; then
echo "Error: locations.sql file not found, exiting.."
exit 1
fi

echo "Creating locations table"
gunzip -c ${DEST_FOLDER}/locations.sql.gz | PGPASSWORD=${PASS} psql -h $HOST -U $DB_USER -d $DB_NAME -v ON_ERROR_STOP=1

echo "Cleaning"
rm -f $DEST_FOLDER/locations.sql.gz
gunzip -c ${SQL_FILE} | PGPASSWORD=${PASS} psql -h "$HOST" -U "$DB_USER" -d "$DB_NAME" -v ON_ERROR_STOP=1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my understanding was putting the .gzipped file in the docker-entrypoint dir executes it automatically when catalog starts up... is that correct?

and if so, what does this script here do?