Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

4.2.6 staging #1903

Merged
merged 80 commits into from
Feb 14, 2024
Merged
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
3c756f4
Cast Object timeout as Integer
wwtamu Nov 1, 2023
7df1050
Update maven-assembly-plugin configuration
wwtamu Nov 1, 2023
c9d2405
Bump tinymce from 5.10.7 to 5.10.9
dependabot[bot] Nov 15, 2023
345d6fd
update version
cstarcher Nov 22, 2023
13f7f37
sql query fix
cstarcher Nov 22, 2023
26231a0
replace string concatenation
cstarcher Nov 29, 2023
6adbcab
Fix email workflow rules resetting on restart
wwtamu Dec 23, 2023
95ce645
Switch application initialization to initializing bean
wwtamu Dec 23, 2023
9425e6b
Have abstract email recipient implement email recipient
wwtamu Dec 23, 2023
bf2b392
Make default collections class members final and remove unused methods
wwtamu Dec 28, 2023
8974332
Minor cleanup of integration tests
wwtamu Dec 28, 2023
30fa312
Remove unused imports
wwtamu Dec 28, 2023
f1bf6f9
More cleanup
wwtamu Dec 28, 2023
5c6c0c4
Organize imports and return rule id in payload of create new email wo…
wwtamu Dec 28, 2023
e54aedb
Use mock mvc to call controller in email workflow rule integration tests
wwtamu Dec 28, 2023
0217b72
Initial breaking system data loader test
wwtamu Dec 28, 2023
78309e3
Decouple submission list column default services
wwtamu Dec 28, 2023
2caff3e
Fix broken test due to immutable in memory defaults
wwtamu Dec 28, 2023
55439e4
Decouple system default settings from system data loader
wwtamu Dec 28, 2023
01c8da4
Decouple loading in memory proquest codes from persisting controlled …
wwtamu Dec 28, 2023
319fd14
Decouple initializing depositor services from system data loader
wwtamu Dec 28, 2023
4276717
Reorder system data loader methods according to use in class
wwtamu Dec 28, 2023
47035da
Directly use resource pattern resolver during system data loading
wwtamu Dec 28, 2023
c9f320a
Fix submission controller indentation
wwtamu Dec 28, 2023
0494d5f
Add private constructor for ORCID utility
wwtamu Dec 28, 2023
5d93951
Correct registration error response message
wwtamu Dec 28, 2023
3daa5ae
Only set default columns on user before create/persist
wwtamu Dec 28, 2023
522b885
Fix ORCiD authentication - issue 1877
wwtamu Dec 28, 2023
bcace83
Avoid null pointer exception
wwtamu Dec 28, 2023
c2e8212
Remove redundant superinterface
wwtamu Dec 28, 2023
042395c
Consistently format system default JSON
wwtamu Dec 29, 2023
4ec8d06
Simplify system submission status loading
wwtamu Dec 29, 2023
3abc1d0
Use field profile heritable repo delete and remove orphaned submissio…
wwtamu Dec 29, 2023
635cca0
Keyboard accessible organization triptych
wwtamu Dec 29, 2023
bfa6f92
Use test resources and ignore environment specific config files
wwtamu Dec 29, 2023
a51c6de
Print internal server error stack trace in debug
wwtamu Dec 29, 2023
37e9fd3
Switch to release of weaver and and compiler release property
wwtamu Dec 29, 2023
d1a6ac1
Remove overriding exception handlers causing all exceptions to be 500…
wwtamu Dec 29, 2023
c9a24a7
Move modal service to app to fix close modal - issue 1682
wwtamu Dec 29, 2023
56672ca
Remove unused dependency injection and scope variable
wwtamu Dec 29, 2023
4fc58eb
Close modal and wait before location change on submission delete
wwtamu Dec 29, 2023
086d593
Remove modal backdrop manually - issue 1879
wwtamu Dec 29, 2023
5d03e49
Update file upload to large message with max file size - issue 1883
wwtamu Dec 29, 2023
07e1d7a
Adjust line height of drop zone message
wwtamu Dec 29, 2023
33a88f7
Fix student submission select icon vertical centering
wwtamu Dec 29, 2023
2452c21
Fix conditional textarea input template
wwtamu Dec 29, 2023
fc8c0b0
Refactor test and test entity controlled vocabulary
wwtamu Dec 29, 2023
78d4559
Remove empty tests
wwtamu Dec 29, 2023
3aeea30
Add test messages to application initialization tests
wwtamu Dec 29, 2023
16e320d
Cleanup system data loader and rollback on IOException
wwtamu Dec 29, 2023
1c576cf
Add system data loader test
wwtamu Dec 29, 2023
a4f2f70
Update submission list column when field profile gloss updated
wwtamu Dec 30, 2023
ab9375a
Add data integrity violation exception handler
wwtamu Dec 30, 2023
b440656
Have data integrity violation exception return conflict status
wwtamu Dec 30, 2023
1fed8ad
Merge pull request #1869 from wwtamu/create-deposit-location
cstarcher Jan 8, 2024
7872bab
Merge pull request #1871 from TexasDigitalLibrary/dependabot/npm_and_…
cstarcher Jan 8, 2024
4949188
Merge pull request #1873 from TexasDigitalLibrary/1872-sql-date-fix
cstarcher Jan 8, 2024
a7dd1cd
Created new issue templates
cstarcher Jan 14, 2024
3ba37f3
Merge pull request #1892 from TexasDigitalLibrary/new-issue-templates
cstarcher Jan 14, 2024
3d959cc
Merge pull request #1887 from wwelling/system-data-loader-refactor
smutniak Jan 16, 2024
1e7f15a
Return new list from default submission list column service to avoid …
wwtamu Jan 22, 2024
f024240
Rollback remove field profile on constraint violation exception
wwtamu Jan 22, 2024
a988be4
Add ConstraintViolationException handler to custom response entity ex…
wwtamu Jan 22, 2024
3c1d494
fix 1895 for marc21 and proquest keyword display
smutniak Jan 22, 2024
fd1dd8a
Add default cssUrl for local testing
wwtamu Jan 23, 2024
d061e02
Upgrade @wvr/core to latest patch RC and remove modal service override
wwtamu Jan 23, 2024
b92af7e
Merge pull request #1893 from wwelling/4.2.6_staging-default-submissi…
cstarcher Jan 23, 2024
c106eda
Merge pull request #1894 from wwelling/4.2.6_staging-remove-field-pro…
cstarcher Jan 23, 2024
3081289
Upgrade to @wvr/core 2.2.6 RC 2
wwtamu Jan 23, 2024
7207417
Merge pull request #1898 from wwelling/4.2.6_staging_wvr-core-patch-rc
cstarcher Jan 23, 2024
c2cf9c7
Merge pull request #1897 from TexasDigitalLibrary/1895_marc21fix
cstarcher Jan 23, 2024
2122bd5
Upgrade @wvr/core to 2.2.6 release
wwtamu Jan 24, 2024
c6fc2cf
Merge pull request #1899 from wwelling/4.2.6_staging-wvr-core-release
cstarcher Jan 24, 2024
81c16f9
Set field value identifier for embargos
wwtamu Jan 30, 2024
9414d56
Use ISO 8601 format for embargo lift date for and respect embargo wit…
wwtamu Jan 30, 2024
c8c36cb
Minor cleanup of dspace_mets.xml
wwtamu Jan 31, 2024
cda73f7
added list for repeatable field values
cstarcher Feb 4, 2024
59210ee
removed println
cstarcher Feb 4, 2024
92ea94d
Merge pull request #1902 from TexasDigitalLibrary/1880-excel-repeatab…
cstarcher Feb 5, 2024
c30f186
Merge pull request #1901 from wwelling/4.2.6_staging-dspace-mets-swor…
smutniak Feb 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Decouple loading in memory proquest codes from persisting controlled …
…vocabulary
wwtamu committed Dec 28, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 01c8da4d1c4183eec9d3db7347987fc971441357
92 changes: 86 additions & 6 deletions src/main/java/org/tdl/vireo/service/ProquestCodesService.java
Original file line number Diff line number Diff line change
@@ -1,28 +1,70 @@
package org.tdl.vireo.service;

import java.io.IOException;
import java.io.InputStream;
import java.util.Collections;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;

import javax.annotation.PostConstruct;

import org.apache.poi.hssf.usermodel.HSSFSheet;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.CellType;
import org.apache.poi.ss.usermodel.Row;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.ResourcePatternResolver;
import org.springframework.stereotype.Service;

@Service
public class ProquestCodesService {

// TODO: make map immutable after setting values
private final Logger logger = LoggerFactory.getLogger(ProquestCodesService.class);

private final Map<String, Map<String, String>> codes;

public ProquestCodesService() {
@Autowired
private ResourcePatternResolver resourcePatternResolver;

ProquestCodesService() {
codes = new HashMap<String, Map<String, String>>();
}

public void setCodes(String key, Map<String, String> codes) {
checkCreateCodes(key);
this.codes.put(key, codes);
@PostConstruct
void init() throws IOException {
loadProquestLanguageCodes();
loadProquestDegreeCodes();
loadProquestSubjectCodes();
}

public Map<String, String> getCodes(String key) {
checkCreateCodes(key);
return codes.get(key);
return Collections.unmodifiableMap(codes.get(key));
}

private void loadProquestLanguageCodes() throws IOException {
logger.info("Loading proquest language codes");
setCodes("languages", getProquestCodes("language_codes.xls"));
}

private void loadProquestDegreeCodes() throws IOException {
logger.info("Loading proquest degree codes");
setCodes("degrees", getProquestCodes("degree_codes.xls"));
}

private void loadProquestSubjectCodes() throws IOException {
logger.info("Loading proquest subjects codes");
setCodes("subjects", getProquestCodes("umi_subjects.xls"));
}

private void setCodes(String key, Map<String, String> codes) {
checkCreateCodes(key);
this.codes.put(key, codes);
}

private void checkCreateCodes(String key) {
@@ -31,4 +73,42 @@ private void checkCreateCodes(String key) {
}
}

private Map<String, String> getProquestCodes(String xslFileName) throws IOException {
Map<String, String> proquestCodes = new HashMap<String, String>();

Resource resource = resourcePatternResolver.getResource("classpath:/proquest/" + xslFileName);

try (
InputStream file = resource.getInputStream();
HSSFWorkbook workbook = new HSSFWorkbook(file);
) {
HSSFSheet sheet = workbook.getSheetAt(0);

Iterator<Row> rowIterator = sheet.iterator();
while (rowIterator.hasNext()) {
Row row = rowIterator.next();

String code = null, description = "";

Iterator<Cell> cellIterator = row.cellIterator();
while (cellIterator.hasNext()) {

Cell cell = cellIterator.next();
if (cell.getCellType() == CellType.STRING) {
String cellValue = cell.getStringCellValue();
if (code == null) {
code = cellValue;
} else {
description = cellValue;
}
}
}

proquestCodes.put(code, description);
}
}

return proquestCodes;
}

}
174 changes: 45 additions & 129 deletions src/main/java/org/tdl/vireo/service/SystemDataLoader.java
Original file line number Diff line number Diff line change
@@ -2,23 +2,16 @@

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.apache.commons.io.IOUtils;
import org.apache.poi.hssf.usermodel.HSSFSheet;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.CellType;
import org.apache.poi.ss.usermodel.Row;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
@@ -216,14 +209,8 @@ public void loadSystemData() {
logger.info("Loading system controlled vocabularies");
loadControlledVocabularies();

logger.info("Loading system Proquest language codes");
loadProquestLanguageCodes();

logger.info("Loading system Proquest degree codes");
loadProquestDegreeCodes();

logger.info("Loading system Proquest subject codes");
loadProquestSubjectCodes();
logger.info("Loading system Proquest subjects codes controlled vocabulary");
loadProquestSubjectCodesControlledVocabulary();

logger.info("Loading system Submission List Columns");
loadSubmissionListColumns();
@@ -237,51 +224,7 @@ public void loadSystemData() {
logger.info("Finished loading system data");
}

private void loadControlledVocabularies() {

List<File> controlledVocabularyFiles = new ArrayList<File>();
try {
controlledVocabularyFiles = fileIOUtility.getResourceDirectoryListing("classpath:/controlled_vocabularies/");
} catch (IOException e) {
e.printStackTrace();
}

for (File vocabularyJson : controlledVocabularyFiles) {
try {
ControlledVocabulary cv = objectMapper.readValue(vocabularyJson, ControlledVocabulary.class);

// check to see if Controlled Vocabulary exists, and if so, merge up with it
ControlledVocabulary persistedCV = controlledVocabularyRepo.findByName(cv.getName());

if (persistedCV == null) {
persistedCV = controlledVocabularyRepo.create(cv.getName());
}

for (VocabularyWord vw : cv.getDictionary()) {

VocabularyWord persistedVW = vocabularyRepo.findByNameAndControlledVocabulary(vw.getName(), persistedCV);

if (persistedVW == null) {
persistedVW = vocabularyRepo.create(persistedCV, vw.getName(), vw.getDefinition(), vw.getIdentifier(), vw.getContacts());
persistedCV = controlledVocabularyRepo.findByName(cv.getName());
} else {
persistedVW.setDefinition(vw.getDefinition());
persistedVW.setIdentifier(vw.getIdentifier());
persistedVW.setContacts(vw.getContacts());
persistedVW = vocabularyRepo.save(persistedVW);
}
}

} catch (JsonParseException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

}
}


private void loadOrganizationCategories() {
try {
@@ -917,23 +860,54 @@ private void loadDocumentTypes() {
}
}

private void loadProquestLanguageCodes() {
proquesteCodesService.setCodes("languages", getProquestCodes("language_codes.xls"));
}
private void loadControlledVocabularies() {

private void loadProquestDegreeCodes() {
proquesteCodesService.setCodes("degrees", getProquestCodes("degree_codes.xls"));
}
List<File> controlledVocabularyFiles = new ArrayList<File>();
try {
controlledVocabularyFiles = fileIOUtility.getResourceDirectoryListing("classpath:/controlled_vocabularies/");
} catch (IOException e) {
e.printStackTrace();
}

for (File vocabularyJson : controlledVocabularyFiles) {
try {
ControlledVocabulary cv = objectMapper.readValue(vocabularyJson, ControlledVocabulary.class);

private void loadProquestSubjectCodes() {
Map<String, String> subjectCodes = getProquestCodes("umi_subjects.xls");
// check to see if Controlled Vocabulary exists, and if so, merge up with it
ControlledVocabulary persistedCV = controlledVocabularyRepo.findByName(cv.getName());

if (persistedCV == null) {
persistedCV = controlledVocabularyRepo.create(cv.getName());
}

for (VocabularyWord vw : cv.getDictionary()) {

VocabularyWord persistedVW = vocabularyRepo.findByNameAndControlledVocabulary(vw.getName(), persistedCV);

if (persistedVW == null) {
persistedVW = vocabularyRepo.create(persistedCV, vw.getName(), vw.getDefinition(), vw.getIdentifier(), vw.getContacts());
persistedCV = controlledVocabularyRepo.findByName(cv.getName());
} else {
persistedVW.setDefinition(vw.getDefinition());
persistedVW.setIdentifier(vw.getIdentifier());
persistedVW.setContacts(vw.getContacts());
persistedVW = vocabularyRepo.save(persistedVW);
}
}

proquesteCodesService.setCodes("subjects", subjectCodes);
} catch (JsonParseException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

createSubjectControlledVocabulary(subjectCodes);
}
}

private void createSubjectControlledVocabulary(Map<String, String> subjectCodes) {
private void loadProquestSubjectCodesControlledVocabulary() {
Map<String, String> subjectCodes = proquesteCodesService.getCodes("subjects");

// check to see if Controlled Vocabulary exists, and if so, merge up with it
ControlledVocabulary persistedCV = controlledVocabularyRepo.findByName("Subjects");
@@ -957,64 +931,6 @@ private void createSubjectControlledVocabulary(Map<String, String> subjectCodes)
persistedVW = vocabularyRepo.save(persistedVW);
}
}

}

private Map<String, String> getProquestCodes(String xslFileName) {
Map<String, String> proquestCodes = new HashMap<String, String>();
Resource resource = resourcePatternResolver.getResource("classpath:/proquest/" + xslFileName);

InputStream file = null;
try {
file = resource.getInputStream();
} catch (IOException e) {
e.printStackTrace();
}

if (file != null) {

HSSFWorkbook workbook = null;
try {
workbook = new HSSFWorkbook(file);
} catch (IOException e) {
e.printStackTrace();
}
if (workbook != null) {

HSSFSheet sheet = workbook.getSheetAt(0);

Iterator<Row> rowIterator = sheet.iterator();
while (rowIterator.hasNext()) {
Row row = rowIterator.next();

String code = null, description = "";

Iterator<Cell> cellIterator = row.cellIterator();
while (cellIterator.hasNext()) {

Cell cell = cellIterator.next();
if (cell.getCellType() == CellType.STRING) {
String cellValue = cell.getStringCellValue();
if (code == null) {
code = cellValue;
} else {
description = cellValue;
}
}
}

proquestCodes.put(code, description);
}
}

try {
file.close();
} catch (IOException e) {
e.printStackTrace();
}

}
return proquestCodes;
}

private void loadPackagers() {
21 changes: 20 additions & 1 deletion src/test/java/org/tdl/vireo/service/SystemDataLoaderTest.java
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
package org.tdl.vireo.service;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.util.Map;

import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
@@ -45,7 +49,6 @@ public void testLoadSystemData() throws Exception {
}

private void assertPersistedSystemData(boolean isReload) {

assertEquals(8, this.defaultSettingsService.getTypes().size(),
isReload
? "Incorrect number of default setting types after reload"
@@ -69,6 +72,10 @@ private void assertPersistedSystemData(boolean isReload) {
isReload
? "Incorrect number of default filters after reload"
: "Incorrect number of default filters");

assertProquestCodes(82, "languages", isReload);
assertProquestCodes(1219, "degrees", isReload);
assertProquestCodes(288, "subjects", isReload);
}

private void assertSettingsType(int expected, String type, boolean isReload) {
@@ -78,4 +85,16 @@ private void assertSettingsType(int expected, String type, boolean isReload) {
: String.format("Incorrect number of default %s settings", type));
}

private void assertProquestCodes(int expected, String key, boolean isReload) {
Map<String, String> codes = this.proquesteCodesService.getCodes(key);
assertNotNull(codes,
isReload
? String.format("Missing proquest %s codes after reload", key)
: String.format("Missing proquest %s codes", key));
assertEquals(expected, codes.size(),
isReload
? String.format("Incorrect number of proquest %s codes after reload", key)
: String.format("Incorrect number of proquest %s codes", key));
}

}