Merge changes in Rc/v2.1.0 to main branch

[mingstat]

Improved code quality through refactoring for better readability and maintainability, resolved partial matching issues for file names without extensions, and enhanced the load_data() function with new env_var and print_file_paths arguments for greater flexibility.

[ml-ebs-ext]

Whole paths would be better, because file_paths can point to different folders with files that share the same name.

[mingstat]

The use of base file names as list names is maintained to ensure backward compatibility with existing code that relies on the legacy load_data() function. For users who need the full file paths, this information is stored in the metadata attributes of each data frame in the returned list.

[ml-ebs-ext]

Users of legacy load_data() retain the old behavior because of this statement that happens at the end of that function:

names(data_list) <- file_names

Users of load_data_files() would benefit from seeing the exact path they provided as names of the output list.

[ml-ebs-ext]

See discussion in internal chat about possibility of removing both the paths and the extensions as well as preventing repeat entries in the resulting list. The

names(data_list) <- file_names

at the end of load_data() should make that function immune to implementing this change.

[ml-ebs-ext]

I don't get the rewrite from toupper to tolower, the change in the error message, the moving of code around and the obvious comments. They only open the doors for a bug to creep in and make code review a longer process for no discernible benefit.

[mingstat]

The change to lowercase file extensions (.rds and .sas7bdat) was made to align with common R package conventions, as seen in packages like {pins} and {haven}.

[ml-ebs-ext]

The old toupper is only used to do a case-insensitive check of the file extension. It's a non-user-facing implementation detail. Changing the logic is unnecessary.

[ml-ebs-ext]

There's a lot of new code here and I haven't reviewed it closely. The difference that strikes me the most is that the old toupper case-insensitive match behavior is gone. I imagine this can have an impact under Windows. Since load_data has been rewritten to list files through this function, we need a good reason to deviate from the old behavior.

[mingstat]

The current code follows the case-sensitive behavior of readRDS() and haven::read_sas() to avoid ambiguity and risk of matching the wrong file.

[ml-ebs-ext]

This piece of code belongs to the create_data_list currently on the main branch:

# Case insensitive file name match
uppercase_file_name <- toupper(paste0(x, ext))

match_count <- sum(uppercase_candidates == uppercase_file_name)
if (match_count > 1) {
  stop(paste("create_data_list(): More than one case-insensitive file name match for", file_path, x))
}

It is there to warn against an edge-case scenario in which a folder contains two files that share the same name but differ in case. That is not a problem under linux, but we still want to warn users against that situation, because running the same code with the same data files under case-insensitive windows file systems could lead to the loading of different files.

This check is no longer in the rewritten dv.loader and it should be, unless the team decides otherwise.

My suggestion here would be to take the original logic of create_data_list and adapt it minimally to follow the old filename-matching logic, so that we don't throw away useful behavior on a rewrite.

[mingstat]

Thank you for raising this important point about case-sensitivity. I agree we should discuss with the team to determine if we want to to make any changes to the current behavior.

[mattkorb]

This variable is not used anywhere. I think it can be removed.

[mingstat]

Good catch. I've removed the unused file_ext variable since it's now handled in the get_file_paths() function.