Importer errors: FATAL: PCDATA invalid Char value 11

[bkiahstroud]

Story

Affected importer IDs

• 92
• 117
• 280
• 281
• 282
• 283
• 284
• 285
• 286
• 287
• 288
• 289

[afred]

This one is due to vertical tabs present in the source xml. We are fixing it at the source and will notify here once the importers can be retried.

[afred]

I gathered a list of AAPB IDs from the filenames printed in the error messages. I'm hoping that these are the only ones that need to be looked at, rather than the 10K files for each importer.
aapb_ids_vertical_tab_in_pbcore.txt

[bkiahstroud]

Script to switch broken XML files with the fixed ones:

fixed_xml_files = Dir.glob(Rails.root.join('tmp', 'imports', 'i129-fixed-xml', '*.xml'))
imp_ids = [92, 117, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289]
imp_dirs = []
results = {}

imp_ids.each do |id|
  imp_dirs << Rails.root.join('tmp', 'imports', Bulkrax::Importer.find(id).name)
end

fixed_xml_files.each do |f|
  filename = File.basename(f)
  imp_dirs.each do |dir|
    true_path = File.join(dir, filename)
    next unless File.exist?(true_path)

    results[filename] = true_path
    break
  end
end

results.each do |filename, path|
  FileUtils.mv(path, Rails.root.join('tmp', 'imports', 'i129-broken-xml'))
  FileUtils.cp(Rails.root.join('tmp', 'imports', 'i129-fixed-xml', filename), path)
end

[bkiahstroud]

All the effected Importers have been rerun and no longer error 🙌