Code

Check for cross submits#68

Code

Open

silentninja wants to merge 6 commits intocommoncrawl:2.xcommoncrawl/news-crawl:2.xfrom silentninja:feat/cross-submitsilentninja/news-crawl:feat/cross-submitCopy head branch name to clipboard

+349-7

Commits

Commits on Feb 21, 2025

• Add cross submit verification for sitemaps

  silentninja

  committed
  4c854e0

  Copy full SHA for 4c854e0

Commits on Feb 26, 2025

• Add cross submit verification check for sitemap indexes

  silentninja

  committed
  0a58e3b

  Copy full SHA for 0a58e3b

Commits on Mar 3, 2025

• Use URI instead of URL

  silentninja

  committed
  8cf52f5

  Copy full SHA for 8cf52f5

• Revert accidental changes

  silentninja

  committed
  29e5f3d

  Copy full SHA for 29e5f3d

Commits on Mar 4, 2025

• Allow lenient cross submits as config

  silentninja

  committed
  3a2bd46

  Copy full SHA for 3a2bd46

Commits on Mar 5, 2025

• use crawlercommons.domains.EffectiveTldFinder instead of org.apache.http.conn.util.PublicSuffixMatcher to get the hostnames when checking for cross submit

  silentninja

  committed
  72a74c3

  Copy full SHA for 72a74c3