Fixed issue where migration.py passes a Content object to remove_content(), which ends up breaking pulpcore's remove_content() further down the line with a traceback stating, 'Content' object has no attribute 'count'.
Pulpcore's remove_content() will not face this issue anymore as it will now always receive a QuerySet object from migration.py. #550
Migration may stuck for many days when migrating an environment with massive number of rpms. This commit fixed the issue. #568
When pre-migrating errata, make query in batch to prevent the BSON too large error. #572
Fix invalid BSON size error when migrating docker contents
The size limit for a single document in Mongodb is 16MB so migrating many docker tags could exceed this limit. This commit fixes this issue by fetching the docker tags in a specified batch size instead of fetching all and returning a large results into a single document. #578
- Added steps to remove the plugin and announced its EOL. Pulp-2to3-migration plugin reaches its EOL on December 31, 2022. The last supported pulpcore version is 3.19. #534
- Added the DEB_COMPONENT_BATCH_SIZE setting, so users can individually control the batch size for this one to many type. #9564
Try parsing advisory dates as unix timestamps before giving up. #483
Fixed a bug in the deb migration that caused migrated repos to lack content needed for structured publications in some circumstances. #484
Fixed DistributionTree (kickstart tree) migration for pulp_rpm 3.17+.
Alma Linux 8 or CentOS 8 repos migration will no longer fail with: No declared artifact with relative path "images/boot.iso" (or ".treeinfo"). #491
Escape django template syntax in xml when rendering filelist #496
Fixed local variable 'item' referenced before assignment #497
Taught mongo-connection how to deal with Really Big Queries. This addresses the cause of exceptions like pymongo.errors.OperationFailure: Sort exceeded memory limit of 104857600 bytes when migrating large collections. #511
Fixed
'NoneType' object has no attribute 'delete'error during migration re-runs. #8968deb_components now use a default batch size of 50 to avoid CursorNotFound errors during pre_migration for this one to many type. #9564
- Added
signing_servicefor (apt-)repository in Complex-MigrationPlan. This allows for a signing-service created before the migration to automatically be attached to the migrated repositories. #9517
- Fixed distibution tree migration issue "‘NoneType’ object has no attribute ‘url’". #8862
- Add batch_size to a pulp 2 query for content pre-migration of errata. #9451
- Added a note about migration when Pulp 3 runs on a FIPS-enabled machine. #7992
- Filtered out incomplete publications for the pulp2repositories/ endpoint. #9242
- Fixed migration of sles_auth_token #9254
- Fixed bug where sync tasks would open a lot of DB connections #9283
- Fixed remigration of publications after an unsuccessful or interrupted run. #9295
- Fixed modular errata migration. #8874
- Add migration of gpgkeys-field for verification of debian-repos. #8970
- Fix migration for any clients that have applied the fix for #8893. #8982
- Strip leading slashes from Relative URL for deb-migrations. #9184
- Dropped support for Python 3.6 and 3.7. #9038
- Fixed a bug causing deb migrations to fail if there are Pulp 2 importers without a releases field. #8928
- Started migrating relationship between repo and importer to Pulp 3. #8626
Taught 2to3 migration about changes in task-dispatching and Distributions.
NOTE: To minimize any potential impact from this change, please make sure you've done a 2to3 migration after any Pulp2 changes to distributors before upgrading to this release. #8522
- Fixed failure when encountering overlapping content. #593
- Migration may stuck for many days when migrating an environment with massive number of rpms. This commit fixed the issue. #568
- When pre-migrating errata, make query in batch to prevent the BSON too large error. #572
Fixed issue where migration.py passes a Content object to remove_content(), which ends up breaking pulpcore's remove_content() further down the line with a traceback stating, 'Content' object has no attribute 'count'.
Pulpcore's remove_content() will not face this issue anymore as it will now always receive a QuerySet object from migration.py. #550
- Taught mongo-connection how to deal with Really Big Queries. This addresses the cause of exceptions like pymongo.errors.OperationFailure: Sort exceeded memory limit of 104857600 bytes when migrating large collections. #511
- Escape django template syntax in xml when rendering filelist #496
- Fixed local variable 'item' referenced before assignment #497
- Fixed distribution tree migration for repositories with partial .treeinfo (e.g. most of CentOS 8 repositories). #489
- Fixed
'NoneType' object has no attribute 'delete'error during migration re-runs. (backported from #8968) #9612
- Fixed migration of sles_auth_token (backported from #9254) #9456
- Add batch_size to a pulp 2 query for content pre-migration of errata. (backported from #9451) #9482
- Fixed distibution tree migration issue "‘NoneType’ object has no attribute ‘url’". (backported from #8862) #9372
- Fixed remigration of publications after an unsuccessful or interrupted run. (backported from #9295) #9296
- Filtered out incomplete publications for the pulp2repositories/ endpoint. (backported from #9242) #9297
- Fixed a bug causing deb migrations to fail if there are Pulp 2 importers without a releases field. (backported from #8928) #8945
- Added migration of gpgkeys-field for verification of debian-repos. (backported from #8970) #9017
- Fixed migration for any clients that have applied the fix for #8893. (backported from #8982) #9109
- Fixed modular errata migration. (backported from #8874) #9173
- Strip leading slashes from Relative URL for deb-migrations. (backported from #9184) #9188
Fixed migration of CentOS8 distribution trees. #8566
Fixed a problem when migrating one plugin after another will remove publications/distributions of the first one. #8686
Ensure a checksum type of a package is used for publications when a checksum type was not explicitly configured in Pulp 2.
If you plan to perform sync from the migrated Pulp 3 to a Pulp 2 instance, this fix is important, otherwise you can ignore it. If you've already started migration of the RPM plugin to Pulp 3, reset the migration for it and start again. #8725
Fix migration of on_demand distribution (kickstart) trees when they do no have any images, e.g. CentOS 8 High Availability repo. #8817
Pulp2 Content that does not have downloaded flag is marked as downloaded. #8863
- Stopped migrating unsupported metadata, like .zck, which could have been imported into some old Pulp 2 version. #8400
- Handled overlapping paths which might come from old pulp2 repositories. #8582
- Handled properly skipping of corrupted or missing docker content. #8594
- Fixed a bug in the deb pipeline that was preventing successfull skipping of corrupted content for migrations with
skip_corrupted=True. #8612 - Fixed migration of Pulp 2 deb importers without configured components or architectures #8613
- Fixed total counters for content migration. #8621
- Fixed Debian importer migration on a re-run. #8653
- Added a new configuration option CONTENT_PREMIGRATION_BATCH_SIZE to be able to adjust the content batch size if the system is slow. #8470
- Fixed the case when some Pulp 2 content was removed and cleaned up between migration re-runs. #7887
- Fixed a migraiton failure when no importer was specified in the migration plan. #8382
- Fixed errata re-migration when it's been copied to a new repo in Pulp2 between migration re-runs. #8417
- Fixed the case when listing pulp2content/ endpoint using bindings would fail if content didn't have a storage path in Pulp 2. #8436
- Fixed a migration failure when no importer was specified in the migration plan. #8389
- Fixed re-migration cases when a distributor on an importer changed in the migration plan between re-runs. #7889
- Ensure trailing slash is present when assembling the url. #8321
- Fixed pulp2content unique-constraint to correctly handle a null repo-id field. #8329
- Added docs on the value of fast storage volumes and adjusting the worker timeout values for both Pulp 2 and Pulp 3. #7847
- Updated docs to pulp-cli. #8254
- Added the ability to migrate additional Debian content types needed for structured publishing. #7865
- The default configuration now contains the ALLOWED_CONTENT_CHECKSUMS setting with all checksum types supported in Pulp 2. #8266
- Fixed the re-run times when repositories/importers/distributors haven't changed much since the last run. #7779
- Fixed an edge-case failure in erratum-migration when doing repeated migrations. #8166
- Fixed distributor re-migration case when it was changed in Pulp 2 between migration plan runs. #8195
- Fixed openapi schema for reset/ endpoint. Bindings no longer require plan parameter. #8211
- Added a note that ALLOWED_CONTENT_CHECKSUMS is strongly recommended to allow all supported checksum types, and can be adjusted after the migration. #8266
Multi-artifact content aren't (not) skipped properly when some artifacts are unavailable. #7681
Taught pre-migration to order content by last-updated.
This lets a migration recover reliably from fatal errors during migration attempts. NOTE: this fix assumes the Pulp2 instance is at least at 2.21.5. Earlier versions are missing an index in the Mongo database that makes the ordering possible. #7781
Fix an error migrating module content with no "stream" or "profile" information specified, as is allowed by the spec. #7846
No longer generate sqlite metadata when publishing unless the Pulp 2 configuration specified to do so. #7851
Fixed Pulp2Content serialization when filters are applied. #7994
Taught rpm to warn and continue if a Distribution is missing a treeinfo file. #8084
- Added an option to skip corrupted or missing Pulp 2 content. #7538
- Added a reset/ endpoint to be able to run migration from scratch. #7714
- Added support to migrate Debian packages (tech preview). #7863
- Fixed distribution serialization. #7809
- Fixed a bug where RPM content metadata is not properly migrated to Pulp 3. #7625
- Publications and Distributions are re-created on migration re-run for repos that contain mutable content, and pre-migrated mutable content is no longer deleted and recreated every time. #7280
- Fixed a bug where PULP_MANIFEST was being created outside of the worker's working directory. #7693
- Sped up repository pre-migration by skipping the repository content relations pre-migration when nothing changed in a repository. #7694
- Made content migration significantly faster on low-spec machines w/ HDD backed database storage. #7699
- Fix the bindings for publication and distribution hrefs fields on pulp2repositories API. #7679
- Fixed a distribution migration case when a repository in Pulp 2 has been recreated. #7080
- Stopped logging warnings if at least one LCE per content migrated. #7193
- Fixed metadata checksum type configuration re-migration. #7417
- Fixed re-migration issue when pulp 2 importer changed a feed. #7418
- Fixed validation of the distributor missing resources in the migration plan. #7488
- Fix custom metadata migration when the same metadata is present under different paths in different repositories. #7489
- Fixed high memory usage when migrating large amounts of content (300,000+). #7490
- Removed comps content types from future_types. #7518
- Fixed migration of lazy multi-artifact content not present in a repository in the plan. #7562
- Added GroupProgressReport tracking during the migration. #6769
- Make the migration plugin compatible with pulp_container 2.0 #7365
- Significantly improved performance of partial migrations (when some content / repos has been migrated already). #6111
- Fixed migration of a distribution tree if it has a treeinfo and not .treeinfo #6951
- Fixed cause of view_name warnings during (re)start of Pulp services. #7154
- Marked all Pulp2LCEs as migrated for distribution tree migration. #7260
- Updated migration of file remote url to point to the Manifest. #7264
- Fix exceptions thrown by content migration not being bubbled up through the task. #6469
- Add support for migrating SLES12+ repos which require auth token. #6927
- Fixed distribution tree migration when a distribution tree is present in multiple repositories. #6950
- Fix a bug where errata were not always migrated for new repositories. #7092
- Fix yum metadata files not being migrated. #7093
- Fix an issue causing extremely high memory usage as # of content scale up. #7152
- Fixed a bug where migrated repositories could have multiple different copies of an errata. #7165
- Fixed distribution tree re-migration. #6949
- Fixed RPM migration when its remote is not migrated. #7078
- Migrate checksum_type configuration for an RPM publication. #6813
- Fixed Ruby bindings generation. #7016
- Slightly improve performance by allowing repos to be migrated in parallel. #6374
- As a user, I can track Remotes and not remigrate them on every run. #6375
- Track Publications and Distributions, recreate if needed and not on every run. #6376
- Expose pulp3_repository_version on pulp2content if it is available. #6580
- Ensure that only one migration plan can be run at a time. #6639
- Fixed UnboundLocalError during migration of a repo with a custom name. #6640
- Fix an issue where a migration with many plugin types would crash on execution. #6754
- Fixed distribution creation when a distributor is from a repo which is not being migrated. #6853
- Fixed migration of a sub-set of previously migrated repos. #6886
- Handle already-migrated 're-created' pulp2 repos #6887
- Fixed marking of old distributors, when distributor only is migrated without the repo. #6932
- Fixed case when a publication is shared by multiple distributions. #6947
- Set pulp3_repo relation for all the cases, including remigration. #6964
- Fixed incorrect pulp3_repo_version href for advisories after remigration. #6966
- Fix comps migration when repo is recreated between the migration runs. #6980
- Migrate errata content. #6178
- As a user I can migrate comps content into pulp3. #6358
- As a user I can migrate SRPMS. #6388
- Improve performance by looking only at lazy content types and not through all the migrated content. #6499
- Set properly relative_path Pulp2YumRepoMetadataFile content_artifact. #6400
- Migrate RPM packages to Pulp 3. #6177
- Add custom repo metadata migration. #6283
- As a user I can migrate modules and modules-defaults #6321
- Add awaiting for docker DC resolution and do not use does_batch. #6084
- Do not pre-migrate schema1 docker tags when there are 2 tags with same name witin a repo. #6234
- Moved README to readthedocs website. #6145
- Add pulp3_repository_href to pulp2repositories api. #6053
- Make pulp2 importer optional. #6056
- Migrate empty repos if the migration plan specifies them. #6070
- Handling missing plugin modules #5820
- Fix migration of multiple plugins. #5978
- Add error message for the importers that cannot be migrated. #5984
- Fix the bindings for publication and distribution hrefs fields on pulp2repositories API. #6049
- Fix rendering of the pulp2repositories after a failed migration. #6058
- Handle case when repos are removed and re-created. #6062
- Fix docker repo migration with a custom distributor. #6097
- Fix blobs and manifests relations on migration re-run. #6099
- As a user, I can provide a Migration Plan.
- Migrate iso content.
- Migration plan resources are validated against MongoDB (i.e. that they exist). #5319
- Migrate on_demand content. #5337
- Migrate Pulp 2 repositories into Pulp 3 repo versions. #5342
- As a user, I can migrate Pulp 2 distributor into publication/distribution in Pulp 3 #5343
- Migrate docker content. #5363
- Migration plans are respected. #5450
- Mark and take into account changed or removed pulp2 resources. #5632
- Adding a new endpoint to query the Pulp2-Pulp3 mapping for resources. #5634
- Update get_pulp3_repository_setup so repos are grouped by plugin type. #5845
- Migrate only those repo types that belong to the plugin that is being migrated #5485
- Fix bug preventing the serializer from accepting non-JSON data #5546
- Prevent migration of importers/distributors with an empty config. #5551
- Specify pulp2_distributor_repository_ids instead of distributor_ids #5837
- Importer or distributor can be migrated even if their repository is not. #5852
- Fix "local variable 'pulp2repo' referenced before assignment". #5899
- Fix repository type identification. #5957
- All requested repositories are migrated regardless of the time of the last run or a migration plan change. #5980
- Change __id, __created, __last_updated, __href to pulp_id, pulp_created, pulp_last_updated, pulp_href #5457