~"backend-weight::3" ### Problem to solve Currently, the [Cleanup policy for tags](https://docs.gitlab.com/ee/user/packages/container_registry/#managing-project-expiration-policy-through-the-ui) feature is only allowed for projects created after 12.8. We would like to turn on the feature for all projects, but there are performance concerns because tag deletion can be slow. ### Intended users * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) * [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) ### Further details ### Proposal When we turn on the policy feature for all projects, we need a way to prevent the sidekiq queue from backing up and slowing things to a halt, especially if there are cases when there are large amounts of tags to be cleaned up for a given policy on it's first-time running. To prevent thousands of tags from being queued by one project, we will add throttling to the queue or service. ### Permissions and Security - There are no permissions changes required for this change. ### Documentation - Add a section to the [Container Registry Admin docs](https://docs.gitlab.com/ee/administration/packages/container_registry.html) that jobs will be throttled and how to troubleshoot when something goes wrong. ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> ### What does success look like, and how can we measure that? We are able to enable container expiration and retention policies for all projects on GitLab.com and self-managed instances regardless of which container registry is being used. ### Other concerns * Before turning on the expiration policy feature for all projects, please open a scalability issue: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/new?issuable_template=Review%20Request ### Links / references - [Allow certain Sidekiq jobs to be throttled](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7292)