DataWarehouse on CKI Project

Datawarehouse Data Retention Policy

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

The purpose of this document is to define the retention periods for each of the personal data elements or records processed (ie., collected, used, transmitted, shared, stored, archived) by the CKI Datawarehouse governed by the requirements established in Red Hat Corporate Records Management Policy and in compliance with “Data Retention” principle described in Red Hat Global Privacy Policy.

Scope

The scope of this retention policy is limited to personal data elements or records processed by the CKI Datawarehouse and do not apply to upstream or downstream systems/services that could process the same personal data elements or records. Data retention periods of the personal data processed by those upstream or downstream systems/services shall be governed by the data retention policies of those respective systems/services.

DataWarehouse Authorization and Permission

Mon, 01 Jan 0001 00:00:00 +0000

TL;DR

There are 3 visibility (Policy) levels: public, internal, retrigger;
Objects (Issue and KCIDBCheckout + children) visibility is defined by the Policy they are created with;
Admins grant permissions to Users using DW Groups; preferably through LDAP group links;
DW Groups required to triage internal objects: Triagers, policy_public_write, policy_internal_read, policy_internal_write.
CKI team members can be assigned to the DW Group CKI Team, or be enabled as a superuser.

DataWarehouse uses Django Auth Module with Users, Groups and Permissions that limit operations. On top of that, we have object-level permission, combining Groups with our custom PolicyAuthorizationBackend to distinguish between Public and Internal (Red Hat) data, and also data from retriggered pipelines.

Issues

Mon, 01 Jan 0001 00:00:00 +0000

DataWarehouse provides a way to tag failures on Checkouts, Builds and Tests.

Glossary

Issue

A certain anomaly is represented on the DataWarehouse with an Issue entry. Each Issue represents a problem that can be found across different pipelines.

Example: ‘Timeout while doing X task’

Issue Kind

Issues are grouped on Issue Kinds. Each kind represents a different category of failures.

Example: ‘Infrastructure Issue’, ‘Kernel Bug’

Microservices

Mon, 01 Jan 0001 00:00:00 +0000

DataWarehouse provides a way to let external services KCIDB objects updates.

Design

To communicate between the DataWarehouse and the services running, RabbitMQ is used as the message broker.

DataWarehouse publishes all the events described on this page to the cki.exchange.datawarehouse exchange, which is forwarded to cki.exchange.webhooks. Services that want to consume these messages must create a queue and subscribe to this exchange. Using the cki-lib MessageQueue implementation is strongly recommended.

Bot Protection

Mon, 01 Jan 0001 00:00:00 +0000

Bot Protection

DataWarehouse uses Anubis to protect against bots and malicious traffic. Anubis is deployed as a sidecar container that sits in front of the application and provides intelligent bot detection and mitigation.

Requests to the API endpoints (/api/) are always allowed.

Integration

Anubis sits in front of Nginx as the entrypoint service, with Nginx serving as the reverse proxy behind it. The request flow is:

Client → Anubis → Nginx → Django application

Feature flags

Mon, 01 Jan 0001 00:00:00 +0000

Some features are enabled or disabled by setting feature flags on the deployment environment.

These configurations allow customization of certain behaviours of the Datawarehouse.

Variable Name	Default	Description
`FF_CACHE_ANONYMOUS`	`False`	Cache anonymous requests
`FF_DEL_NOT_CONFIRMED_USERS_OLDER_THAN_DAYS`	`30`	Amount of days before not confirmed accounts are deleted
`FF_EXTERNAL_AVATAR_ENABLED`	`True`	Use external sevice for user avatars. By default Librabatar is used
`FF_LDAP_GROUP_SYNC`	`False`	Enable LDAP groups synchronization
`FF_NOTIFY_ISSUE_REGRESSION`	`False`	Enable sending of Issue Regression emails
`FF_SAML_ENABLED`	`False`	Enable SAML2
`FF_OIDC_ENABLED`	`False`	Enable OIDC
`FF_SIGNUP_ENABLED`	`False`	Enable the creation of new accounts on the UI

LDAP Synchronization

Mon, 01 Jan 0001 00:00:00 +0000

It’s possible to synchronize DataWarehouse groups with external LDAP groups by enabling the FF_LDAP_GROUP_SYNC feature flag.

Groups configured to be synced by the LDAP integration are completely managed by it. This means that users manually added to these Django groups will be removed when the cron task runs.

Changes to the LDAP group are reflected when the cron updater runs. Newly added users to the LDAP group automatically get added to the Django group and removed users from the LDAP group are automatically removed from the Django group. This does not affect users added via extra_users.

Enabling SSO SAML2

Mon, 01 Jan 0001 00:00:00 +0000

It’s possible to configure Datawarehouse to rely on a SAML2 IdP for logging in. The djangosaml2 library is used.

Configuration

Certain values are necessary to be configured for SAML2 to work. The following parameters are set by environment variables:

Name	Type	Description
`SAML_ENABLED`	`bool`	Enable/Disable SAML2.
`SAML_CONFIG`	`yaml`	Data necessary for the SAML2 configuration.

In addition to this, certain directories and files specified in the configuration need to be mounted on the given locations.

Caching Views

Mon, 01 Jan 0001 00:00:00 +0000

To improve responsiveness and reduce the load DataWarehouse supports caching anonymous requests, which can be enabled with the FF_CACHE_ANONYMOUS flag.

By default, Django uses local memory caching if no other backend is configured. This approach utilizes per-process memory to store the data.

For production environments, it is possible to use the MEMCACHED_SERVER environment variable to configure the Memcached backend. The format expected is HOST:PORT.

Example

export FF_CACHE_ANONYMOUS=true
export MEMCACHED_SERVER=localhost:11211

manage.py delete_non_public

Mon, 01 Jan 0001 00:00:00 +0000

Delete Non-Public Data Command

The delete_non_public management command is a powerful utility for cleaning up non-public data from the DataWarehouse database. This command is primarily used for data sanitization, testing environments, or preparing data for public distribution.

DESTRUCTIVE OPERATION: This command permanently deletes data from the database. There is no undo functionality.

Usage

usage: manage.py delete_non_public [-h] [--dry-run] [--force]

Delete non-public data from the DataWarehouse (KCIDBCheckouts, Issues, and Users)

options:
  -h, --help            show this help message and exit
  --dry-run             Show what would be deleted without actually deleting anything
  --force               Force deletion without confirmation

Example output

$ python manage.py delete_non_public

Items to be deleted:
- Non-public KCIDBCheckouts: 88319
- Non-public Issues: 182
- All Users: 475

WARNING: This will permanently delete 88976 items from the database.
Are you sure you want to continue? [y/N]: y

Deleted 25427 objects related to Issues (cascaded:
  {'datawarehouse.IssueRegex': 157, 'datawarehouse.IssueOccurrence': 25088, 'datawarehouse.Issue': 182})

2025-07-02T20:11:17.897000 - [DEBUG] - cki.datawarehouse.management.commands.delete_non_public - Deleting orphaned artifacts...
Deleted 47553 objects related to orphaned Artifacts

Using checkout batch size: 245 based on max iid: 245653

2025-07-02T20:13:38.488000 - [DEBUG] - cki.datawarehouse.management.commands.delete_non_public - Batch 1: Deleting KCIDBCheckouts from 0 to 245...
Batch 1: Deleted 54066 objects related to KCIDBCheckouts (cascaded:
  {'datawarehouse.IssueOccurrence': 28, 'datawarehouse.KCIDBTest_maintainers': 17504, 'datawarehouse.KCIDBTest_provenance': 22840, 'datawarehouse.KCIDBBuild_provenance': 403, 'datawarehouse.Report_addr_to': 263, 'datawarehouse.Report_addr_cc': 203, 'datawarehouse.KCIDBCheckout_patches': 220, 'datawarehouse.KCIDBCheckout_contacts': 20, 'datawarehouse.KCIDBCheckout_provenance': 141, 'datawarehouse.KCIDBTest': 11753, 'datawarehouse.Report': 139, 'datawarehouse.KCIDBBuild': 410, 'datawarehouse.KCIDBCheckout': 142})
2025-07-02T20:13:43.633000 - [DEBUG] - cki.datawarehouse.management.commands.delete_non_public - Batch 2: Deleting KCIDBCheckouts from 245 to 490...
Batch 2: Deleted 46484 objects related to KCIDBCheckouts (cascaded: {'datawarehouse.IssueOccurrence': 20, 'datawarehouse.KCIDBTest_maintainers': 14495, 'datawarehouse.KCIDBTest_provenance': 19820, 'datawarehouse.KCIDBBuild_provenance': 392, 'datawarehouse.Report_addr_to': 244, 'datawarehouse.Report_addr_cc': 130, 'datawarehouse.KCIDBCheckout_patches': 505, 'datawarehouse.KCIDBCheckout_contacts': 9, 'datawarehouse.KCIDBCheckout_provenance': 141, 'datawarehouse.KCIDBTest': 10054, 'datawarehouse.Report': 138, 'datawarehouse.KCIDBBuild': 392, 'datawarehouse.KCIDBCheckout': 144})
...1h later...
Batch 1001: ...

Total deleted KCIDBCheckout objects: 62497358
2025-07-02T22:15:47.444000 - [DEBUG] - cki.datawarehouse.management.commands.delete_non_public - Deleting orphaned artifacts...
Deleted 5972906 objects related to orphaned Artifacts
Deleted 475 User objects
Successfully deleted all non-public items.

What Gets Deleted

The command removes the following types of data:

Database

Mon, 01 Jan 0001 00:00:00 +0000

DataWarehouse uses PostgreSQL as database backend.

Configuration via environment variables

Name	Secret	Required	Description
`DB_NAME`	no	no	Database name. Defaults to `datawarehouse`.
`DB_USER`	no	no	Username. Defaults to `datawarehouse`.
`DB_PASSWORD`	yes	yes	Password.
`DB_HOST`	no	yes	Hostname.
`DB_PORT`	no	no	Port. Defaults to `5432`.

Schema

The database schema changes frequently. Running the following command it’s possible to get an up to date visualization.

Podman:

podman exec -it datawarehouse-web ./manage.py graph_models -a > dw.dot
dot -Tx11 dw.dot

Docker Compose:

docker-compose exec web ./manage.py graph_models -a > dw.dot
dot -Tx11 dw.dot

That will give you a nice diagram of the Django models used in the app. You might have to install the graphviz package if you haven’t already.