Managing data with DVC

For data analysis, and especially for machine learning, it is extremely valuable to be able to reproduce different versions of analyses that were performed with different data sets and parameters. However, in order to obtain reproducible analyses, both the data and the model (including algorithms, parameters, etc.) must be versioned. Due to the size of the data, versioning data for reproducible analyses is a bigger problem than versioning models. Tools such as DVC help with data management by allowing users to transfer data to a remote data storage location using a Git-like workflow. This simplifies the retrieval of specific versions of data to reproduce an analysis.

DVC was developed to enable the sharing and traceable management of ML models and data sets. It uses its own system for storing files with support for SSH and HDFS, among others.

Tip

cusy seminar: Storing code and data in a versioned and reproducible manner

See also

• Get Started with DVC

• Documentation

• Git Repository

Comparison with related technologies

git-annex

git-annex focuses more on discovering and using datasets, which are then easily managed with Git. DVC, on the other hand, stores the data generated at each step of the pipeline in .dvc files, which can then be managed by Git. DVC also provides handy tools for manipulating and visualising data pipelines, see for example dvc status. Finally, dvc remote can also be used to specify external dependencies.

Workflow management systems such as Airflow and Luigi

DVC focuses on data science workflows and modelling; therefore, DVC pipelines are much lighter, easier to create and modify than with Airflow and Luigi. However, DVC lacks advanced features such as execution monitoring, optimisation and fault tolerance. DVC is also a pure command line tool with no graphical user interface, and it does not run daemons or servers. CML attempts to fill some of these gaps in a lightweight manner with GitHub, GitLab, or Bitbucket. However, DVC and CML are well suited for iterative machine learning processes; and once a good model has been found with the two, you are still free to integrate the pipeline into Luigi or Airflow.

Installation

DVC can be installed with uv. Please note, however, that you must explicitly specify the extras. These can be [ssh], [s3], [gs], [azure], and [oss] or [all]. For ssh, the command looks like this:

$ uv add dvc[ssh]

Alternatively, DVC can also be installed via other package managers:

Debian/Ubuntu

$ sudo wget https://dvc.org/deb/dvc.list -O /etc/apt/sources.list.d/dvc.list
$ sudo apt update
$ sudo apt install dvc

macOS

$ brew install iterative/homebrew-dvc/dvc