kurtcms.org

Digitalocean API: Create and Rotate Volume Snapshot

Posted on June 22, 2022 — 4 Minutes Read

Among the modern internet wonders is the cloud, and of all the cloud service providers none quite compares to DigitalOcean (DO) for its simplicity and affordability. From mature compute products such as scalable virtual machines, known as Droplet, to the latest serverless platform, aptly named Functions, DO provides all the richness and flavours that a developer would need from a cloud, without the excessive offering and pricing complexity that is common among the market leaders. As such DO has been a personal favourite, and has been the Infrastructure as a Service (IaaS) of choice behind this website, and the go-to platform for other personal projects.

At times there may arise a need to backup a certain directory on a Droplet, such as the directory that Docker mounts its volumes in for persisting data across and between container, whether for housing the database and contents for a website, or for storing the outputs and logs of an app; attaching a Volume block storage to the Droplet, and have it mounted under such directory where the critical data will be stored, of which a snapshot may be taken anytime for safekeeping, is a simple yet wonderful solution, that combines the field-proven reliability and resiliency of a traditional compute and storage architecture, with the agility and programmability of the cloud enabled by software abstraction and automation. Tapping into the Application Programming Interface (API) available for the Volume block storage product, what follows will be a Python app that calls the DigitalOcean (DO) API for an automated creation and rotation of snapshot from a volume. The rest of the code is containerised with Docker Compose for a modular and cloud-native deployment that fits in any microservice architecture, and is shared on Github for reference and further development. With the Python app containerised with Docker Compose, deployment is as simple as:

Download a copy of the app;
Create the environment variables for authentication and modify the crontab if needed; and
Docker Compose or build and run the image manually to start the app, or alternatively run the Python script as a standalone service.

Git Clone

Download a copy of the app with git clone. Be sure to pass the --recurse-submodules argument to initialise and update each submodule in the repository.

$ git clone --recurse-submodules https://github.com/kurtcms/do-api-volume-snapshot /app/do-api-volume-snapshot/

Environment Variables

The app expects the base URL, the API token, the DO volume ID and the number of volume snapshot to keep, as environment variables in a .env file in the same directory.

Be sure to create the .env file.

$ nano /app/do-api-volume-snapshot/.env

And define the credentials accordingly.

DO_BASE_URL = 'https://api.digitalocean.com/v2'
DO_TOKEN = '(redacted)'

# ID of the volume
DO_VOLUME_ID = '(redacted)'

# Number of volume snapshot to keep
DO_SNAPSHOT = 1

Crontab

By default the app is scheduled with cron to create and rotate the volume snapshot everyday, with stdout and stderr redirected to the main process for Docker logs.

Modify the crontab if a different schedule is required.

$ nano /app/do-api-volume-snapshot/crontab

Docker Container

Packaged as a container, the app is a standalone, executable package that may be run on Docker Engine. Be sure to have Docker installed.

Docker Compose

With Docker Compose, the app may be provisioned with a single command.

Install Docker and Docker Compose with the Bash script that comes with app.

$ chmod +x /app/do-api-volume-snapshot/docker-compose/docker-compose.sh \
    && /app/do-api-volume-snapshot/docker-compose/docker-compose.sh

Start the containers with Docker Compose.

$ docker-compose -f /app/do-api-volume-snapshot/docker-compose.yml up -d

Stopping the containers is as simple as a single command.

$ docker-compose -f /app/do-api-volume-snapshot/docker-compose.yml down

Build and Run

Otherwise the Docker image can also be built manually.

$ docker build -t do_api_volume_snapshot /app/do-api-volume-snapshot/

Run the image with Docker once it is ready.

$ docker run -it --rm --name do_api_volume_snapshot do_api_volume_snapshot

Standalone Python Script

Alternatively the do_api_volume_snapshot.py script may be deployed as a standalone service.

Dependencies

In which case be sure to install the following required libraries for the vco_api_main.py:

$ pip3 install requests python-dotenv

Cron

The script may then be executed with a task scheduler such as cron that runs it everyday for example.

$ (crontab -l; echo "0 0 * * * /usr/bin/python3 /app/do-api-volume-snapshot/do_api_volume_snapshot.py") | crontab -

Docker Logs

A message will be printed on the creation and the deletion of volume snapshot.

Snapshot 2022-10-21-00-00-02 is created at 2022-10-21T00:00:02Z
Snapshot 2022-10-20-00-00-02 is removed at 2022-10-21T00:00:06Z

Thoughts

Whether it is creating and rotating a Volume snapshot, deploying and scaling a Droplet, or adding or removing node pool from a Kubernetes cluster, wondrous automations can be accomplished with the DO API.