Migrate data from Redis to a Key Value Store instance

In this tutorial, you learn how to migrate data from an existing STACKIT Redis instance to a target Key Value Store instance.

Requirements

• Valkey CLI / Redis CLI: (CLI client for database connections)
• gunzip: (Command-line utility used to decompress files)
• openssl: (Toolkit for SSL and TLS protocols, offering secure network communication)
• CF CLI version 7: (CLI for Cloud Foundry, simplifying app and space management)

Before the migration

Install tools

To ensure a smooth migration, proceed with installing the tools listed in the requirements. We recommend consulting the official documentation of each tool for installation. Depending on your operating system, you can utilize package managers such as apt for Linux, homebrew for macOS, or choco (Chocolatey) for Windows to facilitate the installation process.

After installation, check that each tool is operational by invoking a version command (e.g., valkey-cli -v).

Isolate the source database

It is highly recommended to isolate the source Redis instance before starting the migration. This avoids potential data inconsistencies caused by the database receiving additional data while simultaneously processing the migration.

The goal is to avoid client connections to ensure that no additional data is written to the source instance during the migration process. This can be achieved by unbinding/disconnecting all applications from your source Redis instance.

Create a backup of the source database

We strongly recommend triggering a manual backup of your source database as a fail-safe in the event of unexpected data loss. This backup must be stored securely until the consistency of the target database has been confirmed post-migration.

1. Open the Service Dashboard to manually trigger a backup of your source Redis instance. Ensure you have an encryption password set and stored safely.
2. Download the latest backup from the Redis Service Dashboard and store it securely.

For more details, refer to the backup documentation for your source instance.

Create a target STACKIT Key Value Store database

Create an empty Key Value Store instance that acts as the target for the data migration.

Create a target instance with the same capabilities (RAM/Storage/Nodes) or greater than your source instance to ensure it can handle the dataset.

Do not insert any data into the target instance until the migration and consistency checks are fully completed.

Create credentials

Create credentials for your target Key Value Store database using the STACKIT Portal or the stackit key-value-store credentials create CLI command. If you are using the Cloud Foundry Runtime, you can generate these through the CF CLI.

Database connection strings

STACKIT Portal

1. Open your project in the STACKIT Portal.
2. Navigate to Databases > Key Value Store.
3. Choose your instance and go to the overview.
4. On the sidebar, select Credentials.
5. Click on the service key entry you want to view to retrieve your connection strings.

Cloud Foundry CLI

1. Target your desired organization and space:

   cf target -o [your-organization-name] -s [your-space-name]

2. List the services to find your target Key Value Store instance (appcloud-keyvalue8):

   cf services

3. Create a service key for your Data Service and view it:

   cf create-service-key [your-key-value-instance-name] [your-key-name]
   cf service-key [your-key-value-instance-name] [your-key-name]

Reformat the obtained service keys as database connection strings if they are not already provided in URI format:

• Connection String Format: rediss://[username]:[password]@[hosts-fqdn]:[port]

Always use the exact value of host from your credentials for both the source and target databases!

Retrieve the backend host for migration

This step is required only when migrating to an exposed STACKIT Key Value Store Data Service created through the Portal.

The MIGRATE command requires a direct TCP connection from the source to the target database. Using exposed frontend hosts (e.g., kvd297348-main-3.data.eu01.onstackit.cloud) as the target cause the migration to fail. Instead, you must use the backend host for the target database.

You do not need to order a Cloud Foundry Runtime Service to execute this step; you only need to log in to the CF CLI to extract the internal route.

1. Log in with the CF CLI. Consult Interact with Cloud Foundry for detailed instructions.
2. Select your Cloud Foundry organization. A typical organization has the naming schema: stackit_portal_prod_[portal_project_name]_id.

   cf target -o stackit_portal_prod_my_cloud_4eFgBq5P

3. List your Data Service Instances using cf services to identify your target database by name. The correct backend instance contains -internal as a name suffix.
4. Create a service key for the internal backend:

   cf create-service-key [target-instance-name-internal] [your-key-name]

5. Show the service key and store the JSON output, as the host provided here is your true backend route:

   cf service-key [target-instance-name-internal] [your-key-name]

Handling TLS certificates

Depending on where your instances are hosted, you require a specific CA (Certificate Authority) certificate to establish your secure TLS connection during the migration.

Exposed services (STACKIT portal)

If your instance uses public certificates issued by SwissSign:

• Public projects: Modern operating systems natively trust SwissSign. You can usually connect by using the --tls flag. If your system does not trust it, download the SwissSign Root and Intermediate CAs from the SwissSign Repository and pass them through the --cacert flag.
• Internal projects (Schwarz): Corporate devices natively trust the internal network certificates by default.

Cloud Foundry runtime

If your database runs within the Cloud Foundry runtime, it uses a private Certificate Authority. You must extract this certificate from your service key to connect. Locate the cacrt field in the JSON output, replace the \n characters with actual line breaks, and save the output to a file named cf_ca.pem. Pass this using --cacert /path/to/cf_ca.pem.

Migration process

Main strategy: Migration by MIGRATE command

The main strategy to migrate the whole dataset from the source to the target is based on the MIGRATE command. Using the CLI, we connect to the existing instance and pipe the keys directly to the target instance.

Based on internal tests, migrating 1 GB takes about 50 seconds on average, but this depends on network latency and dataset structure. Plan the migration process accordingly.

If your source database contains keys with names containing special characters (e.g., breakline characters), this specific CLI pipe migration process may fail. In that case, use the Alternative Script strategy below.

Because Key Value Store requires a username and password, the target command uses AUTH2. Run the following command, replacing the placeholders with your actual values:

valkey-cli -u "[source-db-connection-string]" -n [db-num] \
--tls --sni [source-frontend-hosts-fqdn] \
--cacert [/path/to/ca-cert.pem] --raw KEYS '*' | \
xargs valkey-cli -u "[source-db-connection-string]" -n [db-num] \
--tls --sni [source-frontend-hosts-fqdn] --cacert [/path/to/ca-cert.pem] \
MIGRATE [target-backend-hosts-fqdn] [target-backend-port] "" [db-num] [timeout] \
COPY AUTH2 [target-username] [target-password] KEYS > migration-output-checker.txt

Explanation of the command

• Source CLI: We connect to the source (-u "[source-db-connection-string]") and use --raw KEYS '*' to dump a raw list of all keys.
• xargs: Takes the list of keys from the source and feeds them to the next command.
• Target CLI (MIGRATE): We connect to the source again, but trigger the MIGRATE command, pointing it directly at the target backend ([target-backend-hosts-fqdn]).
• COPY AUTH2: Because the target is a Key Value Store instance, we must pass AUTH2 followed by the target’s [username] and [password].

Alternative strategy: Migration by script

If the MIGRATE feature is failing due to special characters in keys, you can use a Python script called copy_redis.py to copy and verify all keys from the source to the target database: copy_redis.zip. Follow the instructions provided in the README file inside the zip.

Migration through Cloud Foundry SSH tunnel

To migrate data into a Cloud Foundry Runtime Key Value Store Data Service, an SSH forward tunnel is required to securely access the internal backend.

1. Bind the Application to the Target Service:

   cf bind-service [your-cf-app-name] [your-key-value-service-instance]

2. Restage the application:

   cf restage [your-cf-app-name]

3. Create the SSH tunnel:

   cf ssh [your-cf-app-name] -L 6379:[target-backend-host]:[target-backend-port]

4. Run the migration: With the tunnel open in one terminal, you can run the migration command in a second terminal, using localhost:6379 as the target for the MIGRATE command.

Check for migration errors

The migration-output-checker.txt file contains the output of the MIGRATE command. Verify that the migration ran successfully by checking the file’s contents:

cat migration-output-checker.txt | grep --invert-match "OK"

The output of this command must be empty. If it is not, it returns an error message detailing what went wrong.

Error message	Possible cause	Troubleshooting
NOKEY	The key does not exist in the source instance.	Double-check if the first part of the command (--raw KEYS '*') ran.
IOERR error or timeout	A communication problem between the two instances.	Verify that you used the correct target backend host (not the frontend/exposed host) for the migration.
ERR invalid password or WRONGPASS	Authentication failed on the target.	Ensure you are using AUTH2 with the correct username and password for the target Key Value Store instance.

Check database consistency

Check the database consistency between the original source database and the target database to ensure all data was migrated.

Compare data between the instances

Run the INFO KEYSPACE command against both hosts. The keys and expires values must be equal.

On the source DB:

valkey-cli -u "[source-db-connection-string]" --tls --sni [source-frontend-hosts-fqdn] --cacert [/path/to/ca_cert.pem] INFO KEYSPACE

On the target DB:

valkey-cli -u "[target-db-connection-string]" --tls --sni [target-frontend-hosts-fqdn] --cacert [/path/to/ca_cert.pem] INFO KEYSPACE

Next, compare memory usage. The values should be identical (with minor differences allowed due to engine overhead).

valkey-cli -u "[db-connection-string]" --tls INFO MEMORY | grep "used_memory:\|used_memory_human"

Compare keys between the instances

This step exports and compares the literal key names from both instances.

Get all source keys and sort them:

valkey-cli -u "[source-db-connection-string]" --tls --raw KEYS '*' > source.txt && sort --parallel=2 -uo source-sorted.txt source.txt

Get all target keys and sort them:

valkey-cli -u "[target-db-connection-string]" --tls --raw KEYS '*' > target.txt && sort --parallel=2 -uo target-sorted.txt target.txt

Ensure both files are not empty, then compare them:

diff source-sorted.txt target-sorted.txt

The output must be completely empty, indicating zero differences between the datasets.

Assets

Download the Python migration script: copy_redis.zip.