Create your first Intake and send data to it

Prerequisites

• You have a STACKIT customer account: Create a customer Account
• You have a STACKIT user account: Create a user account
• You have a STACKIT project: Create a project
• You have a Dremio instance
• You have configured Dremio for Intake access: Configure Dremio for STACKIT Intake integration
• You have installed the STACKIT CLI on your system: User Guide
• You have a running terminal/command shell

Set up environment variables

To make the commands easier to run, export the following environment variables in your terminal. This prevents you from having to manually replace values in every command.

• PROJECT_ID: Your project ID from STACKIT Portal.
• DREMIO_CATALOG_URI: The API endpoint for your Dremio Iceberg catalog. You can get this from the Dremio section in the STACKIT portal. Usually: https://dremio-<your-dremio-instance-name>-catalog.data-platform.stackit.run/iceberg/
• DREMIO_WAREHOUSE: The name of the Dremio Iceberg catalog warehouse to connect. Usually: catalog-s3
• DREMIO_TOKEN_ENDPOINT: The URL to request Dremio authentication tokens. Usually: https://dremio-<your-dremio-instance-name>.data-platform.stackit.run/oauth/token
• DREMIO_PERSONAL_ACCESS_TOKEN: Your secret key for authentication. See how to get one in this guide.

export PROJECT_ID="YOUR_PROJECT_ID"
export DREMIO_CATALOG_URI="YOUR_DREMIO_CATALOG_URI"
export DREMIO_WAREHOUSE="YOUR_DREMIO_WAREHOUSE"
export DREMIO_TOKEN_ENDPOINT="YOUR_DREMIO_TOKEN_ENDPOINT"
export DREMIO_PERSONAL_ACCESS_TOKEN="YOUR_DREMIO_PERSONAL_ACCESS_TOKEN"

Authentication: using stackit CLI

To interact with the STACKIT Intake API, you need to authenticate your requests. This guide uses STACKIT CLI that handles authentication for you.

stackit auth login

Create an Intake Runner

The Intake Runner is the engine for your data ingestion. You must create a runner before you can create an Intake. To create an Intake Runner, issue a call to STACKIT CLI with your desired configuration, pressing y if prompted for a confirmation:

• displayName: A human-readable name for your runner.
• maxMessageSizeKiB: The maximum size, in Kibibytes, for a single message.
• maxMessagesPerHour: The maximum number of messages the runner can process per hour.

stackit beta intake runner create \
  --display-name "mytestrunner" \
  --max-message-size-kib 1000 \
  --max-messages-per-hour 1000

The command will return the details of the new runner, something like:

Are you sure you want to create an Intake Runner for project "..."? [y/N] y
Creating STACKIT Intake Runner instance ✓
Created Intake Runner for project "...". Runner ID: ca72a2f8-89bf-4800-88e9-166a82f5c5ff

Capture the id from the response. You will need this Intake Runner ID for the next steps. You can export it as a variable for convenience: export INTAKE_RUNNER_ID="<PASTE_THE_ID_HERE>"

Check your Intake Runner

To get the latest status of your runner:

stackit beta intake Runner describe $INTAKE_RUNNER_ID

You will see all the properties of your runner:

ATTRIBUTE              │ VALUE
────────────────────────┼───────────────────────────────────────────────────────────────────────
 ID                     │ ca72a2f8-89bf-4800-88e9-166a82f5c5ff
 Name                   │ mytestrunner
 State                  │ active
 Created                │ 2025-10-16 16:33:04 +0000 UTC
 Labels                 │ <nil>
 Description            │
 Max Message Size (KiB) │ 1000
 Max Messages/Hour      │ 1000
 Ingestion URI          │ ca72a2f8-89bf-4800-88e9-166a82f5c5ff.intake.eu01.onstackit.cloud:9094

Create an Intake

An Intake is the data pipe that connects the stream of data ingested to your target Dremio Iceberg table.

To create an Intake, send a POST request specifying the intakeRunnerId from the previous step and the details of your Dremio catalog.

stackit beta intake create \
  --display-name "myintaketest" \
  --runner-id "$INTAKE_RUNNER_ID" \
  --catalog-uri "$DREMIO_CATALOG_URI" \
  --catalog-warehouse "$DREMIO_WAREHOUSE" \
  --catalog-table-name "mytesttable" \
  --catalog-partitioning manual \
  --catalog-partition-by "id" \
  --catalog-auth-type "dremio" \
  --dremio-token-endpoint "$DREMIO_TOKEN_ENDPOINT" \
  --dremio-pat "$DREMIO_PERSONAL_ACCESS_TOKEN"

If you do not provide a namespace, the default intake Dremio folder will be used. Notice that if your DREMIO_PERSONAL_ACCESS_TOKEN does not have the correct permissions for the folder selected, the Intake data will not be able to show up correctly in Dremio. Check the configuration required in this Dremio guide.

partitionBy is optional and is evaluated only at table creation. When you point your Intake to an existing table, partitionBy will not modify the partitioning.

Column data types are inferred from the JSON payload. Because of this automatic inference, most transform partitions such as day(ts) can be used only if you create the table ahead of time and explicitly specify both the schema and the partition transform. Once a column type is fixed (e.g. TIMESTAMP), Iceberg automatically casts conforming string literals from the JSON into the correct type during ingest.

The CLI call will return the ID of the Intake. Capture the id from the response, as you will need it to create users.

export INTAKE_ID="<PASTE_THE_INTAKE_ID_HERE>"

Check your Intake

To get the latest status of a specific intake:

stackit beta intake describe $INTAKE_ID

You will see all the properties of the Intake:

ATTRIBUTE            │ VALUE
──────────────────────┼───────────────────────────────────────────────────────────────────────
 ID                   │ 27ea0ab9-e514-442a-a2d0-e8db0edcc2dc
 Name                 │ myintaketest
 State                │ active
 Runner ID            │ ca72a2f8-89bf-4800-88e9-166a82f5c5ff
 Created              │ 2025-10-16 16:50:41 +0000 UTC
 Labels               │ <nil>
──────────────────────┼───────────────────────────────────────────────────────────────────────
 Ingestion URI        │ ca72a2f8-89bf-4800-88e9-166a82f5c5ff.intake.eu01.onstackit.cloud:9094
 Topic                │ intake-27ea0ab9-e514-442a-a2d0-e8db0edcc2dc
 Dead Letter Topic    │ deadletter-intake-27ea0ab9-e514-442a-a2d0-e8db0edcc2dc
 Undelivered Messages │ 0
──────────────────────┼───────────────────────────────────────────────────────────────────────
 Catalog URI          │ https://dremio-intake-catalog.data-platform.stackit.run/iceberg/
 Catalog Warehouse    │ catalog-s3
 Catalog Namespace    │ intake
 Catalog Table Name   │ mytesttable
 Catalog Partitioning │ manual
 Catalog Partition By │ id

Create an Intake user

To send data to your Intake, you must create an Intake user with the intake type and provide a secure password (12 characters minimum, with at least one upper case, lower case, and special character, as well as at least one number). Consult Create and manage Intake users to get more details.

stackit beta Intake user create --intake-id "$INTAKE_ID" \
  --display-name "myIntakeTopicUser" \
  --password "SuperSecr3t12**??Passw0rd" \
  --type intake

The response will include a the ID of the created user. Record it for future use:

export INTAKE_USER_ID="<PASTE_THE_INTAKE_USER_ID_HERE>"

Capture the id from the response.

export INTAKE_USER_ID="<PASTE_THE_INTAKE_USER_ID_HERE>"

Check the Intake user

You can look at the details of the created user:

stackit beta Intake user describe "$INTAKE_USER_ID" --intake-id "$INTAKE_ID"

You should see a response like this:

ATTRIBUTE                │ VALUE
──────────────────────────┼──────────────────────────────────────────────────
 ID                       │ 8d06eb8c-1ef6-49be-9518-77ed2c7900c8
 Name                     │ myIntakeTopicUser
 State                    │ active
 Created                  │ 2025-10-16 16:57:25 +0000 UTC
 Labels                   │ <nil>
 Type                     │ intake
 Username                 │ intake-user-8d06eb8c-1ef6-49be-9518-77ed2c7900c8
──────────────────────────┼──────────────────────────────────────────────────
 Java Client Config       │
──────────────────────────┼──────────────────────────────────────────────────
 librdkafka Client Config │

The password you provide is not stored in a retrievable format. Ensure you store it securely.

Create a deadletter Intake user

To inspect messages that fail processing, create a user with read-only access to the Dead Letter Queue (DLQ) by setting the type to dead-letter.

stackit beta Intake user create --intake-id "$INTAKE_ID" \
  --display-name "myDeadletterTopicUser" \
  --password "SuperSecr3t12**??Passw0rd" \
  --type dead-letter

This user will only have permission to read from the DLQ topic. Make sure to capture the id from the response:

export INTAKE_USER_DLQ_ID="<PASTE_THE_INTAKE_USER_DLQ_HERE>"

Check Your Intake users

List all Intake users for an Intake in your project:

stackit beta Intake user list --intake-id "$INTAKE_ID"

Send data to an Iceberg table via your Intake

After completing the setup, you can verify that everything works by sending a test message to your Intake topic using a Kafka client like kcat.

echo '{"message": "hello world from kcat!", "id": 123}' | \
kcat -b $INTAKE_RUNNER_ID.intake.eu01.onstackit.cloud:9094 \
  -t intake-"$INTAKE_ID"  \
  -P \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanisms=SCRAM-SHA-512 \
  -X sasl.username=intake-user-"$INTAKE_USER_ID" \
  -X sasl.password='<YOUR_SECURE_PASSWORD_FOR_INTAKE_USER>'

After sending the message, query your target table in Dremio UI to confirm that the record has been successfully ingested.

You can also read from the Dead-Letter topic:

kcat -b $INTAKE_RUNNER_ID.intake.eu01.onstackit.cloud:9094 \
  -t deadletter-intake-"$INTAKE_ID"  \
  -C \
  -X security.protocol=SASL_SSL \
  -X sasl.mechanisms=SCRAM-SHA-512 \
  -X sasl.username=intake-user-"$INTAKE_USER_DLQ_ID" \
  -X sasl.password='<YOUR_SECURE_PASSWORD_FOR_DEADLETTER_USER>'

Clean up

You can delete the Intake Runner with the following commands. This will also delete the Intakes and Intake users.

stackit beta Intake user delete ""$INTAKE_USER_ID" --intake-id "$INTAKE_ID"
stackit beta Intake user delete "$INTAKE_USER_DLQ_ID" --intake-id "$INTAKE_ID"
stackit beta intake delete "$INTAKE_ID"
stackit beta intake runner delete "$INTAKE_RUNNER_ID"

You can list all runners and wait for yours to disappear:

stackit beta intake runner list

While deleting, the runner is in state deleting.

Congratulations, you created your first Intake pipeline and sent data to it. From here you can dig deeper with the How-Tos.