Skip to content
Products
Platform
Developer Tools
Portal

Create and Manage Intakes

Prerequisites

Section titled “Prerequisites”

To begin creating and managing your Intakes via the STACKIT CLI, ensure you have the following in place:

Create an Intake

Section titled “Create an Intake”

An Intake defines the data pipeline that connects your data stream to the target Dremio Iceberg table.

Decide on Dremio Connection and Partitioning

Section titled “Decide on Dremio Connection and Partitioning”

The Intake must be configured with details about the Dremio Iceberg REST catalog into which to write the data stream. Also, one needs to decide about the partitioning scheme:

  • none: Data is not partitioned. This is the default. For production workloads, this may not be advisable since table optimizations such as compactions would conflict with the arrival of new data.
  • intake-time: Data is partitioned by day based on the time Intake received the data. This time is reflected in the field __intake_ts which Intake automatically adds to each received message. This allows you to perform optimizations on older partitions while data is still arriving in the current day’s partition.
  • manual: You can specify an arbitrary field of your data stream to use for partitioning.

  1. Open your project in the STACKIT Portal.

  2. Navigate to Data & AI > Intake.

  3. In sidebar, click on Intakes.

  4. In the topbar, click on Create Intake.

  5. Fill in the fields:

    FieldDescriptionOptionsDefaultOptional
    Namea human-readable name for the IntakeString, not longer than 32 characters
    Descriptiona textual description of the Intake’s purposeString, not longer than 1024 characters""✓
    Intake Runnerthe ID of the Intake Runner into which the Intake will be deployedUUID of the runner
    Table namethe name of the Iceberg table to receive the data streamString, not longer than 1024 characters”intake-<INTAKE_ID>“✓
    Iceberg warehousethe name of the warehouse in the catalogUsually catalog-s3
    Table namespacethe namespace in which the Iceberg table gets created in the catalogString, not longer than 1024 characters”intake”✓
    Table partitioningthe partition scheme to applyOne of:<br>• none<br>• intake-time<br>• manualnone✓
    Dremio PATthe PAT of the user to access DremioString, not longer than 1024 characters
    Iceberg catalog endpointthe URL of the Dremio Iceberg REST catalog endpointUsually https://dremio-<your-dremio-instance-name>-catalog.data-platform.stackit.run/iceberg/
    Dremio token endpointthe URL of the Dremio OAuth 2 token endpointUsually https://dremio-<your-dremio-instance-name>.data-platform.stackit.run/oauth/token

View an Intake

Section titled “View an Intake”
  1. Open your project in the STACKIT Portal.
  2. Navigate to Data & AI > Intake.
  3. In sidebar, click on Intakes.
  4. Click on the Intake you want to view.
  5. The details of the Intake will be displayed, including its configuration and status.

List all Intakes

Section titled “List all Intakes”
  1. Open your project in the STACKIT Portal.
  2. Navigate to Data & AI > Intake.
  3. In sidebar, click on Intakes.
  4. A list of all Intakes associated with your project will be displayed, along with their statuses and key details.

Update an Intake

Section titled “Update an Intake”

  1. Open your project in the STACKIT Portal.

  2. Navigate to Data & AI > Intake.

  3. In sidebar, click on Intakes.

  4. Click on the Intake you want to update.

  5. Click on the Edit button.

  6. Modify the desired fields and click Save to apply the changes:

    FieldDescriptionOptionsDefaultOptional
    Namea human-readable name for the IntakeString, not longer than 32 characters
    Descriptiona textual description of the Intake’s purposeString, not longer than 1024 characters""✓
    Table namethe name of the Iceberg table to receive the data streamString, not longer than 1024 characters”intake-<INTAKE_ID>“✓
    Iceberg warehousethe name of the warehouse in the catalogUsually catalog-s3
    Table namespacethe namespace in which the Iceberg table gets created in the catalogString, not longer than 1024 characters”intake”✓
    Table partitioningthe partition scheme to applyOne of:<br>• none<br>• intake-time<br>• manualnone✓
    Dremio PATthe PAT of the user to access DremioString, not longer than 1024 characters
    Iceberg catalog endpointthe URL of the Dremio Iceberg REST catalog endpointUsually https://dremio-<your-dremio-instance-name>-catalog.data-platform.stackit.run/iceberg/
    Dremio token endpointthe URL of the Dremio OAuth 2 token endpointUsually https://dremio-<your-dremio-instance-name>.data-platform.stackit.run/oauth/token

Delete an Intake

Section titled “Delete an Intake”

  1. Open your project in the STACKIT Portal.

  2. Navigate to Data & AI > Intake.

  3. In sidebar, click on Intakes.

  4. Click on the three dots to the right of the Intake you want to delete.

  5. Select Delete.

  6. Confirm the deletion by entering the Intake name.

  7. Press Delete.

    Deleting an Intake is an irreversible operation. It will also delete all associated Intake Users.