FAQ
Is STACKIT Intake a managed Kafka service offering?
Section titled “Is STACKIT Intake a managed Kafka service offering?”No. While STACKIT Intake leverages the Apache Kafka Protocol for data ingestion, it is not a general-purpose, managed Kafka service. The key differentiator is that Intake is a one-way, ingestion-only pipeline; you cannot consume data from it. It is designed to act as a data sink, writing messages directly into your Dremio Iceberg tables, not as a message broker for general-purpose inter-service communication or stream processing.
What data formats does STACKIT Intake support?
Section titled “What data formats does STACKIT Intake support?”STACKIT Intake is designed to ingest data in JSON format only. The service automatically infers the data types from your JSON payloads and manages the schema of the target Apache Iceberg table, supporting automatic schema evolution as new fields are introduced.
How does STACKIT Intake handle table schemas?
Section titled “How does STACKIT Intake handle table schemas?”If you do not pre-create the target table in Dremio, STACKIT Intake will automatically generate it upon receiving the first JSON message. The schema of this table is inferred from the structure and data types of that initial message. STACKIT Intake only supports additive schema evolution. This allows you to add new columns to your schema, but it does not permit changing the data type of an existing column. For reliable and consistent ingestion, all subsequent messages should conform to this inferred schema.
How is the throughput capacity for an Intake Runner enforced?
Section titled “How is the throughput capacity for an Intake Runner enforced?”The throughput capacity you define is not a strict per-hour limit. Instead, the maxMessagesPerHour and maxMessageSizeKiB metrics are used to calculate the size of an internal buffer that stores messages for up to 24 hours. The Intake Runner enforces capacity by blocking new messages from being ingested if this buffer fills up, regardless of how quickly it filled. Once messages are successfully flushed to Dremio, buffer space is freed, and the Intake Runner resumes accepting messages. This mechanism allows the system to absorb short, high-volume spikes without dropping messages.
Can I increase or decrease the throughput capacity of an Intake Runner after it’s been created?
Section titled “Can I increase or decrease the throughput capacity of an Intake Runner after it’s been created?”You can increase the capacity of an existing Intake Runner if your data volume grows. However, decreasing the capacity is not supported at the moment. To reduce the throughput, you must delete and then recreate the Intake Runner and its Intakes, pointing to your existing Dremio tables.
Does STACKIT Intake store my data permanently?
Section titled “Does STACKIT Intake store my data permanently?”No. STACKIT Intake is designed as a temporary data buffer, not a permanent data store. It holds messages for up to 24 hours to ensure reliable delivery to Dremio. Once the data is successfully written to your Dremio Iceberg table, it is removed from the Intake’s buffer.
Is my data secure in transit and at rest?
Section titled “Is my data secure in transit and at rest?”Yes. Data is secured in transit to the Intake Runner via the Apache Kafka Protocol, which uses SASL authentication and SSL/TLS encryption. Data is then held in a temporary, managed buffer. The permanent storage and security of your data are handled by your Dremio-managed Iceberg tables within the STACKIT Data Platform.