Kafka

Install

See the Install guide for the full setup, including Windows PowerShell.

curl -fsSL https://install.skippr.io/install.sh | shClick to copy

Consumes messages from a Kafka topic.

Configuration

source:
  kind: kafka
  brokers: "localhost:9092"
  topic: events
  group_id: skippr-consumer

Field	Default	Description
brokers	(required)	Kafka bootstrap servers
topic	(required)	Topic to consume
group_id	auto-generated	Consumer group ID
auto_offset_reset	earliest	earliest or latest
security_protocol		Security protocol
sasl_mechanism		SASL mechanism
sasl_username / sasl_password		SASL credentials
mode	stream	stream or batch

Namespace: kafka.{topic}

CLI

skippr connect source kafka \
  --brokers localhost:9092 \
  --topic events \
  --group-id skippr-consumer \
  --auto-offset-reset earliest \
  --security-protocol SASL_SSL \
  --sasl-mechanism PLAIN \
  --sasl-username ${KAFKA_SASL_USERNAME} \
  --sasl-password ${KAFKA_SASL_PASSWORD} \
  --mode stream

Flag	Description
--brokers	Kafka bootstrap servers
--topic	Topic to consume
--group-id	Consumer group ID
--auto-offset-reset	Offset reset policy such as earliest or latest
--security-protocol	Broker security protocol
--sasl-mechanism	SASL mechanism
--sasl-username	SASL username
--sasl-password	SASL password
--mode	Consumption mode

Authentication

Kafka supports unauthenticated local development as well as secured broker setups. For security best practices, we strongly advise against storing SASL credentials in skippr.yaml. Use environment variable interpolation instead: replace the sasl_username or sasl_password value with your own ${ENV_VAR} reference.

The relevant part of skippr.yaml looks like this:

source:
  kind: kafka
  security_protocol: SASL_SSL
  sasl_mechanism: PLAIN
  sasl_username: ${KAFKA_SASL_USERNAME}
  sasl_password: ${KAFKA_SASL_PASSWORD}

Set the env vars before running skippr:

macOS / Linux

export KAFKA_SASL_USERNAME="myuser"
export KAFKA_SASL_PASSWORD="mypassword"

Windows PowerShell

$env:KAFKA_SASL_USERNAME = "myuser"
$env:KAFKA_SASL_PASSWORD = "mypassword"

Windows Command Prompt

set KAFKA_SASL_USERNAME=myuser
set KAFKA_SASL_PASSWORD=mypassword

Permissions or Network Requirements

The runner must be able to reach the bootstrap brokers, join the configured consumer group, and read the selected topic. For secured clusters, make sure broker ACLs allow topic consumption and group commits. For Debezium-based CDC, see CDC Sources -- Kafka.

Troubleshooting

Symptom	Fix
connection or SASL failures	Verify broker addresses, security protocol, SASL settings, and network access to the cluster.
messages are not arriving	Check the topic name, consumer group, and whether the producer is publishing to the expected topic.

CDC Support

Kafka supports real-time Change Data Capture via Debezium envelope parsing. When paired with a supported warehouse destination, Skippr reconciles those mutations to exactly-once final state.

See CDC Sources -- Kafka for prerequisites, configuration, and resume behavior.

Next steps

• Pair this source with a destination from Destination Connectors.
• For Debezium-based CDC behavior, see CDC Sources and CDC Operations.

Install

See the Install guide for the full setup, including Windows PowerShell.

curl -fsSL https://install.skippr.io/install.sh | shClick to copy