[Kafka] Add optional ingress TCP proxy for Kafka external access by shay79il · Pull Request #265 · mlrun/ce

shay79il · 2026-02-25T09:54:19Z

Add an alternative to NodePort for external Kafka connectivity that provides the same behavior as the old Bitnami Kafka setup.

Changes:

Add kafka-ingress-tcp.yaml template that creates tcp-services ConfigMap for ingress-nginx
Add kafka.ingress.* values (disabled by default)
When enabled, external clients can connect via ingress on port 9094 -> kafka...domain:9094
This option is disabled by default.

Users can choose between:

NodePort (default): port 30094
Ingress TCP proxy: port 9094

CEML-651

Copilot

Pull request overview

Adds a Helm-chart option to expose Kafka externally via an ingress-nginx TCP proxy (as an alternative to NodePort), by generating a tcp-services ConfigMap entry and introducing new kafka.ingress.* values.

Changes:

Added kafka.ingress.* values (disabled by default) to configure ingress-nginx TCP proxy behavior.
Added a Helm template to create a tcp-services ConfigMap mapping for Kafka external access.
Updated Kafka listener defaults to use an external NodePort listener on 9094 and bumped chart version.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
charts/mlrun-ce/values.yaml	Adds ingress TCP proxy configuration values and updates the default external listener to NodePort.
charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml	Introduces a hook-created ConfigMap intended to configure ingress-nginx TCP proxying to Kafka.
charts/mlrun-ce/Chart.yaml	Bumps chart version to `0.11.0-rc.12`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

charts/mlrun-ce/values.yaml

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

charts/mlrun-ce/values.yaml

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

charts/mlrun-ce/values.yaml

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

charts/mlrun-ce/values.yaml

Copilot · 2026-02-25T12:11:38Z

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ $configMapName }}
+  namespace: {{ $ingressNamespace }}


The template creates a ConfigMap in a different namespace (ingress-nginx) than the release namespace. This requires cluster-level permissions (ability to create ConfigMaps in any namespace), which may not be available in restricted environments or with least-privilege RBAC configurations.

When Helm runs with namespace-scoped permissions (e.g., when users can only manage resources in their own namespace), this template will fail because it tries to create a resource in the ingress-nginx namespace.

Consider documenting this RBAC requirement clearly in the values.yaml comments and potentially providing a pre-flight check or clearer error message. The existing namespace lookup (line 30-33) partially addresses this but only checks if the namespace exists, not if the user has permissions to create ConfigMaps there.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

charts/mlrun-ce/values.yaml

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

charts/mlrun-ce/values.yaml

Add an alternative to NodePort for external Kafka connectivity that provides the same behavior as the old Bitnami Kafka setup. Changes: - Add kafka-ingress-tcp.yaml template that creates tcp-services ConfigMap for ingress-nginx - Add kafka.ingress.* values (disabled by default) When enabled, external clients can connect via ingress on port 9094 -> kafka.<ns>.<cluster>.domain:9094 - This option is disabled by default. Users can choose between: - NodePort (default): port 30094 - Ingress TCP proxy: port 9094 [CEML-651](https://iguazio.atlassian.net/browse/CEML-651)

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-26T09:44:20Z

charts/mlrun-ce/values.yaml

+    # External listener for direct NodePort access (not used by ingress TCP proxy).
+    # The ingress TCP proxy (kafka.ingress) routes to the internal 'client' listener (9092),
+    # while this NodePort listener (9094/30094) is for clients connecting directly to nodes.
+    - name: external


The listener name 'external' is inconsistent with the comment on line 762 stating it's 'not used by ingress TCP proxy'. The previous name 'internal' better reflected that this listener is used by the ingress proxy routing to internal services. Consider renaming to clarify its actual usage, or update the comment to better explain why it's named 'external' when the ingress proxy doesn't use it.

Copilot · 2026-02-26T09:44:20Z

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml

+{{- if hasKey $existingData $externalPort }}
+{{- $existingTarget := index $existingData $externalPort }}
+{{- if ne $existingTarget $kafkaService }}
+{{- fail (printf "kafka.ingress.externalPort %s is already mapped in ConfigMap %s/%s to '%s' for another service/namespace. The ingress-nginx tcp-services ConfigMap is shared cluster-wide, so a given external port can only be used by one TCP service across all namespaces. Either reuse the existing mapping or choose a different kafka.ingress.externalPort value." $externalPort $ingressNamespace $configMapName $existingTarget) }}


The error message contains multiple sentences that reduce readability during troubleshooting. Consider restructuring to have a clear primary error followed by actionable guidance, such as: 'Port %s already mapped to %s in %s/%s. Choose a different kafka.ingress.externalPort value or reuse the existing mapping.'

Suggested change

{{- fail (printf "kafka.ingress.externalPort %s is already mapped in ConfigMap %s/%s to '%s' for another service/namespace. The ingress-nginx tcp-services ConfigMap is shared cluster-wide, so a given external port can only be used by one TCP service across all namespaces. Either reuse the existing mapping or choose a different kafka.ingress.externalPort value." $externalPort $ingressNamespace $configMapName $existingTarget) }}

{{- fail (printf "Port %s already mapped to %s in %s/%s. Choose a different kafka.ingress.externalPort value or reuse the existing mapping." $externalPort $existingTarget $ingressNamespace $configMapName) }}

shay79il requested review from GiladShapira94, Copilot and tal-haim February 25, 2026 09:54

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch from f8d2240 to 4871920 Compare February 25, 2026 09:56

Copilot started reviewing on behalf of shay79il February 25, 2026 09:57 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

GiladShapira94 reviewed Feb 25, 2026

View reviewed changes

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml Outdated Show resolved Hide resolved

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch from 4871920 to 4bbe017 Compare February 25, 2026 11:02

shay79il requested a review from Copilot February 25, 2026 11:02

Copilot started reviewing on behalf of shay79il February 25, 2026 11:03 View session

shay79il requested a review from GiladShapira94 February 25, 2026 11:03

Copilot AI reviewed Feb 25, 2026

View reviewed changes

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch from 4bbe017 to e501fb5 Compare February 25, 2026 12:05

shay79il requested a review from Copilot February 25, 2026 12:05

Copilot started reviewing on behalf of shay79il February 25, 2026 12:06 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch 2 times, most recently from 66a3dba to da79a38 Compare February 25, 2026 12:30

shay79il requested a review from Copilot February 25, 2026 12:30

Copilot started reviewing on behalf of shay79il February 25, 2026 12:31 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

charts/mlrun-ce/values.yaml Outdated Show resolved Hide resolved

charts/mlrun-ce/templates/kafka/kafka-ingress-tcp.yaml Outdated Show resolved Hide resolved

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch from 188c76c to 8d90bf6 Compare February 25, 2026 12:43

shay79il requested a review from Copilot February 25, 2026 12:50

Copilot started reviewing on behalf of shay79il February 25, 2026 12:51 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

charts/mlrun-ce/values.yaml Outdated Show resolved Hide resolved

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch 3 times, most recently from ae95a3c to dcd41e0 Compare February 25, 2026 13:32

shay79il force-pushed the CEML-651-ce-kafka-connectivity-issue branch from dcd41e0 to 9a42ca2 Compare February 26, 2026 09:35

shay79il requested a review from Copilot February 26, 2026 09:35

Copilot started reviewing on behalf of shay79il February 26, 2026 09:36 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

GiladShapira94 closed this Feb 26, 2026

Conversation

shay79il commented Feb 25, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shay79il commented Feb 25, 2026 •

edited by atlassian bot

Loading