Declarative deployment of Apache Kafka on Kubernetes with Calisti

If you are a frequent reader of our blog, or if you’ve been using the open source Koperator, you might already be familiar with Calisti, our product that delivers Apache Kafka as a service on Kubernetes.

Check out Calisti in action on your own clusters:

Register for an evaluation version and run a simple install command!

As you might know, Cisco has recently acquired Banzai Cloud. Currently we are in a transitional period and are moving our infrastructure. Contact us so we can discuss your needs and requirements, and organize a live demo.

Evaluation downloads are temporarily suspended. Contact us to discuss your needs and requirements, and organize a live demo.

smm install -a --install-sdm --no-demo-cluster --kubeconfig <path-to-k8s-cluster-kubeconfig-file>

or read the documentation for details.

Take a look at some of the Kafka features that we've automated and simplified through Calisti and the Koperator, through the Calisti Blog

The first Calisti version was merely a CLI tool which deployed these components (see the list below), but we knew that merely deploying these components in isolation would not be enough, as they have to be both configured properly, and work together as desired. Also, certain components should be deployed in a specific order for everything to work as expected. The orchestration of the deployment and configuration of these components is a process that lends itself to error when done manually, especially if there are many components.

tl;dr:

Calisti is now a CLI, a control plane (including a Calisti operator) and a backend. We added support to Calisti for describing the desired state of the whole Kafka on Kubernetes ecosystem, including describing all the components and their configuration in a declarative manner. The Calisti operator takes care of the rest. We have also opensourced a framework which eases writing declarative installers.

Orchestrating harmony

Calisti' feature set is fairly large - you can read more by checking Calisti's blog - as well the number of components employed under the hood:

• istio-operator to manage an Istio service mesh for Kafka and related components to take advantage of the benefits provided by Istio.
• zookeeper-operator to manage a Zookeeper cluster for Kafka clusters
• prometheus-operator to manage Prometheus and Grafana for monitoring
• Koperator to manage Kafka clusters
• Calisti backend to provide functionalities like Kubernetes RBAC integration with Kafka ACL, disaster recovery using volume snapshots and MirrorMaker 2, management of monitoring artifacts (service monitors, pod monitors, Grafana dashboards), etc
• kafka-minion to provide Kafka consumer monitoring

The Calisti CLI makes it easy for the user to set up the entire Calisti eco-system by setting up and configuring all of the above-listed components. The user has only to execute one single command:

smm install -a --install-sdm

The CLI offers a few way of helping you get started and to begin experimenting with Calisti. While the CLI commands are easy to use, they do not offer the possibility of fine-tuning each of your components. They also might not be a good fit for automation or CD tools.

Day Zero – Setup Calisti with CLI imperative commands

Even when a user has little or very little knowledge of Calisti, it should be easy for them to spin up the Kafka on Kubernetes system with all its features enabled, so that the user can start experimenting with it.

After installing the CLI, and running the

smm install -a --install-sdm

command, a Calisti cluster is deployed with all the features enabled and a Kafka demo cluster. Behind the scenes, the default settings used to deploy the components are collected into a custom resource, called ApplicationManifest, and the reconcile flow deploys and configures the components according to these settings. The advantage of storing these settings in a custom resource is that we can later track what components were deployed/removed and with what settings. Also, it enables running the reconcile flow in operator mode which is useful for any unattended deployment/removal/reconfiguration of the components used by Calisti (e.g. automation and CD tools).

Day One - CLI declarative mode

Now let's say the user is already familiar with Calisti and wants to fine-tune the configuration of various components. The user provides the configuration settings for each service declaratively in a file and passes it to the CLI.

calisti reconcile --from-file <path-to-file>

In this scenario, the ApplicationManifest is read from a file, allowing users to declaratively provide custom configuration for the various components involved.

Day Two - Operator mode

Here, let's assume that instead of the reconcile flow being executed on the user's machine, it runs on the Kubernetes cluster as an operator that watches the ApplicationManifest custom resource. Any changes made to the watched custom resource triggers the same reconcile flow as in the previously described modes. For the operator mode deploy the Calisti Control Plane into the Kubernetes cluster with Helm:

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/

helm install banzaicloud-stable/calisti-control-plane

ApplicationManifest custom resource

The ApplicationManifest custom resource holds the desired component deployment and configuration settings.

apiVersion: Calisti.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: Calisti-apps
spec:
  istioOperator: # Istio operator and Istio mesh related settings.
    enabled: true # Whether to deploy or remove Istio operator component. Defaults to true.

    # Whether to create an Istio mesh. Defaults to true. For an Istio mesh with
    # custom settings set this to false and create a https://github.com/banzaicloud/istio-operator/blob/release-1.5/config/samples/istio_v1beta1_istio.yaml custom resource with your settings.
    createDefaultMesh: true
    namespace: istio-system # The namespace to deploy Istio operator into. Defaults to istio-system.
    # Settings override in YAML format. For the list of overrideable settings see https://github.com/banzaicloud/istio-operator/blob/release-1.5/deploy/charts/istio-operator/values.yaml
    valuesOverride:
  kafkaOperator: # Koperator related settings.
    enabled: true # Whether to deploy or remove Koperator component. Defaults to true.
    namespace: kafka # The namespace to deploy Koperator into. Defaults to kafka.
    valuesOverride:
  Calisti: # Calisti backend related settings.
    enabled: true # Whether to deploy or remove Calisti backend component. Defaults to true.
    namespace: Calisti-system # The namespace to deploy Calisti backend into. Defaults to Calisti-system.
    # Settings override in YAML format. For the list of overrideable settings see https://banzaicloud.com/docs/Calisti/overview/
    valuesOverride:
  monitoring: # Monitoring related settings
    grafanaDashboards: # Grafana dashboards related settings
      enabled: true # Whether to deploy ConfigMaps with Grafana dashboards for the components.
      label: # The label to apply to the Grafana dashboard ConfigMaps. It defaults to "app.kubernetes.io/Calisti_managed_grafana_dashboard"
    prometheusOperator: # Prometheus operator related settings
      enabled: true # Whether to deploy or remove Prometheus operator component. Defaults to true.
      namespace: Calisti-system # The namespace to deploy Prometheus operator into. Defaults to Calisti-system.
      # Settings override in YAML format. For the list of overrideable settings see Prometheus operator Helm chart version 8.11.2
      valuesOverride:
  kafkaMinion: # Kafka Minion related settings
    enabled: true # Whether to deploy Kafka Minion for all Kafka clusters.
    # Settings override in YAML format. For the list of overrideable settings see Kafka Minion Helm chart at https://github.com/banzaicloud/kafka-minion-helm-chart
    valuesOverride:
  zookeeperOperator: # Zookeeper operator and Zookeeper cluster related settings
    enabled: true # Whether to deploy or remove Zookeeper operator component. Defaults to true.
    # Whether to create a default 3 ensemble Zookeeper cluster. Defaults to true. For a Zookeeper cluster with custom settings set this to false and create a https://github.com/pravega/zookeeper-operator/tree/v0.2.6#deploy-a-sample-zookeeper-cluster custom resource with your settings.
    createDefaultCluster: true
    namespace: zookeeper # The namespace to deploy Zookeeper operator into. Defaults to zookeeper.
    # Settings override in YAML format. For the list of overrideable settings see https://github.com/pravega/zookeeper-operator/blob/v0.2.6/charts/zookeeper-operator/values.yaml
    valuesOverride:

The settings applied to the components are the result of merging default settings + valuesOverride + managed settings.

The managed settings are settings that cannot be changed by users, since inadequately setting them may lead to malfunction. Let’s take Kafka Minion as an example, which is used to gather consumer-related Kafka metrics.

kafkaMinion:
  valuesOverride: |-
    resources: 
      limits: 
        memory: 256Mi 
    kafka: 
      brokers: kafka-1:9092,kafka-2:9092,kafka-3:9092

The value for kafka.brokers specified in valuesOverride has no effect as that is computed dynamically by Calisti and automatically updated when the list of brokers is changed due to an upscale or downscale.

You might be wondering why we allow removing the Koperator and calisti backend core components as without these most functionalities are rendered useless. The reason is that we wanted to continue to be able to handle rare and unforeseen events that might require temporarily removing the core components.

The ApplicationManifest custom resource is the owner of these deployed components, thus, if removed, the Kubernetes Garbage Collector will remove all deployed components.

Try it out

Let's try out the operator mode through an example.

Install the Calisti control plane

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/

helm install banzaicloud-stable/calisti-control-plane

Deploy ApplicationManifest custom resource

Deploy an ApplicationManifest that enables all components:

kubectl apply -f-<<EOF
apiVersion: Calisti.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: applicationmanifest-sample
spec:
  istioOperator:
    enabled: true
    createDefaultMesh: true
    namespace: istio-system
  kafkaOperator:
    enabled: true
    namespace: kafka
  Calisti:
    enabled: true
    namespace: Calisti-system
  monitoring:
    grafanaDashboards:
      enabled: true
    prometheusOperator:
      enabled: true
      namespace: Calisti-system
  kafkaMinion:
    enabled: true
  zookeeperOperator:
    enabled: true
    createDefaultCluster: true
    namespace: zookeeper

Check the status

The status of the ApplicationManifest provides information on the progress and status of the deployment of the enabled components, as well as the overall status of the reconcile flow:

---
status:
  components:
    istioOperator:
      meshStatus: Available
      status: Reconciling
  status: Reconciling

---
status:
  components:
    istioOperator:
      meshStatus: Available
      status: Available
    kafkaOperator:
      status: Available
    monitoring:
      status: Available
    calisti:
      status: Available
    zookeeperOperator:
      clusterStatus: Available
      status: Available
  status: Succeeded

Update the settings of one of the components

Let's set a new password for Grafana:

kubectl apply -f-<<EOF
apiVersion: calisti.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: applicationmanifest-sample
spec:
  istioOperator:
    enabled: true
    createDefaultMesh: true
    namespace: istio-system
  kafkaOperator:
    enabled: true
    namespace: kafka
  calisti:
    enabled: true
    namespace: calisti-system
  monitoring:
    grafanaDashboards:
      enabled: true
    prometheusOperator:
      enabled: true
      namespace: calisti-system
      valuesOverride: |-
        grafana:
          adminPassword: my-new-password
  kafkaMinion:
    enabled: true
  zookeeperOperator:
    enabled: true
    createDefaultCluster: true
    namespace: zookeeper
EOF

In the status section we can see that the status for monitoring has changed to Reconciling.

---
status:
  components:
    istioOperator:
      meshStatus: Available
      status: Available
    kafkaOperator:
      status: Available
    monitoring:
      status: Reconciling
    calisti:
      status: Available
    zookeeperOperator:
      clusterStatus: Available
      status: Available
  status: Reconciling

After the new configuration was successfully applied, the status changes to Available

---
status:
  components:
    istioOperator:
      meshStatus: Available
      status: Available
    kafkaOperator:
      status: Available
    monitoring:
      status: Available
    calisti:
      status: Available
    zookeeperOperator:
      clusterStatus: Available
      status: Available
  status: Succeeded

About Calisti

Calisti is the software for setting up and operating production-ready Kafka clusters on Kubernetes, leveraging a Cloud-Native technology stack. Calisti includes Zookeeper, Kafka operator, Istio and many other components that are installed, configured, and managed to operate a production-ready Kafka cluster on Kubernetes.