- Multi-Cluster, Multi-Schema, Multi-Connect.
- Multi-Tenancy, RBAC, all the Authz - LDAP, SAML, etc.
- Very fast streaming multi-topic search (~1M messages / minute)
- Built in JQ search filters (we implemented a subset of JQ in the browser and JVM)
- Live Kafka Streams / Consumer Group topology visualisation (including RocksDB metrics for state stores, etc)
- Offset management, resetting, scheduled mutations.
- Consumer Group read and Topic write freshness / n-minutes inactivity metrics.
- Heaps of admin functions like Staged Mutations, Temporary RBAC policies, Data Masking for PII, etc.
- Prometheus / OpenMetrics endpoints.
- kREPL (slightly mad programmatic interface to Kafka)
- 7k commits, 100k docker pulls, 97.4% Clojure.
- Unlimited users, unlimited features.
- Commercially supported, annual licenses or available as hourly-metered (0.16c/hr) on the AWS Marketplace or Red Hat Marketplace.
https://kpow.io/how-to/manage-kafka-visibility-with-multi-te...
kPow gives you the ability to restrict visibility of Kafka resources down to a group or topic level by user-role, then builds completely internally consistent synthetic cluster views for each tenant.
We have a number of large enterprise customers who requested multi-tenancy support, it sounded tough at first but due to how kPow is built (we compute our own metrics / no JMX) and the fact that we use Clojure in both the front/back end it turned out to be fairly straight forward to implement.
Control centre eats its own dog food. All its data is on topics. That means if your cluster is unhealthy you can see healthy stats until the control centre consumers have caught up. Which means if you have a lot of data and load, control centre can start lagging and in one case I had some very hard debugging wondering why control centre saying the cluster was unhealthy but the cluster was up. It turned out control centre was lagging by 24 hrs so I was seeing the previous days outage.
Confluent recommended putting control centre on its own cluster for this very reason. One that isn’t cheap, a minimal prod cluster is 3 brokers+zookeepers and secondly what then monitors your monitoring cluster.
First - kPow also stores all telemetry in internal topics (and also keeps an audit-log in another topic for data governance). Setting up multi-cluster is fairly easy and kPow stores data in the first configured cluster - so like Confluent's suggestion you could have a dedicated kPow cluster. We really only recommend this if/when you get up to a large number of connected Kafka resources (Cluster, Schema, or Connect) and/or you have cluster/s that are very busy with other work.
Second - from v85 (next week) our v3 compute engine make kPow monitoring functionally independent of topic load / performance of the underlying cluster. We do still persist to internal topics for recovery, etc, but live stats and control are only dependent on your Kafka cluster responding to AdminClient requests (a handful every minute).
If you would like a trial license or guided tour, just say.
Another question: how might this compare to the Confluent web GUI? Obviously this is open source which is great.
The frontend is a React app in Typescript.
Currently we're using Kowl, and we're looking into Cruise Control.
One limiting factor is that many of these tools doesn't support authentication and RBAC, Kowl does, and that has been a requirement for at least some of our projects, where not everyone is allowed to view certain topics.
Disclaimer: I'm one of the Kowl authors. I do not intend to hijack that post for Kowl content though!
Links for the lazy. The GP page (https://lenses.io) certainly has a lot of good examples.
https://en.wikipedia.org/wiki/Corporate_Memphis
https://t-artmagazine.com/what-is-corporate-memphis-and-why-...
Kafdrop is great, especially if you're already running everything else in Docker, just add a few lines to your docker-compose and get a free web UI for troubleshooting/introspecting.
I like the multi-cluster management feature of this project. Unfortunately I don't know react (angular + typescript is ok with me), so I'm wondering if I should try this one knowing that I cannot modify it. The reasons I modified kafdrop are also the features I like to see in a new Kafka UI.
Positive:
- Multi-cluster support is a great feature! That's the main thing missing from Kowl, in my opinion
- The UI feels clean
- Message streaming feels smooth, and the search feature is a great addition. Also love the ability to see a message for a given partition/offset, and that JSON messages are nicely rendered as JSON!
- The CleanupPolicy logic doesn't handle "delete,compact" (I see bug number 829 addressed a slightly related issue, maybe that logic could be expanded [1])
- When viewing Cluster -> Topic -> Consumers, it seems like far more is being loaded than just consumer groups for that particular topic. It takes much longer to load than, e.g., Kowl does for pulling the same information
- Similar feedback for when clicking on an individual consumer group (example URL path: "/ui/clusters/dev/consumer-groups/my-group"). It takes a very long time to load
- If any permissions issues are encountered while loading the individual consumer group, the entire request fails. That wouldn't necessarily be an issue, but it is when you're loading all consumer groups, and not just the one requested
- I don't see any docs on how to access a topic that is secured with certificates (Kowl's relevant config [2])
- If a schema registry uses a self-signed certificate (or one signed by a corporation's CA), there's no way to pass the certificate through a config or skip validation (Kowl's relevant config [3]). As it is, you get a 500 error in the API call, but the UI doesn't show an error
1. https://github.com/provectus/kafka-ui/blob/master/kafka-ui-a...
2. https://github.com/cloudhut/kowl/blob/10776b81e45abfa716548d...
3. https://github.com/cloudhut/kowl/blob/10776b81e45abfa716548d...
Hi, db3pt0
Thanks for detailed feedback!
- The CleanupPolicy
fixed issue 925 (https://github.com/provectus/kafka-ui/issues/925)
- When viewing Cluster -> Topic -> Consumers, it seems like far more is being loaded than just consumer groups for that particular topic.
It takes much longer to load than, e.g., Kowl does for pulling the same information
There is only one way to filter consumer groups by topic, get all consumer groups, enrich it with current members, committed offsets, and then filter them by topic.
If you have a lot of consumer groups, this might take time to do it. We are thinking on this issue, and will try to improve it in the next versions.
(https://github.com/provectus/kafka-ui/issues/927)
- Similar feedback for when clicking on an individual consumer group (example URL path: "/ui/clusters/dev/consumer-groups/my-group"). It takes a very long time to load
This looks rather strange. For single consumer we are getting consumer group description and then enrich it with topic info (end offsets). This works fast enough.
- If any permissions issues are encountered while loading the individual consumer group, the entire request fails. That wouldn't necessarily be an issue, but it is when you're loading all consumer groups, and not just the one requested
Thx for describing this. We created issue for this. (https://github.com/provectus/kafka-ui/issues/928)
- I don't see any docs on how to access a topic that is secured with certificates (Kowl's relevant config [2])
You have to pass ssl config stores to kafka client (we'll add docs. in the next versions https://github.com/provectus/kafka-ui/issues/929)
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION = /var/private/ssl/kafka.server.keystore.jks
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD = test1234
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEY_PASSWORD = test1234
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION = /var/private/ssl/kafka.server.truststore.jks
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD = test1234
- If a schema registry uses a self-signed certificate (or one signed by a corporation's CA), there's no way to pass the certificate through a config or skip validation (Kowl's relevant config [3]). As it is, you get a 500 error in the API call, but the UI doesn't show an error
Same as above (https://github.com/provectus/kafka-ui/issues/930)
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION = /var/private/ssl/kafka.server.truststore.jks
KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_PASSWORD = test1234We’ve now rolled it out to our 400+ engineers who use it every day to help them gain visibility and insight to their data and topics, manage offsets, amount other things. It handles HUGE clusters with no issue, and the RBAC features are great.
I’ve tried them all. While there are some ok alternatives, I honestly find them toys compared to kPow. Do yourself a favor and check it out!