> I'm curious to know where you've seen success with Kafka.
Much the same as you, as a replacement for brittle data pipelines where multiple services are generating data, and then a cron moves it across a network file system, or the apps are pushing to something like ES directly, which has caused data loss when their ES wasn't configured correctly, etc. etc.
The nice thing about Kafka is that it decouples those dependencies, which gives you room to scale the ingestion side, and also allows for failures without data loss.
The one caveat though, is that Kafka becomes the dependency, but in my years of using Kafka in anger (since 0.8), I've only ever encountered full cluster unavailability a few times.
Memorable ones:
* a sysop who decided that the FD limit for processes on the boxes running their self-managed Kafka should be a nice low number, which prevented Kafka from opening sufficient sockets
* a 3 node cluster, where to "prevent data loss", they were using acks=all and had configured replicas to be 3, and min insync replicas to be 3, and then a broker had gone down. Yeah, not great that.
* A 2.5 stretch cluster (brokers and ZKs in two AZs, tie-breaker ZK in a 3rd) suffered a gradual network partition when an AWS DC in Frankfurt overheated, and they ended up with two leaders for the same topic partition in each of their main AZs, that were still accepting writes as they could hit minISR within their own DCs. There was some other weirdness in how they interacted with the tie-breaker ZK. And when the network partition was ended, neither of them could be elected leader. Had to dump the records from both brokers from the time the partition began, roll the brokers to allow an unclean leader election, then roll them again to turn it off once the election had succeeded, and then the client got to sift through the data and figure out what to do with it - as they weren't relying on absolute ordering, it was reasonably trivial for them to just write it again IIRC.
So for absolutely essential do not lose data, if Kafka connectivity was lost, the app should make itself unready, and then fail over to writing any uncommitted data to disk for a side-car to upload to S3 or similar.
Kafka's great, but you have to code deliberately for it, which is where I saw the biggest mistakes occur - strapping Kafka to ActiveMQ with Camel and expecting Kafka to work the same as ActiveMQ...
Much the same as you, as a replacement for brittle data pipelines where multiple services are generating data, and then a cron moves it across a network file system, or the apps are pushing to something like ES directly, which has caused data loss when their ES wasn't configured correctly, etc. etc.
The nice thing about Kafka is that it decouples those dependencies, which gives you room to scale the ingestion side, and also allows for failures without data loss.
The one caveat though, is that Kafka becomes the dependency, but in my years of using Kafka in anger (since 0.8), I've only ever encountered full cluster unavailability a few times.
Memorable ones: * a sysop who decided that the FD limit for processes on the boxes running their self-managed Kafka should be a nice low number, which prevented Kafka from opening sufficient sockets
* a 3 node cluster, where to "prevent data loss", they were using acks=all and had configured replicas to be 3, and min insync replicas to be 3, and then a broker had gone down. Yeah, not great that.
* A 2.5 stretch cluster (brokers and ZKs in two AZs, tie-breaker ZK in a 3rd) suffered a gradual network partition when an AWS DC in Frankfurt overheated, and they ended up with two leaders for the same topic partition in each of their main AZs, that were still accepting writes as they could hit minISR within their own DCs. There was some other weirdness in how they interacted with the tie-breaker ZK. And when the network partition was ended, neither of them could be elected leader. Had to dump the records from both brokers from the time the partition began, roll the brokers to allow an unclean leader election, then roll them again to turn it off once the election had succeeded, and then the client got to sift through the data and figure out what to do with it - as they weren't relying on absolute ordering, it was reasonably trivial for them to just write it again IIRC.
So for absolutely essential do not lose data, if Kafka connectivity was lost, the app should make itself unready, and then fail over to writing any uncommitted data to disk for a side-car to upload to S3 or similar.
Kafka's great, but you have to code deliberately for it, which is where I saw the biggest mistakes occur - strapping Kafka to ActiveMQ with Camel and expecting Kafka to work the same as ActiveMQ...