https://www.jesse-anderson.com/2022/04/the-reasons-for-data-mesh-on-pulsar/
Data mesh is quickly becoming a way for companies to roll out their data strategy. If you haven’t already learned about data mesh, I suggest doing so. It comes with organizational and technical changes. I think a crucial part of your data mesh revolves around the choice of publish/subscribe technologies.
At the crux of data mesh is a desire for flexibility. This flexibility extends from the creation of data products (publish) all the way to the consumption of data products (subscribe). Greater flexibility in data mesh means that teams creating and consuming data products have even less coordination between themselves.
To help us look at the pub/sub needs for data teams, we’ll compare Apache Pulsar and Apache Kafka.
There isn’t a vast difference between publishing data in Pulsar or Kafka from a cursory look. They both have similar publish APIs. However, Kafka lacks a built-in schema registry, whereas Pulsar has a built-in schema registry that prevents incorrect schema usage. Data mesh calls for discovery and schema services. However, I still believe these should be built into the technology itself and not be added on.
Another significant difference between Kafka and Pulsar that isn’t as well known is that once data is published into a Kafka topic, it can’t be consumed in any other way than through reading that partition. This manifests when teams have to choose between wanting an ordered topic or random ordering, such as a round-robin layout. Before a team can answer this question, the team needs to know how the data will be consumed. In a data mesh scenario, the team producing the data product may be able to talk to the team or predict all the ways their data will be consumed in current or future use cases. In these situations, the team would be forced to publish their data product twice, once into an ordered topic and again into a random topic. The Kafka solution adds complexity in both the operations, by doubling the data and load, and programmatically by having to know which topic is ordered or random.
With Pulsar, this choice is dramatically different. Pulsar allows for different methods of consumption that can be chosen by the consumer. In this case, the producer can produce in an ordered manner. The data product consumer can choose from several ways of maintaining order or randomized data. Pulsar is flexible enough to let the downstream consuming team choose for themselves.
A common use case for pub/sub is messaging. With messaging, consumption requires the ability to individually acknowledge that a message was consumed and processed. Kafka’s messaging support entails workarounds whereas Pulsar supports this natively. Once again, Kafka requires more coordination and operational considerations. Pulsar allows for the consumer of the data product to make the choice.
I always recommend using managed services whenever you are running on the cloud. There are two main managed services for Pulsar on the major cloud providers. StreamNative has StreamNative Cloud and DataStax has Astra Streaming.
I think a critical feature of the technologies for data mesh is flexibility. We don’t want to have to complicate our technical landscape with the various limitations of our core technologies. One of those core technologies is our choice of pub/sub. Data teams won’t know ahead of time all of the ways that the data products will be consumed. We can clearly see that Pulsar will make it far easier for data mesh while Kafka will complicate it. When doing data mesh, I highly recommend Pulsar as the pub/sub of choice.
In 2021 I had the pleasure to first get to know and speak with Zhamak Dheghani, Director of Emerging Technologies at ThoughtWorks, in season one of the Data Dream Team series. Zhamak is a software engineer and architect who is (in)famously known as the founder of the data mesh concept, a paradigm shift in how we manage data-driven value at scale.
I interviewed Zhamak last season as more of an introduction to Data Mesh. I was happy to be able to welcome Zhamak back to The Soda Podcast, following our first conversation in 2021 that I would now categorize as an introduction to data mesh. This episode is different to others in the Data Dream Team series. When Zhamak and I talked about this episode, it was clear that she didn’t want to have just another conversation introducing data mesh. We also had the good fortune that the universe was aligned with our desire to delve into data mesh, with her freshly published book, Data Mesh: Delivering Data-Driven Value at Scale, giving us a great base to have an in-depth discussion and not just another interview.
Our discussion happens over two parts (part 1 and part 2) – we tallied over two hours of good conversation and so thought it best to break it up.