A Consul Service Mesh Integration Case Study with Presto

This is a follow-up post to the presentation we held yesterday at Hashitalks 2020 with the message that we’re actually leveraging these technologies as part of a wider initiative to build a state-of-the-art data platform.

Take a look at: Hashitalks 2020 YouTube Stream

Take a look at: Slideshare from our presentation

Presto

Presto is a distributed SQL tool that has become a key element in our toolbox because of its flexibility and a growing number of plugins.

Consul Connect

Consul Connect is the service-mesh implementation provided by Hashicorp. At its simplest, it allows you to easily secure communication between all services in your ecosystem by using sidecar proxies that handle any inbound/outbound connections, while the services themselves will only talk to their own proxy through the loopback interface.

When dealing with opinionated services or applications (meaning that they will impose an architectural burden on you), the traditional blueprint can be severely strained, for example, because the coordinator will use proprietary methods to differentiate among its different workers/brokers, thus rendering the cluster unable to communicate internally through loopback.

There’s another option though called “Connect Native Integration”, which we ended up exploring and adopting when we found ourselves facing a similar challenge with Presto:

In many cases, the features offered by this kind of “opinionated” application, outweigh by far the architectural shortcomings; Kafka/Zookeper would be another good example of this.

So, native integration becomes a very viable option for keeping architectural integrity within your Control Plane while still leveraging such technologies as part of your ecosystem.

Connect-Native integrating Presto

Native integration is basically mutual TLS + an API Authorization call, so we decided to contribute some code to the Presto open-source distribution prestosql.io, in order to make the Certificate handling functionality pluggable as well.

So, there’s an open pull-request towards the Presto master (with a quite good outlook) and a ConsulConnect plugin (which does the API call) we have also developed and should be the next merge request.

The resulting architecture is something we really like because it is a clean and organized way to solve a very challenging problem; also, it works beautifully!!

Key Takeaways

All in all, we have learned a lot of lessons throughout this journey, so these are our 3 main conclusions:

• State-of-the-art should be “simple as that”
• Refactor to reduce, redesign to refine
• Fortune favors the bold

All of the latter speaks to the importance of simplicity and also the benefits of embracing a mindset that allows our tech initiatives to be iterated upon, improved, and re-shaped while striving to deliver top quality & value.

Thanks for reading!!