DevOps Solution for Big Data Platform Over AWS

The client had two major challenges:
1) To implement a continuous pipeline for Kafka service deployment, and
2) to have real-time data processing.
After the consulting phase, they chose Nagarro’s solution to leverage AWS cloud and Kops to provision Kubernetes cluster automatically and to auto-deploy all the Big Data services.

We automated the Kubernetes cluster setup with Kops and deployed Kafka services through many providers. We created a data pipeline to auto-deploy UI layers & backend components. We developed CI CD pipelines to auto setup monitoring tools on infrastructure, and automatically configure business metrics. We built workflow automation and scheduling by Apache Airflow and created automation programs to encrypt Kubernetes secrets.

Automation of the data pipeline reduced data processing time from 1 day to just a few minutes. This helped the client to perform real-time data analytics and BI reporting. CI CD pipelines for microservice deployments achieved faster delivery and reduced 100% efforts of manual deployment. Automation to encrypt Kubernetes Secrets brings security as service. The same security was implemented in more than 10 accounts.