Introduction Cost Efficiency and Portability are the main reason to migrate Apache Spark workloads from managed services like AWS EMR, Azure Databricks, or HDInsight to Kubernetes. You can learn more about the migration process from AWS EMR to K8s in the following article. However, there are also potential pitfalls with leaving…

Introduction Amazon EC2 provides a broad portfolio of compute instances, including many that are powered by the latest-generation Intel and AMD processors. AWS Graviton2 processors add even more choice. …

Introduction One of the 5 pillars of the Well-Architectured Framework is Cost Optimization. The Cost Optimization pillar focuses on avoiding unnecessary costs, selecting the most appropriate resource types, analyzing spend over time, scaling in/out in order to meet business needs without overspending. …

Introduction ESG research found that 43% of respondents considering cloud as their primary deployment for Apache Spark. And it makes a lot of sense because the cloud provides scalability, reliability, availability, and massive economies of scale. Another strong selling point of cloud deployment is a low barrier of entry in the…

Introduction COVID19 made a huge impact on the world. Every single aspect of our lives went through a massive change and just like many other software development teams in the world, our team went fully remotely. I have no idea when our office will be open, and probably, offices may never…

Introduction Monitoring and alerting is a mandatory part of any software system running in a production environment. To keep software systems healthy, to optimize performance and resource utilization, you need a unified operational view, real-time granular data, and historical reference. …

Introduction One of the well-known problems in parallel computational systems is data skewness. Usually, in Apache Spark, data skewness is caused by transformations that change data partitioning like join, groupBy, and orderBy. For example, joining on a key that is not evenly distributed across the cluster, causing some partitions to be…

“Define user behavior with Python code, and swarm your system with millions of simultaneous users” — locust.io Introduction Vertica is a very powerful analytic database designed to handle massive amounts of data. When configured right, Vertica enables a great query performance even in very intensive scenarios. Some Vertica users report that…