Episode 104: An Infrastructure Rewrite Story with Guest Paul Stack

Guest Paul Stack shares his experiences in transforming a massive logging infrastructure from a monolithic architecture to a microservices-based system. He discusses the challenges faced during the transition, the implementation of Kafka for log management, and the importance of data retention and cost management. The conversation highlights the evolution of logging practices and the empowerment of teams through decentralization, ultimately leading to a more efficient and effective infrastructure. Show Notes: Paul's LinkedIn: https://www.linkedin.com/in/stack72/ Paul's Github: https://github.com/stack72 Chapters 00:00 Introduction to Logging Infrastructure Challenges 02:58 Transitioning from Monolith to Microservices 06:09 Building a Robust Logging System 08:58 Overcoming Data Overload with Kafka 11:52 Scaling and Resilience in Logging Infrastructure 15:02 Cost Management and Optimization Strategies 17:34 Breaking Down the Monolith 20:07 Empowering Teams Through Decentralization 23:00 The Evolution of Logging and Monitoring 28:02 Navigating the Shift in IT Infrastructure 31:06 Lessons Learned from the Past