Episode 46: Workshopping Live with Dustin Rea

In this conversation, Dustin Rea discusses the challenges faced by one of his clients who has a CRM. The CRM is built on a legacy codebase with issues in quality, scalability, and deliverability. The team has been working on improving the system, but there are still problems with emails, SMS, payments, and automations. The company is mission-driven and relies on the CRM to run their business, so rebuilding customer trust is crucial. They have made some improvements in infrastructure and email deliverability, but there is still work to be done. The conversation covered several topics related to database architecture and system design. The main themes include database optimization, handling heavy reads and writes, improving error handling and incident response, and enhancing observability. The speakers discussed the need for a purpose-driven database, implementing caching with Redis, tracking heavy queries, and addressing inconsistencies in the email service. They also mentioned the importance of logging and monitoring tools like Sentry and Grafana. Overall, the conversation highlighted the challenges and potential solutions for improving the performance and reliability of the system. Takeaways The client's CRM system is built on a legacy codebase and has issues with quality, scalability, and deliverability. The team has been working on improving the system, but there are still problems with emails, SMS, payments, and automations. Rebuilding customer trust is crucial for the company, as the CRM is core to their business. Improvements have been made in infrastructure and email deliverability, but there is still work to be done. Optimizing database performance is crucial for handling heavy reads and writes. Implementing caching with Redis can help improve performance and reduce load on the database. Tracking and optimizing heavy queries is important for identifying and resolving performance issues. Improving error handling and incident response processes can help address inconsistencies and ensure reliable system operation. Enhancing observability through logging and monitoring tools can provide valuable insights for troubleshooting and improving system performance. Chapters 00:00 Introduction and Background 01:12 The Current State of the White Label CRM 06:11 Goals and Expectations for the Project 07:54 User Trust and Importance of the CRM 10:44 Focus on Email and SMS Issues 13:30 Importance of User Support and Community 16:36 Email and SMS Deliverability Challenges 19:46 Database Structure and Performance Challenges 20:13 Improving Write Efficiency 21:02 Using SQS and SNS for Message Queuing 22:03 Controlling Flow with Multiple Readers 23:05 Writing to Separate Databases 24:01 Tracking Heavy Queries 25:37 Separating Permissioning and Heavy Reads 26:08 Caching with Redis 27:05 Improving Email Service Reliability 29:02 Monitoring and Alerting 30:28 Improving Data Logging and Validation 32:15 Implementing a Defined Process for Email Service 33:09 Recording and Rebuilding Failed Email Requests 35:11 Setting Up Aggregated Logs and Monitoring