Faez Hasnan

Building a High-Volume Consumer Transactions System

Introduction

We developed a transaction processing system aimed at efficiently handling high volumes of consumer purchase data. The project utilizes Kotlin and Spring Boot as the backend technologies, while leveraging containerization with AWS ECS and Terraform for scalable infrastructure. The system is designed to apply business rules, calculate cashbacks, and maintain accounting ledgers, with a target throughput of around 100000 transactions per second.

Problem Statement

A leading e-commerce platform, faced a significant challenge in processing the ever-increasing influx of consumer purchase data. The existing system was implemented in PHP and delivered many years ago, and there is a recent appetite for maintaining accurate accounting ledgers. The need for a robust, high-throughput transaction processing system became paramount to ensure an accurate financial records.

Design and Architecture

The aforementioned challenges were addressed, by opting for a modern, containerized infrastructure using AWS ECS and Terraform. This approach provides the flexibility and scalability necessary to handle the projected transaction volumes.

The backend of the system was developed using Kotlin and Spring Boot, a powerful framework for building microservices. The combination of these technologies allowed for rapid development, easy integration with existing systems, and efficient resource utilization.

System Functionality

The transaction processing system was designed to handle the following key functionalities.

Data Ingestion

The system integrates with data sources, such as click captures and online payment gateways, to receive consumer purchase data in real-time. The data is ingested into the system through well-defined APIs, ensuring data integrity and ample security.

Business Rules and Cashback Calculation

The system applies complex business rules to determine cashback eligibility based on predefined criteria, such as product categories, customer segments, and promotional campaigns. Advanced algorithms are employed to accurately calculate cashbacks, taking into account variables such as transaction value and loyalty programs.

Accounting Ledgers

To maintain accurate financial records, the system updates accounting ledgers in real-time, ensuring consistency and transparency. By leveraging event-driven architecture and asynchronous processing, the system efficiently handles ledger updates while maintaining high performance.

Scalability and Performance

To achieve the desired throughput of 100000 transactions per second, the system utilizes horizontally scalable infrastructure. AWS ECS provides the ability to dynamically scale the number of containers based on workload demand. Load balancing and auto-scaling configurations ensure optimal resource allocation, reducing latency and bottlenecks during peak transaction periods.

Implementation and Deployment

The development process involved an iterative approach, employing agile methodologies to ensure flexibility and adaptability to changing requirements. Continuous integration and continuous deployment (CI/CD) pipelines facilitated automated testing and deployment of code changes.

Infrastructure provisioning was achieved using Terraform, an infrastructure-as-code tool, which allowed for version control and reproducibility. The use of containerization with Docker and orchestration with AWS ECS simplified the deployment process, enabling easy scalability and management of application containers.

Results and Benefits

The implementation of the transaction processing system yielded significant improvements in processing times, accuracy, and scalability. Key benefits include the following.

Increased Throughput

The system successfully achieved the target throughput of around 120000 transactions per second, ensuring efficient processing of high transaction volumes without compromising performance.

Enhanced Accuracy

The application of business rules and cashback calculation algorithms improved the accuracy of cashback distributions, resulting in higher customer satisfaction and merchant loyalty.

Real-time Financial Reporting

By maintaining accounting ledgers in real-time, the system provided up-to-date financial reports, enabling timely decision-making and accurate payouts.