Built on Pulsar: A Commercial Consent Management System for 80 Million Citizens

Pulsar Virtual Summit Europe 2021
A Commercial Consent
Management System for 80
Million Citizens
Shivam Arora
Solution Architect
Çetin Yalçın Güleç
Solution Architect

Softtech
Today and Tomorrow of the Use-Case.
Our approach and System Design.
Challenges and Solutions.
Other Use-Cases on Pulsar.
Agenda

is a leading technology company with growing global footprint.
Core clients
>60+ 10+
Locations
1,500+
Employees
~65 m $
In total revenues
San Francisco China
Softtech by numbers…
915
Product
Development
165
Subject Matter Experts &
Business Architects
140
Professional
Services
44
Operation
Support
San Francisco Maxitech
TUTOM R&D Center
Kadıköy Garage
ITU Technopark
İş Kule
Kolektif House - Innovation Office
Frankfurt, Maxi Digital GMBH
Softtech Shanghai
Istanbul
Ankara
Frankfurt
Hacettepe Technopark
ODTÜ Kuzey Kıbrıs Kampüsü
Eskişehir Technical University
Eskişehir
Banking Finance Capital Markets Retail Health & Medical Insurance Agriculture Logistics Mobile
Industries

Softtech serves next generation digital financial institutions
internationally
Innovation
Talent
Delivery Hub
Professional
Services
Products
Mobile
Banking
Quick – Omni Channel Platform
Internet
Banking
Branches Mobile Virtual POS
Digital Onboarding
Data
Front End and Channel
Verticals
Retail
Banking
Commercial
SME Banking
Treasury
Capital Markets
Card payment system
Plateau (Cloud Ready Infrastructure)
Big Data
Analytics
Platform
‘High skill team
of professionals
delivering
products and
services,
building
capabilities for
clients’
Point of Business
1 Digital channels with mobile first customer journeys (Productizing
Isbank Georgia and Isbank AG)
1
2
Multi channel management system to deploy capabilities
seamlessly across channels (Quick)
2
3 4
Creating an ecosystem of partners to leveraging innovation hub of
Maxitech and Softtech China (i.e. Digital onboarding with
biometrics)
3
Solutions from internal innovation (Virtual POS, Point of Business
- POB)
4
Next Generation data platform deploying big data analytics
architecture
5
Digital core banking solutions for retail and commercial customers
(i.e. Productizing KGF solution)
6
Loyalty Programs
Core Banking 5
Integration & API Platform
6
Vertical solutions that are productized addressing specific
capabilities for financial institutions (i.e. Treasury, capital markets,
cards, loyalty programs)
Plateau is a cloud ready microservices based development
framework enabling XaaS solutions
Building blocks of our proposition
BPA
ESM
RPA
Data Science &
AI
Cloud
Consultancy
UX

Today and Tomorrow of the use case.

Today and tomorrow of the use case
o Create a database which provides the
management of 80M+ receivers’ consents
for commercial electronic messages from
100K+ companies
o All 100K+ service provides will register
receivers’ electronic consents in a scalable
system (1M write & 5M read in a minute)
o Service Providers are responsible to keep
the database up to date
o Intermediary service providers have to chech
the DB (read only) if receiver has authorized
the provider to send commercial information
(SMS || Email)
o Which requires very fast read executions in
order not to block LOBs
o Intermediary service providers will read the
information from the microservices located in
their datacenters
o All the personal information must be
encrypted and stored

80 M * 100K Consent database
The intermediary service provider
The firm which is obligated to screen the
status of the consent before allowing the
message be directed to the recipient.
Solution Partners
Solution partners provide integration services for
their customers to reach and consume services.
Service Provider
All individuals and legal entities sending
commercial electronic messages.
The responsibility of the service provider is to
ensure compliance with the legislation.
Process must be easy and secure management
of consents.
Public
There should be a more efficient and
effective complaint management system.
Recipient
It is either consumer or a an individual on
behalf of a business.
Recipients manage all of their consents via
an account and easily change the status of
their consents.

Business Requirements
Information will be distributed to
Datacenters Records in 60 seconds Records in 60 seconds
5 1M 5M
The system must be
available to encrypt and write
The system must be available
to read and decrypt

Our approach and system design.

High Level Requirements
o Highly consistent, reliable and scalable architecture ( Hybrid )
o High throughput with low read latency.
o Multi datacenter API deployment.
o Cloud first architecture.
o Extensible architecture from business point of view.
o Real-Time dashboards to understand the health of the business.
o Emphasis on open-source (and commercial open-source software) as much as possible

Our architecture in Nutshell
o Highly scalable for sync and
async request
o Microservices
o Event driven architecture
o Autonomous software agents
o End–To–End monitoring

Why Pulsar ?
o Cloud first approach. Deployment(Helm) charts are production ready*
o Pulsar provides both queueing and log semantics.
o Selective acknowledgments and redelivery of individual messages.
o Multi-tier architecture where each layer can be scaled as needed.
o Segment-Centric Storage.
o Smooth cluster expansion, and node failure recovery.
* We required to change few properties on production.

Pulsar Usage In Our System
o Pulsar cluster is deployed on Kubernetes across multiple data center (Active-Active)
o Both Shared and Exclusive subscription are being used.
o Our system consume/produce JSON messages.
o Pulsar metrics are being shipped to central Prometheus server.
o Currently our production Pulsar cluster is running with following configuration –
3 brokers and zookeeper node
6 bookie nodes
285 Topics
3000+ Producers
50+ Subscription

Typical API request flow

Allows us to focus on business logic
Sets up producer/consumer, thus eliminating boilerplate code
Each agent can be scaled independently
Highly maintainable and testable
Dependent on each other yet run independently.
Benefits of Atomic Agents

Challenges and Solutions
Problem Solution/Outcomes
No message backlog but topic size was increasing to alarming rate.
Since message acknowledgement is best effort by default in
Pulsar. Thus under heavy load its possible to see spike in topic
storage size. To avoid this -
• Increase JVM size of broker nodes
• Unload topic and handle message duplication on
consumer side
• Use negative acknowledgement instead of ack timeout.
Kubernetes worker node crash
In Initial stage of production we lost all kubernetes worker node
and total message backlog was around 500M. After system
came back, one bookie did not came up but no message loss
was observed.
(Bookie Ledger Configuration E=3, Qw=2, Qa=2)
Instant traffic burst on system causes DB bottleneck
We stored all async API request messages/events on topics by
increasing bookie nodes from 3 to 12 each having 1TB
storage, once DB got stabilized we processed events and
decommissioned extra bookie nodes.
Ad-Hoc requirement for reporting (To be implemented)

Stats
Stat #4
peak time Unique visitor registered from
UI so far.
Unique emails are stored.
Stat #5 Stat #6
10M 2M 180M
Stat #1
TRX are being processed
daily
Consents have been
registered so far
Unique phone numbers are
stored.
Stat #2 Stat #3
450M 6.5B 115M

Technology Stack (current)

Other use cases on Apache Pulsar

#1 Event Hub For Turkey’s Largest Bank
Features
o Provide client (Java, .Net ve Python) and rest service for
applications. Removing boiler plate code for connected
application
o Whole system will be run on OpenShift cluster.
o Event routing using Pulsar function.
o Management (Topic Access control, topic management
etc.) and monitoring of whole system.
o Provide pull service for consumer application.
o Replicate mainframe transaction to open system for
existing application.
o Kafka protocol on Pulsar.

#2 Design Comlex Event Processing Engine around Pulsar
Features
o Event routing and simple data transformation
using Pulsar function.
o Delayed Messaging or scheduled messaging
o Scalability and reliability
o Multiple subscription type
o Message replay using consumer seek.

Thank You

Built on Pulsar: A Commercial Consent Management System for 80 Million Citizens

More Related Content

What's hot (20)

Similar to Built on Pulsar: A Commercial Consent Management System for 80 Million Citizens (20)

More from StreamNative (20)

Recently uploaded (20)

Built on Pulsar: A Commercial Consent Management System for 80 Million Citizens