6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Open in app

Search

Trading System Design using MicroServices

8 min read · Oct 9, 2023

Bhavin Follow

Listen Share More

Photo by Nathan Dumlao on Unsplash

Views , Designs and Opinions are my own.

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 1/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

This blog should be consider for learning purposes only.

I started out with just backtesting using pandas data frame and calculating
percent change of returns based on the frequency I was testing and quickly
realized that this apporach will not work in reality. In the course of 1.5 years since
then I have learned a lot and evolved my trading systems, backtesting strategies and
my whole approach to systematic trading. With the help of this blog I would like to
share my experience and learning with someone starting out on the same journey.

We will start out with identifying components of a trading system. They varies based
on characteristics of strategies employed, execution speed, asset class traded etc. I
have listed components of my trading system based on the following characteristic:

Components:

Backtesting Engine:

This component has 2 subcomponents. one is the backtesting framework. You need
a backtesting framework which can keep track of every order that is generated for
your backtest. Track transaction costs. It helps to keep track of available funds as of
the backtesting date during a backtest run. Especially important if your strategy
considers shorting, because the stock might not be available to borrow in reality as
of the backtest date etc.

Framework:

which you can pick from one of the open source ones available. I have listed few
open source frameworks available in the appendix below. I have decided to use bt
because it lets you inject various modules in your backtest and this fits well with
tapping into various microservices as and when you need an output from one. It also

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 2/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

let you defined hierarchy of strategies so you can backtest variable allocation across
multiple strategies.

Pipeline:

I had to build a custom framework around the chosen backtesting framework to

execute various stages in my backtesting pipeline. This will more specifically
depend on the backtesting framework you have chosen but I would recommend
having a custom build out so it lets you test various pipelines ( combinations of
stages ).

Trading strategies:

No trading system is complete without one or more trading strategy. Trading

strategies can be as simple as SMA cross over to more complex like finding
cointegrated series or partial series between one vs one or one vs multiple stocks.

Ideally you would backtest each strategy individually and then different
combinations of 1 or more strategies . You would also want to allocate % of fund to
each strategy or combinations to strategies and this could also vary from one time
frame to another.

Risk Management Framework:

you might also want to setup risk management rulebook based on allocation. for eg:
I will not allocate more than 20% of account size to one strategy and within the
strategy not more then 5% will be allocated to on name. You can also expand this to
run market neutral allocation, sector allocation limits , limits based on market cap
of names, geographic area of business etc.

Optimizer:

Optimizers help you determine weightage for each name in your future portfolio
based on risk management framework and performance requirement to meet your
portfolio goals.

Optimizers also acts as filters in a way that at the end of the optimization chain few
names with allocation below your threshold will drop off. ( you will of course
rebalance after they drop off ).

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 3/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

most of my backtesting pipeline comprises of optimizers. Filtering via optimization

is a better approach in some respect in my opinion then filtering out names in the
strategy and making strategy more restrictive. Restrictive strategies might
outperform in certain market regims but would give back all of the gain when
regims suddenly shifts.

I have used weights as the language for optimizers to communicate to backtesting

and trading components.

DataSets:

Datasets are the life blood of your trading system. Having accurate data is the most
important ingredient in your trading system setup. you can get price action and
fundamental data set for US equities fairly cheaply by spending $30-$70/month. You
will have to do your due diligence and make sure the data you have is accurate by
cross checking with your brokers charts. For fundamental dataset you can go to
EDGAR and cross check. If you are scraping data by your self, you want to put alerts
and statistical check on downloads every run.

If you are pulling in public datasets like TSA, google trends, wiki trends, Flight
status. you want to be mindful of restatements when integrating in your trading
systems.

Its very difficult to get an accurate dataset even for price action at a retail traders
budget. You want to double , tripple check your datasets from time to time basis
during your backtests.

order executor (optional)

You can choose to fully automate your trading by have an order executior that will
submit your trade orders directly to your broker. I choose to get an email of the
orders and then I play them manually. Implementing this component requires that
you have thoroughly tested the executor code. you will also need to build out order
monitoring etc if you decide to implement order executor.

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 4/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

In this section, I will go over my trading system setup, various components and
microservices

Strategies:

I maintain confluence workspace where I document various strategies and mark the
matured ones so I can run backtests against them and integrate in trading system. I
have a structure while documenting my strategies which includes 3 sections,
Strategy Overview, Strategy, Entry and Exits definition , I also like to keep some
housekeeping items like notebook link used to develop it , backtest performance
etc. ( QS-1a doesnt work any more ).

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 5/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Microservices

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 6/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 7/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 8/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

PnL Reporting MicroService:

input: trade blotter and epoch

output: returns pnl and performance stats for all names

This microservice expect that you pass in the order executed (trade blotter) and it
will calculate PnL up to the current time of the backtest and return PnL for each
name, its performance etc. Trivial as it may sound but this is process is complicated
where you have to keep pnl of a position that was flipped from short to long or vice
versa?

Mean Variance Weighting MicroService:

input: current portfolio and epoch of the backtest

output: weights from optimized portfolio closest to the efficient frontier.

type : lagging

This microservice implements Mean-Variance framework to optimize portfolio. It

take list of names and current epoch of backtest. It draws 5000 samples where in
each sample randomly weights all of the names from [0–1] and then finds the most
optimized portfolio that is closest to the efficient frontier. It returns the weights of
that optimized portfolio back in the response.

Market Direction Weighting MicroService:

input: epoch

output: long vs short weighting

type : lagging

This microservice implements weighting skew towards either long side or short side
based on SPY performance (not yet implemented :2s vs 10s curve, M2 supply etc.)

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 9/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Returns long and short weights. This helps the optimizer determine how much to
skew long or short positions.

Sector Weighting MicroService:

input: portfolio and epoch

output: portfolio weights based on sector weighting

type: lagging

This microservice takes in portfolio from backtest and epoch, it maps the names in
portfolio to sectors and then lookups sector indices. using the sector indices price
action as of the epoch. it constructs covariance metric and using this covariance
metric it will generate weights of each sector compared to others. This weights will
then be mapped back to portfolio names. Names and weights will be returned back.
This helps the pipeline optimizer apply sector skew if it exists a the time.

Earning Performance Weighting MicroService:

input : portfolio and epoch

output: portfolio weights based on fundamental performance

type: lagging

This microservice take in portfolio and epoch from the backtest. It tries to look up
fundamental dataset for each name in the portfolio and generates average
performance based on few relevant KPIs for past 4 quarters (q/q). ( for now I use eps
, but I later plan to add name specific KPIs like ASM for airline names, GMV for
retail names etc ). This is then normalized across portfolio and the weight are
returned back.

Alternative Datasets Weighting MicroService:

input : portfolio and epoch

output: portfolio weights based on metrics derived from alternative data sets.

type: leading

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 10/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

This microservice takes in portfolio and epoch. it tries to lookup alternative dataset
and see if there is an existing mapping between the portfolio name and the dataset,
then it tries to compute performance of the metric that was derived from the alt
dataset. This performance is then normalized across names and returned back in
terms of weights.

Weighting Based on ML MicroService:

input: portfolio and epoch

output: portfolio weights based on ML ( Random Forest for now )

type: leading

This microservice takes in portfolio and epoch. It tried to generate following

features: avg return for past x days , std of past x days return, few other indicators
and it tried to calculate forward returns for 1 week as output marked by 1 or -1
indicating long or short position. Finally all names in portfolio are weighted 1 or -1
meaning either to go long or short. and the weights are returned back

This microservice has lot of potential to use various ML techniques and try to weight
to capture forward returns. You could also use kpis from any of the datasets that we
have already used in weighting above as features to the algorithm.

Backtesting Pipeline

Pipeline starts with choosing a strategy to run the backtest for. Each stage of the
backtesting pipeline then tries to tap into one or more microservices and adjusts the
weights which in turn reflects the $ allocations in terms of number of shares to
purchase.

Then we apply a filter stage where we drop of names that falls below certain
threshold of weight value eg: (0.01). We limit max weight for any one name not to
exceed threshold ( eg: .05) and rebalance to normalize across.

Framework then spits out final order which either you can choose to pass on to
order executor or email / notify yourself , so you can take action.

you would execute this pipeline every day after market close.

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 11/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

you could run multiple pipelines for each strategy and receive multiple orders for
which you could choose to take action or not This way you can diversify across
multiple strategies.

Appendix:

Backtesting Frameworks:

1. bt | https://pmorissette.github.io/bt/

2. backtrader | https://www.backtrader.com/

3. zipline | https://zipline.ml4trading.io/#

Links to ML Algos used in this article:

1. Random Forest | https://en.wikipedia.org/wiki/Random_forest

Trading System Backtesting Microservices Alternative Data

Follow

Written by Bhavin
36 followers · 50 following

[Data]* [Explorer, Engineer, Scientist]

No responses yet

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 12/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Moses Ongwenyi

What are your thoughts?

More from Bhavin

In TDS Archive by Bhavin

Airflow Design Pattern to manage multiple airflow projects

separate code from monolithic airflow deployment…

Nov 5, 2019 178 2

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 13/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Bhavin

COVID-19 by Numbers
I use to track COVID-19 dashboard from John Hopkins on a daily basis. which gave a fair view
into the situation. But what if you wanted a…

Apr 21, 2020 7

See all from Bhavin

Recommended from Medium

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 14/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

In Stackademic by Anurag Goel

Backtesting the Short Strangle Strategy on Nifty

Note: Don’t directly use this strategy anywhere because there is no guarantee in the real
market, this is just an backtesting result . No…

Dec 27, 2024 6

A Passionate Programmer - A Technology Enthusiast

Real-Time Push Notifications with Spring Java Microservices

Hi All , This story is open to everyone; non-member readers can click this link to read the story.

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 15/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Dec 29, 2024 22 1

Quantum Anomaly

Decoding Redis
Redis (REmote DIctionary Server) is an open-source, in-memory, key-value data store, often
used as a cache, message broker, real-time…

Jun 13 3 1

In Javarevisited by Veenarao

System Design CheatSheet for Interview

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 16/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Dear Readers, I am summarizing the commonly asked concepts in system design interviews.
These questions are asked in almost all the system…

Dec 23, 2024 1.5K 28

Rob Hutton

Mastering Messaging Patterns in Event-Driven Systems: Practical

Implementation for Developers
Learn how messaging patterns like Pub/Sub, Event-Driven Architecture, and Message
Queueing optimize distributed systems.

Feb 8 10

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 17/18
6/23/25, 10:22 AM Trading System Design using MicroServices | by Bhavin | Medium

Kevinnjagi

LangGraph: Building Stateful AI Agents

Stateful AI Agents refers to AI agents that can remember information from previous
interactions (stateful) to make decisions or solve…

Feb 17

See more recommendations

https://medium.com/@datajedi/trading-system-design-using-microservices-256cda0dc60a 18/18