The Quant Stack: Pairs Wiki

Pairs Trading Papers - Review

Quant Arb — Sat, 11 Nov 2023 15:23:03 GMT

Introduction

I’ve got quite a few organized papers on the topic, so I figured I’d share them today, and then do a bit of a literature review in terms of where I see the space.

It’s important to abstract away what pairs trading really is, and that’s just a way to find relationships between assets to price them. The prior article on lead-lag relationships is a distilled example of this but with a bit less hedging. In this case, we are saying that one will do all, or most of, the moving as opposed to convergence between the two equally. Convergence is what we focus on today, but of course, we can draw parallels to lead-lag and other areas because they are really all quite close.

If you enjoy this, feel free to subscribe. The subscriber number going up makes me write more content and you get the content I write delivered straight to you.

Here is a brief overview of our sections:

Copulas
Optimization Based
1. Multi-Objective Optimization
2. SDP
3. SMRP
4. Misc
Bivariate
Multivariate
Lead-Lag (DTW, we focused on graph-based approaches in the last article so these papers are all Dynamic Time Warping)
OU / Stochastic Control
Regime Shift
Metrics
Cointegration
Math Reading
HFT Pairs
Clustering & Pre-Selection
Misc
Notes from me

Please remember that this is simply a write-up of maybe 100 or so papers from my paper archive. There are thousands of papers in this folder that I’ve pulled from so I shall try my best here, but papers may be misplaced.

Copulas

Copulas are simply a model for expressing a relationship between two assets. There are roughly 3 areas where copulas exist in the literature:

Low Dimension
High Dimension
Vine Copulas

The lower dimension copulas are the ones you’ll read about in most popular papers. It’s generally seen as a dead approach, but there is often alpha in doing simple things well so I wouldn’t say it’s worth skipping entirely. Simpler markets where you optimize txn costs certainly could see this prevail, but there are also more advanced methods in the literature now.

Higher dimensional copulas are great for performance and certainly represent an improvement by allowing us to incorporate more assets. Sadly, the assumptions are often a bit too rigid as we go up in dimension.

Vine copulas are the solution we need for higher dimensions and are the most cutting-edge niche in the field of copulas.

There’s also some interesting work I did on using Kernel copulas, but I never finished that paper and it kind of became a rabbit hole so we won’t go into that. Math is hard kids. Do check out the topic though in your own time - it was fun.

Copula Ar Model

742KB ∙ PDF file

Download

Analyzing Dependent Data With Vine Copulas A Practical Guide With R By Claudia Czado (z Lib

8.36MB ∙ PDF file

Download

Dvinecopula Quantileregression

807KB ∙ PDF file

Download

Cvines Maxlikelihood Fxrates

600KB ∙ PDF file

Download

Dependencemodelling Vinecopulahandbook

4.35MB ∙ PDF file

Download

Oilhedging Canoncialvinecopulas

1.97MB ∙ PDF file

Download

Mixture Dvinecopulas Dependencemodelling

2.71MB ∙ PDF file

Download

Rainfall Copula Stochastic

7.72MB ∙ PDF file

Download

Regularvinecopulaefinancialapplications

456KB ∙ PDF file

Download

Regularvinecopula Financialapplications

456KB ∙ PDF file

Download

Simulating Copulas Stochastic Models, Sampling Algorithms, And Applications By Jan Frederik Mai, Matthias Scherer (z Lib

6.81MB ∙ PDF file

Download

Statarb Vinecopulas

767KB ∙ PDF file

Download

Vinecopulastrucationgoodnessoffit

632KB ∙ PDF file

Download

Vinecopulas Statarb

767KB ∙ PDF file

Download

Var Dvinecopula Garch Multivariate

1.11MB ∙ PDF file

Download

Vine Copula Autoencoders

9.68MB ∙ PDF file

Download

Optimization Based

As many of you will already know, I’ve written a LOT about this part of pairs trading on the research blog and I’ll link to past articles as well as papers here so I don’t end up repeating myself. There’s a ton of cool tricks and it’s great to be able to walk through the strategies with code and real examples, not just anecdotes from years of fucking around and finding out.

We have multi-objective-based optimization which is what the name says. We have SDP (Semi Definite Programming) which is a convex optimization method I’ve written a bit about / have shared code for in prior articles. There are SMRPs (Sparse Mean Reverting Portfolios) which I have spent a lot of time poking around in and even published a working strategy on it. We also have a nice misc section. Most of these I could explain again but you could also just read prior articles for a much better job instead of me skimming through it.

Starting with prior articles:

For the papers:

Multi-Objective:

Boostingml Weaklycointegrated

775KB ∙ PDF file

Download

Mean Variance Cardinality Evolutionary

1MB ∙ PDF file

Download

Intelligentpairstradinggeneticalgorithm

1.81MB ∙ PDF file

Download

Pairsgeneticalgo Intelligent

1.81MB ∙ PDF file

Download

Evolutionary Pairs Multi Objective

3.92MB ∙ PDF file

Download

Advancedoptimization Bollingerbands Correlation

5.4MB ∙ PDF file

Download

SDP:

Mrp Variance Threshold Cuturi

285KB ∙ PDF file

Download

Tradeoffsbetweensparsityandvol Sdp

411KB ∙ PDF file

Download

SMRP (HMM):

Sparsemrp Ar Hmm

645KB ∙ PDF file

Download

Ar Hmm Secondaryeffects Smrp

645KB ∙ PDF file

Download

Parralell Ar Hmm Smrp

1.12MB ∙ PDF file

Download

SMRP Cont.

103 L1

584KB ∙ PDF file

Download

Sparse&fast Mrp

471KB ∙ PDF file

Download

Novel Smrp 2022

603KB ∙ PDF file

Download

Mrp Cyclical Coordinate Descent

1.58MB ∙ PDF file

Download

Mean Reverting Portfolio With Budget Constraint

2.1MB ∙ PDF file

Download

Optimizaingsmrp Nn Feedforward

3.06MB ∙ PDF file

Download

Smrp Simulated Annealing

826KB ∙ PDF file

Download

Sparsemrp Penalized Likelihood

595KB ∙ PDF file

Download

Sipos Smrp

1.84MB ∙ PDF file

Download

Smrp Full Algorithm System

2.72MB ∙ PDF file

Download

Zhaozhoupalomar Tsp2019 Mrp Leverage

4.46MB ∙ PDF file

Download

Misc:

Spca Smrp

286KB ∙ PDF file

Download

Smrp Trading With Lstm

1.28MB ∙ PDF file

Download

Mixedinteger Nn Solving

4.81MB ∙ PDF file

Download

Bear with me and maybe skip the neural networks. NNs were fun topics, but they never yielded much results (KISS - Keep it simple stupid).

Bivariate

2 assets, 1 portfolio. Simple as that. Here goes:

Aus Pairs Trading

165KB ∙ PDF file

Download

Bivariateapproachpairs

210KB ∙ PDF file

Download

Pairs Trading And Idiosyncratic Cash Flow Risk

372KB ∙ PDF file

Download

Dynamicmodelling Mr Spreads Statarb

507KB ∙ PDF file

Download

Pairs Relativevaluearbrule

319KB ∙ PDF file

Download

Hmm Ica Pairs

134KB ∙ PDF file

Download

Vecm Pairs Trading

257KB ∙ PDF file

Download

Equitypairstrading

769KB ∙ PDF file

Download

Pairstradinggeneralstatemodels

1.23MB ∙ PDF file

Download

Pairstradingpptx Bivariate

73.5KB ∙ PDF file

Download

Stochasticmodelling Cointegrated

9.19MB ∙ PDF file

Download

Multivariate:

This has some interesting relations to the optimization section as well, but I did keep them separate since they generally appear to be two distinct literatures. A lot of the papers in this area tend to reference each other whilst the optimization section tends to reference others in that section. That said, they’re similar:

Vmat Multi Pairs

397KB ∙ PDF file

Download

Zhaozhouwangpalomar Ssp18 Mrp Portdesign

1.01MB ∙ PDF file

Download

Multivariatepairstrading M Ofakind

182KB ∙ PDF file

Download

Mpo Game Approach

2.64MB ∙ PDF file

Download

Multi Pairs Statistical Learning

2.63MB ∙ PDF file

Download

Statistical Arbitrage In The Us Equities Market

3.03MB ∙ PDF file

Download

Lead-Lag (DTW)

The above article gives quite a few papers on the subject, especially in the graph-based approaches but here are quite a few papers from my dive into DTW:

Derivative Dtw

146KB ∙ PDF file

Download

Temporallogicbasedmethods

281KB ∙ PDF file

Download

Leadlag Indepth Fx

649KB ∙ PDF file

Download

Symmetric Thermal Optimal Path And Timedependent Leadlag Relatio

910KB ∙ PDF file

Download

Ftse100 Leadlag Intraday

313KB ∙ PDF file

Download

Ftw

263KB ∙ PDF file

Download

Non Parametric Optimalthermalcausalpath Zhou2006

877KB ∙ PDF file

Download

Optimalcausalpath

1.4MB ∙ PDF file

Download

Optimalthermalpath

747KB ∙ PDF file

Download

Leadlagnetworks Dynamicpatterns

1.72MB ∙ PDF file

Download

Evidence Dtw Covid Metals Relationship

5.51MB ∙ PDF file

Download

Varyingleadlag Generalizecausalityalgorithm

11.1MB ∙ PDF file

Download

OU / Stochastic Control

Some people seem to really enjoy this area, maybe it is the math, but I always found that it was too conservative and the assumptions were too rigid to work well in practice. Bollinger bands for the win imo. That said, you can find some very well-behaved spreads, closer to the pure arb part of the trade that this applies well to so I think it’s worth learning:

Analytical Solutions For Optimal Statistical Arbitrage Trading

348KB ∙ PDF file

Download

Pairs Trading Optimal Thresholds And Profitability

535KB ∙ PDF file

Download

Finding The Optimal Pre Set Boundaries For Pairs Trading Strategy

245KB ∙ PDF file

Download

Optimal Pairs Stochcontrol

506KB ∙ PDF file

Download

Pairstrading Spread Variability

614KB ∙ PDF file

Download

Bertramspairs Boundedrisk

1.11MB ∙ PDF file

Download

Optimalmrp Stoploss Transactioncosts

350KB ∙ PDF file

Download

Ou Dynamic Mean Variance

737KB ∙ PDF file

Download

Ou Pairs

317KB ∙ PDF file

Download

Optimalfastmrpstochastic

396KB ∙ PDF file

Download

Optimaltradingstrats Levydriven Ou

2.01MB ∙ PDF file

Download

Optimalportfoliomrp

769KB ∙ PDF file

Download

Realtime Stochastic Portfoliooptimization

2.99MB ∙ PDF file

Download

Multiasset Optimal Ou Pairs

2.74MB ∙ PDF file

Download

Optimalmrp Textbook

3.31MB ∙ PDF file

Download

Regime Shifts

Honestly, you can find simple proxy metrics that do the job just as well, but there is a bit of a literature. I’ve spent my time reading and testing these to know they don’t work that well so now hopefully you won’t need to either. This is a paper dump though so here they are:

Statarbhmm Regimeshifting

114KB ∙ PDF file

Download

Flexible Hft Regimeshift Pairs

847KB ∙ PDF file

Download

Structuralbreakpairstrading

2.69MB ∙ PDF file

Download

A Machine Learning Approach To Regime Modeling Two Sigma Webpage Pdf

3.7MB ∙ PDF file

Download

Metrics

I like this part a lot. It’s a fun area and thinking about what these metrics are truly describing is an integral part of the process when it comes to building good relationships between assets:

Test For White Noise

32.4KB ∙ PDF file

Download

White Noise Tests

179KB ∙ PDF file

Download

Crossingstat

349KB ∙ PDF file

Download

Introducing Hurst Exponent In Pair Trading

591KB ∙ PDF file

Download

2 Stage Hurst Fast

480KB ∙ PDF file

Download

Persistencebaseddecomposition

456KB ∙ PDF file

Download

Fasthurst

596KB ∙ PDF file

Download

A Fast Estimation Algorithm On The Hurst Parameter

501KB ∙ PDF file

Download

Pairs Persistencedecomposition

227KB ∙ PDF file

Download

A Cooperative Dynamic Approach To Pairs Trading

2.79MB ∙ PDF file

Download

Statarb Multiasset Using Tsa

1.09MB ∙ PDF file

Download

Cointegration

This is the one everyone knows. There are some interesting approaches to it that have come out recently but I think they’re still too close to what is a dead trade. We can do well finding underdeveloped markets / fundamentally robust pairs and using these methods, but there’s a lot more thought that goes into it:

Kalman Pairs Statarb

441KB ∙ PDF file

Download

Cointegration And Subset Correlation Hedging

929KB ∙ PDF file

Download

Ms&e 448 Final Project Statistical Arbitrage

945KB ∙ PDF file

Download

Cointegration Enhancedindextracking

2.1MB ∙ PDF file

Download

Jse Partialcoint Statarb

2.12MB ∙ PDF file

Download

Avellanedaleestatarb2008

2.59MB ∙ PDF file

Download

Math Reading

You should probably get good at math if you want to be a quant, especially if you are interested in a model-focused area like pairs trading. Here’s some material to help you along:

Highordercrossings Timeseries

193KB ∙ PDF file

Download

Truncated Mathematics

630KB ∙ PDF file

Download

Var Ma Process

534KB ∙ PDF file

Download

Sgep Smooth

458KB ∙ PDF file

Download

Generalized Eigen

151KB ∙ PDF file

Download

On The Simulation And Estimation Of The

1.63MB ∙ PDF file

Download

Boxtiao Canonical Bio 77

574KB ∙ PDF file

Download

Sparserecovery Conicprogramming

152KB ∙ PDF file

Download

Eigenvalues And Gep

151KB ∙ PDF file

Download

Nonsymmetric Sparse Matrix Factorizations

3.03MB ∙ PDF file

Download

Sparsereducedrankregularization

760KB ∙ PDF file

Download

Levelcrossing

2.6MB ∙ PDF file

Download

Box Leung

6.66MB ∙ PDF file

Download

HFT Pairs

There’s a bit of literature on pairs trading for high-frequencies, but with all things HFT, you need to learn from practice. The literature tends to be less relevant for HFT strategies owing to the very practical and market feedback-oriented development process behind them which academics/theory-loving individuals tend not to go for. Either way here are a few, but I do recommend reading some of my material on this which is a bit more practically focused (although it is still quite mid-frequency in its scope), and of course testing for yourself:

Statarb Pairs Hft

832KB ∙ PDF file

Download

Statarbstrats Hft

658KB ∙ PDF file

Download

Pairs Trading Using Hft In Omx Baltic Market

723KB ∙ PDF file

Download

Twostagecorr Coint Dynamic Hft Pairs

924KB ∙ PDF file

Download

Intradaypairs Hft Oil

622KB ∙ PDF file

Download

Statarb Hft Gunderson

1.96MB ∙ PDF file

Download

Pairstrading Jumpdiffusion Hft

888KB ∙ PDF file

Download

Ml Hft Pairs

1.7MB ∙ PDF file

Download

Hft Copula Pairs

1.38MB ∙ PDF file

Download

Statarb Lob Imbalance

2.58MB ∙ PDF file

Download

Hft Pairs Micro Nanosecond

891KB ∙ PDF file

Download

Clustering and Pre-Selection

I never dove too much into this from a theory/models perspective because my approaches were usually based on fundamental similarities behind assets such as industries, but there is a machine learning side of things that is quite interesting and that I have explored a little:

Multi Factor Statistical Arbitrage Model

1.14MB ∙ PDF file

Download

Cluster Based Statistical Arbitrage Strategy

1.88MB ∙ PDF file

Download

Dataselectionpairsavoidoverfitting

2.82MB ∙ PDF file

Download

Misc.

I don’t have more papers to share, but I do want to give a shout-out to financialnoob who has written a decent bit on pairs trading and shared a lot of code. I think it’s a great resource for those diving into the subject and covers most of the simpler methods that I perhaps have glazed over.

Simple methods do well and often are the most robust way to do pairs trading. Don’t write them off. I worked on pairs trading at a more model-focused time in my life and hence my thoughts and writings often flow into this area, but my current research represents more of a balance.

It isn’t very pairs trading oriented anymore, but I’ve had a few looks around in the years since pairs trading and have enjoyed the application of simpler models to dumber markets.

Both sides of the coin are fun and teach you a lot.

https://medium.com/@financialnoob

He’s also got a Github which saves me from writing code on these methods (mostly ones I haven’t covered yet anyway) and as a bonus it’s all free. Enjoy :)

Notes from me

Please excuse any mislabelling of papers, I’ve put a lot of work in but as you can see there are a lot of them to go through, and as with all papers - I skim. Hopefully, my commentary has provided guidance and my articles that exist so far will provide more practical insights into this area of strategies.

On the point of skimming, do it. Have a skim and then come back to what you enjoy. There’s a lot to get through and most papers don’t require you to read the full thing to get the general idea. If you really like the idea then of course dive in and code it up / test it out - it’s a great way to learn, but do be diligent with your time. It’s not a great idea to just sit there and read them all day, in full - you’ll forget a lot of the parts anyway if you do that.

Thinking about stationarity the right way

Quant Arb — Fri, 23 Jun 2023 05:49:52 GMT

Introduction

A Real Pairs Trading Strategy

Quant Arb — Sun, 11 Jun 2023 05:35:16 GMT

Introduction

In this article, we will go into detail about an equities strategy I built in 2020. This was traded live for a few months and delivered 30% returns on a 2.7 Sharpe (over about 4 months) until I stopped trading equities (crypto is more exciting).

My journey into quant started with options and volatility trading (really early days). I quickly realized I knew nothing about volatility trading and instead fell in love with the field of pairs trading. This was a field I researched for a while and even wrote papers in. The strategy we discuss today trades a portfolio of equities.

There are a few optimizations which add to the performance that I’ve left out the exact details on, but for most of them, I still cover what they achieve so readers can come up with their own ways to benefit from the ideas behind them.

General Strategy Overview

This strategy is built around multivariate pairs trading and uses the optimization approach to pairs trading. It is quite a complex strategy, but all of the alpha comes from simple and elegant ideas. This is the way to do it; the complexity of a strategy is usually inverse to alpha. You could do this very simply; it would just suck. Each idea that adds complexity has clear reasoning - we aren’t randomly slapping an LSTM on it or anything.

We generate a portfolio of assets (4-6) with some optimization objectives for the spread and portfolio characteristics. Reduce turnover with some simple methods. Repeat this step 10-30x to generate 10-30 portfolios. Use a validation technique to pick the best ones for robustness, then trade the top portfolios live.

Putting this into a list of steps:

Identify the universe of assets to trade (liquid, has short margin, and maybe even some filters for characteristics like removing meme stonks).
Generate portfolios by optimizing for an ensemble of objectives.
Sit still for a short out-of-sample “validation” period where we do not trade.
Compare the in-sample portfolios vs. validation period portfolios and ensure we didn’t overfit (ex. - ultra mean-reverting now barely mean-reverting OR stationary portfolio now drifting off into space).
Trade the selected portfolios in a 3rd and final test set using a slightly modified Bollinger Bands implementation.
Continuously monitor for the decay of statistical properties and shut down decayed portfolios.

We’ll walk through this list and the considerations in the following sections.

Universe Selection (1)

This part starts with the basics - what assets are not going to cause issues. We want assets with a decent amount of liquidity and availability of short margin. That’s the first step, and it usually just means we trade only S&P500 stocks because margin availability is like a $250,000 dataset.

After this, we run into an issue with universe size. 500 assets are far too many to optimize over because we’ll overfit, so we shrink it down. Our universe size is actually a parameter for the strategy and has a pretty big impact on results. This should be informed by the timeframe of mean-reversion we want to capture.

Longer-term pairs trading warrants a smaller universe as there is a higher risk of overfitting, and inversely shorter-term pairs trading lets us have larger universes as we look for more transitory relationships than robust ones. Hence we fit more.

My previous articles on pairs trading have explored this and used real data to show it happening in practice. Feel free to check them out.

As a related component to universe selection, we also need to think about the size of our train, validation, and test periods in terms of samples. I can give the same algorithm hourly data and daily data, and I’ll get different results even if the length of the train, validation, and test periods are the same in terms of time. Sampling less frequently removes noise, and as a rule of thumb, there is more noise per data point as we go down timeframes.

When we are doing short-term pairs trading, we need more samples due to the higher noise per sample. With long-term pairs trading, this just adds noise and confuses our optimization algorithm with relationships that are significant but transitory. With short-term pairs trading, we can just use validation to know when the portfolio has decayed and take advantage of these transitory relationships, whereas long-term pairs trading does not capture these relationships because they’re gone by the time we get to the test set. The data agrees with this observation of optimal sample sizes when you test it out - the more you do, the more intuition you build.

The universe can be randomly split into equal slices of a certain size, but there are smarter ways to do things. I won’t cover the exact details, but here are some pointers:

Clustering algorithms (DBSCAN recommended) based on:
1. Factor exposures
2. Fundamentals
3. Time series characteristics (Hurst, volatility, mkt cap adj. volume, etc.)
Maximize similarity (correlation and the mix) as an objective using a greedy algorithm to shrink 500 → 50 (arbitrary example).
Group by industry
Top/bottom XX by some characteristic (use the metrics above in 1c).
Group by beta to things like political polls or exposure to macroeconomic variables (ex. CPI)

Depending on the level of optimization done to create the universe, 20-70 is optimal for this strategy. 50-70 being no optimization (randomly split into buckets). 20-30 being a fair bit of optimization in our universe selection. Play around with it and see what feels right.

I’ve given a few ideas, and there is not a specific one that is the correct option. Test them, see what works, and ensemble the versions that work for maximum diversification points.

Optimization Objectives (2a)

We have primary and secondary objectives. Our primary objectives capture what makes us money, and our secondary objectives capture what loses us money.

Mean-reversion makes us money. Simple as that. Here are our metrics:

Hurst
Portmanteau
VAR(1) Predictability
Crossing Statistic
SNVR

Greedy Method for SMRPs (PART 2)

Quant Arb — Wed, 29 Mar 2023 00:38:58 GMT

Introduction

We recently explored the use of the Greedy algorithm for generating Sparse Mean Reverting Portfolios (SMRPs) using the Portmanteau criterion. In this article, we will expand a little on this method and look at 2 other metrics which take a slightly different perspective when optimizing. These two metrics are:

Predictability (Box-Tiao Canonical Decomposition)
Crossing Statistic

Code will be included so that you can test this out for yourself. I’ll also add some commentary on how these metrics can be useful when used together.

Pairs Trading Framework & Process

Quant Arb — Thu, 23 Mar 2023 03:28:33 GMT

Introduction

As many of you will have seen, recent articles have focused a fair bit on pairs trading methods. After finishing a 3 part series where we went from non-sparse mean-reverting portfolios up to sparse mean-reverting portfolios I thought it was about time to give a proper guide on all the different nuances of pairs trading.

In this article, we will take a different perspective than we normally chose, and cover pairs trading, specifically building mean-reverting portfolios, as a start-to-finish framework. Many statistical arbitrage managers will run these strategies alongside others, and this is by no means an exclusive area to specialize in.

DISCLAIMER: This is going to be quite, if not a lot, anecdotal. It is based on a lot of research into the space, and I even ended up writing a paper on some sparse eigenportfolio math during my adventures. I don’t often work with these strategies anymore since migrating from equities, but I hope to share the knowledge regardless.

Index

This is quite a long article as it is a full guide to pairs trading and all the lessons/anecdotes I’ve accumulated over the years. You’ll be happy to find that there is an index to help organize all of this content:

Introduction
Defining Pairs Trading
Pairs Trading as an Optimization Problem
Robust or Non-Robust: A Temporal Question
The Parameters That Matter
How Data Parameters Drive Modelling Choices
Short-Term High-Turnover Pairs Trading
Long-Term Low-Turnover Pairs Trading
The Different Flavors of Mean Reversion
Optimization Metrics (Mean-Reversion)
Optimization Metrics (Alternative Objectives)
Methods for Optimization
Non-Optimization Focused Approaches
Conclusion

Defining Pairs Trading

I think most readers have a general idea of what pairs trading is, and if you don’t - Google exists, but I want to really dig into the question of what is pairs trading?

For many, it is simply trading the spread between two cointegrated assets, but this is no longer true or an effective definition. Modern pairs trading can involve a variety of assets, with a mix of long and short exposures. They don’t have to be cointegrated or stationary either. You can generally think of pairs trading as the art of generating, trading, and managing portfolios that exhibit both predictability and efficient tradeability. Predictability in this sense refers to the fact that our portfolios follow a (usually mean-reverting) pattern that is robust and stable. This is not to be confused with the VAR(1) (BTCD) approach where predictability means something different.

Where does pairs trading end? This isn’t an easy question either. If I take a KO - PEP spread and decide to only trade one leg, the one I expect to do the moving, am I still pairs trading? I could be capturing 90% of the variance of the equivalent (although hedged) pairs trade but would this still be a pairs trade?

In most cases, the scenario where only one asset will move occurs because one asset is driving all of the variance between the two of them. There is a leader and a lagger. This is a lead-lag trade, one that has many similarities to pairs trading, and with a lot of room to bridge ideas, but is definitely still its own distinct area.

Pairs Trading as an Optimization Problem

Pairs trading in my view is best thought of as an optimization problem. We have 3 primary objectives we need to optimize for:

Predictability (Well Behaved Mean-Reversion)
Trading Costs Relative to Profit (Liquidity, Variance, Notional vs. Net Ratio)
Stability of Statistical Properties (Stationarity - although often this is worth giving up to get a much better deal on parts 1 & 2)

You could very well sit down and create a metric for mean reversion then slap some assets into a neural network and shit out a mean-reverting portfolio. That doesn’t guarantee that your portfolio will be robust, it will likely be overfit in fact, and will decay completely in live testing. Often a healthy dose of heuristic methods and well-thought-out processes will provide the best results with our limited data.

We have already framed it as a problem like this in previous articles. Our SDP article optimized for mean-reversion using a VAR(1) framework (objective 1) and made trading efficient by inducing sparsity and variance constraints (objective 2). It was also convex, which means it had far less ability to overfit, helping us achieve objective 3. An earlier article took a more aggressive stance and used the portmanteau statistic to achieve objective 1, but only implemented some controls on the liquidity requirements of our input universe to help aid objective 2. It was definitely not the most robust technique, but as we will discuss later, robust != optimal for many cases.

Robust or Non-Robust - A Temporal Question

In my view, it was never a question of how can we make our portfolio robust always. Unstable portfolios add cost, but you can still make money if the other objectives get optimized really well. Below is a portfolio that is constructed from two fractional Brownian motion processes. One has a Hurst exponent of 0.3, the other has a Hurst exponent of 0.9. What this means is that the 0.9 Hurst process is our drift (trend) which we do not want, and the 0.3 Hurst process is the mean-reversion (we want this). The trend can be thought of as noise that damages our PNL, but we can still adapt to it. The use of techniques like Bollinger Bands allow us to trade non-stationary spreads, with the error term from non-stationarity as a cost.

The cause of loss for Bollinger Band based approaches is what I call lag error. I have covered lag error in a thread on my Twitter:

Moving back to the concept of robustness. We cannot have everything, and there is usually a question of what model is right for the job. If you are trading higher frequency timeframes, you need to optimize for trading costs, and you will likely trade short-lasting effects which will quickly disappear. This is fine. We already have a framework that is robust to instability. Instead of non-stationary behavior making it impossible to trade it just acts as a variable cost against our revenue from mean-reversion. More effective spread trading models and optimization for well-behaved mean-reversion lower this cost.

The reason I say this is a temporal problem is that as you increase your timeframe, you have less opportunity to diversify, both in the correlation of your spreads and the law of large numbers with trades. This is partially because correlations increase as the timeframe increases (so you have less ability to form multiple diversified pairs portfolios). The other issue is that you will make fewer trades so the law of large numbers applies less. Above both of these, the key reason why it makes sense to pick more flexible and less robust models for shorter timeframes is because the relationships are more complex and unstable on shorter timeframes. To detect these we use models which fit more but do not create as robust portfolios, in a lot of cases the extra mean reversion we gain outweighs the lag error introduced.

The Parameters That Matter

I wouldn’t say any of these are necessarily parameters as you might normally think of them, but they are under your control. Here is a quick list of the data-related ones:

Universe Size
Data Frequency
Training Period Length
Test Period Length
Validation Period Length / Use

It is important to use the right data. How large should the starting universe you pick from be? Should we filter for liquidity to reduce turnover costs? What about the timeframe - there is a big difference between 5-minute bars and daily bars. You also get to play around with the length of your training, validation, and test sets.

The validation set is an optional extra set you add between the training and testing sets. You might generate 30 portfolios, then for a validation period you measure their ADFs, rank them on ADFs, take the top 3, and trade the top 3 in test. Here we are sacrificing the number of portfolios / “freshness” (much like an apple, portfolios decay quickly, especially when trained on short-term data) for hoped stability.

What about model-related choices we can control? Here’s a quick list:

Portfolio Sparsity
Metrics / Combination To Fit The Task
Optimization Model

There is definitely a “right” set of metrics to get a certain type of mean reversion. This is important and should come inside of the overall situation context. All of these questions are part of this context, but typically our first list is what drives the choices we make in our second list.

Portfolio sparsity isn’t something that greatly affects the mean-reversion of portfolios, a 4-asset portfolio might behave the same as a 6-asset portfolio, but in the long run, the extra cost to trade 2 more assets can add up. It is really a question of inducing sparsity, which is just a branch of the overall objective of reducing turnover costs, but at the cost of the mean-reversion or stability the portfolio exhibits. If assets do not meaningfully contribute to the portfolio there are methods like Truncation contribution to test whether they are noise and exclude them.

How Data Parameters Drive Modelling Choices

The data parameters directly affect how you then model it. This is a pretty basic concept, but we will walk through the 3 different modeling choice subjects and how they are affected by different data. This all reverts to the different focuses you have with different types of pairs trading.

The next two sections will examine short-term high-turnover pairs trading and long-term low-turnover pairs trading to give insights about what are the right choices and what problems should you focus on solving based on your data/timeframe of effect.

Short-Term High-Turnover Pairs Trading

With short-term high-turnover pairs trading:

Transaction costs are important and thus we care a lot about:
- Spread
- Exchange Fees
- Sparsity (2 legs is easier to limit fill than 10)
- Volatility (larger moves, easier to beat costs)
- Notional vs. net exposure ratio (If I have to spend $1000 notional to buy a $100 net exposure portfolio that is effectively a 10x multiplier on trading costs as most of the exposure I buy cancels out). Called SNVR (standardized notional volume ratio when we adjust for volatility, otherwise it is just NVR).
Mean reversion effects are far stronger.
Effects are generally less robust.
Fleeting / more complex relationships require more fitting.
More data to work with.
Less information per data point (this combines with the above point to create larger train/test datasets in terms of data points).
Low correlation between different pairs portfolios.
Higher Sharpe / reward → much harder. A high turnover means you are temporally diversified. If you make 1000s of trades they will average out and your risk will be lower, this is boosted by low correlations. This means the reward is larger compared to long-term methods, but that slight edge per trade is competitive.

The implication of some of these is quite substantial. If you know that finding highly mean-reverting portfolios is easier, then you are fine paying some of that away as lag error and using bolling bands / dynamic mean based methods of capturing the mean-reversion. If your mean-reversion is weaker (like long term) you need to focus on robustness because you cannot afford to pay for the lag error of a non-robust portfolio.

There is huge importance in trading costs and optimizing that. How many elements should your portfolio contain? Can we use limit orders to execute? You might have 6 assets, but if 1 asset is 40% of the portfolio it is perhaps worthwhile trying to get that leg in as a limit order, but not always.

Think of short-term high-turnover pairs trading as the following equations:

The average move captured puts volatility, but also our ability to capture it (lag error) at the front of the equation. We then think of mean-reversion strength & quality as the probability of this being in our favor and multiply as such. Trading costs are subtracted net of all this.

Long-Term Low-Turnover Pairs Trading

With long-term low-turnover pairs trading:

Low turnover means transaction costs matter far less
Mean reversion tends to be weaker, but more robust
Risk premium / generally easier to make money in
Fewer data points to work with so requires less fitting
Portfolios tend to be correlated / diversification is harder
News can affect prices more, creating tail risk

As with longer-term strategies in general it is easier to make a little bit of money, but many many times harder to make a lot of money in a low-risk manner. The weaker mean-reversion strength also means that methods like stochastic control are not suitable to trade the spread as they will fail to fit properly.

It is important to understand the nature of these effects and that they are different when looking at the short-term and the long-term. The best way to think of it in my view is that with long-term effects you are focusing on collecting a small risk premium each time and not taking downside. For short-term pairs trading it is the other way around. Your default starting point is losing small amounts of money because of transaction costs. Once you cross that threshold of beating fees your profit rockets.

Longer-term strategies tend to have fewer data points to work with and there is less time for losses to average out, not to mention the fact that portfolios have higher correlations killing diversification. This all culminates to require the use of simpler models when generating long-term portfolios and focusing on how can we keep these portfolios stable. When each trade matters / has a larger impact on the final PNL you need to focus on improving quality since you cannot rely on quantity / averaging out.

The Flavors of Mean Reversion

Ideal mean reversion should have minimal lag error. This means it should have one single frequency and stable properties.

What do we mean by frequency? This is where fractal methods like Hurst can come in handy. More intuitively, “how does price cluster around different price levels?”, and “how many levels are there?”. If our process has 3 states, up, down, and at mean if it is only able to be in one of those 3 states and the prices for those states are constant we get a portfolio like this:

The above chart clearly only has 3 prices it ever visits.

Since we have no lag error we can capture these moves really well. If the price is at our upper band we have close to 1 probability that the price will then go down. In normal settings, however, the probability curve does not look like this.

The different frequencies that mean reversion or generally cyclical behavior occur on primarily influence how we trade the spread.

This is an example of a mean-reverting process with 2 different frequencies. We either capture the short-period mean reversion or the longer-period mean reversion, but each will work against us.

If we capture the longer period we will just have all this extra mean-reversion add noise to our PNL curve. If we are capturing the short period, one of our bands will be more likely to get hit because of the longer period reversion and that will introduce a large degree of lag error.

This is not really an issue for longer-term pairs trading as portfolios tend to exhibit simpler mean-reversion. Whether this is a fundamental nature of mean-reversion on this timescale or a reflection of the fact that our models must be simpler I can’t say, but either argument would be valid.

For short-term pairs portfolios, there is definitely an issue with multi-fractal mean reversion. Models have far more data and tend to be able to fit more making this a common type of mean-reversion to encounter.

Mean reversion may also be thought of in the context of white noise or stationarity, but this is a topic with plenty of literature to dig through so I will leave this to the reader.

Finally, I will list some metrics to classify mean reversion:

Half-Life (speed of mean-reversion)
Crossing Statistic (how many times it crosses the mean)
Volatility (self-explanatory)
Kurtosis (We want bounded mean-reversion, not infrequent mega deviations)
Quantile Ratios (Absolute the z-scores, then calculate quantiles, compare the ratio of different quantiles, maybe 95% vs. 99%, is the difference large, this acts as a more fine-tuned version of kurtosis when controlling tails)
Stationarity (non-stationary white noise is a whole different ballpark than standard ADF-created portfolios)
Drift Term (directly looking at drift / change in price instead of stationarity)
Information discreteness of smoothed process (after smoothing, i.e. removing the noise/ mean-reversion, what is the % of data points that had a positive % change vs % of data points with a negative % change? This tells us whether it is non-stationary with a constant smooth drift or if it is all over the place).

Optimization Metrics (Mean-Reversion)

We’ve already given some metrics which classify the behavior of mean-reversion and are some custom metrics I developed whilst working on pairs trading years ago, but here I’ll list through some metrics for mean-reversion that are stand-alone. The previous list was metrics that help you assess the quality of mean-reversion, but most of them still don’t optimize for mean-reversion directly, only the qualities we would want to see if we had mean-reversion. Hence, most of these optimizations are multi-objective.

This is by no means a complete list and should only serve as inspiration. The key is to focus on how these methods work and understand what we are actually capturing.

ADF
KPSS
Portmanteau
Hurst
Crossing Statistic
AR-HMM (similar idea as BTCD, Levendovszky & Sipos wrote the literature here)
Zivot Andrews
BTCD - Box Tiao Canonical Decomposition (also called Variance Ratio or VAR(1))
Kanaya, 2011
Delft et al, 2017
Basu et al, 2009
Delft and Eichler, 2018
Vogt & Dette, 2015

There is existing literature finding semi-definite relaxations for a lot of these metrics so that we can optimize for them in a non-sparse manner before using greedy search to make it sparse, but some of them do not have such relaxations published. One such example is the Hurst exponent, which is a shame because it performs so well once relaxed. Pick up a copy of Boyd’s Convex Optimization and work through it yourself, it isn’t too hard if you have the background down.

Mean reversion strength and robustness are often 2 different tests. In my experience, it is always the tests like Portmanteau which scarcely produce stationarity but also perform the best. You just need to control for stationarity/robustness of the portfolio after the fact in a validation set.

Here is an example with the use of a validation set:

Train 30 portfolios using portmanteau as your objective (train set is 2x as long as test), then for a period equal in length to our test set we wait and watch the portfolios, we then rank them all on their ADF scores or some stationarity metric, and then take the top 3. We run these 3 live in our test set.

Our validation set created a trade-off where we were able to reduce the chance of overfitting by having a period where portfolios could decay after being trained and then can be thrown out before actually trading them. We also improve stability by picking the most stationary portfolios. This comes at the cost of information decay. Instead of being run right after training, when the optimization is most relevant, we had to wait for the validation set meaning that some good portfolios potentially lost mean-reversion strength. For me, the validation set was a great choice in most cases that I worked on, but I would only use it for short-term pairs trading where mean-reversion comes first and stationarity comes second. Longer term you cannot afford to wait because this is a timespan of multiple months, you wouldn’t want to trade a portfolio trained over a year ago.

Optimization Metrics (Alternative Objectives)

We have already listed quite a few alternative metrics under the area of quality of mean-reversion so the remaining metrics focus on transaction costs/practicality.

Our goal is to reduce the amount of notional needed to get a net exposure as this increases costs. If I need 10x net in terms of notional volume to capture a 10% move in the spread, I am effectively paying 10x fees. If I need 3x the net exposure in terms of notional then I have less than 1/3 the cost if I was to capture the same move.

I won’t go ahead and list them all as they are pretty self-explanatory, but you should focus on what drives costs. This should also be a part of the execution. If it is 1 asset long against 3 others short then you can probably execute that 1 asset as a limit order / using actual smart execution where you have some time to execute the order. If it was 3 vs 3 then you have executed 16.5% of the portfolio if you fill one, but if this is 3v1 and you execute the 1 then you have nailed 50% of the portfolio in one fill. If there are 4 orders you get quite a complicated problem of unhedged exposures whilst some legs have been filled. This is another complex problem to solve, one that is rewarded with alpha.

There is also a significant literature on optimization for leverage / budget constraints as well as other real-world constraints that users will deal with for optimizing a full suite of pairs trading portfolios. Literature on this is included at the end.

Methods For Optimization

There are many methods to optimize your portfolio. I will list some of them, specifically the ones I have tended to look at, but will probably ignore others. Do your own research here, but note that the right model will not make you any alpha, but the wrong model will sure as shit prevent you from finding it.

LASSO (brute force)
Simulated Annealing (Dual annealing is better)
Basinhopping (I prefer this to the one above / they are very similar)
Greedy
Truncation
SDP (Semi-Definite Programming)
Non-sparse solving as GEP (used in methods like Greedy / Truncation as a sub-problem)
Feed Forward Neural Networks (They don’t work, but I will give them the mention)
Cyclical Coordinate Descent
Other Brute Force-Based Approaches
DSPCA (SDP is an extension of this)
DC-PCA (difference of convex functions algorithm, DCA)

Relevant literature can be found at the end.

Non-Optimization Focused Approaches

We have tackled pairs trading within the framework of optimization, but there are other ways to think of it as well. I prefer thinking of it as a robust optimization challenge and have found a lot of success this way, but there are also approaches that model the relationships between assets instead of optimizing for a final goal.

The primary approach here is copulas, which can be extended into higher dimensions with vine copulas. This is an approach that performs well and should not be discounted, but it is not something we will dive too deep into.

There are also models which use graphs, but those lean more into the lead-lag literature where we model it as a network.

Optimization methods still play a key role in these as we need to identify our starting universe. This is often done by creating a portfolio using the greedy method and then fitting a copula / vine copula / whatever you want to model to the constituents.

Additional Areas of Study

We have not covered the trading of the spread, universe selection, or regime shift models (the existing literature sucks, just ask yourself “is this portfolio different enough OOS compared to IS that I actually care.” If the mean reversion is slightly lower but it is still stable then it is fine, if the mean reversion is gone and it is unstable then it is useless. There is no need to use a fancy HMM model, a simple metric should serve as an easy proxy).

Universe selection is also an extensive topic and as we saw in the last articles it certainly plays a role in the final output. This can be done by grouping based on industry, or using a simple method to optimize for some metric (maybe average correlation between assets, can significantly speed up this process by taking the eigenportfolios and using the correlation vs the main eigenportfolios instead of pairwise, this reduces noise / overfitting and makes compute faster - I’d still use greedy alongside this though).

There are also machine learning-based methods to universe selection like clustering algorithms (HDBSCAN / DBSCAN is the go-to as K-means produces groups that tend to look like a Voronoi diagram), but at the end of the day the metrics you cluster by are still the most important part.

Conclusion

I hope to end on the note that finding alpha is hard. I gave a list of custom metrics I developed over my time, but to find your own alpha you need to take the same route of understanding these effects and how they behave. How can I isolate effects that improve performance and then capture them without hurting other objectives?

We also didn’t dive much into the problem of trading spreads as this is a whole article on its own.

This should serve as a complete introduction to creating portfolios for pairs trading, I’ll leave it up to the reader to form their own ideas on how to then trade these portfolios as well as come up with novel modifications (coming from an area of understanding/observation of the effects) to find new alpha.

If you enjoyed the post consider subscribing! There are also a few articles I’ve previously written that take a more practical dive into different methods so feel free to check them out.

Literature

“Three l1 based nonconvex methods in constructing sparse mean reverting portfolios”

“Mean-Reverting Portfolio With Budget Constraint”

“Efficient computation of mean reverting portfolios using cyclical coordinate descent”

“A Novel Optimization Approach to Sparse Mean-Reverting Portfolios”

“Optimizing a portfolio of mean-reverting assets with transaction costs via a feedforward neural network”

“Constructing sparse and fast mean reverting portfolios”

“Optimizing Sparse Mean Reverting Portfolios with AR-HMMs in the Presence of Secondary Effects”

“Parallel Optimization of Sparse Portfolios with AR-HMMs”

“Optimal Mean-Reverting Portfolio With Leverage Constraint for Statistical Arbitrage in Finance”

“Sparse mean-reverting portfolios via penalized likelihood optimization”

“Sparse, mean reverting portfolio selection using simulated annealing”

“Developing a fully automated algo-trading system”

“Optimizing sparse mean reverting portfolios”

“A Penalty Decomposition Algorithm with Greedy Improvement for Mean-Reverting Portfolios with Sparsity and Volatility Constraints”

“A simplified approach to parameter estimation and selection of sparse, mean reverting portfolios”

“Mean-Reverting Portfolio Design via Majorization-Minimization Method”

“Identifying Small Mean Reverting Portfolios”

“POLYNOMIAL TIME HEURISTIC OPTIMIZATION METHODS APPLIED TO PROBLEMS IN COMPUTATIONAL FINANCE”

“Detection of mean-reverting phenomenon from American stocks with unsupervised techniques”

“Trading sparse, mean reverting portfolios using VAR(1) and LSTM prediction”

“Solving Mixed Integer Programs Using Neural Networks”

“Mean Reversion with a Variance Threshold”

“Mean-Reverting Portfolios Tradeoffs between Sparsity and Volatility”

“An Intelligent Model for Pairs Trading Using Genetic Algorithms”

“An Advanced Optimization Approach for Long-Short Pairs Trading Strategy Based on Correlation Coefficients and Bollinger Bands”

“Discovery of multi-spread portfolio strategies for weakly-cointegrated instruments using boosting-based optimization”

“Evolutionary multi-objective optimization for multivariate pairs trading”

“Multi-asset pair-trading strategy: A statistical learning approach”

“M of a kind: A Multivariate Approach at Pairs Trading”

“Statistical arbitrage in the US equities market”

“OPTIMAL PORTFOLIO DESIGN FOR STATISTICAL ARBITRAGE IN FINANCE”

“Multivariate Pair Trading by Volatility & Model Adaption Trade-of”

Greedy Method for SMRPs

Quant Arb — Mon, 20 Mar 2023 22:33:36 GMT

Introduction

Truncation Method for SMRPs

Quant Arb — Thu, 16 Mar 2023 02:40:10 GMT

Introduction

We will be building off the previous article on non-sparse synthetic portfolios and look at the truncation method for generating sparse mean reverting portfolios. This is part of a 3 article series where we start with a non-sparse method and then climb into some heuristic approaches to sparsity. Our first heuristic approach is the truncation method. This is quite an easy method to understand so this should be a comfortable introduction; I’ll also be including code so that readers can follow along themselves.

As with before, we will be using a pre-defined set of stocks in the S&P500 that are all in the same industry. Keeping the data the same should help for easier comparison.

Data

Since the data is the exact same data as the previous article, with the exact same train and test periods, there is no need to explain it again. Readers are free to reference back to that for more information on how the data is organised.

Methodology

The Truncation method is probably the simplest of all. To start, we calculate the non-sparse solution with all the tickers as we did in the previous article. Then we normalise our weights (for the minimum eigenvalue weights) by the price of each asset. Take the absolute value, rank them largest to smallest, adjust them so that they total to 1 (thus, our weights are the % contribution to the notional volume needed to trade this portfolio), then select our top x tickers by contribution.

We can then retrain on this new, truncated, set of assets or we can skip this step and leave the weights unchanged (using the real weights of course, not the adjusted & absoluted wieghts as those only represent each asset’s contribution).

For our example, we will take the top 5 assets by contribution. We end up only discarding about 5-10% of the notional volume, but remove more than half the assets. This lets us achieve a relatively sparse portfolio, and we will see that the portfolio that wasn’t retrained is stronger OOS. Less is more in many cases, and reducing the amount of fitting we perform is a great way to ensure robust portfolios. When the discarded % is this low we are able to do this, but only because these components are noise that have no effect on our OOS portfolio.

The simplicity of this method makes it easy to understand / implement. Our next article will cover the greedy method which is a little bit more work, but is a far better method.

Truncation is super fast to compute, making it the fastest way to build sparse mean-reverting portfolios. This is ideal for working with large datasets, but otherwise greedy is superior, something we will show in the next article for this series.

Implementation (CODE)

I am starting off from the same place as before (reference previous article) with the exact same data, exact same weights, and exact same non-sparse portfolio.

eigvals, wgts = portmanteau_gep(df_train, 10)

This is the same method we coded up last article, and when we plotted our minimum eigenvalue portfolio we returned this:

We start by creating our adjustment factor, we then use this to form our contribution list as explained in the methodology.

The reason we multiply by adjustment factor divided by itself (basically all ones) is to take our weights from a numpy array with no tickers attached to a pandas array where the index is our ticker names. They are aligned in the same order so this is fine. Doing this lets us select the unadjusted weights by ticker which is easier. My approach is a very dirty way to do it, but hey it works. I also have infrequently used this model in my research due to the superiority of the greedy method (hence the dirty code), but insights about how non-sparse methods can overfit weights are revealed a lot more intuitively in our results so well worth learning.

Finally, normalize the contribution list so that all values total 1 (thus representing the % of notional volume each ticker contributes).

# Take mean of each asset, then divide by average price to normalize values.
adjustment_factor = df_train.mean() / df_train.mean().mean()

# Divide our weights by our adjustment factor, absolute them, then sort.
contribution_list = abs(wgts[0] / adjustment_factor)
contribution_list = contribution_list.sort_values(ascending=False);

# putting our weights into pandas with tickers attached (dirty af)
ticker_weights = wgts[0] * (adjustment_factor / adjustment_factor)

# Normalizing so all total to 1
contribution_list /= contribution_list.sum()

Below is our contribution list, we take the top 5, which means we are only losing about 10% of the notional volume, contributions which are effectively noise.

sparse_tickers = contribution_list[:5].index.tolist()

sparse_weights = ticker_weights.loc[sparse_tickers].values.tolist()

We take the top 5 tickers to get our sparse tickers, then we use the ticker weights (our actual weights which haven’t been changed short of the redundant transform - multiplying by 1 - we did to add tickers as the index).

The above list is our final set of tickers, our sparse tickers. Plotting the portfolio (using the original wieghts) we get the below results out of sample (we have been using df_train so far, this chart is df_test).

Our above porfolio is actually more robust than the prior portfolio because we have removed the useless assets. As shown below when we take the remaining assets and use them instead, they create a drift term which actually hurts our results:

Some mean reversion, but mostly drift. This is likely why our sparse portfolio appears much more stationary out of sample than our non-sparse portfolio.

Looking at our retrained portfolio we quickly re-run the code, as shown below:

eigvals, wgts = portmanteau_gep(df_train[sparse_tickers], 10)

and then we plot our new portfolio and achieve this porfolio out of sample:

The portfolio in-sample:

We only have one example here, and the jury is out on whether retraining is a good idea, it probably is in my view, but we ran into the same issue that we previously were trying to avoid where a large majority of the weights are basically noise because the non-sparse method finds a weight for every asset in the portfolio, overfitting some when they should just be 0. This results in a few being very small weights which barely contribute to the portfolio, but help overfit towards our objective of minimising portmanteau (only in-sample, out of sample this overfitting comes to life and they actually hurt more than they help as we saw).

Conclusion

We looked at a very simple method for making portfolios more robust and reducing turnover. We also showed the ugly side of non-sparse portfolios where the majority of the weights are overfit noise which do not contribute significantly to the final portfolio. This is a fantastic example of how quick and dirty methods can vastly improve performance by avoiding unneccessary fitting.

Non-Sparse Synthetic Portfolios

Quant Arb — Wed, 08 Mar 2023 03:21:12 GMT

Introduction

Building on previous articles in the area of synthetic portfolios, we will explore a convex method for building synthetic portfolios without introducing constraints like sparsity. In future research, we will look at 2 heuristic approaches that allow us to build on these methods and introduce sparsity constraints. In this article, we once again look at the portmanteau criterion as our chosen method for generating non-sparse synthetic portfolios. I won’t dive too much into the theory here since you can read up on that yourself (I’ll butcher it anyways), but I will walk through a code implementation of this. We do not produce any backtests or strategies here, but we do present a tool every quant should have.

Data

We would have a pretty hard time trying to form a robust portfolio using a large number of portfolio constituents. There is a much larger probability of overfitting, and in other cases, this can be impractical for computing reasons. The GEP (generalized eigenvalue problem) framework used to solve this optimization is highly compute efficient but the downside is that it cannot produce a sparse output.

Our universe consists of S&P500 stocks in the form of daily close prices for each asset. We ensure that our universe is part of the S&P500 for 3 main reasons:

Greater ability to trade portfolios in size due to better book depth.
We don’t need short-margin availability datasets as margin is probably available.
Fake mean-reversion from the bid/ask bounce is less pronounced because of tighter spreads.

Starting with the first point, we need to ensure that our portfolios will be able to absorb some decent size without creating a large impact. This is important for many traders as something you can only put $10,000 behind but makes 60% a year is probably not even worth coding up despite the great APR.

The next two concerns are designed to avoid situations where our backtests aren’t realistic for live trading. This is either from assets being unavailable to short or from mistaking the price bouncing between the bid and ask as actual mean reversion. Most bars use the last trade to form the close. If this trade is a buy and in the next candle it is a sell we will see the price change by the size of the bid/ask spread. The actual fair value could have remained unchanged during this period.

To form our portfolios we should select the optimal assets ahead of time. Otherwise, we will not be likely to generate a robust portfolio. In our case, we use the below list of tickers in the electrical utility category. This is an area that can be expanded upon greatly by looking at other categories/themes or even by applying clustering algorithms to form your portfolios.

Minimizing Portmanteau Statistic as a GEP

Formulating our optimization as a generalized eigenvalue problem (GEP), we can greatly reduce the computational complexity and quickly find an optimal solution. If this sounds a bit too complicated you’ll be happy to hear we are basically using PCA.

For background reading, this is one of the best tutorials on the topic:

https://arxiv.org/pdf/1903.11240.pdf

We will be using the portmanteau statistic (more info in the link below) which we have covered in previous articles. This tests for white noise using autocorrelation. As a result of this, you have to specify the number of lags, in our case, we will use 10, but it is generally recommended to go lower for mean-reversion (what we optimize for here), and higher for momentum.

https://en.wikipedia.org/wiki/Ljung%E2%80%93Box_test

We will need to solve the problem:

Modify w (our matrix of weights for each asset) to minimize the portmanteau statistic with lag p. M represents the autocovariance matrix of lag i.

We know p ahead of time and it is a parameter we define ourselves, this is the lag parameter that was mentioned before.

Implementation (Optimization)

The first function we need to define is one that produces our autocovariance matrix. This is done below:

def autocov_matrix_calc(arr, p):
    m = arr.shape[0]
    arr_demeaned = arr - np.nanmean(arr, axis=0)
    return 1 / (m - p - 1) * arr_demeaned[p:].T @ arr_demeaned[:m - p];

Next, we rearrange our optimization problem to produce a matrix we can find the eigenvalues/vectors of:

import pandas as pd;
import numpy as np;
from scipy.linalg import sqrtm;

def portmanteau_gep(df, lags):    
    df_normalized = df - df.mean(0);
    rho = df_normalized.cov().values
    rho_inv_sqrt = np.linalg.inv(sqrtm(rho))
    
    pmt_matrix = 0;
    for i in range(1, lags):
        autocov = autocov_matrix_calc(df.values, i)
        pmt_matrix += np.square(rho_inv_sqrt @ autocov @ rho_inv_sqrt)
    pmt_matrix /= lags;

    eigenvalues, eigenvectors = np.linalg.eig(pmt_matrix)
    asc = np.argsort(eigenvalues)
    eigenvalues, eigenvectors = eigenvalues[asc], eigenvectors[:, asc]

    wgts = rho_inv_sqrt @ eigenvectors

    return np.real(eigenvalues), np.real(wgts)

The above code goes through the entire optimization logic and is all that you need to test out this model for yourself. We start by de-meaning our data. Then, we take the covariance matrix denoted as rho. After taking the inverse square root of this we can now use it to build our matrix.

This matrix is denoted as “pmt_matrix” and is produced by dot multiplying the inverse square root of our covariance matrix (twice) against the autocovariance matrix and then squaring it. We do this for each lag below our lag parameter.

After that, we divide by the number of lags so that it is an average, not a sum, and then our matrix is finished. The hard part is over now; we can simply take the eigenvalues and eigenvectors of our matrix and we are almost finished.

Finally, we sort our eigenvalues and eigenvectors. Our smallest eigenvalue's corresponding eigenvector can be dot multiplied with the inverse square root of our covariance matrix “rho_inv_sqrt” to give us the weights that minimize our portmanteau statistic. This is the weights for our mean-reverting portfolio. We can do the opposite of this to get a momentum portfolio, although, in my experience momentum portfolios are best formed using other methods and approaches not covered in this article.

Implementation (Generating Portfolios)

Applying this method using a lag parameter of 10 to our training dataset spanning 2017-01-01 to 2018-06-01 we produce this portfolio out of sample:

Our above portfolio exhibits high degrees of mean-reversion but has a hard time with stationarity. This metric is best used for short-term mean-reversion strategies so daily data isn’t the most optimal way to use this.

The (not shown) momentum portfolio (maximum eigenvalue) is quite mean-reverting since we used a short lag and portmanteau is best used for mean-reversion.

Conclusion

The main objective for this article was to present GEPs as a foundation for future articles. In the future, we will look at the truncation and greedy methods for adapting this method to sparse portfolios using heuristics that save on compute.

We’ve also managed to present a method for determining the optimal portfolio given a pre-selected group of assets. This is useful for strategies that work by finding groups of assets that are similar. We can use industry categories or more statistical approaches involving clustering or plain simple top z assets with the largest/smallest metric x. Hopefully, this article is useful for readers in search of a new tool they can use when developing strategies in this area.

Semi-Definite Programming For SMRPs

Quant Arb — Mon, 27 Feb 2023 03:19:20 GMT

Introduction

In the previous research article, we explored the use of Monte-Carlo Minimization (MCM) as a non-convex method for finding mean-reverting portfolios. Here we will use Semi-Definite Programming (SDP) to form mean-reverting portfolios with 2 major constraints. These are variance and sparsity.

SDP is a convex method which makes it a lot more robust compared to MCM. This means that it is far better suited to longer-term pairs trading portfolios as these prioritize forming robust portfolios whereas shorter-term portfolios are best built using models that can find more complex, but also shorter-lasting relationships.

We were able to produce a strategy with significant alpha in equities markets. Avid readers are welcome to solve the optimal spread trading problem in other ways and form their own strategies after building on this work.

Teasing results, we were able to generate significant alpha OOS using a robust methodology that makes overfitting far harder.

The Models

In addition to SDP, we will also be making use of Sparse Principal Component Analysis (SPCA) to enforce sparsity within our weight vector. Sparsity in this case means reducing the number of assets in our portfolio. We also want to add a minimum variance constraint to ensure that our portfolio is volatile enough to beat fees. To trade the spread we will use a set of Bollinger Bands. This is a very simple approach as our main focus is on portfolio generation methods.

The Data

For this analysis, we shall use data from 2017-01-01 to 2018-01-01 for our training dataset, and 2018-01-01 to 2018-06-01 for our OOS dataset. This is equities data which comes from 50 randomly sampled S&P500 constituents.

Pre-Processing

We start by taking our data and arranging it into an array of column vectors containing daily close prices for our assets. Below is what our training (top image) and testing (bottom image) datasets look like:

We then proceed to normalize our data. We first subtract the mean of each column, then we divide each column by the standard deviation of that column. We then get this Numpy array (for our training dataset)

VAR(1) Non-Sparse Optimization

We will give a quick example of how to optimize for a non-sparse portfolio using VAR(1) predictability as our optimization objective. There will also be no attempt to optimize for a minimum variance here.

We start by taking the autocovariance matrix of our normalized data using a lag of 0 and doing this again with a lag of 1. I’m sure some readers will have figured this out, but using a lag of 0 is just a normal covariance matrix.

We are effectively solving for the matrix x which minimizes our variance ratio, this is expressed as the below problem:

We know that minimizing this variance ratio is the same as finding the minimum generalized eigenvalue 𝜆 in the equation:

The portfolio that thus minimizes the variance ratio 𝑣(𝑥) is given by:

Where 𝑧 is the eigenvector that corresponds to the minimum eigenvalue of the matrix denoted:

Rho is our covariance matrix, and A is the coefficients of our vector autoregressive model.

Expressing this as code we perform the below series of operations. cov_matrix is our covariance matrix (autocovariance matrix with lag 0) and autocov_matrix is our autocovariance matrix with lag 1.

We then take the eigenvalues of this matrix and then grab the eigenvector (our portfolio weights) corresponding to the minimum eigenvalue.

Our eigenvalues look like this:

Taking the dot product of our previously mentioned normalized training dataset with our minimum_eigen variable (the weights) we get the portfolio below:

Semi-Definite Programming & SPCA

Now that we have generated our non-sparse portfolio without minimum variance constraints we will now create a portfolio that is both sparse and meets minimum variance constraints using Semi-Definite Programming and SPCA.

Giving a glimpse into our optimization problem:

This is all done using CVXPY which is quite an easy task so I won’t walk through this part too much. We use a matrix norm penalty which is equivalent to the alpha parameter of our SPCA to induce sparsity.

We get the target selection of eigenvectors given below from solving with SDP:

After we apply SPCA we get:

This array (which is our portfolio weights) can be used to generate the below portfolio:

Applying these weights to our out of sample dataset we get:

Trading The Spread

We will be applying a simple 20-period moving average as our rolling mean, with bands that are +/- 2SD (computed as a 20-period rolling standard deviation). This is just Bollinger Bands for the degens out there. It is far from the optimal way to do it, but trading the spread is not the focus of this article.

Doing this, our in-sample and OOS portfolios look like this:

Results

Trading these thresholds by getting short the spread if it is above the upper band, then exiting at the mean, AND getting long the spread if it is below the lower band, then exiting at the mean results in the below performance:

Conclusion

We were able to show that there is significant alpha in pairs trading still, but this does not come from outdated methods static thresholds, and ADF. We can further improve this performance by creating additional optimization metrics which can improve the volatility, trading costs, and mean-reversion characteristics of our portfolio.

Eager users may also want to explore my other articles on portfolio generation for pairs trading and may attempt to build on this using methods like pattern matching and stochastic control for more optimal pairs trading (these methods are only optimal for higher resolution data, they are too much / will suck for a longer-term portfolio like the one shown).

If you want further reading on this topic I’ve added some papers below.

Papers

BOX, G. E., and G. C. TIAO. “A Canonical Analysis of Multiple Time Series.” Biometrika, vol. 64, no. 2, 1977, pp. 355–365., https://doi.org/10.1093/biomet/64.2.355.

D'Aspremont, Alexandre. “Identifying Small Mean-Reverting Portfolios.” Quantitative Finance, vol. 11, no. 3, 2011, pp. 351–364., https://doi.org/10.1080/14697688.2010.481634.

Deadman, Edvin, et al. “Blocked Schur Algorithms for Computing the Matrix Square Root.” Applied Parallel and Scientific Computing, 2013, pp. 171–182., https://doi.org/10.1007/978-3-642-36803-5_12.

Fogarasi, Norbert, and Janos Levendovszky. “Sparse, Mean Reverting Portfolio Selection Using Simulated Annealing.” Algorithmic Finance, vol. 2, no. 3-4, 2013, pp. 197–211., https://doi.org/10.3233/af-13026.

Göncü, Ahmet, and Erdinç Akyıldırım. “Statistical Arbitrage with Pairs Trading.” International Review of Finance, vol. 16, no. 2, 2016, pp. 307–319., https://doi.org/10.1111/irfi.12074.

Monte-Carlo Minimization for Synthetic Portfolios

Quant Arb — Fri, 10 Feb 2023 07:19:41 GMT

Introduction

Monte-Carlo Minimization (MCM), also known as Basinhopping is a popular non-convex optimization algorithm for global optimization. It is derived from the chemical physics literature and is accompanied by Dual-Annealing as its primary alternative.

Today we will apply it to digital asset markets to form synthetic mean-reverting portfolios by optimizing for multiple different mean-reversion tests. This article merely is meant to provide a methodology for generating these portfolios and we will not explore the optimal trading problem needed to form a full strategy. There are also many backtesting caveats that are beyond the scope of this article that would need to be explored separately first.

Data

We use these below assets in our analysis, but this is entirely an arbitrary selection. This research blog will likely explore universe selection methods in future articles. Research relating to optimization and mean-reverting portfolios will be explored in a mix of both free and paid articles, alternating each time.

The table below gives an overview of the dates used. There hasn’t been much of a significant change in the effectiveness of this technique, so we can use older data. This was also influenced by this dataset being the first one I found on my computer.

These are close prices arranged as a matrix which is usually required for such optimization problems, especially the more complicated convex methods we will explore in later articles.

Normalization

In the interest of avoiding as much look-ahead bias as possible, we standardize our training and test sets individually.

This is the below-standardized training set:

For clarity, we are using the below formula where x is our asset close prices, mu is the mean of our prices, and sigma is the standard deviation of our prices. For shorter-term synthetic portfolios standardization is ideal, but for longer-term portfolios, it can make our portfolios less robust as volatility changes over time.

We also show our testing dataset that has been standardized here.

Objective Function

We will be optimizing for mean-reversion using the portmanteau statistic which is used to test for the presence of white noise in time series.

It is formulated below:

As some readers may notice, we are using the 1978 Box-Leung formulation of the statistic and not the original 1970 Box-Pierce formulation shown below:

For further improvements, the additional options of the Hosking portmanteau test formulation (1980) or the Li and Mcleod formulation (1981) could be used.

Hosking:

Li and Mcleod:

The final two I suggested are not appropriate for this use case because of their increased complexity. One of the primary reasons that the Portmanteau statistic was chosen was because it is efficient to compute when compared to statistics like Augmented Dickey-Fuller. Despite its computational efficiency, it is still vastly superior to ADF when used for lower timeframes.

Portfolio Optimization

We use the SciPy Optimize library which includes both Monte-Carlo Minimization (named as Basinhopping in the library) and Dual-Annealing. There is already some literature on the use of Simulated Annealing for the formation of mean-reverting portfolios, but since its release, Dual-Annealing has proven a much more effective and popular solution. Overall, it is regarded that Monte-Carlo Minimization is more effective than Dual-Annealing in chemical physics, but it is yet to be determined whether this is the case for financial markets.

Using the library with custom-defined optimization metrics we get the following results:

We then generate this mean-reverting portfolio. The training set is shown in blue with the out-of-sample data shown in red.

Taking another arbitrary data window, but here halving the size of the train and test sets respectively we were able to achieve similar mean-reversion.

Final Remarks

We find in our analysis that whilst our portfolios do tend to decay out of sample they remain highly mean-reverting. This is a trend that was found consistently with other non-convex optimization approaches I have used before to do this. This effect typically only occurs for shorter timeframes. Usually, you get a total decay when you do not use ultra-robust methods for creating mean-reverting portfolios over longer timeframes. This method is likely one of if not the least robust of them all.

It is however a novel application of a very underrated algorithm to financial markets and we were able to generate highly mean-reverting portfolios both in and out of sample despite seeing some decay. The dynamics of short-term pairs trading are very different from longer-term pairs trading portfolios that the literature focuses on, and for shorter-term pairs trading portfolios stationarity / robustness is less of a concern as we quickly generate new portfolios, capturing temporally local effects.

It is important to note that as a result of using assets that are not top 15 liquid coins, there will be lower capacity in this strategy. We also have to trade many assets which further increases the costs of trading large sizes. This is likely only appropriate for smaller capital bases. The volatility and increased mean-reversion exhibited by medium-sized coins are ideal for our research. You may also maximize the objective to try and form index portfolios for different areas such as DeFi tokens etc, this is best done with PCA-based methods which we will explore in one of the next articles!

Eager readers may want to continue experimenting with this research topic and finding ways to trade these mean-reverting portfolios optimally and cost-effectively.

If you enjoyed reading this, then feel free to subscribe. The next article will be paid as I alternate between a paid and unpaid article each time.

The Quant Stack: Pairs Wiki

Pairs Trading Papers - Review

Introduction

Copulas

Optimization Based

Bivariate

Multivariate:

Lead-Lag (DTW)

OU / Stochastic Control

Regime Shifts

Metrics

Cointegration

Math Reading

HFT Pairs

Clustering and Pre-Selection

Misc.

Notes from me

Thinking about stationarity the right way

Introduction

A Real Pairs Trading Strategy

Introduction

General Strategy Overview

Universe Selection (1)

Optimization Objectives (2a)

Greedy Method for SMRPs (PART 2)

Introduction

Pairs Trading Framework & Process

Introduction

Index

Defining Pairs Trading

Pairs Trading as an Optimization Problem

Robust or Non-Robust - A Temporal Question

The Parameters That Matter

How Data Parameters Drive Modelling Choices

Short-Term High-Turnover Pairs Trading

Long-Term Low-Turnover Pairs Trading

The Flavors of Mean Reversion

Optimization Metrics (Mean-Reversion)

Optimization Metrics (Alternative Objectives)

Methods For Optimization

Non-Optimization Focused Approaches

Additional Areas of Study

Conclusion

Literature

Greedy Method for SMRPs

Introduction

Truncation Method for SMRPs

Introduction

Data

Methodology

Implementation (CODE)

Conclusion

Non-Sparse Synthetic Portfolios

Introduction

Data

Minimizing Portmanteau Statistic as a GEP

Implementation (Optimization)

Implementation (Generating Portfolios)

Conclusion

Semi-Definite Programming For SMRPs

Introduction

The Models

The Data

Pre-Processing

VAR(1) Non-Sparse Optimization

Semi-Definite Programming & SPCA

Trading The Spread

Results

Conclusion

Papers

Monte-Carlo Minimization for Synthetic Portfolios

Introduction

Data

Normalization

Objective Function

Portfolio Optimization

Final Remarks

Further Reading