Over my career I have held the title of Head of Quantitative Research at various hedge funds and proprietary trading firms, overseeing the trading of significant amounts of capital and the development of advanced research pipelines. Coming into a firm and dictating research direction is quite the task, and one that has given me many lessons which I thought I would share today.

This is not as niche a role as it might sound, as one of it’s core tasks is to build and maintain a successful research pipeline which is broadly applicable in many contexts. You may not be managing tens of people, but if you are looking to make money in quantitative trading, with the exception of a few rare latency- or developer-driven strategies, you will need a successful research operation within the firm. Even within latency teams there will be research pipelines around latency optimisation. Almost every firm and every desk has to approach the task of developing these capabilities in its beginnings, and maintain them as it grows.

What follows are the four things I have found matter most: the people, the tooling, the data, and the relentless sense of urgency that ties them all together. I have also included some code examples for the eager reader.

The Personal Component

The people on your team are your greatest resource, which is exactly why this is the first section. The difference between a team of A players and a team of B players is the difference between making and losing money as a firm. I have only seen B player teams succeed at uncompetitive trades (carry, momentum, etc), and in the ultra-competitive world of market making, etc I have never seen a successful team that I wouldn’t deem to be almost all A players.

The first complication is remote work. Some people work very well remotely, but most do not. It is mostly senior, experienced researchers and developers who do well in a remote context — not juniors. Juniors typically need more guidance, management, and mentoring, which usually means it is vastly preferable to have them working in person. I have found remote work to be a barrier even when the person clearly works very hard and puts in long hours, simply because they require a lot of guidance on tasks that would be much better achieved with them sitting nearby in the office. New teams, and large teams also suffer extra from remote work, when there is an intense build out associated with the creation of a new pod or desk then it is critical to have tight communication loops. In the past, I have not found remote work to be a successful for large teams or for teams where heavy communication is required (a very intense buildout is occurring, made especially worse if it involves a large team doing said buildout).

That said, if you strictly mandate in-office hiring you will miss out on a lot of great talent. When an exceptional researcher comes across my desk, for me it is usually every 9–12 months, and always through my own network. It is almost always a remote hire (though this is specific to my network having a lot of remote people; if you know more people, this will happen more often). You want to take advantage of these people when they become available, because it is very hard to find A players precisely when you want to hire them. The best talent becomes available when it happens, not when it is convenient for you. It is wise to keep an open hiring policy for exactly these moments. The best talent does not come from a job posting on LinkedIn! It is my view that you need an exceptionally well run team (often tight knit and with everyone independently motivated towards the mission) to do it fully remote and for in person teams it is harder to get the talent but easier to manage the talent and ensure productivity.

Another point I have found is that for cases when you are specifically hiring for a role you do not have the capabilities for in your own team, then raw intelligence is not enough on its own; you need people with the specific knowledge the problem requires. If your problem is options execution, you need to find someone who has already learnt options execution at an established player. It is much harder to hire someone smart without that experience and have them figure out how to do options execution from scratch. There is a huge amount of information that lives in the minds of those who have worked at certain firms, and the only way to get access to it is to have worked there yourself or to hire someone who has. This is why firms like to hire ex-Citadel / JS / SIG / Jump people. It is probably true that the idea of some magic Citadel alpha that every Citadel researcher knows is a falsehood - but generally, if you want to do well, you need to hire people who have learnt how to do the task at a successful operation.

Build the Tooling First

Everyone wants to sit down and immediately start hunting for alpha, but you need to start with the tooling and research infrastructure before any meaningful analysis can occur.

This is a double-edged sword, and it should not be confused with the premature development of infrastructure. Many of the most successful trading operations I know built initially slow and messy systems, only to refactor them into well-polished (and slow-to-develop) mature systems later, after they had already generated significant PnL. This even happened on my own desk at a firm I was at. The early mistake is the opposite one: failing to develop any proper research infrastructure. Realistically it only takes a couple of weeks for a very motivated team to build great tooling to regularly scrape, analyse, and manipulate data. Anything longer and you are overcomplicating it. An easy-to-use, effective research pipeline is extremely important to the success of any team.

Some core rules, the things you should never let yourself do:

• Never scrape or pre-process data you already knew you’d need. All data should be automatically scraped and pre-processed as part of a daily cron job. Manually pulling data you could have anticipated is a huge time killer.

• Never write a backtester per alpha. Your testing tooling should be standardised, optimised, and pulled from a library. If your backtester can’t handle a particular type of alpha logic, upgrade the backtester — do not spin up a new one for each alpha. That will massively slow down your research process.

• Pre-merge data that is often merged. If, say, you use the top 50 cryptocurrencies by market capitalisation and want their 1d OHLCV bars, you should not be loading many separate files and writing a for-loop with logic that filters a rolling top-50 list and scrapes the dates. That is slow. Have them already merged into one file. If you find yourself merging a particular set of data often (e.g. all top-50 coins, 1h data), create that merged file and generate it as a cron job.

• Never load data outside your tooling. You should have a fast data loader where you specify the symbol, date range, data type, and provider, and it loads. Behind the scenes that is a filter finding the relevant files, looping through them, loading, and concatenating — but you should not be rewriting that logic every time. Better yet, optimise it: use pd.read_csv with engine='pyarrow' (this is not the default) for CSV files, and use Polars for Parquet files, which is fastest even once you account for converting back to pandas. Polars and pandas have similar merge times, so either is fine for the merge.

• Create notebook templates. If you follow a fixed set of steps when testing an alpha, build a dummy template notebook so that all you have to do is define the alpha and its input datasets and hit run. Then you only need to copy one notebook instead of pulling 20 cells from various notebooks.

Once you have a setup where you write as little code as possible and get results in your hands as fast as possible, you have a great research pipeline. Iteration speed is everything. If you can write the code faster - because more has been abstracted away , and it runs faster, because the code is optimised and the data is pre-generated, then you can test more ideas. If you can do 10x as much work as the next person, you win. That is the real secret to a successful research operation: find ways to do more with the same time.

The Data

Great data does three things: it exists, it is correct, and it is easy to work with.

It exists. This is the most important of the three. If you didn’t bother to collect the data in the first place, then either you cannot complete the analysis at all, or you have to reconstruct it from historical sources that may be less accurate — and you will waste a lot of time doing it. Imagine you are running an arbitrage book and need to calculate historical arbitrages. It is fairly expensive to compute many combinations at the tick level, so now you have to scrape and pre-process Tardis data, write out the arbitrage logic, generate the data, probably wait half the day (if not longer) for it to finish, and only then do you have your arbitrages. If instead you had simply been tracking every arbitrage all along, you could have filtered them straight out of a dataset. If you don’t yet have that dataset but know you are likely to need it often, you can pre-generate it historically — but the best data to collect is the easiest, which is usually real-time data.

It is correct. Write tests for your data so that when they are violated you investigate, note an error, or manually bypass it. Think of it like a unit test: you should always be checking for anomalies. Market cap dropping 90% in an hour and then recovering shortly after, for example, is very strange. Sudden changes in price are alarming, and should be extra alarming when they come back in an extreme timeline. If you are using alternative datasets, you often need to apply a delay if the data is not truly point-in-time, ideally it is PIT, but not always, and timestamps may be left-labelled rather than right-labelled. Clean up your data before you work with it, and ideally have that done before you ever need to use it.

It is easy to work with. At one firm I was at, we had been lazy with some of our data collection and dumped it into DynamoDB instead of a proper time-series database. As we scaled up, it grew into the terabytes, and, the $30,000-a-month AWS bill aside, it was completely unusable, because DynamoDB makes you scan the entire database to work with it. That is an extreme example, and we did eventually fix the infrastructure, but the lesson stands: store your data properly and apply your pre-processing up front. If you are storing all arbitrage opportunities, bucket them at the file level so that specific dates or tickers can be read in isolation without loading extras. And if, for example, you only trade opportunities over 20% APR, keep a separate >20% APR database too, it will be much smaller, and since that is a common dataset to load, it saves you from re-running an expensive filter. Running expensive filters twice is not ideal; try to avoid it.

Delays Kill

There is a mile of difference between the pod that stays up until 2am until the issue is fixed, and the pod that will get to it the next morning when it’s convenient. This isn’t a difference of work ethic, it’s a difference of urgency. Work ethic matters, but the pods that do well have an extreme sense of urgency about everything they do, and it makes them work extremely efficiently. There is no option to spend two weeks on a piece of analysis; it needs to be done today, and so they find a way to get roughly the same result for 10% of the effort. The 80/20 rule, in practice.

Allowing any delays into the research pipeline kills that sense of urgency, and with it all the productivity gains that come along with it. This is partly cultural and will depend heavily on who you hire, but it also comes down to your pipeline.

That is what much of this article has really been about. Tooling should make things easy, and it should be fast. Waiting on slow code is, in many cases, a fixable problem. Now that we have LLMs, writing your tooling out in Rust is often far quicker than waiting on slow Python. For example, I took one of my forecasting models from 8–12 hours down to 15–20 minutes by rewriting it in optimised Rust, and it took me no more than an hour to do using Claude. There is always a way to make code run faster, and the days when optimising would take longer than the time it saved are gone, you should be taking advantage of that (to the extent that it makes sense, don’t optimise a 20 second task).

Get these four right: the people, the tooling, the data, and the urgency that connects them, and you have the foundations of a research operation that actually makes money.

Code Examples

A standard data loader for Tardis data which assumes you pre-process raw trades into trade bars at various frequencies (parquet) and save the raw files in their native .csv.gz format that Tardis uses. It loads parquet with Polars (fastest), and .csv.gz with Pandas Pyarrow (fastest):

from datetime import datetime
from pathlib import Path
from typing import Literal

import pandas as pd
import polars as pl

ROOT_DIR = "..."

OutputFrame = Literal["pandas", "polars"]
MissingPolicy = Literal["skip", "raise"]


def load_tardis_data(
    start_date: str | datetime,
    end_date: str | datetime,
    *,
    market: str,
    instrument: str,
    kind: str = "trades",
    freq: str | None = "klines_1min",
    root: str = ROOT_DIR,
    fields: list[str] | None = None,
    missing: MissingPolicy = "raise",
    frame: OutputFrame = "pandas",
) -> pd.DataFrame | pl.DataFrame:
    """Load local Tardis data for a date range.

    ``freq=None`` loads unprocessed compressed CSV files with pandas/PyArrow.
    Any concrete frequency loads processed parquet with a lazy Polars scan.
    """
    storage_key = "unprocessed" if freq is None else freq
    paths = [
        Path(get_data_path(day, kind, storage_key, market, instrument, base_dir=root, vendor="tardis"))
        for day in pd.date_range(start_date, end_date, freq="1D")
    ]
    present = [path for path in paths if path.exists()]

    if missing == "raise" and len(present) != len(paths):
        raise FileNotFoundError(next(str(path) for path in paths if not path.exists()))

    if freq is None:
        parts = [pd.read_csv(path, engine="pyarrow", usecols=fields) for path in present]
        data = pd.concat(parts, ignore_index=True) if parts else pd.DataFrame(columns=fields)
        return pl.from_pandas(data) if frame == "polars" else data

    if not present:
        return pl.DataFrame(schema=fields) if frame == "polars" else pd.DataFrame(columns=fields)

    scan = pl.scan_parquet([str(path) for path in present])
    data = scan.select(fields).collect() if fields else scan.collect()
    return data if frame == "polars" else data.to_pandas()

And then you should have some sort of daily scraping script like this:

from datetime import date, datetime, timedelta
from pathlib import Path
from typing import Iterable

import pandas as pd

StorePath = str | Path

def previous_utc_day(now: datetime | None = None) -> date:
    """Return the most recent complete UTC day."""
    clock = now or datetime.utcnow()
    return (clock.date() - timedelta(days=1))


def active_binance_perps(day: date) -> list[str]:
    """Return Binance Futures perpetual contracts available on the requested day."""
    catalog = get_tardis_exchange_symbols("binance-futures")
    live = catalog[
        (catalog["type"] == "perpetual")
        & (catalog["availableSince"].dt.date <= day)
        & (catalog["availableTo"].dt.date > day)
    ]
    symbols = live.index.str.upper().sort_values().to_list()
    return symbols


def fetch_binance_futures_day(
    day: date,
    destination: StorePath,
    symbols: Iterable[str],
    datasets: Iterable[str] = ("trades", "quotes", "derivative_ticker", "liquidations"),
) -> None:
    """Download one Binance Futures Tardis day and write raw plus derived files."""
    stamp = day.isoformat()
    for symbol in symbols:
        for dataset in datasets:
            run_tardis_download(
                start=stamp,
                end=stamp,
                base_dir=str(destination),
                symbol=symbol,
                data_type=dataset,
                exchange="binance-futures",
                resample_label="right",
            )


def main() -> None:
    """Run a minimal daily Binance Futures Tardis ingestion."""
    run_day = previous_utc_day()
    root = Path("/mnt/hdd-storage/data")
    universe = active_binance_perps(run_day)
    fetch_binance_futures_day(run_day, root, universe)


if __name__ == "__main__":
    main()

It can feel like a chore to make sure all your datasets scrape regularly, get pre-processed, are maintained in an organised collection with lots of accessibility via tooling, but it more than pays off in productivity terms. When you need the data, it will feel like a breeze. This is only a basic set of examples, and you will need to write a lot more than this, but its a start.

Authors Note

For the first time I have decided to let Claude do a bit of a refactor (on a fully human written draft) on some of the phrasing because I felt it was very poorly structured. I think it retains most of its original character, and provides a more enjoyable reading experience. I have only allowed minimal edits from my entirely hand written draft, but if anyone thinks its significantly worse please drop me a message.

Introduction

A recent paper called “Learning To Rank” has gained quite a lot of attention, and rightfully so, it addresses a really interesting problem for statistical arbitrage managers - how to get your forecast to align with what you are actually trading. In reality, we are not purely concerned with mean squared error but instead with the relative performance of assets. So, to fight this, the paper proposes that we should be learning to rank assets and not learning to predict assets. This makes sense for statistical arbitrage managers as you will usually hold a portfolio with some assets long and some assets short. You care mostly about how they perform relative to one another, and not their absolute performance (both assets could go up 1000s of % but I only make money on the difference).

To tackle this, and get us closer to predicting the difference, the paper suggests that we implement one of 3 different options. It presents 3 LTR models:

• LambdaMART

• ListNet (Neural Network)

• ListMLE (Neural Network

The issue is that NONE OF THEM WORK as they have described it. BUT I will explain how to fix this performance issue and why the version described in the paper underperforms the benchmark, but our improved version does significantly better than the benchmark.

Selecting A Model

The paper presents 3 options for models. Two are based on neural networks and one is a decision tree. From my testing it was immediately very clear that the best performing model was the decision tree. This is fairly expected and usually ends up being the case in the world of quant, especially when you are not working with high-frequency data. Thus, we go with LambdaMART.

LambdaMART gets its name from two ideas, LambdaRank - which is a smart way of computing gradients that directly optimize ranking metrics like NDCG, and MART (Multiple Additive Regression Trees) - gradient boosted decision trees.

LambdaMART works by sequentially building an ensemble of decision trees that optimise for a ranking objective. This does not directly predict returns, and instead focuses on learning a scoring function which orders assets from most attractive to least attractive - which in our case is the rank of forward returns. During the training process, LambdaRank computes gradients which encourage incorrectly ordered pairs of assets to swap places, with larger emphasis placed on swaps that improve ranking metrics (NDCG for example).

Why LambdaMART Fails

In many real world scenarios that use LTR models, this is an excellent objective. Unfortunately, this grossly misaligns with what actually generates PnL in a statistical arbitrage managers context.

Read more

In this article, we will extend the work of the first article (below) and develop a full feature set for our forecasting model. This will be fairly basic, and will not be the final feature set. We will do some selection in the next article, develop a factor model, and then use that factor model in article 4 to develop advanced features and build out the final forecasting set. We were able to find many features that scored over 5 Sharpe pre-fees, and now have our set of features for article 3.

This will be a fairly short article where we simply test our alpha candidates and select the best, but it will provide us with our feature set to be used in the next few articles which we will then use to finally develop our forecast.

Index

1. Introduction

2. Index

3. Alphas

4. Analysis

5. Conclusions

Read more

In this new 5-part series of articles, I will be developing real HFT alphas using the research pipeline discussed in the previous article on HFT Alpha Research. In the first article (today’s article), we will present a basic set of features which explain a large portion of the overall variance. In the next article, we will expand this selection of features and use some more advanced features engineered from full depth orderbook data. In the 3rd article, we will expand our universe from BTC, ETH & SOL, to the top 30 assets by rolling 90d volume rank, and develop a cross-sectional factor model for the HFT timeframe. In the 4th article, we will use this factor model to test additional features, and we will also perform feature selection to form our final feature set. In the final article, we will test various models for forecasting forward returns. We use an incredibly large dataset of 1 year of second level tick data to perform this analysis and provide code to replicate the entire process at home. The final forecast aims to be worthy of being considered a starting point for market making and development. Moreso, we aim to show readers how to research a set of alphas, with the theory behind alphas explained in detail. If you have not already read the prior article on methodology please see the below article:

By the end of this article, we will have found 7 features for the 5s timeframe and 5 features for the 15s timeframe. See the lists at the end for the final feature set.

Index

1. Introduction

2. Index

3. The Data

4. The Alphas

5. Analysis

6. Conclusions

The Data

Our analysis focuses solely on USD-M Binance Futures for USDT pairs. This is the most liquid market for trading digital assets. We will use the range of data from 2025-01-01 to 2025-12-31. Our data is tick level and encompasses every orderbook update, and every trade that occurred over that duration. We use BTCUSDT, ETHUSDT, and SOLUSDT for this analysis. In the later articles we will expand the universe, but shrink the date range (to keep things computationally feasible - full depth data is massive)

Data is where it all starts, and having a high quality data provider is one of the most important things to consider when doing HFT analysis. It is typical of most HFT firms that data is collected internally using custom data scrapers to ensure that the data has representative latency statistics of their actual setup. In lieu of this, we will be using Tardis, which is a high quality institutional dataset for tick data in digital asset markets. We use the below endpoints:

1. trades

2. quotes

3. incremental_book_L2

We pre-process the trade and quote data into an OHLCV dataset, made of 5s bars, where we have volume data derived from trades, and open, high, low, close derived from quote mid-price. We also use quotes to get our top of book best bid/ask which will be one of the features in our article today. Please note that the quote dataset from Tardis is generated from the book deltas feed and not from the quote feed.

Then, we take our OHLCV dataset, and calculate the close to close mid-price returns. From here we shift these returns to create 5s forward returns. Additionally, we generate a disjoint 15s return which runs from t+5s to t+15s. This lets us separate effects on the 5s timeframe from the 15s timeframe without needing to use a markout analysis.

The Alphas

We will be testing the below alphas:

Read more

Introduction

I have found many proprietary HFT and MFT alphas in my time as a quant and have monetised them as part of different desks / books. In today’s article, I will do a walkthrough of how I personally approach it and explain how to find alpha in the HFT domain. We will walk through how to research short term alphas for either MFT execution or skewing into as part of an HFT strategy.

We will cover backtesting, factor modelling, residualization, correlation testing, horizon analysis, forecasting, monetisation, and more.

This is one of the core ways to drive profitability for any HFT desk and is a skill which can make or break the performance of a pod based on whether the managers know how to develop alphas or not.

Index

1. Introduction

2. Index

3. The Research Data Setup

4. HFT Factors

5. Backtesting Signals

6. Similarity Analysis

7. Horizon Analysis

8. Forecasting

9. Monetisation

The Research Data Setup

The first part we need to cover is our research data. We need to produce various lookahead targets to forecast. A reasonable setup is:

1s, 3s, 5s, 15s, 1min, 5min, 15min

Below 1s timeframes can also be forecasted, although timeframes like 100ms forecast often require low latency and the shorter you go the more you rely on breakdowns of the events that transpired as opposed to detailed statistical analysis of features performances.

Low latency events are often very deterministic in their signal and you can clearly assert why certain behaviours are happening with only a handful of samples. I’ve written about this before in one of my prior articles:

When constructing our return targets it is wise to always use right labelled, left closed aggregation. We should also use disjoint return intervals, there should be no overlap. For example if our timeframes were 1min, 5min, and 15min we would do:

t+0 to t+1

t+1 to t+5

t+5 to t+15

We would NOT do:

t+0 to t+1

t+0 to t+5

t+0 to t+15

This is because we want to be able to distinguish the effects and their timeframes. If I test a very strong alpha with roughly 1minute of strength it will show as working on 1min, 5min, and 15min if I do not use disjoint returns. However, if I test 5_15 as my interval then I will not see it, nor will I see it in 1_5 so I know exactly that it only works for 1min. Additionally, we should not use trade bars to get our returns. We need to use mid-price to calculate our return targets. This is because there will be significant bid/ask bounce in illiquid names which will make the trade bars have a much stronger reversal effect than actually exists.

HFT Factors

What is a factor model on the HFT timeframes, and why do we need it for our alpha research pipeline? The idea of risk premiums disappears when we enter the HFT world. All of our factors will be in the high single or even double digit Sharpe ranges pre-fees, and unmonetizable post-fees (at least on their own). We should not think of factors here as some sort of inherent risk premium, but instead as any other alphas. Simply alphas we deem very important.

Read more

Today’s article will be a concise analysis of some fill data I’ve collected on Binance Linear Futures for a top 50 universe comprising about 35,000 real fills. These were collected recently, only over the last 4 days, by ping-ponging the book with the intention of collecting data.

This is supplied with the intention of helping traders make decisions about making into their positions better. We make many interesting findings around the best ways to quote and how to improve fills.

The quotes are updated fairly fast using low-latency code colocated in AWS Tokyo nearby the Binance server in the data centre. This however is not necessarily reflective of the fastest systems out there, and only a reasonable job of latency optimisation.

The article is rather short and simply presents some interesting results; another new article is coming out soon!

Read more

Introduction

In today’s article, we will walk through 6 novel latency tricks for crypto exchanges which so far are entirely unseen. To date, none of these latency optimisations have been published for the public to view and are exclusively accessible to Quant Arb readers. I hope you all enjoy!

Read more

Introduction

I think the ways the industry works are very mysterious at times, and by that I mean the quant trading industry (proprietary trading or quantitative hedge funds). I have sat in a PM-like seat multiple times now and thought I’d sit down to talk about the differences in firm structures, how the compensation typically looks, and what are the smart choices to make as a PM.

Index

1. Introduction

2. Index

3. IC vs PM

4. Compensation

5. Cost Attributions

6. External vs. Internal

7. Final Words

IC vs PM

I think this is one of the biggest differences in how things are setup at firms. There is, of course, a middle ground between these two types of setups but it is a great classifier for thinking about firm structures. These are: IC - individual contributor and PM - portfolio manager.

In individual contributor roles there is much more collaboration and each individual works on improving the business as a whole. It is much more stable as a job since as long as you do your job well you are not very likely to lose that job (doing your job poorly is a different story of course). There is no risk that the PM above you blows up and you lose your job or you simply fail due to some other assortment of reasons despite working hard, and being smart. The base pay is typically higher, but the bonuses and control is much more limited. These are firms like HRT, Jane Street, Jump, XTX, etc where you hear about these staggeringly large compensations, but it is mostly in the form of base and a fairly bounded (both on the up and down side) bonus. Using HRT as an example, their entire codebase is entirely open for all employees and there is no silo-ing between teams. This makes for a great learning environment, and removes one large risk / challenge of being a PM which is knowledge acquisition (if you are never mentored by the right people, and get under a bad PM it is very hard to be a successful PM purely from scratch). On the downside, it also means that your efforts get blended into the efforts of others, and the career progression ladder requires you to be a lot more collaborative. For most of these IC type firms, if you want to move up you need to become management and start running internal teams / becoming a head of some sort of project / division. The kind of PnL tied compensation that happens on the PM side just isn’t do-able without working very closely with many others in these IC roles, and even in this case it will be a discretionary bonus and the effective % you get will be a lot lower than if you were a PM.

For PM structured businesses, you will either be a portfolio manager, sub-PM, or working under a PM (their researcher / developer). The experience here can be wildly different depending on how good of a PM you are or how good the PM you are under is. These are usually the characters you hear about having made 10s or even in some cases hundreds of millions of dollars. The bonuses given by the PM often can also be large for the people working below these successful PMs, but more likely the money made by the people under said successful PM will come from them taking that knowledge the PM has passed on and becoming a PM themselves (hopefully successfully). If they become a PM under the PM (as a sub-PM) then all parties are usually happy and the PM takes some chunk (either explicitly or discretionarily) of the sub-PMs would-be cut if they were a full PM. This is not to say the sub-PM has their own cut - usually they are simply part of the PnL of the main PM (which is how they get their backing in the first place — via the main PM backing them with their own PnL on the line) and as such if the main PM loses a lot of money then there is no PnL cut to pay out to the sub-PM even if they did well.

For PMs, the PnL is often a direct formula, especially in the more pod like shops such as Tower, but in firms like QRT, PMs receive discretionary bonuses in most cases. That said, when it is discretionary there is usually much more communication between teams whereas in some firms such as Tower there is very little communication. In some cases you can get fired for talking to other pods about their book! Millennium is the largest “PM” shop and some of its pods are incredibly large, to the point of being practically their own firm. One of these is WorldQuant (WQ) which is actually a pod inside of Millennium. Within WQ it is structured such that the data team prepares data, researchers engineer features (alphas) and then PMs handle forecasting and portfolio construction, and finally execution teams deal with putting the positions on. In WQ, you can be one of 3 types of PMs:

1. EQW

2. Specialist

3. Independent

EQWs can trade any asset class and this is more senior than specialist; you typically have to work towards this position. Specialists can only trade a very limited range of assets. Independent PMs can trade whatever they want typically and come in with their own infrastructure. They often see a large chunk of the pipeline from data to execution (although not always all of it) unlike the other two types of PMs who will only perform forecasting and portfolio construction. This is just a breakdown of the structure at one firm, every firm has its own structure and most PMs tend to lean more towards the independent PM setup with perhaps some help on execution and data provided but often their own signals. This is also an MFT specific setup, HFT is by definition almost always an independent PM-like setup. Tower however does have a latency team, tooling, and infrastructure provided to PMs (as far as I know for the crypto side) so there is help!

I find IC roles tend to be more for big HFT shops and PM roles tend to lean more MFT. This is the nature of the business: you can’t do equities HFT with a 3 person team, but you can do MFT in equities with a team of that size (provided legal, operations, etc are provided like any normal pod setup). I have seen small teams like this in crypto (and other asset classes) but some types of trading require really large teams to be competitive and there isn’t much bargaining ability for an explicit PnL cut when you have a team that large and with lots of infrastructure. You can’t leave with the team easily and the bargaining chips are gone. If an MFT (or HFT in an asset class where small teams are practical) PM leaves with his researchers and developers below him, short of non-competes, and NDA/IP protections they will still be able to rebuild at a new firm once the garden leave is up.

Compensation

I am not as well positioned to give an idea of what IC roles pay since they vary and I have mostly worked in PM positions, but recruiters often publish reports on industry compensations if you can get your hands on it. I know Selby Jennings has one (although I thought it was quite high, probably for top funds / NYC).

I think it is worth bringing up that most of the industry does NOT get paid 300k+ base salary. This is what Citadel, JS, etc pays, and even then this their top offices where the best talent is. If you are in a smaller office or on a team which does not drive as much PnL for the firm you may get paid less. If you are junior and are at a small or medium firm (non-top tier), you will not break $200,000 (base) a year, and without a very significant amount of PnL being generated there’s a high chance the total compensation remains below this mark. In fact, I have seen firms pay less than $100k base in some rare cases, and in many common cases traders/quants paid $120,000 to $160,000 (TC). There are a lot of people who want to work in the industry afterall. A large part of the industry does not pay these huge sums of money you will see online as base salary. It would be irresponsible I think if I did not mention this.

For PM roles, I’ll cut straight to it. Typical compensation I’ve seen and negotiated around has been $200,000 - $350,000 USD for base compensation, and I have seen it go up to about $500,000 for others. I expect established teams who have consistently made VERY significant PnL have boosted this up much higher, but this is typically a reasonable expectation of compensation (crypto specific). This is a range I have heard from others and have seen in my own career as a common range.

For PnL cuts, SMA deals typically sit in the 20-30% cut range, and PMs get around:

• 5% at WQ (quite low, but lots of infrastructure and all signals provided)

• 15-20% at Millennium

• 10-20% is reasonable for firms like QRT, Point72 etc

Proprietary trading firms pay a lot more:

20-50% is the usual range. I think most offers sit around 25-35% and if you do better you will work up towards 40%, and if you are a top team you will get 50%. It doesn’t go beyond 50% as far as I’ve seen, but I know pods where it is 50/50 their capital and the firm’s capital so the effective cut then rises significantly. Often the charge for running your own capital is 0-20%, some firms offer deals where it’s entirely your own capital and you can use their top fee accounts etc.

Cost Attributions

One of the benefits of being an internal PM at a firm is that you get all your costs paid for you, but they do still come back to you… Costs will be taken out of your PnL or your cut depending on your deal. Often some costs are paid entirely by the firm (this can be only operations/compliance or can go as far as you only paying for your salary - but I’ve never seen someone pay nothing, although I’m sure it has happened for a lucky pod somewhere out there).

The term “draw” is what comes out of your PnL cut and usually refers to your salary, with top of the line (business expenses) expenses coming out of the PnL itself. Costs may include:

• Latency Tech (latency line providers like Avellacom, Mckay Brothers, and BSO for HFT pods - crypto specific names)

• Servers (research or production servers)

• Datasets

• Salaries of your team

• Operations / Compliance / Legal (often charged as a cost of “hey you pay X for our internal legal/ops team and this is mandatory”) (one of the more likely items to not be charged)

• Office costs (sometimes charged on a per seat basis if you are not renting your own office)

This obviously does not apply to IC roles. I find it is almost the opposite way around where IC roles often give very strong benefits (rules like you can expense up to £100+ if you are doing an activity with 2 or more other colleagues which can be things like surfing lessons) whereas PM seats do not need to woo hires with special benefits — it’s eat what you kill, and if you kill a lot you eat a lot (big fat bonus) and if not you starve (get fired).

On the note of getting fired, I have seen some people make it 2 years without making any money before getting fired! But I think after about a year your time is up at most shops, and often you should be aiming to have something making money before 6-9 months since that’s when the heat turns up! Realistically you should at the 3 month mark have something to show for yourself. It will depend on what your setup is and how much infrastructure needs to be built, how desperate the firm is (not ideal), how well you can make up excuses (also not an ideal factor to have to use!), and whether you have come in with prior infrastructure which can speed up deployment time. Your time will eventually run out if you don’t make money as a PM or the PM that you are under doesn’t make money so ideally try to make money! If you are under a PM and they get fired you will either be fired or moved to another team. Depends on the firm, and how well others thought of you. I know great quants who were sub-PMs or researchers under an unsuccessful PM and sadly lost their jobs despite doing everything right on their part.

External v.s. Internal

We have so far talked about internal PMs, but there are also external PMs. It is usually some sort of SMA deal where you get to keep your IP, but they pay you some relatively small base in the form of management fees (they also may not) and they often times will cover your business expenses provided they are not excessive. It is a good deal for the firm because they are paying much less than usual (you will not get away with asking for 100s of thousands in costs, maybe up to 20-50k in costs although good chance it’s even less and then get 2% of the capital under management if they even pay that out). Sometimes, no such costs are covered and you are out of luck, but it is a reasonable setup if you’ve got low costs, some savings to live on, and want to be in a PM seat with full ownership of the IP in the end (which lets you move around with minimal friction and thus ensure your terms are fair / you are not locked in).

Final Words

I hope this was insightful and provided some information to newcomers in this industry or maybe even those who have been in it for a bit. I may write a second part about some other thoughts on this matter if the audience enjoys this article a lot.

Disclaimer: I may be off about one specific firm doing something, things change! but this is written to the best of my knowledge about the industry having worked in it for a fair while in PM-level roles.

In the last article, we covered 3 working alphas, 2 weaker alphas and one strong alpha. Now we will present a medium strength alpha and a second strong alpha then show how to combine them to form a very profitable trading strategy both before and after real world trading costs. In the final part to this series (the next article), we will explain how to trade and monetise the alphas although the explanation today will be more than sufficient to make money trading (even the first article presented enough alpha standalone to make money).

We combine to form the below curve:

Which when we rebalance every 72h we achieve 2.7 Sortino, and 110 bps on dollars traded!

Much like the prior article this will be fairly short. There is little reason to ramble on when I am merely presenting working alphas to the reader, I will resume more educational material in further articles in this series (research processes, and data pipelines to find said alphas).

Read more

In this article, the first in a three part series, we will present 3 working options alphas where we use options data to predict perpetual prices. Over the three articles we will present 5 alphas which achieve a combined performance above 2 Sharpe combined and 110 bps on dollars traded (All assets have sub-5 bps trading costs) on 72h rebalance (~1.5 Sharpe at 72h rebalance, ~2 Sharpe at 1h rebalance).

In the second article, we will share 2 more alphas ranging between 1 and 2 Sharpe, and show how to combine them. Then we will analyse the signal as a fully monetizable strategy in the 3rd article. These alphas use options data to predict perpetual prices of BTC, ETH, SOL, and XRP on Binance, one of the 3 we will present today is shown below:

The alphas are fairly uncorrelated and orthogonal (not perfectly, but still very good for all being from the same dataset type) as seen in the scree plot below:

I think this series will be a real treat for readers as we present real working alpha which can actually be monetized now with no part left unexplained (other than data aggregation methods as this is very extensive, we will explain how to create the features of course though).

As with the previous article, I will not bore you with excessive writing as these are not complicated ideas and the work has already been done by myself in finding them, I will provide the feature, how to replicate it, and the performance.

Read more

In this article we will detail 2 (two!) proprietary orderbook alphas which achieve over 3 sortino when combined and over 2 Sharpe each individually. To date, there is no public literature documenting this alpha and is an entirely novel creation presented exclusively to readers of the Quant Arb Substack. We achieve the below performance as a raw signal using cross sectional z-score portfolio construction:

The Alpha

Read more

Introduction

Personally, I feel as though the whole world of options statistical arbitrage has always been fairly confusing for most people. In my view, there’s 4 main ways to approach statistical arbitrage strategies using options (that I’ve worked with at least). I’m not talking about pairs trading here for those who may be a little confused, I’m instead talking about statistical medium frequency alphas that trade options. I will give the frameworks for modelling these effects and the ways in which alpha is often discovered.

This is perhaps one of the areas most shrouded in mystery (as if doing good statistical arbitrage research wasn’t hard enough). I’ll break it down in this article the ways that I’ve always approached the problem (all of which I’ve found to be fairly successful, although some a bit more than others)

We will talk only about strategies that directly trade options, and not about strategies that use information from options markets to trade. You can do that and it works well, although for crypto options effects are a fairly weak in my opinion and I’ve always had trouble monetising the metrics I found that worked partially because you can only really trade BTC/ETH (so it’s either BTC/ETH relative value or a time series strategy, which rules out cross sectional which would be my preferred way do it if I could), but mostly because the options market in crypto is a much smaller part of the total volume compared to markets like equities where options flows are big game. Anyways, you can still use options as signals in linear stuff, but that’s a story for another time.

Read more

Introduction

Alphas make up the core of what drives a large majority of firms trading PnL. You can have complicated quoting rules, great models, amazing optimizers, advanced vol curves, or fast latency, but the gist of most trading firms edge can be compiled into some alpha representation. Even execution relies heavily on “execution alphas” in order to deliver highly effective execution costs to trading strategies. In this article, we will cover how to structure and trade alphas in the context of MFT, HFT, execution trading, arbitrage, and options (OMM + options MFT).

Out of all the ways to find an edge, one stands out the most and it is having very strong alphas signals. HFT strategies use this, MFT strategies use this, options strategies use this, execution algorithms use this - practically all advanced trading operations will have some set of proprietary alphas they use to inform their understanding of where price is going. The only exception will be arbitrage, but even that we will see becomes quite alpha like in some cases.

MFT Alphas

This is where everyone hears about alphas mostly, and when I say alphas I am specifically talking about features or “formulaic alphas”. You can generate logical alphas as well, but I find these to be very inefficient. The pipeline I will mention, is a very well known pipeline - one I have talked about in the past. You take in data, you engineer it into your features or “alphas” which is where the true profitability lives and then from here everything else serves as a performance enhancer which builds on your alphas. This is your forecasting, and then your optimizer. I like to view each block in the pipeline as a converter. We convert data into alphas (feature engineering), alphas into forecasts (forecasting / ML), forecasts into target portfolios (portfolio optimization) and target portfolios into individual trades (execution - which we will go over in the next section). The reason logical alphas are inefficient is because they bypass the portfolio optimization and forecasting stages and directly output a target portfolio in many cases. You can include them in the pipeline, but it is quite tricky to use binary outputs, and the best case is that you find a way to represent the idea as a formulaic alpha or run it separately from your formulaic pipeline (they don’t tend to like to work together and this is often what has to occur).

For MFT, it couldn’t be more clear that the alphas are the foundation. In logical form they are the whole strategy and in formulaic form they are the base of what makes all the money. If you do not have good alphas, you will find it impossible to make any money trading in any reasonably competitive market. Unless you find some horribly inefficient market (in which case finding & accessing said market is the alpha itself because such a thing is so rare!) you will rely on alphas to drive profits. Even in the case of a horribly inefficient market you will still need to have some alphas although they may be extremely basic such as quoting around Binance on a small exchange (in this case Binance’s price is the feature itself). This is a common mistake among beginner quants - they falsely believe that you can save bad features with advanced & complicated machine learning. Mentally you should view what happens after the alphas as a multiplier, ESPECIALLY machine learning as it is the most multiplier like of them all. 2 * 0 is still 0! Portfolio optimization is also important and can save you from trading yourself to death, but the boosts from here on are very incremental - and the alpha is driven by your alphas (aptly named!).

Execution

How does really high-performance execution work? Well, at some level it is very much an HFT problem, since you are trying to get limit orders filled with great markouts (hence having some HFT overlap), but at a higher timeframe it becomes an alpha problem. Say we want to do a 1h TWAP. We can choose to speed up or slow down our execution based on our view of liquidity and our view of price. Part of this is deciding the parameters of the execution algorithm itself, this is fairly simple for a market order TWAP as we can take the alpha decay of our signals we want to trade and the curve for how our execution improves as we wait longer and find the optimal intersection through interval iteration where we maximize total edge (inclusive of trading costs). From here, we can figure out the optimal number of chunks and how aggressive to be on limit orders through similar forms of analysis. This gets us to a fairly basic execution setup.

Read more

Introduction

When it comes to HFT strategies, you often reach a point where latency cannot be ignored. Simply having great alphas and modelling is not enough — you also need to be competitive on the latency front. Most of this is concentrated on the cloud network engineering front, so in today's article, we catalogue many different latency tricks.

We will dive into >10 different latency optimisations in this article to help you improve your latency setup.

Latency touches all elements of HFT strategies. For arbitrage strategies you are often competing against a couple other participants on any given market and having even a mild latency edge is often enough to thrash the competition, ensuring you aren’t left picking up their crumbs. It’s not just arbitrage strategies that need low latency, market making (which most arbitrage strategies eventually lead to since making into opportunities is the optimal approach), needs low latency in order to ensure your quotes are up to date with the latest estimates of global fair value (which a large part is just data feeds of various exchanges and getting that fast). Even statistical strategies often have latency requirements when they’re run on the HFT timescales.

This is all to say that latency is extremely important in the HFT world, and by not learning how to optimize it — you are missing out on a valuable skill which can make the difference between thrashing your competition.

Index

1. Latency Tricks:

   1. WS Rotate

   2. FIX feed

   3. Machine Gun Orders

Read more

Introduction

There is no doubt that regardless of whatever area of quant you end up in, you will end up having to do some degree of forecasting. Whether that is forecasting returns (for stat arb strategies), funding rates (for funding arb strategies), volumes (for execution strategies), or even parameters is a vol curve, it is a problem that comes up time and time again in the work of quants. Today, me and are going to be walking through our thoughts on how to do forecasting properly with some practical tips. We focus primarily on return forecasting in a statistical arbitrage manager context.

[This article is available to readers of either of our publications in it’s entirety so feel free to subscribe to either!]

Index

[Quant Arb]

1. Introduction

2. Index

3. What are we forecasting

4. Implicit Forecasts (and why they work!)

5. Models

6. Features come first

7. What doesn’t work

   1. Dropping features at the model level

   2. Dimensionality reductions

   3. Lots of bad features

8. Forecasting isn’t the best edge…

[Systematic Long Short] - On Combining Forecasts

9. The Limits Of Diversification

10. Optimal Forecast Weighting

11. When Forecast Combining Breaks Down

12. A Few Simple Heuristics…

Read more

So far we have covered how to get fair value when doing options market making, but we are yet to cover the complex parts relating to quote sizing, skewing, spreads, quoting OTC, quoting illiquids, quoting multiple exchanges, and the risks that we skew to avoid outside of the usual greeks (which are well known).

So today, in the 4th article in our options market making series, we will focus on completing the pipeline.

Read more

Introduction

Half the work with arbitrage trading is figuring out where the opportunities are. You can make your system as advanced as you want, but if you can’t find the exchanges and products where all the returns are concentrated then you won’t be very profitable.

In this article, I explain what to look for when deciding on whether to add an exchange and even when you do integrate an exchange the reasons why the initial research showing arbs could be unrealistic (wash flow, internalized flow, etc).

Then, finally — the part most of you are deeply interested in, I explicitly share where the best arbitrage opportunities currently are and can actually be captured based on my own private research.

Read more

Introduction

So you want to do market making? Perhaps you’ve seen some screenshots of those double digit Sharpe charts doing huge returns, and want to do something a bit more advanced than a pure arbitrage based approach? Well, you’re in luck. In this article, we’ll break down each component of a proper market making system and how to do it successfully.

There is obviously a large gap between doing it very well on a small exchange and doing it very well on Binance. My definition of successful will lean towards the small exchange definition since if I had alpha on Binance then I’d be making a fortune. I’ve done that in past roles (not with the same team anymore to do it again), and it’s no small feat. We had some of the best latency tech out there, plus a large team and it still took lots of work.

In today’s article, we dig into how to do market making effectively and continue some of the points from the market making for dummies article with more advanced tricks as well as recaps of simpler parts (but with expanded information on how to do them properly that didn’t cross my mind to include when writing the past article).

Read more

Introduction

Read more

Introduction

This article is a mix of various thoughts I have about how to work well with HFT data and perform high quality quantitative research. HFT data is interesting to deal with because it’s cumbersome, slow, and often messy. You want to find out what happened but you need to dig through 20 different data types from your internal logs, and on top of that often the dataset is big enough to keep your server forever if you try to process any attempt at a multi-year backtest.

Well, that’s roughly what I aim to talk about in this article. I can’t promise this will be a detailed tutorial on how to be an HFT researcher, you won’t get anywhere near that far from reading articles - in fact, you’ll need to get working with the data itself if you want to travel that far (and probably get a bit of mentorship along the way as we all tend to get), but I do think this article provides insights that can only be acquired through many years of working with the data (even if to truly become a pro you need to spend some time with the data), I would argue that a lot of this information would otherwise take ages of toiling around to figure out. Part of my professional experience has involved running research in HFT operations and as part of that I have gained insights into how to organize the research process in order for it to produce useful results. I do hope this article is useful for those in the industry who have to work with HFT data regularly. These are observations from my experience working in various HFT operations.

Read more