Max Halford

Lower your warehouse costs via DuckDB transpilation

maxhalford25@gmail.com (Max Halford) — Thu, 12 Mar 2026 00:00:00 +0000

Many people seem to admire DuckDB. But most of us are stuck with our traditional warehouses, because they’re entrenched in our data stacks and IT landscape. This is with good reason: BigQuery, Snowflake, ClickHouse and co. are great software. But they’re not cheap, and keeping a warehouse’s monthly bill under control is non-trivial.

What if you could get the best of both worlds? Tables could be kept in your warehouse of choice, but computed with DuckDB. There’s been some discussion on multi-engine warehouses – see this article by Julien Hurault and Sung Won Chung. But their proposition is to assign each table to one engine. Multi-engine stacks can be useful, and SQLMesh provides support for it. But to me it sounds too sophisticated, and I’m not convinced it’s what most practitioners want/need.

Text classification with Python 3.14's zstd module

maxhalford25@gmail.com (Max Halford) — Fri, 06 Feb 2026 00:00:00 +0000

Python 3.14 introduced the compression.zstd module. It is a standard library implementation of Facebook’s Zstandard (Zstd) compression algorithm. It was developed a decade ago by Yann Collet, who holds a blog devoted to compression algorithms.

I am not a compression expert, but Zstd caught my eye because it supports incremental compression. You can feed it data to compress in chunks, and it will maintain an internal state. It’s particularly well suited for compressing small data. It’s perfect for the classify text via compression trick, which I described in a previous blog post 5 years ago.

Solving Détrak with brute force

maxhalford25@gmail.com (Max Halford) — Mon, 02 Feb 2026 00:00:00 +0000

Détrak is a simple board game. There’s a 5x5 grid, and players need to place 12 domino-like pieces to cover the grid completely. The domino symbols are determined by rolling two dice. The dice are six-sided, and their rolls are shared by all players. Points are scored based on adjacencies of matching symbols.

The goal is to find the optimal placement of the pieces to maximize the score based on the rolled symbols. What makes it competitive is that everyone has to work with the same rolls. This involves logical thinking and spatial reasoning. But you also have to juggle luck and risk, because you can’t predict the dice rolls.

Nostalgia for a time I didn’t experience

maxhalford25@gmail.com (Max Halford) — Sat, 17 Jan 2026 00:00:00 +0000

Our everyday vocabulary fails to capture subtle emotions. Some languages developed to fill in these gaps. An example that became popular is Schadenfreude, which means taking pleasure in someone else’s misfortune. It has an opposite in German, Fremdscham, which is the vicarious embarrassment you feel when witnessing someone else’s humiliation. Its Finnish equivalent myötähäpeä is slightly more well known.

I like these words that capture complex feelings. When I was in high school, Wittgenstein made a lasting impression on my very naive mind. He argued that language can only picture facts about the world, and that ethics, aesthetics, and the mystical lie beyond what language can express. I think this is fascinatingly true. I’m convinced that a share of our daily sorrows come from our inability to put them in words. It reassures me when I stumble on a word that expresses a feeling I can’t explain.

Row level lineage at Carbonfact

maxhalford25@gmail.com (Max Halford) — Fri, 09 Jan 2026 00:00:00 +0000

No pain no startup

maxhalford25@gmail.com (Max Halford) — Mon, 27 Oct 2025 00:00:00 +0000

I’ve been working at Carbonfact for close to 4 years. Two other people and I were the first hires. I got to build a large share of the initial systems. I’ve been involved in many business decisions, but I’ve mostly actively contributed to technical aspects.

I had to decide on a lot of things at first. What data warehouse to use, the internal data models, how to organize our Python code, our definition of self-serve analytics, and even seemingly mundane things like Pandas vs. Polars. There was a lot to do and it was fun. There were no pushbacks because there was nobody else to push back. I could just build what I thought was best. I even pushed to the main code branch, because nobody else had to review my code. There was always something to build as the startup garnered traction. It felt painless, in a naive sort of way.

Scraping Google Calendar events

maxhalford25@gmail.com (Max Halford) — Sun, 12 Oct 2025 00:00:00 +0000

At my day job we deal with enterprise customers. They pay us a subscription fee, and in return we help them in various ways to reduce their carbon footprint. To keep the boat afloat, we need to make some money. We shouldn’t spend more money than we make. So we need to keep track of our revenue and costs. Our gross margin is (revenue - cost) / revenue, where the cost is mostly the salaries of our employees.

Warmshowers sparks joy

maxhalford25@gmail.com (Max Halford) — Sun, 24 Aug 2025 00:00:00 +0000

I have a friend with whom I like to go travelling on a bicycle. We usually go for a couple of weeks, and we’ve been doing it regularly over the past five years. We’ve cycled in France and England so far. Both countries have many affordable campsites, and there’s always a decent hotel/Airbnb not too far for when it’s raining.

But this year we travelled through Switzerland, which is prohibitively expensive, and somewhat lacking in terms of places to camp. We were aware of Warmshowers and decided to give it a try. It was fantastic, and opened our eyes on what travelling and connecting with people can be like. I don’t think we’ll ever travel like we used to anymore.

Do LLMs identify fonts?

maxhalford25@gmail.com (Max Halford) — Wed, 30 Jul 2025 00:00:00 +0000

Spoiler: not really

dafont.com is a wonderful website that contains a large collection of fonts. It’s more comprehensive and esoteric than Google Fonts. One of its features is a forum where users can ask for help identifying fonts – check out this poor fellow who’s been waiting for over two years and bumped his thread. I thought it would be interesting to see if an LLM could do this task, so I scraped the forum and set up a benchmark.

Thoughts on DuckLake

maxhalford25@gmail.com (Max Halford) — Mon, 09 Jun 2025 00:00:00 +0000

DuckLake is the new data lake/warehouse from the makers of DuckDB. I really like the direction they’re taking. I’m hopeful it has the potential to streamline the data engineering workflow for many people, vastly reducing costs along the way.

I’m a bit of a nut and don’t use SQLMesh or dbt. Instead, I built lea a few years ago, and we still use it at Carbonfact. I would probably pick SQLMesh if I had to start over, but lea allows me to explore new ideas, so I’m sticking to it for now.

The total derivative of a metric tree

maxhalford25@gmail.com (Max Halford) — Tue, 06 May 2025 00:00:00 +0000

A metric tree is a visual way to organize a complex metric. Count gives a good introduction here. Abhi Sivasailam gave a popular talk at Data Council 2023 if watching videos is your thing. Ergest Xheblati is someone to follow if you want to go deeper. There’s also a recent article from Lightdash. Finally, there’s this article by Timo Dechau, but it’s behind a paywall. The concept has a homonym, so beware when you browse for it.

Minimizing the runtime of a SQL DAG

maxhalford25@gmail.com (Max Halford) — Sat, 08 Feb 2025 00:00:00 +0000

I recently looked into reducing the runtime of Carbonfact’s SQL DAG. Our DAG is made up of roughly 160 SQL queries. It takes about 10 minutes to run with BigQuery, using on-demand pricing. It’s decent. However, the results of our DAG feed customer dashboards, and we have the (bad) habit of refreshing the DAG several times a day. Reducing the runtime by a few minutes can be a nice quality-of-life improvement.

Hard data integration problems at Carbonfact

maxhalford25@gmail.com (Max Halford) — Thu, 02 Jan 2025 00:00:00 +0000

Carbonfact’s customers are clothing brands and factories. Our mission is to measure (and ultimately reduce) the carbon footprint of their products. We need primary data to do this: purchase orders, bills of materials, energy consumption data, etc.

Each customer has a unique IT setup, which makes it challenging to scale to hundreds/thousands of customers. Our success as a business depends on our ability to not reinvent the wheel for each customer.

Introducing icanexplain @ PyData Paris 2024

maxhalford25@gmail.com (Max Halford) — Thu, 26 Sep 2024 00:00:00 +0000

@daily_cache implementation in Python

maxhalford25@gmail.com (Max Halford) — Tue, 27 Aug 2024 00:00:00 +0000

I spend a lot of time at Carbonfact working on datasets shared by our customers. We typically set things up so that our customers can export data automatically. They usually deposit files to a GCP bucket, with a script, once a day. We then have an ETL script for each customer that runs afterwards to fetch their latest data and process it.

During development, I load customer data to my laptop and work on it. The datasets can be quite heavy, and it takes time to fetch them, so I cache them to save some time. Python has something for this in its standard library:

LCA software: exit the matrix

maxhalford25@gmail.com (Max Halford) — Sun, 09 Jun 2024 00:00:00 +0000

Measuring the environmental impact of a product is done using life cycle assessment (LCA). This is a methodology that breaks down a product’s life cycle into stages (LCI), and measures the impact of each stage on the environment (LCIA).

There are a few pieces of LCA software to choose from. The leading ones are SimaPro, GaBi, openLCA, and Umberto. These are all proprietary software, and they’re expensive. But there’s a free and open source alternative: Brightway.

Cutting up shoes to measure their footprint

maxhalford25@gmail.com (Max Halford) — Fri, 17 May 2024 00:00:00 +0000

Our mission at Carbonfact is to measure the environmental impact of clothes. This involves a lot of steps. The main one is to determine what materials a product is made of, along with each material’s mass. This is straightforward for most clothes like jumpers and pants. These are typically made of a single fabric, such as cotton or polyester. The mass of each material is roughly the same as the product’s mass.

A training set for bike sharing forecasting

maxhalford25@gmail.com (Max Halford) — Thu, 04 Apr 2024 00:00:00 +0000

Last night I went to a Toulouse Data Science meetup. The talks were about generative AI and information retrieval, which aren’t topics I’m knowledgeable about. However, one of the speakers was a friend of mine, so I went to support him. Toulouse is my hometown, so I bumped into a few people I knew. It was a nice evening.

Fast Poetry and pre-commit with GitHub Actions

maxhalford25@gmail.com (Max Halford) — Tue, 27 Feb 2024 00:00:00 +0000

This is a short post to share a GitHub Actions pattern I use to setup Poetry and pre-commit. These two tools cover most of my Python development needs. I use Poetry to manage dependencies and pre-commit to run code checks and formatting. The setup is fast because it caches the virtual environment and the .local directory.

I like to use custom actions for this type of stuff. These are base actions that can be re-used in multiple workflows. I have a custom action to install the Python environment. Here’s the action file:

Decomposing funnel metrics

maxhalford25@gmail.com (Max Halford) — Thu, 14 Dec 2023 00:00:00 +0000

Funnel metrics as products

I talked about metric decomposition in a previous article, and how it can be used to explain why metrics change values over time. That article explained how to decompose a sum, as well as a ratio. In this article, I’ll explain how to decompose a product.

revenue = impressions * click_rate * conversion_rate * spend

The decomposition in this article isn’t limited to funnels. It can be applied to any metric that is expressed as a product of factors. For instance, at Carbonfact, we decompose the carbon footprint of a clothing line as so:

Efficient ELT refreshes

maxhalford25@gmail.com (Max Halford) — Fri, 01 Dec 2023 00:00:00 +0000

A tenant of the modern data stack is the use of ELT (Extract, Load, Transform) over ETL (Extract, Transform, Load). In a nutshell, this means that most of the data transformation is done in the data warehouse. This has become the de facto standard for modern data teams, and is epitomized by dbt and its ecosystem. It’s a great time to be a data engineer!

We at Carbonfact fully embrace the ELT paradigm. In fact, our whole platform is powered by BigQuery, which acts as our single source of truth. We have a main BigQuery dataset where we materialize several SQL views that power what our customers see.

Online machine learning on the road @ IDE+A, TH Köln

maxhalford25@gmail.com (Max Halford) — Thu, 26 Oct 2023 00:00:00 +0000

Sh*t flows downhill, but not at Carbonfact

maxhalford25@gmail.com (Max Halford) — Mon, 16 Oct 2023 00:00:00 +0000

I’m writing this after watching the talk Joe Reis gave at Big Data LDN. It’s called Data Modeling is Dead! Long Live Data Modeling! It’s an easy-to-watch short talk that calls out on a few modern issues in the data world.

I’d like to bounce off one of Joe’s slides:

I’m aligned with Joe that many issues stem from the lack of unison between engineering and data teams. A fundamental aspect of the Modern Data Stack is to replicate/copy production data into an analytics warehouse. For instance, copying the production PostgreSQL database into BigQuery.

Answering "Why did the KPI change?" using decomposition

maxhalford25@gmail.com (Max Halford) — Wed, 09 Aug 2023 00:00:00 +0000

Edit – I published a notebook here that deals with the case where dimension values may (dis)appear from one period of time to the next. The notebook decomposes a ratio, but the logic is also valid for decomposing a sum.

Edit 2 – I’ve stumbled on this article by Shao Zhifei which provides a good derivation of the ratio decomposition formula. I contacted Shao Zhifei on LinkedIn, and he told me they heavily use these formulas at Grab. He also pointed out a typo in the ratio decomposition formula which I have now fixed.

Measuring the carbon footprint of pizzas

maxhalford25@gmail.com (Max Halford) — Sun, 25 Jun 2023 00:00:00 +0000

Making environmentally friendly decisions can only be done with the right information. At Carbonfact, we’ve realized a big challenge is the lack of information about industrial processes. We tackle that slowly but surely by gathering data from various sources, and making it available to our customers.

Regarding food, the French government has a great initiative called Agribalyse. It’s a free database of environmental footprints for various food products. It includes raw ingredients straight out from the farm, as well as ready to eat dishes from the supermarket. It’s a great initiative, as it allows anyone to do their own research and make informed decisions.

Graph components with DuckDB

maxhalford25@gmail.com (Max Halford) — Sat, 03 Jun 2023 00:00:00 +0000

Introduction

Graph problems are quite common. However, it’s rare to have access to a database offering graph semantics. There are graph databases, such as Neo4j and GraphX, but it’s difficult to justify setting one of those up. One could simply use networkx in Python. But that only works if the graph fits in memory.

From a practical angle, the fact is that people are querying data warehouses in SQL. There are many good reasons to write graph algorithms in SQL. And anyway, one may argue that graphs are a special case of the relational model.

For analytics, don't use dynamic JSON keys

maxhalford25@gmail.com (Max Halford) — Thu, 11 May 2023 00:00:00 +0000

I love the JSON format. It’s the kind of love that grows on you with time. Like others, I’ve been using JSON everywhere for so many years, to the point where I just take it for granted.

I suppose the main thing I like about JSON is its flexibility. You can structure your JSONs without too much care. There will always be a way to consume and manipulate it. But I have discovered a bit of anti-pattern, which I believe is worth raising awareness about. Especially when you’re doing analytics.

Metric correctness doesn't matter, consistency does

maxhalford25@gmail.com (Max Halford) — Fri, 28 Apr 2023 00:00:00 +0000

According to the United Nations, the 15th of November was the day we crossed 8 billion humans on the planet. How can they be so sure of that? Surely there has to be some margin of error, meaning it could have happened on the 14th or 16th. Then again, does it matter?

I would argue almost all metrics we look at are incorrect. For instance, I work at a company whose goal is to measure the carbon footprint of clothing items. I can tell you first hand our measurements are stock full of assumptions. In the sustainability world, it’s not surprising to get reports like this one:

Online gradient descent written in SQL

maxhalford25@gmail.com (Max Halford) — Tue, 07 Mar 2023 00:00:00 +0000

Edit – this post generated a few insightful comments on Hacker News. I’ve also put the code in a notebook for ease of use.

Introduction

Modern MLOps is complex because it involves too many components. You need a message bus, a stream processing engine, an API, a model store, a feature store, a monitoring service, etc. Sadly, containerisation software and the unbundling trend have encouraged an appetite for complexity. I believe MLOps shouldn’t be this complex. For instance, MLOps can be made simpler by bundling the logic into your database.

Using SymPy in Python doctests

maxhalford25@gmail.com (Max Halford) — Wed, 15 Feb 2023 00:00:00 +0000

A program which compiles and runs without errors isn’t necessarily correct. I find this to be especially true for statistical software, both as a developer and as a user. Small but nasty bugs creep up on me every week. I keep sane in the membrane by writing many unit tests 🐛🔨

I make heavy use of doctests. These are unit tests which you write as Python docstrings. They’re really handy because they kill two birds with one stone: the unit tests you write for a function also act as documentation.

Online active learning in 80 lines of Python

maxhalford25@gmail.com (Max Halford) — Sun, 22 Jan 2023 00:00:00 +0000

Active learning is a way to get humans to label data efficiently. A good active learning strategy minimizes the number of necessary labels, while maximizing a model’s performance. This usually works by focusing on samples where the model is unsure of its prediction.

In a batch setting, the model is periodically retrained to learn from the freshly labeled samples. However, the training time is usually too prohibitive for this to happen each time a new label is provided. This isn’t the case with online models, because they are able to learn one sample at a time. Active and online learning naturally fit together.

Are Airbnb guests less energy efficient than their host?

maxhalford25@gmail.com (Max Halford) — Tue, 17 Jan 2023 00:00:00 +0000

TLDR

I compared the energy consumption of Airbnb guests versus their host, in the same apartment, during 2022. It appears that guests do in fact consume more energy than hosts. The data I used is available to any Airbnb host. I also open-sourced all the code I wrote for this analysis.

Introduction

European energy prices have soared in 2022. It’s gone to the point where some Airbnb hosts have become reluctant to rent, believing their guests are too wasteful and cost too much. You can see this by scrolling Airbnb groups on Facebook.

The future of River

maxhalford25@gmail.com (Max Halford) — Tue, 13 Dec 2022 00:00:00 +0000

Source

When I see tweets like this one, I’m both happy because people are aware of River, but also irked because it’s really difficult to make production-grade open source software.

We just had a developer meeting a week ago. We planned what we will work on during the first half of 2023. I thought it would be worthwhile to give a high-level view of how we envision River’s future. If not to be comprehensive, at least to reassure potential users that River is alive and kicking 🤺

Parsing garment descriptions with GPT-3

maxhalford25@gmail.com (Max Halford) — Sun, 20 Nov 2022 00:00:00 +0000

The task

You’ll have heard of GPT-3 if you haven’t been hiding under a rock. I’ve recently been impressed by Nat Friedman teaching GPT-3 to use a browser, and SeekWell generating SQL queries from free-text. I think the most exciting usecases are yet to come. But GPT-3 has a good chance of changing the way we approach mundane tasks at work.

I wrote an article a couple of months ago about a boring task I have to do at work. I got a few interesting suggestions by email. Raphaël suggested a question-answering model here. I’ve slowly become convinced that GPT-3 might be the perfect tool for the job. It’s cold and miserable where I am, so I thought it would be opportune to take GPT-3 for a spin 🧙

Dynamic on-screen TV keyboards

maxhalford25@gmail.com (Max Halford) — Sun, 25 Sep 2022 00:00:00 +0000

This article has some interactive keyboards, therefore I recommend reading it from your computer rather than your phone.

On-screen TV keyboards

I’ve recently been spending time at my brother’s place. We usually eat in front of TV. I’ve thus found myself typing stuff on the Netflix/Amazon/Plex TV apps. The typing happens through a remote controller, which is slower than typing with ones fingers. However, the TV apps usually suggest the correct show/movie after five or six keystrokes, so it’s not that bad.

NLP at Carbonfact: how would you do it?

maxhalford25@gmail.com (Max Halford) — Tue, 06 Sep 2022 00:00:00 +0000

The task

I work at a company called Carbonfact. Our core value proposal is computing the carbon footprint of clothing items, expressed in carbon dioxide equivalent – $kgCO_2e$ in short. For instance, we started by measuring the footprint of shoes – no pun intended. We do these measurements with life cycle analysis (LCA) software we built ourselves. We use these analyses to fuel higher-level tasks for our clients, such as carbon accounting and sustainable procurement.

A life cycle analysis is essentially a recipe, the output of which is a carbon footprint assessment. Like any recipe, an LCA necessitates ingredients. In a cradle-to-gate scenario, this includes everything that is needed to make the product: the materials, the mass, the manufacturing methods, the transport between factories, etc. In our experience, the biggest impact on a product’s footprint come from the materials which it is made of.

Matrix inverse mini-batch updates

maxhalford25@gmail.com (Max Halford) — Wed, 24 Aug 2022 00:00:00 +0000

The inverse covariance matrix, also called precision matrix, is useful in many places across the field of statistics. For instance, in machine learning, it is used for Bayesian regression and mixture modelling.

What’s interesting is that any batch model which uses a precision matrix can be turned into an online model. That is, provided the precision matrix can be estimated in a streaming fashion. For instance, scikit-learn’s elliptic envelope method could have an online variant with a partial_fit method.

A rant against dbt ref

maxhalford25@gmail.com (Max Halford) — Tue, 28 Jun 2022 00:00:00 +0000

Disclaimer

Let me be absolutely clear: I think dbt is a great tool. Although this post is a rant, the goal is to be constructive and suggest an improvement.

dbt in a nutshell

dbt is a workflow orchestrator for SQL. In other words, it’s a fancy Make for data analytics. What makes dbt special is that it is the first workflow orchestrator that is dedicated to the SQL language. It said out loud what many data teams were thinking: you can get a lot done with SQL.

First IRL meetup with the River developers

maxhalford25@gmail.com (Max Halford) — Thu, 09 Jun 2022 00:00:00 +0000

River is a Python software for doing online machine learning. It’s the result of a merger in early 2020 between creme and scikit-multiflow. Saulo Mastelini, Jacob Montiel, and myself are the three core developers. But there are many more people who contribute here and there!

This week Saulo Mastelini and I got to meet in person. This is worth mentioning because Saulo is originally from Brazil, whereas I’m based in Europe. We connected and I’m glad to think of him as a good friend from now on. Of course we were not alone: some friends of mine from university also joined the fun. These are people who initially contributed to creme, back in what we already call the old days! Each one of them has their own areas of expertise, and contributed to various parts of the codebase.

Online machine learning with River @ GAIA

maxhalford25@gmail.com (Max Halford) — Thu, 07 Apr 2022 00:00:00 +0000

Fuzzy regex matching in Python

maxhalford25@gmail.com (Max Halford) — Mon, 04 Apr 2022 00:00:00 +0000

Fuzzy string matching in a nutshell

Say we’re looking for a pattern in a blob of text. If you know the text has no typos, then determining whether it contains a pattern is trivial. In Python you can use the in function. You can also write a regex pattern with the re module from the standard library. But what about if the text contains typos? For instance, this might be the case with user inputs on a website, or with OCR outputs. This is a much harder problem.

OCR spelling correction is hard

maxhalford25@gmail.com (Max Halford) — Sun, 06 Mar 2022 00:00:00 +0000

I recently saw SymSpell pop up on Hackernews. It claims to be a million times faster than Peter Norvig’s spelling corrector. I think it’s great that there’s a fast open source solution for spelling correction. But in my experience, the most challenging aspect of spelling correction is not necessarily speed.

When I worked at Alan, I mostly wrote logic to extract structured information from medical documents. After some months working on the topic, I have to admit I hadn’t cracked the problem. The goal was to process >80% of documents with no human interaction, but when I left we had only reached 35%. However, I developed a good understanding of what made this task so difficult.

Comic book panel segmentation

maxhalford25@gmail.com (Max Halford) — Sat, 05 Mar 2022 00:00:00 +0000

Edit (2023-05-26) – I’ve learnt about the Kumiko project, which is exactly devoted to slicing comic book panels. There’s even a live tool. I discovered it thanks to being pinged on this issue.

Motivation

I’ve recently been reading some comic books I used to devour as a kid. Especially those from the golden era of francophone comics: Thorgal, Lanfeust, XIII, Tintin, Largo Winch, Blacksad, Aldebaran, etc.

It’s not easy to get my hands on many of them. Luckily enough I found a website called ReadComicOnline which is delightfully profuse. It gives access to comics for free under the murky “fair use” copyright doctrine. I’m very doubtful about the legality of the website, but I’m still using it for lack of a better option.

Online machine learning in practice @ PyData PDX

maxhalford25@gmail.com (Max Halford) — Wed, 09 Feb 2022 00:00:00 +0000

The online machine learning predict/fit switcheroo

maxhalford25@gmail.com (Max Halford) — Thu, 06 Jan 2022 00:00:00 +0000

Why I’m writing this

Fact: designing open source software is hard. It’s difficult to make design decisions which don’t make any compromises. I like to fall back on Dieter Rams’ 10 principles for good design. I feel like they apply rather well to software design. Especially when said software is open source, due to the many users and the plethora of use cases.

I had to make a significant design decision for River. It boils down to the fact that making a prediction with a model pipeline is a stateful operation, whereas users understandably expect it to be pure with no side-effects. This regularly comes up on the issue tracker, as you can see here, here, and here.

Weighted sampling without replacement in pure Python

maxhalford25@gmail.com (Max Halford) — Fri, 24 Dec 2021 00:00:00 +0000

I’m working on a problem where I need to sample k items from a list without replacement. The sampling has to be weighted. In Python, numpy has random.choice method which allows doing this:

import numpy as np

n = 10
k = 3

np.random.seed(42)
population = np.arange(n)
weights = np.random.dirichlet(np.ones_like(population))

np.random.choice(population, size=k, replace=False, p=weights)

array([0, 9, 8])

I’m always wary of using numpy without thinking because I know it incurs some overhead. This overhead is usually meaningful when small amounts of data are involved. In such a case, a pure Python implementation may be faster.

Online machine learning in practice @ Applied AI

maxhalford25@gmail.com (Max Halford) — Fri, 17 Dec 2021 00:00:00 +0000

Online machine learning in practice @ LVMH

maxhalford25@gmail.com (Max Halford) — Fri, 10 Dec 2021 00:00:00 +0000

Web scraping, upside down

maxhalford25@gmail.com (Max Halford) — Thu, 11 Nov 2021 00:00:00 +0000

Motivation

Web scraping is the art of extracting information from web pages. A web page is essentially an amalgamation of HTML tags. Usually, we’re looking for a particular piece of information on a given web page. This may be done by fetching the HTML content of the page in question, and then running some HTML parsing logic. It’s quite straightforward.

There are many tools in the wild to perform web scraping. For instance, in Python, you may use requests in combination with Beautiful Soup. You can also automate some of the more mundane aspects of scraping by using Scrapy.

One year at Alan

maxhalford25@gmail.com (Max Halford) — Tue, 26 Oct 2021 00:00:00 +0000

Context

Today marks the 1 year anniversary since I started working at Alan. It’s my first real job, and certainly the place where I grew up the most professionally. I’m writing this post to summarise what I did and what I learnt at Alan.

Alan is a special company. It has a unique culture that is starting to become famous in France. I won’t expand on the way things work at Alan, and will simply focus on the way I experienced it. Let me just say this: it works. The pace at which stuff gets shipped is insane. And yet, it’s a healthy environment to be working in. Alaners are some of the kindest and wisest human beings I’ve had the chance to meet.

Manipulating ephemeral data with git

maxhalford25@gmail.com (Max Halford) — Thu, 07 Oct 2021 00:00:00 +0000

Dashboards and GROUPING SETS

maxhalford25@gmail.com (Max Halford) — Fri, 10 Sep 2021 00:00:00 +0000

Motivation

At Alan, we do almost all our data analysis in SQL. Our data warehouse used to be PostgreSQL, and have since switched to Snowflake for performance reasons. We load data into our warehouse with Airflow. This includes dumps of our production database, third-party data, and health data from other actors in the health ecosystem. This is raw data. We transform this into prepared data via an in-house tool that resembles dbt. You can read more about it here.

Homoglyphs: different characters that look identical

maxhalford25@gmail.com (Max Halford) — Thu, 19 Aug 2021 00:00:00 +0000

A wild homoglyph appears

For instance, can you tell if there’s a difference between H and Η? How about N and Ν? These characters may seem identical, but they are actually different. You can try this out for yourself in Python:

>>> 'H' == 'Η'
False

>>> 'N' == 'Ν'
False

Indeed, these all represent different Unicode characters:

>>> ord('H'), ord('Η')
(72, 919)

>>> ord('N'), ord('Ν')
(78, 925)

Η in fact represents the capital Eta letter, while Ν is a capital Nu. In fact, entering H or Η in Google will produce different results. The same goes for N and Ν.

Automated document processing at Alan

maxhalford25@gmail.com (Max Halford) — Thu, 10 Jun 2021 00:00:00 +0000

Text classification by data compression

maxhalford25@gmail.com (Max Halford) — Tue, 08 Jun 2021 00:00:00 +0000

Edit – I posted this on Hackernews and got some valuable feedback. Many brought up the fact that you should be able to reuse the internal state of the compressor instead of recompressing the training data each time a prediction is made. There’s also some insightful references to data compression theory and its ties to statistical learning

Edit (2025-06-29) – *Python 3.14 introduces compression.zstd, which implement’s Facebook’s Zstandard compression algorithm, as discussed in the comments section below.

Reducing the memory footprint of a scikit-learn text classifier

maxhalford25@gmail.com (Max Halford) — Sun, 11 Apr 2021 00:00:00 +0000

Context

This week at Alan I’ve been working on parsing French medical prescriptions. There are three types of prescriptions: lenses, glasses, and pharmaceutical prescriptions. Different information needs to be extracted depending on the prescription type. Therefore, the first step is to classify the prescription. The prescriptions we receive are pictures taken by users with their phone. We run each image through an OCR to obtain a text transcription of the image. We can thus use the text transcription to classify the prescription.

An overview of dataset time travel

maxhalford25@gmail.com (Max Halford) — Wed, 07 Apr 2021 00:00:00 +0000

TLDR

You’re a data scientist. The engineers in your company overwrite data in the production database. You want to access overwritten data to train your models. How?

I thought time travel only existed in the movies

You’re probably right, expect maybe for this guy.

I want to discuss a concept that’s been on my mind for a while now. I like to call it “dataset time travel” because it has a nice ring to it. But the association of “time travel” and “data” has already been used elsewhere. It’s not something I’m pulling out from thin air. Essentially, what I want to discuss is the ability to view a dataset at any given point in the past. Having this ability is powerful, as it allows answering important business questions. As an example, let’s say we have a database table called users. We might ask the following question:

The challenges of online machine learning in production @ Itaú Unibanco

maxhalford25@gmail.com (Max Halford) — Fri, 26 Feb 2021 00:00:00 +0000

Quelle est l’empreinte écologique du Big Data? @ Toulouse Tech

maxhalford25@gmail.com (Max Halford) — Fri, 22 Jan 2021 00:00:00 +0000

Organising a Kaggle InClass competition with a fairness metric

maxhalford25@gmail.com (Max Halford) — Thu, 21 Jan 2021 00:00:00 +0000

Some context

I co-organised a data science competition during the second half of 2020. This was in fact the 5th edition of the “Défi IA”, which is a recurring event that happens on a yearly basis. It is essentially a supervised machine learning competition for students from French speaking universities and engineering schools. This year was the first time that Kaggle was used to host the competition. Before that we used a custom platform that I wrote during my student years. You can read more about this here.

Converting Amazon Textract tables to pandas DataFrames

maxhalford25@gmail.com (Max Halford) — Thu, 14 Jan 2021 00:00:00 +0000

I’m currently doing a lot of document processing at work. One of my tasks is to extract tables from PDF files. I evaluated Amazon Textract’s table extraction capability as part of this task. It’s very well documented, as is the rest of Textract. I was slightly disappointed by the examples, but nothing serious.

I wanted to write this short blog post to share a piece of code I use to convert tables extracted through Amazon Textract to pandas.DataFrames. I’ll be using the following anonymised image as an example:

What my PhD was about

maxhalford25@gmail.com (Max Halford) — Wed, 06 Jan 2021 00:00:00 +0000

I defended my PhD thesis on the 12th of October 2020, exactly 3 years and 11 days after having started it. The title of my PhD is Machine learning for query selectivity estimation in relational databases. I thought it would be worthwhile to summarise what I did. Note sure anyone will read this, but at least I’ll be able to remember what I did when I grow old and senile.

Computing cross-correlations in SQL

maxhalford25@gmail.com (Max Halford) — Tue, 17 Nov 2020 00:00:00 +0000

Introduction

I’m currently working on a problem at work where I have to measure the impact of a growth initiative on a performance metric. Hypothetically, this might answer the following kind of question:

I’ve spent X amount of money, what is the impact on the number of visitors on my website?

Of course, there are many measures that can be taken to answer such a question. I decided to measure the correlation between the initiative and the metric, with the latter being shifted forward in time. This measure is called the cross-correlation. It’s different from serial correlation, which is the correlation of a series with a shifted version of itself.

Unsupervised text classification with word embeddings

maxhalford25@gmail.com (Max Halford) — Sat, 03 Oct 2020 00:00:00 +0000

Edit – since writing this article, I have discovered that the method I describe is a form of zero-shot learning. So I guess you could say that this article is a tutorial on zero-shot learning for NLP.

Edit – I stumbled on a paper entitled “Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings” which proposes something very similar. The paper is rather well written, so you might want to check it out. Note that they call the tech -> technology trick “label enrichment”.

Focal loss implementation for LightGBM

maxhalford25@gmail.com (Max Halford) — Sun, 20 Sep 2020 00:00:00 +0000

Edit (2021-01-26) – I initially wrote this blog post using version 2.3.1 of LightGBM. I’ve now updated it to use version 3.1.1. There are a couple of subtle but important differences between version 2.x.y and 3.x.y. If you’re using version 2.x.y, then I strongly recommend you to upgrade to version 3.x.y.

Motivation

If you’re reading this blog post, then you’re likely to be aware of LightGBM. The latter is a best of breed gradient boosting library. As of 2020, it’s still the go-to machine learning model for tabular data. It’s also ubiquitous in competitive machine learning.

A few intermediate pandas tricks

maxhalford25@gmail.com (Max Halford) — Mon, 17 Aug 2020 00:00:00 +0000

I want to use this post to share some pandas snippets that I find useful. I use them from time to time, in particular when I’m doing time series competitions on platforms such as Kaggle. Like any data scientist, I perform similar data processing steps on different datasets. Usually, I put repetitive patterns in xam, which is my personal data science toolbox. However, I think that the following snippets are too small and too specific for being added into a library.

A brief introduction to online machine learning @ Hong Kong Machine Learning Meetup

maxhalford25@gmail.com (Max Halford) — Wed, 10 Jun 2020 00:00:00 +0000

The correct way to evaluate online machine learning models

maxhalford25@gmail.com (Max Halford) — Sun, 07 Jun 2020 00:00:00 +0000

Motivation

Most supervised machine learning algorithms work in the batch setting, whereby they are fitted on a training set offline, and are used to predict the outcomes of new samples. The only way for batch machine learning algorithms to learn from new samples is to train them from scratch with both the old samples and the new ones. Meanwhile, some learning algorithms are online, and can predict as well as update themselves when new samples are available. This encompasses any model trained with stochastic gradient descent – which includes deep neural networks, factorisation machines, and SVMs – as well as decision trees, metric learning, and naïve Bayes.

Online machine learning with decision trees @ Toulouse AOC workgroup

maxhalford25@gmail.com (Max Halford) — Thu, 07 May 2020 00:00:00 +0000

Server-sent events in Flask without extra dependencies

maxhalford25@gmail.com (Max Halford) — Mon, 04 May 2020 00:00:00 +0000

Server-sent events (SSE) is a mechanism for sending updates from a server to a client. The fundamental difference with WebSockets is that the communication only goes in one direction. In other words, the client cannot send information to the server. For many usecases this is all you might need. Indeed, if you just want to receive notifications/updates/messages, then using a WebSocket is overkill. Once you’ve implemented the SSE functionality on your server, then all you need on a JavaScript client is an EventSource. Trust me, it’s very straightforward.

I got plagiarized and Google didn't help

maxhalford25@gmail.com (Max Halford) — Fri, 17 Apr 2020 00:00:00 +0000

One of my most popular articles is the one on target encoding. It gets a fair amount of mentions on Kaggle discussions and I see it pop up from time to time in other contexts. It also brings me around 2500 unique monthly viewers. That’s quite a chunk of people for an unambitious blogger like me. Up to a few months ago, my article was on the first page of Google when you typed in searches such as “target encoding python” and “bayesian target encoding”. This was purely organic and it felt nice to have a relevant article, even though that’s not the main reason why I blog.

Our solution to the IDAO 2020 qualifiers

maxhalford25@gmail.com (Max Halford) — Sun, 12 Apr 2020 00:00:00 +0000

Speeding up scikit-learn for single predictions

maxhalford25@gmail.com (Max Halford) — Tue, 31 Mar 2020 00:00:00 +0000

It is now common practice to train machine learning models offline before putting them behind an API endpoint to serve predictions. Specifically, we want an API route which can make a prediction for a single row/instance/sample/data point/individual (call it what you want). Nowadays, we have great tools to do this that take care of the nitty-gritty details, such as Cortex, MLFlow, Kubeflow, and Clipper. There are also paid services that hold your hand a bit more, such as DataRobot, H2O, and Cubonacci. One could argue that deploying machine learning models has never been easier.

Machine learning for streaming data with creme

maxhalford25@gmail.com (Max Halford) — Thu, 26 Mar 2020 00:00:00 +0000

Global explanation of machine learning with sensitivity analysis @ MASCOT-NUM

maxhalford25@gmail.com (Max Halford) — Tue, 10 Mar 2020 00:00:00 +0000

Bayesian linear regression for practitioners

maxhalford25@gmail.com (Max Halford) — Wed, 26 Feb 2020 00:00:00 +0000

Motivation

Suppose you have an infinite stream of feature vectors $x_i$ and targets $y_i$. In this case, $i$ denotes the order in which the data arrives. If you’re doing supervised learning, then your goal is to estimate $y_i$ before it is revealed to you. In order to do so, you have a model which is composed of parameters denoted $\theta_i$. For instance, $\theta_i$ represents the feature weights when using linear regression. After a while, $y_i$ will be revealed, which will allow you to update $\theta_i$ and thus obtain $\theta_{i+1}$. To perform the update, you may apply whichever learning rule you wish – for instance most people use some flavor of stochastic gradient descent. The process I just described is called online supervised machine learning. The difference between online machine learning and the more traditional batch machine learning is that an online model is dynamic and learns on the fly. Online learning solves a lot of pain points in real-world environments, mostly because it doesn’t require retraining models from scratch every time new data arrives.

Under-sampling a dataset with desired ratios

maxhalford25@gmail.com (Max Halford) — Tue, 17 Dec 2019 00:00:00 +0000

Introduction

I’ve just spent a few hours looking at under-sampling and how it can help a classifier learn from an imbalanced dataset. The idea is quite simple: randomly sample the majority class and leave the minority class untouched. There are more sophisticated ways to do this – for instance by creating synthetic observations from the minority class à la SMOTE – but I won’t be discussing that here.

I checked out the imblearn library and noticed they have an implementation of random under-sampling aptly named RandomUnderSampler. It contains a sampling_strategy parameter which gives some control over the sampling. By the default the observations are resampled so that each class is equally represented:

The benefits of online machine learning @ Quantmetry

maxhalford25@gmail.com (Max Halford) — Tue, 29 Oct 2019 00:00:00 +0000

The benefits of online machine learning @ Element AI

maxhalford25@gmail.com (Max Halford) — Wed, 23 Oct 2019 00:00:00 +0000

Finding fuzzy duplicates with pandas

maxhalford25@gmail.com (Max Halford) — Mon, 16 Sep 2019 00:00:00 +0000

Duplicate detection is the task of finding two or more instances in a dataset that are in fact identical. As an example, take the following toy dataset:

	First name	Last name	Email
0	Erlich	Bachman	eb@piedpiper.com
1	Erlich	Bachmann	eb@piedpiper.com
2	Erlik	Bachman	eb@piedpiper.co
3	Erlich	Bachmann	eb@piedpiper.com

Each of these instances (rows, if you prefer) corresponds to the same “thing” – note that I’m not using the word “entity” because entity resolution is a different, and yet related, concept. In my experience there are two main reasons why data duplication may occur:

A smooth approach to putting machine learning into production

maxhalford25@gmail.com (Max Halford) — Sat, 13 Jul 2019 00:00:00 +0000

Putting machine learning into production is hard. Usually I’m doubtful of such statements, but in this case I’ve never met anyone for whom everything has gone smoothly. Most data scientists might agree that there is a huge gap between their local environment and a live environment. In fact, “productionalizing” machine learning is such a complex topic that entire companies have risen to address the issue. I’m not just talking about running a gigantic grid search and finding the best model, I’m talking about putting a machine learning model live so that it actually has a positive impact on your business/project. Off the top of my head: Cubonacci, H2O, Google AutoML, Amazon Sagemaker, and DataRobot. In other words people are making money off businesses because data scientists and engineers are having a hard putting their models into production. In my opinion if a data scientist can’t put her model into production herself then something is wrong. Life should be simpler.

The benefits of online machine learning @ Airbus Bizlab

maxhalford25@gmail.com (Max Halford) — Fri, 28 Jun 2019 00:00:00 +0000

Machine learning incrémental: des concepts à la pratique @ Toulouse Data Science Meetup

maxhalford25@gmail.com (Max Halford) — Tue, 28 May 2019 00:00:00 +0000

Skyline queries in Python

maxhalford25@gmail.com (Max Halford) — Tue, 21 May 2019 00:00:00 +0000

Imagine that you’re looking to buy a home. If you have an analytical mind then you might want to tackle this with a quantitative. Let’s suppose that you have a list of potential homes, and each home has some attributes that can help you compare them. As an example, we’ll consider three attributes:

The price of the house, which you want to minimize
The size of the house, which you want to maximize
The city where the house if located, which you don’t really care about

Some houses will be objectively better than others because they will be cheaper and bigger. However, for some pairs of houses the comparison will not be as clear. It might be that house A is more expensive than house B but is also larger. In data analysis this set of best houses which are incomparable with each other is called a skyline. As they say, a picture is worth a thousand words, so let’s draw one.

Online machine learning with creme @ PyData Amsterdam

maxhalford25@gmail.com (Max Halford) — Sat, 11 May 2019 00:00:00 +0000

SQL subquery enumeration

maxhalford25@gmail.com (Max Halford) — Mon, 06 May 2019 00:00:00 +0000

I recently stumbled on a rather fun problem during my PhD. I wanted to generate all possible subqueries from a given SQL query. In this case an example is easily worth a 1000 thousand words. Take the following SQL query:

SELECT *
FROM
 customers AS c,
 purchases AS p,
 shops AS s
WHERE
 p.customer_id = c.id AND
 p.shop_id = s.id
AND
 c.nationality = 'Swedish' AND
 c.hair = 'Blond' AND
 s.city = 'Stockholm'

Here all the possible subqueries that can be generated from the above query.

An approach based on Bayesian networks for query selectivity estimation @ DASFAA

maxhalford25@gmail.com (Max Halford) — Mon, 22 Apr 2019 00:00:00 +0000

Morellet crosses with JavaScript

maxhalford25@gmail.com (Max Halford) — Sun, 03 Feb 2019 00:00:00 +0000

The days I’m working on a deep learning project. I hate it but I promised myself to give it a real try. My scripts are taking a long time so I decided to do some procedural art while I waited. This time I’m going to reproduce the following crosses made by François Morellet. I saw them the last I went to the Musée Pompidou with some friends from university. I don’t have any smartphone anymore so one my friends was kind enough to take a few pictures for me, including this one. The painting is called Violet, bleu, vert, jaune, orange, rouge.

Streaming groupbys in pandas for big datasets

maxhalford25@gmail.com (Max Halford) — Wed, 05 Dec 2018 00:00:00 +0000

If you’ve done a bit of Kaggling, then you’ve probably been typing a fair share of df.groupby(some_col). That is, if you’re using Python. If you’re handling tabular data, then a lot of your features will revolve around computing aggregate statistics. This is very true for the ongoing PLAsTiCC Astronomical Classification challenge. The goal of the competition is to classify objects in the sky into one of 14 groups. The bulk of the available data is a set of so-called light curve. A light curve is a sequence of brightness measures observations along time. Each light curve is filtered at different passbands. The idea is that there is one light curve per passband and per object and that the shape of each light curve should tell us what kind of object we’re looking at. Yada yada.

Target encoding done the right way

maxhalford25@gmail.com (Max Halford) — Sat, 13 Oct 2018 00:00:00 +0000

When you’re doing supervised learning, you often have to deal with categorical variables. That is, variables which don’t have a natural numerical representation. The problem is that most machine learning algorithms require the input data to be numerical. At some point or another a data science pipeline will require converting categorical variables to numerical variables.

There are many ways to do so:

Label encoding where you choose an arbitrary number for each category
One-hot encoding where you create one binary column per category
Vector representation a.k.a. word2vec where you find a low dimensional subspace that fits your data
Optimal binning where you rely on tree-learners such as LightGBM or CatBoost
Target encoding where you average the target value by category

Each and every one of these method has its own pros and cons. The best approach typically depends on your data and your requirements. If a variable has a lot of categories, then a one-hot encoding scheme will produce many columns, which can cause memory issues. In my experience, relying on LightGBM/CatBoost is the best out-of-the-box method. Label encoding is useless and you should never use it. However if your categorical variable happens to be ordinal then you can and should represent it with increasing numbers (for example “cold” becomes 0, “mild” becomes 1, and “hot” becomes 2). Word2vec and others such methods are cool and good but they require some fine-tuning and don’t always work out of the box.

Stella triangles with JavaScript

maxhalford25@gmail.com (Max Halford) — Thu, 26 Apr 2018 00:00:00 +0000

Around the same time last year I visited the San Francisco Museum of Modern Art. Frank Stella’s compositions really caught my eye. When I saw them I started thinking about how I could write a computer program to imitate his work. In this post I’m going to attempt to reproduce his so-called V Series.

Nice and simple right? Indeed in a lot of his work Frank Stella uses straight lines without much randomness. There are quite a few prints in the V Series. However in each one of them the common denominator is a single triangle. If we have a routine for drawing one triangle then we can use to make compositions. As always let’s start by creating a canvas.

Unknown pleasures with JavaScript

maxhalford25@gmail.com (Max Halford) — Mon, 24 Jul 2017 00:00:00 +0000

No this blog post is not about how nice JavaScript can be, instead it’s just another one of my attempts at reproducing modern art with procedural generation and the HTML5 <canvas> element. This time I randomly generated images resembling the cover of the album by Joy Division called “Unknown Pleasures”.

According to Wikipedia, this somewhat iconic album cover is based on radio waves. I saw a poster of it in a bar not long ago and decided to reproduce the next time I had some time to kill.

Subsampling a training set to match a test set - Part 1

maxhalford25@gmail.com (Max Halford) — Mon, 19 Jun 2017 00:00:00 +0000

Edit – it’s 2022 and I still haven’t written a part 2. That’s because I believe this problem is easily solved with adversarial validation.

Some friends and I recently qualified for the final of the 2017 edition of the Data Science Game competition. The first part was a Kaggle competition with data provided by Deezer. The problem was a binary classification task where one had to predict if a user was going to listen to a song that was proposed to him. Like many teams we extracted clever features and trained an XGBoost classifier, classic. However, the one special thing we did was to subsample our training set so that it was more representative of the test set.

Docker for data science @ HelloFresh Berlin

maxhalford25@gmail.com (Max Halford) — Thu, 01 Jun 2017 00:00:00 +0000

Halftoning with Go - Part 2

maxhalford25@gmail.com (Max Halford) — Mon, 20 Mar 2017 00:00:00 +0000

The next stop on my travel through the world of halftoning will be the implementation of Weighted Voronoi Stippling as described in Adrian Secord’s 2002 paper. This method is more involved than the ones I detailed in my previous blog post, however the results are quite interesting. Again, I did the implementation in Go.

Notice the black dot in the middle of the white square?

Overview

I found a fair amount of resources about the method, most of them being implementations of Adrian Secord’s paper. However, not many of these resources went into the nitty-gritty details which are not obvious for beginners in image processing. Before delving into the code, I want to go through some concepts that may seem obvious to some readers but that I judge worthy of detailing.

Grid paintings à la Mondrian with JavaScript

maxhalford25@gmail.com (Max Halford) — Sat, 04 Mar 2017 00:00:00 +0000

I was at a laundrette today and had just finished my book so I had some time to kill. Naturally I devised an algorithm for generating drawings that would resemble the grid-like paintings that Piet Mondrian made famous. With the benefit of hindsight I guess I could indulge in saner activities while waiting for my laundry to dry!

I went through different ideas but in the end I settled on a recursive approach. My idea is to divide a rectangle into two smaller ones and then to do the same with each sub-rectangle. Every time a rectangle is generated and is then filled with a random color; like Mondrian I use yellow, red, blue, black and white.

A short introduction and conclusion to the OpenBikes 2016 Challenge

maxhalford25@gmail.com (Max Halford) — Thu, 26 Jan 2017 00:00:00 +0000

During my undergraduate internship in 2015 I started a side project called OpenBikes. The idea was to visualize and analyze bike sharing over multiple cities. Axel Bellec joined me and in 2016 we won a national open data competition. Since then we haven’t pursued anything major, instead we use OpenBikes to try out technologies and to apply concepts we learn at university and online.

Before the 2016 summer holidays one of my professors, Aurélien Garivier mentioned that he was considering using our data for a Kaggle-like competition between some statistics curriculums in France. Near the end of the summer, I sat down with a group of professors and we decided upon a format for the so-called “Challenge”. The general idea was to provide student teams with historical data on multiple bike stations and ask them to do some forecasting which we would then score based on a secret truth. The whole thing lasted from the 5th of October 2016 till the 26th of January 2017 when the best team was crowned.

Challenge Big Data @ TSE

maxhalford25@gmail.com (Max Halford) — Mon, 09 Jan 2017 00:00:00 +0000

Halftoning with Go - Part 1

maxhalford25@gmail.com (Max Halford) — Sun, 27 Nov 2016 00:00:00 +0000

Recently I stumbled upon this webpage which shows how to use a TSP solver as a halftoning technique. I began to read about related concepts like dithering and stippling. I don’t have any background in photography but I can appreciate the visual appeal of these techniques. As I understand it these techniques were first invented to save ink for printing. However nowadays printing has become cheaper and the modern use of these technique is mostly aesthetic, at least for images.

Predire la disponibilité des Velib' @ Toulouse Data Science Meetup

maxhalford25@gmail.com (Max Halford) — Wed, 30 Mar 2016 00:00:00 +0000

Recursive polygons with JavaScript

maxhalford25@gmail.com (Max Halford) — Fri, 25 Mar 2016 00:00:00 +0000

I like modern art, I enjoy looking at the stuff that was made at the beginning of the 20th century and thinking how it is still shaping today’s style. I’m not an expert, it’s just a hobby of mine. I especially like the Centre Pompidou in Paris, it’s got loads of fascinating stuff. While I was going through the galleries it struck me that some of the paintings were very geometrical. In fact they were so geometrical that a machine could have produced them! I’m not talking about artificial intelligence but rather a set of rules that could be given to a programming language. Through a series of blog posts I would like to try to emulate some works with my computer. I realize it’s a waste of time but it’s a good opportunity for me to learn some more JavaScript and refreshen my geometry. I also want to insist on making these drawings random, not deterministic.

The Naïve Bayes classifier

maxhalford25@gmail.com (Max Halford) — Thu, 10 Sep 2015 00:00:00 +0000

The objective of a classifier is to decide to which class (also called label) to assign an observation based on observed data. In supervised learning, this is done by taking into account previous classifications. In other words if we know that certain observations are classified in a certain way, the goal is to determine the class of a new observation. The first group of observations on which the classifier is built is called the training set.

An introduction to genetic algorithms

maxhalford25@gmail.com (Max Halford) — Sun, 02 Aug 2015 00:00:00 +0000

The goal of genetic algorithms (GAs) is to solve problems whose solutions are not easily found (ie. NP problems, nonlinear optimization, etc.). For example, finding the shortest path from A to B in a directed graph is easily done with Djikstra’s algorithm, it can be solved in polynomial time. However the time to find the smallest path that joins all points on a non-directed graph, also known as the Travelling Salesman Problem (TSP) increases exponentially as the number of points increases. More generally, GAs are useful for problems where an analytical approach is complicated or even impossible. By giving up on perfection they manage to find a good approximation of the optimal solution.

Setting up a droplet to host a Flask app

maxhalford25@gmail.com (Max Halford) — Tue, 14 Jul 2015 00:00:00 +0000

After having worked for some weeks on the OpenBikes website, it was time to put it online. Digital Ocean seemed to provide a good service and so I decided to give it a spin. Their documentation is quite good but it doesn’t cover exactly everything for setting up Flask. In this post I simply want to record every single step I took.

OpenBikes is a project with a Flask backend and a few upstart jobs. It lives at the openbikes.co domain name. In this blog post I will list every step it takes to make it happen on Ubuntu 14.04 with Apache (it’s robust and easy to setup). I didn’t always say when to use sudo before the commands to avoid clutter, however you can safely use it everywhere.

Visualizing bike stations live data

maxhalford25@gmail.com (Max Halford) — Wed, 03 Jun 2015 00:00:00 +0000

Recently some friends and I decided to launch openbikes.co, a website for visualizing (and later on analyzing) urban bike traffic. We have a lot of ideas that we will progressively implement. Anyway, the point is that all of it started with me fiddling about with the JCDecaux API and the leaflet.js library and I would like to share it with you. Shall we?

Presentation

In this post I want to show you the tools and the code to get a fully functional website for visualizing live data. In this particular case we will display bike stations in Toulouse, however I will keep the scripts as general as possible so they are easily modifiable for different data. Before starting here is a glimpse of the end result:

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

Incremental machine learning

## Online machine learning with creme

### Max Halford

#### 11th of May 2019, Amsterdam

</div>

???

Hello!

---

### Outline

.bullets[
1. You're not doing ML the correct way
2. Cool kids do online learning
3. Introducing `creme`, the fresh kid on the block
4. Forecasting League of Legends match durations
5. Future work
]

???

My goal is to convince you that the way most people do machine learning isn't the way to go in production scenarios. Online learning is more suitable and nullifies many issues. I'll introduce a new library me and some friends have been working on, and I'll show you we're using for a pet project
trying to forecast the duration of League of Legends matches.

---

### Batch learning

.bullets[
1. Collect features $X$ and labels $Y$
2. Train a model on $(X, Y)$
3. Save the model somewhere
4. Load the model to make predictions
]

With code:

```python
>>> model.fit(X_train, y_train)
>>> dump(model, 'model.json')
>>> model = load('model.json')
>>> y_pred = model.predict(X_test)
```

???

The most common form of machine learning is called batch learning. It's what you do when you use scikit-learn and do Kaggle competitions. I won't dwell on this too long as I'm pretty sure you're all acquainted with it. In short: load, fit, predict.

---

### Batch machine learning in production

???

There isn't a clear pattern as to how to put into production a batch learning system. Typically this involves training a model periodically, and then serializing it before saving it somewhere. API calls can then be issues to make predictions. This is such a difficult thing to get right that startups and tech giants are building products around it.

---

### Batch machine learning in production is hard

???

Works fine for Kaggle cases and when you have a clear split between train and test. But things are not so smooth in production.

---

background-color: #2ac380
class: center, middle, white

# Models have to be retrained from scratch with new data

---

background-color: #dbaaa8
class: center, middle, white

# Ever increasing computational requirements

---

background-color: #1f282d
class: center, middle, white

# Models are static and rot faster than bananas 🍌

---

background-color: #e69138
class: center, middle, white

# Features that you develop locally might not be available in real-time

???

This is more subtle. Sometimes it isn't possible to recreate features for past instances. If you don't store information for an instance at every particular point in time, then it isn't possible to reproduce the true state during your feature extraction process. In other words you need a solid data engineering team that cares about time.

---

> It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.

???

And just like Winnie the Pooh, we're spending too much time banging our heads to be able to think about a better way of doing things.

---

background-color: #607bd4
class: middle, white

## Online learning

.bigbullets[
- Data comes from a stream
- Models learn 1 observation at a time
- Observations do not have to be stored
- Features and labels are dynamic
]

---

background-color: #FF7F50
class: middle, white

## Different names, same thing

.bigbullets[
- Online learning
- Incremental learning
- Sequential learning
- Iterative learning
- Out-of-core learning
]

---

background-color: #008080
class: middle, white

## Applications

.bigbullets[
- Time series forecasting
- Spam filtering
- Recommender systems
- Ad placement
- Internet of things
- Basically, <span style="text-decoration:underline">anything event based</span>
]

---

### Online learning in a nutshell

---

background-color: #e66868ff
class: middle, white

## Why is batch learning so popular?

.bigbullets[
- Taught at university
- (Bad) habits
- Hype
- Kaggle
- Library availability
- Higher accuracy
]

---

.bullets[
- Python library for doing online learning
- API heavily inspired from `sklearn` and easy to pick up
- Focus on feature extraction, not just learning
- First commit in January 2019
- Version `0.1.0` released earlier this week
]

---

### Observations

Representing an observation with a `dict` is natural:

```python
x = {
 'date': dt.datetime(2019, 4, 22),
 'price': 42.95,
 'shop': 'Ikea'
}
```

- Values can be of any type
- Feature names can be used instead of array indexes
- Python's standard library plays nicely with `dict`s

---

### Targets

A target's type depends on the context:

```python
# Regression
y = 42

# Binary classification
y = True

# Multi-class classification
y = 'setosa'

# Multi-output regression
y = {
 height: 29.7,
 width: 21
}
```

---

### Streaming data

```python
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

for x, y in X_y:
 print(x, y)
```

- `X_y` is a **generator** and consumes a tiny amount of memory
- The point is that we want to **handle data points one by one**
- Source depends on your use case (can be Kakfa producer, CSV file, HTTP requests, etc.)

---

### Training with `fit_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
 model.fit_one(x, y)
```

Every `creme` estimator has a `fit_one` method

---

### Predicting with `predict_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* y_pred_before = model.predict_one(x)
 model.fit_one(x, y)
* y_pred_after = model.predict_one(x)
```

- Classifiers also have a `predict_proba_one` method
- Transformers have a `transform_one` method
- Training and predicting phases are inter-leaved

---

### Monitoring performance

```python
from creme import linear_model
from creme import metrics
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

* metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
* metric.update(y, y_pred)
 print(metric)
```

Validation score is available for free! No need to do any cross-validation. You can also use `online_score` from the `model_selection` module.

---

### Composing estimators is easy

```python
from creme import linear_model
from creme import metrics
from creme import preprocessing
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

scale = preprocessing.StandardScaler()
lin_reg = linear_model.LogisticRegression()

* model = scale | lin_reg # Pipeline shorthand

metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
 metric.update(y, y_pred)
 print(metric)
```

---

### `creme`'s current modules

.left-column[
- `cluster`
- `compat`
- `compose`
- `datasets`
- `dummy`
- `ensemble`
- `feature_extraction`
- `feature_selection`
- `impute`
- `linear_model`
]
.right-column[
- `model_selection`
- `multiclass`
- `naive_bayes`
- `optim`
- `preprocessing`
- `reco`
- `stats`
- `stream`
- `tree`
- `utils`
]

---

### Online mean

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$

```python
>>> mean = creme.stats.Mean()

>>> mean.update(5)
>>> mean.get()
5

>>> mean.update(10)
>>> mean.get()
7.5
```

---

### Online variance

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$
3. $s\_{i+1} = s\_i + (x - \mu\_{i}) \times (x - \mu\_{i+1})$ (running sum of squares)
4. $\sigma\_{i+1} = \frac{s\_{i+1}}{n}$

```python
>>> variance = creme.stats.Variance()

>>> X = [2, 3, 4, 5]
>>> for x in X:
... variance.update(x)
>>> variance.get()
1.25

>>> numpy.var(X)
1.25

```

???

This is called Welford's algorithm, it can be extended to skew and kurtosis

---

### Standard scaling

With the mean and the variance, we can scale incoming data so that it has zero mean and unit variance.

```python
>>> scaler = creme.preprocessing.StandardScaler()

>>> for x in [2, 3, 4, 5]:
... print(x, 'becomes', scaler.fit_one(x)['x'])
2 becomes 0.0
3 becomes 0.9999999999999996
4 becomes 1.224744871391589
5 becomes 1.3416407864998738

```

---

### Linear regression (1)

Model is $y_t = \langle w_t x_t \rangle + b_t$. The weights $w_t$ can be learnt with any online gradient descent algorithm, for example:

- Stochastic gradient descent (SGD)
- Adam
- RMSProp
- Follow the Regularized Leader (FTRL)

```python
from creme import linear_model
from creme import optim

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01)
)
```

---

### Linear regression (2)

The intercept term $b_t$ is difficult to learn

Some people (Léon Bottou, scikit-learn) suggest to use a lower learning rate for the intercept than for the weights (heuristic but okay)

`creme` uses any running statistic from the `creme.stats` module, which is a powerful trick

```python
from creme import linear_model
from creme import optim
from creme import stats

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01),
 intercept=stats.RollingMean(42)
)
```

---

### Online aggregations

```python
>>> import creme

>>> X = [
... {'place': 'Taco Bell', 'revenue': 42},
... {'place': 'Burger King', 'revenue': 16},
... {'place': 'Burger King', 'revenue': 24},
... {'place': 'Taco Bell', 'revenue': 58}
... ]

>>> agg = creme.feature_extraction.Agg(
... on='revenue',
... by='place',
... how=creme.stats.Mean()
... )

>>> for x in X:
... print(agg.fit_one(x).transform_one(x))
{'revenue_mean_by_place': 42.0}
{'revenue_mean_by_place': 16.0}
{'revenue_mean_by_place': 20.0}
{'revenue_mean_by_place': 50.0}
```

---

### Bagging (1)

Bagging is a popular and simple ensemble algorithm:

1. Pick $m$ base models (usually identical copies)
2. Train each model on a sample with replacement
3. Average the predictions of each model on the test set

Each observation is sampled $K$ times where $K$ follows a binomial distribution:

$$P(K=k) = {n \choose k} \times (\frac{1}{n})^k \times (1 - \frac{1}{n})^{n-k}$$

As $n$ grows towards infinity, $K$ can be approximated by a Poisson(1):

$$P(K=k) \sim \frac{e^{-1}}{k!} $$

---

### Bagging (2)

`ensemble.BaggingClassifier` is very simple:

```python
def fit_one(self, x, y):

for estimator in self.estimators:
 for _ in range(self.rng.poisson(1)):
 estimator.fit_one(x, y)

return self

def predict_proba_one(self, x):
 y_pred = statistics.mean(
 estimator.predict_proba_one(x)[True]
 for estimator in self.estimators
 )
 return {
 True: y_pred,
 False: 1 - y_pred
 }
```

---

### League of Legends match duration forecasting (1)

---

### League of Legends match duration forecasting (2)

---

### Architecture

---

### Django model

```python
from django.db import models
from picklefield.fields import PickledObjectField

class CremeModel(models.Model):
 name = models.TextField(unique=True)
 pipeline = PickledObjectField()

class Meta:
 db_table = 't_models'
 verbose_name_plural = 'models'

def fit_one(self, x, y):
 self.pipeline.fit_one(x, y)
 return self

def predict_one(self, x):
 return self.pipeline.predict_one(x)
```

---

### Code for predicting

```python
match = fetch_match(match_id)

model = models.CremeModel.objects.get(name='My awesome model')

duration = model.predict_one(match.raw_info)

match.predicted_ended_at = match.started_at + predicted_duration
match.predicted_by = model
match.save()
```

---

### Code for training

When the match ends, it has a `true_duration` property.

```python
model = models.CremeModel.objects.get(id=match.predicted_by.id)

model.fit_one(
 x=match.raw_info,
 y=match.true_duration.seconds
)

model.save()
```

---

### Code for calculating performance

Just some Django black magic.

```python
duration = ExpressionWrapper(
 Func(F('predicted_ended_at') - F('ended_at'), function='ABS'),
 output_field=fields.DurationField()
)

agg = models.Match.objects.exclude(ended_at__isnull=True)\
 .annotate(duration=duration)\
 .aggregate(Avg('duration'))

avg_error = agg['duration__avg']
```

---

### Benefits of online learning

.bullets[
- No need to schedule model training
- Easy to debug and to monitor
- You're not "far" from production
- Way more fun than batch learning
]

---

### Future work

.left-column[
.bullets[
- Decision trees (nearly there!)
- Gradient boosting
- Bayesian linear models
- More feature extraction
- More models
- More benchmarks
- Many issues [on GitHub](https://github.com/creme-ml/creme/issues)
]
]
.right-column[

]

---

### If you want to contribute

- [creme-ml.github.io](https://creme-ml.github.io/)
- [github.com/creme-ml](https://github.com/creme-ml/)
- You can shoot emails to [maxhalford25@gmail.com](mailto:maxhalford25@gmail.com)
- Get in contact if you want to try `creme` and want advice
- Spread the word!

---

# Thanks for listening!

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

Machine learning incrémental: des concepts à la pratique

## Machine learning incrémental: des concepts à la pratique

### Max Halford

#### 28 mai 2019

#### Toulouse Data Science Meetup

</div>

???

Hello!

---

### Outline

.bullets[
1. You're doing machine learning the wrong way 😱
2. Cool kids do online learning 😎
3. Introducing `creme`, a Python lib for online learning 🐍
4. Bike stations forecasting demo 🚲 🔮
]

---

### PyData Amsterdam 2019 🐍 🇳🇱 🧀

---

### Batch learning

.bullets[
1. Collect features $X$ and labels $Y$
2. Train a model on $(X, Y)$
3. Save the model somewhere
4. Load the model to make predictions
]

With code:

```python
>>> model.fit(X_train, y_train)
>>> dump(model, 'model.json')
>>> model = load('model.json')
>>> y_pred = model.predict(X_test)
```
---

### Batch machine learning in production

---

background-color: #2ac380
class: center, middle, white

# Models have to be retrained from scratch with new data ➰➰➰

---

background-color: #663399
class: center, middle, white

# Models needs increasing amounts of power 🔌

---

background-color: #1f282d
class: center, middle, white

# Models are static and "rot" faster than bananas 🍌

---

background-color: #e69138
class: center, middle, white

# Models that work locally don't always work in production 😭

---

> It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.

???

And just like Winnie the Pooh, we're spending too much time banging our heads to be able to think about a better way of doing things.

---

background-color: #607bd4
class: middle, white

## Online learning

.bigbullets[
- Data comes from a stream
- Models learn 1 observation at a time
- Features and labels are dynamic
]

---

## Everything changes 💥

</div>
</div>

---

background-color: #008080
class: middle, white

## Different names, same thing 🤷

.bigbullets[
- Online learning
- Incremental learning
- Sequential learning
- Iterative learning
- Continuous learning
- Out-of-core learning
]

---

background-color: #FF7F50
class: middle, white

## Applications

.bigbullets[
- Time series forecasting
- Spam filtering
- Recommender systems
- Ad placement
- Internet of things
- Basically, <span style="text-decoration:underline">anything event based</span>
]

---

### Online learning in a nutshell

---

background-color: #e66868ff
class: middle, white

## Why is batch learning so popular?

---

# Questions?

---

.bullets[
- Online machine learning library for Python 🐍
- Easy to pick up API inspired by `sklearn`
- Written with production scenarios in mind
- First commit in January 2019
- Version `0.2.0` released yesterday
]

---

#### scikit-learn

```python
from sklearn import datasets
from sklearn import linear_model

X, y = datasets.load_boston(return_X_y=True)
model = linear_model.LinearRegression()

model.fit(X, y)
```

#### creme

```python
from creme import linear_model
from creme import stream
from sklearn import datasets

X_y = stream.iter_sklearn_dataset(datasets.load_boston)
model = linear_model.LinearRegression()

for x, y in X_y:
 model.fit_one(x, y)
```

---

### Features

Representing a set of features using a `dict` is natural:

```python
x = {
 'date': dt.datetime(2019, 4, 22),
 'price': 42.95,
 'shop': 'Ikea'
}
```

- Values can be of any type
- Feature names can be used instead of array indexes
- Python's standard library plays nicely with `dict`s

---

### Targets

A target's type depends on the context:

```python
# Regression
y = 42

# Binary classification
y = True

# Multi-class classification
y = 'setosa'

# Multi-output regression
y = {
 height: 29.7,
 width: 21
}
```

---

### Streaming data

```python
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

for x, y in X_y:
 print(x, y)
```

- `X_y` is a **generator** and consumes a tiny amount of memory
- The point is that we only need one data point at a time
- Source depends on your use case (CSV file, Kafka consumer, HTTP requests)

---

### Training with `fit_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* model.fit_one(x, y)
```

Every `creme` estimator has a `fit_one` method

---

### Predicting with `predict_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* y_pred = model.predict_one(x)
 model.fit_one(x, y)
```

- Classifiers also have a `predict_proba_one` method
- Transformers have a `transform_one` method
- Training and predicting phases are inter-leaved

---

### Progressive validation 💯

```python
from creme import linear_model
from creme import metrics
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
* metric.update(y, y_pred)
 print(metric)
```

Validation score is available for free! No need for cross-validation. You can also use `online_score` from the `model_selection` module.

---

### Composing estimators is easy

```python
from creme import compose
from creme import linear_model
from creme import preprocessing

scale = preprocessing.StandardScaler()
lin_reg = linear_model.LogisticRegression()

# You can do this...
model = compose.Pipeline([
 ('scale', scale),
 ('lin_reg', lin_reg
])

# Or this...
model = scale | lin_reg
```

---

# Questions?

---

### Online mean

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$

```python
>>> mean = creme.stats.Mean()

>>> mean.update(5)
>>> mean.get()
5

>>> mean.update(10)
>>> mean.get()
7.5
```

---

### Online variance

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$
3. $s\_{i+1} = s\_i + (x - \mu\_{i}) \times (x - \mu\_{i+1})$
4. $\sigma\_{i+1} = \frac{s\_{i+1}}{n}$

```python
>>> variance = creme.stats.Variance()

>>> X = [2, 3, 4, 5]
>>> for x in X:
... variance.update(x)
>>> variance.get()
1.25

>>> numpy.var(X)
1.25

```

???

This is called Welford's algorithm, it can be extended to skew and kurtosis

---

### Standard scaling

Using the mean and the variance, we can rescale incoming data.

```python
>>> scaler = creme.preprocessing.StandardScaler()

>>> for x in [2, 3, 4, 5]:
... features = {'x': x}
... scaler.fit_one(features)
... new_x = scaler.transform_one(features)['x']
... print(f'{x} becomes {new_x})
2 becomes 0.0
3 becomes 0.9999999999999996
4 becomes 1.224744871391589
5 becomes 1.3416407864998738

```

---

### Linear regression (1)

Model is $y_t = \langle w_t x_t \rangle + b_t$. The weights $w_t$ can be learnt with any online gradient descent algorithm, for example:

- Stochastic gradient descent (SGD)
- Adam
- RMSProp
- Follow the Regularized Leader (FTRL)

```python
from creme import linear_model
from creme import optim

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01)
)
```

---

### Linear regression (2)

Some people (Léon Bottou, scikit-learn) suggest to use a lower learning rate for the intercept than for the weights (heuristic but okay)

`creme` uses any running statistic from the `creme.stats` module, which is a powerful trick

```python
from creme import linear_model
from creme import optim
from creme import stats

lin_reg = linear_model.LinearRegression(
 optimizer=optim.Adam(lr=0.01),
 intercept=stats.RollingMean(42)
)
```

---

### Online aggregations

```python
>>> import creme

>>> X = [
... {'meal': 🍕, 'sales': 42},
... {'meal': 🍔, 'sales': 16},
... {'meal': 🍔, 'sales': 24},
... {'meal': 🍕, 'sales': 58}
... ]

>>> agg = creme.feature_extraction.Agg(
... on='sales',
... by='meal',
... how=creme.stats.Mean()
... )

>>> for x in X:
... print(agg.fit_one(x).transform_one(x))
{'sales_mean_by_meal': 42.0}
{'sales_mean_by_meal': 16.0}
{'sales_mean_by_meal': 20.0}
{'sales_mean_by_meal': 50.0}
```

---

### Bagging (1)

Each observation is sampled $K$ times where $K$ follows a binomial distribution:

$$P(K=k) = {n \choose k} \times (\frac{1}{n})^k \times (1 - \frac{1}{n})^{n-k}$$

As $n$ grows towards infinity, $K$ can be approximated by a Poisson(1):

$$P(K=k) \sim \frac{e^{-1}}{k!} $$

This leads to a simple and efficient online algorithm.

---

### Bagging (2)

`ensemble.BaggingClassifier` is very simple:

```python
def fit_one(self, x, y):

for estimator in self.estimators:
 for _ in range(self.rng.poisson(1)):
 estimator.fit_one(x, y)

return self

def predict_proba_one(self, x):
 y_pred = statistics.mean(
 estimator.predict_proba_one(x)[True]
 for estimator in self.estimators
 )
 return {
 True: y_pred,
 False: 1 - y_pred
 }
```

---

### Decision trees 🌳

- A version of Hoeffding trees is being implemented
- Basic idea:
 - Start with a leaf 🍃
 - Find the leaf where an observation belongs 🔎
 - Update the leaf's sufficient statistics 📊
 - Measure information gain every so often 🔬
 - Split when the information gain is good enough 🍂
- Mondrian trees 👨‍🎨 are another possibility but they only work for continuous attributes

---

# Questions?

---

### `creme`'s current modules

<div style="display: flex; justify-content: space-around;">
 <ul>
 <li><pre>cluster</pre></li>
 <li><pre>compat</pre></li>
 <li><pre>compose</pre></li>
 <li><pre>datasets</pre></li>
 <li><pre>dummy</pre></li>
 <li><pre>ensemble</pre></li>
 <li><pre>feature_extraction</pre></li>
 </ul>
 <ul>
 <li><pre>feature_selection</pre></li>
 <li><pre>impute</pre></li>
 <li><pre>linear_model</pre></li>
 <li><pre>model_selection</pre></li>
 <li><pre>multiclass</pre></li>
 <li><pre>naive_bayes</pre></li>
 <li><pre>optim</pre></li>
 <li><pre>plot</pre></li>
 </ul>
 <ul>
 <li><pre>preprocessing</pre></li>
 <li><pre>proba</pre></li>
 <li><pre>reco</pre></li>
 <li><pre>stats</pre></li>
 <li><pre>stream</pre></li>
 <li><pre>tree</pre></li>
 <li><pre>utils</pre></li>
 </ul>

</div>

---

### Cool stuff in `creme` we skipped 😢

.bullets[
- Clustering
- Factorization machines
- Feature selection
- Passive-aggressive models
- Recommender systems
- Histograms
- Skyline queries
- Fourier transforms
- Imputation
- Naive Bayes
]

---

### Alternative frameworks

---

### Benefits of online learning

.bullets[
- No need to schedule model training
- Easy to monitor
- You're very close to production
- Way more fun than batch learning
]

---

### Current work

.bullets[
- Decision trees (nearly there)
- Gradient boosting (easy)
- Bayesian linear models (part of my PhD)
- Latent Dirichlet Allocation (ask Raphael)
- Many issues [on GitHub](https://github.com/creme-ml/creme/issues)
]

---

### What next?

- [creme-ml.github.io](https://creme-ml.github.io/)
- [github.com/creme-ml](https://github.com/creme-ml/)
- You can send emails to [maxhalford25@gmail.com](mailto:maxhalford25@gmail.com)
- Get in touch if you want help and/or advice
- Starring us on GitHub helps a lot 🌟

---

# Thanks for listening!

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

The Benefits of Online Learning

## The Benefits of Online Learning

### (and other shenanigans)

#### Max Halford

</div>

???

Hello!

---

## Outline

---

### A bit about me

.right-column[
- 3rd year PhD student in Toulouse
- PhD on Bayesian networks applied to database cost models
- Topics of interest:
 - Online machine learning
 - Systems for machine learning
 - Machine learning for systems
 - Competitive machine learning
 - Fair and explainable learning
- Into opensource (mostly Python)
- Kaggle Master (rank 247)
]

---

### Batch learning in a nutshell

.bullets[
1. Collect features $X$ and labels $Y$
2. Train a model on $(X, Y)$
3. Save the model somewhere
4. Load the model to make predictions
]

With code:

```python
>>> model.fit(X_train, y_train)
>>> dump(model, 'model.json')
>>> model = load('model.json')
>>> y_pred = model.predict(X_test)
```

---

## Lambda architecture

---

background-color: #e66868ff
class: middle, white

## Batch learning in production has issues

1. Models are retrained from scratch with new data 🕒
2. Models require powerful machines 💰
3. Models are static and "rot" faster than bananas 🍌
4. Models that work in development don't always work in production 🤷
]

???

- [Continuum: a platform for cost-aware low-latency continual learning](https://blog.acolyer.org/2018/11/21/continuum-a-platform-for-cost-aware-low-latency-continual-learning/)
- [Applied machine learning at Facebook: a datacenter infrastructure perspective](https://blog.acolyer.org/2018/12/17/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/)

As we looked at last month with Continuum, the latency of incorporating the latest data into the models is also really important. There’s a nice section of this paper where the authors study the impact of losing the ability to train models for a period of time and have to serve requests from stale models. The Community Integrity team for example rely on frequently trained models to keep up with the ever changing ways adversaries try to bypass Facebook’s protections and show objectionable content to users. Here training iterations take on the order of days. Even more dependent on the incorporation of recent data into models is the news feed ranking. “Stale News Feed models have a measurable impact on quality.” And if we look at the very core of the business, the Ads Ranking models, “we learned that the impact of leveraging a stale ML model is measured in hours. In other words, using a one-day-old model is measurably worse than using a one-hour old model.” One of the conclusions in this section of the paper is that disaster recovery / high availability for training workloads is key importance. (Another place to practice your chaos engineering ;) ).

---

## Banana rotting time

---

### "If everyone's doing it, it's got to be the best way, right?"

---

## Kappa architecture

???

This looks great, but our favorite models such as LightGBM, can be updated incrementally.

---

background-color: #b5ddd1
class: middle

## Online machine learning

.bigbullets[
- Subdiscipline of machine learning
- Data is a stream, potentially infinite
- Models learn from one observation at a time
- Features and labels can be dynamic
- Can be supervised or unsupervised
]

---

background-color: #2ac380
class: middle, white

## Different names, same thing 🤔

.bigbullets[
- Online learning
- Incremental learning
- Sequential learning
- Iterative learning
- Continuous learning
- Out-of-core learning
]

---

background-color: #607bd4
class: middle, white

## Benefits of online machine learning

.bigbullets[
1. Models don't have to be retrained
2. Nothing is too big
3. Online models (usually) adapt to drift
4. Model development is closer to reality
5. Training can be monitored in real-time
]

---

background-color: #e69138
class: middle, white

## Applications

.bigbullets[
- Time series forecasting
- Spam filtering
- Recommender systems
- Ad placement
- Internet of things
- Basically, <span style="text-decoration:underline">anything event based</span>
]

---

background-color: #FF7F50
class: middle, white

## Why is batch learning so popular?

---

> It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.

???

And just like Winnie the Pooh, we're spending too much time banging our heads to be able to think about a better way of doing things.

---

.bullets[
- Online machine learning library for Python 🐍
- Easy to pick up API inspired by scikit-learn
- Written with production scenarios in mind
- First commit in January 2019
- Version `0.4.3` released this week (with wheels!)
]

---

### PyData Amsterdam, May 2019 🐍 🇳🇱 🧀

---

### Features

Representing a set of features using a `dict` is natural:

```python
x = {
 'date': dt.datetime(2019, 4, 22),
 'price': 42.95,
 'shop': 'Ikea'
}
```

- Values can be of any type
- Feature names can be used instead of array indexes
- Web request payloads are dictionaries

---

### Targets

A target's type depends on the context:

```python
# Regression
y = 42

# Binary classification
y = True

# Multi-class classification
y = 'setosa'

# Multi-output regression
y = {
 'height': 29.7,
 'width': 21
}
```

---

### Streaming data

```python
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

for x, y in X_y:
 print(x, y)
```

- `X_y` is a **generator**, it doesn't hold data in memory
- Source depends on your use case (CSV file, Kafka consumer, HTTP requests)

---

### Training with `fit_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* model.fit_one(x, y)
```

Every `creme` estimator has a `fit_one` method

---

### Predicting with `predict_one`

```python
from creme import linear_model
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

for x, y in X_y:
* y_pred = model.predict_one(x)
 model.fit_one(x, y)
```

- Classifiers also have a `predict_proba_one` method
- Transformers have a `transform_one` method
- Training and predicting phases are inter-leaved

---

### Progressive validation 💯

```python
from creme import linear_model
from creme import metrics
from creme import stream

X_y = stream.iter_csv('some/csv/file.csv')

model = linear_model.LogisticRegression()

metric = metrics.Accuracy()

for x, y in X_y:
 y_pred = model.predict_one(x)
 model.fit_one(x, y)
* metric.update(y, y_pred)
 print(metric)
```

Validation score is available for free! No need for cross-validation. You can also use `online_score` from the `model_selection` module.

---

### Composing estimators is easy

counts = feature_extraction.CountVectorizer()
tdidf = feature_extraction.TFIDFVectorizer()

scale = preprocessing.StandardScaler()

log_reg = linear_model.LogisticRegression()

model = (counts + tdidf) | scale | log_reg

model.draw()
```
]

---

### Online mean

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$

```python
>>> mean = creme.stats.Mean()

>>> mean.update(5)
>>> mean.get()
5

>>> mean.update(10)
>>> mean.get()
7.5
```

---

### Online variance

For every incoming $x$, do:

1. $n = n + 1$
2. $\mu\_{i+1} = \mu\_{i} + \frac{x - \mu\_{i}}{n}$
3. $s\_{i+1} = s\_i + (x - \mu\_{i}) \times (x - \mu\_{i+1})$
4. $\sigma\_{i+1} = \frac{s\_{i+1}}{n}$

```python
>>> variance = creme.stats.Variance()

>>> X = [2, 3, 4, 5]
>>> for x in X:
... variance.update(x)
>>> variance.get()
1.25

>>> numpy.var(X)
1.25

```

???

This is called Welford's algorithm, it can be extended to skew and kurtosis

---

### Standard scaling

Using the mean and the variance, we can rescale incoming data.

```python
>>> scaler = creme.preprocessing.StandardScaler()

```

In practice, works better than normalized gradient descent 😲

---

### Linear models

Model is:

$$\hat{y}_t = f(w_t, x_t) + b_t$$

Update weights with gradients:

$$w_{t+1} = u(w_t, x_t, \partial L(y_t, \hat{y}_t))$$

Many models can be derived, for example:

- Use Hinge loss for SVM
- Add L1/L2 regularisation for LASSO/ridge
- Add interactions for factorization machines

---

### Many online optimizers to choose from

.bullets[
- Stochastic gradient descent (SGD)
- Passive-Aggressive (PA)
- ADAM
- RMSProp
- Follow the Regularized Leader (FTRL)
- Approximate Large Margin Algorithm (ALMA)
]

<div align="center">
Many variants of each, as you know
</div>

---

### Bayesian linear models

We want the posterior target distribution on the target:

$$\color{forestgreen} p(y\_t | x\_t) \color{black} \propto \color{crimson} p(y_t | w_t, x_t) \color{royalblue} p(w_t)$$

We first need to compute the posterior distribution of the weights:

$$\color{blueviolet} p(w\_{t} | w\_{t-1}, x\_t, y\_t) \color{black} \propto \color{crimson} p(y\_t | w\_{t-1}, x\_t) \color{royalblue} p(w\_{t-1})$$

This is old-school Bayesian learning, it is different from and predecesses the Monte-Carlo mumbo-jumbo.

---

### Online belief updating

Before any data comes in, the model parameters follow the initial distribution we picked, which is $\color{royalblue} p(w\_0)$. Next, once the first observation $(x\_0, y\_0)$ arrives, we can obtain the distribution of $w\_1$:

$$\color{blueviolet} p(w\_1 | w\_0, x\_0, y\_0) \color{black} \propto \color{crimson} p(y\_0 | w\_0, x\_0) \color{royalblue} p(w\_0)$$

Once the second observation $(x\_1, y\_1)$ is available, the distribution of the model parameters is obtained in the same way:

$$\color{blueviolet} p(w\_2 | w\_1, x\_1, y\_1) \color{black} \propto \color{crimson} p(y\_1 | w\_1, x\_1) \color{royalblue} p(w\_1) \propto \color{crimson} p(y\_1 | w\_1, x\_1) \color{blueviolet} p(w\_1 | w\_0, x\_0, y\_0)$$

The $\propto$ symbol means there is an analytical formula that can be derived.

---

### Nearest neighbors

.bullets[
- Three parameters:
 1. The distance function
 2. The number of neighbors
 3. The window size
- .green[Naturally adapts to drift]
- .red[Lazy]
]

---

### Decision trees 🌳

- A version of Hoeffding trees is being implemented
- Basic idea:
 - Start with a leaf 🍃
 - Find the leaf where an observation belongs 🔎
 - Update the leaf's "sufficient statistics" 📊
 - Measure information gain every so often 🔬
 - Split when the information gain is good enough 🍂
- Mondrian trees 👨‍🎨 are another possibility but they only work for continuous attributes

---

### Decision trees 🌳

Quality criterion of split $x < t$ can be evaluated with:

$$P(y \mid x < t) = \frac{P(x < t \mid y) \times P(y)}{P(x < t)}$$

and:

$$P(y \mid x \geq t) = \frac{(1 - P(x < t \mid y)) \times P(y)}{1 - P(x < t)}$$

- For classification, $P(x < t \mid y)$ is a set of online CDFs and $P(y)$ is a PMF.
- For regression, $P(x < t \mid y)$ is a 2D CDF and $P(y)$ is a PDF
- All these distributions can be updated online

---

### Bagging

Each observation is sampled $K$ times where $K$ follows a binomial distribution:

$$P(K=k) = {n \choose k} \times (\frac{1}{n})^k \times (1 - \frac{1}{n})^{n-k}$$

As $n$ grows towards infinity, $K$ can be approximated by a Poisson(1):

$$P(K=k) \sim \frac{e^{-1}}{k!} $$

This leads to a simple and efficient online algorithm:

```python
for model in models:
 for _ in range(random.poisson(λ=1)):
 model.fit_one(x, y)
```

---

### (S)(N)(AR)(I)(MA)(X)

ARMA model is defined as so:

$$\hat{y}\_t = \sum\_{i=1}^p \alpha\_i y\_{t-i} + \sum\_{i=1}^q \beta\_i (y\_{t-i} - \hat{y}\_{t-i}) $$

Classically, Kalman filters are used to find the weights $\alpha\_i$ and $\beta\_i$. But $y\_{t-i}$ and $\hat{y}\_{t-i}$ can also be [seen as features in an online setting](https://dl.acm.org/citation.cfm?id=3016160):

- Seasonality can be handled online
- Any online learning model can be used
- Detrending by differencing can be done online
- Heteroscedasticity can be handled online
- Exogenous variables can be added

---

### Online aggregated features

```python
>>> import creme

>>> X = [
... {'meal': 'tika masala', 'sales': 42},
... {'meal': 'kale salad', 'sales': 16},
... {'meal': 'kale salad', 'sales': 24},
... {'meal': 'tika masala', 'sales': 58}
... ]

>>> agg = creme.feature_extraction.Agg(
... on='sales',
... by='meal',
... how=creme.stats.Mean()
... )

>>> for x in X:
... print(agg.fit_one(x).transform_one(x))
{'sales_mean_by_meal': 42.0}
{'sales_mean_by_meal': 16.0}
{'sales_mean_by_meal': 20.0}
{'sales_mean_by_meal': 50.0}
```

---

background-color: #008080
class: middle, white

## There is much more

.bullets[
- Half-space trees for anomaly detection
- $k$-means clustering
- Latent Dirichlet allocation (LDA)
- Expert learning
- Stacking
- Recommendation systems
- See [creme-ml.github.io/api](https://creme-ml.github.io/api.html)
]

---

### Alternative frameworks

---

### Binary classification benchmark with default parameters

<div align="center">
.pure-table.pure-table-striped[
| Library | Method | Accuracy | Average fit time | Average predict time |
|---------|------------|----------|------------------|----------------------|
| creme | LogisticRegression | 0.61810 | .green[26μs] | .green[10μs] |
| creme | PAClassifier | 0.55009 | 35μs | 22μs |
| creme | DecisionTreeClassifier | 0.64663 | 356μs | 15μs |
| creme | RandomForestClassifier | .green[0.65915] | 3ms, 972μs | 208μs |
| Keras on TF (CPU) | Dense | 0.61840 | 463μs | 534μs |
| PyTorch (CPU) | Linear | 0.61840 | 926μs | 621μs |
| scikit-garden | MondrianTreeClassifier | .red[0.53875] | 864μs | 208μs |
| scikit-garden | MondrianForestClassifier | 0.60061 | .red[9ms, 773μs] | .red[1ms, 233μs] |
| scikit-learn | SGDClassifier | 0.56161 | 420μs | 116μs |
| scikit-learn | PassiveAggressiveClassifier | 0.55009 | 398μs | 114μs |
]
</div>

---

### Linear regression benchmark

<div align="center">
.pure-table.pure-table-striped[
| Library | Method | MSE | Average fit time | Average predict time |
|---------|------------|----------|------------------|----------------------|
| creme | LinearRegression | 23.035085 | 18μs | 4μs |
| Keras on TF (CPU) | Dense | 23.035086 | 1ms, 208μs | 722μs |
| PyTorch (CPU) | Linear | 23.035086 | 577μs | 187μs |
| scikit-learn | SGDRegressor | 25.295369 | 305μs | 108μs |
]
</div>

---

background-color: #1f282d
class: middle, white

## Current work (1)

.bullets[
- Boosting, many methods but no clear winner:
 - [Online Bagging and Boosting (Oza-Russell, 2005)](https://ti.arc.nasa.gov/m/profile/oza/files/ozru01a.pdf)
 - [Online Gradient Boosting (Beygelzimer, 2015)](https://arxiv.org/pdf/1506.04820.pdf)
 - [Optimal and Adaptive Algorithms for Online Boosting (Beygelzimer, 2015)](http://proceedings.mlr.press/v37/beygelzimer15.pdf)
- Mixture models through expectation-maximization:
 - [Recursive Parameter Estimation Using Incomplete Data (Titterington, 1982)](https://apps.dtic.mil/dtic/tr/fulltext/u2/a116190.pdf)
 - [A View of the EM Algorithm that Justifies Incremental, Sparse, and other Variants (Neal-Hinton, 1998)](http://www.cs.toronto.edu/~fritz/absps/emk.pdf)
 - [Online EM Algorithm for Latent Data Models (Cappé-Moulines 2009)](https://hal.archives-ouvertes.fr/hal-00201327/document)
]

---

background-color: #1f282d
class: middle, white

## Current work (2)

.bullets[
- Field-aware factorization machines (FFM):
 - [Factorization Machines (Rendle, 2010)](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf)
 - [Field-aware Factorization Machines for CTR Prediction (Juan et al., 2016)](https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf)
 - [Field-aware Factorization Machines in a Real-world Online Advertising System (Juan-Lefortier-Chappelle, 2017)](https://arxiv.org/pdf/1701.04099.pdf)
- Metric learning:
 - [Online and Batch Learning of Pseudo-Metrics (Shwartz-Singer-Ng, 2004)](https://ai.stanford.edu/~ang/papers/icml04-onlinemetric.pdf)
 - [Information-Theoretic Metric Learning (Davis et al., 2007)](http://www.cs.utexas.edu/users/pjain/pubs/metriclearning_icml.pdf)
 - [Online Metric Learning and Fast Similarity Search (Jain et al., 2009)](http://people.bu.edu/bkulis/pubs/nips_online.pdf)
]

---

background-color: #85144b
class: middle, white

## You can help

.bigbullets[
- Use it and tell us about it
- Share it with others
- Take on issues on GitHub
- Become a core contributor
]

---

# Thanks for listening!

An introduction to symbolic regression

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

An introduction to symbolic regression

Max Halford - PhD student IRIT/IMT

Toulouse Data Science Meetup - December 2017

.center[ .left-column[![tds_logo](/assets/img/presentations/tds_logo.jpeg)] .right-column[![xgp_logo](/assets/img/presentations/xgp_logo.png)] ] --- layout: true # Symbolic regression --- ## Quick overview - The goal is to evolve "programs" with selection, mutation, and crossover - Selection keeps programs that perform well - Mutation changes a piece of the program - Crossover combines two programs --- --- ## Example programs --- ## Kaggle Titanic top 1% 🚢 2 years ago [scirpus](https://www.kaggle.com/scirpus) posted a [solution](https://www.kaggle.com/scirpus/genetic-programming-lb-0-88) to the [Kaggle Titanic competiton](https://www.kaggle.com/c/titanic) ```python y_raw = ((np.minimum(((((0.058823499828577 + X['Sex']) - np.cos((X['Pclass'] / 2.0))) * 2.0)), ((0.885868))) * 2.0) +np.maximum(((X['SibSp'] - 2.409090042114258)), (-(np.minimum((X['Sex']), (np.sin(X['Parch']))) * X['Pclass']))) +(0.138462007045746 * ((np.minimum((X['Sex']), (((X['Parch'] / 2.0) / 2.0))) * X['Age']) - X['Cabin'])) +np.minimum(((np.sin((X['Parch'] * ((X['Fare'] - 0.720430016517639) * 2.0))) * 2.0)), ((X['SibSp'] / 2.0))) +np.maximum((np.minimum((-np.cos(X['Embarked'])), (0.138462007045746))), (np.sin(((X['Cabin'] - X['Fare']) * 2.0)))) +-np.minimum(((((X['Age'] * X['Parch']) * X['Embarked']) + X['Parch'])), (np.sin(X['Pclass']))) +np.minimum((X['Sex']), ((np.sin(-(X['Fare'] * np.cos((X['Fare'] * 1.630429983139038)))) / 2.0))) +np.minimum(((0.230145)), (np.sin(np.minimum((((67.0 / 2.0) * np.sin(X['Fare']))), (0.31830988618379069))))) +np.sin((np.sin(X['Cabin']) * (np.sin((12.6275)) * np.maximum((X['Age']), (X['Fare']))))) +np.sin(((np.minimum((X['Fare']), ((X['Cabin'] * X['Embarked']))) / 2.0) * -X['Fare'])) +np.minimum((((2.675679922103882 * X['SibSp']) * np.sin(((96) * np.sin(X['Cabin']))))), (X['Parch'])) +np.sin(np.sin((np.maximum((np.minimum((X['Age']), (X['Cabin']))), ((X['Fare'] * 0.31830988618379069))) * X['Cabin']))) +np.maximum((np.sin(((12.4148) * (X['Age'] / 2.0)))), (np.sin((-3.0 * X['Cabin'])))) +(np.minimum((np.sin((((np.sin(((X['Fare'] * 2.0) * 2.0)) * 2.0) * 2.0) * 2.0))), (X['SibSp'])) / 2.0) +((X['Sex'] - X['SibSp']) * (np.cos(((X['Embarked'] - 0.730768978595734) + X['Age'])) / 2.0)) +((np.sin(X['Cabin']) / 2.0) - (np.cos(np.minimum((X['Age']), (X['Embarked']))) * np.sin(X['Embarked']))) +np.minimum((0.31830988618379069), ((X['Sex'] * (2.212120056152344 * (0.720430016517639 - np.sin((X['Age'] * 2.0))))))) +(np.minimum((np.cos(X['Fare'])), (np.maximum((np.sin(X['Age'])), (X['Parch'])))) * np.cos((X['Fare'] / 2.0))) +np.sin((X['Parch'] * np.minimum(((X['Age'] - 1.5707963267948966)), ((np.cos((X['Pclass'] * 2.0)) / 2.0))))) +(X['Parch'] * (np.sin(((X['Fare'] * (0.623655974864960 * X['Age'])) * 2.0)) / 2.0)) +(0.31830988618379069 * np.cos(np.maximum(((0.602940976619720 * X['Fare'])), ((np.sin(0.720430016517639) * X['Age']))))) +(np.minimum(((X['SibSp'] / 2.0)), (np.sin(((X['Pclass'] - X['Fare']) * X['SibSp'])))) * X['SibSp']) +np.tanh((X['Sex'] * np.sin((5.199999809265137 * np.sin((X['Cabin'] * np.cos(X['Fare']))))))) +(np.minimum((X['Parch']), (X['Sex'])) * np.cos(np.maximum(((np.cos(X['Parch']) + X['Age'])), (3.1415926535897931)))) +(np.minimum((np.tanh(((X['Cabin'] / 2.0) + X['Parch']))), ((X['Sex'] + np.cos(X['Age'])))) / 2.0) +(np.sin((np.sin(X['Sex']) * (np.sin((X['Age'] * X['Pclass'])) * X['Pclass']))) / 2.0) +(X['Sex'] * (np.cos(((X['Sex'] + X['Fare']) * ((8.48635) * (63)))) / 2.0)) +np.minimum((X['Sex']), ((np.cos((X['Age'] * np.tanh(np.sin(np.cos(X['Fare']))))) / 2.0))) +(np.tanh(np.tanh(-np.cos((np.maximum((np.cos(X['Fare'])), (0.094339601695538)) * X['Age'])))) / 2.0) +(np.tanh(np.cos((np.cos(X['Age']) + (X['Age'] + np.minimum((X['Fare']), (X['Age'])))))) / 2.0) +(np.tanh(np.cos((X['Age'] * ((-2.0 + np.sin(X['SibSp'])) + X['Fare'])))) / 2.0) +(np.minimum((((281) - X['Fare'])), (np.sin((np.maximum(((176)), (X['Fare'])) * X['SibSp'])))) * 2.0) +np.sin(((np.maximum((X['Embarked']), (X['Age'])) * 2.0) * (((785) * 3.1415926535897931) * X['Age']))) +np.minimum((X['Sex']), (np.sin(-(np.minimum(((X['Cabin'] / 2.0)), (X['SibSp'])) * (X['Fare'] / 2.0))))) +np.sin(np.sin((X['Cabin'] * (X['Embarked'] + (np.tanh(-X['Age']) + X['Fare']))))) +(np.cos(np.cos(X['Fare'])) * (np.sin((X['Embarked'] - ((734) * X['Fare']))) / 2.0)) +((np.minimum((X['SibSp']), (np.cos(X['Fare']))) * np.cos(X['SibSp'])) * np.sin((X['Age'] / 2.0))) +(np.sin((np.sin((X['SibSp'] * np.cos((X['Fare'] * 2.0)))) + (X['Cabin'] * 2.0))) / 2.0) +(((X['Sex'] * X['SibSp']) * np.sin(np.sin(-(X['Fare'] * X['Cabin'])))) * 2.0) +(np.sin((X['SibSp'] * ((((5.428569793701172 + 67.0) * 2.0) / 2.0) * X['Age']))) / 2.0) +(X['Pclass'] * (np.sin(((X['Embarked'] * X['Cabin']) * (X['Age'] - (1.07241)))) / 2.0)) +(np.cos(((((-X['SibSp'] + X['Age']) + X['Parch']) * X['Embarked']) / 2.0)) / 2.0) +(0.31830988618379069 * np.sin(((X['Age'] * ((X['Embarked'] * np.sin(X['Fare'])) * 2.0)) * 2.0))) +((np.minimum(((X['Age'] * 0.058823499828577)), (X['Sex'])) - 0.63661977236758138) * np.tanh(np.sin(X['Pclass']))) +-np.minimum(((np.cos(((727) * ((X['Fare'] + X['Parch']) * 2.0))) / 2.0)), (X['Fare'])) +(np.minimum((np.cos(X['Fare'])), (X['SibSp'])) * np.minimum((np.sin(X['Parch'])), (np.cos((X['Embarked'] * 2.0))))) +(np.minimum((((X['Fare'] / 2.0) - 2.675679922103882)), (0.138462007045746)) * np.sin((1.5707963267948966 * X['Age']))) +np.minimum(((0.0821533)), (((np.sin(X['Fare']) + X['Embarked']) - np.cos((X['Age'] * (9.89287))))))) y_pred = 1 / (1 + np.exp(-y_raw)) ``` 0.88516 (top 1% as of today) on the public leaderboard 😵 --- ## Types of nodes - Constants - Variables - Functions Huge search space because the shape of the model is included in the search space 🙊 --- ## Algorithm (high-level) ➰ ```python programs = make_random_programs() for i in range(generations): evaluate(programs) new_programs = select(programs) crossover(new_programs) mutate(new_programs) programs = new_programs ``` --- ## Mutation --- ## Crossover 💏 --- ## Evaluation 💯 --- ## Selection --- ## Initialization --- ## Pros 😺 - Flexible model - Built-in feature selection - Can optimise non-differentiable metrics - Useful for stacking - Makes you look cool 😎 --- ## Cons 😿 - Black box model - No mathematical foundation - Requires a lot of CPU power - Non-deterministic and volatile --- layout: true .center[![xgp_logo](/assets/img/presentations/xgp_logo.png)] --- ## Enter XGP 🎉 - Written in [Go](https://golang.org/) - Optimization done with [gago](https://github.com/MaxHalford/gago) - SIMD operations thanks to [gonum](https://www.gonum.org/) ⚡ - Readable code 📖 - Very customizable (with sensible defaults!) - Command line interface (CLI) 💻 - Python bindings (scikit-learn API) 🐍 --- ## Command line interface (CLI) 💻 ```sh $ xgp fit train.csv ``` ```sh $ xgp predict test.csv ``` --- ## Python package 🐍 ### Regression 📈 ```python model = xgp.XGPRegressor() model.fit(X_train, y_train) y_pred = model.predict(X_test) ``` --- ## Python package 🐍 ### Feature extraction 🔬 ```py model = xgp.XGPTransformer() model.fit(X_train, y_train) train_gp_features = model.predict(X_train) test_gp_features = model.predict(X_test) ``` --- ## Future work 🔮 - Binary and multi-class classification - Boosting (promising) - Caching to handle large datasets - More bindings (R, Ruby, ...) - Extensive testing - Documentation and examples --- layout: false class: center, middle # Thanks! ✌ [github.com/MaxHalford](https://github.com/MaxHalford/)

Bio

maxhalford25@gmail.com (Max Halford) — Mon, 01 Jan 0001 00:00:00 +0000

Hello ✌️

I’m Head of Data at Carbonfact, where we measure the carbon footprint of clothing items 🍃. Before that I worked for Alan, a health insurance company. My PhD topic was about applying machine learning – Bayesian networks in particular 🕸️ – to query optimisation in relational databases 🤖. My current areas of interest revolve around online machine learning 🍥, document processing 🔬, as well as tooling and good practices for data analytics 📊 and engineering 📦