Extracting the parameters and type-annotations

To get the parameters of the function, we can use the signature function of the inspect module.

def add(a: int, b: int) -> int:
    """Adds two integers together"""
    return a + b

This will return the entire signature for both the input parameters and the return type.

import inspect

signature = inspect.signature(add)
signature

<Signature (a: int, b: int) -> int>

We can get a dictionary of the parameters of the function from the signature

signature.parameters

mappingproxy({'a': <Parameter "a: int">, 'b': <Parameter "b: int">})

We can access each parameter from the dictionary. It will return an object that has many useful properties

a = signature.parameters["a"]
a

<Parameter "a: int">

We can now easily access the name of the parameter, its default value as well as the type annotation.

print("Name of parameter: ", a.name)
print("Default value: ", a.default)
print("Type annotation: ", a.annotation)

Name of parameter:  a
Default value:  <class 'inspect._empty'>
Type annotation:  <class 'int'>

This means that if a parameter has a default value of inspect._empty, it’s a required parameter`

a.default == inspect._empty

True

The type annotation is of particular interest to us. It will return the type directly

a.annotation

<class 'int'>

a.annotation == int

True

We can also get the type annotation for the return statement i.e. output of the function using the signature itself

signature.return_annotation

<class 'int'>

Extracting the docstring

To get the docstring, we can use the __doc__ attribute in the function

def add(a: int, b: int) -> int:
    """Adds two integers together"""
    return a + b

add.__doc__

'Adds two integers together'

An alternate approach is to use inspect module itself.

import inspect

inspect.getdoc(add)

'Adds two integers together'

Extracting the function name

This is relatively simple as python already provides a __name__ attribute on each function.

def add(a: int, b: int) -> int:
    """Adds two integers together"""
    return a + b

add.__name__

'add'

Extracting the parameter descriptions from the docstring

We can make use of a third-party library called docstring_parser as the format of docstrings can vary a lot.

!pip install docstring_parser -qqq

def add(a: int, b: int) -> int:
    """
    Adds two integers together.

    Args:
        a (int): The first integer.
        b (int): The second integer.

    Returns:
        int: The sum of a and b.
    """
    return a + b

from docstring_parser import parse

doc = parse(add.__doc__)
{param.arg_name: param.description for param in doc.params}

{'a': 'The first integer.', 'b': 'The second integer.'}

Approach 1: Pure Python

This is the approach implemented in the OpenAI Swarm library. In this, we can use all introspection feature discussed above to write the conversion function from scratch.

!pip install git+https://github.com/openai/swarm.git -qqq

def add(a: int, b: int) -> int:
    """Adds two integers together"""
    return a + b

Swarm has a utility function called function_to_json that converts a python function into a JSON schema.

from swarm.util import function_to_json

function_to_json(add)

{
    'type': 'function',
    'function': {
        'name': 'add',
        'description': 'Adds two integers together',
        'parameters': {
            'type': 'object',
            'properties': {'a': {'type': 'integer'}, 'b': {'type': 'integer'}},
            'required': ['a', 'b']
        }
    }
}

As seen above, we first need some mapping to convert the parameter types from Python to the equivalent JSON schema data type.

Based on this, the implementation is quite simple and reuses all the concept we discussed before.

We take the function signature and extract the parameter types for each paramter as well as get the function name and docstring. Using this, we construct the JSON Schema at the end.

# Source: https://github.com/openai/swarm/blob/9db581cecaacea0d46a933d6453c312b034dbf47/swarm/util.py#L31
import inspect


def function_to_json(func) -> dict:
    # A mapping of types from python to JSON
    type_map = {
        str: "string",
        int: "integer",
        float: "number",
        bool: "boolean",
        list: "array",
        dict: "object",
        type(None): "null",
    }

    try:
1        signature = inspect.signature(func)
    except ValueError as e:
        raise ValueError(
            f"Failed to get signature for function {func.__name__}: {str(e)}"
        )

    parameters = {}
2    for param in signature.parameters.values():
        try:
            param_type = type_map.get(param.annotation, "string")
        except KeyError as e:
            raise KeyError(
                f"Unknown type annotation {param.annotation} for parameter {param.name}: {str(e)}"
            )
        parameters[param.name] = {"type": param_type}

3    required = [
        param.name
        for param in signature.parameters.values()
        if param.default == inspect._empty
    ]

    return {
        "type": "function",
        "function": {
4            "name": func.__name__,
5            "description": func.__doc__ or "",
            "parameters": {
                "type": "object",
                "properties": parameters,
                "required": required,
            },
        },
    }

function_to_json(add)

{
    'type': 'function',
    'function': {
        'name': 'add',
        'description': 'Adds two integers together',
        'parameters': {
            'type': 'object',
            'properties': {'a': {'type': 'integer'}, 'b': {'type': 'integer'}},
            'required': ['a', 'b']
        }
    }
}

Approach 2: Pydantic

from pydantic import BaseModel


class Add(BaseModel):
    a: int
    b: int


Add.model_json_schema()

{
    'properties': {'a': {'title': 'A', 'type': 'integer'}, 'b': {'title': 'B', 'type': 'integer'}},
    'required': ['a', 'b'],
    'title': 'Add',
    'type': 'object'
}

from pydantic import create_model

a = create_model("Add", a=(int, ...), b=(int, ...))
a.model_json_schema()

{
    'properties': {'a': {'title': 'A', 'type': 'integer'}, 'b': {'title': 'B', 'type': 'integer'}},
    'required': ['a', 'b'],
    'title': 'Add',
    'type': 'object'
}

from pydantic import create_model

a = create_model("Add", **{"a": (int, ...), "b": (int, ...)})
a.model_json_schema()

{
    'properties': {'a': {'title': 'A', 'type': 'integer'}, 'b': {'title': 'B', 'type': 'integer'}},
    'required': ['a', 'b'],
    'title': 'Add',
    'type': 'object'
}

import inspect

from pydantic import create_model


def add(a: int, b: int) -> int:
    """Adds two integers together"""
    return a + b


def schema(f):
    kws = {
        name: (
            # Get the type annotation
            parameter.annotation,
            # Check if parameter is required or optional
            ... if parameter.default == inspect._empty else parameter.default,
        )
        for name, parameter in inspect.signature(f).parameters.items()
    }
    # Pass the function name and parameters to get a pydantic model
    p = create_model(f"`{f.__name__}`", **kws)

    # Convert to JSON Schema
    schema = p.model_json_schema()
    return {
        "type": "function",
        "function": {
            "name": f.__name__,
            "description": f.__doc__,
            "parameters": schema,
        },
    }


schema(add)

{
    'type': 'function',
    'function': {
        'name': 'add',
        'description': 'Adds two integers together',
        'parameters': {
            'properties': {
                'a': {'title': 'A', 'type': 'integer'},
                'b': {'title': 'B', 'type': 'integer'}
            },
            'required': ['a', 'b'],
            'title': '`add`',
            'type': 'object'
        }
    }
}

from pydantic import TypeAdapter


def add(a: int, b: int) -> int:
    """Adds two integers together"""
    return a + b


def schema(f):
    schema = TypeAdapter(f).json_schema()
    return {
        "type": "function",
        "function": {
            "name": f.__name__,
            "description": f.__doc__,
            "parameters": schema,
        },
    }


schema(add)

{
    'type': 'function',
    'function': {
        'name': 'add',
        'description': 'Adds two integers together',
        'parameters': {
            'additionalProperties': False,
            'properties': {
                'a': {'title': 'A', 'type': 'integer'},
                'b': {'title': 'B', 'type': 'integer'}
            },
            'required': ['a', 'b'],
            'type': 'object'
        }
    }
}

Approach 3: Decorators

Most agent libraries wrap conversion approaches like above as decorators (e.g. smolagents) to make them easier to use.

For example, we can make a decorator called tool, which, when applied to a function, will add a json_schema method to that function.

def tool(func):
    func.json_schema = lambda: function_to_json(func)
    return func

We can mark out functions with the decorator.

@tool
def add(a: int, b: int) -> int:
    """Adds two numbers"""
    return a + b

And can use the json_schema method to get the schema directly and use it downstream in LLM API.

add.json_schema()

{
    'type': 'function',
    'function': {
        'name': 'add',
        'description': 'Adds two numbers',
        'parameters': {
            'type': 'object',
            'properties': {'a': {'type': 'integer'}, 'b': {'type': 'integer'}},
            'required': ['a', 'b']
        }
    }
}

Distinct n-grams (Distinct-k)

Li et al. (2016) proposed distinct-k to evaluate their technique for increasing diversity in sequence-to-sequence models. It builds on the type-token ratio concept from linguistics.

They calculate diversity as the ratio of the number of unique n-grams to the total n-grams occurring in the entire generated dataset. As shown below, the two texts contain only 5 unique words out of a total of 9 words and thus, the diversity score is only 55% (0.55).

However, if all the synthetic texts were unique, we would get a diversity score of 100% (1.0).

We can extend this same idea from unigrams to bigrams, trigrams, and any higher-order n-grams. There are two approaches.

In the first approach, we report the diversity score separately for different n-grams. Li et al. (2016) do this for unigrams and bigrams as distinct-1 and distinct-2. While Padmakumar and He (2023) report diversity scores up to 4-grams separately in their paper that shows instruction-tuned models have lower diversity compared to base models.

Alternatively, we can report a single diversity score by combining the scores for different n-grams. Li et al. (2022) take the product of the diversity score for unigrams, bigrams, trigrams, and four-grams as a single final score, while Meister et al. (2023) take the sum of the diversities.

The library diversity by Shaib et al. (2024a) provides an easy way to compute the distinct-k metric:

shell

pip install diversity

from diversity import ngram_diversity_score

texts = ['As an AI language model', 'As an AI model']

ngram_diversity_score(texts, 1)

0.556

N-gram Entropy (Ent-n)

Zhang et al. (2018) introduced this metric, and Jagfeld et al. (2018) also used it to evaluate the diversity of template to natural language generation.

The intuition behind it is that in an ideal case, LLM generates texts that are all unique and no n-grams is repeated more than once.

We can measure this by collecting all the unique bigrams in the text and calculating their count and the relative frequency. This yields a probability distribution over the bigrams.

For the highest diversity, all the texts would be unique and thus the probability distribution over the bigrams would be uniform, resulting in the highest entropy. Therefore, the entropy of the n-gram distribution serves as a diversity metric, as shown below.

Given the distribution of bigrams, we can calculate the entropy easily as shown below.

import math

probs = [0.25, 0.25, 0.25, 0.25]
-sum(p * math.log(p) for p in probs)

1.3862

However, let’s take another case where there is lots of repetition e.g. “Play the music” being generated 100 times. In such a case, the bigrams “Play the” and “the music” dominate the frequency distribution. As such, the entropy reduces, and thus, the diversity score drops.

We can also extend this idea to higher-order n-grams similar to the distinct n-grams metric. Tevet and Berant (2020) calculate and report entropy separately for unigram, bigram, and trigrams.

While Oraby et al. (2018) combine all unique unigrams, bigrams, and trigrams and then use the entropy of the resulting distribution as the diversity.

This metric can be implemented in code as shown below.

import math
from collections import Counter


def generate_ngrams(words, n: int):
    return [" ".join(words[i : i + n]) for i in range(len(words) - n + 1)]


def ngram_entropy(texts: list[str], n: int = 2) -> float:
1    ngrams = []
    for text in texts:
        words = text.split()
        ngrams.extend(generate_ngrams(words, n))

2    ngram_counts = Counter(ngrams)
    total_ngrams = sum(ngram_counts.values())

3    ngram_frequencies = [count / total_ngrams for ngram, count in ngram_counts.items()]

4    entropy = -sum(freq * math.log(freq) for freq in ngram_frequencies)
        
    return entropy

1

Step 1: Generate n-grams from input texts

2

Step 2: Count the frequency of each n-gram

3

Step 3: Calculate the frequency of each n-gram

4

Step 4: Calculate entropy

texts = ["Call an Uber", "Play the music"]

print("Unigram entropy:", ngram_entropy(texts, n=1))
print("Bigram entropy:", ngram_entropy(texts, n=2))
print("Trigram entropy:", ngram_entropy(texts, n=3))

Unigram entropy: 1.7917594692280547
Bigram entropy: 1.3862943611198906
Trigram entropy: 0.6931471805599453

import math
from collections import Counter


def generate_ngrams(words: list[str], n: int) -> list[str]:
    return [" ".join(words[i : i + n]) for i in range(len(words) - n + 1)]


def normalized_ngram_entropy(texts: list[str], n: int = 2) -> float:
    ngrams = []
    for text in texts:
        words = text.split()
        ngrams.extend(generate_ngrams(words, n))

    ngram_counts = Counter(ngrams)
    total_ngrams = sum(ngram_counts.values())

    ngram_frequencies = [count / total_ngrams for ngram, count in ngram_counts.items()]
    entropy = -sum(freq * math.log(freq) for freq in ngram_frequencies)

    uniform_frequencies = [1/len(ngrams) for _ in range(len(ngrams))]
    ideal_entropy = -sum(freq * math.log(freq) for freq in uniform_frequencies)

    diversity = entropy / ideal_entropy
    return diversity

texts = ["Call an Uber", "Play the music"]

print("Unigram diversity:", normalized_ngram_entropy(texts, n=1))
print("Bigram diversity:", normalized_ngram_entropy(texts, n=2))
print("Trigram diversity:", normalized_ngram_entropy(texts, n=3))

Unigram diversity: 1.0
Bigram diversity: 1.0
Trigram diversity: 1.0

Compression Ratio

Shaib et al. (2024a) proposed this metric by adapting the concept of the compression ratio, originally used to evaluate compression algorithms, as a diversity metric.

Compression ratio calculates the ratio of the size of the compressed file to its original size. A high compression ratio indicates the file was highly compressible and thus had higher redundancy, indicating lower diversity in the file contents.

To apply this concept to texts, we can compress them using an algorithm like Gzip and then calculate the compression ratio. A higher ratio indicates lower diversity in the text. Thus, the greater the compression ratio, the less diverse the generated texts.

Thus, diversity can be calculated as the reciprocal of the compression ratio to get a score between 0 and 1.

If all the texts are unique, then the compressed file size would be the same as the original file size and thus the compression ratio and the diversity both would be 1.

We can implement this in code using the diversity library.

shell

pip install diversity

from diversity import compression_ratio

texts = ['Call an Uber'] + ['Play the music'] * 100

compression_ratio(texts)

16.258

Embedding Diversity

Tevet and Berant (2020) proposed this metric, which considers diversity as the dissimilarity of text embeddings.

The metric calculates sentence embeddings for all generated texts using an encoder (e.g. sentence-transformers).

Then, we calculate the cosine similarity between all the unique pairs and take the average to get a similarity score.

To convert the similarity into diversity, we can either take the negation of the average cosine similarity (Tevet and Berant, 2020) or take the cosine distance i.e.  (Young et al. (2024); Hayati et al. (2024))

DCScore

This metric was proposed in a paper currently under review for ICLR 2025 (Anonymous, 2024).

The metric, similar to embedding diversity, also starts by calculating the pairwise similarity between all the text embeddings but has a unique take on formulating the diversity.

To understand the intuition, let’s look at the first row of the pairwise similarity matrix. Here, the numbers 1.0, 0.75, and 0.2 mean that the text is 100% similar to itself, 75% similar to some other text, and 20% similar to the final text. Hypothetically, we would have wanted the text to only be 100% similar to itself and 0% similar to everything else for maximum diversity.

Thus, we want some relative measure of similarity of the text to itself in comparison to others. The authors use softmax for this. Softmax converts the cosine similarities into relative probabilities of the text belonging to itself and others. When we apply softmax, we see that the first text is only belonging 45% to itself. Thus, the softmax probability of the text belonging to itself can be a measure of diversity.

To calculate the diversity of the dataset overall, we simply take the mean of the diagonal of the pairwise matrix after applying softmax. Thus, we get a diversity score of 0.47 in the example above.

The implementation is simple and fits in a few lines of code. We can swap the embedding model as needed.

shell

pip install sentence-transformers scipy numpy

import numpy as np
from scipy.special import softmax
from sentence_transformers import SentenceTransformer

1model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def dcscore(texts: list[str]) -> float:
2    text_embeddings = model.encode(texts, normalize_embeddings=True)
3    pairwise_matrix = text_embeddings @ text_embeddings.T
4    softmax_matrix = softmax(pairwise_matrix, axis=1)
5    score = np.mean(np.diag(softmax_matrix))
    return score

score = dcscore(['Play the music', 'Start the music', 'Call an Uber'])
print(score)

1

Load the MiniLM Sentence-BERT model

2

Generate embeddings for the sentences

3

Calculate pairwise cosine similarity

4

Apply softmax on the row level for each text

5

Take the mean of the scores in the diagonal

0.47264108

Cluster Inertia

Du and Black (2019) proposed this metric, reusing the inertia metric used to compute the quality of clustering as the diversity.

The metric clusters embeddings of the LLM-generated texts into 10 clusters and measures the inertia. Inertia is the sum of the squared distances between all points in a cluster and its centroid.

We can treat the inertia as a proxy for diversity because if the texts are diverse, they would be far apart from the centroid and thus the squared distance from the cluster centroid will be larger.

In code, this can be accomplished as shown below:

import numpy as np
from sklearn.cluster import KMeans

# Text embeddings for 1024 synthetic texts
text_embeddings = np.random.rand(1024, 768)

# Run clustering
kmeans = KMeans(n_clusters=10, random_state=42)
kmeans.fit(text_embeddings)

# Get the inertia
k.inertia_

64556.00644871439

Compression Ratio - Part of Speech (CR-POS)

Shaib et al. (2024b) proposed this metric to detect the repetition of syntactic templates in LLM-generated texts.

It reuses the idea of Compression Ratio but applies it to syntactic representation instead of the raw text. This works by applying a part-of-speech tagger to the text to get the POS tag for each token.

We apply a POS tagger to all the synthetically generated texts and get their syntactic representation as strings.

Then, the process is the same as the regular compression ratio. We concatenate the POS-tagged strings of all the texts together, compress the text using gzip, and then compare the ratio of the original file size with the compressed file size.

If the compression ratio is high, it indicates a large repetition of syntactic templates in the generated texts. Thus, the diversity will be low.

We can compute diversity directly by taking the reciprocal of the compression ratio and get a score between 0 and 1.

1. Clone the code locally

The code for the concept described above has been implemented at https://github.com/amitness/bluesky-arxiv.

First, make a fork of my repo from https://github.com/amitness/bluesky-arxiv and then clone your repo locally.

# Replace with the link to your repo
git clone git@github.com:amitness/bluesky-arxiv.git

Install the required libraries via the requirements.txt file in your virtual environment.

pip install -r requirements.txt

2. Setup Cloudflare pages

We will need a Cloudflare page to host the data in the format needed by Bluesky.

You can create an account on Cloudflare pages. Once the account is created, go to Workers and Pages > Overview from the left sidebar on the dashboard.

You should see two tabs: Workers and Pages. Click the Pages tab.

Then, click the “Upload Assets” button.

Then enter a name for the project. Cloudflare will provide you a unique domain based on it. Click Create Project.

You will be shown a page below that allows you to upload a zip file or a folder. At this stage, just upload a random folder from your device at least one file in it. Once you’re done, click Deploy site.

Once the site is deployed, you should see a message below with the URL of your domain.

In the repo that you cloned locally, change the SERVICE_DOMAIN variable in config.py file to the domain you got above from Cloudflare.

config.py

# Domain provided by Cloudflare pages
SERVICE_DOMAIN = "bluesky-1tj.pages.dev"

3. Initialize a custom feed on Bluesky

Now, we will initialize a custom feed programmatically on Bluesky.

In the repo, you will find a config.py file. You have to change a few configurations inside it.

First, change the HANDLE to your bluesky handle.

config.py

# YOUR bluesky handle
# Ex: user.bsky.social
HANDLE: str = "amitness.com"

Then you need to generate an app password for Bluesky. It’s available at https://bsky.app/settings/app-passwords and will allow us to get programmatic access to Bluesky in Python.

You can set a name to denote what the password is going to be used for. Here I set it to custom-feed.

Then you will receive your app password. Take note of it in a safe place as you won’t be able to access it again.

Now you can set the BLUESKY_APP_PASSWORD environment variable to your password.

export BLUESKY_APP_PASSWORD=...

This will be read by the setup_feed.py script.

config.py

# YOUR bluesky password, or preferably an App Password (found in your client settings)
# Ex: abcd-1234-efgh-5678
PASSWORD = os.environ["BLUESKY_APP_PASSWORD"]

Next, you can modify the name of your custom feed, a description and the slug. Here is what I have set.

config.py

# A short name for the record that will show in urls
# Lowercase with no spaces.
# Ex: whats-hot
RECORD_NAME: str = "arxiv-feed"

# A display name for your feed
# Ex: What's Hot
DISPLAY_NAME: str = "Papers"

# (Optional) A description of your feed
# Ex: Top trending content from the whole network
DESCRIPTION: str = dedent(
    """
 Latest ML research papers and preprints from arxiv.org discussed on Bluesky.
    
 Logic:
 - Fetch arxiv preprints & filters out non-ML via arxiv API
 - Ranks the items using hackernews algorithm
 """
).strip()

Here is how it will render up on Bluesky app later on.

Once everything above is setup, now you can run the script.

python setup_feed.py

This will initialize our custom feed on Bluesky. If everything was set up correctly, you will get an output for the value of FEED_URI.

Update the FEED_URI in config.py file with this value.

config.py

# Feed URI generated by running `python setup_feed.py`
FEED_URI = "at://did:plc:bpuq5cgmyvssgi3iwsyvd4gn/app.bsky.feed.generator/arxiv-feed"

Your feed has been created and now it needs to be populated before you can start using it in the app.

4. Setup Skyfeed

In this step, we will build an initial feed using the interface of the Skyfeed app.

You can signup on skyfeed.app using your Bluesky handle and the app password you created in previous step.

After logging in, go to the top-right and click Create Feed to create a new feed

You will see bunch of options. Since our goal is to filter out all the posts on Bluesky in past 24 hours that mention arxiv.org or aclanthology.org, we can set up the options as such.

First, the Input field specifies how many posts to capture. We will specify the Entire Network and set the time to 24 hours because we want to run a regex over all posts on Bluesky indexed in the past 24 hours. Depending on your usecase, you can modify this part.

As seen below, it yields 6 million posts in the past 24 hours.

Now, we will filter those 6 million posts to only get items that mention either the arxiv.org or the aclanthology.org links. This can be achieved with the below regex and can be pasted in the RegEx field. Make sure the Post Text and Link items are green as we want to search only in the post text and links.

(arxiv.org/.+)|(aclanthology.org/.+)

Here is how it should look after everything is set up correctly.

With this setup, we can now publish the feed as shown below by clicking Update Feed button and clicking Publish in the popup. This will create a feed that can be accessed via Bluesky now.

You should see the link to your published skyfeed as shown below.

Copy the portion as shown above to the SKYFEED_DID variable in config.py. We will be further filtering this feed now using Python in the next steps.

config.py

# Skyfeed path
SKYFEED_DID = "did:plc:bpuq5cgmyvssgi3iwsyvd4gn/feed/aaagg56kp5qzi"

5. Feed Generation in Python

With the above steps done, we can build out the feed generation logic. The main crux of the logic is present in generate_feed.py file. Let’s understand how it works:

generate_feed.py

def main():
 did_data = {
        "@context": ["https://www.w3.org/ns/did/v1"],
        "id": f"did:web:{config.SERVICE_DOMAIN}",
        "service": [
 {
                "id": "#bsky_fg",
                "type": "BskyFeedGenerator",
                "serviceEndpoint": f"https://{config.SERVICE_DOMAIN}",
 }
 ],
 }
    write_json(did_data, "./_site/.well-known/did.json")

 feed_generator_data = {
        "encoding": "application/json",
        "body": {"did": config.SERVICE_DID, "feeds": [{"uri": config.FEED_URI}]},
 }

    write_json(feed_generator_data, "./_site/xrpc/app.bsky.feed.describeFeedGenerator")

generate_feed.py

# Fetch latest posts and prepare data in the format expected by Bluesky protocol
post_uris = fetch_latest_posts()

feed_skeletion = {"feed": [{"post": uri} for uri in post_uris]}
write_json(feed_skeletion, "./_site/xrpc/app.bsky.feed.getFeedSkeleton")

generate_feed.py

def fetch_latest_posts():
 client = Client()
 client.login(config.HANDLE, config.PASSWORD)

 data = client.app.bsky.feed.get_feed(
 {
1            "feed": config.SKYFEED_PATH,
            "limit": 100,
 },
        timeout=100,
 )

 feed = data.feed
    for _ in range(2):
 data = client.app.bsky.feed.get_feed(
2 {"feed": config.SKYFEED_PATH, "limit": 100, "cursor": data.cursor},
            timeout=200,
 )
 feed.extend(data.feed)

3 bool_filter = thread_map(filter_item, feed)
 filtered_feed = compress(feed, bool_filter) 
4 sorted_feed = rank_posts(filtered_feed)
 post_uris = [item.post.uri for item in sorted_feed]
    return post_uris

generate_feed.py

def hackernews_score(item, gravity: float = 2.5):
 hours_passed = (
 datetime.now(timezone.utc) - parse_date(item.post.indexed_at)
 ).total_seconds() / 3600
    if hours_passed >= 12:
        return 0
    else:
 points = (
 item.post.like_count
            + item.post.quote_count
            + item.post.reply_count
            + item.post.repost_count
 )
 score = points / ((hours_passed + 2) ** (gravity))
        return score


def rank_posts(feed):
    return sorted(feed, key=hackernews_score, reverse=True)

6. Running periodically via GitHub Actions

To run our script periodically for free, we can leverage Github Actions. This will fetch the feed from Skyfeed, perform the filtering and re-ranking, and push the resulting data to Cloudflare pages every 30 minutes.

The schedule for the cron job is defined in the build_and_deploy.yml file and can be modified there as needed.

.github/workflows/build_and_deploy.yml

name: Build and deploy site to cloudflare

on:
  push:
    branches:
 - main
  schedule:
 - cron: '*/30 * * * *'

Crontab.guru is a great website to visualize what the cron syntax does.

To enable the actions in your forked GitHub repo, goto “Settings > Secrets and Variables” and click “New Repository Secret” and set these three variables one by one

• BLUESKY_APP_PASSWORD
• CLOUDFLARE_ACCOUNT_ID
• CLOUDFLARE_API_TOKEN

You can get your “CLOUDFLARE_ACCOUNT_ID” by logging in to Cloudflare Pages and then getting the value from the right sidebar as shown below.

To get the CLOUDFLARE_API_TOKEN, create a new token from https://dash.cloudflare.com/profile/api-tokens as shown below.

Once all three secret variables have been set up, you can enable GitHub actions in your forked repo as shown below.

The action should automatically run every 30 minutes now. As such, it will fetch the latest posts from skyfeed, perform the filtering and generate the final set of posts to be displayed on Bluesky and deploy that to Cloudflare.

7. Access your feed

Your feed will be listed on your profile now at bsky.app/feeds and can be pinned to the homepage as well.

You can find the link from the address bar when the feed is open and share it with others.

a. Specific character

We can simply specify the character in the regular expression and it will match all instances in the text.

For example, a regular expression given below will match all instances of ‘a’ in the text. You can use any of the small and capital alphabets.

'a'

You can also use any digits from 0 to 9 and it will work as well.

'3'

Note that regex is case-sensitive by default and thus the following regex won’t match anything.

'A'

b. White space character

We can detect special characters such as whitespace and newlines using special escape sequences.

Besides the common ones above, we have:

• \r for carriage return
  
• \f for form feed
  
• \e for escape

c. Special sequences

Regex provides a bunch of built-in special symbols that can match a group of characters at once. These begin with backslash \.

import re

>>> re.findall(r'.', 'line 1\nline2')
['l', 'i', 'n', 'e', ' ', '1', 'l', 'i', 'n', 'e', '2']

d. Character sets

These are patterns starting with [ and ending with ] and specify the characters that should be matched enclosed by brackets.

For example, the following pattern matches any of the characters ‘a’, ‘e’, ‘i’, ‘o’, and ‘u’.

You can also replicate the functionality of \d using the below pattern. It will match any digits between 0 to 9.

Instead of specifying all the digits, we can use - to specify only start and end digits. So, instead of [0123456789], we can do:

For example, [2-4] can be used to match any digits between 2 to 4 i.e. (2 or 3 or 4).

You can even use the special characters we learned previously inside the brackets. For example, you can match any digit from 0 to 9 or whitespace as:

Below, I have listed some useful common patterns and what they mean.

e. Anchors

Regex also has special handlers to make the pattern only match if it’s at the start or end of the string.

We can use the ^ anchor to match patterns only at the start of a line. For example:

Similarly, we can use the $ anchor after the character to match patterns only if it’s the end of the line. For example:

f. Escaping metacharacters

Consider a case where we want to exactly match the word “Mr. Stark”.

If we write a regex like Mr. Stark, then it will have an unintended effect. Since we know dot has a special meaning in a regex.

So, we should always escape the special metacharacters like ., $ etc. if our goal is to match the exact character itself.

Here is the list of metacharacters that you should remember to escape if you’re using them directly.

^ $ . * + ? { } [ ] \ | ( )

a. Naive repetition

Using only what we have learned so far, a naive way would be to just repeat the pattern. For example, we can match two-digit numbers by just repeating the character-level pattern.

\d\d

b. Quantifiers

Regex provides special quantifiers to specify different types of repetition for the character preceding it.

Need for raw strings

To specify a regular expression in Python, we precede it with r to create raw strings.

pattern = r'\d'

To understand why we precede with r, let’s try printing the expression \t without r.

pattern = '\t'
print(pattern)

You can see how when we don’t use raw string, the string \t is treated as the escape character for tab by Python.

Now let’s convert it into raw string. We get back whatever we specified.

pattern = r'\t'
print(pattern)
\t

Using re module

To use re module, we can start by importing the re module as:

import re

import re
re.findall(r'\d', '123456')

['1', '2', '3', '4', '5', '6']

import re

match = re.match(r'batman', 'batman is cool')
print(match)

<re.Match object; span=(0, 6), match='batman'>

print(match.group())

batman

import re

match = re.match(r'batman', 'The batman is cool')
print(match)

None

import re

match = re.search(r'batman', 'the batman is cool')
print(match.group())

batman

Intuition

The authors observed that in a good representation space, semantically similar data points should be close together.

In regular supervised classification, the information that images are semantically similar is encoded through labels annotated by humans. A model trained on such labels would have a representation space that groups semantically similar images.

Thus, with pre-text tasks in self-supervised learning, the objective is implicitly learning a metric that makes the same category images similar and different category images dissimilar. Hence we can provide a robust estimate of the learned representation if we could encode semantically related images to the same labels in some way.

1. Pretext task

Here we choose some deep network architecture and train it on some pretext task of our choice on some dataset. We can take features from some intermediate layer after the model is trained.

2. K-means Clustering

For all the unlabeled images in the dataset, we compute the feature vectors from the pretext task model. Then, we run K-means clustering to group semantically similar images. The idea is that the cluster centers will be aligned with categories in ImageNet.

In the paper, the authors ran K-means on a single Titan X GPU for 4 hours to cluster 1.3M images into 2000 categories.

3. Pseudo-labeling

The cluster centers are treated as the pseudo-label. We can use either the same dataset as the above step or use a different dataset itself. Then, we compute the feature vectors for those images and find the closest cluster center for each image. This cluster center is used as the pseudo-label.

4. Training on Pseudo-labels

We take the model architecture that will be used for downstream tasks and train it to classify the unlabeled images into the pseudo-labels. Thus, the target architecture will learn a new representation such that it will map images that were originally close in the pre-trained feature space to close points.

a. Increase complexity of pretext task (Jigsaw++)

To evaluate their method, the authors took an old puzzle-like pretext task called “Jigsaw” where we need to predict the permutation that was used to randomly shuffle a 3*3 square grid of image.

They extended the task by randomly replacing 0 to 2 number of tiles with tile from another random image at some random locations. This increases the difficulty as now we need to solve the problem using only the remaining patches. The new pretext task is called “Jigsaw++”.

In the paper, they use 701 total permutations which had a minimum hamming distance of 3. They apply mean and standard deviation normalization at each image tile independently. They also make images gray-scale 70% of the time to prevent the network from cheating with low-level statistics.

b. Use a deeper network to solve pretext task

The authors used VGG-16 to solve the pretext task and learn representations. As VGG-16 has increased capacity, it can better handle the increased complexity of the “Jigsaw++” task and thus extract better representation.

c. Transfer Knowledge back to AlexNet

The representations from VGG-16 are clustered and cluster centers are converted to pseudo-labels. Then, AlexNet is trained to classify the pseudo-labels.

d. Finetune AlexNet on Evaluation datasets

For downstream tasks, the convolutional layers for the AlexNet model are initialized with weights from pseudo-label classification and the fully connected layers were randomly initialized. The pre-trained AlexNet is then finetuned on various benchmark datasets.

e. Results

Using a deeper network like VGG-16 leads to better representation and pseudo-labels and also better results in benchmark tasks. It got state of the art results on several benchmarks in 2018 and reduced the gap between supervised and self-supervised methods further.

1. How does the number of clusters affect the performance?

The network is not significantly affected by the number of clusters. The authors tested AlexNet trained on pseudo-labels from a different number of clusters on the task of object detection.

2. How is this different from Knowledge Distillation?

Knowledge transfer is fundamentally different from knowledge distillation. Here, the goal is to only preserve the cluster association of images from the representation and transfer that to the target model. Unlike distillation, we don’t do any regression to the exact output of the teacher.

3. Can you use different datasets in clustering vs predicting pseudo-labels?

Yes, the method is flexible and you can pre-train on one dataset, cluster on another, and get pseudo-labels for the third one.

The authors did an experiment where they trained clustering on representations for ImageNet and then calculated cluster centers on the “Places” dataset to get pseudo-labels. There was only a small reduction (-1.5%) in performance for object classification.

1. Preparing Dataset

To understand this use case, let’s take a subset of 100 movie reviews from the SST-2 dataset which are labeled as positive and negative.

import pandas as pd

df = pd.read_csv('http://bit.ly/dataset-sst2', 
                 nrows=100, sep='\t', names=['text', 'label'])

df['label'] = df['label'].replace({0: 'negative', 1: 'positive'})

The dataset has a column containing the text and a label indicating whether it’s positive or negative opinion.

We will introduce noise into our dataset by corrupting five of the responses with random text. It will act as an outlier for our example.

df.loc[[10, 27, 54, 72, 91], 'text'] = 'askgkn askngk kagkasng'

2. Generating Embeddings

Now, we will compute sentence embeddings for the headlines using the sentence-transformers package. First, let’s install it using pip.

shell

!pip install sentence-transformers

Next, we will create a helper function to return a NumPy array of sentence embeddings given a list of sentences.

from sentence_transformers import SentenceTransformer

sentence_bert_model = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')

def get_embeddings(sentences):
    return sentence_bert_model.encode(sentences,
                                    batch_size=32, 
                                    show_progress_bar=True)

Using the above function, we can generate sentence embeddings for our data as shown below.

e = get_embeddings(df['text'])
# shape: (100, 768)

3. Exporting to Embedding Projector Format

Embedding Projector requires two TSV files to load our custom embeddings. - output.tsv: This file should contain the embeddings without any headers. - metadata.tsv: This file should contain the original text and labels for the embeddings

Let’s first generate the output.tsv file for our sentence embeddings from the previous step.

# Convert NumPy array of embedding into data frame
embedding_df = pd.DataFrame(e)

# Save dataframe as as TSV file without any index and header
embedding_df.to_csv('output.tsv', sep='\t', index=None, header=None)

To generate metadata.csv, we simply save our original dataframe.

# Save dataframe without any index
df.to_csv('metadata.tsv', index=False, sep='\t')

4. Importing into Embedding Projector

We first go to https://projector.tensorflow.org/.

On the left-hand sidebar, click the Load button.

Then, for the first Choose file button, upload the output.tsv file and for the second Choose file button, upload the metadata.tsv file.

After uploading both files, click outside and you should see the sentence embedding projection. The dimensions of embeddings are reduced to 3D by default using PCA.

Let’s switch to 2D by turning off the checkbox for ‘Component #3’ in the bottom part of sidebar.

On the 2D visualization, we can see how the random text is far from other groups of text as an outlier. On hovering the point, we see the text askgkn askngk kagkasng.

5. Useful Features in Projector

a. Pre-processing

In this step, we lowercase the text and remove low informative words such as stop words from the text.

b. Candidate Generation

We split the remaining terms by space and punctuation symbols to get a list of possible keywords.

c. Candidate Scoring

We can count the number of times each term occurs to get a score for each term.

d. Final Ranking

We can sort the keywords in descending order based on the counts and take the top N keywords as the output.

Drawback of Naive Counting

This method has an obvious drawback of only focusing on frequency. But, generic words are likely to be very frequent in any document but are not representative of the domain and topic of the document. We need some way to filter out generic terms.

a. Pre-processing

In this step, we lowercase the text and split the document into sentences.

b. Candidate Generation

We generate 1-gram, 2-gram, and 3-grams candidate phrases from each sentence such that they don’t contain any punctuations. These are our list of candidate phrases.

c. Candidate Scoring

Now, for each candidate keyword “w”, we calculate the TF-IDF score in the following steps.

First, the term frequency(TF) is calculated simply by counting the occurrence of the word.

Then, the inverse document frequency(IDF) is calculated by dividing the total number of documents by the number of documents that contain the word “w” and taking the log of that quantity.

Finally, we get the TF-IDF score for a term by multiplying the two quantities.

d. Final Ranking

We can sort the keywords in descending order based on their TF-IDF scores and take the top N keywords as the output.

How RAKE works

Let’s apply RAKE on a toy example document to understand how it works:

c. Candidate Scoring

Next, the frequency of all the individual words in the candidate keywords are calculated. This finds words that occur frequently.

	deep	learning	subfield	ai	useful
Word Frequency:	1	1	1	1	1

Similarly, the word co-occurrence count is calculated and the degree for each word is the total sum. This metric identifies words that occur often in longer candidate keywords.

	deep	learning	subfield	ai	useful
deep	1	1	0	0	0
learning	1	1	0	0	0
subfield	0	0	1	0	0
ai	0	0	0	1	0
useful	0	0	0	0	1
degree:	1 + 1 = 2	1 + 1 = 2	1	1	1

Then, we divide the degree by the frequency for each word to get a final score. This score identifies words that occur more in longer candidate keywords than individually.

	deep	learning	subfield	ai	useful
Score =	2 / 1 = 2	2 / 1 = 2	1 / 1 = 1	1 / 1 = 1	1 / 1 = 1

Drawbacks of RAKE

• If the stop word list used in RAKE is not exhaustive, it would treat continuous long text as a phrase and give very long phrases.
• Multi-word expressions that contain stop-words could be missed. For example, mention of a brand called “Good Day” could be missed if “good” is present in the stop word list.

Using RAKE in Python

We can use the rake-nltk library to use it in Python as shown below.

shell

pip install rake-nltk

from rake_nltk import Rake
rake = Rake()

text = 'Deep Learning is a subfield of AI. It is very useful.'
rake.extract_keywords_from_text(text)

print(rake.get_ranked_phrases_with_scores())

[(4.0, 'deep learning'), (1.0, 'useful'), (1.0, 'subfield'), (1.0, 'ai')]

How YAKE works:

Using YAKE in Python

To apply YAKE, we will use the pke library. First, we need to install the library and its dependencies using the following command:

shell

pip install git+https://github.com/boudinfl/pke.git
python -m nltk.downloader stopwords
python -m spacy download en

Then, we can use YAKE to generate keywords of maximum length 2 as shown below.

from pke.unsupervised import YAKE
from nltk.corpus import stopwords

document = "Machine learning (ML) is the study of computer algorithms that improve automatically through experience. It is seen as a subset of artificial intelligence."

# 1. Create YAKE keyword extractor
extractor = YAKE()

# 2. Load document
extractor.load_document(input=document,
                        language='en',
                        normalization=None)


# 3. Generate candidate 1-gram and 2-gram keywords
stoplist = stopwords.words('english')
extractor.candidate_selection(n=2, stoplist=stoplist)

# 4. Calculate scores for the candidate keywords
extractor.candidate_weighting(window=2,
                              stoplist=stoplist,
                              use_stems=False)

# 5. Select 10 highest ranked keywords
# Remove redundant keywords with similarity above 80%
key_phrases = extractor.get_n_best(n=10, threshold=0.8)
print(key_phrases)

You get back a list of top-10 keywords and their scores. The highest ranked keyword has the lowest score.

[('machine learning', 0.01552184797949213),
 ('computer algorithms', 0.04188746641162499),
 ('improve automatically', 0.04188746641162499),
 ('machine', 0.12363091320521931),
 ('learning', 0.12363091320521931),
 ('experience', 0.12363091320521931),
 ('artificial intelligence', 0.18075564686791562),
 ('study', 0.2005079697193566),
 ('computer', 0.2005079697193566),
 ('algorithms', 0.2005079697193566)]

1. Precision@k

This metric quantifies how many items in the top-K results were relevant. Mathematically, this is given by:

For our example, precision@1 = 1 as all items in the first 1 results is relevant.

Similarly, precision@2 = 0.5 as only one of the top-2 results are relevant.

Thus, we can calculate the precision score for all k values.

k	1	2	3	4	5
Precision@k

A limitation of precision@k is that it doesn’t consider the position of the relevant items. Consider two models A and B that have the same number of relevant results i.e. 3 out of 5.

For model A, the first three items were relevant, while for model B, the last three items were relevant. Precision@5 would be the same for both of these models even though model A is better.

2. Recall@k

This metric gives how many actual relevant results were shown out of all actual relevant results for the query. Mathematically, this is given by:

For our example, recall@1 = 0.33 as only one of the 3 actual relevant items are present.

Similarly, recall@3 = 0.67 as only two of the 3 actual relevant items are present.

Thus, we can calculate the recall score for different K values.

k	Recall@k
1
2
3
4
5

3. F1@k

This is a combined metric that incorporates both Precision@k and Recall@k by taking their harmonic mean. We can calculate it as:

Using the previously calculated values of precision and recall, we can calculate F1-scores for different K values as shown below.

Metric	Precision@k	Recall@k	F1@k
k=1	1	1/3
k=2	1/2	1/3
k=3	2/3	2/3
k=4	1/2	2/3
k=5	3/5	1

1. Mean Reciprocal Rank(MRR)

This metric is useful when we want our system to return the best relevant item and want that item to be at a higher position. Mathematically, this is given by:

where: - denotes the total number of queries
- denotes the rank of the first relevant result

To calculate MRR, we first calculate the reciprocal rank. It is simply the reciprocal of the rank of the first correct relevant result and the value ranges from 0 to 1.

For our example, the reciprocal rank is as the first correct item is at position 1.

Let’s see another example where the only one relevant result is present at the end of the list i.e. position 5. It gets a lower reciprocal rank score of 0.2.

Let’s consider another example where none of the returned results are relevant. In such a scenario, the reciprocal rank will be 0.

For multiple different queries, we can calculate the MRR by taking the mean of the reciprocal rank for each query.

We can see that MRR doesn’t care about the position of the remaining relevant results. So, if your use-case requires returning multiple relevant results in the best possible way, MRR is not a suitable metric.

2. Average Precision(AP)

Average Precision is a metric that evaluates whether all of the ground-truth relevant items selected by the model are ranked higher or not. Unlike MRR, it considers all the relevant items.

Mathematically, it is given by:

where:

• is an indicator function which is 1 when the item at rank K is relevant.
  
• is the Precision@k metric

For our example, we can calculate the AP based on our Precision@K values for different K.

To illustrate the advantage of AP, let’s take our previous example but place the 3 relevant results at the beginning. We can see that this gets a perfect AP score than the above example.

3. Mean Average Precision(MAP)

If we want to evaluate average precision across multiple queries, we can use the MAP. It is simply the mean of the average precision for all queries. Mathematically, this is given by

where

• is the total number of queries
  
• is the average precision for query q.

1. Cumulative Gain (CG@k)

This metric uses a simple idea to just sum up the relevance scores for top-K items. The total score is called cumulative gain. Mathematically, this is given by:

For our example, CG@2 will be 5 because we add the first two relevance scores 3 and 2.

Similarly, we can calculate the cumulative gain for all the K-values as:

While simple, CG doesn’t take into account the order of the relevant items. So, even if we swap a less-relevant item to the first position, the CG@2 will be the same.

2. Discounted Cumulative Gain (DCG@k)

We saw how a simple cumulative gain doesn’t take into account the position. But, we would normally want items with a high relevance score to be present at a better rank.

Consider an example below. With the cumulative gain, we are simply adding the scores without taking into account their position.

An item with a relevance score of 3 at position 1 is better than the same item with relevance score 3 at position 2.

So, we need some way to penalize the scores by their position. DCG introduces a log-based penalty function to reduce the relevance score at each position. For 5 items, the penalty would be

1
2
3
4
5

Using this penalty, we can now calculate the discounted cumulative gain simply by taking the sum of the relevance score normalized by the penalty. Mathematically, this is given by:

To understand the behavior of the log-penalty, let’s plot ranking position in x-axis and the percentage of relevance score i.e.  in the y-axis. As seen, in position 1, we don’t apply any penalty and score remains unchanged. But, the percentage of score kept decays exponentially from 100% in position 1 to 63% in position 2, 50% in position 3, and so on.

Let’s now calculate DCG for our example.

1	3
2	2
3	3
4	0
5	1

Based on these penalized scores, we can now calculate DCG at various k values simply by taking their sum up to k.

k	DCG@k
DCG@1
DCG@2
DCG@3
DCG@4
DCG@5

There is also an alternative formulation for DCG@K that gives more penalty if relevant items are ranked lower. This formulation is preferred more in industry.

While DCG solves the issues with cumulative gain, it has a limitation. Suppose we a query Q1 with 3 results and query Q2 with 5 results. Then the query with 5 results Q2 will have a larger overall DCG score. But we can’t say that query 2 was better than query 1.

3. Normalized Discounted Cumulative Gain (NDCG@k)

To allow a comparison of DCG across queries, we can use NDCG that normalizes the DCG values using the ideal order of the relevant items.

Let’s take our previous example where we had already calculated the DCG values at various K values.

k	DCG@k
DCG@1
DCG@2
DCG@3
DCG@4
DCG@5

For our example, ideally, we would have wanted the items to be sorted in descending order of relevance scores.

Let’s calculate the ideal DCG(IDCG) for this order.

				IDCG@k
1	3		3 / 1 = 3	3
2	3		3 / 1.5849 = 1.8927	3+1.8927=4.8927
3	2		2 / 2 = 1	3+1.8927+1=5.8927
4	1		1 / 2.3219 = 0.4306	3+1.8927+1+0.4306=6.3233
5	0		0 / 2.5849 = 0	3+1.8927+1+0.4306+0=6.3233

Now we can calculate the NDCG@k for various k by dividing DCG@k by IDCG@k as shown below:

	DCG@k	IDCG@k	NDCG@k
1	3	3
2	4.2618	4.8927
3	5.7618	5.8927
4	5.7618	6.3233
5	6.1486	6.3233

Thus, we get NDCG scores with a range between 0 and 1. A perfect ranking would get a score of 1. We can also compare NDCG@k scores of different queries since it’s a normalized score.

1. Test Types

These are the columns in the previous matrix. There are 3 types of tests proposed in the CheckList framework:

2. Linguistic Capabilities

These are the rows in the CheckList matrix. Each row contains a specific linguistic capability that applies to most NLP tasks.

Let’s understand examples of capabilities given in the original paper. The authors provide a lot of examples to help us build a mental model of how to test new capabilities relevant to our task and domain.

3. Test Generation

The paper’s authors have open-sourced a software tool that can generate test cases at scale based on the ideas above.

The tool provides three approaches to write test cases:

To generate templates, you can either brainstorm them from scratch or generalize patterns from your existing data.

a. Pseudo-label

Lee (2013) proposed a very simple and efficient formulation called “Pseudo-label” in 2013.

The idea is to train a model simultaneously on a batch of both labeled and unlabeled images. The model is trained on labeled images in usual supervised manner with a cross-entropy loss. The same model is used to get predictions for a batch of unlabeled images and the maximum confidence class is used as the pseudo-label. Then, cross-entropy loss is calculated by comparing model predictions and the pseudo-label for the unlabeled images .

The total loss is a weighted sum of the labeled and unlabeled loss terms.

To make sure the model has learned enough from the labeled data, the term is set to 0 during the initial 100 training steps. It is then gradually increased up to 600 training steps and then kept constant.

b. Noisy Student

Xie et al. (2019b) proposed a semi-supervised method inspired by Knowledge Distillation called “Noisy Student” in 2019.

The key idea is to train two separate models called “Teacher” and “Student”. The teacher model is first trained on the labeled images and then it is used to infer the pseudo-labels for the unlabeled images. These pseudo-labels can either be soft-label or converted to hard-label by taking the most confident class. Then, the labeled and unlabeled images are combined together and a student model is trained on this combined data. The images are augmented using RandAugment as a form of input noise. Also, model noise such as Dropout and Stochastic Depth are incorporated in the student model architecture.

Once a student model is trained, it becomes the new teacher and this process is repeated for three iterations.

a. π-model

This model was proposed by Laine and Aila (2017) in a conference paper at ICLR 2017.

The key idea is to create two random augmentations of an image for both labeled and unlabeled data. Then, a model with dropout is used to predict the label of both these images. The square difference of these two predictions is used as a consistency loss. For labeled images, we also calculate the cross-entropy loss. The total loss is a weighted sum of these two loss terms. A weight w(t) is applied to decide how much the consistency loss contributes in the overall loss.

b. Temporal Ensembling

This method was also proposed by Laine and Aila (2017) in the same paper as the pi-model. It modifies the π-model by leveraging the Exponential Moving Average(EMA) of predictions.

The key idea is to use the exponential moving average of past predictions as one view. To get another view, we augment the image as usual and a model with dropout is used to predict the label. The square difference of current prediction and EMA prediction is used as a consistency loss. For labeled images, we also calculate the cross-entropy loss. The final loss is a weighted sum of these two loss terms. A weight w(t) is applied to decide how much the consistency loss contributes in the overall loss.

c. Mean Teacher

This method was proposed by Tarvainen and Valpola (2017). The general approach is similar to Temporal Ensembling but it uses Exponential Moving Average(EMA) of the model parameters instead of predictions.

The key idea is to have two models called “Student” and “Teacher”. The student model is a regular model with dropout. And the teacher model has the same architecture as the student model but its weights are set using an exponential moving average of the weights of student model. For a labeled or unlabeled image, we create two random augmented versions of the image. Then, the student model is used to predict label distribution for first image. And, the teacher model is used to predict the label distribution for the second augmented image. The square difference of these two predictions is used as a consistency loss. For labeled images, we also calculate the cross-entropy loss. The final loss is a weighted sum of these two loss terms. A weight w(t) is applied to decide how much the consistency loss contributes in the overall loss.

d. Virtual Adversarial Training

This method was proposed by Miyato et al. (2019). It uses the concept of adversarial attack for consistency regularization.

The key idea is to generate an adversarial transformation of an image that will change the model prediction. To do so, first, an image is taken and an adversarial variant of it is created such that the KL-divergence between the model output for the original image and the adversarial image is maximized.

Then we proceed as previous methods. We take a labeled/unlabeled image as first view and take its adversarial example generated in previous step as the second view. Then, the same model is used to predict label distributions for both images. The KL-divergence of these two predictions is used as a consistency loss. For labeled images, we also calculate the cross-entropy loss. The final loss is a weighted sum of these two loss terms. A weight is applied to decide how much the consistency loss contributes in the overall loss.

e. Unsupervised Data Augmentation

This method was proposed by Xie et al. (2019a) and works for both images and text. Here, we will understand the method in the context of images.

The key idea is to create an augmented version of a unlabeled image using AutoAugment. Then, a same model is used to predict the label of both these images. The KL-divergence of these two predictions is used as a consistency loss. For labeled images, we only calculate the cross-entropy loss and don’t calculate any consistency loss. The final loss is a weighted sum of these two loss terms. A weight w(t) is applied to decide how much the consistency loss contributes in the overall loss.

a. MixMatch

This holistic method was proposed by Berthelot et al. (2019).

To understand this method, let’s take a walk through each of the steps.

1. For the labeled image, we create an augmentation of it. For the unlabeled image, we create K augmentations and get the model predictions on all K-images. Then, the predictions are averaged and temperature scaling is applied to get a final pseudo-label. This pseudo-label will be used for all the K-augmentations.

   

2. The batches of augmented labeled and unlabeled images are combined and the whole group is shuffled. Then, the first N images of this group are taken as , and the remaining M images are taken as .

   

3. Now, Mixup is applied between the augmented labeled batch and group . Similarly, mixup is applied between the M augmented unlabeled group and the group. Thus, we get the final labeled and unlabeled group.
   

4. Now, for the labeled group, we take model predictions and compute cross-entropy loss with the ground truth mixup labels. Similarly, for the unlabeled group, we compute model predictions and compute mean square error(MSE) loss with the mixup pseudo labels. A weighted sum is taken of these two terms with weighting the MSE loss.

   

b. FixMatch

This method was proposed by Sohn et al. (2020) and combines pseudo-labeling and consistency regularization while vastly simplifying the overall method. It got state of the art results on a wide range of benchmarks.

As seen, we train a supervised model on our labeled images with cross-entropy loss. For each unlabeled image, weak augmentation and strong augmentations are applied to get two images. The weakly augmented image is passed to our model and we get prediction over classes. The probability for the most confident class is compared to a threshold. If it is above the threshold, then we take that class as the ground label i.e. pseudo-label. Then, the strongly augmented image is passed through our model to get a prediction over classes. This prediction is compared to ground truth pseudo-label using cross-entropy loss. Both the losses are combined and the model is optimized.

If you want to learn more about FixMatch, I have an article that goes over it in depth.