Benjamin Heinzerling

@benbenhh

Natural Language Understanding Team Deputy Director @RIKEN_AIP, Associate Professor @TohokuNLP, based in Sendai, Japan. PhD @UniHeidelberg @HITStudies

Sendai, Japan.

github.com/bheinzerling

Replies Reposts Media

Pinned
Benjamin Heinzerling
@benbenhh
Jan 31, 2019
We've released a new version of our pretained byte pair embeddings in 275 languages. Now with pip install, automatic download of embeddings and sentencepiece models, convenient subword segmentation, and tons of pretty UMAP visualizations. nlp.h-its.org/bpemb
1
Benjamin Heinzerling
@benbenhh
Mar 26, 2024
LLMs give the right answer to questions like "When was Karl Popper born?" but do they have the right internal representation? We find evidence that they do: There are directions in activation space that have a causal, interpretable effect on the expression of numeric properties
1
Benjamin Heinzerling
@benbenhh
Jul 3, 2022
There is nothing "sudden" about the increase in validation accuracy. The apparent sudden increase is just an artifact of the x-axis log scale. A linear scale reveals a steady increase. That's still a really cool finding, but "grokking" is a bit of a misnomer
Awni Hannun
@awnihannun
Jul 1, 2022
Read a bit about Grokking recently. Here's some learnings: "Grokking" is a curious neural net behavior observed ~1 year ago (arxiv.org/abs/2201.02177). Continue optimizing a model long after perfect training accuracy and it suddenly generalizes. Figure:
5
Benjamin Heinzerling
@benbenhh
May 28, 2022
It's kinda ridiculous to have these million dollar LMs and then NLP researchers are like: the current SOTA prompt for recipe generation is "I want to cook a yummy meal" but we found that average recipe tastiness increased 1.3% if you append "#potatosalad" and a cow emoji
4
Benjamin Heinzerling
@benbenhh
Apr 22, 2021
If language models are knowledge bases, how much knowledge can they store? How robustly can they be queried? In our paper we give some answers to these questions. Poster: bheinzerling.github.io/publication/he… Paper: aclweb.org/anthology/2021… If you're at #EACL2021, poster session starting now
1