Johannes von Oswald

Research Interests

My research is focused on learning algorithms for neural networks to tackle problems with continuous and sparse data. Under these assumptions, the trade-off between model bias and sample efficiency lies at the heart of my interests in biological and artificial neural networks, statistical learning theory and reinforcement learning. In particular, during my PhD I am developing tools around a class of neural network architectures termed HyperNetworks that can incorporate model design and its optimization into a single framework.

I am gratefull for the support I receive from the Swiss Data Science Center.

At the moment, I am doing an internship at Google Zurich with Alexander Mordvintsev and Max Vladymyrov.

Selected Publications

Transformers learn in-context by gradient descent
(joint work with Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov), 2022.

A contrastive rule for meta-learning
(joint work Nicolas Zucchet, Simon Schug, Johannes von Oswald, Dominic Zhao and João Sacramento
36th Conference on Neural Information Processing Systems (NeurIPS), 2022.

Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
(joint work Seijin Kobayashi and Pau Vilimelis Aceituno)
36th Conference on Neural Information Processing Systems (NeurIPS), 2022.

The least-control principle for learning at equilibrium
(joint work with Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi and João Sacramento)
36th Conference on Neural Information Processing Systems (NeurIPS), 2022.

Learning where to learn: Gradient sparsity in meta and continual learning
(joint work with Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet and João Sacramento)
35th Conference on Neural Information Processing Systems (NeurIPS), 2021.

On the reversed bias-variance tradeoff in deep ensembles
(joint work with Seijin Kobayashi and Benjamin Grewe)
ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning, 2021

Neural networks with late-phase weights
(joint work with Seijin Kobayashi, Benjamin Grewe and João Sacramento)
International Conference on Learning Representations (ICLR), 2021

Continual learning with hypernetworks
(joint work with Christian Henning, Benjamin Grewe and João Sacramento)
International Conference on Learning Representations (ICLR), 2020

Meta-learning via hypernetworks
(joint work with Dominic Zhao, Seijin Kobayashi, João Sacramento)
NeurIPS Workshop on Meta-Learning , 2020


Check Google scholar for an updated and complete publication history.

Please do not hesitate to contact me if you are interested in doing a research project, Bachelor or Master thesis with me.