This seminar provides an introduction to Natural Language Processing (NLP), covering fundamental concepts such as tokenization, Named Entity Recognition (NER), and sentiment analysis. We will show how many NLP tasks can be translated into regular data science problem through the use of embeddings and introduce different embedding strategies for words and sentences obtained at different stages of history. Finally, we will explore the current industrial applications of Large Language Models (LLMs) including prompt engineering strategies, RAG and chatbots. Attendees will gain insights into how LLMs work, their advantages, and real-world use cases.

Lecture 1. Intro and theory for shallow networks 

• Perceptron convergence theorem
• Universal approximation theorem
• Approximation rates for shallow neural networks
• Barron spaces

• Advantages of additional hidden layers
• Deep ReLU networks
• Misclassification error for image deformation models

• Optimization in machine learning
• Weight balancing phenomenon
• Analysis of dropout
• Benign overfitting
• Grokking

In the first part of the course course, I will review some models of statistical mechanics and discuss a central question in the study of disordered systems, the so-called disorder relevance. This question is quite broad and mainly consists in determining whether the properties of a system are stable under small (random) perturbations or whether they are affected by the presence of a small noise.

For the rest of the course I will develop at length an example: the Directed Polymer Model. The Directed Polymer Model is based on a Simple Random Walk interacting with a random environment, and it has seen an incredible activity (and important progress) over the past decade. In fact, even though the model is very simply defined (it is a disordered version of the simple random walk), it exhibits a wide range of behaviours and in particular it undergoes a phase transition as the intensity of disorder varies. I will first describe important features of this model, for instance showing the presence of a phase transition and discussing the role of the dimension in the question of disorder relevance. I will then present very recent result on several fronts:
• the recent characterization of the phase transition in dimension d≥3,
• the construction of a disordered scaling limit and its relation to the Stochastic Heat Equation,
• the case of the critical dimension d=2.