Overview of Adaptive Learning-Rate-Based Optimisation Algorithms for Neural Networks


Efficient learning algorithms are crucial for any machine learning algorithm in health care and other applications. A very crucial parameter for gradient-based optimisation algorithms is the learning rate, which determines how big changes in the network's parameters are within each optimisation step. Within the last few years, the concept of an adaptive learning rate has achieved great improvements in performance for learning algorithms.

Task The task for this topic is to point out the importance of the learning rate in optimization algorithms and give an overview of the concepts and advantages of existing adaptive learning-rate-based algorithms like RMSProp, ADADelta and Adam. For this purpose it will be necessary to search for, read, understand and sum up existing literature on the topic.
Utilises None, literature review only
Requirements Basic knowledge of neural networks, proficiency with calculus
Languages German or English

Manuel Milling (