In my previous post I showed how to build a maze-solving Q-learner using a table-based Q-learning method. In this post, I show how to solve the same maze using DQN (Deep Q-Learning). The code for this example program is found here.
In a reinforcement learning problem, an Agent interacts with an Environment by evaluating its State, taking Actions, and receiving Rewards. The goal is to learn which Actions provide the most Reward over the course of time.
As in the previous article, we’re going to attempt to solve the maze problem. …
Q-learning is an algorithm in which an agent interacts with its environment and collects rewards for taking desirable actions.
The simplest implementation of Q-learning is referred to as tabular or table-based Q-learning. There are tons of articles, tutorials, etc. already available on the web which describe Q-learning so I won’t go into excruciating detail here. Instead, I want to show how efficiently table-base Q-learning can be done using tinymind. In this article, I will describe how tinymind implements Q-learning using C++ templates and fixed-point (Q-format) numbers as well as go thru the example in the repo.
Years ago, I began writing C++ template libraries with the goal of instantiating and running neural networks and machine learning algorithms within embedded systems. I did not have an FPU (floating point unit), GPU (graphics processing unit), or any vectorized instructions(e.g. SIMD) at my disposal so I needed to ensure I could run these algorithms with only a very simple CPU. Memory to store code and data was also highly constrained in my environment. I didn’t have room to store code and data that I wasn’t going to use.
My inspiration for these libraries was Andrei Alexandrescu’s Modern C++ Design…
Q-Format is a binary, fixed-point number format. Tinymind contains a template library which allows us to specify the number of integer as well as the number of fractional bits as template parameters. Q-Format can be used in lieu of floating point numbers when fixed-point precision is adequate (or when we don’t have an FPU at all!). See Q-Format for a deep dive.
Tinymind contains a C++ template library for defining and using Q format values. The template class is called QValue. The template declaration is here:
template< unsigned NumFixedBits, unsigned NumFractionalBits, bool QValueIsSigned, template<typename, unsigned> class QValueRoundingPolicy = TruncatePolicy >…
Currently working for Intel as a Principal Engineer in the Non-Volatile Memory Solutions Group. My primary interests are machine learning and neural networks.