Introduction

In my previous post I showed how to build a maze-solving Q-learner using a table-based Q-learning method. In this post, I show how to solve the same maze using DQN (Deep Q-Learning). The code for this example program is found here.

Reinforcement Learning

In a reinforcement learning problem, an Agent interacts with an Environment by evaluating its State, taking Actions, and receiving Rewards. The goal is to learn which Actions provide the most Reward over the course of time.

Image for post
Image for post

The Maze Problem

As in the previous article, we’re going to attempt to solve the maze problem. …


Introduction

Q-learning is an algorithm in which an agent interacts with its environment and collects rewards for taking desirable actions.

Image for post
Image for post

The simplest implementation of Q-learning is referred to as tabular or table-based Q-learning. There are tons of articles, tutorials, etc. already available on the web which describe Q-learning so I won’t go into excruciating detail here. Instead, I want to show how efficiently table-base Q-learning can be done using tinymind. In this article, I will describe how tinymind implements Q-learning using C++ templates and fixed-point (Q-format) numbers as well as go thru the example in the repo.

The Maze Problem

A common table-based Q-learning…


Introduction

Years ago, I began writing C++ template libraries with the goal of instantiating and running neural networks and machine learning algorithms within embedded systems. I did not have an FPU (floating point unit), GPU (graphics processing unit), or any vectorized instructions(e.g. SIMD) at my disposal so I needed to ensure I could run these algorithms with only a very simple CPU. Memory to store code and data was also highly constrained in my environment. I didn’t have room to store code and data that I wasn’t going to use.

My inspiration for these libraries was Andrei Alexandrescu’s Modern C++ Design


Introduction

Q-Format is a binary, fixed-point number format. Tinymind contains a template library which allows us to specify the number of integer as well as the number of fractional bits as template parameters. Q-Format can be used in lieu of floating point numbers when fixed-point precision is adequate (or when we don’t have an FPU at all!). See Q-Format for a deep dive.

Tinymind QValue

Tinymind contains a C++ template library for defining and using Q format values. The template class is called QValue. The template declaration is here:

template< unsigned NumFixedBits, unsigned NumFractionalBits, bool QValueIsSigned, template<typename, unsigned> class QValueRoundingPolicy = TruncatePolicy >…

Dan McLeran

Currently working for Intel as a Principal Engineer in the Non-Volatile Memory Solutions Group. My primary interests are machine learning and neural networks.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store