Reinforcement learning (RL) is an area of machine learning which allows AI based systems to make decisions in constantly changing context using trial and error in order to increase rewards shared by all in response to feedback by individual actions. In RL situation feedback is reference to either negative or positive concept that is reflected in rewards or consequences.
Reinforcement Learning (RL)
RL improves AI driven systems through replicating natural intelligence and mimicking human cognitive abilities. This type of learning method helps computers to make important choices that yield astonishing results with respect to tasks they are assigned without intervention of humans or requirement to program explicitly AI system.
Some known RL methods that have added subtle dynamic element to conventional ML methods include Monte Carlo state action reward state action (SARSA) and Q learning. AI models trained over reinforcement learning algorithms have defeated human counterparts in several video games and board games including chess and Go.
Technically speaking RL implementations may be divided into three categories:
- Policies based policy based RL strategy aims at maximizing rewards for system through predictable policies strategies and methods.
- Value based value based RL implementation aims to maximize variable value functions involved in learning process.
- Model based approach based on models lets you create an imaginary setting that is based on an environment. Additionally all system agents work within virtual parameters.
A common reinforcement learning model could be represented as:
In figure above computers could represent an agent within certain condition (St). computer takes actions (At) within context to reach certain target. Based on performing action agent will receive feedback in form of reward (R) or punishment (R).
The Benefits of Reinforcement Learning
Reinforcement learning solves myriad of issues that conventional ML algorithms cannot solve. RL is well known for its capability to complete tasks on its own utilizing every possible avenue which can be compared with AI or artificial general intelligence (AGI).
The main advantages of RL include:
- is focused on goal of long term success Most algorithmic approaches to ML divide issues into subproblems that are addressed separately without focusing on primary issue. But RL is more about reaching goal over time without breaking problem down into smaller tasks thus maximizing reward.
- Data collection is simple It doesnt require an autonomous data collection procedure. Since agent is operating in context learning data is collected dynamically through agents responses and experiences.
- is program that operates in an ever changing unstable and unpredictable environment Techniques for RL are based around an adaptive framework that grows with time while agent continues to interact with surrounding. In addition as environment changes limitations RL algorithms tweak and modify themselves in order to better perform.
How Does Reinforcement Learning Work?
The principle behind reinforcement learning relies on function of reward. We will explore RL process with aid of an illustration.
Well assume that you plan to instruct you pet (dog) specific tricks.
- Because your dog isnt able to comprehend language of us we have to use different approach.
- The scenario is one in which pet is required to complete an exact task and then give rewards (such as reward) to animal.
- When animal is faced with similar scenario and is faced with similar situation it attempts to complete exactly same thing that before earned him reward and with more excitement.
- The animal learns from positive experiences it has had and repeats these actions until its now aware of what do when specific circumstance is encountered.
- In same way as dog it becomes conscious of what to keep out of situation.
Use for case
In preceding case
- Your dog (dog) is an agent moving around your home and that is your environment. state of being is dogs posture that is sitting. This may be transformed into walking if you speak an appropriate phrase.
- The shift from walking to sitting occurs as agent reacts to words you speak when they are within surroundings. policy allows agents to act within certain condition and then expect more favorable outcomes.
- When dog is in another state (walk) and then walks pet receives reward (dog meal).
RL Stepwise Workflow
The process of reinforcement learning includes training agent while looking at following aspects:
- Environment
- Reward
- Agent
- Training
- Deployment
Lets go over each of them by examining each one in depth.
Step 1: Define/create Environment
The RL procedure starts by delineating conditions in which an agent will be actively. term “environment” could refer to actual physical system or an artificial space. When location of environment has been determined then experiments can be initiated with RL procedure.
Step 2: Define Amount of Reward
As next step you must define your agents reward. This is an indicator of performance for agent and allows agents to assess quality of their work against their targets. Furthermore providing an appropriate reward to agent might require several iterations before you can determine appropriate reward to reward particular action.
Step 3: Define Agent
After rewards and environment are set you may make an agent who outlines guidelines for each as well as RL learning algorithm. procedure can comprise below steps:
- Make use of appropriate neural networks and lookup tables to reflect policies
- Pick most suitable RL learning algorithm.
Step 4: Verify Agents Training/validation
Validate and train your agent in order to improve policies for training. In addition you should focus on structure of rewards RL structure and then continue to train agent. RL training can be time consuming that can last from between few minutes and days depending upon use. Therefore when you have complicated collection of apps speedier training can be accomplished by utilizing system design where multiple processors GPUs and other computing devices run in same way in parallel.
Stage 5: Apply Guidelines
In system that is RL enabled policy acts as decision making element that is implemented with C C++ or CUDA development software.
In course of implementing these rules reviewing first steps of RL process is often necessary to make sure that best decisions are made or outcomes arent achieved.
Reinforcement Learning Algorithms
RL algorithms are essentially divided into two categories models based algorithms and model free ones. Further subdividing them algorithms are classified under off policy as well as on policy categories.
A model based algorithm is one in which there is clearly defined RL model that is able to learn from current states as well as actions and state shifts that result from actions. Therefore they save state and actions data to be used in future. However model free algorithm operates by trial and error which eliminate requirement to store state and actions data in memory.
Off policy and on policy algorithms can be understood better with aid of these mathematical symbols:
The letters symbolizes state of system while letter “a” represents actions and symbol “p” represents likelihood of determining reward. Q(s A) function can be useful for process of prediction and provides potential rewards for agents in sense of understanding and learning from state as well as actions and changes in state.
So on policy system uses Q(s the) function to gain knowledge from current state of affairs and activities in contrast to off policy which focuses on understanding [Q(s an)[Q(s a)] through random state and actions.
Furthermore it is true that furthermore Markov decision making process is focused on present state of affairs which can help predict future states more instead of relying on previous information about state. It is result that likelihood of state is influenced by actual state conditions more than on procedure that results in present condition. Markov property plays vital function in reinforcement learning.
Well now look at most important RL algorithms:
1. Q learning
Q learning is an off policy model free algorithm that teaches through random acts (greedy practice). word “Q” in Q learning is reference to effectiveness of actions which maximize reward generated by algorithmic procedure.
Q learning algorithms use reward matrix as way to record rewards earned. For instance for reward of 50 points matrix is created that assigns an amount at position 50 which is used to indicate reward of 50. This value is updated with techniques like value iteration method and policy iteration.
It refers to process of improving or refining policy through measures that increase values function. When value iteration is conducted where values that make up value function get changed. Mathematically speaking Q learning can be expressed as formula:
Q(sa) = (1 a).Q(sa) + a. (R + g.max(Q(S2a)).
Where
alpha = rate of learning
gamma = discount factor
R = reward
S2 = next state.
Q(S2a) = value to be expected in future.
2. SARSA
The State Action Reward State Action (SARSA) algorithm is an on policy method. Therefore it doesnt adhere to shrewd approach of Q learning. In contrast SARSA learns from present state of affairs and actions taken for implementation of RL procedure.
3. Deep Q Network (DQN)
In contrast to SARSA and Q learning SARSA deep Q network is neural network and doesnt rely on arrays of 2D. These algorithms fail when it comes to predicting and updating states theyre unaware of. state they are unaware of is generally in form of unknown state values.
Therefore that In DQN 2D arrays are replaced with neural networks that allow for effective calculation of state values as well as states that are associated with state transformations and speeding process of learning RL.
Uses of Reinforcement Learning
Reinforcement learning was created to increase amount of rewards received by agents when they complete particular job. RL can benefit wide range of situations and scenarios in real life that include autonomous cars robots surgeons and AI robots.
The following are some of key ways that reinforcement learning can be applied every day life which define field of AI.
1. Controlling Self Driving Vehicles
In order for vehicles to function completely autonomously in urban settings They require significant amount of help by model based models which simulate all possibilities of scenarios and scenes they could encounter. RL can come in handy when needed as models are taught in fluid setting in which all possibilities are studied and then sorted out during process of learning.
The ability to learn from past experiences This makes RL ideal choice for cars with self driving capabilities that require to make best decision making on fly. Many variables like management of driving zones controlling flow of traffic observing vehicle speed and preventing accidents are easily handled with RL method.
A group of researchers have designed similar simulation for autonomous machines like automobiles and drones at MIT and is dubbed DeepTraffic. This project is free source platform that creates algorithms through mixing RL deep learning as well as computers with restrictions.
2. Resolving Issue of Energy Consumption
As rate of growth in AI technology administration have ability to handle serious problems including energy use in present. In addition growing amount of IoT gadgets and increasing use of industrial commercial and corporate systems has made servers more alert.
In course of gaining recognition its been discovered that RL agents with no prior understanding of servers conditions can be adept at controlling physical conditions surrounding servers. This data comes from multiple sensors that gather temperatures energy as well as other information to aid in development of neural deep networks and in process aiding in coolness of data center as well as controlling energy usage. Most often Q learning network (DQN) techniques are employed for these cases.
3. Control of Traffic Signals
Growing demand and urbanization of vehicles in cities has raised alarms for authorities who are trying to reduce traffic congested urban settings. solution is reinforcement learning. RL models provide control of traffic lights in accordance with current traffic situation in particular area.
It is fact that model takes into account traffic coming in multiple directions and is able to learn adjust and alters traffic signal signals within urban traffic networks.
4. Healthcare
RL has significant role within medical field as it is primary way that DTRs (Dynamic Treatment Regimes) have assisted medical professionals with handling health of patients. DTRs make use of series of actions to come up an end to end solution. This process could include these actions:
- Find out status of patients life
- Select type of treatment to be used.
- Find right dosage of medication in accordance with state of patient.
- Determine dosage timers as well as other dosage timings.
Through this process of making decisions doctors are able to fine tune their treatment plan and identify complicated diseases such as depression diabetes and cancer. Furthermore DTRs may assist in delivering therapies at correct timing and without any issues caused by delayed action.
5. Robotics
It is field in which one helps robot learn to emulate human behaviour while performing an activity. Todays robotics machines arent able to demonstrate morality social skills or common sense in performing task. When this happens AI sub fields such as deep learning and RL are able to be combined (Deep Reinforcement Learning) to achieve better results.
Deep RL is vital to robots with warehouse navigation while providing important product components packaging assembly of product as well as defect inspection. In particular deep RL models have been trained on multimodal information which are crucial in determining fractures missing parts scratches and overall damages to warehouses by analyzing images and comparing billions of information points.
Furthermore deep RL assists in management of inventory since personnel are able to identify empty containers and replenish containers immediately.
6. Marketing
RL aids organizations to maximize their customer satisfaction and improve their business plans to meet long term objectives. Marketing is one of areas where RL helps in giving individual recommendations to customers through predicting their decisions as well as their reactions in relation to specific products or services.
The RL trained bots are also trained to consider various aspects including changing customers mindsets and adjusts its user needs in response to their actions. Businesses can offer specific and high quality recommendations that in turn increases their profits.
7. Gaming
Agents learning reinforcement learn to adjust to game environment while they apply logic they have learned through their experience and attain desired outcomes through series of actions.
In case of Googles DeepMind for instance AlphaGo beat standard Go player during October. 2015. It was an incredible leap for AI models at day. Apart from creating games like AlphaGo using deep neural networks RL agents can be used to test games and for bugs detection in game environment. Bugs that could be problem are quickly identified because RL performs multiple runs with no external interference. As an example companies in gaming industry like Ubisoft make use of RL to identify bugs.