ImitationLearningASurveyofLearningMethods.pdf_福昕ofd转pdf资源-CSDN下载

需积分: 10 130 浏览量 2019-07-29 09:36:52 上传评论收藏 952KB PDF 举报

《Imitation Learning: A Survey of Learning Methods.pdf》这份文档是一篇关于模仿学习方法的综述，旨在介绍和分类模仿学习的各种技术，并讨论在不同学习阶段的设计选择。文档指出，模仿学习是一种模仿人类行为以完成特定任务的学习技术，其中学习机器（agent）通过观察和行为之间的映射，从示范中学习并执行任务。该领域虽然历史悠久，但最近由于计算和感知能力的进步以及对智能应用日益增长的需求，受到了越来越多的关注。模仿学习的范式之所以受欢迎，是因为它有助于简化复杂任务的教学过程，几乎不需要对任务的专家知识，也不需要对任务特定的显式编程或设计奖励函数。现代传感器能够快速收集和传输大量数据，而具有高计算能力的处理器能够迅速处理这些感知数据，并及时映射到相应的动作上。这种能力为很多需要实时感知和反应的人工智能应用打开了大门，例如人形机器人、自动驾驶车辆、人机交互和电脑游戏等。文档强调了模仿学习在实现过程中面临的挑战，这需要特定的算法来有效地学习模型，而这种学习方式有自己的问题。例如，在机器人和游戏领域，由于这些领域的文献最为丰富，并提供了广泛的问题和方法论，因此，特别关注了这两个领域的学习方法。在机器人技术中，模仿学习能够用于训练机器人进行特定的任务，如操纵物体、导航环境等。而游戏领域的应用，如利用模仿学习技术训练电脑玩家来模仿真实玩家的行为，或解决游戏中遇到的特定挑战。文档还讨论了设计和评估模仿学习任务的方法。这些方法被分类和回顾，并特别强调了在不同学习阶段的设计选择。最终，文章讨论了如何将模仿学习与其他学习方法结合使用，以及如何在多个领域中应用模仿学习技术。文档作者们来自罗伯特戈登大学的计算科学与数字媒体学院以及伯明翰城市大学的计算与数字技术学院，他们共同强调了在实际应用中模仿学习所面临的挑战，并提出了潜在的解决方案和未来的研究方向。以上内容仅为文档部分内容的解读与归纳，完整内容更全面地覆盖了模仿学习的原理、方法分类、技术挑战以及在多个应用领域中的实际应用案例。该文档为机器学习和人工智能爱好者提供了宝贵的学习资源，对于希望深入了解模仿学习这一领域的人来说，它是一个很好的起点。

资源推荐

资源详情

资源评论

Imitation Learning: A Survey of Learning Methods

Ahmed Hussein, School of Computing Science and Digital Media, Robert Gordon University

Mohamed Medhat Gaber, School of Computing and Digital Technology, Birmingham City University

Eyad Elyan, School of Computing Science and Digital Media, Robert Gordon University

Chrisina Jayne, School of Computing Science and Digital Media, Robert Gordon University

Imitation learning techniques aim to mimic human behavior in a given task. An agent (a learning machine)

is trained to perform a task from demonstrations by learning a mapping between observations and actions.

The idea of teaching by imitation has been around for many years, however, the ﬁeld is gaining attention

recently due to advances in computing and sensing as well as rising demand for intelligent applications. The

paradigm of learning by imitation is gaining popularity because it facilitates teaching complex tasks with

minimal expert knowledge of the tasks. Generic imitation learning methods could potentially reduce the

problem of teaching a task to that of providing demonstrations; without the need for explicit programming

or designing reward functions speciﬁc to the task. Modern sensors are able to collect and transmit high

volumes of data rapidly, and processors with high computational power allow fast processing that maps the

sensory data to actions in a timely manner. This opens the door for many potential AI applications that

require real-time perception and reaction such as humanoid robots, self-driving vehicles, human computer

interaction and computer games to name a few. However, specialized algorithms are needed to effectively

and robustly learn models as learning by imitation poses its own set of challenges. In this paper, we sur-

vey imitation learning methods and present design options in different steps of the learning process. We

introduce a background and motivation for the ﬁeld as well as highlight challenges speciﬁc to the imitation

problem. Methods for designing and evaluating imitation learning tasks are categorized and reviewed. Spe-

cial attention is given to learning methods in robotics and games as these domains are the most popular in

the literature and provide a wide array of problems and methodologies. We extensively discuss combining

imitation learning approaches using different sources and methods, as well as incorporating other motion

learning methods to enhance imitation. We also discuss the potential impact on industry, present major

applications and highlight current and future research directions.

CCS Concepts:

General and reference → Surveys and overviews;

Computing methodologies →

Learning paradigms; Learning settings; Machine learning approaches; Cognitive robotics; Control

methods; Distributed artiﬁcial intelligence; Computer vision;

General Terms: Design, Algorithms

Additional Key Words and Phrases: Imitation learning, learning from demonstrations, intelligent agents,

learning from experience, self-improvement, feature representations, robotics, deep learning, reinforcement

learning

ACM Reference Format:

Ahmed Hussein, Mohamed M. Gaber, Eyad Elyan, and Chrisina Jayne, 2016. Imitation Learning: A Survey

of Learning Methods. ACM Comput. Surv. V, N, Article A (January YYYY), 35 pages.

DOI: http://dx.doi.org/10.1145/0000000.0000000

Author’s addresses: A. Hussein, E. Elyan, and Chrisina Jayne School of Computing Science and Digital

Media, Robert Gordon University, Riverside East, Garthdee Road, Aberdeen AB10 7GJ, United Kingdom

M. M. Gaber, School of Computing and Digital Technology, Birmingham City University, 15 Bartholomew

Row, Birmingham B5 5JU, United Kingdom

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted

without fee provided that copies are not made or distributed for proﬁt or commercial advantage and that

copies bear this notice and the full citation on the ﬁrst page. Copyrights for components of this work owned

by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or repub-

lish, to post on servers or to redistribute to lists, requires prior speciﬁc permission and/or a fee. Request

permissions from [email protected].

 YYYY ACM. 0360-0300/YYYY/01-ARTA $15.00

DOI: http://dx.doi.org/10.1145/0000000.0000000

ACM Computing Surveys, Vol. V, No. N, Article A, Publication date: January YYYY.

A:2 A. Hussein et al.

1. INTRODUCTION

In recent years, the demand for intelligent agents capable of mimicking human behav-

ior has grown substantially. Advancement in robotics and communication technology

have given rise to many potential applications that need artiﬁcial intelligence that can

not only make intelligent decisions, but is able to perform motor actions realistically

in a variety of situations. Many future directions in technology rely on the ability of

artiﬁcial intelligence agents to behave as a human would when presented with the

same situation. Examples of such ﬁelds are self-driving vehicles, assistive robots and

human computer interaction. For the latter especially, opportunities for new applica-

tions are growing due to recent interest in consumer virtual reality and motion capture

systems

. In these applications and many robotics tasks, we are faced with the prob-

lem of executing an action given the current state of the agent and its surroundings.

The number of possible scenarios in a complex application is too large to cover by ex-

plicit programming and so a successful agent must be able to handle unseen scenarios.

While such a task may be formulated as an optimization problem, it has become widely

accepted that having prior knowledge provided by an expert is more effective and ef-

ﬁcient than searching for a solution from scratch [Schaal 1999] [Schaal et al. 1997]

[Bakker and Kuniyoshi 1996] [Billard et al. 2008]. In addition, optimization through

trial and error requires reward functions that are designed speciﬁcally for each task.

One can imagine that even for simple tasks, the number of possible sequences of ac-

tions an agent can take grows exponentially. Deﬁning rewards for such problems is

difﬁcult, and in many cases unknown.

One of the more natural and intuitive ways of imparting knowledge by an expert

is to provide demonstrations for the desired behavior that the learner is required to

emulate. It is much easier for the human teacher to transfer their knowledge through

demonstration than to articulate it in a way that the learner will understand [Raza

et al. 2012]. This paper reviews the methods used to teach artiﬁcial agents to perform

complex sequences of actions through imitation.

Imitation learning is an interdisciplinary ﬁeld of research. Existing surveys focus

on different challenges and perspectives of tackling this problem. Early surveys re-

view the history of imitation learning and early attempts to learn from demonstra-

tion [Schaal 1999] [Schaal et al. 2003]. In [Billard et al. 2008] learning approaches

are categorized as engineering oriented and biologically oriented methods, [Ijspeert

et al. 2013] focus on learning methods from the viewpoint of dynamical systems, while

[Argall et al. 2009] address different challenges in the process of imitation such as

acquiring demonstrations, physical and sensory issues as well as learning techniques.

However, due to recent advancements in the ﬁeld and a surge in potential applications,

it is important at this time to conduct a survey that focuses on the computational meth-

ods used to learn from demonstrated behavior. More speciﬁcally, we review artiﬁcial

intelligence methods which are used to learn policies that solve problems according

to human demonstrations. By focusing on learning methods, this survey addresses

learning for any intelligent agent, whether it manifests itself as a physical robot or

a software agent (such as games, simulations, planning, etc. ). The reviewed litera-

ture addresses various applications, however, many of the methods used are generic

and can be applied to general motion learning tasks. The learning process is catego-

rized into: creating feature representations, direct imitation and indirect learning. The

methods and sources of learning for each process are reviewed as well as evaluation

metrics and applications suitable for these methods.

In the last two years the virtual reality market has attracted major technology companies and billions

of dollars in investment and is still rapidly growing. http://www.fastcompany.com/3052209/tech-forecast/vr-

and-augmented-reality-will-soon-be-worth-150-billion-here-are-the-major-pla?partner=engadget

ACM Computing Surveys, Vol. V, No. N, Article A, Publication date: January YYYY.

Imitation Learning: A Survey of Learning Methods A:3

Imitation learning refers to an agent’s acquisition of skills or behaviors by observing

a teacher demonstrating a given task. With inspiration and basis stemmed in neuro-

science, imitation learning is an important part of machine intelligence and human

computer interaction, and has from an early point been viewed as an integral part in

the future of robotics [Schaal 1999]. Another popular paradigm is learning through

trial and error; however, providing good examples to learn from expedites the process

of ﬁnding a suitable action model and prevents the agent from falling into local min-

ima. Moreover, a learner could very well arrive on its own at a suitable solution, i.e one

that achieves a certain quantiﬁable goal, but which differs signiﬁcantly from the way

a human would approach the task. It is sometimes important for the learner’s actions

to be believable and appear natural. This is necessary in many robotic domains as well

as human computer interaction where the performance of the learner is only as good

as a human observer’s perception of it. It is therefore favorable to teach a learner the

desired behavior from a set of collected instances. However, it is often the case that

direct imitation of the expert’s motion doesn’t sufﬁce due to variations in the task such

as the positions of objects or inadequate demonstrations. Therefore imitation learning

techniques need to be able to learn a policy from the given demonstrations that can

generalize to unseen scenarios. As such the agent learns to perform the task rather

than deterministically copying the teacher.

The ﬁeld of imitation learning draws its importance from its relevance to a variety

of applications such as human computer interaction and assistive robots. It is being

used to teach robots of varying skeletons and degrees of freedom (DOF) to perform

an array of different tasks. Some examples are navigational problems, which typically

employ vehicle-like robots, with relatively lower degrees of freedom. These include

ﬂying vehicles [Sammut et al. 2014] [Abbeel et al. 2007] [Ng et al. 2006], or ground

vehicles [Silver et al. 2008] [Ratliff et al. 2007a] [Chernova and Veloso 2007a] [Ollis

et al. 2007]. Other applications focus on robots with higher degrees of freedom such as

humanoid robots [Mataric 2000a] [Asfour et al. 2008][Calinon and Billard 2007a] or

robotic arms [Kober and Peters 2010][Kober and Peters 2009b][M

ulling et al. 2013].

High DOF humanoid robots can learn discrete actions such as standing up, and cyclic

tasks such as walking [Berger et al. 2008]. Although the majority of applications target

robotics, imitation learning applies to simulations [Berger et al. 2008] [Argall et al.

2007] and is even used in computer games [Thurau et al. 2004a] [Gorman 2009] [Ross

and Bagnell 2010].

Imitation learning works by extracting information about the behavior of the teacher

and the surrounding environment including any manipulated objects, and learning

a mapping between the situation and demonstrated behavior. Traditional machine

learning algorithms do not scale to high dimensional agents with high degrees of free-

dom [Kober and Peters 2010]. Specialized algorithms are therefore needed to create

adequate representations and predictions to be able to emulate motor functions in hu-

mans.

Similar to traditional supervised learning where examples represent pairs of fea-

tures and labels, in imitation learning the examples demonstrate pairs of states and

actions. Where the state represents the current pose of the agent, including the posi-

tion and velocities of relevant joints and the status of a target object if one exists (such

as position, velocity, geometric information, etc.). Therefore, Markov decision processes

(MDPs) lend themselves naturally to imitation learning problems and are commonly

used to represent expert demonstrations. The Markov property dictates that the next

state is only dependent on the previous state and action, which alleviates the need

to include earlier states in the state representation [Kober et al. 2013]. A typical im-

itation learning work ﬂow starts by acquiring sample demonstrations from an expert

which are then encoded as state-action pairs. These examples are then used to train

ACM Computing Surveys, Vol. V, No. N, Article A, Publication date: January YYYY.

A:4 A. Hussein et al.

a policy. However, learning a direct mapping between state and action is often not

enough to achieve the required behavior. This can happen due to a number of issues

such as errors in acquiring the demonstrations, variance in the skeletons of the teacher

and learner (correspondence problem) or insufﬁcient demonstrations. Moreover, the

task performed by the learner may slightly vary from the demonstrated task due to

changes in the environment, obstacles or targets. Therefore, imitation learning fre-

quently involves another step that requires the learner to perform the learned action

and re-optimize the learned policy according to its performance of the task. This self-

improvement can be achieved with respect to a quantiﬁable reward or learned from

examples. Many of these approaches fall under the wide umbrella of reinforcement

learning.

Figure 1 shows a workﬂow of an imitation learning process. The process starts by

capturing actions to learn from, this can be achieved via different sensing methods.

The data from the sensors is then processed to extract features that describe the state

and surroundings of the performer. The features are used to learn a policy to mimic

the demonstrated behavior. Finally the policy can be enhanced by allowing the agent

to act out the policy and reﬁne it based on its performance. This step may or may not

require the input of a teacher. It might be intuitive to think of policy-reﬁnement as

a post learning step, but in many cases it occurs in conjunction with learning from

demonstrations.

Fig. 1. Imitation learning ﬂowchart

Imitation learning applications face a number of diverse challenges due to their

interdisciplinary nature:

— Starting with the process of acquiring demonstrations, whether capturing data from

sensors on the learner or the teacher, or using visual information, the captured sig-

nals are prone to noise and sensor errors. Similar problems arise during execution,

ACM Computing Surveys, Vol. V, No. N, Article A, Publication date: January YYYY.

Imitation Learning: A Survey of Learning Methods A:5

when the agent is sensing the environment. Noisy or unreliable sensing will result

in erroneous behavior even if the model is adequately trained. [Argall et al. 2009]

survey the different methods of gathering demonstrations and the challenges in each

approach.

— Another issue concerning demonstration is the correspondence problem [Dauten-

hahn and Nehaniv 2002]. Correspondence is the matching of the learner’s capabil-

ities, skeleton and degrees of freedom to that of the teacher. Any discrepancies in the

size or structure between the teacher and learner need to be compensated for dur-

ing training. Often in this case a learner can learn the shape of the movement from

demonstrations, then reﬁne the model through trial and error to achieve its goal.

— A related challenge is the problem of observability where the kinematics of the

teacher are not known to learner [Schaal et al. 2003]. If the demonstrations are not

provided by a designated teacher, the learner may not be aware of the capabilities

and possible actions of the teacher; it can only observe the effects of the teacher’s

actions and attempt to replicate them using its own kinematics.

— The learning process also faces practical problems as traditional machine learning

techniques do not scale well to high degrees of freedom [Kober and Peters 2010].

Due to the real-time nature of many imitation learning applications, the learning

algorithms are restricted by computing power and memory limitations; especially in

robotic applications that require on-board computers to perform the real-time pro-

cessing.

— Moreover, complex behaviors can often be viewed as a trajectory of dependent micro

actions which violates the independent and identically distributed (i.i.d.) assumption

adopted in most machine learning practices. The learned policy must be able to adapt

its behavior based on previous actions and make corrections if necessary.

— The policy must also be able to adapt to variations in the task and the surround-

ing environment. The complex nature of imitation learning applications dictate that

agents must be able to reliably perform the task even under circumstances that vary

from the training demonstration.

— Tasks that entail human-robot interaction pose a new set of challenges. Naturally

safety is a chief concern in such applications [De Santis et al. 2008] [Ikemoto et al.

2012], and measures need to be taken to prevent injury of the human partners and

insure their safety. Moreover, other challenges concern the mechanics of the robot,

such as its ability to react to the humans’ force and adapt to their actions.

In the next section we formally present the problem of imitation learning and discuss

different ways to formulate the problem. Section 3 describes the different methods for

creating feature representations. Section 4 reviews direct imitation methods for learn-

ing from demonstrations. Section 5 surveys indirect learning techniques and presents

the different approaches to improve learned policies through optimizing reward func-

tions and teachers’ behaviors. The paradigms for improving direct imitation through

indirect learning are also discussed. Section 6 reviews the use of imitation learning in

multi-agent scenarios. Section 7 describes different evaluation approaches and Section

8 shows the potential applications to utilize imitation learning. Finally, we present a

conclusion and discuss future directions in Section 9.

2. PROBLEM FORMULATION

In this section we formalize the problem of imitation learning and introduce some

preliminaries and deﬁnitions.

DEFINITION 1. The process of imitation learning is one by which an agent uses in-

stances of performed actions to learn a policy that solves a given task.

ACM Computing Surveys, Vol. V, No. N, Article A, Publication date: January YYYY.

剩余34页未读，继续阅读

评论收藏

内容反馈

xcmax

粉丝: 29

Imitation Learning A Survey of Learning Methods.pdf

最新资源

Imitation Learning A Survey of Learning Methods.pdf

模仿学习（Imitation Learning）

模仿学习Imitation Learning最新论文2018

ICML 2018强化学习tutorial: Imitation Learning

模仿学习（imitation_learning）【62页ppt】.zip

Global overview of Imitation Learning.pdf

A Survey of Hand Crafted and Deep Learning Methods for Image

A Survey of Deep Learning Methods for Relation Extraction

A Survey of Hand Crafted and Deep Learning Methods for Image Aes

imitation learning.pdf

imitation-learning:模仿学习算法

Third-Person Imitation Learning, OpenAI, 2017.pdf模仿学习

Generative Adversarial Imitation Learning 生成对抗的模仿学习

模仿学习论文 One-Shot Imitation Learning .Yan Duan

EnsembleDAgger A Bayesian Approach to Safe Imitation Learning.pdf

Ensemble Machine Learning.Methods and Applications.2013.pdf

Ensemble Methods for Machine Learning 9781617297137.pdf

A Survey on 3D Hand Pose Estimation Cameras, Methods, and Datasets.pdf

Explainable Deep Learning Methods in Medical Diagnosis：A Survey.

模仿学习论文 无模式的模仿学习 Model-Free Imitation Learning with Policy Optimization

imitation-learning-master.zip_Python 深度学习_imitation learning_模仿学

Mind A Quarterly Review Of Psychology And Philosophy.pdf

深度强化学习 - Imitation Learning

2020年机器学习深度学习下载地址.txt

The Deep Learning Compiler A Comprehensive Survey.pdf

Multimodality in Meta-Learning A Comprehensive Survey.pdf

A Comprehensive Survey on Transfer Learning.pdf

A brief Survey of Deep Reinforcement Learning.pdf

mysql 问题排查

我需要研究给出 一阶RC滤波的波特图 其电阻电容值对波特图幅频响应的关系

最新资源

模仿学习论文　One-Shot Imitation Learning .Yan Duan

模仿学习论文无模式的模仿学习 Model-Free Imitation Learning with Policy Optimization

我需要研究给出一阶RC滤波的波特图其电阻电容值对波特图幅频响应的关系