arXiv:1708.02392v1  [cs.RO]  8 Aug 2017
Learning Human-Robot Collaboration Insights through the Integration
of Muscle Activity in Interaction Motion Models
Longxin Chen, Juan Rojas, Shuangda Duan, and Yisheng Guan.
Abstract— Recent progress in human-robot collaboration
(HRC) makes fast and fluid interactions possible, even when
human observations are partial and occluded. Methods like
Interaction Probabilistic Movement Primitives (ProMPs) model
human Cartesian trajectories through motion capture systems.
However, such representation does not properly model tasks
where similar motions are used to handle different objects. As
such, under current approaches, a robot would not be able
to properly adapt its pose and dynamics for proper handling.
We propose to integrate the use of Electromyography (EMG)
into the Interaction ProMP framework and utilize EMG-
based muscular signals to augment the human observation
representation. The contribution of our paper is the increased
capacity to discern tasks that have similar trajectories but
ones in which different tools are utilized and require the
robot to adjust its pose for proper handling. Multidimensional
Interaction ProMPs are used with an augmented vector that
integrates muscle activity. Augmented time-normalized trajec-
tories are used in training to learn correlation parameters
and robot motions are predicted by finding a best weight
combination and temporal scaling for a task. Collaborative
single task scenarios with similar motions but different objects
were used and compared. For one experiment only joint angles
were recorded, for the other EMG signals were additionally
integrated. Task recognition was computed for both tasks.
Observation state vectors with augmented EMG signals were
able to completely identify differences across tasks, while the
baseline method failed every time. Integrating EMG signals
into collaborative tasks significantly increases the ability of the
system to recognize nuances in the tasks that are otherwise
imperceptible, up to 74.6% in our studies. Furthermore, the
integration of EMG signals for collaboration also opens the
door to a wide class of human-robot physical interactions based
on haptic communication that have been largely unexploited in
the field. Supplemental information including video, code, and
results analysis can be found at [1].
I. INTRODUCTION
Interest in HRC has significantly increased in recent years.
The promise of synergistically combining the best of what
robots and humans have to offer has led to numerous
studies. However, many challenges remain in facilitating
programming robot collaborative partners. The variety of
tasks in which a human needs assistance is practically
unlimited. Robots must easily learn and adapt to unstructured
scenarios. Recent progress in HRC now makes fast and
fluid interactions possible, even when human observations
are partial and occluded.
Numerous approaches used to generate robot motion in
response to human motion observations have relied on the
All authors are with the School of Electromechanical Engineering in
Guangdong University of Technology in Guangzhou, China.
Fig. 1.
A robot collaborator is empowered when it is able to discern
different tasks that consist of similar human trajectories. In this figure, three
tasks are shown where a human uses a similar trajectory to hand over three
distinct objects to the robot. In each case, we augmented the observation
vector with EMG-based muscular activity signals that enabled to robot to
discern across tasks and choose the correct robot response.
use of joint angle or Cartesian trajectory information. Works
like Dynamic Movement Primitives (DMP) [2]–[4], Interac-
tive Meshes [5]–[7], and Interaction ProMPs [8]–[10] use
motion capture systems to record human motion trajectories.
However these systems are unable to properly model tasks
where similar motions are used to perform different tasks,
such as that of passing, holding, or coordinating motion of a
human using different tools with different shape and inertial
properties. As such, under current approaches, a robot is
unable to properly adapt its pose and dynamics when two
tasks with similar motion but different objects are used.
In this paper we explore techniques that enable increased
task recognition discernment given human observations. Par-
ticularly, we explore the impact of integrating EMG-based
muscular activity signals when used alongside motion trajec-
tories in the Interaction ProMP framework. The contribution
of this paper is the discernment of tasks that have similar
motion trajectories, but ones in which objects of different
shapes and dynamics are used. Better action recognition
also leads to more natural interactions as a robot can adjust
its pose and dynamics to minimize (mental, emotional, and
physical) load placed on the human to compensate for poor
adjustment on the robot’s part. Fig. 1 illustrates a hand-over
interaction in a collaborative task.
Multidimensional Interaction ProMPs are used with an
augmented state vector that integrates EMG-based muscle
activity. This works builds on the Phase Estimation approach
of [8]. Provided a set of human-robot collaborative task
demonstrations, time aligned trajectory way-points and EMG
signals are parameterized into a lower dimensional weight
space as a linear combination of basis functions. A Gaussian
distribution is built from the set of weight vectors obtained
in training and a normal distribution is also built from
time-scaling values used to normalize training data yielding
a probabilistic movement primitive. As for the Interaction
segment, the correlation of all human-robot data dimensions
is computed and the robot motion is inferred by computing
a posterior probability distribution over the weights condi-
tioned on the partial augmented human observation. The
weight distribution requires a new mean and covariance
from the partial observations, both of which are computed
by using a Kalman filter. For task recognition, the task
with the highest posterior probability for new observations
given the task probability is selected. In Maeda et al. ’s
work, temporal variance is included in the model. A phase
ratio needs to be computed from the sparse sequence of
observations, to determine an associated observation matrix
to finally condition and do prediction.
To test the effects of EMG signals in Interaction ProMPs,
three distinct hand-over tasks were performed, all of which
consisted of similar motions but used different objects.
Experiments were done with and without EMG-signals. Task
recognition was reported for both scenarios for different
number of demonstrations and observation ratios. Integrating
EMG signals into collaborative tasks significantly increases
the ability of the system to recognize nuances in the tasks
that were otherwise imperceptible, up to 74.6%. EMG signals
recognized tasks better in 11/12 of our comparative studies
and did it overwhelmingly better. We also purport that
user-loads (mental, physical, and emotional) would diminish
significantly as humans would not need to adjust their
handling to make up for the robot’s deficiency. Finally, the
integration of EMG signals in HRC opens the door to a
wide class of human-robot physical interactions based on
haptic communication that have been largely unexplored in
the field. Supplemental information including video, code,
and results analysis can be found at [1].
II. RELATED WORK
HRC poses a dual problem: one of action recognition
and movement generalization. This section we discuss the
conventional interaction motion models and previous works
related to the use of EMG signals in HRC.
In [2]–[4], DMPs are introduced as a time-dependent
movement representation. DMPs comprise a proportional-
derivative (PD) controller and a non-linear forcing function.
Based on the DMP framework, Interaction Primitives (IPs)
capture the variance of DMP parameters and generate a
probability distribution. The probabilistic model learns the
inter-agent correlations and allows to generalize skills in
HRC.
In [7], Interaction meshes (IM) were used to learn
human-robot interactions from human-human demonstra-
tions. IMs capture spatio-temporal relationships between the
body movements of two interacting partners. For any given
time-step, an IM represents a pair of postures in the human-
human demonstration. IMs allow to transfer a collaborative
skill from one pair of partners to another (i.e. a human-
robot pair) given the set of IMs. IMs are coupled with
Hidden Markov Models (HMMs) to have both the ability
to generate robot motions (through the IMs) and perform
task recognition (through the HMMs). HMMs have been a
popular modeling approach in which the process is assumed
to be Markov and consist of unobserved hidden states that
are inferred [11], [12]. Furthermore, IMs can be deformed
to adapt to varying trajectory observations in the interacting
partners [5], [6], [13].
In [14], [15], ProMPs were introduced as an alternative to
DMPs. ProMPs are a time-dependent movement representa-
tion that do not need a forcing function, instead trajectories
are approximated by a weighted sum of time-dependent basis
functions. More recently, Maeda et al. proposed Interaction
ProMPs based on ProMPs for HRC [8]–[10]. Interaction
ProMPs capture temporal and spatial variances of motion
trajectories as well as correlations across all human and robot
dimensions. The model can recognize executed tasks and
generate corresponding robot motion given human motion
observations. That is, both motion generalization and action
recognition are jointly implemented in the framework.
All previous works are limited in that they only model-
ing motion trajectories. In situations where different tasks
are executed with similar trajectories, these techniques are
unable to discern across tasks. This is important given that
in collaboration, it is not uncommon to to perform similar
motions with different tools. Consider any kind of hand-over
task, the same motion is used for a variety of tools that
have unique shape and dynamical properties. Thus, it is of
significant interest to explore techniques that enable greater
insight into tasks with similar spatio-temporal relationships
in motion trajectories. In our work, we propose to integrate
the use of EMG-based muscular activity in the previously
presented Interaction ProMP model. By integrating EMG
signals, the system is able to gain insights unavailable in
spatio-temporal trajectory patterns in motion trajectories.
Muscular activity contains signatures that differentiate both
pose and dynamical patterns hence providing key information
to our model.
We note that there seem to be no other works in which
EMG-signals are used to model and classify human motions
within HRC. Some studies like that of Reed et al. [16],
measured human force profiles in human-human interactions,
where humans developed a specialization of roles. Later
when a human-interacted with robots, no specialization took
place according to the force profiles. This is an example
where human force feedback was used, but not to affect the
response of the collaborative robot. Peterne et al. [17], used
EMG signals to estimate human partner fatigue in human-
human collaborative tasks. Kulic et al. [18], use human phys-
iological signals like heart rate, perspiration rate, and facial
muscle contract to measure the body-language interaction
between a human and a robot. A robot manipulator was
conditioned to move to different distances from the human,
and the physiological response was measured. This study
is similar to our current work in that human signals are
modeled, but differ in the this study did not use them to
tell the robot how to move. Instead the goal was simply to
model the affective state of the human given a robot motion.
III. METHODOLOGY
In HRC tasks, Interaction ProMPs generate a robot col-
laborative motion based on the prediction from a set of
partial human movement observations. The approach also
works in multi-task scenarios. Our work explains the steps
need to integrate and process EMG-based muscle activities
in addition to motion trajectory data.
A. Probabilistic Movement Primitives for a Single Dimen-
sion
ProMPs summarize patterns across demonstrations in a
probabilistic manner. They are able to capture correlations
across all data dimensions and describe variations in which
movements can be executed leading to a probability distri-
bution over trajectories. Representing variance information
correctly is critical as it reflects the importance of single
time steps for a movement execution. For each time step, a
single dimensional position is represented by yt ∈R1 and a
trajectory of T time steps as y1:T . We adopt linear regression
with n Gaussian basis functions ψ to represent one motion
trajectory. The probability of observing a trajectory y1:T
given an underlying weight vector ω is given as a linear
basis function model:
yt = ψT
t ω + ǫy,
p(y1:T |ω) =
T
Y
1
N(yt|ψT
t ω, σy),
(1)
where, ǫy ∼N(0, σy) models zero-mean i.i.d. Gaussian
noise. The set ψ = [(ψt)1, (ψt)2, ..., (ψt)N]T
∈RN×1
contains values of each of the basis function at time t. Given
a basis function, one can compute ω for each trajectory y1:T
using linear regression as:
ω = (ΨT
1:T Ψ1:T )−1Ψ1:T y1:T ,
(2)
where,
Ψ1:T =


(ψ1)1
· · ·
(ψ1)N
...
...
...
(ψT )1
· · ·
(ψT )N


(3)
The ω vector can compactly represent a single trajectory.
Having a set of motion trajectories, we can compute a proba-
bility distribution over the weights ω. To capture the variance
across trajectories in different demonstrations, we define θ
as a parameter that governs the distribution of weight vectors
in the set ω and we assume that ω ∼N(µω, Σω), that is
θ = (µω, Σω).
The trajectory distribution p(y1:T ; θ) can now be com-
puted by marginalizing out the weight vector ω. The dis-
tribution p(y1:T ; θ) defines a Hierarchical Bayesian Model
(HBM) whose parameters are given by the observation noise
variance σy and the parameters θ of p(ω; θ). For now, we
can compute the probability distribution of a position at a
given time from the distribution of ω as
p(yt|θ) =
Z
p(yt|ω)p(ω|θ)dω
= N(yt|ψT
t µω, ψT
t Σωψt + σy).
(4)
The above framework captures spatial correlations from a set
of demonstrations. To cope with demonstrations of different
durations, the training set must be time aligned (done through
resampling in this work).
B. Correlating Muscular Activity into Interaction Motion
Model
In this section, we extend ProMPs to a multidimensional
setting and compute the correlation for the full set of data-
dimensions for human and robot across demonstrations.
Previous works assume that human-motion collaborative-
task trajectories differ spatio-temporally from one another.
Under this assumption, the use of Cartesian information
from human motion capture systems has been sufficient to
distinguish different tasks. However, if the assumption is
violated and different tasks share similar trajectories, the
task recognition system is bound to fail. We consider the
introduction of EMG-based muscular activities as part of the
observed state in Interaction ProMPs. EMG signals are easily
integrated as a temporal sequence. With them, we attempt to
infer future robot responses from human observations (now
Cartesian pose and EMG information), with more nuanced
insights into the collaborative task. Namely, the ability to
discern different tasks with similar pose observations but
with distinct muscular activities.
Now we introduce the mathematical model for Interaction
ProMPs with the augmented EMG-signals. For human ob-
servations, consider p pose dimensions and e EMG signal
channels, while for robot observations, consider j joint an-
gles. Each collaborative demonstration consists of (p+e+j)
dimensions in the training trajectories. For HRC, the state
vector yt at time t is the concatenation of the (p+e) human
observations and the j joints of the robots, such that
yt = [yH
1,t, ...yH
p,t, yH
1,t, ...yH
e,t, yR
1,t, ...yR
j,t]
T ,
(5)
where, the upper script (.)H refers to the human pose
and EMG signal, and (.)R refers to the robot joint angle
configuration. The weight vector ω for each demonstration
is the concatenation of all weight vectors involved in the
demonstration. Thus, all the interaction dimensions involved
in the task are correlated as:
ωi = [(ωH
1 )
T , ..., (ωH
p )
T , (ωH
1 )
T , ..., (ωH
e )T , (ωR
1 )
T ..., (ωR
j )
T ]
T
.
(6)
And, as in the single dimensional case, the weight vector is
given as a linear regression model:
p(yt|ω) = N(yt|HT
t ω, Σy),
(7)
where, the Ht = diag((ψT
t )1, ..., (ψT
t )(p+e), (ψT
t )1, ..., (ψT
t )j)
is the time-dependent basis matrix for the positions.
Given the (partial) observations, we can compute the pos-
terior distribution of both human and robot trajectories using
a Kalman Filter. Where observations only contain human
motion, thus robot observations are set to zero yielding:
yo
t = [yH
1,t, ...yH
p,t, yH
1,t, ...yH
e,t, yR
1,t, ...yR
j,t]
T .
(8)
To contrast with a complete observation sequence [t : t′], the
notation [t −t′] ∈Rs×(p+e) is used to indicate a sequence
s of partial observations in the interval (some measurements
in the interval are missing). Observations can be considered
as modulations to via-points. The operation is done by
conditioning the ProMPs to reach a certain state yo
t−t′ at
time (t−t′). The conditioning adds a desired observation to
xt−t′ = [yo
t−t′, Σo
y] to the probabilistic model and applying
Bayes theorem. Kalman filtering is used to compute the
posterior distribution as:
µnew
ω
= µω + K(yo
t−t′ −Ht−t′µω),
Σnew
ω
= Σω −K(Ht−t′Σω).
(9)
Here, K = ΣωHT
t−t′(Σo
y + Ht−t′ΣωHT
t−t′)
−1. And, since
missing observations exist, for each time step of the obser-
vation matrix Ht−t′, the latter is set as:
Ht−t′ =


(ψT
t )1
· · ·
0
0
· · ·
0
0
...
0
0
...
0
0
· · ·
(ψT
t )(p+e)
0
· · ·
0
0
· · ·
0
01
· · ·
0
0
...
...
0
...
0
0
· · ·
0
0
· · ·
0j


(10)
with Ht−t′ ∈R(p+e+j)×(p+e+j)N .
C. Phase estimation
It’s natural for a human to execute repetitions of a specific
task with different speeds. The latter leads to uncertainty in
the duration of the demonstration. To capture such spatial
variation correctly, time alignment must been done. What’s
more, phase (or progress) analysis of human observations
during testing must be estimated to aligning them to the
trained spatial models. In our work, each demonstration was
resampled yielding a nominal duration Tnorm. As in [8],
we assume that the ith demonstration also has a constant
temporal change in relation to the nominal duration and can
define a scaling factor in Eqtn. 11 to index all demonstrations
by the nominal time index.
αi = Ti/Tnorm.
(11)
For phase estimation in testing, Maeda’s single phase tem-
poral model is used. And a distribution over phase rations
from different demonstrations are modeled according to a
normal distribution and set as the phase prior. We assume
α ∼N(µα, σα). In testing, given a human observation yo
t−t′,
the posterior for the phase is computed as:
p(α|yo
t−t′, θ) ∝p(yt−t′|α, θ)p(α),
(12)
where the p(α) is the prior probability of the scaling factor
α as previously discussed. Additionally, the likelihood for a
specific task is given as:
p(yt−t′|α, θ) =
Z
p(yo
t−t′|ω, α)p(ω)dω.
(13)
For one specific task, given the human observations yo
t−t′ the
most probable scaling factor is:
α∗= arg max
α
p(α|yo
t−t′, θ)
(14)
The best fit scaling factor α∗
k for each task is selected to get
the set {α∗
k, θk}. Then, task recognition is done based on
this set.
D. Task Recognition
We model a set of k demonstrations from a probabilistic
perspective and compute the posterior distribution of a task
given human signal observations according to Eqtn. 15
p(k|yo
t−t′) ∝p(yo
t−t′|θk, α∗)p(k),
(15)
where, p(k) is the task’s prior probability and can be
determined by the specific circumstances of an experiment.
The likelihood of each component given the model θ is:
p(yo
t−t′; θk, α∗) =
Z
p(yo
t−t′|Ho
t−t′ω, Σy)p(ω; θk)dω.
(16)
A task is selected by choosing the posterior with the highest
probability:
k∗= arg max
k
p(k|yo
t−t′)
(17)
Fig. 2, summarizes the multiple task recognition problem.
Motion and EMG signals from a human and robot states
(joint angles or Cartesian) are captured. After demonstrating
a collaborative task, we generate the probabilistic model for
each task to represent multiple demonstrations using our
method. For clarity sakes, sensor data is abstracted to a
single dimension in the Figure. Note how human motion
look similar across tasks. This condition leads to a situation
where the likelihood for multiple tasks is very similar to each
other, rendering it difficult to select a task with any certainty.
IV. EXPERIMENTS AND RESULTS
Our experimental testbed used a dual-armed upper-torso
anthropomorphic Baxter robot, a Myo wearable armband and
ROS Indigo in Linux Ubuntu 14.04. Kinesthetic teaching
was used to drive Baxter in collaborative tasks. The Myo
armband is composed of eight stainless steel EMG sensors
and a nine axis IMU. Raw EMG and IMU data, along
with motion, orientation, and rotation data are streamed over
blue tooth. The band is placed around the forearm, as such
it measures muscle signals in the forearms’ anterior and
posterior superficial muscles. Such data can play a vital
complimentary role to motion data.
To test the effects of EMG signals in Interaction ProMPs,
three distinct hand-over tasks, but ones with similar human
motions, were tested with and without EMG-signals. Namely,
EMG signals
IMU signals
task 1
observations
EMG signals
task 2
task n
task 3
IMU signals
robot joints
observations
EMG signals
IMU signals
robot states
observations
EMG signals
IMU signals
robot joints
observations
EMG signals
IMU signals
robot joints
robot joints
k* = arg max p (k | y )
k
o
...
Fig. 2.
Multidimensional Interaction ProMPs are used with an augmented state vector that integrates EMG-based muscle activity. Phase aligned trajectory
way-points and EMG signals are parameterized into a lower dimensional weight space as a linear combination of basis functions. The correlation of all
human-robot data is computed and the robot motion is inferred by computing a posterior probability distribution over the weights conditioned on the partial
augmented human observation (shown in green circles). For task recognition, the task with the highest posterior probability of new observations given a
task’s probability is selected.
(i) passing an aluminum rod, (ii) passing a wrench, and
(iii) passing a measuring tape. For the aluminum rod and
measuring tape tasks, the human beings by grasping the
corresponding object and then proceeds to pass them to the
robot, the robot executes a parallel motion and picks the
object. For the wrench task, it is the human who receives
the tool from the Baxter robot. Each of the three tasks
was repeated 10, 15, and 20 times respectively for training
and an additional 10 trials for testing. The different number
of training trials was set to study the impact of training
trials with EMG signals. All trial data was time aligned by
resampling. Fig 1, shows a snap shot for each of the three
experiments at the time the tool is handed over. For the three
tasks the human motion is nearly the same: each experiment
has the human standing in approximately the same location
and the arm pose is also started in approximately the same
location. Such assumptions are really realistic given that a
work site has an established working environment. Similar
motions are attempted by a single user each time. This sets
the stage to measure the task recognition ability when using
the EMG signals. We report results for experiments with
orientation and orientation with EMG data (we did not in fact
use a Cartesian trajectory due to the noisiness of IMU motion
data). Both under different amounts of human observations:
10% and 20% of the duration of the task.
A. Results
We present results in a set of tables. Each table presents the
results according to the number of training demonstrations
as well as the human observation ratio for the task, and the
recognition accuracy result for the three tasks with and with-
out EMG data. Table I shows results for 20 demonstrations
and 10% observation ratio. Table II: 15 and 10% respectively,
Table III 10 and 10% respectively, and Table IV 10 and 20%
respectively.
TABLE I
NUMBER OF DEMONSTRATION: 20, OBSERVATION RATIO: 0.1
Task
w/out EMG
with EMG
Aluminum Rod
0.90
1.00
Wrench
0.60
1.00
Measuring Tape
0.10
0.70
We note that out of the 12 measurements that we made
(different number of demonstrations & observation ratios by
TABLE II
NUMBER OF DEMONSTRATION: 15, OBSERVATION RATIO: 0.1
Task
without EMG
with EMG
Aluminum Rod
0.60
1.00
Wrench
0.60
0.90
Measuring Tape
0.10
0.70
TABLE III
NUMBER OF DEMONSTRATION: 10, OBSERVATION RATIO: 0.1
Task
without EMG
with EMG
Aluminum Rod
0.00
0.50
Wrench
0.00
0.80
Measuring Tape
0.80
0.70
TABLE IV
NUMBER OF DEMONSTRATION: 10, OBSERVATION RATIO: 0.2
Task
without EMG
with EMG
Aluminum Rod
0.30
1.00
Wrench
1.00
1.00
Measuring Tape
0.90
1.00
the three tasks), 11 out of the 12 tasks or 91.6%, experiments
with EMG signals out-classified those without. Not only so,
if we average classification rates across all experiments, we
see that without EMG signals we had an accuracy of get a
sum of 49.2%, while for the augmented EMG signals we
get an accuracy of 85.8%. That is 74.6% more accurate
recognition (see our supplemental information for details
[1]). In summary, integration of EMG signals not only is
correct more than without, but is also does it overwhelmingly
better. We believe this too would have significant effects
in user-load (mental, emotional, and physical) as the robot
would handle tasks in ways that do not require the human
to adjust its handling, thus enhancing the overall experience.
However, this is was not formally measured in this study.
We noted that during task recognition inference, there
is a strong dependence on the prior. That is, observations
make a small contribution. For motion trajectory only demos,
failed task recognition predictions result in wrong robot col-
laborative motions. But with the integration of EMG-based
muscular signals to human motion observations, the distinct
EMG signatures disambiguate task recognition yielding large
probabilistic differences across tasks.
V. DISCUSSION
Our work demonstrates that the integration of EMG-
based muscular activity into Interaction ProMPs for tasks
with similar motions significantly increased task recognition
discernment. It was shown that for three different hand-over
tasks (including human-to-robot and robot-to-human passes)
with different number of training demonstrations and differ-
ent number of human observation ratios, experiments with
EMG-signals overwhelmingly outperformed those without,
that is by 74.6%.
This result shows that human muscular activity can sig-
nificantly augment a robot’s insight into human service tasks
and improve its task recognition. This in turn allows a
robot to improve how it handles an object: it’s end-pose
at the time of the hand-over and possibly its dynamics. In
doing so, hand-overs and numerous other tasks would place
a lower user-load on the human: mentally, physically, and
emotional. If the robot does not need to adjust his own pose
upon a handover because the robot has correctly reached
an object and thereafter properly handled, the human would
be at greater ease. We leave it to future work to show
the quantitative effects of this work. While the proposed
methodology of our work did not differ from that in [8],
we believe that the knowledge and insight gained from our
analysis of a rarely used biometric signal in HRC offers a
relevant insight to the field. We estimated this may be the first
work that studies the impact of muscular activity in human
robot collaboration tasks.
There are a number of enhancements we set as future
work. First, is to explore more compelling cases for the use of
muscular-based EMG-signals in physical human interaction.
The authors believe that a wide array of possibilities can
exist through haptic communication with the robot. That is,
through direct physical touch. EMG can serve as a primary
signal, especially if finger motion cannot be tracked or visual
occlusion prevents identifying small nuanced haptic motions.
Other improvements to the current work include the use of
non-parametric methods to estimate an optimal number of
basis functions in modeling trajectories. This will result in
better modeling, particularly when tasks have more complex
dynamics. Bayesian estimation can also yield more confident
beliefs in computing relevant parameters as opposed to MAP
estimates.
VI. CONCLUSION
We proposed the integration of EMG-based muscular
activity into the Interaction ProMP framework to augment
the human observation representation. A probabilistic model
containing the variance of human and robot motion and (fore-
arm) muscle activity was used. Motion Primitive’s temporal
distribution were modeled through a Hierarchical Bayesian
Model with Gaussian distributions. A temporal sequence
distribution is obtained from demonstrations and the cor-
relation across all dimensions jointly modeled and used to
generate a corresponding robot motion from the observation
of human action signals. The result was an increased capacity
to discern tasks with similar trajectories but different tools
aiding the robot to improve object handling and reducing
user-load.
VII. ACKNOWLEDGEMENTS
This work is supported by “Major Project of the Guang-
dong Province Department for Science and Technology
(2014B090919002), (2016B0911006).”
REFERENCES
[1] L. Chen, “Supplement to learning human-robot collaboration insights
through the integration of muscular activity in interaction motion mod-
els,” 2017. [Online]. Available: http://www.juanrojas.net/ipromp_emg/
[2] H. B. Amor, G. Neumann, S. Kamthe, and O. Kroemer, “Interaction
primitives for human-robot cooperation tasks,” in IEEE International
Conference on Robotics and Automation, 2016, pp. 2831–2837.
[3] S. Schaal, Dynamic Movement Primitives -A Framework for Motor
Control in Humans and Humanoid Robotics.
Springer Tokyo, 2006.
[4] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal,
“Dynamical movement primitives: learning attractor models for motor
behaviors.” Neural Computation, vol. 25, no. 2, pp. 328–373, 2013.
[5] D. Vogt, B. Lorenz, S. Grehl, and B. Jung, “Behavior generation
for interactive virtual humans using contextâ ˘A ˇRdependent interaction
meshes and automated constraint extraction,” Computer Animation &
Virtual Worlds, vol. 26, no. 3-4, pp. 227–235, 2015.
[6] D. Vogt, S. Stepputtis, S. Grehl, B. Jung, and H. B. Amor, “A system
for learning continuous human-robot interactions from human-human
demonstrations,” in IEEE International Conference on Robotics and
Automation, 2017.
[7] D. Vogt, S. Stepputtis, R. Weinhold, B. Jung, and H. B. Amor,
“Learning human-robot interactions from human-human demonstra-
tions (with applications in lego rocket assembly),” in Ieee-Ras Inter-
national Conference on Humanoid Robots, 2017.
[8] G. Maeda, M. Ewerton, G. Neumann, R. Lioutikov, and J. Peters,
“Phase estimation for fast action recognition and trajectory generation
in human–robot collaboration,” The International Journal of Robotics
Research, p. 0278364917693927, 2017.
[9] M. Ewerton, G. Neumann, R. Lioutikov, H. B. Amor, J. Peters, and
G. Maeda, “Learning multiple collaborative tasks with a mixture of
interaction primitives,” in Robotics and Automation (ICRA), 2015
IEEE International Conference on.
IEEE, 2015, pp. 1535–1542.
[10] G. J. Maeda, G. Neumann, M. Ewerton, R. Lioutikov, O. Kroemer,
and J. Peters, “Probabilistic movement primitives for coordination of
multiple humanâ ˘A¸Srobot collaborative tasks,” Autonomous Robots, pp.
1–20, 2016.
[11] S. Calinon and A. Billard, “Gesture recognition and reproduction for a
humanoid robot using hidden markov models,” in Ami/pascal/im2/m4
Workshop on Multimodal Interaction and Related Machine Learning
Algorithms, 2004.
[12] S. Calinon, F. Guenter, and A. Billard, “On learning the statistical
representation of a task and generalizing it to various contexts,” in
Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE
International Conference on.
IEEE, 2006, pp. 2978–2983.
[13] E. S. L. Ho, T. Komura, and C. L. Tai, “Spatial relationship preserving
character motion adaptation,” in ACM SIGGRAPH, 2010, p. 33.
[14] A. Paraschos, C. Daniel, J. Peters, and G. Neumann, “Probabilistic
movement primitives,” Advances in Neural Information Processing
Systems, pp. 2616–2624, 2013.
[15] D. Koert, G. Maeda, R. Lioutikov, G. Neumann, and J. Peters,
“Demonstration based trajectory optimization for generalizable robot
motions,” in Ieee-Ras International Conference on Humanoid Robots,
2017, pp. 515–522.
[16] K. B. Reed and M. A. Peshkin, “Physical collaboration of human-
human and human-robot teams,” IEEE Transactions on Haptics, vol. 1,
no. 2, pp. 108–120, 2008.
[17] L. Peternel, N. Tsagarakis, D. Caldwell, and A. Ajoudani, “Adaptation
of robot physical behaviour to human fatigue in human-robot co-
manipulation,” in IEEE RAS International conference on humanoid
robotics, 2016.
[18] D. Kulic and E. A. Croft, “Affective state estimation for human–robot
interaction,” IEEE Transactions on Robotics, vol. 23, no. 5, pp. 991–
1000, 2007.