25 th   Iranian Conference on Electrical Engineering (ICEE) May 2-4, 2017, Tehran, Iran 2017 IEEE 1  Brain Computer Interface for Gesture Control of a Social Robot: an Offline Study  Reza Abiri  Department of Mechanical, Aerospace, and Biomedical Engineering University of Tennessee, Knoxville Knoxville, TN 37996, USA rabiri@vols.utk.edu  Griffin Heise  Department of Mechanical, Aerospace, and Biomedical Engineering University of Tennessee, Knoxville Knoxville, TN 37996, USA gheise@vols.utk.edu  Xiaopeng Zhao  Department of Mechanical, Aerospace, and Biomedical Engineering University of Tennessee, Knoxville Knoxville, TN 37996, USA xzhao9@utk.edu  Yang Jiang  Department of Behavioral Science College of Medicine University of Kentucky Lexington, KY 40356, USA yjiang@uky.edu  Fateme Abiri  Department of Computer Engineering Ferdowsi University of Mashad Mashad, Iran fateme.abiri@yahoo.com  Abstract — Brain computer interface (BCI) provides promising applications   in   neuroprosthesis   and   neurorehabilitation   by controlling computers and robotic devices based on the patient’s intentions.   Here, we have developed a novel BCI platform that controls a personalized social robot using noninvasively acquired brain   signals.   Scalp   electroencephalogram   (EEG)   signals   are collected   from   a   user   in   real-time   during   tasks   of   imaginary movements. The imagined body kinematics are decoded using a regression model to calculate the user-intended velocity. Then, the decoded kinematic information is mapped to control the gestures of a social robot. The platform here may be utilized as a human-robot- interaction   framework   by   combining   with   neurofeedback mechanisms to enhance the cognitive capability of persons with dementia.  Keywords —Brain   Computer   Interface,   EEG,   Social   Robot, Human-Robot Interaction.  I. INTRODUCTION The field of human-robot interaction has been significantly enriched with the integration of Brain computer interface (BCI), in which the subject can manipulate the environment in a desired way   compatible   with   his/her   intention   through   the   brain activities [1]. For example, integrating BCI into exoskeletons, rehabilitation   robots,   and   prosthetics   has   shown   increased efficiency of rehabilitation due to direct intention of patient in rehabilitation progress [2-8]. In   BCI   and   particularly   in   noninvasive   approaches, electroencephalography   (EEG)   based   paradigms   are   more convenient and portable than other neuroimaging techniques such as electrocorticography (ECoG), magnetoencephalography (MEG), and magnetic resonance imaging (MRI) [1]. Many different EEG paradigms have been developed using external stimulations,   sensorimotor   rhythms,   or   imaginary   motor movements. The main drawback for systems on sensorimotor rhythms is the lengthy training time (several weeks to several months)   required   for   the   subjects   to   achieve   satisfactory performance. In cases with external stimulations, a fatigue phenomenon has been reported by subjects and researchers, although it should be noted that this paradigm is not reflecting the user’s intention to control a device. Another issue concerning these paradigms is the discrete control of cursor directions due to switching among imagined movements of several large body parts [9] or switching among multiple paradigms [10]. The alternative   system   based   on   imaginary   movement,   as   first designed by Bradberry et al. [11], has the capability to minimize
2 the training time (~20 minutes for two dimensional cursor control). Many researchers have employed EEG paradigms to control robotic systems. Sensorimotor rhythms have been utilized by various authors to control remote robotic systems [12], virtual and real quadcopters [13-15], and robotic arms [16-18]. Using an external   stimulation   paradigm/hybrid   paradigm,   researchers demonstrated the control of a prosthetic arm [19], artificial arm [20], and an exoskeleton for rehabilitation of the hand [21]. Besides the brain-controlled robots such as mobile robots [22], controlling humanoid and social robots has become of interest in BCI. Social robots are autonomous robots that can interact and communicate with humans. For example, by employing the aforementioned EEG paradigms, some researchers controlled the movements of humanoid robots such as NAO through direct control approaches [23-28]. However, no previous work had been   reported   on   manipulating   a   humanoid/social   robot   in cognitive training for patients with cognitive deficits. In this work, we develop a novel neurofeedback-based noninvasive BCI system for possible applications in cognitive enhancement. In contrast to previous studies on computer-based neurofeedback systems, the platform here is based on interaction with a social robot. The interaction with a robot may better engage and motivate user participation in specified tasks and thus enhance the targeted rehabilitation program. An initial testing of the developed   platform   is   conducted   using   the   imagined   body kinematics   scheme   originally   proposed   in   [11]   to   control different gestures of a social robot. II. MATERIALS AND METHODS  A. Experimental protocol  Before   controlling   the   social   robot,   the   subjects   are instructed to use a BCI system in a cursor control task. The experiment served   two objectives; first, a regression model was  developed   to   extract   imagined   body   kinematics   from   the subject’s   brainwaves.   Second,   the   experiment   helped   to familiarize the subject with BCI concepts.   The institutional Review Board of the University of Tennessee approved the experimental procedure and 5 subjects (4 male, 1 female) participated   in   the   experiments   after   signing   the   informed consent. For the experiments, a PC with dual monitor was provided. One monitor for the experimenter and another for the subjects. During the experiments, EEG signals were acquired by using an Emotiv EPOC device with 14 channels and through BCI2000 software (with 128 sampling time, high pass filter at 0.16Hz, and low pass filter at 30Hz). The cursor control task included three phases. Phase 1 was the training phase. The subject was asked to sit comfortably in a fixed chair with hands resting in the lap. The subject’s face was kept at an arm’s length from the   monitor. The subject   was instructed to track   the movement (up-down/right-left) of a computer cursor, whose movement was controlled by a practitioner in a random manner. Meanwhile, the subjects were instructed to imagine the same matched velocity movement with their right index fingers. The training phase consisted of 5 trials, each of which lasted 60 seconds. Phase 2 was the calibration phase, during which a decoder model was constructed to model the velocity of the cursor as a function of the EEG waves of the subject. For more accurate   reconstruction   and   prediction   of   the   imagined kinematics at each point, 5 previous points (time lag) of EEG data were also included in the decoding procedure. Then, the developed decoder was fed into BCI2000 software to test the performance of the subject in phase 3 (test phase). In the test phase, the subject was asked to move the cursor using their imagination to a target that randomly appeared at the edges of the monitor.  B. Decoding  Many   decoding   methods   for   EEG   data   have   been investigated by researchers in the frequency and time domains. Most sensorimotor-rhythms-based studies are developed in the frequency domain [9, 13-15, 17, 18, 29-32]. Meanwhile, in the time domain, researchers employed regression models as a common decoding method for decoding EEG data for offline decoding [33-37] and real-time implementation [11]. Some nonlinear methods such as the Kalman filter [38] and the particle filter model [39] were also applied in decoding EEG signals for offline analysis. Many previous works confirmed that among kinematics   parameters   (position,   velocity),   velocity encoding/decoding shows the most promising and satisfactory validation in prediction [33, 34, 36]. Hence, we were motivated to decode and map the acquired EEG data to the observed velocities in x and y directions. In other words, the aim was to reconstruct the subject’s trajectories off-line from EEG data and obtain a calibrated decoder. For this purpose, all the collected data was transferred to MATLAB software for analyzing and developing a decoder. Here, based on a regression model for output velocities at time sample   ᡲ   in x direction ( ᡳ䙰ᡲ䙱 ) and y direction ( ᡴ䙰ᡲ䙱 ), the equations can be presented as follows:  ᡳ 䙰 ᡲ 䙱   =   ᡓ ⡨ け   +   㔳   㔳   ᡔ ぁ十け ᡗ ぁ 䙰 ᡲ   −   ᡣ 䙱  〒  十 ⢀ ⡨  〕  ぁ ⢀ ⡩  (1)  ᡴ 䙰 ᡲ 䙱   =   ᡓ ⡨げ   +   㔳   㔳   ᡔ ぁ十げ ᡗ ぁ [ ᡲ   −   ᡣ ]  〒  十 ⢀ ⡨  〕  ぁ ⢀ ⡩  (2)  where   ᡗ ぁ [ ᡲ   −   ᡣ ]   is the measured voltage for EEG electrode   ᡦ   at time lag   ᡣ   and for the total number of EEG sensors   ᡀ   = 14   and total lag number   ᠷ   = 5 . Based on a previously published study [11],   for   more   accurate   reconstruction   of   the   imagined kinematics, 5 previous points (time lag) of EEG data were included in the decoding and prediction of present value. The choice of 5 lag points is the tradeoff between accuracy and computational efficiency. The parameters   ᡓ   and   ᡔ   are calculated by feeding the data using least mean square error. The data collected in the training sessions was fed to equations 1 and 2 without any further filtering and the final developed decoder was employed to test and control the cursor on the monitor. The upper part of Fig. 1 shows a simple schematic of this procedure.
3  C. Robot interface design  Figure 1 shows a schematic of the proposed neurofeedback- based   human-robot-interaction   platform. The   decoded   brain activity   signals   collected   from   the   previous   cursor   control experiment   are   used   in   the   offline   mode   to   control   the movements of a social robot. Here, an affordable social robot called “Rapiro” [40] is chosen to be controlled by controlled cursor position data. An Arduino and a Raspberry Pi board placed in the robot enabled us to make communication with the robot and send the command signals from the PC through Simulink [41]. Rapiro robot is a humanoid robot kit with 12 servo motors with an Arduino compatible controller board. Its capabilities for performing and controlling multitask can be extended by employing a Raspberry Pi board assembled in the head of the robot. Rapiro was selected to provide neurofeedback by executing movements, playing sounds, and flicking lights corresponding to specific commands which are extracted from decoding EEG signals. The Simulink program was compatible with making communication with the social robot and it coped with   sending   commands   to   the   social   robot.   Here,   it   was programmed such that if the controlled cursor position (which was fed offline to the robot) was positive, the social robot showed right hand movement as neurofeedback for the subject ;  if the value was negative, the left hand movement will be the neurofeedback from the robot for the subject. III. RESULTS As mentioned in the decoding section, we used five points in EEG memory data to provide a more accurate estimation for parameters of imagined body kinematics. As an example, Fig. 2 shows a plot of results from a subject during the horizontal movement training phase. It illustrates a good match between the observed cursor velocity (real values) and decoded velocity from subject’s   collected   EEG   data   using   the   regression   model. Meanwhile, Table 1 shows the results for all 5 subjects during the control of the cursor in the test phase. Four subjects each conducted 6 trials of vertical movement and 6 trials of horizontal movement. One subject conducted 6 trials of vertical movement and did not conduct horizontal movements. The total success rate of hitting the appeared random targets shows higher accuracy in horizontal   movement   compared   to   vertical   movement.   The subjects also reported that it was easier for them to hit the targets in the horizontal direction. This result is inconsistent with the results   in   other   literature   [11].   Here,   the   one   dimensional movement was employed to test the developed platform in offline mode. The two dimensional movement and real-time control will be the next steps in research. After performing the test phase by the subjects in the cursor control application, the recorded data (cursor position) for this phase   are   collected   and   they   are   applied   to   control   the movements of the social robot. Figure 3 shows a series of recorded data of cursor positions that was sent in the offline mode to the Simulink to control the different parts of the social robot (e.g. right hand and left hand of social robot). Figure 3 illustrates the cursor position controlled by a subject during horizontal trials. Center of the screen, where the cursor started to move, is located at the origin (0, 0). Positive values indicate the controlled cursor is on the right side of the screen and negative values show the cursor is on the left side of the screen. After a pre-run time, the trials began and RT (Right Target) or LT (Left Target) appeared on screen. The subject had a limited time (15s) to hit the targets or the next trial would begin. In this run, the subject hit all the targets and as it is shown in Fig. 3, in all 6 trials the subject  Fig. 1. A schematic of neurofeedback-based BCI platform by engaging human- robot interaction in offline mode.  Table. 1. Results of cursor movement using imagined body kinematics.  Vertical Direction   Horizontal Direction  Number of Trials   30   24  Success Rate (standard deviation) 83.3% (+/- 11.7%)   100% (+/- 0%)  moved the cursor to the right side (positive values) for RT and left   side   (negative   values)   for   LT. The   subjects   showed   a satisfactory performance in control of the cursor during the trials except for some fluctuation at the beginning of each trial, in which the subject is managing to guide the cursor in the correct direction corresponding to the appeared target. This fluctuation is clear at the first trial in which the subject first went to the opposite direction (negative values) and then guided the cursor to the correct direction (positive values) to hit the RT. The recorded cursor position data was fed to Simulink to control the movements of the social robot in the offline mode. As a simple experiment, it was programmed such that the social robot showed right   hand   movement   for positive   values   of controlled cursor position and left hand movement for negative values   of   controlled   cursor   position.   Fig.   3   shows   the experimental results. In the beginning of each trial, there was a short period of time during which the robot showed incorrect hand movement, but the robot movement was quickly corrected and thereafter remained consistent with the user’s intentions.
4 These results confirmed the validity of a platform that can be used to provide real-time neurofeedback for the subject. Here, we controlled the social robot in an offline mode. In the next step of our work, we will make direct interaction between subject and social robot and as a result, provide direct neurofeedback from the robot for the subject.  Fig. 2. Observed cursor velocity during horizontal movement training and estimated/decoded values from EEG signals by employing regression model  Fig. 3. Recorded values of controlled-cursor position during one run (6 trials) of cursor control in horizontal direction by a subject. RT: Right Target appeared. LT: Left Target appeared.  IV. CONCLUSION Brain-robot interaction has become of interest in recent years and many studies demonstrated robotic control using invasive or noninvasive brain signals. Here, as a pilot study, we presented   a   novel   neurofeedback-based   BCI   platform   as   a testbed for cognitive training for the patient with cognitive deficits. The proposed platform is designed based on a human- robot interaction approach. For initial testing of platform, a new EEG paradigm based on continuous decoding of imagined body kinematics was used. The BCI paradigm was first applied in a computer cursor control experiment, which showed high rate of success in one-dimension of cursor control. Then, the controlled data from the cursor control task was fed into Simulink to control right hand and left hand movements of our social robot in the developed platform. The work here serves as a feasibility study to confirm the applicability of the platform for possible future development   and   testing   with   cognitive   algorithms   and   by patients. In the   future, the   system   will   be integrated   with neurofeedback   exercises   to   improve   cognitive   training   for patients of cognitive disorders [42-45]. Interestingly, we note that the cursor control tasks have higher accuracy in horizontal directions than in vertical directions. The discrepancy between the accuracy in horizontal and vertical controllability probably bears psychological and behavioral significance and is worthy of further investigation in future studies. One hypothesis is that horizontal   eye   movement   may   be   easier   than   vertical   eye movement   and   thus   affect   the   cursor   movement   task correspondingly. While Bradberry et al. showed the cursor movement tasks are not the results of eye movement, there may exist secondary effects due to eye movement. ACKNOWLEDGMENTS This work was in part supported by a NeuroNET seed grant from University of Tennessee to XZ and by a Department of Defense grant USUHS HU0001-11-1-0007 to YJ. REFERENCES  1.   Nicolas-Alonso, L.F. and J. Gomez-Gil,   Brain computer interfaces, a review.   Sensors, 2012.   12 (2): p. 1211-1279. 2.   Kiguchi, K. and Y. Hayashi.   Motion Estimation Based on EMG and EEG Signals to Control Wearable Robots . in  Systems,   Man,   and   Cybernetics   (SMC),   2013   IEEE International Conference on . 2013. IEEE. 3.   Luth, T., et al.   Low level control in a semi-autonomous rehabilitation robotic system via a brain-computer interface . in   Rehabilitation Robotics, 2007. ICORR 2007. IEEE 10th International Conference on . 2007. IEEE. 4.   Contreras-Vidal,   J.L.   and   R.G.   Grossman.   NeuroRex:   A clinical   neural   interface   roadmap   for   EEG-based   brain machine interfaces to a lower body robotic exoskeleton . in  Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE . 2013. IEEE. 5.   Frisoli, A., et al.,   A New Gaze-BCI-Driven Control of an Upper Limb Exoskeleton for Rehabilitation in Real-World Tasks.   Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 2012.   42 (6): p. 1169- 1179. 6.   Blank, A., et al.   A pre-clinical framework for neural control of   a   therapeutic   upper-limb   exoskeleton .   in   Neural Engineering   (NER),   2013   6th   International   IEEE/EMBS Conference on . 2013. 7.   Agashe, H.A., et al.,   Global cortical activity predicts shape of hand during grasping.   Frontiers in neuroscience, 2015.   9 . 8.   Agashe,   H.   and   J.L.   Contreras-Vidal.   Observation-based training for neuroprosthetic control of grasping by amputees . in   Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE . 2014. IEEE. 9.   Xia, B., et al.,   A combination strategy based brain–computer interface for two-dimensional movement control.   Journal of neural engineering, 2015.   12 (4): p. 046021. 10.   Allison, B.Z., et al.,   A hybrid ERD/SSVEP BCI for continuous simultaneous two dimensional cursor control.   Journal of neuroscience methods, 2012.   209 (2): p. 299-307. 11.   Bradberry, T.J., R.J. Gentili, and J.L. Contreras-Vidal,   Fast attainment of computer cursor control with noninvasively acquired brain signals.   Journal of neural engineering, 2011.  8 (3): p. 036010.
5  12.   Chakraborti, T., et al.   Implementation of EEG based control of remote robotic systems . in   Recent Trends in Information Systems (ReTIS), 2011 International Conference on . 2011. IEEE. 13.   Royer, A.S., et al.,   EEG control of a virtual helicopter in 3- dimensional space using intelligent control strategies.   Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 2010.   18 (6): p. 581-589. 14.   Doud, A.J., et al.,   Continuous three-dimensional control of a virtual   helicopter   using   a   motor   imagery   based   brain- computer interface.   PloS one, 2011.   6 (10): p. e26322. 15.   LaFleur, K., et al.,   Quadcopter control in three-dimensional space   using   a   noninvasive   motor   imagery-based   brain– computer interface.   Journal of neural engineering, 2013.  10 (4): p. 046003. 16.   Aiqin, S., F. Binghui, and J. Chaochuan.   Motor imagery EEG- based online control system for upper artificial limb . in  Transportation,   Mechanical,   and   Electrical   Engineering (TMEE), 2011 International Conference on . 2011. 17.   Baxter, B.S., A. Decker, and B. He.   Noninvasive control of a robotic   arm   in   multiple   dimensions   using   scalp electroencephalogram . in   Neural Engineering (NER), 2013 6th International IEEE/EMBS Conference on . 2013. IEEE. 18.   Li,   T.,   et   al.,   Brain–machine   interface   control   of   a manipulator using small-world neural network and shared control strategy.   Journal of neuroscience methods, 2014.   224 : p. 26-38. 19.   Muller-Putz,   G.R.   and   G.   Pfurtscheller,   Control   of   an electrical prosthesis with an SSVEP-based BCI.   IEEE Trans Biomed Eng, 2008.   55 (1): p. 361-4. 20.   Horki, P., et al.,   Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb.   Medical & biological engineering & computing, 2011.   49 (5): p. 567-577. 21.   Pfurtscheller, G., et al.,   Self-paced operation of an SSVEP- Based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid BCI.   Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 2010.   18 (4): p. 409-414. 22.   Bi, L., X.-A. Fan, and Y. Liu,   EEG-based brain-controlled mobile robots: a survey.   Human-Machine Systems, IEEE Transactions on, 2013.   43 (2): p. 161-176. 23.   Bouyarmane, K., et al.,   Brain-machine interfacing control of whole-body   humanoid   motion.   Frontiers   in   systems neuroscience, 2014.   8 . 24.   Li, W., C. Jaramillo, and Y. Li.   Development of mind control system   for   humanoid   robot   through   a   brain   computer interface .   in   Intelligent   System   Design   and   Engineering Application (ISDEA), 2012 Second International Conference on . 2012. IEEE. 25.   Bell, C.J., et al.,   Control of a humanoid robot by a noninvasive brain–computer   interface   in   humans.   Journal   of   neural engineering, 2008.   5 (2): p. 214. 26.   Bryan, M., et al.   An adaptive brain-computer interface for humanoid robot control . in   Humanoid Robots (Humanoids), 2011 11th IEEE-RAS International Conference on . 2011. IEEE. 27.   Choi, B. and S. Jo,   A low-cost EEG system-based hybrid brain-computer interface for humanoid robot navigation and recognition.   PloS one, 2013.   8 (9): p. e74583. 28.   Li, M., et al.   An adaptive P300 model for controlling a humanoid robot with mind . in   Robotics and Biomimetics (ROBIO), 2013 IEEE International Conference on . 2013. IEEE. 29.   Wolpaw, J.R., et al.,   An EEG-based brain-computer interface for   cursor   control.   Electroencephalography   and   Clinical Neurophysiology, 1991.   78 (3): p. 252-259. 30.   Wolpaw,   J.R.   and   D.J.   McFarland,   Control   of   a   two- dimensional   movement   signal   by   a   noninvasive   brain- computer interface in humans.   Proceedings of the National Academy of Sciences of the United States of America, 2004.  101 (51): p. 17849-17854. 31.   McFarland,   D.J.,   W.A.   Sarnacki,   and   J.R.   Wolpaw,  Electroencephalographic (EEG) control of three-dimensional movement.   J Neural Eng, 2010.   7 (3): p. 036007. 32.   Hazrati, M.K. and U.G. Hofmann.   Avatar navigation in Second   Life   using   brain   signals .   in   Intelligent   Signal Processing (WISP), 2013 IEEE 8th International Symposium on . 2013. IEEE. 33.   Bradberry,   T.J.,   R.J.   Gentili,   and   J.L.   Contreras-Vidal,  Decoding   three-dimensional   hand   kinematics   from electroencephalographic signals.   Conf Proc IEEE Eng Med Biol Soc, 2009.   2009 : p. 5010-3. 34.   Bradberry,   T.J.,   R.J.   Gentili,   and   J.L.   Contreras-Vidal,  Reconstructing Three-Dimensional Hand Movements from Noninvasive Electroencephalographic Signals.   The Journal of Neuroscience, 2010.   30 (9): p. 3432-3437. 35.   Antelis, J.M., et al.,   On the usage of linear regression models to reconstruct limb kinematics from low frequency EEG signals.   PloS one, 2013.   8 (4): p. e61976. 36.   Ofner, P. and G.R. Muller-Putz,   Decoding of velocities and positions of 3D arm movement from EEG.   Conf Proc IEEE Eng Med Biol Soc, 2012.   2012 : p. 6406-9. 37.   Ubeda, A., et al.   Linear decoding of 2D hand movements for target selection tasks using a non-invasive BCI system . in  Systems   Conference   (SysCon),   2013   IEEE   International . 2013. 38.   Lv, J., Y. Li, and Z. Gu,   Decoding hand movement velocity from electroencephalogram signals during a drawing task.  Biomed Eng Online, 2010.   9 : p. 64. 39.   Zhang, J., et al.,   Nonlinear EEG Decoding Based on a Particle Filter Model.   BioMed Research International, 2014.  2014 . 40.   RAPIRO. Available from:   http://www.rapiro.com/ .  41.   Abiri,   R.,   et   al.   A   Real-Time   Brainwave   Based   Neuro- Feedback System for Cognitive Enhancement . in   ASME 2015 Dynamic Systems and Control Conference . 2015. American Society of Mechanical Engineers. 42.   McBride, J., et al.,   Resting EEG discrimination of early stage Alzheimer’s disease from normal aging using inter-channel coherence network graphs.   Annals of biomedical engineering, 2013.   41 (6): p. 1233-1242. 43.   McBride,   J.,   et   al.,   Scalp   EEG-based   discrimination   of cognitive deficits after traumatic brain injury using event- related Tsallis entropy analysis.   Biomedical Engineering, IEEE Transactions on, 2013.   60 (1): p. 90-96. 44.   McBride, J.C., et al.,   Spectral and complexity analysis of scalp EEG characteristics for mild cognitive impairment and early Alzheimer's disease.   Computer methods and programs in biomedicine, 2014.   114 (2): p. 153-163. 45.   McBride, J.C., et al.,   Sugihara causality analysis of scalp EEG for detection of early Alzheimer's disease.   NeuroImage: Clinical, 2015.   7 : p. 258-265.