arXiv:1706.05340v1 [cs.RO] 16 Jun 2017  Robotic Ironing with 3D Perception and Force/Torque Feedback in Household Environments  David Estevez, Juan G. Victores, Raul Fernandez-Fernandez and Carlos Balaguer  Abstract — As   robotic   systems   become   more   popular   in household environments, the complexity of required tasks also increases. In this work we focus on a domestic chore deemed dull by a majority of the population, the task of ironing. The presented algorithm improves on the limited number of previ- ous works by joining 3D perception with force/torque sensing, with emphasis on finding a practical solution with a feasible implementation in a domestic setting. Our algorithm obtains a point cloud representation of the working environment. From this   point   cloud,   the   garment   is   segmented   and   a   custom Wrinkleness Local Descriptor (WiLD) is computed to determine the location of the present wrinkles. Using this descriptor, the most suitable ironing path is computed and, based on it, the manipulation algorithm performs the force-controlled ironing operation. Experiments have been performed with a humanoid robot platform, proving that our algorithm is able to detect successfully wrinkles present in garments and iteratively reduce the wrinkleness using an unmodified iron.  I. INTRODUCTION An increasing demand exists for robots to be capable of assisting people with everyday domestic chores. A great deal of robotic research has been focusing on aspects of tasks in kitchen [1][2] and even bathroom [3] scenarios. However, certain domestic tasks, specially those related with deformable objects such as textiles (e.g. laundry, ironing, etc), remain a very big challenge for robots. The level of complexity is high for these tasks given the deformations and   almost   infinite   possible   configurations these   objects can adopt. Solutions in perception algorithms working with garments include the use of models which may fail to fit unknown garments [4], or fiducial markers which may even include large numbers of QR identifiers on a single garment [5]. Regarding tasks that require advanced perception and manipulation, the ironing task is a particularly salient ex- ample. If performed incorrectly, it can induce even more wrinkles. Focus groups and systematic studies have described iron- ing as a   strongly disliked domestic chore and one that would be preferably automated [6]. However, literature on robotic ironing is scarce, especially in the domain of real robotic platforms. The EPSRC grant “A Feasibility Study into Robotic Ironing” from the past decade resulted in a number of theoretical analysis, such as a study on algorithms for folding and unfolding garments as well as contemporary robotic gripper solutions that could be of aid during the ironing task [7], and a mathematical model of an ironing  All of the authors are members of the Robotics Lab research group within the Department of Systems Engineering and Automation, Universidad Carlos III de Madrid (UC3M).   destevez@ing.uc3m.es  path derived from a protractor-and-tracing-paper tracking of a human demonstration [8]. More recently, Kormushev et al. incorporated an actual physical robot and force feedback to the robotic ironing process [9]. Ironing paths were extracted from user demon- strations by kinesthetic teaching, and force profiles were extracted from demonstrations via a haptic device. Finally, Li et al. [10] included 3D perception in the ironing process to be able to directly treat wrinkles, while using a 6 cm thick foam to compensate not using force feedback for the actual physical robot setup. The perception system required a dark- room setup with controlled orthogonal light sources, as well as requiring an image of each garment without wrinkles (in the same exact pose) for each wrinkled garment to iron. In this paper, a robotic ironing system that encompasses 3D perception and force/torque feedback evaluated on a full-sized humanoid robot is presented. The algorithms and implementation present the following set of advantages with respect to predecessors in robotic ironing.  •   Standard room light conditions can be used, without special illumination setup or control requirements.  •   The   perception   algorithm   is   garment-agnostic   and model-agnostic, so information of the garment or even its category is not needed.  •   Force-feedback is incorporated within the ironing loop, avoiding the use of foams and similar mechanisms.  •   The ironing loop with force-feedback is based on infor- mation from the 3D perception algorithm. Figure 1 depicts the experimental setup using no special lighting conditions, an arbitrary garment, and an unmodified iron and ironing board with the humanoid robot TEO.  Fig. 1: Experimental setup for full-sized humanoid robotic ironing with 3D perception and force/torque feedback.
The complete robotic ironing algorithm will be described in section II. The experimental setup and results will be given in section III, leading to the final section IV of conclusions. II. ALGORITHM Our algorithm has two main components: a 3D perception- based ironing path generation and an iterative path-following ironing control. These components are designed to work sequentially, applying first the perception component to com- pute the most suitable ironing path, and then using the manip- ulation component to control the iron while performing the ironing trajectory. This sequence can be applied iteratively until no wrinkles are present in the garment patch currently placed on the ironing board. Both components are focused in the feasibility of its implementation in a domestic setting. For that purpose, the perception component only requires an RGB-D sensor, and the manipulation component works with an unmodified iron.  A. 3D perception-based ironing path generation  Our perception component consists of three main stages. In the first stage, the working scene is scanned with the KinectFusion [11] algorithm to obtain a high-resolution 3D representation of the garment and the ironing board. The garment is then segmented from the ironing board, and a custom descriptor is computed in the second stage. Finally, in the last stage, the computed descriptor is used to find the wrinkled areas and to determine the most suitable ironing path. The first stage starts by obtaining a 3D representation of   the   working   scene.   As   RGB-D   sensors   typically   do not have enough resolution on a single depth frame to determine precisely the presence and location of wrinkles, the KinectFusion algorithm is used to integrate data from different perspectives. The resulting 3D mesh is processed and the ironing board plane is located using RANSAC and the following prior about its location and orientation: the plane with a higher z component that has a normal aligned with the global z axis is selected. The ironing board plane contains points from both the ironing board top surface and the garment to be ironed. It is assumed that the points are linearly separable based on color, fulfilling the following requirement for two sets of points   X 0   and   X 1 :  n ∑  i =1  ω i x i   > k   ∀ x   ∈   X 0   (1a)  n ∑  i =1  ω i x i   < k   ∀ x   ∈   X 1   (1b) Where   ω 1 , ω 2 , . . . , ω n , are weights and   k   is a real number. Therefore, we can partition the set of   n   points contained in the ironing board plane into   k   =   k G   +   k B   different clusters,  k G   of them belonging to the garment and   k B   belonging to the ironing board. If uniform color and a Lambertian reflectance model are assumed in both the garment and the ironing board surface, the total number of clusters to consider is  k   = 1 + 1 = 2 . To improve separability we convert the color from the RGB space to the HSV space. We can then iteratively minimize the within-cluster sum of squares using the following expression for   k   clusters:  arg min  S k ∑  i =1  ∑  ~ x ∈ S i  ‖ ~ x   −   μ i ‖ 2   (2) Where   k   is the number of clusters,   S i   is the   i -th cluster,  ~ x   is the feature vector, and   μ i   is the mean of the points in  S i . The feature vector includes both the spatial location of the point and the HSV color components:  ~ x   = ( x, y, z, h, s, v )   (3) Once the clusters are obtained, the garment cluster is labeled accordingly based   on the   spatial   location of   its centroid. This cluster is further cleaned using Euclidean clustering to discard smaller sub-clusters within the garment cluster.   Figure   2   depicts the   steps   of   this   segmentation process. The next stage computes a custom Wrinkleness Local Descriptor (WiLD) that is used in the last stage to locate the wrinkles. The WiLD descriptor is computed using the following equation: WiLD ( ~ i ) = 1  k   ·   ∑  ~ j ∈ K  ~ n ~ i   ·   ~ n ~ j   (4) Where    ̄ n ~ i   denotes the normal vector computed at point  ~ i ,   ~ i   is the point for which the descriptor is being computed and   K   is the set of   k   neighboring points within a radius  r . This descriptor is bounded between 0 and 1, with values closer to 1 representing planar regions and values closer to 0 corresponding to sharp edges. Wrinkles are regions not as sharp as edges so for them the descriptor values lie in intermediate values, which depend on the abruptness of the wrinkle. The different values obtained for this descriptor are projected onto a 2D image, using the garment plane as the projection plane, and z-buffering for managing possible occlusions. A segmentation mask is also generated, assigning values of 1 where a garment point was projected and 0 where no point was projected. This WiLD 2D image and the corresponding mask are the input for the last stage. This stage starts by performing a binary closing over the segmentation mask for filling possible artifacts due to the projection. The erosion applied during the binary closing is larger than the dilation. The reason behind this operation is to remove part of the border, since it will be always curved to some extent due to the shape of the ironing board and the effect of the gravity pulling down on the garment. The contour of the resulting mask is extracted and stored. The WiLD image is then normalized and thresholded to obtain the wrinkled regions:  wrinkle ( ~ x ) =  {  1 ,   if   L t   <   WiLD ( ~ x )   < H t  0 ,   otherwise   (5)
(a) Colored 3D point cloud obtained with KinectFusion  (b) Ironing board plane segmentation result  (c) Segmented garment  Fig. 2: 3D analysis for the garment segmentation stage. Where   L t   and   H t   are the lower and higher thresholds. The Radius-Based Surface Descriptor (RSD) [12] is an alternative descriptor that encodes the radial relationship of the point and its neighborhood, describing the curvature of the surface at a given point. To obtain the ironing path, the binary region is skele- tonized using the Zhang-Suen algorithm. The pixels from this skeleton are converted into a graph, and the start and end points of the ironing path are selected according to the following criteria:  •   Start point: leaf node of the graph that is closest to the garment contour.  •   Ending point: leaf node of the graph that is furthest from the garment contour. Once the start and ending points are selected, the path is found by applying a depth-first search on the skeleton graph. Figure 3 depicts this process.  B. Iterative path-following ironing control  Manipulation for ironing is performed by an iterative force-controlled path-following algorithm that uses a set of   w   =   { w (0) , ..., w ( N   ) }   visually-obtained waypoints as references. Each waypoint   w ( i )   is described by its Cartesian components   ( w ( i )  x   , w ( i )  y   , w ( i )  z   ) . An initial set of waypoints   p   =   { p (0) , ..., p ( N   ) }   is fixed, aimed at moving the iron to an end position over the garment with the iron perpendicular to the surface of the ironing board. The end position is at a fixed height, directly on top of the first waypoint given by the perception system. This is achieved by replacing the components   p ( N   )  x   and   p ( N   )  y   with  w (0)  x   and   w (0)  y   . A vertical descent stage is performed until contact with the ironing board is detected. The descent is either in small com- manded joint increments   ∆ q cmd   given a desired Cartesian increment vector   ∆ x d   = (0 ,   0 ,   − x dz )   using the Levenberg- Marquardt algorithm [13] to perform inverse kinematics, or in a velocity control loop    ̇ q cmd   =   J †  A ( q )   ·    ̇ x d , where    ̇ q cmd  is the commanded joint velocity vector,   J †  A ( q )   is a Moore- Penrose pseudoinverse that uses singular value decompo- sition based on householders rotations with updated joint positions   q   at each iteration, and    ̇ x d   = (0 ,   0 ,   −    ̇ x dz )   is the desired Cartesian velocity vector. The iron descent stage ceases when the force/torque sensor measurements reach above   a   f d   =   (( f dx , f dy , f dz ) ,   ( t dx , t dy   , t dz ))   threshold, which indicates contact. The ironing path   w   =   { w (0) , ..., w ( N   ) } , received from the vision system, must be followed to iron the detected wrin- kles. The iterative force-controlled path-following algorithm is mandated by the control vector   ~ c   at each time step   ( k ) :  ~ c ( k )   =        c ( k )  x   =   c t   ·   cos ( atan 2(( w ( i )  y   −   y ( k ) ) ,   ( w ( i )  x   −   x ( k ) )))  c ( k )  y   =   c t   ·   sin ( atan 2(( w ( i )  y   −   y ( k ) ) ,   ( w ( i )  x   −   x ( k ) )))  c ( k )  z   =   c ( k − 1)  z   +   K f   ·   ( f dz   −   f   ( k )  z   )  (6) Where   ( c ( k )  x   , c ( k )  y   , c ( k )  z   )   are the Cartesian components of  ~ c ( k ) ,   c t   is a user-defined constant corresponds to the desired tangential component of   ~ c ,   x ( k )   and   y ( k )   are Cartesian com- ponents of the current position, and   K f   is the proportional controller gain on the vertical axis. The control vector   ~ c   may be used to as   ∆ x d   to compute a small commanded joint increment   ∆ q cmd   through inverse kinematics as previously, or used in the differential form    ̇ x d   to compute    ̇ q cmd   through its premultiplication of   J †  A ( q ) . Finally, the vertical ascent stage is performed analogous to the descent using   ∆ x d   = (0 ,   0 ,   + x dz )   or    ̇ x d   = (0 ,   0 ,   +  ̇ x dz )  for a certain amount of distance or time. III. EXPERIMENTS For the validation of the proposed algorithm, a series of ex- periments were performed on a robotic platform. This section describes the experimental setup in which the experiments are framed, the experiments, and the results obtained from
Fig. 3: Ironing path extraction from WiLD image. Images correspond (from top to bottom) to wrinkle region selection, skeletonization of selected wrinkle region and ironing path. The image colors correspond to the value of the WiLD de- scriptor for each point. The start and end points of the ironing path are marked with blue and green points, respectively. the experiments. The algorithm has been open sourced and is available online 1 .  A. Experimental Setup  The experimental setup is set to emulate the situation of a robot deployed in a real household environment. For this purpose, a real ironing table was setup with real garments, and a common unmodified iron was used. The robotic platform used for the experiments was TEO, the humanoid robot from the Robotics Lab of Universidad Carlos III de Madrid [14]. Since grasping was not the main purpose of this work, the robot hand was removed, and the iron was installed directly in the right arm using custom 3D printed parts. This robot has a JR3 force/torque sensor in each wrist. For 3D perception the robot head is equipped with an ASUS Xtion PRO LIVE RGB-D sensor. The pan and tilt degrees of freedom of the robot’s neck were used to move  1 https://github.com/roboticslab-uc3m/textiles  the camera while scanning the scene with the KinectFusion algorithm, to generate the disparity required by the algorithm. We used the PCL implementation 2   of KinectFusion for our experiments. After the scan, the 3D scene is introduced in our per- ception algorithm and the ironing path points are obtained. Points are converted from the image frame to the robot frame using the corresponding homogeneous transformation matri- ces. The robot starts performing a preprogrammed approxi- mation trajectory to avoid collisions with the ironing board, and proceeds to descend until the iron contacts the garment. This is detected by setting the   f d   force/torque threshold, which was hand-crafted compensating the deviation between the vertical axis of the sensor and the normal opposition forces generated by the ironing board.  B. Experiments  For evaluation of the algorithm presented in this paper, two different sets of experiments were designed: one for the perception algorithm, and other for the complete algorithm. The values of the different parameters used for the two sets of experiments were the following ones: RANSAC threshold = 0.02, normal estimation radius = 0.02, WiLD neighborhood radius = 0.03, WiLD upper threshold = 0.95, WiLD lower threshold = 0.4, erosion structuring element size = 11. The first set of experiments evaluates the accuracy of the perception algorithm to detect wrinkles. 10 trials were performed with different garments, 5 with 1 wrinkle and 5 with 2 wrinkles. Only the perception component was run for these experiments, performing only the 3D scan of the garment and the wrinkled region extraction. Our algorithm is compared to RSD using the Jaccard Similarity Index (JSI) as a metric:  JSI ( A, B ) =   | A   ∩   B |  | A   ∪   B |   (7) WiLD and RSD extracted regions are compared against a hand-labeled ground truth of the scan using JSI. Computa- tion times for both descriptors were also computed with a GNU/Linux computer equipped with a Intel(R) Core(TM) i7- 4790 CPU @ 3.60GHz processor and NVidia GeForce GTX 960 graphics card using PCL-1.7. Table I shows the results of this set of experiments. On average, WiLD presents a JSI over 25% better than the RSD descriptor, while being more than 40% faster. The assumption regarding precision in terms of JSI is that WiLD has been developed for wrinkles and does not attempt to be as general as RSD. On the other hand, the simple and multi-threaded implementation of WiLD enables much faster processing. The second set of experiments evaluates the performance of the whole algorithm, including both the perception com- ponent and the manipulation component. The values of the  f d   parameter used are ((0,0,-200),(0,25,0)) sensor Internal Units (IU). For each experiment, a garment is manually  2 http://pointclouds.org/documentation/tutorials/ using_kinfu_large_scale.php
TABLE I: Perception Algorithm: Results  Experiment   #1   #2   #3   #4   #5   #6   #7   #8   #9   #10   Mean  RSD   JSI (%)   36.67   32.51   44.90   30.75   41.12   42.27   48.84   38.30   31.07   34.60   38.10  Time (s)   52.1271   50.2347   52.4690   52.3757   53.2446   50.3053   49.7287   50.4271   49.6802   51.6080   51.2200  WiLD   JSI (%)   68.41   60.98   67.39   59.65   61.94   68.64   64.02   65.56   67.03   60.64   64.43  Time (s)   36.0774   34.5110   36.6981   36.6254   38.0608   34.8852   34.888   35.3923   34.2363   35.3987   35.6770  placed on the ironing board. To ensure a homogeneous initial state for all the experiments, the garment is laid mostly flat and two wrinkles are manually generated randomly over the garment. This ensures all the trials start with the same number of wrinkles. The algorithm is performed iteratively, and the number of iterations and elapsed times are recorded for each iteration and trial. The results for the 5 experiments are shown in table II. While intermediate wrinkleness vary, convergence to zero wrinkles on only 2 iterations proves to be a very satisfactory result. As a metric for the progress towards the goal of a com- pletely ironed garment, a measure of the total wrinkleness of the presented garment is computed through the following equation: wrinkleness   =  ∑  ~ x ∈ G   wrinkle ( ~ x )  | G |   (8) Where   G   is the set of points belonging to the garment and  | G |   represents the cardinality of that set. This value of wrinkleness is tracked for each iteration and trial, and shown in Figure 4.  Fig. 4: Wrinkleness on each experiment trial vs iterations. Iteration 0 corresponds to the initial wrinkleness. Zero wrin- kleness corresponds to a situation where no wrinkles between the WiLD thresholds were found. Figure 5 shows a sequence of still frames describing one of the ironing operations performed during the experiments. A video of the algorithm, including a subset of the performed experiments, has been submitted as complementary material. IV. CONCLUSIONS AND FUTURE WORK In this work we present a robotic ironing method for a   humanoid robot using   unmodified human tools in   an unmodified domestic environment. This work is focused on  Fig. 5: Ironing operation being performed during an experi- ment. providing a practical method that has a feasible implemen- tation in a domestic setting. For that purpose it proposes a robotic ironing system that encompasses 3D perception and force/torque feedback. Garment data is obtain by applying our segmentation stage to a colorful 3D scan of the work- ing environment. A custom WiLD descriptor is computed to detect wrinkled regions. Once these regions have been computed, the most suitable ironing path is estimated. The ironing path is used in conjunction with an iterative path- following ironing control to perform the ironing operation over the actual garment. This system has been evaluated on a full-sized humanoid robot. Results show that this system is capable of performing successfully ironing operations over simple garments, and it is very promising to be included in a complete ironing pipeline. Due to the great diversity of garment shapes, textures, materials and decorative elements, ironing is a challenging
TABLE II: Ironing Algorithm: Results  Experiment   #1   #2   #3   #4   #5  Number of iterations   2   2   2   2   2  Avg. time (s) / iteration   71.827   67.803   72.561   75.684   63.684  task to automate. Our work, as a first approach to a practical robot ironing process, is focused in ironing simple garments. As future work we propose to extend the proposed algorithm with detection of different elements present in garments, such as buttons, zippers or other decorative elements. These elements are typically to be avoided when ironing. Different garment parts, such   as collars, cuffs or   pockets require different ironing approaches depending on their size and location, and their detection would benefit the proposed method. Finally, a more relaxed color uniformity constraint would allow this algorithm to work with garments with more than one colors, or even decorative patterns. Regarding   manipulation,   position   and   velocity   control have been tested with force/torque feedback. While velocity control   reduces the   jerk   of   the   movements, it   must   be bounded to avoid excessive joint space velocities when the determinant of the Jacobian is near zero. There is obvious room for testing torque control using the transposed Jacobian matrix   J T   which may enable enhancements through active compliance.   In   addition,   the   current   approach   could   be extended by including the remaining steps of the ironing process to have a fully automated process. One of these steps is the manipulation of the garment for its placement in the ironing board. Once the current garment patch is correctly ironed, this garment has to be removed and, if necessary, placed again in the next configuration. ACKNOWLEDGMENT This   work   was   supported   by   RoboCity2030-III-CM project (S2013/MIT-2748), funded by Programas de Activi- dades I+D in Comunidad de Madrid and EU and by a FPU grant funded by Ministerio de Educacin, Cultura y Deporte. It was also supported by the anonymous donor of a red hoodie used in our initial trials. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the NVIDIA Titan X GPU used for this research. R EFERENCES [1]   M. Beetz, U. Klank, I. Kresse, A. Maldonado,   L. M ̈ osenlechner, D. Pangercic, T. R ̈ uhr, and M. Tenorth, “Robotic roommates making pancakes,” in   Humanoid Robots (Humanoids), 2011 11th IEEE-RAS International Conference on .   IEEE, 2011, pp. 529–536. [2]   T. Asfour, N. Vahrenkamp, D. Schiebener, M. Do, M. Przybylski, K. Welke, J. Schill, and R. Dillmann, “Armar-iii: Advances in hu- manoid grasping and manipulation,”   Journal of the Robotics Society of Japan , vol. 31, no. 4, pp. 341–346, 2013. [3]   A. J. Huete, J. G. Victores, S. Martinez, A. Gim ́ enez, and C. Balaguer, “Personal autonomy rehabilitation in home environments by a portable assistive robot,”   IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews , vol. 42, no. 4, pp. 561–570, 2012. [4]   X. Hu, Y. Bai, S. Cui, X. Du, and Z. Deng, “Review   of Cloth Modeling,” in   ISECS International Colloquium on Computing, Com- munication, Control, and Management , vol. 4.   Sanya: IEEE, 2009, pp. 338 – 341. [5]   C.   Bersch,   B.   Pitzer,   and   S.   Kammel,   “Bimanual   robotic   cloth manipulation for laundry folding,” in   2011 IEEE/RSJ International Conference on Intelligent Robots and Systems , sep 2011, pp. 1413– 1419. [6]   D. Stewardson, D. McGeeney, C. Burdon, T. Foulweather, and F. Gray, “Market research into an automated ironing machine,” Internal Report RIWP2. Newcastle, UK: University of Newcastle upon Tyne, Tech. Rep., 2003. [7]   J.   Dai,   P. Taylor,   H.   Liu,   and   H.   Lin,   “Folding   algorithms   and mechanisms synthesis for robotic ironing,”   International Journal of Clothing Science and Technology , vol. 16, no. 1/2, pp. 204–214, 2004. [8]   J. S. Dai, P. M. Taylor, P. Sanguanpiyapan, and H. Lin, “Trajectory and orientation analysis of the ironing process for robotic automation,”  Intl Journal of Clothing Science and Technology , vol. 16, no. 1-2, pp. 215–226, 2004. [9]   P. Kormushev, S. Calinon, and D. G. Caldwell, “Imitation Learning of Positional and Force Skills Demonstrated via Kinesthetic Teaching and Haptic Input,”   Advanced Robotics , vol. 25, no. 5, pp. 581–603, 2011. [10]   Y. Li, X. Hu, D. Xu, Y. Yue, E. Grinspun, and P. Allen, “Multi- Sensor Surface Analysis for Robotic Ironing,” in   IEEE International Conference on Robotics and Automation (ICRA) , Stockholm, 2016. [11]   R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon, “KinectFusion: Real-time dense surface mapping and tracking,” in  10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) , 2011, pp. 127–136. [12]   Z.-C. Marton, D. Pangercic, N. Blodow, and M. Beetz, “Combined 2D-3D categorization   and classification   for multimodal perception systems,”   The International Journal of Robotics Research , vol. 30, no. 11, pp. 1378–1402, 2011. [13]   D. W. Marquardt, “An algorithm for least-squares estimation of non- linear parameters,”   Journal of the society for Industrial and Applied Mathematics , vol. 11, no. 2, pp. 431–441, 1963. [14]   S. Mart ́ ınez, C. A. Monje, A. Jard ́ on, P. Pierro, C. Balaguer, and D. Mu ̃ noz, “Teo: Full-Size Humanoid Robot Design Powered By a Fuel Cell System,”   Cybernetics and Systems , vol. 43, no. 3, pp. 163– 180, 2012.