A System For Symbiotic Collaborative Service Robots

Posted on March 22, 2017 in Research • 10 min read

Preamble: This article is derived from a post-doctoral research proposal I recently submitted and provides insights on what's next in service robotics: human-robot interaction, multi-robot systems, cognitive capabilities.

Example of service robot manipulating an object.


Despite the increasing demand for service robots and the rich theoretical and practical contributions in all aspects of their development, cognitive capabilities remain an open challenge. Cognition relies on sensory information to drive decision-making and human-robot interaction through learning and reasoning. Nevertheless, these capabilities are addressed independently and their joint potential is not fully realized. In this research, I propose a unifying approach to learning and reasoning for decision-making and human-robot interaction in multi-robot systems. The method will be integrated and evaluated experimentally as part of a complete system for symbiotic and collaborative service robots.


The problem of domestic service robots has been investigated extensively in recent years thanks to a pressing commercial demand and scientific competitions including the RoboCup @Home initiative [6], the RoCKIn @Home challenge [16] and the European Robotics League for Service Robots (ERL-SR) [8]. Each competition defines a set of scenarios and user stories in dynamic and partially-known environments. The robots are then tested against a series of capabilities.

Robot capabilities. Service robots are expected to provide a combination of the following capabilities: navigation (move safely), mapping (spatial and semantic map both on-line and off-line), person recognition (detect and identify people), person tracking (track people’s position over time), object recognition (detect and identify objects), object manipulation (grasp, move and place objects), speech recognition (recognize and translate the spoken language to text), gesture recognition (recognize hand gestures), cognition (situation awareness, learning, and reasoning to drive human-robot interaction and decision-making), human-robot interaction (multimodal interaction between humans and robots). In [6], Iocchi et. report the results of prior editions of the RoboCup @Home competition summarizing interesting findings, reviewing the best technical solutions and highlighting trends.

  • Navigation and mapping capabilities are well understood and can be regarded as solved problems. Robust implementations are publicly available as modular components in the ROS navigation stack. The optimization of trajectories and the integration of spatial data with semantic information remain open challenges.

  • Person/gesture recognition and tracking capabilities are implemented by using robust and widely adopted libraries such as OpenCV for computer vision, OpenNI for hands and body interaction, PCL for point cloud processing. Robust identification and tracking in crowded environments remain open problems.

  • Object recognition is addressed as a twofold problem: Identification of planes in the 3D scene where objects might be located, and object identification on the candidate planes. Methods for robust object manipulation are provided by the MoveIt! library5.

  • Speech synthesis (text-to-speech) can be regarded as a solved problem and is provided by open-source libraries such as MaryTTS6, eSpeak7, and Festival8. Speech recognition (speech-to-text) is usually implemented using grammar-based methods, such as the freely available library PocketSphinx9. Robust speech recognition in noisy environments remains an open challenge. Meaning and semantics are extracted using methods for computational semantics [10] such as context-free grammar parsers [10], lambda-calculus [17] and grammar learning [11].

  • Cognition depends on perceptual skills and requires the integration of learning, reasoning, decision-making and human-robot interaction capabilities. Perception in robotic systems depends on sensory data, and therefore it is intrinsically uncertain. Recent studies investigate the combination of probabilistic learning and logical inference for decision-theoretic planning [2, 18, 26, 19]. Partially observable Markov decision processes (POMDPs) are the most popular model for sequential actions under uncertainty. POMDPs have been extended to support logical reasoning using Answer Set Programming (ASP) [25, 17] and first-order logic [15] with promising results.

  • Human-robot interaction relies on the other capabilities to implement interactions between humans and robots [4, 5, 7]. E.g., the verbal request “Can you please bring me a snack?” triggers several sub-modules, including task planning, navigation, object detection and manipulation, speech recognition and cognition. Symbiotic collaboration [20] overcomes robot limitations by identifying and delegating sub-tasks to humans. E.g., the robot is aware of its limitations and asks for help: “Can you please open the door for me?”. Spoken dialog systems study the problem of conversational speech interactions and are therefore relevant to the development of symbiotic collaboration. POMDPs have been used successfully in this field [22, 24].

Networked robot systems. Multi-Robot Systems have been extensively studied in the last decades [23] and their potential advantages over single-robot systems include: better spatial distribution; better overall system performance; cheaper and simpler hardware; better overall system reliability, flexibility, and scalability; better system robustness thanks to data fusion and redundancy.

  • Robots are either homogeneous (identical capabilities) or heterogeneous (different capabilities). Their collective behavior is either cooperative or competitive: cooperative robots share information to reach a joint goal while competitive robots compete independently to reach the same goal.

  • Coordination addresses resource conflicts (e.g., two colliding trajectories) and is either static (e.g., “turn always right”) or dynamic (e.g., “communicate to resolve conflict”). Communication is either explicit or implicit: explicit communication refers to the direct exchange of information (e.g., network messages), implicit communication refers to information that can be inferred by sensing the environment (e.g., the location of the other robots is inferred by vision-based tracking).

  • Task planning and navigation are high-level challenges of multi-robot systems. Prior works studying the adoption of POMDPs for decision-making in multi-robot systems include [12, 20, 2].

In [13], the authors demonstrate the first use of heterogeneous robots in the RoboCup @Home competition. In [2], the authors propose a heterogeneous security system of robots and networked security cameras. Methods for navigation in multi-robot formations are presented in [21, 7]. Recently, the concept of robot-as-a-service (RaaS) has been introduced [3, 9]. The benefits of RaaS are twofold: faster development or robotic applications by taking advantage of shared APIs and more efficient use of computational resources.

Robot Operating System. ROS is a popular modular and distributed platform for robotics research [14]. Its communication layer provides a publish/subscribe and service messaging system. Nodes can publish or subscribe to topics and can provide or call services. Topics have anonymous publish/subscribe semantics and are intended for unidirectional, streaming communication. Services are similar to RPC requests and are defined by a pair of messages: one for the request and one for the reply. The ROS system is a set of ROS nodes interconnected through topics and services. Each node provides certain capabilities to the robot. E.g., navigation. Nodes are organized into packages, whose goal is to be reusable components. The ROS community has contributed several high-quality, open-source packages.

Recently, ROS 2.0 [1] is being introduced to overcome some of the limitations of ROS:

  • native support for teams of multiple robots
  • direct communication with electronic devices such as sensors and actuators
  • real-time system capabilities
  • resilience to network issues
  • robust deterministic node launch
  • introspection and orchestration

Problem statement

Service robots are moving beyond research labs, and are now being deployed in homes, offices and hospitals. They will interact with untrained or minimally trained people in everyday environments to provide support in daily activities: elderly-care robots will help, monitor and provide assistance in daily living; domestic robots will help in household chores; rehabilitation and edutainment robots will provide assistance through social and physical interaction; security robots will provide remote monitoring and security services. In summary, we are heading toward a pervasive use of multi-robot systems in human-populated environments. Robots are expected to learn, reason, share knowledge and interact with humans as part of their decision-making procedures. Furthermore, spoken dialog systems will enable symbiotic collaboration between humans and robots. As a driving example, consider the following scenario:

Two robots are deployed at an hospital, namely Alice and Bob. A doctor asks Alice to get the medicine for a patient. The request is forwarded to Bob, that is closer to the medicine cabinet. Bob reaches the cabinet, and asks for help to a nearby nurse (Bob is not able to open the cabinet). Bob then brings the medicine to the doctor.

The collaboration between the two robots has reduced the overall execution time of the request. In addition, the symbiotic interaction between Bob and the nurse has been instrumental to overcome a robot limitation and increase the overall efficiency of the hospital organization. Both multi-robot task planning and human-robot interaction rely on spatial and semantic knowledge, statistical learning and logical inference. Nevertheless, these capabilities are usually addressed independently and their joint full potential is not fully realized. How can we combine them into a unifying model? The present research plan aims to answer this question.


The primary goal of this research is to design a networked system of symbiotic and collaborative service robots using a unifying approach to learning and reasoning for cooperative task planning and human-robot interaction. In addition, the system will address practical technical challenges arising in real-world robot deployments. The contributions of this research are the following:

  • The development of a single principle approach to learning and reasoning for decision-making and human-robot interaction in multi-robot systems. A promising research direction is the combination of hierarchical partially observable Markov decision processes (POMDPs) and answer set programming (ASP). POMDPs are widely adopted to model multi-robot decision-making and human-robot dialogues. ASP is a well studied model to represent and perform logical inference with incomplete world knowledge. The General Purpose Service Robot (GPSR) tests in ERL-SR and RoboCup @Home competitions will be used as driving scenarios and in the experimental evaluation.

  • The integration of existing and novel methods into a real-world system of networked service robots based on the ROS system. The platform will provide these capabilities: navigation and mapping (ROS navigation stack), person recognition/tracking and gesture recognition (OpenCV, OpenNI and ROS/PCL libraries), object recognition and manipulation (MoveIt!), speech processing (MaryTTS and PocketSphinx), single and multi-robot decision-making, symbiotic interaction and human-robot dialogues.

  • The integration of existing and novel solutions to support multiple robots and networked electronic devices, resilience to network issues, robust software introspection and orchestration. These features are part of the development roadmap of ROS 2.0 and the project will contribute to its official repositories. The resulting system will participate and will be evaluated in international robotic competitions, including ERL-SR and RoboCup @Home.


Service robots are being deployed at an increasing pace in several application domains. In this research plan, I reviewed prior studies on service robot capabilities, multi-robot systems, and human-robot interaction. Cognitive skills rely on learning and reasoning to drive decision-making and human-robot interaction. Albeit their tight relationship, there is no off-the-shelf solution to integrate their knowledge and inference models to exploit their joint full potential.

In this study, I propose a unifying approach to be evaluated as part of a complete system for symbiotic and collaborative service robots. The system will be tested in real-world scenarios and will compete in international competitions.


  • [1] Ros 2.0 design. http://design.ros2.org. Accessed: 2017-02-28.
  • [2] A. Ahmad and P. Lima. Dataset suite for benchmarking perception in robotics. In IROS 2015 * Workshop: Open Forum on Evaluation of Results, Replication of Experiments and Benchmarking in * Robotics Research, 2015.
  • [3] Y. Chen, Z. Du, and M. Garc ́ıa-Acosta. Robot as a service in cloud computing. In Service * Oriented System Engineering (SOSE), 2010 Fifth IEEE International Symposium on, pages 151–158. IEEE, 2010.
  • [4] T. Fong, I. Nourbakhsh, and K. Dautenhahn. A survey of socially interactive robots. Robotics and autonomous systems, 42(3):143–166, 2003.
  • [5] M. A. Goodrich and A. C. Schultz. Human-robot interaction: a survey. Foundations and trends in human-computer interaction, 1(3):203–275, 2007.
  • [6] L. Iocchi, D. Holz, J. Ruiz-del Solar, K. Sugiura, and T. Van Der Zant. Robocup@ home: Analysis and results of evolving competitions for domestic and service robots. Artificial Intelligence, 229:258–281, 2015.
  • [7] A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis. Human interaction with robot swarms: A survey. IEEE Transactions on Human-Machine Systems, 46(1):9–26, 2016.
  • [8] P. U. Lima, D. Nardi, G. K. Kraetzschmar, R. Bischoff, and M. Matteucci. Rockin and the european robotics league-building on robocup best practices to promote robot competitions in europe. 2016.
  • [9] B. Liu, Y. Chen, A. Hadiks, E. Blasch, A. Aved, D. Shen, and G. Chen. Information fusion in a cloud computing era: a systems-level perspective. IEEE Aerospace and Electronic Systems Magazine, 29(10):16–24, 2014.
  • [10] J. Lunenburg, S. van den Dries, R. Appeldoorn, R. Wijnands, and M. van de Molengraft. Tech united eindhoven@ home 2016 team description paper. Joao Pessoa, Brazil, 2014.
  • [11] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language commands to a robot control system. In Experimental Robotics, pages 403–415. Springer, 2013.
  • [12] J. V. Messias, M. T. Spaan, and P. U. Lima. Gsmdps for multi-robot sequential decision-making. In AAAI, 2013.
  • [13] S. Meyer zu Borgsen, T. Korthals, and S. Wachsmuth. Tobi-team of bielefeld the human-robot interaction system for robocup@ home 2016. 2016.
  • [14] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng. Ros: an open-source robot operating system. In ICRA workshop on open source software, volume 3, page 5. Kobe, 2009.
  • [15] S. Sanner, K. Kersting, et al. Symbolic dynamic programming for first-order pomdps. In AAAI, volume 10, pages 1140–1146, 2010.
  • [16] S. Schneider, F. Hegger, A. Ahmad, I. Awaad, F. Amigoni, J. Berghofer, R. Bischoff, A. Bonarini, R. Dwiputra, G. Fontana, et al. The rockin@ home challenge. In ISR/Robotik 2014; 41st International Symposium on Robotics; Proceedings of, pages 1–7. VDE, 2014.
  • [17] W. Shuai, J. Liu, X. Wang, F. Zhou, and X. Chen. Wrighteagle@ home 2016 team description paper.
  • [18] M. Sridharan and M. Gelfond. Using knowledge representation and reasoning tools in the design of robots. In IJCAI Workshop on Knowledge-Based Techniques for Problem Solving and Reasoning (KnowProS), volume 10, pages 1288–1302, 2016.
  • [19] S. Srivastava, S. Russell, A. Pfeffer, and C. R. Analytics. First-order models for pomdps. In UAI Workshop on Statistical Relational Artificial Intel ligence (StarAI). Citeseer, 2012.
  • [20] M. M. Veloso, J. Biswas, B. Coltin, and S. Rosenthal. Cobots: Robust symbiotic autonomous mobile service robots. In IJCAI, page 4423, 2015.
  • [21] A. Wasik, J. N. Pereira, R. Ventura, P. U. Lima, and A. Martinoli. Graph-based distributed control for adaptive multi-robot patrolling through local formation transformation. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, pages 1721–1728. IEEE, 2016.
  • [22] J. D. Williams and S. Young. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393–422, 2007.
  • [23] Z. Yan, N. Jouandeau, and A. A. Cherif. A survey and analysis of multi-robot coordination. International Journal of Advanced Robotic Systems, 10(12):399, 2013.
  • [24] S. Young, M. Gaˇsi ́c, B. Thomson, and J. D. Williams. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179, 2013.
  • [25] S. Zhang, M. Sridharan, and J. L. Wyatt. Mixed logical inference and probabilistic planning for robots in unreliable worlds. IEEE Transactions on Robotics, 31(3):699–713, 2015.
  • [26] S. Zhang and P. Stone. Corpp: Commonsense reasoning and probabilistic planning, as applied to dialog with a mobile robot. In AAAI, pages 1394–1400, 2015.