Jens Schmüdderich, "Multimodal learning of grounded concepts in embodied systems", 2010.

The research of methods equipping a technical system with the ability to learn mental concepts of objects, properties, or actions is an important step towards the understanding of intelligence. Moreover, for everyday interaction, a system acquiring knowledge by learning is obviously advantageous compared to a system that requires programming. A special challenge arises from the variety of different characteristics whose association forms our understanding of a concept. For example the concept of a table might comprise a decisive collection of planarity, height, size, the action of placing something on the surface, and the speech label used to refer to the table. In the last decade the growing awareness that concepts must be linked to the real world has led to several approaches capable of learning concepts from interaction. However, most of these systems require supervision during the learning process; others lack the scalability required to span the variety of possible associations forming a concept. An important research question that has been vastly neglected concerns the visual perception: How can a system segregate objects from its surrounding, if it lacks any knowledge about their appearance? In recent approaches this question has been avoided by constraining the learning scenario to more or less static platforms observing objects on a uniformly colored table. This does not only limit the concepts to be learned but also prevents natural interaction. The contribution of this work covers these three different aspects: It presents an unsupervised mechanism for the learning of multimodal concepts, a generic framework for visual perception linking these concepts to the real world, and a system embedding the above on a humanoid robot acting autonomously in dynamic scenes. An important prerequisite for the achieved unsupervised learning from interaction consists in the reliable discrimination between new and known concepts. It bases on a biologically plausible combination of visual, auditory, and behavioral information. To ensure the scalability required to cover the variety of different concepts, a strict modularization with a common representation scheme for communication is employed. In order to link visual concepts to physical objects in the environment and overcome the environmental constraints used in related approaches, a perceptual framework is presented, providing the system with generic, category unspecific object representations. The instantiation of these representations uses a late fusion of different universal cues, capable of segregating arbitrary objects from the environment.\\ One of the employed cues is visual motion. Its extraction poses a challenge to existing methods for movement estimation, because the abrupt movement changes of the camera induced by a walking biped robot disturb the commonly applied estimation of egomotion models. To solve this problem, a novel approach combining visual depth, optical flow, and proprioception is introduced. Its modelfree computation make it a generic approach for motion estimation for moving platforms. Finally, embedding the concept learning and the visual perception on a humanoid robot leads to learning in natural, dynamic scenarios. Existing methods to generate robot behaviors from the mentioned object representations afford autonomous reactions of the robot to its environment. Coupling information about these behaviors to the concept learning enable the robot to learn labels for its own actions. All the presented approaches are embedded on the humanoid robot Asimo and extensively evaluated in typical interaction scenarios. The resulting system is capable of learning concepts for positions, sizes, planar surfaces, movement, and robot behaviors from interaction with a human tutor. The acquired concepts can include online learned speech-labels, and afford the use of synonyms.