@phdthesis {pub4816,
	title = {Randomizing Physic Simulations for Robot Learning},
	author = {Fabio Muratore},
	year = {2021},
	month = {November},
	abstract = {The ability to mentally evaluate variations of the future may well be the key to intelligence.
Combined with the ability to reason, it makes humans excellent at handling new and
complex situations. If we want robots to solve varying tasks autonomously, we need to
endow them with such kind of {\textquoteleft}mental rehearsal{\textquoteright}. Physics simulations allow to predict how
the environment will change depending on a sequence of actions. For example, robots can
simulate multiple control policies in diff erent simulations instances, collect the results,
and subsequently reason about which policy to execute in the real world. As such physics
simulations are highly customizable, they enable generating vast amounts of diverse data
at relatively low cost. Therefore, they make it possible to apply deep learning methods
for physical systems despite the exorbitant demand for data. Since state-of-the-art deep
learning methods come with few guarantees, it is essential to test them in many simulated
scenarios before deployment on the real system.
Over the last decade, the speed and modeling power of general purpose physics engines
increased significantly. State-of-the-art simulators feature rigid body, soft body, and
fluid dynamics, as well as massive GPU-based parallelization. Despite the impressive
progress, simulations will always remain an idealized model of the real world, thus are
inevitably fl awed. Typical error sources are unmodeled physical phenomena, or suboptimal
parameter values of the underlying generative model. These discrepancies between the
real and the simulated world are summarized by the term {\textquoteleft}reality gap{\textquoteright}. This gap can
manifest in various ways when learning from simulations. In the best case, it is only
a performance drop, e.g., a lower success rate, or a reduced tracking accuracy. More
likely, the learned policy is not transferable to the robot because of unknown friction
eff ects, which lead to underestimating the friction in simulation. Thus, the commanded
actions are in this case not strong enough to get the robot moving. Another reason for
failure are small parameter estimation errors, which can quickly lead to unstable system
dynamics. This case is particularly dangerous for the human and the robot. For these
reasons, bridging the reality gap is the essential step to endow robots with the ability to
learn from simulated experience.
In this thesis, we will tackle the challenge of learning robot control policies from
simulations such that the results can be (directly) transferred to the real world. We focus
on scenarios where the source domain is a randomized simulator and the target domain
is either a different simulation instance (sim-to-sim) or the physical robot (sim-to-real).
We strive to answer the following research questions:
1. How can we quantitatively estimate the transferability of a control policy from one
domain to another?
2. Does randomizing the simulator during learning make the resulting policy more
robust against modeling imperfections?
3. How do we adapt the randomized simulator based on real-world evaluations?
4. Can we infer the source domain parameter distribution from data and subsequently
use it for learning?
5. What are the necessary assumptions and technical requirements to learn robot
control policies from randomized simulations?
Despite the recent popularity of sim-to-real methods, the fi rst question has been unan-
swered up to this point in time. As a consequence, state-of-the-art algorithms can not
make a quantitative statement about the transferability of the resulting control policies.
Moreover, they stop training according to some heuristic like a fixed number of iterations,
which can lead to a waste of computation time. In Chapter 3, we derive the simulation
optimization bias as a measure of the reality gap, and show that policies learned from
a source domain are optimistically biased in terms of their performance in the target
domain, even if they originate from the same distribution. To mitigate this problem,
we propose a policy search algorithm which estimates simulation optimization bias and
continues training until an estimated upper confidence bound on this bias is below a given
threshold. Thus, the resulting policy satisfies a probabilistic guarantee on the performance
loss when transferring the policy to a different environment from the same source domain
distribution. Moreover, our sim-to-real evaluations answer the second question with a
clear {\textquotedblleft}yes{\textquotedblright}.
Straightforwardly learning from randomized source domains shows the tendency to be
slower and have lower performance at the nominal model than methods which close the
sim-to-real loop by adapting the domain parameter distribution. Therefore, we tackle the
third question in Chapter 4 by introducing a policy search algorithm which incorporates
Bayesian optimization to adapt the domain parameter distribution based on real-world
data. The sample-efficiency of Bayesian optimization allows to update the distribution{\textquoteright}s
parameters, including the uncertainty, while only requiring few evaluations on the physical
device. Most notably, the data yielded from these evaluations can be very scarce, e.g., a
scalar return value per trial. This way, the connection between distribution over simulator
parameters and the target domain performance is captured by a probabilistic model. At
the same time we can eliminate the common assumption of knowing the distribution{\textquoteright}s
mean and variance a priori.},
	publisher = {Internal Report},
	booktitle = {TU Darmstadt}
}