MoGaze: A Dataset of Full-Body Motions that Includes Workspace Geometry and Eye-Gaze
This tutorial describes how to get started using the MoGaze dataset with our pybullet based library humoro.
There is an ipython notebook version of this tutorial available.
The installation is tested on Ubuntu 18.04.
The packages python3 and python3-pip need to be installed and upgraded (skip if python already is installed):
sudo apt install python3
sudo apt install python3-pip
python3 -m pip install --upgrade pip --user
For parts of the software qt5 is used, it can be installed using:
sudo apt install qt5-default
Clone the repository:
git clone https://github.com/PhilippJKratzer/humoro.git
The python requirements can be installed using:
cd humoro
python3 -m pip install -r requirements.txt --user
Finally, you can install humoro system-wide using:
sudo python3 setup.py install
Download and extract the dataset files:
wget https://github.com/humans-to-robots-motion/mogaze/raw/master/data/mogaze.7z
7z x mogaze.7z
Let’s first have a closer look into the human data only. We can load a trajectory from file using the following:
from humoro.trajectory import Trajectory
full_traj = Trajectory()
full_traj.loadTrajHDF5("mogaze/p1_1_human_data.hdf5")
The trajectory contains a data array, a description of the joints and some fixed joints for scaling:
print("The data has dimension timeframe, state_size:")
print(full_traj.data.shape)
print("")
print("This is a list of jointnames (from the urdf) corresponding to the state dimensions:")
print(list(full_traj.description))
print("")
print("Some joints are used for scaling the human and do not change over time")
print("They are available in a dictionary:")
print(full_traj.data_fixed)
To play the trajectory using the pybullet player, we spawn a human and add a trajectory to the human
from humoro.player_pybullet import Player
pp = Player()
pp.spawnHuman("Human1")
pp.addPlaybackTraj(full_traj, "Human1")
A specific frame can be displayed:
pp.showFrame(3000)
Or a sequence of frames can be played using:
pp.play(duration=360, startframe=3000)
There is also a possibility to use a Qt5 widget (pp.play_controls()) to allow fast forward and skipping through the file. It has also some options for segmenting the data. We explain it in the segmentation section.
Often it is useful to display multiple human trajectories at the same time. For example, it can be used to show the output of a prediction and the ground truth at the same time.
It can be achieved by spawning a second human and adding a trajectory to it. A trajectory also has an element startframe, which tells the player when a trajectory starts.
pp.spawnHuman("Human2", color=[0., 1., 0., 1.])
# this extracts a subtrajectory from the full trajectory:
sub_traj = full_traj.subTraj(3000, 3360)
# we change the startframe of the sub_traj,
# thus the player will play it at a different time:
sub_traj.startframe = 4000
pp.addPlaybackTraj(sub_traj, "Human2")
pp.play(duration=360, startframe=4000)
There is a helper function to directly spawn the objects and add the playback trajectories to the player:
from humoro.load_scenes import autoload_objects
obj_trajs, obj_names = autoload_objects(pp, "mogaze/p1_1_object_data.hdf5", "mogaze/scene.xml")
pp.play(duration=360, startframe=3000)
You can access the object trajectories and names:
print("objects:")
print(obj_names)
print("data shape for first object:")
print(obj_trajs[0].data.shape) # 7 dimensions: 3 pos + 4 quaternion rotation
The gaze can be loaded the following way. Only a trajectory of gaze direction points is loaded, the start point comes from the “goggles” object.
from humoro.gaze import load_gaze
gaze_traj = load_gaze("mogaze/p1_1_gaze_data.hdf5")
pp.addPlaybackTrajGaze(gaze_traj)
pp.play(duration=360, startframe=3000)
If you want to use the raw gaze data, the direction points need to be rotated by the calibration rotation:
print("calibration rotation quaternion:")
print(gaze_traj.data_fixed['calibration'])
The following loads a small Qt5 Application that displays a time axis with the segmentations. The file is segmented into when an object moves.
Note that opening the QApplication does not allow to spawn any new objects in pybullet.
pp.play_controls("mogaze/p1_1_segmentations.hdf5")
The label “null” means that no object moves at the moment (e.g. when the human moves towards an object to pick it up.
It is also possible to directly use the segmentation file, it contains elements of the form (startframe, endframe, label):
import h5py
with h5py.File("mogaze/p1_1_segmentations.hdf5", "r") as segfile:
# print first 5 segments:
for i in range(5):
print(segfile["segments"][i])
In order to compute positions from the joint angle trajectory, pybullet can be used. We have a small helper class, which can be used like this:
from humoro.kin_pybullet import HumanKin
kinematics = HumanKin()
kinematics.set_state(full_traj, 100) # set state at frame 100
print("position of right wrist:")
wrist_id = kinematics.inv_index["rWristRotZ"]
pos = kinematics.get_position(wrist_id)
print(pos)
The Jacobian can be retreived with:
print(kinematics.get_jacobian(wrist_id))