Building shape and pose models for multple object family

The objective of this tutorial is to learn how to build a shape and pose model for several families of objects from matching meshes. Correspondence means that objects of the same family have been registered using the same reference mesh and that each object is at its original spatial position (position from the acquisition).

Preparation

As in the previous tutorials, we start by importing some commonly used objects and initializing the system.

import ShapeAndPoseModels._
import scalismo.common.{DiscreteField, PointId}
import scalismo.common.interpolation.NearestNeighborInterpolator
import scalismo.geometry._
import scalismo.io.{LandmarkIO, MeshIO}
import scalismo.mesh.TriangleMesh
import scalismo.transformations.Translation
import scalismo.ui.api.ScalismoUI

scalismo.initialize()
implicit val rng = scalismo.utils.Random(42)

val ui = ScalismoUI()

Loading and preprocessing a dataset:

Let’s load (and visualise) a set of first and second lollipop meshes from which we want to model variations in shape and pose. We are interested in the lollipop joint (a joint consisting of a first and second lollipop object), but the framework works with any (object complex) joint with a finite number of objects:

val dataGroup = ui.createGroup("datasets")

  val firstObjectMeshFiles = new java.io.File("LollipopData/first_objects/").listFiles
  val secondObjectMeshFiles = new java.io.File("LollipopData/second_objects/").listFiles

  val firstObjectRotationCenterFiles = new java.io.File("LollipopData/rotation_centers_first_object/").listFiles
  val secondObjectRotationCenterFiles = new java.io.File("LollipopData/rotation_centers_second_object/").listFiles

  val dataFiles = for (i<- 0 to firstObjectMeshFiles.size-1) yield {
    val firstObjectMesh = MeshIO.readMesh(firstObjectMeshFiles(i)).get
    val secondObjectMesh = MeshIO.readMesh(secondObjectMeshFiles(i)).get

    val firstObjectRotCent=LandmarkIO.readLandmarksJson[_3D](firstObjectRotationCenterFiles(i)).get
    val secondObjectRotCent=LandmarkIO.readLandmarksJson[_3D](secondObjectRotationCenterFiles(i)).get

    val firstObjectMeshView = ui.show(dataGroup, firstObjectMesh, "firstObjectMesh"+i)
    val secondObjectMeshView = ui.show(dataGroup, secondObjectMesh, "secondObjectMesh"+i)

    val firstObjectRotCentView = ui.show(dataGroup,firstObjectRotCent, "firstObjectRotCent")
    val secondObjectRotCentView = ui.show(dataGroup,secondObjectRotCent, "secondObjectRotCent")

    (List(firstObjectMesh, secondObjectMesh),
      List(firstObjectRotCent.head.point, secondObjectRotCent.head.point),
      List(firstObjectMeshView,secondObjectMeshView),
      List(firstObjectRotCentView,secondObjectRotCentView)
    ) // return a tuple of the mesh and roation centers with their associated view
  }

Exercise: find the rotation angles (Euler’s angles) of some humeri (hemeri brought to the reference frame) with respect to the first humerus mesh in the dataset.

Computing logarithmic functions from data

As with Single shape and pose models, to study shape variations we need to extract shape and pose variations. This is done by selecting two of the meshes (first and second objects) as references to create Multiple Object Structures, and then using the reference to calculate the Logarithmic Mapping. The logarithmic map is used to compute a sequence of shared deformation fields (containing the shape and posture of both objects) over which the Gaussian process will be computed. This is done simply by calculating a logarithm of the deformation fields for the MultibodyObject[_3D, TriangleMesh] structures created from the rest of the datasets.

   val firstObjectReference=dataFiles.head._1(0)
    val secondObjectReference=dataFiles.head._1(1)
    val firstObjectRotCentRef=dataFiles.head._2(0)
    val secondObjectRotCentRef=dataFiles.head._2(1)

    val referenceMultibodyObject: MultiBodyObject[_3D, TriangleMesh] =  MultiBodyObject(
      List(firstObjectReference,secondObjectReference),
      List(firstObjectRotCentRef,secondObjectRotCentRef),
      List(Translation(EuclideanVector3D(0.0, 0.1, 0.0)).apply(firstObjectRotCentRef),Translation(EuclideanVector3D(0.0, 0.1, 0.0)).apply(secondObjectRotCentRef))

      )

    val expLog=MultiObjectPoseExpLogMapping(referenceMultibodyObject)

    val defFields = for (i <- 0 to dataFiles.size - 1) yield {

      val targMultiBody=
        MultiBodyObject(dataFiles(i)._1,
          dataFiles(i)._2,
          dataFiles(i)._2
        )

      val df = DiscreteField[_3D, MultiBodyObject[_3D, TriangleMesh]#DomainT, EuclideanVector[_3D]](referenceMultibodyObject, referenceMultibodyObject.pointSet.pointsWithId.toIndexedSeq.map(pt => targMultiBody.pointSet.point(pt._2) - pt._1))

      expLog.logMapping(df)
    }

Note that the deformation fields can be interpolated by nearest-neighbour interpolation to ensure that they are defined on all points of the reference mesh.

 val continuousFields = defFields.map(f => f.interpolate(NearestNeighborInterpolator()))

## Building of shape and pose models of multiple object families Now that these shape and pose variations are projected into the tangent (vector) space of the reference multibody object using logathimic mapping, the learning of shape and pose variations for the multibody object family from these deformation fields is performed using a PGA.

 val multiBodyShapeAndPosePGA = MultiBodyShapeAndPosePGA(defFields,expLog)

Sampling of shape and pose model of multiple object families

We can recover random samples of meshes and centres of rotation from the model by calling the sample on the Gaussian process:

    val sample=multiBodyShapeAndPosePGA.sample()
    val randomFirstObjectSample:TriangleMesh[_3D]=sample.objects(0)
    val randomSecondObjectSample:TriangleMesh[_3D]=sample.objects(1)
    val randomFirstObjectRotCent:Point[_3D]=sample.rotationCenters(0)
    val randomSecondObjectRotCent:Point[_3D]=sample.rotationCenters(1)

    ui.show(randomFirstObjectSample, "randomFirstObjectSample")
    ui.show(randomSecondObjectSample, "randomSecondObjectSample")
    ui.show(Seq(Landmark[_3D]("scapula",randomFirstObjectRotCent)), "randomFirstObjectRotCent")
    ui.show(Seq(Landmark[_3D]("humerus",randomSecondObjectRotCent)), "randomSecondObjectRotCent")

Marginalise model of shape and pose of multiple object famlies

One wishes to obtain the shape model only or the pose model only. The marginalisation property of a Gaussian process is used to obtain the distribution for a specific feature class. The shape and pose model is computed by specifying the origin of the specific object family, from the shape and pose models can be computed as discussed in tutorial 4. We can obtain this distribution, by calling the marginal method on the model. We calculate the shape and pose model of the second object.

    val referenceDomainWithPoseParam=DomainWithPoseParameters(secondObjectReference,
      secondObjectRotCentRef,
      Translation(EuclideanVector3D(0.0, 0.1, 0.0)).apply(secondObjectRotCentRef)
    )

    val singleExpLog=SinglePoseExpLogMapping(referenceDomainWithPoseParam)
    val secondObjectTransitionModel:ShapeAndPosePGA[TriangleMesh]=multiBodyShapeAndPosePGA.transitionToSingleObject(referenceDomainWithPoseParam,singleExpLog)


    val singleObjectsample=secondObjectTransitionModel.sample()
    val secondObjectSample:TriangleMesh[_3D]=singleObjectsample.domain
    ui.show(secondObjectSample, "secondObjectSample transition model")

Posterior model of shape and pose of multiple object families

Similar to the posterior model in tutorial 4, we use Gaussian processes for the regression tasks for shape models and multiple object families. The framework also allows regression to any feature included in the domain. One practical application of this regression is to constrain a model to the desired relative pose as well as the reconstruction of a partial shape. To calculate the regression, the Gaussian process model assumes that deformation is only observed up to a certain uncertainty, which can be modelled by a normal distribution. The observed data is specified in terms of points and their identifiers. In the example below, we assume that the joint shape in the third position in the training data is observed and compute the posterior model (pose model).

  val littleNoise=0.2 // variance of the Gaussian noise N(0,0.2)
  val observedData :List[IndexedSeq[(PointId, Point[_3D])]]=List(defFields(2).valuesWithIds.map(v=>(v._2,v._1.shapeVec.toPoint)).toIndexedSeq)//getting the shape of third joint
  val MultiBodyShapeAndPosePosterior: MultiBodyShapeAndPosePGA[TriangleMesh] = multiBodyShapeAndPosePGA.posterior(observedData,littleNoise)