Building Shape and Pose models for a Single object family

The aim of this tutorial is to learn how to build a static shape and pose model for a single object from matching meshes. In correspondence, here means objects have been registered using the same reference mesh and each object is at its original spatial position (position from the acquisition) #Preparation As in the previous tutorials, we start by importing some commonly used objects and initializing the system.

  import multobjectmodels.{DomainWithPoseParameters, ShapeAndPosePDM, SinglePoseExpLogMapping}
  import scalismo.common.{DiscreteField, PointId}
  import scalismo.common.interpolation.NearestNeighborInterpolator
  import scalismo.geometry._
  import scalismo.io.{LandmarkIO, MeshIO}
  import scalismo.mesh.TriangleMesh
  import scalismo.statisticalmodel.PointDistributionModel
  import scalismo.transformations.Translation
  import scalismo.ui.api.ScalismoUI

  scalismo.initialize()
  implicit val rng = scalismo.utils.Random(42)
  val ui = ScalismoUI()

Loading and preprocessing a dataset:

Let us load (download here) and visualize a set of objects meshes based on which we would like to model shape and pose variation:

 val dataGroup = ui.createGroup("datasets")

   val meshFiles = new java.io.File("LollipopData/second_objects/").listFiles
    val rotationCenterFiles = new java.io.File("LollipopData/rotation_centers_second_object/").listFiles


  val dataFiles = for (i<- 0 to meshFiles.size-1) yield {
      val mesh = MeshIO.readMesh(meshFiles(i)).get
      val rotcent=LandmarkIO.readLandmarksJson[_3D](rotationCenterFiles(i)).get
      val meshView = ui.show(dataGroup, mesh, "mesh"+i)
      val rotCentView = ui.show(dataGroup, mesh, "rotCent"+i)
      (mesh, rotcent, meshView, rotCentView) // return a tuple of the mesh and rotation centers with their associated view
    }

You can see that the meshes are at different positions in space and they are in correspondence. This means that for each point on one of the second lollipop objects meshes, we can identify the corresponding point on the other meshes. The corresponding points are identified by the same point identifier.

Exercise: find the rotation angles of some meshes with respect to the fist meshes in the dataset.

Building logarithmic functions from data

In order to study shape variations, we need to factor the shape variations and the pose variations (rotation and translation). This is done by selecting one of the meshes as a reference to create a DomainWithPoseParamter, and then using the reference to compute the logarithmic mapping. The logarithmic map is used to compute a sequence of deformation fields on which the Gaussian process will be computed. This is done simply by computing a logarithm of the deformation fields for the DomainWithPoseParamter created from the rest of the datasets.

  val reference = dataFiles.head._1
    val refRotCent=dataFiles.head._2


    val referenceDomainWithPoseParam=DomainWithPoseParameters(reference,
      refRotCent.head.point,
      Translation(EuclideanVector3D(0.0, 0.1, 0.0)).apply(refRotCent.head.point)
    )

    val singleExpLog=SinglePoseExpLogMapping(referenceDomainWithPoseParam)

    val defFields = for (i <- 0 to dataFiles.size - 1) yield {

      val targDomainWithPoseParam=DomainWithPoseParameters(dataFiles(i)._1,
        dataFiles(i)._2.head.point,
        dataFiles(i)._2.head.point
      )

      val df=DiscreteField[_3D, ({ type T[D] = DomainWithPoseParameters[D, TriangleMesh] })#T, EuclideanVector[_3D]](referenceDomainWithPoseParam,
        referenceDomainWithPoseParam.pointSet.pointsWithId.toIndexedSeq.map(pt =>targDomainWithPoseParam.pointSet.point(pt._2) - pt._1))

      singleExpLog.logMappingSingleDomain(df)

    }

Note that the deformation fields can be interpolated by nearest-neighbour interpolation to ensure that they are defined on all points of the reference mesh.

 val continuousFields = defFields.map(f => f.interpolate(NearestNeighborInterpolator()))

## Single shape and pose model building Now that these shape and pose variations are projected into the tangent (vector) space of the reference using logathimic mapping, learning the shape and pose variations from these deformation fields is done using a PCA. This is essentially the PGA, which is the PCA in tangent space. The model is compiled using the ShapeAndPosePDM class.

val shapeAndPoseModel: ShapeAndPosePGA[TriangleMesh] = ShapeAndPosePGA(defFields,singleExpLog)

Single shape and pose model sampling

We can retrieve random samples of meshes and rotation centers from the model by calling sample on the Gaussian process:

val sample=shapeAndPoseModel.sample()
    val randomMeshSample:TriangleMesh[_3D]=sample.domain
    val randomRotCent:Point[_3D]=sample.rotCenter
    ui.show(randomMeshSample, "randomMeshSample")
    ui.show(Seq(Landmark[_3D]("",randomRotCent)), "randomRotCent")

    //Single shape and pose model marginalisation
    val MarginalshapeModel: PointDistributionModel[_3D, TriangleMesh]=shapeAndPoseModel.shapePDM
    val MarginalPoseFromShapeAndPose: ShapeAndPosePGA[TriangleMesh]=shapeAndPoseModel.PosePGA

Single shape and pose model marginalisation

One wishes to obtain the shape model only or the pose model only. The marginalisation property of a Gaussian process allows to obtain the distribution for a specific class of features. The shape model is calculated as a point distribution model, while the pose model is again a ShapeAndPoseModel, but with the shape set to the mean shape. We can obtain this distribution, by calling the marginal method on the model :

 val MarginalshapeModel: PointDistributionModel[_3D, TriangleMesh]=shapeAndPoseModel.shapePDM
 val MarginalPoseFromShapeAndPose: ShapeAndPosePDM[TriangleMesh]=shapeAndPoseModel.PosePGA