Abstract
The need for 3D and 4D (i.e., 3D + time) shape analysis arises in many branches of science ranging from anatomy, bioinformatics, medicine, and biology to computer graphics, multimedia, and virtual and augmented reality. In fact, shape is an essential property of natural and man-made 3D objects. It deforms over time as a result of many internal and external factors. For instance, anatomical organs such as bones, kidneys, and subcortical structures in the brain deform due to natural growth or disease progression; human faces deform as a consequence of talking, executing facial expressions, and aging. Similarly, human body actions and motions such as walking, jumping, and grasping are the result of the deformation, over time, of the human body shape. The ability to understand and model (1) the typical shape and deformation patterns of a class of 3D objects, and (2) the variability of these shapes and deformations within and across object classes has many applications. For example, in medical diagnosis and biological growth modeling, one is interested in measuring the intensity of pain from facial deformations, and in distinguishing between normal growth and disease progression using the shape of the body and its deformation over time. In computer vision and graphics, the ability to statistically model such spatiotemporal variability can be used to summarize collections of 3D objects and their animation, and simulate animations and motions. Similar to 3D morphable models, these tools can also be used in a generative model for synthesizing large corpora of labeled longitudinal 3D shape data, e.g., 4D faces, virtual humans, and various objects. In this talk, I will share the research undertaken by my group and collaborators in the area of statistical analysis and modelling of static (i.e., 3D) and dynamic (i.e., 4D) shapes. I will first highlight the importance of this topic for various applications ranging from biology and medicine to computer graphics and virtual/augmented reality. I will then structure my talk into three parts. The first one focuses on 3D shapes that bend, stretch, and change in topology. I will introduce our mathematical framework, termed Square Normal Fields (SRNF) [6, 10-12, 15], which provides (1) an efficient representation of 3D shapes, (2) an elastic metric for quantifying shape differences between objects, (3) mechanisms for computing correspondences and geodesics between such shapes, and (4) methods for characterizing populations of 3D shapes using generative models. I will consider both shapes that bend and stretch [6, 10-12, 15] but also those that change their structure and topology [18-22]. The second part of the talk will focus on 4D shapes, i.e., 3D shapes that move and deform as the result of normal growth or disease progression [9, 14]. I will summarize the latest solutions we developed for the statistical analysis of the spatio-temporal variability in such 4D shape data and highlight their applications in various fields. The third part of this talk will focus on the role statistical 3D and 4D shape models played and have to play in the era of Deep Learning and Generative AI. I will particularly highlight their importance and the role they played in advancing the field of 3D and 4D reconstruction and generation from images, videos, and text [1-5, 7, 8, 13, 16, 17]. I will conclude the talk by sharing insights into potential future developments in and applications of statistical 3D and 4D shape models.