Human Body Pose Estimation and Shape Reconstruction

My first projects were about reconstructing people from images. I began with animatable SMPL body models, then turned to clothing: how to represent separate garments, keep their layers from intersecting, and infer their shape from a single view. That work led to Layered-Garment Net and to my later interest in partially hidden hands and objects.

Progress log

2022 — Layered-Garment Net

In Layered-Garment Net (LGN), we reconstructed separate clothing layers from a single image while accounting for how they cover the body and one another. The paper appeared at ACCV 2022.

Spring 2021 — Garment data and reconstruction benchmarks

I set up comparisons of existing garment reconstruction methods. I prepared BCNet garment meshes, generated the multi-view images and joint annotations needed by MGN, and tested reconstructions with MGN, PIFu, PIFuHD, and Octopus. Because the methods produced meshes with different topology, I worked on camera alignment and surface-distance measures for the comparison.

With Alakh, I also worked on a synthetic dataset of SMPL bodies wearing separate garment layers. We selected clothing combinations and poses, then began rendering them from multiple views. Toward the end of the semester, I started a viewer for implicit surfaces represented with Hermite radial basis functions (HRBFs).

2020–2021 — PAVM and garment modeling

Before LGN, I worked with PAVM to represent bodies and garments. I animated the model in Unity, tested skinning weights for garments such as skirts, and prepared garment segmentation data. This was an early attempt to model how separate garments move with the body.

2019 — Animatable SMPL reconstruction

I adapted reconstructed SMPL meshes for use in Unity. That included mapping vertices between the reconstruction and Unity models, applying shape offsets and textures, and testing whether the resulting body could be animated. I also built a mobile prototype that recorded a video, retrieved reconstruction output from a server, and displayed the model.

Why layers matter

A single reconstructed surface cannot represent both a jacket and the shirt hidden beneath it. From one image, we have to infer surfaces we cannot see while keeping the garments from cutting through one another in 3D. In Layered-Garment Net (LGN), we represented the body and each garment layer with its own signed distance field. A garment indication field identifies openings and overlap, so the model can enforce the right covering order where layers meet.

LGN produced distinct garment surfaces from one image while reducing intersections between layers. The same problem of recovering partly hidden 3D shapes later reappeared in my hand-object interaction work.

Selected work: “Layered-Garment Net: Generating Multiple Implicit Garment Layers from a Single Image,” ACCV 2022. Paper · Code