Latest Artificial Intelligence (AI) Research Proposes ECON, A Method To Reconstruct Detailed Clothed 3D Humans From A Color Image

Future video games, movies, mixed reality, telepresence and the “metaverse” will rely heavily on human avatars. We need to accurately reconstruct detailed 3D persons from color images taken in the field to create realistic and customized avatars at scale. Due to the difficulties involved, this issue has yet to be resolved. People dress differently, accessorize differently, and position their bodies in various, often innovative ways. A decent reconstruction technique should capture them precisely while standing up to creative clothing and poses. These techniques require a more specific understanding of the anatomy of the human body and thus tend to overfit the positions observed in the training data.

As a result, people often create misshapen shapes or disembodied limbs for pictures of unfamiliar poses; see the second row of Figure 1. The third and fourth rows of Figure 1 show how follow-up work adjusts the IF using a shape provided in advance by an explicit body model to account for such artifacts, but it can limiting the applicability to new clothing while weakening shape details. In other words, robustness, generality and detail can all be traded off. However, the robustness of explicit anthropomorphic body models and the adaptability of IF to capture various topologies is what we want.

In light of this, we note two important facts: (1) Deriving a 3D geometry with comparably precise features is still difficult, even if it is very simple to derive detailed 2D normal maps from color photographs. Using meshes, we can precisely imply “geometry-aware” 2D maps that we can lift into 3D. (2) It is possible to think of a body model as a low-frequency “canvas” that “guides” the sewing of finely detailed surface sections. We create ECON, a revolutionary technique for “explicitly dressed people obtained from normal people,” with these considerations in mind. An RGB image and a derived SMPL-X body are the input to ECON. Then it produces a 3D person wearing freeform clothing with an advanced degree of detail and robustness (SOTA).

Step 1: Normal front and back rebuild. Using a conventional image-to-image translation network, we predict front and back clothed human normal maps from the input RGB image, conditional on the body estimation.

Step 2: Reconstruction of the front and back surfaces. To create accurate and coherent front-back 3D surfaces, MF, MB, we use the previously predicted normal maps and the corresponding depth maps produced from the SMPL-X mesh. To achieve this, we extend the recently published BiNI method and create a new optimization strategy to achieve three objectives for the resulting surfaces:

1. Their high frequency components correspond to normal clothed people.
2. Their low-frequency components and discontinuities correspond to SMPL-X.
3. The depth values ​​on their silhouettes are coherent with each other and match the SMPL-X based depth maps.

The occluded and “profile” sections of the two output surfaces, MF and MB, lack geometry, making them detailed but incomplete.

Step 3: Complete the 3D shape. The SMPL-X mesh and the two d-BiNI surfaces, MF and MB, are the two inputs to this module. The goal is to “paint” the missing geometry. Existing solutions struggle to solve this problem. On the one hand, Poisson reconstruction naively “fills” gaps without benefiting from a shape distribution prior, leading to “blobby” shapes.

However, data-driven methods need help with (self-)occlusion-related missing pieces and lose information available in provided high-quality surfaces, leading to degenerate geometries. We overcome the limitations above in two steps: (1) For SMPL-X to regularize shape “filling”, we extend and redirect our IF-Nets to be conditioned on the SMPL-X body. Triangles near MF and MB are discarded, while the remaining triangles are kept as “filler spots”. (2) Using Poisson reconstruction, we connect the front and back surfaces as well as the “fill patches”; note that the gaps between them are small enough for a universal technique.

ECON combines the best features of explicit and implicit surfaces to produce strong and detailed 3D reconstructions of clothed people. As seen at the bottom of Figure 1, the outcome is a complete 3D shape of a clothed person. We assess ECON using real photos and known benchmarks (CAPE, Renderpeople). According to a quantitative study, ECON outperforms SOTA. Qualitative findings show that ECON generalizes more effectively than SOTA to a wide range of poses and wears, even when the topology is extremely loose or complicated. This is supported by perceptual research, which shows that ECON is highly favored over opponents in difficult positions and loose clothing when competing with PIFuHD in fashion photos. Code and models are accessible on GitHub.

Look at the Paper, Codeand Project. All credit for this research goes to researchers on this project. Also, don’t forget to join our Reddit page and disagreement channelwhere we share the latest AI research news, cool AI projects, and more.