Shap-E: A Generative Model for 3D Assets with NeRF Capability

OpenAI has unveiled its latest project, Shap-E, a conditional generative model for creating 3D assets. Unlike other 3D generative models that produce a single output representation, Shap-E can directly generate the parameters of implicit functions, which can be rendered as both textured meshes and neural radiance fields (NeRF) using single text prompts. Shap-E is open-source and available on GitHub, complete with model weights, inference code, and samples.

Shap-E Training Process

Shap-E is trained in two stages. First, an encoder is trained to deterministically map 3D assets into the parameters of an implicit function. Second, a conditional diffusion model is trained on the encoder's outputs. This training process allows Shap-E to generate complex and diverse 3D assets in just a few seconds when trained on a large dataset of paired 3D and text data.

Performance and Limitations

Shap-E converges faster and produces comparable or better sample quality than Point·E, despite modeling a higher-dimensional, multi-representation output space. However, there are some limitations to Shap-E's capabilities. The 3D objects generated may appear pixelated and rough, and while models can be generated with a single text, Shap-E is currently limited to producing objects with single object prompts and simple attributes. The model struggles with multiple attributes, as mentioned in the paper.

Relation to Point-E and DALL-E

OpenAI recently released Point-E, dubbed "3D DALL-E 2," which used the same diffusion technique found in DALL-E and Point-E. Shap-E also leverages this diffusion technique, but instead of point cloud diffusion as in Point-E, users can now generate NeRF-capable textured meshes.

Conclusion

OpenAI's Shap-E is a significant step forward in the development of generative models for 3D assets. By enabling the generation of textured meshes and NeRFs with single text prompts, Shap-E opens up new possibilities for 3D asset creation. Despite its limitations, Shap-E demonstrates the potential for further advancements in this area. You can explore Shap-E and its open-source resources on GitHub.