Introducing 3D Tiles Next, Streaming Geospatial to the Metaverse
Today we are thrilled to announce that the 3D Tiles Next open specification is available for community feedback.
At the SIGGRAPH conference over six years ago, we introduced the initial 3D Tiles specification for streaming massive heterogeneous 3D geospatial datasets. Since then we have been inspired by how the community has built apps, exporters, APIs, and engines with 3D Tiles to grow an open and interoperable 3D geospatial ecosystem.
Power plant CAD model from Bentley Systems in CesiumJS, point cloud from Government of Victoria in Unreal Engine, San Francisco photogrammetry model from Aerometrex in O3DE, OSM buildings in STK, NASA Perseverance Rover at Jezero Crater in Three.js
This collective experience building with 3D Tiles, combined with the continued growth of 3D geospatial data availability, especially semantic metadata, and increasing user interest in digital twins and the metaverse has led to this next generation of the 3D Tiles specification.
The core of 3D Tiles Next is still streaming interactive 3D at scale. 3D Tiles Next is a set of extensions to allow the developer community to better:
- stream semantic metadata efficiently,
- run massive simulations and analytics via spatial indexes, and
- integrate with the glTF software and extension ecosystem.
The architecture of new draft extensions, in green, for 3D Tiles Next and glTF.
The extensions for 3D Tiles Next enable streaming game-like geospatial experiences to the open metaverse across devices, including today’s and tomorrow’s VR and AR wearables. 3D Tiles Next takes advantage of 5G and compute at edge and in the cloud, all to raise the bar for 3D geospatial experiences in an open and interoperable ecosystem. In developing 3D Tiles Next, we collaborated closely with Maxar to ensure it will serve the modeling and simulation community’s needs for efficient runtime terrain for training, simulation, and operations.
Left: example sources of 3D assets and metadata that are becoming more widely available. Right: runtime engines that already support 3D Tiles Next or have support in progress.
In 3D Tiles Next, we placed a large emphasis on metadata due to its increasing availability and users’ needs. Just as today’s vehicle, sensor, compute, and photogrammetry trends have enabled us to capture the real world with ever-increasing geometry and texture resolution, today’s AI algorithms are increasingly more semantically-rich metadata to augment 3D models.
From the start of 3D Tiles, we saw the need to go beyond visualization by storing these metadata, such as a building ID or point classification, along with geometry and texture. These metadata are often used for styling and filtering, such as applying a classification-based color ramp to differentiate foliage from structures, and for user interaction such as showing the number of floors in a building on mouseover.
3D Tiles Next introduces new metadata extensions for:
- a well-defined type system and more encoding options, such as binary or JSON;
- more storage granularity options, such as per-texel and per-tile; and
- semantic meaning added to metadata, facilitating an ecosystem of domain-specific semantic extensions.
Similar to how 3D Tiles 1.0 used a Batch Table to efficiently store metadata, 3D Tiles Next enables one of the most significant performance advantages of 3D Tiles: batching. Many logical features such as individual buildings, including their metadata, can be pre-batched into one mesh and one draw call at runtime for the graphics API, but they can still be uniquely identified, leading to little per-tile runtime processing and low CPU overhead during rendering.
Metadata type system, encoding options, and example semantic specifications.
The new 3D Metadata Specification defines a type system schema and methods for encoding metadata.
3D Tiles 1.0 relied on JSON’s type system; 3D Tiles Next is more strongly typed, including types for classes, vectors, and matrices. Metadata can be encoded:
- in binary, such as when it would be stored per-vertex for different vertices in the same mesh, or
- in JSON, such as when it may be used per-tile, or
- in raster for fine-grained per-texel granularity.
Using 3DTILES_metadata and EXT_mesh_features (a glTF extension), 3D Tiles Next is designed to efficiently store metadata at varying granularities:
- per tileset
- per tile or tile content group
- per feature
- per GPU instance
- per vertex
- per texel
Example metadata at various granularities.
The finest granularity, per texel, allows metadata to vary even more frequently than the geometry. For example, consider the side of a building that may be modeled with two triangles and then have a metadata texture applied that defines what is glass, brick, or stone. Each of these will interact differently in analytics, such as how rays bounce in RF propagation, and in rendering, such as mapping metadata to glTF’s physically-based rendering (PBR) materials.
Street level photogrammetry of San Francisco Ferry Building from Aerometrex. Left: per-texel colors showing the feature classification, e.g., roof, sky roof, windows, window frames, and AC units . Right: classification is used to determine rendering material properties, e.g., make the windows translucent.
3D Tiles Next is designed to enable game-like metaverse experiences in a digital representation of the entire world. At that scale, not only do we need to represent every city, every building, every room, every door knob, and each object's metadata, we also need to know the semantic meaning of the metadata so that different applications know how the metadata influences interactions. The coefficient of friction for concrete vs. grass will impact the speed of a vehicle, for example, and knowing which way a door opens impacts a crowd simulation.
3D Tiles Next provides metadata foundations to facilitate an ecosystem where domain experts can create 3D Tiles extensions to define a dictionary of domain-specific metadata and each property’s semantics. For example, earthworks construction may define semantics for materials in a stockpile, e.g., stone or clay, and for pre-computed parameters to accelerate volume and area computations at runtime.
3D Tiles Next optimizes spatial subdivision, enabling fast spatial queries, faster analysis, and greater interoperability with GIS and simulation ecosystems.
Since 3D Tiles 1.0, we have seen increasing interest in running large-scale simulations with many agents in a real-world 3D model, and also in performing analytics, such as line of sight or RF propagation, in similar environments. Not only does the spatial data structure of 3D Tiles Next optimize streaming for visualization, but it also optimizes large-scale simulation and analytics, where algorithms like ray tracing and nearest neighbor searches benefit from spatial subdivision.
3D Tiles scales to such massive models because of this spatial subdivision. A model is divided into many streamable pieces called tiles, and then a hierarchy of combined low-detail tiles are created to enable cloud-based view-dependent streaming using a technique called out-of-core hierarchical level of detail.
In 3D Tiles 1.0, spatial subdivision is explicitly defined in one or more tileset.json files defining the spatial data structure, i.e., each tile’s bounding volume, content, and child tiles, recursively to the leaf tiles.
This design is more general purpose than traditional 2D map tiling using global quadtrees. The 3D Tiles approach allows the flexibility to create a wide array of spatial data structures so that many different 3D tiling pipelines can be developed to generate optimized 3D tilesets for the given input. For example, a city model defined by extruded footprints and a model of the same city collected by LIDAR may have different subdivisions based on each model’s data density. In both cases, the generated 3D tileset can be streamed by the same 3D Tiles runtime engine as long as the subdivision follows the spatial coherence rule: the content of child tiles need to be spatially inside the parent tile’s bounding volume. This is a key enabler of the 3D Tiles ecosystem: a 3D Tiles runtime engine can stream 3D Tiles from any source.
Examples of different spatial subdivisions: KD-Tree, loose quadtree, and octree.
With 3D Tiles Next, the 3DTILES_implicit_tiling extension introduces support for well-known spatial indexes in a 3D tileset without the need to explicitly list tile bounding volumes. Enabling random access to any tile, or any number of tiles needed to display or analyze a region, this enables optimizations for:
- Large-scale simulation of agents running on a single server or across multiple compute units where efficient k-nearest neighbors and range queries are beneficial
- Analytics based on ray tracing, ray marching, and spatial queries where a uniform spatial index enables efficiency gains
- Partial updates for when just a subset of a 3D tileset needs to be updated, such as a stockpile in a construction site, or a new building in a city model
New runtime streaming algorithms such as optimized streaming for agents running through a city; for a digital twin of a construction site changing over time; or for a flight simulator. These examples all benefit from random access to tiles enabling use-case-specific algorithms separate from general-purpose top-down hierarchical level of detail.
Implicit tiling supports massive simulations such as a crowd of people in a city. Spatial indexes enable efficient queries for nearby objects.
Implicit tiling concisely represents a 3D tileset’s spatial data structure, including:
- octree or quadtree subdivision;
- the root tile’s geometric error and bounding volume, e.g., to support both global and local tilesets;
- each tile’s availability, so the runtime requests only tiles that exist, and other per-tile data stored in a binary Morton z-order curve; and
- template URIs to define the URI for random access to tiles.
Left: implicit tiling structure in tileset JSON. Right: four levels of subdivision with Morton Z-order curve overlay.
Implicit tiling, of course, also saves storage space and network bandwidth compared to explicit tiling with tileset.json. The size saving is beneficial, but the real motivation for implicit tiling is that it enables random access to any given tile in the 3D tileset and guarantees that the tileset has a uniform subdivision, both of which enable new optimizations like those listed above.
In addition to enabling new algorithms, we designed implicit tiling with interoperability in mind. We want to enable integration with 2D GIS and traditional modeling and simulation ecosystems using open specifications such as CDB, WMTS, and TMS. Implicit tiling can represent the subdivision used in these specifications and others.
3D Tiles Next also includes the 3DTILES_bounding_volume_S2 extension, which works with both implicit and explicit tiling to define a new bounding volume type for S2 cells. This is a cube-based global subdivision where each tile at the same level has roughly equal area and there is minimal distortion at the poles, especially compared to quadtree subdivision, where tiles near the poles become very thin.
The S2 Hilbert curve on the WGS84 ellipsoid.
A common geospatial use case is to group objects with the same metadata together into a layer such as a layer each for highways, points of interest, and buildings in a city. The layer can then be styled, toggled, etc., together.
With 3D Tiles Next, 3DTILES_multiple_contents can be used to efficiently support layers by defining a layer’s metadata in a tile’s group and then assigning multiple contents, e.g., glTF assets, to the group.
Using 3DTILES_content_gltf in 3D Tiles Next, a tile now directly references a .gltf or .glb file; instead of referencing a .b3dm or .i3dm with embedded glTF.
This will enable the 3D Tiles communities and glTF communities to more easily benefit from each other's work. The 3D Tiles community can use glTF tools such as validators, converters, pipelines, modeling tools, and runtime loaders; and the glTF community benefits from contributions to these tools and new glTF tools.
3D Tiles Next deepens the synergy between 3D Tiles and glTF specifications and ecosystems by:
- providing direct glTF references from 3D Tiles,
- using glTF for point clouds, and
- better leveraging glTF extensions such as texture compression.
3D Tiles 1.0 includes tile formats for Batched 3D Models (.b3dm) and Instanced 3D Models (.i3dm). These binary formats embed binary glTF, along with a header and metadata. It is not a coincidence that the bitstream layout of a .glb glTF has a striking similarity to a .b3dm, as their initial design occurred concurrently.
Left: 3D Tiles 1.0 with .b3dm to reference glTF. Right: 3D Tiles Next with direct glTF reference using EXT_mesh_features to replace the Batch Table for storing metadata.
Using 3DTILES_content_gltf, a tile now directly links to a .gltf or .glb file. If features in the tile have metadata, it is referenced by the glTF model using the EXT_mesh_features glTF extension described above, replacing the need for .b3dm.
If a tile would benefit from instancing; for example, because there are many trees in different locations at different scales, the existing EXT_mesh_gpu_instancing glTF extension is used, replacing the need for .i3dm. EXT_mesh_features can also be used with instancing to define per-instance metadata, such as tree type.
Left: 3D Tiles 1.0 with .i3dm for instancing glTF. Right: 3D Tiles Next with direct glTF reference using EXT_mesh_gpu_instancing for instancing and EXT_mesh_features to replace the Batch Table for storing metadata.
Our hope is that this move to directly reference glTF and then add glTF extensions and utilize extensions from the glTF community enables both 3D Tiles and glTF to leverage each other's work and lead to more cross-industry and cross-standards-organizations collaboration.
10 billion point cloud from the City of Montreal with ASPRS classification codes (CC-BY 4.0)
Point clouds in 3D Tiles Next now utilize glTF with EXT_meshopt_compression compression to support massive point clouds with billions of points and runtime interactions such as styling, filtering, and displaying per-point metadata.
In 3D Tiles 1.0, point clouds were stored in a .pnts binary file, a feature table for point geometry attributes, a batch table for metadata, and Draco for compression. 3D Tiles Next now leverages the glTF ecosystem for point clouds:
- Point geometry is now stored in glTF, and metadata is stored with EXT_mesh_features
- Vertex and normal compression can utilize EXT_meshopt_compression, which is efficient to encode, transmit, and decode, and stays compressed in GPU memory.
Since 3D Tiles Next no longer uses b3dm, i3dm, and pnts containers, everything that augments a glTF tile content will be a glTF extension, as shown in the examples above. This will facilitate 3D Tiles better contributing to the glTF ecosystem and potentially a glTF Geospatial Profile.
In addition, 3D Tiles Next will continue to use glTF extensions such as KTX 2.0 and PBR Next materials.
KTX 2.0 enables compressed textures for both transmission and runtime use across GPU vendors, enabling versatile optimizations that reduce memory, bandwidth, and power usage. We are especially excited to apply this to 3D geospatial content, where imagery captured via satellites and drones is creating an explosion of textures at global scale.
glTF’s PBR Next initiative has brought together the world’s experts on PBR to advance glTF’s material representation from metal-roughness and specular-gloss to support a wide array of new visual effects such as clearcoat, transmission, and volumetric efforts. For example, transmission will enhance the visual fidelity of glass on windshields and buildings by representing the thin-surface transparency in a physically plausible way that absorbs, reflects, and transmits light.
We look forward to collaborating with the community to finalize the 3D Tiles Next specification over the next several months. The community needs your input to help solidify the specification and needs your effort to start to build out the software ecosystem.
The state of the 3D Tiles Next ecosystem is currently:
-
Draft 3D Tiles and glTF extensions ready for your feedback on GitHub and prototype implementations in your software. These are:
-
Developer resources including the 3D Tiles Next Reference Card, sample data, and the upcoming 3D Tiles Next Tech blog series.
-
Open-source experimental implementation of the 3D Tiles Next spec in CesiumJS 1.87, including demos.
-
Open-source Cesium Native implementation in progress. This will then lead to 3D Tiles Next in Cesium for Unreal and the upcoming Cesium for O3DE.
-
Open-source CDB to 3D Tiles Next converter in progress on GitHub.
-
Cesium ion enhancements planned and in progress
-
3D tiling pipelines outputting 3D Tiles Next
-
3D Tiles 1.0 to 3D Tiles Next pipeline
-
Cesium World Terrain in 3D Tiles Next format and utilizing MESHOPT compression
-
Over the next several months, our focus on 3D Tiles Next is to solidify the draft specifications and continue to facilitate ecosystem growth, keeping specification design pragmatic based on implementation experience.
Working with the Open Geospatial Consortium (OGC) via their OGC Community Standard process and working with Khronos via their 3D Formats Working Group will be essential to facilitating collaboration and adoption.
You can get involved today. The 3D Tiles community needs your feedback on the draft specifications and insights from your implementation experience. Please share your thoughts in the 3d-tiles repo on GitHub.
While preparing this announcement, we reached out to folks we admire who are making significant and sustained contributions to the graphics, geospatial, open source, or open standards communities. We asked for their insights on the metaverse, interoperability, 3D Tiles, open ecosystems, or anything else they wanted to share in the spirit of this 3D Tiles Next community feedback announcement. Their comments signify the spirit of building an open metaverse. We hope that, like us, you're inspired by their vision of the future.
“It's great to see how the energy around the metaverse is helping the world really see the value of geospatial in a whole new light! Seeing the examples on social media of how developers are using the OGC 3D Tiles standard to build worlds and simulations is truly inspirational! And knowing that it's done based on open standards is reassuring as the open metaverse becomes a reality. Kudos to Cesium and team for taking this further with the 3D Tiles Next specification bringing together many of the geospatial topics that keep the OGC community busy and buzzing from semantically-rich metadata for GeoAI, to 3D visualization, to edge computing optimization!” said Nadine Alameh, CEO, Open Geospatial Consortium (OGC)
“To build a healthy metaverse, we need thriving developers and widely deployed open standards. I’m excited to see the developer ecosystem adopt the 3D Tiles format, thereby making it easier, cheaper, and faster to build GIS-based experiences and without locking into a specific platform or proprietary format.” said Matthew Ball, CEO, Epyllion and author of the The Metaverse Primer
“Making large point clouds accessible from a browser or tablet has always been a challenge. With the new 3D Tiles Next format, we can now reference glTF files directly, making way for a new streamable point cloud standard, that is both open and easy to adopt. At its most basic level, 3D Tiles Next can pinpoint a single glTF point cloud tile, on Earth. With a little more code to your existing glTF viewer, you can add the ability to stream multiple glTF tiles, coming from a massive point cloud dataset, thanks to 3D Tiles Next. When we all agree on an open standard, like 3D Tiles Next and glTF, it makes it possible to share, view and generate streamable point cloud datasets, across industries, devices and 3D engines.” said Michael Beale, Senior Software Engineer at Autodesk
“Bentley Systems was an early adopter of the 3D Tiles technology and the Cesium platform. 3D Tiles rapidly became a vital component in Bentley Systems’ iTwin ecosystem. Our ContextCapture reality modeling software generates reality meshes in 3D Tiles format from either images and/or point cloud data. Our iTwin Platform integrates 3D Tiles to allow project teams and owner-operators to create, visualize, and analyze digital twins of infrastructure assets. ” said Ray Bentley, EVP at Bentley Systems
"Nearmap is a leading location analytics company with operations in Australia, New Zealand, the USA and Canada. We operate at petabyte-scale, generating and streaming some of the largest geospatial 2D, 3D and AI datasets in the world to help our customers make faster, smarter and better decisions. Our collaboration with Cesium, and early adoption of 3D Tiles, has helped us to achieve at-scale 3D streaming and interoperability for interactive customer workflows. The introduction of 3D Tiles Next will take this to the next step, allowing streaming of rich metadata alongside our base content. This will enhance many existing applications and enable emerging ones including digital twins and the metaverse." said Dr Tom Celinski, Chief Technology Officer, Nearmap
“We at Aerometrex have been thrilled to provide enhanced 3D datasets to the Cesium team leading up to the release of the 3D Tiles Next announcement.
“We believe that 3D Tiles Next is a major step forward, not only for the geospatial industry but also as an open 3D format which will help bridge the gap between industries.
3D Tiles will be the first open 3D format to support semantic 3D classification and attribution at a pixel texture level.
“For Aerometrex, 3D Tiles Next represents an exciting new phase of potential for the 3D Tiles data specification, where metadata is being thoughtfully integrated into the data's architecture at an impressive number of levels. It will allow us to leverage our AI capability much further, including within existing and emerging game engines.
“With the new standard, we will be able to dynamically apply reflective materials to pre-identified windows and glass surfaces. We'll also be able to remove trees or street furniture across an entire city without having to extract them from the core dataset or populate 3D scenes with foliage based on the underlying classification embedded in the 3D Tiles.
“With Metaverses and Digital Twins being developed worldwide, the industry is looking for ways to build intelligence into core data and we're very excited to leverage 3D Tiles Next to deliver Metaverse-ready and intelligent 3D data to Aerometrex customers and the world.
The potential applications for our customers are vast, ranging from the embedding of AI/ML classification data into the tiles themselves, to the potential of leveraging the same enhanced data across multiple exciting platforms.” said Mark Deuter, Managing Director, Aerometrex
“Maxar is proud to have partnered with Cesium to develop 3D Tiles Next as the foundational format for the U.S. Army’s One World Terrain (OWT) program. 3D Tiles Next is the comprehensive data standard used to store and deliver geospatial terrain data to the U.S. Army’s Synthetic Training Environment (STE) Point of Need (PoN) training systems, preparing warfighters to transition from simulated environments to today’s multidomain operational theater, ” said Mr. Tony Frazier; Maxar Executive Vice President Global Field Operations
"Open standards are fundamental to building large scale virtual worlds and Digital Twins. The work Cesium is doing to advance the 3D Tiles standard is of utmost importance. We look forward to working with Cesium and the community to harmonize 3D Tiles and glTF with Universal Scene Description (USD) and NVIDIA Omniverse.” said Rev Lebaredian, VP Simulation Technology & Omniverse, NVIDIA
"The exponential progress in technology that delivers an immersive 3D experience creates a thirst for content. We look forward to seeing our customers use our FME data integration platform to transform the world's data, no matter what its original format, into 3D Tiles Next. This will allow customers to maximize the value of their data by not just inspecting it, but truly being able to experience it like never before." said Dale Lutz, Co-Founder, Safe Software
“With 3D Tiles Next building upon widely used glTF extensions, including texture compression and geometry compression, we add value to other areas of our visualization outside of 3D Tiles, while at the same time lowering the barrier to 3D Tiles Next adoption in our products.” said Ed Mackey, Advisory Software Developer, AGI, an Ansys Company
"Many development partners have asked about the limitations of how big a spatial world can be. To allow the designer's imagination to run free in the open metaverse, they need to harness tools of mass creation. With the live demonstration of 3D Tiles in the Open 3D Engine (O3DE), we have already seen entire cities unfold in real-time at 60fps. We see 3D Tiles as an excellent instrument to expedite the workflow and adoption of open source technologies like O3DE." said Royal O'Brien, General Manager of Digital Media and Games, The Linux Foundation
“Cesium continues to push the geospatial industry forward, making interactive 3D content available at scale and pioneering the open-standard building blocks that serve as a foundation for the Metaverse.
“With 3D Tiles Next, Cesium is helping deliver on our shared vision of a 3D space that promotes the interoperability of services and the creativity of the community. We’re excited to see how streaming large amounts of metadata will impact the simulation space and shape the future of digital twins.” said Marc Petit, VP and General Manager of Unreal Engine, Epic Games
“The 3D graphics and geospatial communities have long worked together to advance the mutual need to represent the real world in computer graphics. OGC and the Khronos Group have cooperated closely in the past two years advancing practical work through successive Interoperable Simulation and Gaming Sprints. Our communities will benefit from the output of this cooperation as it is reflected in the 3D Tiles Next evolution and the deeper integration of Khronos’ glTF with geospatial content." said Scott Simmons, Chief Standards Officer, Open Geospatial Consortium (OGC)
“glTF’s flexibility enables it to serve as the foundational 3D asset format in multiple vertical markets, including geospatial systems that will be essential building blocks for the metaverse. It is significant that 3D Tiles Next directly leverages advancements in the glTF roadmap to meet expanding GIS needs – maximizing the ongoing synergy between the geospatial and wider 3D communities. Khronos welcomes our productive liaison with the Open Geospatial Consortium; many such co-operations will be needed to create the constellation of standards vital to the open metaverse,” said Neil Trevett, VP Developer Ecosystems at NVIDIA, and President of the Khronos Group.
“We use 3D Tiles at Analytical Graphics for accelerating our line of sight analysis. The bounding volume hierarchies, which are core to 3D Tiles, provide a natural way to cull massive datasets down into just the parts we care about.” said Alex Wood, Advisory Software Developer, AGI, an Ansys Company