Virtual Unwrapping

ARCHIVED

This page has been archived and is no longer maintained. It describes an earlier generation of the Vesuvius Challenge pipeline — tools, data layouts, and results referenced here may have been superseded. See Open Problems for the current state of the pipeline and Prizes for what is open today.

The image data of the Herculaneum scrolls (PHerc 172) look like this:

It's a mess! The scrolls, that survived a volcanic eruption and were buried under lava and mud for centuries, are carbonized, damaged, frayed and, most importantly, still wrapped!

To read a scroll, we first need to unwrap it... virtually!

If, within the chaotic 3D image you are observing, you can isolate a contiguous region of the papyrus sheet, you can approximate it with a 2-manifold—specifically, an orientable genus-0 developable surface. This allows you to construct a bidimensional isometric parametrization¹ on the manifold, thereby defining a local basis.

The 2-manifold is typically represented as a non-watertight triangular mesh. When rendered in the coordinate frame established by its local basis, the surface appears flat, provided that the parametrization is derived from an isometric mapping from 3D to 2D. In this case, local distances, shapes, and angles are preserved, ensuring geometric fidelity in the transformation.

A triangular mesh in 3D space.

By rendering the mesh in 2D and observing the now-flattened papyrus sheet, faint letters begin to emerge — appearing as brighter regions in the figure. These weak ink traces are captured by machine learning, that can enhance their visibility and improve text recovery.

The Open Question

Virtual unwrapping is essential to read the scrolls, but far from trivial.

The open question that has been puzzling our community for more than one year, and Dr. Seales’ research for over twenty-five years, is the following:

Can we find an automated way to isolate the 2-manifold that represents the rolled scroll in the 3D images?

We were awarding $100k to anyone that could solve this problem by the end of 2024. The prize was left unclaimed.

However, both the community and the team worked on two approaches that are currently the state-of-the-art in virtual unwrapping attempts:

• Spiral Fitting
• Surface Tracer

On a very high level, both methods apply two major steps:

1. Extract from the raw data a more manageable intermediate representation (via ML or classic filters).
2. Process this representation into a surface mesh (via either top-down or bottom-up strategies).

Example of intermediate representations.

Top-down approaches solve a global optimization problem: they try to fit a complete surface representing the scroll to the intermediate representation at once.

Bottom-up approaches solve a local optimization problem, building small patches that are then stitched together coherently.

All approaches rely also on the information about the scroll umbilicus. Unfortunately, no method currently manages to fit the intended surface to the data!

Let us delve into the details of how the current state-of-the-art methods work.

Top-down methods

Spiral Fitting

(Overview)
This method leverages the idea that the scroll, in its original form, was a 2D spiral extruded in a 3D shape with approximate cylindrical symmetry. Once flattened, the papyrus sheet is nearly rectangular, with locally perpendicular fibers. The approach attempts to find the transformation that deformed this canonical spiral into the observed data.

An archimedean spiral in red, before (left) and after (right) being fit to the scroll surface predictions (in green).

Intermediate Representations

The method extracts useful intermediate representations from the raw 3D images using machine learning–based semantic segmentation, more specifically nnUNet², to isolate characteristic features that can be used in the fitting step.

Specifically, the extracted intermediate representations are:

• Surface sheets predictions: $\mathcal{S}$
• Vertical papyrus fibers predictions: $\mathcal{F}_v$
• Horizontal papyrus fibers predictions: $\mathcal{F}_h$

These representations are not free from mistakes (e.g., fake mergers or false positives), but they still make the geometry easier to handle than raw data alone.

Vertical fibers (red) and horizontal fibers (blue) highlighted in a papyrus fragment.

Geometry Processing

Given surface predictions $\mathcal{S}$, vertical papyrus fibers $\mathcal{F}_v$, horizontal papyrus fibers $\mathcal{F}_h$, and a canonical non-deformed spiral $\mathcal{S}_0$, the goal is to find a 3D diffeomorphism, i.e. a differentiable and invertible transformation from $\mathcal{S}_0$ to $\mathcal{S}$.

We can represent this transformation as an integrable, parametrized flow field $\mathbf{u}(\mathbf{x})$ in the full 3D volume. The canonical spiral $\mathcal{S}_0$ is displaced by the flow vectors $\mathbf{u}$. To find the best flow field, we minimize a global objective that aligns the deformed canonical surface to the intermediate representation while enforcing fiber directions. For example, by an abuse of notation:

$$\min_{\mathbf{u}(\mathbf{x})} \Bigg[ \int_{\Omega} d\Omega \underbrace{\big\| (\mathcal{S}_0(\mathbf{x}) + \mathbf{u}(\mathbf{x})) - \mathcal{S}(\mathbf{x}) \big\|^2}_{\text{data fidelity}} + \underbrace{\lambda R(\mathbf{u}, \mathcal{F}_v, \mathcal{F}_h)}_{\text{fiber base regularization}} \Bigg]$$

where $\mathcal{S}_0(\mathbf{x}) + \mathbf{u}(\mathbf{x})$ represents the canonical spiral displaced by the flow field, $\Omega$ is the domain in the 3D volume, and $R(\mathbf{u}, \mathcal{F}_v, \mathcal{F}_h)$ enforces that vertical/horizontal fibers in $\mathcal{S}$ match up properly when flattened.

First Automated Segmentation Prize Submission

Spiral fitting was submitted by Prof Paul Henderson for the First Automated Segmentation Prize of 2024. Although it did not meet the needed quality criteria to win, it was the most elegant solution according to the Vesuvius Challenge Team.

FASP - Spiral Fitting submission (almost zero human input)

For more information, visit this repository.

Bottom-up methods

Surface Tracer

(Overview)
Surface Tracer recognizes that, physically, the papyrus sheet is relatively continuous and should not bend abruptly from one point to the next. Instead of a global approach, it grows a surface mesh locally by “tracing” the sheet.

Intermediate Representation

As with Spiral Fitting, a semantic segmentation of the surface sheet is obtained using nnUNet². The predictions may contain fake mergers, holes, and other errors, but they focus on identifying likely surface voxels.

Slice showing Machine Learning prediction of the surface.

Geometry Processing

Starting from one or multiple small patches on the intermediate representation, the method iteratively expands these patches. In each step, a new point on the fringe is chosen to minimize a local objective. A typical local objective might be:

$$\min_{\mathbf{x}_{new}} \Big[ \underbrace{\lambda_{\text{data}} \, d(\mathbf{x}_{new}, \mathcal{S})}_{\text{data fidelity}} +\underbrace{ \lambda_{\text{dist}} \, \|\mathbf{x}_{new} - \mathbf{x}_{\text{fringe}}\| + \lambda_{\text{bend}} \, B(\mathbf{x}_{new}, M_{\text{current}})}_{\text{physical inspired term}} \Big]$$

where:

• $d(\mathbf{x}_{new}, \mathcal{S})$ measures how well the new point sits on the surface predictions,
• $\|\mathbf{x}_{new} - \mathbf{x}_{\text{fringe}}\|$ prevents large jumps,
• $B(\mathbf{x}_{new}, M_{\text{current}})$ measures how adding the new point affects the mesh local curvature (flatness).

Every n iterations, the mesh is smoothed to better approximate a developable surface. Because growing a single mesh is slow, multiple patches are grown in parallel and then merged via a consensus algorithm.

The surface tracer in action.

Finally, human annotators should manually further refine the results to reduce mistakes in the obtained mesh.

First Automated Segmentation Prize Submission

The surface tracer was submitted by Dr Hendrik Schilling and Sean Johnson for the First Automated Segmentation Prize of 2024. Although it did not meet the needed quality criteria to win, it was very close to the target!

FASP - Surface tracer submission (4 hours of human input)

More information here.

---

Conclusion

Both methods—Spiral Fitting and Surface Tracer—share at a very high level a similar pipeline of extracting an intermediate representation and then building a surface mesh via geometry processing. However, each takes a different path to deal with errors and ambiguities in the data:

• Spiral Fitting uses a global approach to align a canonical shape to the segmented surface and fiber directions.
• Surface Tracer locally expands a mesh from patches, carefully balancing data fidelity and smoothness.

Despite these advances, the problem is still unsolved. No method so far manages to perfectly fit the wanted surface to the data. The most common error is sheet switching, i.e. when the fitted surface jumps a winding, resulting in a distorted, non realistic shape. We need your help to virtually unwrap the Herculaneum scrolls.

---

References

Footnotes

1. Rabinovich et al. (2017). Scalable Locally Injective Mappings. ACM Transactions on Graphics (TOG), 36(4), 1. ↩

2. Isensee et al. (2024). nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 488-498). Cham: Springer Nature Switzerland. ↩ ↩²