3D Slide Presentation

Presented by Michael P. Zeleznik, Ph.D. at the XII International Conference on the Use of Computers in Radiation Therapy (ICCR), May 27-30, 1997, Salt Lake City, UT regarding the accuracy of 3D visualization.

IBM Visualization Data Explorer (DX) was used to generate the original 1280 x 1024 True Color images. These have been reduced in quality for this WEB presentation.

Click on the smaller images to enlarge the slide. Please note, the full sized images may take extra time to load due to their size.

3D visualization techniques are commonly employed in RTP.

These are powerful tools, and create very pretty pictures, but a false sense of security exists because the pictures look so much like the reality we THINK they accurately represent. Visualizations can be very misleading, taking extreme liberties with the data, as the visualization community is well aware. They can suppress information about the original data quality, add or lose information, and different visualizations of the same data can disagree.

Powerful tool

False sense of security?

Visualization can mislead:

Suppress orig data quality
Add information.
Loose information.
Different vis can disagree.

Does any of this matter?

Depends on impact on
overall RTP process.

To understand, we must

Define the issues.
Quantify / evaluate them.

Critical when trading ACCURACY for performance.

Whether any of this matters depends on its impact on the overall RTP process. To understand this we must first understand what these issues are, then find a way to quantify and evaluate them. This is critical when trading accuracy for performance as is often the case.

The goal of the paper is to initiate discussion on this problem and on possible solutions. A number of common RTP visualization issues are analyzed in an attempt to characterize the problem, and one method of modeling and evaluating these issues is then presented as a strawman proposal, but only the surface is scratched. This presentation will provide some examples of these issues.

PAPER:

Goal: Initiate discussion
Analyze critical vis issues.
Modeling / Evaluation method
(strawman proposal)
Only surface is scratched.

TODAY:

Present some examples.

The examples will involve dose visualizations from this simple 3 beam plan. The tumor is the cyan structure in the center. The beams are rectangular with no blocks.

This 30% dose isosurface was generated with a marching cubes algorithm. It clearly shows the shape of the beams. One problem with marching cubes is that calculation time is proportional to the number of volume elements (over 300K here). This can be slow - too slow for interactive response, such as turning a dial interactor with the surface growing/shrinking in real-time. Thus, strong incentive exists to reduce the dose volume complexity by spatially filtering it down into a smaller number of elements.

Another problem with marching cubes is that the surface triangle size is determined by the dose voxel size. This surface is much more complex than necessary (over 7600 vertices) which slows down graphics performance. Certainly, a surface simplification algorithm can now be employed, but this further adds to the calculation time, and does so for EACH surface. If, instead, the dose volume is reduced up front there are multiple wins. Not only does the calculation speed up, but the dose voxels become larger, resulting in larger surface triangles and reduced surface complexity! Further, this occurs for ALL surfaces created from that volume. Thus, there is tremendous incentive to work with a reduced dose volume.

Here the dose volume has been reduced by factor of 100. The 3 planes shown simply demonstrate the original and resulting resolutions.

This shows the surface created from the reduced volume along with the original. The surface from the reduced volume is 100X faster to calculate, and is 1/25 as complex, resulting in a 25X speedup in graphics interactivity. But how accurate is it?

Clearly, this reduced volume surface (blue solid) is very different from the original (orange mesh). What does it really represent?

This can be evaluated by dropping the reduced volume surface into the original dose volume, and mapping the dose back onto it. This shows that this "30% surface" actually represents doses varying from 10% to 50% with VERY LITTLE of it being 30% (green)! Note, the full color range now represents 10%-50% dose, shown in the colorbar.

A better approach would be to apply a surface simplification algorithm to the original dose surface, as shown here. This also results in 1/25 the complexity for a 25X graphics speedup, but we are back to the original slow calculation time, PLUS the additional time to simplify the surface (which, for accurate algorithms, can be substantial).

Clearly, this surface (blue solid) is a much better approximation to the original (orange mesh). How accurate is it?

Again mapping the original dose onto this surface, the values now range only from 15% to 35% with MOST of it actually being 30% (green). Note, the full color range now represents 15%-35% dose, shown in the colorbar.

Clearly, working with the original dose volume and then simplifying the surface is much more accurate. The penalty is calculation time. It will not be interactive.

In this demonstration, we had the luxury of evaluating these different results. However, if a particular RTP system simply provided one of these surfaces, how would you know what it really represented? Each would claim to be THE “30% dose surface.” One way of evaluating is to compare it with other dose visualizations. Two possibilities are dose-structure mappings, and dose volume histograms. We will look first at the former.

Here the dose has been mapped onto all structure surfaces. Note the beam footprints exiting the lungs, and the expected high dose at the tumor surface. This is a much simpler operation than dose surfaces, since a calculation only occurs at each structure vertex (13K) rather than at each dose point (300K). Thus, one is more likely to use the original dose volume for this type of operation. If dose surfaces and dose mappings are calculated from different dose volumes, one might see discrepancies by comparing the two.

The 97% dose band has been mapped onto the tumor surface. Red is 97% or greater, blue is below 97%. Since this dividing line represents the 97% dose transition, one would expect the 97% dose surface to intersect the tumor volume at this same boundary.

As expected, the original 97% dose surface intersection aligns very well with the dose map boundary (also at the other end which is not shown).

This is the 97% dose surface from the reduced dose volume, showing very poor alignment of the intersection at both ends. In this type of disagreement, which would you believe? One is tempted to trust the dose mapping since it is more likely to have utilized the original dose volume. But let's take a closer look at what really goes on with dose mapping...

This shows one plane of the dose volume along with the lung surface triangulation, to demonstrate the relative granularity. That fine grain, high resolution dose matrix must be mapped onto those largely spaced vertices. A substantial amount of interpolation will occur, with a corresponding loss of dose information.

That doesn't stop us from creating an arbitrary dose mapping. This 30% dose band was created by first interpolating the dose onto the coarse vertices, and then INTERPOLATING AGAIN between vertices to find the 30% location! How much error is introduced? Well, this should be a rectangular footprint, since the beam was rectangular...

A better way is to map the dose onto a high resolution surface, where the vertex spacing is comparable to the dose matrix grid. The result here is the expected rectangular footprint. However, this surface is 20X more complex, making it slower to calculate and resulting in 20X slower graphics interactivity. This surface can't just be simplified as before, or we’d be back at the previous situation.

However, if the surface is simplified based on both a geometric error AND a DATA ERROR our goal is achieved. The triangles are large except where either the surface or its data varies rapidly. The rectangular footprint is maintained while still reducing surface complexity by 6X, resulting in a 6X graphics speedup. However, calculation time is further increased (significantly) for this simplification step. This would likely preclude interactive dose mapping.

So again, we can have slow and accurate or fast and inaccurate. And again, we had the luxury of evaluating these situations. But if a particular RTP system simply provided one of these mappings, how would you know which it was? What could be said about its accuracy?

Having just compared dose surfaces with dose mappings, we now compare dose surfaces with dose volume histograms (DVH). For example, if the DVH says that 100% of a structure receives a dose of X or greater, the X dose surface would be expected to fully enclose that structure.

We just looked at
dose surface vs. dose mapping

Now look at
dose surface vs. DVH

If DVH says:

100% of struct. receives
dose of Xor greater.

Then:

X dose surface should
fully enclose the struct.

The DVH says that 100% of the tumor receives 95% or greater dose. Thus, a 95% dose surface should fully enclose the tumor.

But the tumor is not fully covered by the 95% dose surface.

The tumor is only covered by a 91% dose surface. Why the discrepancy? To understand this, a much simpler 2D example will be used.

This is a single plane 2D study, a 5x5 dose matrix with 100% dose in center 3x3 pixels and 0% dose elsewhere. The structure contour is the yellow line, exactly enclosing the 100% dose pixels.

As expected, the DVH says that 100% of the structure receives 100% dose (bar graph shown filled in orange).

Now consider dose isolines in this plane. Using pixel data as we have, where would the 0% line be? Since everything inside this line must receive 0% or greater, it must lie at the outside boundary. Similarly, the 100% line would lie at the 100% pixel boundaries. What about the 50% line? Since the pixels have only 2 values, there can be only two lines, 0% and 100%. The 50% line can't lie at the 0% boundary, since everything inside is not 50% or greater, so it must lie at the 100% boundary. In the end, EVERY line except the 0% line would lie at the 100% boundary. Clearly this is not what we expect to see. We are used to thinking in terms of a dose continuum which is created by interpolation from a set of points, not pixels.

This shows dose POINTS instead of PIXELS, where the dose is assumed to exist at the point in the center of each pixel, rather than filling the pixel.

Arbitrary dose lines can now be generated by interpolating between these points. The 100% line goes through 100% points, the 0% line through 0% points, and the others are interpolated in between. Our concept of dose lines (or surfaces in 3D) is based on points, not pixels.

This also allows the familiar representation of a dose continuum. Dose is seen as smoothly varying between the points (except for the rainbow colormap banding artifact). In this display, the area outside the perimeter dose points is assumed to be invalid since there are no points beyond those with which to interpolate.

The structure contour is shown here in yellow. The DVH, based on pixels, said it was covered by 100% dose. But the 100% dose line does not cover it. It is only completely covered by a 25% dose line, a very large discrepancy. What if the DVH were based on dose points? Unless we interpolate in creating the DVH, the result would not change. All points inside the contour are still 100%. However, since we interpolated the dose lines, the same could be done for the DVH.

Here, the original dose matrix has been refined by a factor of 3 in each dimension (subsampled by interpolation). The grid shown here no longer defines the pixel boundaries, but is instead the grid associated with the dose points. The original dose points lie at the intersections of the grid lines. The structure contour is shown in white. It now contains some non-100% dose points which will be reflected in the DVH.

This is the DVH based on the 3x interpolated data. 100% of the structure is now covered by only 45% dose, and only 60% of the structure receives 100% dose.

The dose matrix can be further refined (interpolated), here by 10X. Clearly the structure now includes more dose points with a wider range of non-100% values.

This is the DVH based on the 10x interpolated data. 100% of the structure is now covered by only 30% dose, and only 49% of the structure receives 100% dose.

This is the DVH based on 100x interpolated data, now approaching the integral of differential dose X differential volume for a dose continuum. 100% of the structure is now covered by only 28% dose, and only 45% of the structure receives 100% dose.

The 100x interpolated DVH agrees well with the dose lines. The 25% dose line just covers the structure. The DVH said 28%. The 100% dose line covers 45% of the structure (16 blocks out of 36 blocks). The DVH said 45%.

This simply shows the differences among the DVHs based on 3x, 10x, and 100x interpolation. The pixel based DVH would be a flat line out to right edge.

This example has been a single plane study. Slice thickness did not matter.

Now consider extending the 5x5 dose plane to a 5 plane study. The 3 center planes are the same as before. The two end planes have 0% dose. Thus, we have a simple 3x3x3 volume of 100% dose, with 0% dose outside. The structure is now defined by the 3 contours (in yellow) that each exactly bound the 3x3 100% dose pixels.

In any multiple plane study, each plane represents some fraction of the volume. Here, since the planes are regularly spaced, each represents 1/5 of the volume and the actual thickness would not really matter.

But the planes could be irregularly spaced, as shown here. The middle plane might represent more of the structure volume than the left plane, which might represent more than the right plane.

A common approach is to view each plane as a slab extending halfway to the adjacent plane, as shown here for our center plane. However, the structure "contour" now extends as a "tube" through this slab.

Thus, the entire structure extends through the 3 slabs as shown here. The contours are extruded through the volume.

This extrusion is essentially how the 3D structure surface is created. It is shown here with just the center plane slabbed, clearly extending into the volume between planes. This is very different from the 2D situation. In 2D, the contours completely defined the structure, and did so exactly. In 3D, the structure is assumed to extend beyond those contours in the Z dimension. In the same manner, the dose pixels now become dose voxels.

This is the DVH based on voxel data (bar graph shown filled in orange). 100% of the structure receives 100% dose, as expected since the structure boundary clearly coincides with the 100% dose voxel boundary.

But where would the 100% dose surface be? As we might expect, the voxel based DVH disagrees with the 100% dose surface which goes through the 100% dose points. Here, the structure is shown in translucent yellow, with the 100% dose surface well inside.

As in 2D, the DVH can be calculated using dose points instead of voxels. The 3D dose points are shown here. If we don't interpolate, the DVH will be the same since the structure encloses only 100% dose points. But, as in 2D, this matrix can be refined by interpolation.

Here, the 3D dose points have been refined by 3x in each dimension. Note the new planes interpolated in Z, especially those with values between the 0% end planes and the adjacent planes containing 100%. The extruded structure now includes non-100% dose points from these new planes as well as those from interpolation within the original planes (as in the 2D case), which the DVH will now reflect.

This is the DVH based on the 3x interpolation, showing the effect of including the interpolated dose points.

These are the DVHs based on 3x, 10x, and 100x interpolation of the dose points, with the latter approaching the dose-volume integral. At 100x interpolation, the DVH says 100% of the structure is covered by 18% dose, and only 30% of structure receives 100% dose.

The 18% dose surface is shown as a wire mesh, with the structure as a solid inside. Clearly, it just covers it, with only the very corners of structure penetrating the surface, agreeing with the DVH.

The 100% dose surface is shown as a red surface, with the structure as a wire mesh inside. By rotating this into 2 orthogonal views, the areas of the structure and dose surfaces were measured. The 100% dose surface did in fact fill 30% of the structure volume, as the DVH indicated.

POINT based calcs agree.

What about VOXEL calcs?

DVH
Dose Surface?

Voxel-based dose surface:

Doable with real dose distributions.

We have seen that DVHs and dose surfaces can agree if both are based on dose points with substantial interpolation. What about voxel-based calculations? Although voxel-based dose lines (or surfaces) were degenerate in our simple example, they are quite doable with real dose distributions which contain a wide range of values.

Returning to the original 3 beam plan, this is the 30% voxel-based dose surface. These surfaces are generally more complex than point surfaces, and require flat (not smooth) shading, both resulting in reduced graphics interactivity. Further, they can not be simplified without losing the clean voxel edges. More importantly, what does this really represent? A dose voxel is included if its value is 30% or greater. For example, if two adjacent voxels are 29% and 60%, the 60% one would be included. Thus, although this surface has been colored to reflect 30% dose, it actually represents dose values of 30% OR GREATER. This is very different from a point-based dose surface.

To see what this 30% dose voxel surface actually represents, each voxel has been colored with its dose value. This surface contains values ranging from 30% to 90%! Very little is 30%, which would be the light blue color. Voxel surfaces would have to be color mapped in this manner in order to not be misleading.

This 65% dose voxel surface (blown up) actually ranges from 65% to 95%, with large discontinuities. However, to what spatial error do these dose errors correspond? The maximum spatial error at any point should not be greater than the size of a voxel. In XY this is usually small, but can be substantial in Z (if planes are far apart). This is where one often sees such discrepancies, usually at the end caps of structures.

This is the 65% dose point surface together with the voxel surface (as a wire frame), turned on its side for visibility. As expected, the spatial error is less than the voxel dimensions. The point surface will lie between the centers of the outermost voxels and the voxels just outside of those.

This is just a blow-up of the upper left corner of the previous image, with faceted shading to show the detail. The spatial discrepancy is always less then a voxel dimension.

This view shows the spatial error in Z, normally much more pronounced. On the left, the surfaces align nearly exactly, but on the right they disagree by 1/2 the Z thickness of the voxel. Remember, the whole reason we got into this DVH discussion was the discrepancy between the 95% surface and the DVH; the tumor stuck out in the end cap (Z).

This was the situation we started with.

As you might expect, a 95% dose voxel surface DOES cover the structure. Had we used a voxel surface originally, there would have been no disagreement. Thus, voxel-based DVHs and dose surfaces can agree as well.

We have seen that DVHs and dose surfaces can agree, as long as both are based on voxels or points. But there is more to the story. Point based dose surfaces are created with essentially infinite resolution. That is, dose points are interpolated to the floating point accuracy of the machine, and those coordinates become the dose surface coordinates. With DVHs, dose points are interpolated to a finite grid. Although a 100x interpolation approached the dose-volume integral, it requires 100x in each dimension, or 100 cubed for 3D. This increases calculation time by a factor of 1 million! Even a 4x interpolation results in a 64x slowdown, turning 1 second into 1 minute, or .1 seconds into 6 seconds. As such, one is likely to use a much lower refinement for the DVH, and thus still see disagreement with the (relatively) infinite interpolation of the dose surfaces.

Similar calcs (voxel or point) can agree.

But, dose surface and DVH NOT at same resolution

Dose surface interpolation:

Floating point accuracy.

DVH interpretation:

100x => 1 Million X calcs!
4x => 64 X calcs.

Thus, can still agree

The issue is NOT whether
Voxel or Point calcs are better.

Just that they can DISAGREE.

We might want to use:

Voxel for DVH
Point for dose surface.

The issue here is not whether voxel or point calculations are better. It is simply that they can, and often will disagree.

This concludes the examples, which demonstrated three RTP visualization issues: dose surfaces, dose mappings, and DVHs. The paper discusses others, such as volume data regularization, color mapping, handling of missing data, and smooth vs. flat shading.

These were simple examples of three issues:

Dose surfaces
Dose mappings
Dose volume histograms

Paper discusses others:

Volume data
Color maps
Missing data
Shading

These visualizations could disagree due to:

Raw data reduction
Simplified geometries.
Voxel vs. Point data

How can we know WHICH if ANY, is correct?

These visualizations were seen to disagree due to raw data reduction (as with dose surfaces), use of simplified geometries (as with dose mapping), and voxel vs. point data (as with dose surfaces vs. DVHs). In practice, in the presence of such disagreement, how can we know which (if any) is the more accurate representation of the true data? Unlike this presentation, we generally will have no idea of what is going on behind the scene.

The point of this talk was not the details of the examples, but rather to demonstrate that 1) there are many ways to create a given visualization, 2) it can be very difficult to know what it really means, and as such, 3) it can be difficult to know its impact on the RTP process. To understand this, we must first define these issues, then find a way to quantify and evaluate them.

Many ways to create a visualization

Can be difficult to know what a visualization means.
How can we know its impact on the RTP process

To understand, we must:

Define these issues.
Quantify and evaluate them.

A PROPOSED MODEL

Object-oriented approach

Data types.
Transformations,
Quality Metrics.

Quality Metrics (QM).

Error information,
non-information that is added.
information that is lost,
intrinsic value, ...

Propagate QM, data to observer.

In the paper, one approach to modeling and evaluating these issues is presented as a strawman proposal. The visualization process is modeled in an object-oriented manner, where data undergoes transformations in visualization pipelines. The key element of this model is the Quality Metric (QM) associated with each data type. The QM captures the intrinsic quality of the data, including things like error information (e.g., the dose range represented by a dose surface, or the positional error resulting from surface simplification), non-information that is added (e.g., smooth shading a surface with very coarse triangulation), information that is lost (e.g., reducing the dose volume, or losses due to the interaction of the colormap with human visual perception), or the intrinsic value of a visualization (e.g., displaying dose as raw numbers is certainly accurate, but worthless as a visualization tool). QMs are modified by each transformation, and thus are propagated from the raw data to the final observer, providing a quantitative handle on the meaning of the resulting visualization. Only the surface has been scratched, but hopefully this provides a framework for future discussion.

So, in answer to “What Does it Mean?” Rather than wondering, perhaps we should DEFINE what it means, clearly specifying what these different visualizations are expected to convey, and to what level of accuracy, with RTP systems then conforming to those specifications.