Primitive Processing in Open GL

This chapter is from the book 

Geometry Shaders

The geometry shader is unique in contrast to the other shader types in that it processes a whole primitive (triangle, line, or point) at once and can actually change the amount of data in the OpenGL pipeline programmatically. A vertex shader processes one vertex at a time; it cannot access any other vertex’s information and is strictly one-in, one-out. That is, it cannot generate new vertices, and it cannot stop the vertex from being processed further by OpenGL. The tessellation shaders operate on patches and can set tessellation factors, but have little further control over how patches are tessellated, and cannot produce disjoint primitives. Likewise, the fragment shader processes a single fragment at a time, cannot access any data owned by another fragment, cannot create new fragments, and can only destroy fragments by discarding them. On the other hand, a geometry shader has access to all of the vertices in a primitive (up to six with the primitive modes GL_TRIANGLES_ADJACENCY and GL_TRIANGLE_STRIP_ADJACENCY), can change the type of a primitive, and can even create and destroy primitives.

Geometry shaders are an optional part of the OpenGL pipeline. When no geometry shader is present, the outputs from the vertex or tessellation evaluation shader are interpolated across the primitive being rendered and are fed directly to the fragment shader. When a geometry shader is present, however, the outputs of the vertex or tessellation evaluation shader become the inputs to the geometry shader, and the outputs of the geometry shader are what are interpolated and fed to the fragment shader. The geometry shader can further process the output of the vertex or tessellation evaluation shader, and if it is generating new primitives (this is called amplification), it can apply different transformations to each primitive as it creates them.

The Pass-Through Geometry Shader

As explained back in Chapter 3, “Following the Pipeline,” the simplest geometry shader that allows you to render anything is the pass-through shader, which is shown in Listing 8.16.

Listing 8.16. Source code for a simple geometry shader

#version 430 core

layout (triangles) in;
layout (triangle_strip) out;
layout (max_vertices = 3) out;

void main(void)
{
    int i;

    for (i = 0; i < gl_in.length(); i++)
    {
        gl_Position = gl_in[i].gl_Position;
        EmitVertex();
    }
    EndPrimitive();
}

This is a very simple pass-through geometry shader, which sends its input to its output without modifying it. It looks similar to a vertex shader, but there are a few extra differences to cover. Going over the shader a few lines at a time makes everything clear. The first few lines simply set up the version number (430) of the shader just like in any other shader. The next couple of lines are the first geometry shader-specific parts. They are shown again in Listing 8.17.

Listing 8.17. Geometry shader layout qualifiers

#version 430 core

layout (triangles) in;
layout (triangle_strip) out;
layout (max_vertices = 3) out;

These set the input and output primitive modes using a layout qualifier. In this particular shader we’re using triangles for the input and triangle_strip for the output. Other primitive types, along with the layout qualifier, are covered later. For the geometry shader’s output, not only do we specify the primitive type, but the maximum number of vertices expected to be generated by the shader (through the max_vertices qualifier). This shader produces individual triangles (generated as very short triangle strips), so we specified 3 here.

Next is our main() function, which is again similar to what might be seen in a vertex or fragment shader. The shader contains a loop, and the loop runs a number of times determined by the length of the built-in array, gl_in. This is another geometry shader-specific variable. Because the geometry shader has access to all of the vertices of the input primitive, the input has to be declared as an array. All of the built-in variables that are written by the vertex shader (such as gl_Position) are placed into a structure, and an array of these structures is presented to the geometry shader in a variable called gl_in.

The length of the gl_in[] array is determined by the input primitive mode, and because in this particular shader, triangles are the input primitive mode, the size of gl_in[] is three. The inner loop is given again in Listing 8.18.

Listing 8.18. Iterating over the elements of gl_in[]

for (i = 0; i < gl_in.length(); i++)
{
    gl_Position = gl_in[i].gl_Position;
    EmitVertex();
}

Inside our loop, we’re generating vertices by simply copying the elements of gl_in[] to the geometry shader’s output. A geometry shader’s outputs are similar to the vertex shader’s outputs. Here, we’re writing to gl_Position, just as we would in a vertex shader. When we’re done setting up all of the new vertex’s attributes, we call EmitVertex(). This is a built-in function, specific to geometry shaders that tells the shader that we’re done with our work for this vertex and that it should store all that information away and prepare to start setting up the next vertex.

Finally, after the loop has finished executing, there’s a call to another special, geometry shader-only function, EndPrimitive(). EndPrimitive() tells the shader that we’re done producing vertices for the current primitive and to move on to the next one. We specified triangle_strip as the output for our shader, and so if we continue to call EmitVertex() more than three times, OpenGL continues adding triangles to the triangle strip. If we need our geometry shader to generate separate, individual triangles or multiple, unconnected triangle strips (remember, geometry shaders can create new or amplify geometry), we could call EndPrimitive() between each one to mark their boundaries. If you don’t call EndPrimitive() somewhere in your shader, the primitive is automatically ended when the shader ends.

Using Geometry Shaders in an Application

Geometry shaders, like the other shader types, are created by calling the glCreateShader() function and using GL_GEOMETRY_SHADER as the shader type, as follows:

glCreateShader(GL_GEOMETRY_SHADER);

Once the shader has been created, it is used like any other shader object. You give OpenGL your shader source code by calling glShaderSource(), compile the shader using the glCompileShader() function, and attach it to a program object by calling the glAttachShader() function. Then the program is linked as normal using the glLinkProgram() function. Now that you have a program object with a geometry shader linked into it, when you draw geometry using a function like glDrawArrays(), the vertex shader will run once per vertex, the geometry shader will run once per primitive (point, line, or triangle), and the fragment will run once per fragment. The primitives received by a geometry shader must match what it is expecting based in its own input primitive mode. When tessellation is not active, the primitive mode you use in your drawing commands must match the input primitive mode of the geometry shader. For example, if the geometry shader’s input primitive mode is points, then you may only use GL_POINTS when you call glDrawArrays(). If the geometry shader’s input primitive mode is triangles, then you may use GL_TRIANGLES, GL_TRIANGLE_STRIP, or GL_TRIANGLE_FAN in your glDrawArrays() call. A complete list of the geometry shader input primitive modes and the allowed geometry types is given in Table 8.1.

Table 8.1. Allowed Draw Modes for Geometry Shader Input Modes

Geometry Shader Input Mode	Allowed Draw Modes
points	GL_POINTS
lines	GL_LINES, GL_LINE_LOOP, GL_LINE_STRIP
triangles	GL_TRIANGLES, GL_TRIANGLE_FAN, GL_TRIANGLE_STRIP
lines_adjacency	GL_LINES_ADJACENCY
triangles_adjacency	GL_TRIANGLES_ADJACENCY

When tessellation is active, the mode you use in your drawing commands should always be GL_PATCHES, and OpenGL will convert the patches into points, lines, or triangles during the tessellation process. In this case, the input primitive mode of the geometry shader should match the tessellation primitive mode. The input primitive type is specified in the body of the geometry shader using a layout qualifier. The general form of the input layout qualifier is

layout (primitive_type) in;

This specifies that primitive_type is the input primitive type that the geometry shader is expected to handle, and primitive_type must be one of the supported primitive modes: points, lines, triangles, lines_adjacency, or triangles_adjacency. The geometry shader runs once per primitive. This means that it’ll run once per point for GL_POINTS; once per line for GL_LINES, GL_LINE_STRIP, and GL_LINE_LOOP; and once per triangle for GL_TRIANGLES, GL_TRIANGLE_STRIP, and GL_TRIANGLE_FAN. The inputs to the geometry shader are presented in arrays containing all of the vertices making up the input primitive. The predefined inputs are stored in a built-in array called gl_in[], which is an array of structures defined in Listing 8.19.

Listing 8.19. The definition of gl_in[]

in gl_PerVertex
{
    vec4  gl_Position;
    float gl_PointSize;
    float gl_ClipDistance[];
} gl_in[];

The members of this structure are the built-in variables that are written in the vertex shader: gl_Position, gl_PointSize, and gl_ClipDistance[]. You should recognize this structure from its declaration as an output block in the vertex shader described earlier in this chapter. These variables appear as global variables in the vertex shader because the block doesn’t have an instance name there, but their values end up in the gl_in[] array of block instances when they appear in the geometry shader. Other variables written by the vertex shader also become arrays in the geometry shader. In the case of individual varyings, outputs in the vertex shader are declared as normal, and the inputs to the geometry shader have a similar declaration, except that they are arrays. Consider a vertex shader that defines outputs as

out vec4 color;
out vec3 normal;

The corresponding input to the geometry shader would be

in vec4 color[];
in vec3 normal[];

Notice that both the color and normal varyings have become arrays in the geometry shader. If you have a large amount of data to pass from the vertex to the geometry shader, it can be convenient to wrap per-vertex information passed from the vertex shader to the geometry shader into an interface block. In this case, your vertex shader will have a definition like this:

out VertexData
{
    vec4 color;
    vec3 normal;
} vertex;

And the corresponding input to the geometry shader would look like this:

in VertexData
{
    vec4 color;
    vec3 normal;
    // More per-vertex attributes can be inserted here
} vertex[];

With this declaration, you’ll be able to access the per-vertex data in the geometry shader using vertex[n].color and so on. The length of the input arrays in the geometry shader depends on the type of primitives that it will process. For example, points are formed from a single vertex, and so the arrays will only contain a single element, whereas triangles are formed from three vertices, and so the arrays will be three elements long. If you’re writing a geometry shader that’s designed specifically to process a particular primitive type, you can explicitly size your input arrays, which provides a small amount of additional compile-time error checking. Otherwise, you can let your arrays be automatically sized by the input primitive type layout qualifier. A complete mapping of the input primitive modes and the resulting size of the input arrays is shown in Table 8.2.

Table 8.2. Sizes of Input Arrays to Geometry Shaders

Input Primitive Type	Size of Input Arrays
points	1
lines	2
triangles	3
lines_adjacency	4
triangles_adjacency	6

You also need to specify the primitive type that will be generated by the geometry shader. Again, this is determined using a layout qualifier, like so:

layout (primitive_type) out;

This is similar to the input primitive type layout qualifier, the only difference being that you are declaring the output of the shader using the out keyword. The allowable output primitive types from the geometry shader are points, line_strip, and triangle_strip. Notice that geometry shaders only support outputting the strip primitive types (not counting points—obviously, there is no such thing as a point strip).

There is one final layout qualifier that must be used to configure the geometry shader. Because a geometry shader is capable of producing a variable amount of data per vertex, OpenGL must be told how much space to allocate for all that data by specifying the maximum number of vertices that the geometry shader is expected to produce. To do this, use the following layout qualifier:

layout (max_vertices = n) out;

This sets the maximum number of vertices that the geometry shader may produce to n. Because OpenGL may allocate buffer space to store intermediate results for each vertex, this should be the smallest number possible that still allows your application to run correctly. For example, if you are planning to take points and produce one line at a time, then you can safely set this to two. This gives the shader hardware the best opportunity to run fast. If you are going to heavily tessellate the incoming geometry, you might want to set this to a much higher number, although this may cost you some performance. The upper limit on the number of vertices that a geometry shader can produce depends on your OpenGL implementation. It is guaranteed to be at least 256, but the absolute maximum can be found by calling glGetIntegerv() with the GL_MAX_GEOMETRY_OUTPUT_VERTICES parameter.

You can also declare more than one layout qualifier with a single statement by separating them with a comma, like so:

layout (triangle_strip, max_vertices = n) out;

With these layout qualifiers, a boilerplate #version declaration, and an empty main() function, you should be able to produce a geometry shader that compiles and links but does absolutely nothing. In fact, it will discard any geometry you send it, and nothing will be drawn by your application. We need to introduce two important functions: EmitVertex() and EndPrimitive(). If you don’t call these, nothing will be drawn.

EmitVertex() tells the geometry shader that you’ve finished filling in all of the information for this vertex. Setting up the vertex works much like the vertex shader. You need to write into the built-in variable gl_Position. This sets the clip-space coordinates of the vertex that is produced by the geometry shader, just like in a vertex shader. Any other attributes that you want to pass from the geometry shader to the fragment shader can be declared in an interface block or as global variables in the geometry shader. Whenever you call EmitVertex, the geometry shader stores the values currently in all of its output variables and uses them to generate a new vertex. You can call EmitVertex() as many times as you like in a geometry shader, until you reach the limit you specified in your max_vertices layout qualifier. Each time, you put new values into your output variables to generate a new vertex.

An important thing to note about EmitVertex() is that it makes the values of any of your output variables (such as gl_Position) undefined. So, for example, if you want to emit a triangle with a single color, you need to write that color with every one of your vertices; otherwise, you will end up with undefined results.

EmitPrimitive() indicates that you have finished appending vertices to the end of the primitive. Don’t forget, geometry shaders only support the strip primitive types (line_strip and triangle_strip). If your output primitive type is triangle_strip and you call EmitVertex() more than three times, the geometry shader will produce multiple triangles in a strip. Likewise, if your output primitive type is line_strip and you call EmitVertex() more than twice, you’ll get multiple lines. In the geometry shader, EndPrimitive() refers to the strip. This means that if you want to draw individual lines or triangles, you have to call EndPrimitive() after every two or three vertices. You can also draw multiple strips by calling EmitVertex() many times between multiple calls to EndPrimitive().

One final thing to note about calling EmitVertex() and EndPrimitive() in the geometry shader is that if you haven’t produced enough vertices to produce a single primitive (e.g., you’re generating triangle_strip outputs and you call EndPrimitive() after two vertices), nothing is produced for that primitive, and the vertices you’ve already produced are simply thrown away.

Discarding Geometry in the Geometry Shader

The geometry shader in your program runs once per primitive. What you do with that primitive is entirely up to you. The two functions EmitVertex() and EndPrimitive() allow you to programmatically append new vertices to your triangle or line strip and to start new strips. You can call them as many times as you want (until you reach the maximum defined by your implementation). You’re also allowed to not call them at all. This allows you to clip geometry away and discard primitives. If your geometry shader runs and you never call EmitVertex() for that particular primitive, nothing will be drawn. To illustrate this, we can implement a custom backface culling routine that culls geometry as if it were viewed from an arbitrary point in space. This is implemented in the gsculling example.

First, we set up our shader version and declare our geometry shader to accept triangles and to produce triangle strips. Backface culling doesn’t really make a lot of sense for lines or points. We also define a uniform that will hold our custom viewpoint in world space. This is shown in Listing 8.20.

Listing 8.20. Configuring the custom culling geometry shader

#version 330

// Input is triangles, output is triangle strip. Because we're going
// to do a 1 in 1 out shader producing a single triangle output for
// each one input, max_vertices can be 3 here.
layout (triangles) in;
layout (triangle_strip, max_vertices=3) out;

// Uniform variables that will hold our custom viewpoint and
// model-view matrix
uniform vec3 viewpoint;
uniform mav4 mv_matrix;

Now inside our main() function, we need to find the face normal for the triangle. This is simply the cross products of any two vectors in the plane of the triangle—we can use the triangle edges for this. Listing 8.21 shows how this is done.

Listing 8.21. Finding a face normal in a geometry shader

// Calculate two vectors in the plane of the input triangle
vec3 ab = gl_in[1].gl_Position.xyz - gl_in[0].gl_Position.xyz;
vec3 ac = gl_in[2].gl_Position.xyz - gl_in[0].gl_Position.xyz;
vec3 normal = normalize(cross(ab, ac));

Now that we have the normal, we can determine whether it faces toward or away from our user-defined viewpoint. To do this, we need to transform the normal into the same coordinate space as the viewpoint, which is world space. Assuming we have the model-view matrix in a uniform, simply multiply the normal by this matrix. To be more accurate, we should multiply the vector by the inverse of the transpose of the upper-left 3 × 3 submatrix of the model-view matrix. This is known as the normal matrix, and you’re free to implement this and put it in its own uniform if you like. However, if your model-view matrix only contains translation, uniform scale (no shear), and rotation, you can use it directly. Don’t forget, the normal is a three-element vector, and the model-view matrix is a 4 × 4 matrix. We need to extend the normal to a four-element vector before we can multiply the two. We can then take the dot product of the resulting vector with the vector from the viewpoint to any point on the triangle.

If the sign of the dot product is negative, that means that the normal is facing away from the viewer and the triangle should be culled. If it is positive, the triangle’s normal is pointing toward the viewer, and we should pass the triangle on. The code to transform the face normal, perform the dot product, and test the sign of the result is shown in Listing 8.22.

Listing 8.22. Conditionally emitting geometry in a geometry shader

// Calculate the transformed face normal and the view direction vector
vec3 transformed_normal = (vec4(normal, 0.0) * mv_matrix).xyz;
vec3 vt = normalize(gl_in[0].gl_Position.xyz - viewpoint);

// Take the dot product of the normal with the view direction
float d = dot(vt, normal);

// Emit a primitive only if the sign of the dot product is positive
if (d > 0.0)
{
    for (int i = 0; i < 3; i++)
    {
        gl_Position = gl_in[i].gl_Position;
        EmitVertex();
    }
    EndPrimitive();
}

In Listing 8.22, if the dot product is positive, we copy the input vertices to the output of the geometry shader and call EmitVertex() for each one. If the dot product is negative, we simply don’t do anything at all. This results in the incoming triangle being discarded altogether and nothing being drawn.

In this particular example, we are generating at most one triangle output for each triangle input to the geometry shader. Although the output of the geometry shader is a triangle strip, our strips only contain a single triangle. Therefore, there doesn’t strictly need to be a call to EndPrimitive(). We just leave it there for completeness.

Figure 8.16 shows a the result of this shader.

In Figure 8.16, the virtual viewer has been moved to different positions. As you can see, different parts of the model have been culled away by the geometry shader. It’s not expected that this example is particularly useful, but it does demonstrate the ability for a geometry shader to perform geometry culling based on application-defined criteria.

Modifying Geometry in the Geometry Shader

The previous example either discarded geometry or passed it through unmodified. It is also possible to modify vertices as they pass through the geometry shader to create new, derived shapes. Even though your geometry shader is passing vertices on one-to-one (i.e., no amplification or culling is taking place), this still allows you to do things that would otherwise not be possible with a vertex shader alone. If the input geometry is in the form of triangle strips or fans, for example, the resulting geometry will have shared vertices and shared edges. Using the vertex shader to move shared vertices will move all of the triangles that share that vertex. It is not possible, then, to separate two triangles that share an edge in the original geometry using the vertex shader alone. However, this is trivial using the geometry shader.

Consider a geometry shader that accepts triangles and produces triangle_strip as output. The input to a geometry shader that accepts triangles is individual triangles, regardless of whether they originated from a glDrawArrays() or a glDrawElements() function call, or whether the primitive type was GL_TRIANGLES, GL_TRIANGLE_STRIP, or GL_TRIANGLE_FAN. Unless the geometry shader outputs more than three vertices, the result is independent, unconnected triangles.

In this next example, we “explode” a model by pushing all of the triangles out along their face normals. It doesn’t matter whether the original model is drawn with individual triangles or with triangle strips or fans. As with the previous example, the input is triangles, the output is triangle_strip, and the maximum number of vertices produced by the geometry shader is three because we’re not amplifying or decimating geometry. The setup code for this is shown in Listing 8.23.

Listing 8.23. Setting up the “explode” geometry shader

#version 330

// Input is triangles, output is triangle strip. Because we're going to do a
// 1 in 1 out shader producing a single triangle output for each one input,
// max_vertices can be 3 here.
layout (triangles) in;
layout (triangle_strip, max_vertices=3) out;

To project the triangle outward, we need to calculate the face normal of each triangle. Again, to do this we can take the cross product of two vectors in the plane of the triangle—two edges of the triangle. For this, we can reuse the code from Listing 8.21. Now that we have the triangle’s face normal, we can project vertices along that normal by an application-controlled amount. That amount can be stored in a uniform (we call it explode_factor) and updated by the application. This simple code is shown in Listing 8.24.

Listing 8.24. Pushing a face out along its normal

for (int i = 0; i < 3; i++)
{
    gl_Position = gl_in[i].gl_Position +
                  vec4(explode_factor * normal, 0.0);
}

The result of running this geometry shader on a model is shown in Figure 8.17. The model has been deconstructed, and the individual triangles have become visible.

Generating Geometry in the Geometry Shader

Just as you are not required to call EmitVertex() or EndPrimitive() at all if you don’t want to produce any output from the geometry shader, it is also possible to call EmitVertex() and EndPrimitive() as many times as you need to produce new geometry. That is, until you reach the maximum number of output vertices that you declared at the start of your geometry shader. This functionality can be used for things like making multiple copies of the input or breaking the input into smaller pieces. This is the subject of the next example, which is the gstessellate sample in the book’s accompanying source code. The input to our shader is a tetrahedron centered around the origin. Each face of the tetrahedron is made from a single triangle. We tessellate incoming triangles by producing new vertices halfway along each edge and then moving all of the resulting vertices so that they are variable distances from the origin. This transforms our tetrahedron into a spiked shape.

Because the geometry shader operates in object space (remember, the tetrahedron’s vertices are centered around the origin), we need to do no coordinate transforms in the vertex shader and, instead, do the transforms in the geometry shader after we’ve generated the new vertices. To do this, we need a simple, pass-through vertex shader. Listing 8.25 shows a simple pass-through vertex shader.

Listing 8.25. Pass-through vertex shader

#version 330

in vec4 position;

void main(void)
{
    gl_Position = position;
}

This shader only passes the vertex position to the geometry shader. If you have other attributes associated with the vertices such as texture coordinates or normals, you need to pass them through the vertex shader to the geometry shader as well.

As in the previous example, we accept triangles as input to the geometry shader and produce a triangle strip. We break the strip after every triangle so that we can produce separate, independent triangles. In this example, we produce four output triangles for every input triangle. We need to declare our maximum output vertex count as 12—four triangles times three vertices. We also need to declare a uniform matrix to store the model-view transformation matrix in the geometry shader because we do that transform after generating vertices. Listing 8.26 shows this code.

Listing 8.26. Setting up the “tessellator” geometry shader

#version 430 core

layout (triangles) in;
layout (triangle_strip, max_vertices = 12) out;

// A uniform to store the model-view-projection matrix
uniform mat4 mvp;

First, let’s copy the incoming vertex coordinates into a local variable. Then, given the original, incoming vertices, we find the midpoint of each edge by taking their average. In this case, however, rather than simply dividing by two, we multiply by a scale factor, which will allow us to alter the spikiness of the resulting object. Code to do this is shown in Listing 8.27.

Listing 8.27. Generating new vertices in a geometry shader

// Copy the incoming vertex positions into some local variables
vec3 a = gl_in[0].gl_Position.xyz;
vec3 b = gl_in[1].gl_Position.xyz;
vec3 c = gl_in[2].gl_Position.xyz;

// Find a scaled version of their midpoints
vec3 d = (a + b) * stretch;
vec3 e = (b + c) * stretch;
vec3 f = (c + a) * stretch;

// Now, scale the original vertices by an inverse of the midpoint
// scale
a *= (2.0 - stretch);
b *= (2.0 - stretch);
c *= (2.0 - stretch);

Because we are going to generate several triangles using almost identical code, we can put that code into a function (shown in Listing 8.28) and call it from our main tessellation function.

Listing 8.28. Emitting a single triangle from a geometry shader

void make_face(vec3 a, vec3 b, vec3 c)
{
    vec3 face_normal = normalize(cross(c - a, c - b));
    vec4 face_color = vec4(1.0, 0.2, 0.4, 1.0) * (mat3(mvMatrix) * face_normal
    gl_Position = mvpMatrix * vec4(a, 1.0);
    color = face_color;
    EmitVertex();

    gl_Position = mvpMatrix * vec4(b, 1.0);
    color = face_color;
    EmitVertex();

    gl_Position = mvpMatrix * vec4(c, 1.0);
    color = face_color;
    EmitVertex();

    EndPrimitive();
}

Notice that the make_face function calculates a face color based on the face’s normal in addition to emitting the positions of its vertices. Now, we simply call make_face four times from our main function, which is shown in Listing 8.29.

Listing 8.29. Using a function to produce faces in a geometry shader

make_face(a, d, f);
make_face(d, b, e);
make_face(e, c, f);
make_face(d, e, f);

Figure 8.18 shows the result of our simple geometry shader-based tessellation program.

Note that using the geometry shader for heavy tessellation may not produce the most optimal performance. If something more complex than that shown in this example is desired, it’s best to use the hardware tessellation functions of OpenGL. However, if simple amplification of between two and four output primitives for each input primitive is desired, the geometry shader is probably the way to go.

Changing the Primitive Type in the Geometry Shader

So far, all of the geometry shader examples we’ve gone through have taken triangles as input and produced triangle strips as output. This doesn’t change the geometry type. However, geometry shaders can input and output different types of geometry. For example, you can transform points into triangles or triangles into points. In the normalviewer example, which we’ll describe next, we’re going to change the geometry type from triangles to lines. For each vertex input to the shader, we take the vertex normal and represent it as a line. We also take the face normal and represent that as another line. This allows us to visualize the model’s normals—both at each vertex and for each face. Note, though, that if you want to draw the normals on top of the original model, you need to draw everything twice—once with the geometry shader to visualize the normals and once without the geometry shader to show the model. You can’t output a mix of two different primitives from a single geometry shader.

For our geometry shader, in addition to the members of the gl_in structure, we need the per-vertex normal, and that will have to be passed through the vertex shader. An updated version of the pass-through vertex shader from Listing 8.25 is given in Listing 8.30.

Listing 8.30. A pass-through vertex shader that includes normals

#version 330

in vec4 position;
in vec3 normal;

out Vertex
{
    vec3 normal;
}  vertex;

void main(void)
{
    gl_Position = position;
    vertex.normal = normal;
}

This passes the position attribute straight through to the gl_Position built-in variable and places the normal into an output block.

The setup code for the geometry shader is shown in Listing 8.31. In this example, we accept triangles and produce line strips, each of a single line. Because we output a separate line for each normal we visualize, we produce two vertices for each vertex consumed, plus two more for the face normal. Therefore, the maximum number of vertices that we output per input triangle is eight. To match the Vertex output block that we declared in the vertex shader, we also need to declare a corresponding input interface block in the geometry shader. As we’re going to do the object-space-to-world-space transformation in the geometry shader, we declare a mat4 uniform called mvp to represent the model-view-projection matrix. This is necessary so that we can keep the vertex’s position in the same coordinate system as its normal until we produce the new vertices representing the line.

Listing 8.31. Setting up the “normal visualizer” geometry shader

#version 330

layout (triangles) in;
layout (line_strip) out;
layout (max_vertices = 8) out;

in Vertex
{
    vec3 normal;
}  vertex[];

// Uniform to hold the model-view-projection matrix
uniform mat4 mvp;

// Uniform to store the length of the visualized normals
uniform float normal_length;

Each input vertex is transformed into its final position and emitted from the geometry shader, and then a second vertex is produced by displacing the input vertex along its normal and transforming that into its final position as well. This makes the length of all of our normals one but allows any scaling encoded in our model-view-projection matrix to be applied to them along with the model. We multiply the normals by the application-supplied uniform normal_length, allowing them to be scaled to match the model. Our inner loop is shown in Listing 8.32.

Listing 8.32. Producing lines from normals in the geometry shader

gl_Position = mvp * gl_in[0].gl_Position;
gs_out.normal = gs_in[0].normal;
gs_out.color = gs_in[0].color;
EmitVertex();

gl_Position = mvp * (gl_in[0].gl_Position +
                     vec4(gs_in[0].normal * normal_length, 0.0));
gs_out.normal = gs_in[0].normal;
gs_out.color = gs_in[0].color;
EmitVertex();
EndPrimitive();

This generates a short line segment at each vertex pointing in the direction of the normal. Now, we need to produce the face normal. To do this, we need to pick a suitable place from which to draw the normal, and we need to calculate the face normal itself in the geometry shader along which to draw the line.

As in the earlier example given in Listing 8.33, we use a cross product of two of the triangle’s edges to find the face normal. To pick a starting point for the line, we choose the centroid of the triangle, which is simply the average of the coordinates of the input vertices. Listing 8.33 shows the shader code.

Listing 8.33. Drawing a face normal in the geometry shader

vec3 ab = gl_in[1].gl_Position.xyz - gl_in[0].gl_Position.xyz;
vec3 ac = gl_in[2].gl_Position.xyz - gl_in[0].gl_Position.xyz;
vec3 face_normal = normalize(cross(ab, ac));

vec4 tri_centroid = (gl_in[0].gl_Position +
                     gl_in[1].gl_Position +
                     gl_in[2].gl_Position) / 3.0;

gl_Position = mvp * tri_centroid;
gs_out.normal = gs_in[0].normal;
gs_out.color = gs_in[0].color;
EmitVertex();

gl_Position = mvp * (tri_centroid +
                     vec4(face_normal * normal_length, 0.0));
gs_out.normal = gs_in[0].normal;
gs_out.color = gs_in[0].color;
EmitVertex();
EndPrimitive();

Now when we render a model, we get the image shown in Figure 8.19.

Multiple Streams of Storage

When only a vertex shader is present, there is a simple one-in, one-out relationship between the vertices coming into the shader and the vertices stored in the transform feedback buffer. When a geometry shader is present, each shader invocation may store zero, one, or more vertices into the bound transform feedback buffers. Not only this, but it’s actually possible to configure up to four output streams and use the geometry shader to send its output to whichever one it chooses. This can be used, for example, to sort geometry or to render some primitives while storing other geometry in transform feedback buffers. There are a couple of pretty major limitations when multiple output streams are used in a geometry shader; first, the output primitive mode from the geometry shader for all streams must be set to points. Second, although it’s possible to simultaneously render geometry and to store data into transform feedback buffers, only the first stream may be rendered — the others are for storage only. If your application fits with these constraints, then this can be a very powerful feature.

To set up multiple output streams from your geometry shader, use the stream layout qualifier to select one of four streams. As with most other output layout qualifiers, the stream qualifier may be applied directly to a single output or to an output block. It can also be applied directly to the out keyword without declaring an output variable, in which case it will affect all further output declarations until another stream layout qualifier is encountered. For example, consider the following output declarations in a geometry shader:

out vec4                    foo; // "foo" is in stream 0 (the default).
layout (stream=2) out vec4  bar; // "bar" is part of stream 2.
out vec4                    baz; // "baz" is back in stream 0.

layout (stream=1) out;           // Everything from here on is in stream 1.
out int                     apple;  // "apple" and "orange" are part
out int                     orange; // of stream 1.
layout (stream=3) out MY_BLOCK   // Everything in "MY_BLOCK" is in
stream 3.
{
    vec3                    purple;
    vec3                    green;
};

In the geometry shader, when you call EmitVertex(), the vertex will be recorded into the first output stream (stream 0). Likewise, when you call EndPrimitive(), it will end the primitive being recorded to stream 0. However, you can call EmitStreamVertex() and EndStreamPrimitive(), both of which take an integer argument specifying the stream to send the output to:

void EmitStreamVertex(int stream);

void EndStreamPrimitive(int stream);

The stream argument must be a compile time constant. If rasterization is enabled, then any primitives sent to stream 0 will be rasterized.

New Primitive Types Introduced by the Geometry Shader

Four new primitive types were introduced with geometry shaders: GL_LINES_ADJACENCY, GL_LINE_STRIP_ADJACENCY, GL_TRIANGLES_ADJACENCY, and GL_TRIANGLE_STRIP_ADJACENCY. These primitive types are really only useful when rendering with a geometry shader active. When the new adjacency primitive types are used, for each line or triangle passed into the geometry shader, it not only has access to the vertices defining that primitive, but it also has access to the vertices of the primitive that is next to the one it’s processing.

When you render using GL_LINES_ADJACENCY, each line segment consumes four vertices from the enabled attribute arrays. The two center vertices make up the line; the first and last vertices are considered the adjacent vertices. The inputs to the geometry shader are therefore four-element arrays. In fact, because the input and output types of the geometry shader do not have to be related, GL_LINES_ADJACENCY can be seen as a way of sending generalized four-vertex primitives to the geometry shader. The geometry shader is free to transform them into whatever it pleases. For example, your geometry shader could convert each set of four vertices into a triangle strip made up of two triangles. This allows you to render quads using the GL_LINES_ADJACENCY primitive. It should be noted, though, that if you draw using GL_LINES_ADJACENCY when no geometry shader is active, regular lines will be drawn using the two innermost vertices of each set of four vertices. The two outermost vertices will be discarded, and the vertex shader will not run on them at all.

Using GL_LINE_STRIP_ADJACENCY produces a similar effect. The difference is that the entire strip is considered to be a primitive, with one additional vertex on each end. If you send eight vertices to OpenGL using GL_LINES_ADJACENCY, the geometry shader will run twice, whereas if you send the same vertices using GL_LINE_STRIP_ADJACENCY, the geometry shader will run five times. Figure 8.20 should make things clear. The eight vertices in the top row are sent to OpenGL with the GL_LINES_ADJACENCY primitive mode. The geometry shader runs twice on four vertices each time—ABCD and EFGH. In the second row, the same eight vertices are sent to OpenGL using the GL_LINE_STRIP_ADJACENCY primitive mode. This time, the geometry shader runs five times—ABCD, BCDE, and so on until EFGH. In each case, the solid arrows are the lines that would be rendered if no geometry shader were present.

The GL_TRIANGLES_ADJACENCY primitive mode works similarly to the GL_LINES_ADJACENCY mode. A triangle is sent to the geometry shader for each set of six vertices in the enabled attribute arrays. The first, third, and fifth vertices are considered to make up the real triangle, and the second, fourth, and sixth vertices are considered to be in between the triangle’s vertices. This means that the inputs to the geometry shader are six-element arrays. As before, you can do anything you want to the vertices using the geometry shader; GL_TRIANGLES_ADJACENCY is a good way to get arbitrary six-vertex primitives into the geometry shader. Figure 8.21 shows this.

The final, and perhaps most complex (or alternatively the most difficult to understand), of these primitive types is GL_TRIANGLE_STRIP_ADJACENCY. This primitive represents a triangle strip with every other vertex (the first, third, fifth, seventh, ninth, and so on) forming the strip. The vertices in between are the adjacent vertices. Figure 8.22 demonstrates the principle. In the figure, the vertices A through P represent 16 vertices sent to OpenGL. A triangle strip is generated from every other vertex (A, C, E, G, I, and so on), and the vertices that come between them (B, D, F, H, J, and so on) are the adjacent vertices.

There are special cases for the triangles that come at the start and end of the strip, but once the strip is started, the vertices fall into a regular pattern that is more clearly seen in Figure 8.23.

The rules for the ordering of GL_TRIANGLE_STRIP_ADJACENCY are spelled out clearly in the OpenGL Specification—in particular, the special cases are noted there. You are encouraged to read that section of the specification if you want to work with this primitive type.

Rendering Quads Using a Geometry Shader

In computer graphics, the word quad is used to describe a quadrilateral – a shape with four sides. Modern graphics APIs do not support rendering quads directly, primarily because modern graphics hardware does not support quads. When a modeling program produces an object made from quads, it will often include the option to export the geometry data by converting each quad into a pair of triangles. These are then rendered by the graphics hardware directly. In some graphics hardware, quads are supported, but internally the hardware will do this conversion from quads to pairs of triangles for you.

In many cases, breaking a quad into a pair of triangles works out just fine and the visual image isn’t much different than what would have been rendered had native support for quads been present. However, there are a large class of cases where breaking a quad into a pair of triangles doesn’t produce the correct result. Take a look at Figure 8.24.

In Figure 8.24, we have rendered a quad as a pair of triangles. In both images, the vertices are wound in the same order. There are three black vertices and one white vertex. In the left image, the split between the triangles runs vertically through the quad. The topmost and two side vertices are black and the bottommost vertex is white. The seam between the two triangles is clearly visible as a bright line. In the right image, the quad has been split horizontally. This has produced the topmost triangle, which contains only black vertices and is therefore entirely black, and the bottommost triangle, which contains one white vertex and two black ones, therefore displaying a black to white gradient.

The reason for this is that during rasterization and interpolation of the per-vertex colors presented to the fragment shader, we’re only rendering a triangle. There are only three vertices’ worth of information available to us at any given time, and therefore, we can’t take into consideration the “other” vertex in the quad.

Clearly, neither image is correct, but neither is obviously better than the other. Also, the two images are radically different. If we rely on our export tools, or worse a runtime library, to split quads for us, we do not have any control over which of these two images we’ll get. What can we do about that? Well, the geometry shader is able to accept primitives with the GL_LINES_ADJACENCY type, and each of these has four vertices — exactly enough to represent a quad. This means that by using lines with adjacency, we can get four vertices’ worth of information at least as far as the geometry shader.

Next, we need to deal with the rasterizer. Recall, the output of the geometry shader can only be points, lines, or triangles, and so the best we can do is to break each quad (represented by a lines_adjacency primitive) into a pair of triangles. You might think this leaves us in the same spot as we were before. However, we now have the advantage that we can pass whatever information we like on to the fragment shader.

To correctly render a quad, we must consider the parameterization of the domain over which we want to interpolate our colors (or any other attribute). For triangles, we use barycentric coordinates, which are three-dimensional coordinates used to weight the three corners of the triangle. However, for a quad, we can use a two-dimensional parameterization. Consider the quad shown in Figure 8.25.

Domain parameterization of a quad is two-dimensional and can be represented as a two-dimensional vector. This can be smoothly interpolated over the quad to find the value of the vector at any point within it. For each of the quad’s four vertices A, B, C, and D, the values of the vector will be (0, 0), (0, 1), (1, 0), and (1, 1), respectively. We can generate these values per vertex in our geometry shader and pass them to the fragment shader.

To use this vector to retrieve the interpolated values of our other per-fragment attributes, we make the following observation: The value of any interpolant will move smoothly between vertex A and B and between C and D with the x component of the vector. Likewise, a value along the edge AB will move smoothly to the corresponding value on edge CD. Thus, given the values of the attributes at the vertices A through D, we can use the domain parameter to interpolate a value of each attribute at any point inside the quad.

Thus, our geometry shader simply passes all four of the per-vertex attributes, unmodified, as flat outputs to the fragment shader, along with a smoothly varying domain parameter per vertex. The fragment shader then uses the domain parameter and all four per-vertex attributes to perform the interpolation directly.

The geometry shader is shown in Listing 8.34, and the fragment shader is shown in Listing 8.35 — both are taken from the gsquads example. Finally, the result of rendering the same geometry as shown in Figure 8.24 is shown in Figure 8.26.

Listing 8.34. Geometry shader for rendering quads

#version 430 core

layout (lines_adjacency) in;
layout (triangle_strip, max_vertices = 6) out;

in VS_OUT
{
    vec4 color;
} gs_in[4];

out GS_OUT
{
    flat vec4 color[4];
    vec2 uv;
} gs_out;

void main(void)
{
    gl_Position = gl_in[0].gl_Position;
    gs_out.uv = vec2(0.0, 0.0);
    EmitVertex();

    gl_Position = gl_in[1].gl_Position;
    gs_out.uv = vec2(1.0, 0.0);
    EmitVertex();

    gl_Position = gl_in[2].gl_Position;
    gs_out.uv = vec2(1.0, 1.0);

    // We're only writing the output color for the last
    // vertex here because they're flat attributes,
    // and the last vertex is the provoking vertex by default
    gs_out.color[0] = gs_in[1].color;
    gs_out.color[1] = gs_in[0].color;
    gs_out.color[2] = gs_in[2].color;
    gs_out.color[3] = gs_in[3].color;
    EmitVertex();

    EndPrimitive();

    gl_Position = gl_in[0].gl_Position;
    gs_out.uv = vec2(0.0, 0.0);
    EmitVertex();

    gl_Position = gl_in[2].gl_Position;
    gs_out.uv = vec2(1.0, 1.0);
    EmitVertex();

    gl_Position = gl_in[3].gl_Position;
    gs_out.uv = vec2(0.0, 1.0);

    // Again, only write the output color for the last vertex
    gs_out.color[0] = gs_in[1].color;
    gs_out.color[1] = gs_in[0].color;
    gs_out.color[2] = gs_in[2].color;
    gs_out.color[3] = gs_in[3].color;
    EmitVertex();

    EndPrimitive();
}

Listing 8.35. Fragment shader for rendering quads

#version 430 core

in GS_OUT
{
    flat vec4 color[4];
    vec2 uv;
} fs_in;

out vec4 color;

void main(void)
{
    vec4 c1 = mix(fs_in.color[0], fs_in.color[1], fs_in.uv.x);
    vec4 c2 = mix(fs_in.color[2], fs_in.color[3], fs_in.uv.x);

    color = mix(c1, c2, fs_in.uv.y);
}

Multiple Viewport Transformations

You learned in “Viewport Transformation” back in Chapter 3 about the viewport transformation and how you can specify the rectangle of the window you’re rendering into by calling glViewport() and glDepthRange(). Normally, you would set the viewport dimensions to cover the entire window or screen, depending on whether your application is running on a desktop or is taking over the whole display. However, it’s possible to move the viewport around and draw into multiple virtual windows within a single larger framebuffer. Furthermore, OpenGL also allows you to use multiple viewports at the same time. This feature is known as viewport arrays.

To use a viewport array, we first need to tell OpenGL what the bounds of the viewports we want to use are. To do this, call glViewportIndexedf() or glViewportIndexedfv(), whose prototypes are

void glViewportIndexedf(GLuint index,
                        GLfloat x,
                        GLfloat y,
                        GLfloat w,
                        GLfloat h);

void glViewportIndexedfv(GLuint index,
                         const GLfloat * v);

For both glViewportIndexedf() and glViewportIndexedfv(), index is the index of the viewport you wish to modify. Also notice that the viewport parameters to the indexed viewport commands are floating-point values rather than the integers used for glViewport(). OpenGL supports a minimum⁶ of 16 viewports, and so index can range from 0 to 15.

Likewise, each viewport also has its own depth range, which can be specified by calling glDepthRangeIndexed(), whose prototype is

void glDepthRangeIndexed(GLuint index,
                         GLdouble n,
                         GLdouble f);

Again, index may be between 0 and 15. In fact, glViewport() really sets the extent of all of the viewports to the same range, and glDepthRange() sets the depth range of all viewports to the same range. If you want to set more than one or two of the viewports at a time, you might consider using glViewportArrayv() and glDepthRangeArrayv(), whose prototypes are

void glViewportArrayv(GLuint first,
                      GLsizei count,
                      const GLfloat * v);

void glDepthRangeArrayv(GLuint first,
                        GLsizei count,
                        const GLdouble * v);

These functions set either the viewport extents or depth range for count viewports starting with the viewport indexed by first to the parameters specified in the array v. For glViewportArrayv(), the array contains a sequence of x, y, width, height values, in that order. For glDepthRangeArrayv(), the array contains a sequence of n, f pairs, in that order.

Once you have specified your viewports, you need to direct geometry into them. This is done by using a geometry shader. Writing to the built-in variable gl_ViewportIndex selects the viewport to render into. Listing 8.36 shows what such a geometry shader might look like.

Listing 8.36. Rendering to multiple viewports in a geometry shader

#version 430 core

layout (triangles, invocations = 4) in;
layout (triangle_strip, max_vertices = 3) out;

layout (std140, binding = 0) uniform transform_block
{
    mat4 mvp_matrix[4];
};

in VS_OUT
{
     vec4 color;
}  gs_in[];

out GS_OUT
{
    vec4 color;
} gs_out;

void main(void)
{
    for (int i = 0; i < gl_in.length(); i++)
    {
        gs_out.color = gs_in[i].color;
        gl_Position = mvp_matrix[gl_InvocationID] *
                      gl_in[i].gl_Position;
        gl_ViewportIndex = gl_InvocationID;
        EmitVertex();
    }
    EndPrimitive();
}

When the shader of Listing 8.36 executes, it produces four invocations of the shader. On each invocation, it sets the value of gl_ViewportIndex to the value of gl_InvocationID, directing the result of each of the geometry shader instances to a separate viewport. Also, for each invocation, it uses a separate model-view-projection matrix, which it retrieves from the uniform block, transform_block. Of course, a more complex shader could be constructed, but this is sufficient to demonstrate direction of transformed geometry into a number of different viewports. We have implemented this code in the multipleviewport sample, and the result of running this shader on our simple spinning cube is shown in Figure 8.27.

You can clearly see the four copies of the cube rendered by Listing 8.36 in Figure 8.27. Because each was rendered into its own viewport, it is clipped separately, and so where the cubes extend past the edges of their respective viewports, their corners are cut off by OpenGL’s clipping stage.