In the context of graphics programming, linear algebra can feel like a set of magic rules that, if applied correctly, produce realistic looking simulations of 3D space. Applying these rules is easy but, at the same time, completely unsatisfying without understanding why they work. So, as a first step, I decided to break down one of the simplest, most foundational operations of linear algebra: the dot product.

The dot product is commonly used in graphics calculations that require the angle between two directional vectors. Diffuse lighting is a good example, which produces a brighter surface when a light source is more direct. This computation relies on this mathematical equality:

\[\vec{v1} \cdot \vec{v2} = \|\vec{v1}\|\|\vec{v2}\|cos(\theta)\]

And, the dot product itself is defined as:

\[\begin{bmatrix} x1\\ y1 \end{bmatrix} \cdot \begin{bmatrix} x2\\ y2 \end{bmatrix} = x1*x2+y1*y2\]

I’m using 2-dimensional vector space because it’s easier to visualize and reason about, but this is an arbitrary choice — the same properties hold true in higher (and lower) dimensional space. If there are beings capable of perceiving 4 spacial dimensions, perhaps they write their examples in 3D space. But we’ll stick to two.

In English, the above equalities state that multiplying the components of two vectors along each axis individually, then summing those products results in the cosine of the angle between those vectors multiplied by the magnitude of each vector. This is a nifty “hack” to perform efficient geometry calculations, and it does indeed work, but why is it true? Intuitively, why is the sum of the products of vector components at all related to the angle between them?

A Visual Definition

First, let’s write an informal definition of the dot product that we can visualize. We can derive a visual definition by working backwards from the right side of the first equality above, \(\|\vec{v1}\|\|\vec{v2}\|cos(\theta)\). It is important to note that vectors that represent a direction and a magnitude only, so we can conceptually think of them as starting at the origin. In other words, they don’t have a position. Later, we will use vectors to represent locations in 3D space as well, but, for now, we are strictly talking about vectors in the classical sense (direction + magnitude).

Two Vectors on Coordinate Plane

Here, we have 2 vectors, \(\vec{a}\) and \(\vec{b}\). The angle between them is \(\theta\). Let’s try to visualize the dot product. First, we draw a right triangle by drawing a line that is orthogonal (normal) to either \(\vec{a}\) or \(\vec{b}\) and intersects with the endpoint of the other vector. We’ll choose \(\vec{b}\), but the logic would be the same if we chose \(\vec{a}\) instead, just reversed.

Two Vectors With Perpendicular Line

Now, we can invoke our long-time friend SohCahToa to substitute \(cos(\theta)\) with \(adjacent/hypotenuse\), which in this case is \(a'/\|\vec{a}\|\). We’ve labeled the “adjacent” side of our triangle \(a'\) because we don’t yet have a good name for it. But we’re one step closer to visually understanding the dot product. Let’s substitute and simplify:

\[\vec{a} \cdot \vec{b} = \|\vec{a}\|\|\vec{b}\|cos(\theta) = \frac{\|\vec{a}\|\|\vec{b}\|a'}{\|\vec{a}\|}=\|\vec{b}\|a'\]

If we chose to draw the normal from \(\vec{a}\) instead, we’d get \(\|\vec{a}\|b'\). So, in English, what is the dot product? If we think about a light source pointed directly onto \(\vec{b}\), \(a'\) represents the length of shadow of \(\vec{a}\) that would cast onto \(\vec{b}\). So, the dot product is a number that, if divided by the magnitude of either vector, produces the length of the shadow of the other vector onto it, assuming a direct light source. Visually,

Two Vectors With Lights Casting Shadows

You may be thinking that this description isn’t exactly intuitive like I promised, but we typically represent direction vectors as unit vectors, which have a magnitude of 1. When we are using unit vectors, the definition of the dot product is much simpler, which hopefully helps explain why unit vectors are so useful for representing directions: The dot product is the length of the shadow cast from one unit vector onto the other.

We’ve cracked the case on how to visualize \(\|\vec{v1}\|\|\vec{v2}\|cos(\theta)\) . But, there are still a few insights we need to be able to explain that the sum of the product of vector components represents the same thing, spatially.

Projection

We’ve been talking about “casting shadows”, but what we really mean is projection. To generalize shadow casting, it’s really just “how much of one vector points in the direction of another”. Projection maps one vector space onto another. For example, when you take a picture, the resulting photograph is a 2-dimensional projection of 3D space, where each point on the picture maps to a real point in 3D space depending on its position relative to the camera lens. It is no surprise that this concept is also used to represent 3D graphics on a computer monitor.

There are many different types of projections. The dot product performs one of these projections: the orthogonal projection. This isn’t exactly how we perceive the world (3D graphics use perspective projection to more accurately model the way humans see the world). An orthogonal projection is more like a floorplan or blueprint, where each point is projected directly at the “camera” at 90 degrees.

Let’s imagine a being who perceives the world in 2 dimensions. How would this being see the world? If this being took a picture, what would the resulting photograph look like? Just as our photographs represent a projection into a lower dimensional space (3D to 2D), a 2-dimensional photograph would represent the projection from 2D to 1D. In other words, this being would see the world as a line.

This is more clearly described with a picture. Let’s name this 2D being Pierre. Imagine Pierre is hanging out in his 2D world, looking at a line and couple of points:

2D Being Viewing Shapes

We are able to see Pierre from this perspective because we can perceive a higher dimension than he can, but remember, he has no concept of “up” or “down”. Here’s what Pierre would see, from his perspective:

Shapes From 2D Being's Perspective

Since the dot product is an orthogonal projection, it is most useful to imagine that members of Pierre’s species are shaped like a line, with eyes dotting their entire length. Therefore, Pierre does not experience perspective like we do - if two objects are in front of Pierre, regardless of their position, he will only be able to see them head-on.

Why should we care about Pierre? Because we can re-imagine the dot product’s projection, what we called \(a'\) above, not as a shadow, but what Pierre’s sees when he aligns orthogonally to one vector and looks at the other. In other words, if Pierre is oriented so that \(\vec{b}\) points to his right, the way he perceives \(\vec{a}\) is the projection \(a'\)! Since a picture is worth a thousand words:

Vectors From Different Perspectives Side by Side

Finishing Up

Let’s see how Pierre can help us understand the dot product. The method is as follows: align Pierre with one vector, rotate our perspective so that the Y-axis aligns with the Pierre’s point of view, then take only the X component of the resulting vector. Let’s look at the initial setup, from the perspective of global coordinates (world space).

2D Being Viewing Vectors

Now, let’s rotate our perspective to match Pierre’s. To do this, let’s change our perspective so that \(\vec{b}\) aligns with the X-axis.

2D Being Viewing Vectors Rotated

To calculate \(a'\) after this change in perspective, we can simply rotate a vector of magnitude \(\|\vec{a}\|\) counter-clockwise from the X-axis by an angle of \(\theta\), then take the X component. Therefore, Pierre sees \(\vec{a}\) as a line of length \(\|\vec{a}\|cos(\theta)\).

To expand this expression, let’s do a little bit of trigonometry to express \(cos(\theta)\) in terms of vector components. We’ll use the terms \(a\theta\) and \(b\theta\) to represent the angle by which each vector has been rotated counter-clockwise from the positive X-axis so that \(a\theta-b\theta=\theta\).

\[\begin{align*}\\ cos(\theta)\\ =cos(a\theta-b\theta)\\ =cos(a\theta)cos(b\theta)+sin(a\theta)sin(b\theta)\\ =\frac{ax*bx}{\|\vec{a}\|\|\vec{b}\|}+\frac{ay*by}{\|\vec{a}\|\|\vec{b}\|}\\ =\frac{1}{\|\vec{a}\|\|\vec{b}\|}(ax*bx)+(ay*by) \end{align*}\\\]

We use the cosine subtraction formula to expand, express in terms of vector components using SohCahToa, and pull out a common factor. We still need to scale up by a factor of \(\|\vec{a}\|\) to get the length of the line the Pierre sees, which leaves us with:

\[\|\vec{a}\|cos(\theta) = \frac{(ax*bx)+(ay*by)}{\|\vec{b}\|}\]

Look familiar? We already know that \(a'\) is equal to the dot product divided by \({\|\vec{b}\|}\), and \(a'\) is equal to \(\|\vec{a}\|cos(\theta)\), so the dot product must be equal to \(ax*bx+ay*by\)! As it turns out, although the dot product doesn’t look like it would relate to the angle between two vectors, it’s more or less a restating of the cosine subtraction formula.

NOTE: In retrospect, we could have proven that \(\|\vec{a}\|\|\vec{b}\|cos(\theta)=ax*bx+ay*by\) simply by using the cosine subtraction formula substitution and simplifying, without imagining how a 2-dimensional being sees the world. But, using a visual aid to explore the concept was more interesting and approachable for me!

Additional Thoughts

As I was thinking about this problem, I went down a few paths that didn’t pan out, but provided some interesting insights:

  1. My first instinct was to try to use the Pythagorean Theorem to calculate \(a'\), but I later realized that the Pythagorean Theorem is just a special case of the dot product. Imagine taking the dot product between two equivalent vectors:

    \[\begin{bmatrix} a\\ b \end{bmatrix} \cdot \begin{bmatrix} a\\ b \end{bmatrix} = a^2+b^2\]

    Because the vectors have the same direction, the angle between them is 0, so \(cos(\theta)\) is 1. The magnitudes are also the same, which yields this familiar formula:

    \[a^2+b^2=c^2\]

    In other words, the Pythagorean theorem is simply projecting a vector onto itself!

  2. Regardless of which vector we project onto the other, we get the same dot product when we multiply by the magnitude of the base vector. Therefore, the shadows casted by \(\vec{a}\) and \(\vec{b}\) are proportional to their magnitudes. Or, in mathematical notation:

    \[if \vec{a}\cdot\vec{b}=\|\vec{b}\|a'=\|\vec{a}\|b' then \frac{\|\vec{a}\|}{a'}=\frac{\|\vec{b}\|}{b'}\]

    This isn’t exactly ground breaking and may be obvious to some. But, it could come in handy later.