Computer vision

Kernels & Convolution

A tiny grid of weights, slid across an image, can blur it, sharpen it or find its edges.

What is a kernel?

A kernel (or filter) is a small matrix used to apply effects to an image.

Identity

Blur

Sharpen

What is convolution?

Convolution slides a kernel over an image and computes a weighted sum at every position.

  1. Place the kernel on an image region
  2. Multiply corresponding elements
  3. Sum all the products
  4. The sum is the new pixel value
  5. Slide the kernel to the next position and repeat

$$\text{Output}(x, y) = \sum_{i}\sum_{j} \text{Kernel}(i, j) \times \text{Image}(x + i,\ y + j)$$

\(i, j\) run over the kernel's offsets: −1, 0, 1 for a 3×3 kernel. Strictly this is correlation; true convolution flips the kernel first, which changes nothing for symmetric kernels. cv2.filter2D computes exactly this sum.

One output pixel, by hand

102030 405060 708090
Image patch
×
Kernel
=
10×11020×0030×(−1)−30 40×14050×0060×(−1)−60 70×17080×0090×(−1)−90
Products

Sum = (10 − 30) + (40 − 60) + (70 − 90) = −60

Result: one output pixel value, −60. The kernel then slides one pixel and does it again.

Watch the kernel slide

Input image, 7×7

Kernel

Pixel × weight

Σ = … 

Output, 5×5

No padding here, so a 7×7 image gives a 5×5 output. Shades are the stored 8-bit values: sums below 0 or above 255 are clipped.

Edge handling

?????

Problem: at a border pixel, the kernel extends beyond the image. Those five cells have no pixels, so we must invent values for them.

Zero paddingBORDER_CONSTANT
00abcde00
Border replicationBORDER_REPLICATE
aaabcdeee
ReflectionBORDER_REFLECT_101
cbabcdedc
Wrap aroundBORDER_WRAP
deabcdeab

OpenCV's default is cv2.BORDER_REFLECT_101: it mirrors without repeating the edge pixel.

Custom convolution in OpenCV


          import cv2
          import numpy as np
          # define a 3x3 identity kernel
          kernel = np.array([[0, 0, 0],
                             [0, 1, 0],
                             [0, 0, 0]], dtype=np.float32)
          # apply convolution
          result = cv2.filter2D(img, -1, kernel)
          # ans: -1 means output has the same depth as the input
          # identity kernel returns the original image
          # ans: centre weight = 1, others = 0, so nothing changes
        

Identity

Box blur kernel


          # 3x3 averaging kernel
          kernel = np.ones((3, 3), dtype=np.float32) / 9
          # ans: all weights = 1/9, they sum to 1
          # apply box blur
          blurred = cv2.filter2D(img, -1, kernel)
          # ans: same as cv2.blur(img, (3, 3))
          # 5x5 box blur kernel
          kernel = np.ones((5, 5), dtype=np.float32) / 25
          result = cv2.filter2D(img, -1, kernel)
          # ans: averages 25 pixels, so more blur
        

Sharpening kernel


          # sharpen kernel (enhances edges)
          kernel = np.array([[ 0, -1,  0],
                             [-1,  5, -1],
                             [ 0, -1,  0]], dtype=np.float32)
          # ans: centre = 5, neighbours = -1, enhances differences
          sharpened = cv2.filter2D(img, -1, kernel)
          # ans: edges become more pronounced
          # why does this sharpen?
          # ans: it amplifies the centre and subtracts the neighbours
        

Weights sum to 1: flat areas stay the same,
differences get bigger.

Edge detection kernels


          # horizontal edge detection (Prewitt)
          kernel_h = np.array([[-1, -1, -1],
                               [ 0,  0,  0],
                               [ 1,  1,  1]], dtype=np.float32)
          # ans: detects horizontal edges (top-to-bottom change)
          # vertical edge detection
          kernel_v = np.array([[-1,  0,  1],
                               [-1,  0,  1],
                               [-1,  0,  1]], dtype=np.float32)
          # ans: detects vertical edges (left-to-right change)
          edges_h = cv2.filter2D(gray, -1, kernel_h)
          edges_v = cv2.filter2D(gray, -1, kernel_v)
          # ans: apply to grayscale for best results
        

Horizontal

Vertical

Gaussian kernel


          # 5x5 gaussian kernel approximation
          kernel = np.array([[1,  4,  6,  4, 1],
                             [4, 16, 24, 16, 4],
                             [6, 24, 36, 24, 6],
                             [4, 16, 24, 16, 4],
                             [1,  4,  6,  4, 1]], dtype=np.float32) / 256
          # ans: weights decrease away from the centre (gaussian distribution)
          gaussian = cv2.filter2D(img, -1, kernel)
          # ans: similar to cv2.GaussianBlur, but with a custom kernel
        

Kernel normalization

Why normalize? To keep the output in the valid range, 0 to 255. Divide by the sum of the kernel's weights.


          # unnormalized kernel
          kernel = np.array([[1, 2, 1],
                             [2, 4, 2],
                             [1, 2, 1]], dtype=np.float32)
          # sum of weights
          kernel_sum = kernel.sum()  # = 16
          # ans: sum = 1+2+1+2+4+2+1+2+1 = 16
          # normalize
          kernel_norm = kernel / kernel_sum
          # ans: ensures the output stays in range
        

Every kernel, on a real photo

Original
Identity

Exercises

Answer first, then press → to check.

Exercises 1 · concepts

What does convolution do?

Slides a kernel over the image and computes weighted sums.

Why are kernel sizes odd?

It needs a centre pixel, for symmetry.

What does this kernel do?


            kernel = np.array([[0, 0, 0],
                               [0, 1, 0],
                               [0, 0, 0]])
          

It is the identity kernel: it returns the original image unchanged.

What does a sum of weights equal to 1 ensure?

Brightness is preserved; nothing is amplified.

Exercises 2 · concepts

What kind of kernel detects vertical edges?

One with a [-1, 0, 1] pattern running horizontally.

What is the ddepth parameter of cv2.filter2D?


            result = cv2.filter2D(img, -1, kernel)
          

-1 means the output has the same depth as the source.

What is the difference between blur and sharpen kernels?

Blur averages (all weights positive); sharpen enhances (negative neighbours).

Why apply edge detection to grayscale?

It is simpler: one channel, and edges are changes in intensity.

Exercises 3 · coding

Create a 3×3 identity kernel and apply it.


            kernel = np.array([[0, 0, 0],
                               [0, 1, 0],
                               [0, 0, 0]], dtype=np.float32)
            result = cv2.filter2D(img, -1, kernel)
          

Create a 3×3 box blur kernel and apply it.


            kernel = np.ones((3, 3), dtype=np.float32) / 9
            result = cv2.filter2D(img, -1, kernel)
          

Exercises 4 · coding

Create a sharpen kernel and apply it.


            kernel = np.array([[ 0, -1,  0],
                               [-1,  5, -1],
                               [ 0, -1,  0]], dtype=np.float32)
            sharpened = cv2.filter2D(img, -1, kernel)
          

Create a horizontal edge kernel.


            kernel = np.array([[-1, -1, -1],
                               [ 0,  0,  0],
                               [ 1,  1,  1]], dtype=np.float32)
          

Exercises 5 · coding

Create an emboss kernel and apply it to a grayscale image.


            kernel = np.array([[-2, -1,  0],
                               [-1,  1,  1],
                               [ 0,  1,  2]], dtype=np.float32)
            embossed = cv2.filter2D(gray, -1, kernel)
          

Create an outline kernel and apply it.


            kernel = np.array([[-1, -1, -1],
                               [-1,  8, -1],
                               [-1, -1, -1]], dtype=np.float32)
            outline = cv2.filter2D(img, -1, kernel)
          

Exercises 6 · mixed

Normalize this custom kernel.


              kernel = np.array([[1, 2, 1],
                                 [2, 4, 2],
                                 [1, 2, 1]], dtype=np.float32)
            

              kernel_norm = kernel / kernel.sum()
              # sum = 16, so divide by 16
            

What is the sum after normalization?

1.0, which preserves brightness.

Create a strong sharpen kernel (centre = 9).


            kernel = np.array([[-1, -1, -1],
                               [-1,  9, -1],
                               [-1, -1, -1]], dtype=np.float32)
            # eight neighbours of -1 balance the 9: weights still sum to 1
          

Exercises 7 · mixed

Create a 5×5 average kernel.


            kernel = np.ones((5, 5), dtype=np.float32) / 25
          

Create a left edge detection kernel.


            kernel = np.array([[1,  0, -1],
                               [2,  0, -2],
                               [1,  0, -1]], dtype=np.float32)
            # this is the Sobel-X kernel (sign flipped: bright-to-dark edges are positive)
          

Combine these two edge detections into one image.


            edges_h = cv2.filter2D(gray, -1, kernel_h)
            edges_v = cv2.filter2D(gray, -1, kernel_v)
            edges = cv2.addWeighted(edges_h, 0.5, edges_v, 0.5, 0)  # ans: horizontal and vertical edges, half each
          

Exercises 8 · advanced

Create a horizontal motion blur kernel.


            kernel = np.zeros((5, 5), dtype=np.float32)
            kernel[2, :] = 1/5
            # one row of weights: a horizontal motion blur
          

Apply a convolution with an explicit border mode.


            result = cv2.filter2D(img, -1, kernel, borderType=cv2.BORDER_CONSTANT)
          

Create a custom gaussian-like 3×3 kernel.


            kernel = np.array([[1, 2, 1],
                               [2, 4, 2],
                               [1, 2, 1]], dtype=np.float32) / 16