Split an image into meaningful regions, so every pixel knows which object it belongs to.
Segmentation: partitioning an image into meaningful regions. Goal: group the pixels that belong to the same object or region.
Label each pixel by its class.
Separate the individual objects.
Semantic + instance combined.
Methods: thresholding · clustering · watershed · deep learning
One fixed threshold.
A local threshold per region.
An automatic, optimal threshold.
import cv2
# simple threshold
_, binary = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)
# Otsu's automatic threshold
ret, binary = cv2.threshold(gray, 0, 255,
cv2.THRESH_BINARY + cv2.THRESH_OTSU) # ans:
# ans: ret is the computed threshold value
Otsu tries every threshold and keeps the one that best separates the histogram into two classes (the peak of the pink curve: the between-class variance).
Why adaptive? It handles varying illumination.
# adaptive mean thresholding
adaptive = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C,
cv2.THRESH_BINARY, blockSize=11, C=2) # ans:
# adaptive Gaussian thresholding
adaptive = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY, 11, 2) # ans:
# blockSize: neighbourhood size (must be odd)
# ans: C: a constant subtracted from the mean
Treat the image like a topographic map: intensity = height.
Water fills the basins, starting from markers.
Where two floods meet, a watershed line (dam) separates the regions.
# binary threshold
_, binary = cv2.threshold(gray, 0, 255,
cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)
# noise removal (opening)
kernel = np.ones((3, 3), np.uint8)
opening = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel, iterations=2)
# sure background (dilation)
sure_bg = cv2.dilate(opening, kernel, iterations=3) # ans:
# sure foreground (distance transform + threshold)
dist_transform = cv2.distanceTransform(opening, cv2.DIST_L2, 5) # ans:
_, sure_fg = cv2.threshold(dist_transform, 0.7 * dist_transform.max(), 255, 0) # ans:
# unknown region
sure_fg = np.uint8(sure_fg) # ans:
unknown = cv2.subtract(sure_bg, sure_fg) # ans:
# label markers
_, markers = cv2.connectedComponents(sure_fg) # ans:
# add 1 to all labels (background becomes 1, not 0)
markers = markers + 1 # ans:
# mark the unknown region as 0
markers[unknown == 255] = 0 # ans:
# apply watershed
markers = cv2.watershed(img, markers) # ans:
# mark boundaries (-1) in red
img[markers == -1] = [0, 0, 255] # ans:
opening: a plain threshold sees one blob.GrabCut: iterative segmentation using graph cuts.
A rectangle around the object.
Semi-automatic: minimal user input.
# define a rectangle around the object
rect = (50, 50, 450, 290) # (x, y, width, height) # ans:
# initialise the mask and models
mask = np.zeros(img.shape[:2], np.uint8)
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)
# apply GrabCut
cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 5, cv2.GC_INIT_WITH_RECT) # ans:
# ans: 5 iterations
# extract the foreground
mask2 = np.where((mask == 2) | (mask == 0), 0, 1).astype('uint8') # ans:
result = img * mask2[:, :, np.newaxis] # ans:
Drag a rectangle around a doll. Outside: sure background. Inside: each iteration re-fits both colour models and relabels the pixels.
iteration 0 / 5
Simplified: a small colour mixture per side plays the GMM; a smoothing pass plays the graph cut.
K-means clusters pixels by colour similarity.
Result: segmented regions with k colours.
# reshape to a 2-D array of pixels
pixels = img.reshape((-1, 3)) # ans:
pixels = np.float32(pixels) # ans:
# define criteria and apply k-means
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.2)
k = 3
_, labels, centers = cv2.kmeans(pixels, k, None, criteria, 10,
cv2.KMEANS_RANDOM_CENTERS) # ans:
# convert centres to uint8
centers = np.uint8(centers) # ans:
# map labels to centres
segmented = centers[labels.flatten()] # ans:
segmented = segmented.reshape(img.shape) # ans:
Contours are the boundaries of shapes; findContours detects them all.
# threshold to binary
_, binary = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)
# find contours
contours, hierarchy = cv2.findContours(binary, cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE) # ans:
# draw contours
cv2.drawContours(img, contours, -1, (0, 255, 0), 2) # ans:
# ans: -1 draws all contours
# filter by area
large_contours = [c for c in contours if cv2.contourArea(c) > 1000] # ans:
The dolls are darker than the wall, so this uses THRESH_BINARY_INV: objects must be white for findContours.
Only contours with area above the minimum are drawn: the small blobs inside the patterns drop out.
Answer first, then press → to check.
What is image segmentation?
Partitioning an image into meaningful regions.
What is Otsu's method?
An automatic computation of the optimal threshold.
Why adaptive thresholding?
It handles varying illumination.
What is the watershed algorithm?
It treats the image as a topographic map and floods the basins.
What is GrabCut?
Iterative segmentation using graph cuts and GMMs.
How is k-means used for segmentation?
It clusters pixels by colour similarity.
What are contours?
The boundaries of shapes or objects.
Semantic vs instance segmentation?
Semantic labels pixels by class; instance separates individual objects.
What is the advantage of GrabCut?
Semi-automatic: it needs only a rectangle as input.
# Otsu's thresholding
ret, binary = cv2.threshold(gray, 0, 255,
cv2.THRESH_BINARY + cv2.THRESH_OTSU) # ans:
# adaptive thresholding
adaptive = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY, 11, 2) # ans:
# find contours
_, binary = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)
contours, _ = cv2.findContours(binary, cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE) # ans:
cv2.drawContours(img, contours, -1, (0, 255, 0), 2) # ans:
# k-means with 4 clusters
pixels = img.reshape((-1, 3))
pixels = np.float32(pixels)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.2)
_, labels, centers = cv2.kmeans(pixels, 4, None, criteria, 10,
cv2.KMEANS_RANDOM_CENTERS) # ans:
centers = np.uint8(centers) # ans:
segmented = centers[labels.flatten()].reshape(img.shape) # ans:
# GrabCut segmentation
rect = (50, 50, 450, 290)
mask = np.zeros(img.shape[:2], np.uint8)
bgdModel = np.zeros((1, 65), np.float64)
fgdModel = np.zeros((1, 65), np.float64)
cv2.grabCut(img, mask, rect, bgdModel, fgdModel, 5, cv2.GC_INIT_WITH_RECT) # ans:
mask2 = np.where((mask == 2) | (mask == 0), 0, 1).astype('uint8') # ans:
result = img * mask2[:, :, np.newaxis] # ans: