Week 12
Image Analysis and Audio Signal Processing
Images and sounds are data. This chapter shows how pixels, filters, waves, and spectrograms become numerical representations that computers can analyze.
Pixels
Filters
Signals
Learning Goals
By the End of This Chapter
By the end, you should be able to describe images as pixel arrays, explain smoothing and edge filters, describe amplitude/frequency/phase, and interpret a spectrogram as time-frequency data.
How to Use This Page
Try each explorer first, then run the related script. The explorers build intuition; the code shows how the notebook implements the same ideas.
Source Materials
The notebook supplies the Python examples. The slides add the conceptual sequence: pixels and resolution, convolution, edge filters, sound waves, sampling, spectra, and spectrograms.
Part 1
Digital Images
An image is a grid of pixels. A grayscale image stores one intensity value per pixel, while an RGB image stores three values: red, green, and blue.
\[\text{image shape} = \text{height} \times \text{width} \times \text{channels}\]
Image as a function
The slides describe a grayscale image as a map \(f(x, y)\) from pixel position to intensity. In an 8-bit image, that intensity often ranges from 0 for black to 255 for white.
Pixel Grid Explorer
Each cell is one pixel. Changing the channel view shows why an RGB image has three values per pixel, while grayscale has one intensity value.
Load image and convert to grayscale
from skimage import data, color
image_rgb = data.astronaut()
image = color.rgb2gray(image_rgb)
print("RGB image shape:", image_rgb.shape)
print("Grayscale image shape:", image.shape)
print("Grayscale value range: 0.0 to 1.0")Part 2
Smoothing and Convolution
Image filtering modifies pixel values by looking at nearby pixels. A smoothing filter replaces each pixel with a local average, which reduces noise but also blurs edges.
Key point
A convolution kernel is a small matrix of weights. The kernel slides over the image, multiplies nearby pixel values by those weights, and sums the result.
Average filter
The notebook uses a 3x3 average filter. Every weight is 1/9, so each output pixel is the mean of its 3x3 neighborhood.
Convolution Explorer
The left grid is a small grayscale image patch. The blue square marks the 3x3 neighborhood used to compute the output pixel. Choose a kernel to see how the weighted sum changes.
Create a 3x3 average filter
import numpy as np
average_filter = np.ones((3, 3)) / 9.0
print(average_filter)Apply smoothing convolution
from scipy.ndimage import convolve
filtered_image = convolve(image, average_filter)
print("Filtered image shape:", filtered_image.shape)
print("Effect: local averaging reduces noise and softens edges.")Part 3
Edge Detection
Edges are sudden changes in intensity. Sobel filters estimate gradients, while Canny combines smoothing, gradient detection, edge thinning, and thresholding.
\[\lVert \nabla I \rVert = \sqrt{G_x^2 + G_y^2}\]
Sobel and Canny
Sobel emphasizes local intensity changes by estimating the horizontal gradient \(G_x\) and vertical gradient \(G_y\). Canny adds Gaussian smoothing controlled by \(\sigma\), so a larger \(\sigma\) keeps fewer, smoother edges.
Edge Detector Explorer
Sobel uses gradients and does not use sigma. Canny first smooths the image, so sigma changes which edges remain.
Sobel edge detection
from skimage.filters import sobel
edge_sobel = sobel(image)
print("Sobel edge image shape:", edge_sobel.shape)
print("Output meaning: gradient magnitude at each pixel.")Canny edge detection
from skimage.feature import canny
edge_canny = canny(image, sigma=1.0)
print("Canny edge mask shape:", edge_canny.shape)
print("Data type: bool")
print("sigma: 1.0")Part 4
Sound as a Signal
Sound is a wave. In data analysis, a digital audio file becomes a sequence of amplitude samples measured at a fixed sampling rate.
\[s(t) = A\sin(2\pi f t + \phi)\]
- Amplitude \(A\) controls loudness.
- Frequency \(f\) controls pitch.
- Period \(T = \frac{1}{f}\) is the time for one cycle.
- Phase \(\phi\) shifts where the wave starts.
Waveform Explorer
Load Brahms audio with librosa
import librosa
audio_path = librosa.example("brahms")
audio_data, sample_rate = librosa.load(audio_path, duration=45)
print(f"Audio data shape: {audio_data.shape}, Sample rate: {sample_rate} Hz")Create audio playback object
from IPython.display import Audio
player = Audio(audio_data, rate=sample_rate)
print("Audio object prepared for playback at", sample_rate, "Hz")Plot time-domain waveform
import matplotlib.pyplot as plt
plt.plot(audio_data)
plt.title("Time-Domain Waveform")
plt.xlabel("Samples")
plt.ylabel("Amplitude")
plt.show()
print("Waveform samples:", len(audio_data))Part 5
Spectrograms
A spectrogram shows how frequency content changes over time. It is often the bridge between raw audio and machine learning models.
How to read it
The x-axis is time, the y-axis is frequency, and color represents amplitude in decibels. Each vertical slice summarizes which frequencies are strong during one small moment.
Fourier transform idea behind STFT
A waveform shows how sound moves up and down over time. A Fourier transform asks a different question: which frequency ingredients are mixed into this sound?
Think of a piano chord. Your ear hears one combined sound, but the chord contains several notes. A Fourier transform helps a computer separate a mixed signal into low, middle, and high frequency ingredients.
The Short-Time Fourier Transform, or STFT, repeats this idea for many short pieces of the recording. That is why a spectrogram can show both which frequencies are present and when they become strong.
Spectrum and Spectrogram Explorer
The signal combines 50 Hz, 120 Hz, and 300 Hz components from the slides. Change component strength and energy pattern to see the spectrum bars and spectrogram heatmap respond.
Generate a spectrogram
import librosa.display
D = librosa.stft(audio_data)
D_db = librosa.amplitude_to_db(abs(D))
librosa.display.specshow(D_db, sr=sample_rate)
plt.colorbar(format="%+2.0f dB")
plt.title("Spectrogram")
plt.show()
print("STFT matrix shape:", D.shape)
print("Spectrogram dB shape:", D_db.shape)Practice
Practice Questions
Use these questions to check your understanding and connect the examples to real image and audio data.
Channels
What information is lost when an RGB image is converted to grayscale, and why might that still be useful?
Show answer
Color differences are collapsed into one intensity value, so red, green, and blue information is lost. Grayscale can still be useful because many tasks, such as detecting edges or shapes, depend mostly on brightness patterns.
Filtering
Why does an average filter reduce noise but make edges less sharp?
Show answer
An average filter replaces each pixel with the mean of nearby pixels. Random noise tends to cancel out, but true sudden changes between regions are also blended, so edges become smoother and less sharp.
Edges
How does increasing Canny sigma change the kinds of edges that remain?
Show answer
A larger \(\sigma\) smooths the image more before edge detection. Fine texture and noisy small edges disappear first, while stronger outlines and broad boundaries are more likely to remain.
Wave Parameters
In \(s(t) = A\sin(2\pi f t + \phi)\), what changes in the graph when \(A\), \(f\), or \(\phi\) is adjusted?
Show answer
Increasing \(A\) makes the wave taller, increasing \(f\) creates more cycles per second and shortens the period \(T = 1/f\), and changing \(\phi\) shifts the wave left or right in time.
Spectrograms
Why might a spectrogram be easier for a model to use than raw audio samples?
Show answer
A spectrogram organizes audio by time and frequency, making patterns such as pitch, harmonics, and changing energy easier to see. Raw samples contain the same signal but are harder to interpret directly because frequency structure is implicit.
STFT
Why does the Short-Time Fourier Transform look at short pieces of audio instead of only the whole recording at once?
Show answer
A Fourier transform of the whole recording can tell which frequency ingredients appear overall, but it does not show clearly when they appear. STFT applies the same frequency idea to many short pieces, so the spectrogram can show changes over time.
Discussion
Anonymous Question Box
After exploring the materials, send a question or confusing point for class discussion. Please do not include your name, student ID, email address, or other identifying details.
Reference
Key Terms
| Term | Meaning | Section |
|---|---|---|
| Pixel | The smallest addressable unit in a digital image. | Digital images |
| Channel | One layer of image values, such as red, green, blue, or grayscale intensity. | Digital images |
| Convolution | A weighted local operation using a sliding kernel. | Smoothing |
| Sobel | An edge detector based on horizontal and vertical gradients. | Edges |
| Canny | A multi-step edge detector with smoothing, gradients, thinning, and thresholding. | Edges |
| Sampling rate | The number of audio samples measured per second. | Sound |
| Amplitude | The height or strength of a wave, often related to perceived loudness in audio. | Sound |
| Frequency | The number of wave cycles per second, measured in Hertz. | Sound |
| Period | The time needed for one complete wave cycle; for frequency \(f\), \(T = 1/f\). | Sound |
| Phase | The horizontal shift of a wave that determines where the cycle starts. | Sound |
| Spectrum | A frequency-domain view that shows how strongly different frequencies are present. | Spectrograms |
| Fourier Transform | A method for finding which low, middle, and high frequency ingredients are present in a signal. | Spectrograms |
| Short-Time Fourier Transform | A method that applies the Fourier transform to short pieces of audio so frequency changes can be tracked over time. | Spectrograms |
| Spectrogram | A time-frequency representation of audio intensity. | Spectrograms |