Every watch has a heart rate sensor these days. But what if there's no watch? The phone still has a camera — and it turns out that's enough to estimate heart rate: no external hardware, no cloud, and not a single frame saved. In MeteoHealth I built this feature entirely on system frameworks — AVFoundation for capture, Vision for face detection, Accelerate for spectral analysis. No ML models: under the hood it's pure digital signal processing. Below is how it works, with real production code, and — honestly — where the method's limits are.
How a finger becomes a sensor#
The principle is called photoplethysmography (PPG) — the same one used in pulse oximeters. You place a finger on the rear camera, and the flash (torch) shines light straight through the fingertip. Blood absorbs light, and with every heartbeat the capillary filling changes — and with it the amount of light reaching the sensor. The camera becomes a photodetector: each frame is one sample of the signal.
From each frame we take the mean of the red channel over a central 64×64 region — a direct pass over the raw BGRA buffer, with no Core Image and no intermediate copies:
/// Mean of RED over the central 64×64 region (finger mode). BGRA, R at +2.
private nonisolated static func meanRedCenter(of pixelBuffer: CVPixelBuffer) -> Float? {
CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly) }
guard let baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer) else { return nil }
// ...
var redSum: UInt64 = 0
for y in startY ..< startY + cropSize {
let rowStart = y * bytesPerRow
for x in startX ..< startX + cropSize {
redSum &+= UInt64(bytes[rowStart + x * 4 + 2])
}
}
return Float(redSum) / Float(cropSize * cropSize) / 255.0
}Over 25–30 seconds at 30 fps, a buffer of roughly 750–900 samples accumulates. Each frame is discarded right after averaging — the CMSampleBuffer is never stored, and no files ever appear on disk. That's not a side effect; it's an invariant, spelled out right in the header of the view model.
All the math lives in a separate PPGProcessor struct that knows nothing about AVFoundation or SwiftUI — only Float arrays and timestamps. Thanks to that, the entire pipeline is covered by unit tests: feed it a synthetic 1.2 Hz sine wave and it must return 72 BPM.
One detail for those living in Swift 6: AVCaptureSession and AVCaptureVideoDataOutput are configured not on the MainActor but on a dedicated sessionQueue, so under strict concurrency they are marked nonisolated(unsafe) — with an explicit comment that confinement is guaranteed by the queue:
/// The session and output are confined to sessionQueue (begin/commitConfiguration,
/// start/stopRunning) — not to the MainActor, hence nonisolated(unsafe).
nonisolated(unsafe) let captureSession = AVCaptureSession()Non-uniform time: why resampling matters#
Each frame has become a sample — but not at evenly spaced points in time. The least obvious step of the pipeline isn't the FFT — it's what comes before it. FFT assumes uniform sampling, but CMSampleBuffer timestamps drift: the system drops frames, the inter-frame interval "breathes". If you pretend the samples are uniform, the spectrum smears and the peak shifts. So the first thing the signal goes through is linear interpolation onto a uniform 30 Hz grid:
/// Linear interpolation onto a uniform grid.
private func resample(samples: [PPGSample], to rate: Double) -> [Float] {Then it's DSP classics: detrending by subtracting a moving average (~3 seconds) removes slow brightness drift, and a Hamming window suppresses edge artifacts. The minimum-length gate is honest too — not "number of samples" but the actual duration of the window, robust to dropped frames: a threshold of 8.5 seconds, which at 30 Hz yields at least 256 resampled points for the FFT.
A spectrum instead of counting peaks#
The naive approach is to count beats as local maxima in the time domain. On a noisy camera signal this falls apart: a single finger movement produces "beats" that never happened. The spectral route is more robust: a real-FFT via vDSP (vDSP_fft_zrip → vDSP_zvmags) yields a power spectrum, and the peak is then searched only within the physiological band of 0.7–4.0 Hz — that's 42–240 BPM:
let bandStart = Int((Self.minBPM / 60.0) * Double(paddedLength) / Self.resamplingRate)
let bandEnd = min(spectrum.count - 1,
Int((Self.maxBPM / 60.0) * Double(paddedLength) / Self.resamplingRate))Everything outside the band — breathing, hand tremor, flickering lights — has no influence on peak selection. But finding the maximum isn't enough: you need to know whether it's a real peak or a random bump of noise. For that, SNR is computed — the ratio of the peak to the median of neighboring bins in the same band (the peak itself, with a ±2 bin neighborhood, is excluded):
floorBins.sort()
let floorValue = floorBins.isEmpty ? 1e-9 : Double(floorBins[floorBins.count / 2])
let snr = max(1.0, Double(peakValue) / max(floorValue, 1e-9))
guard snr >= Self.minSNR else {
return .failure(.lowSignalQuality)
}This is the key product decision: if SNR is below the threshold, the app returns a "weak signal" error and suggests trying again — instead of showing a made-up heart rate with a confident face. A bad measurement that's honestly rejected beats a pretty number you can't trust.
The final touch is parabolic interpolation of the peak across three bins. On a short window the FFT grid is coarse (~1.7 BPM per bin), and interpolation gives sub-bin frequency resolution almost for free.
A forehead in the frame: rPPG via Vision#
The second mode works with no finger and no flash: the front camera looks at the face (remote PPG). Blood pulsation subtly changes the skin's hue, and this response is strongest in the green channel. Vision is used for exactly one thing here — finding the face; there is no ML inference on top:
guard let face = request.results?.first,
face.boundingBox.width >= Self.minFaceWidth else {
return nil
}
// "Forehead" ROI: the upper part of the face (Vision: Y axis goes bottom-up → top = larger Y).
let bb = face.boundingBox
let roi = CGRect(
x: bb.minX + bb.width * 0.25,
y: bb.minY + bb.height * 0.68,
width: bb.width * 0.5,
height: bb.height * 0.20
)The ROI is built geometrically from the bounding box: a strip 20% of the face's height at forehead level, spanning the central 50% of its width — where the skin is exposed and eyes and hair interfere the least. The minFaceWidth = 0.15 gate cuts off "face too far away": on a tiny face the ROI degenerates into a handful of pixels.
From there it's the same pass over the raw BGRA buffer under CVPixelBufferLockBaseAddress, just with channel offset +1 (green) instead of +2, and the resulting signal goes into the same PPGProcessor. Privacy is identical to finger mode: a single number is extracted from the frame with a face — the green-channel mean — and the frame dies.
A caveat: rPPG is noticeably more temperamental than contact PPG. Head movement, changing lighting, a shadow on the forehead — all of these are noise of the same magnitude as the signal itself. The same SNR gate saves the day: a questionable measurement doesn't pass.
Baevsky stress — with a disclaimer#
The same signal yields more than just heart rate. From the detrended time-domain signal, RR intervals (the distances between beats) are extracted, and from them the Baevsky Stress Index: SI = AMo / (2·Mo·MxDMn) over an RR histogram with a 50 ms bin. The formula is classical, but its application comes with an honest disclaimer written right in the source:
// WARNING: classic Baevsky is computed over a 5-min recording (100+ beats). On a short PPG window
// (~15 s, ~15–20 beats) the value is INDICATIVE — good for trends, not for diagnostics.The classical methodology requires a five-minute recording; over ~15 seconds of PPG the index is an indicator for watching a trend, nothing more. If there are fewer than 12 valid RR intervals, the calculator returns nil — again, a refusal instead of a guess.
Four Things I Learned About the Camera Signal#
First: the camera is already a sensor, and there's no magic between "raw frame buffer" and "heart rate" — only a signal chain: one sample per frame → resampling onto a uniform grid → detrend → window → FFT → peak in the physiological band. Second: resampling is the step most often skipped, and it's exactly what separates a working FFT pipeline from a pretty one that lies. Third: signal quality is part of the API, not decoration: the SNR gate, plus the duration and face-size gates, turn "sometimes shows nonsense" into "either a number you can trust, or an explicit refusal". And fourth, the framing: all of this is a wellness estimate, not a medical measurement. The method is sensitive to motion and lighting, and the most mature thing such a feature can do is know its limits and tell the user about them plainly.
