How Machines Find the Beat, and Where They Still Fail
Automatic beat tracking is one of the oldest problems in music information retrieval. Reading its failure cases is one of the better ways to understand what human listeners are doing.
Short answer: a beat tracker takes an audio file and returns a list of times where the beats are. It has to solve two coupled problems, the period and the phase, and getting either one wrong ruins the output. Classic systems reduced the audio to an onset strength curve and searched for a periodic sequence in it. Current systems train neural networks to predict beat likelihood directly from a spectrogram. They are very good on steady, percussive music, and they still lose the thread on expressive classical playing, which is precisely where human listeners do not.
Two questions, one answer
Ask where the beats are and you are really asking two things. Period is how far apart consecutive beats sit, which corresponds to the tempo. Phase is where the sequence is anchored in time. A system that gets the period exactly right and the phase off by half a beat produces output that lands on every offbeat, which is worse than useless for anything downstream.
Human listeners solve both at once and mostly without noticing. Our guide to beat entrainment covers what that feels like from the inside; this article is about what it takes to do the same thing from a waveform.
The classic pipeline
For roughly two decades the standard architecture had three stages, and it is worth knowing because the neural systems still contain recognisable versions of all three.
Onset detection. The audio is converted into a one-dimensional signal that rises when something new begins. The usual trick is to compute a spectrogram and measure how much energy increased in each frequency band between frames, then sum the increases. Meinard Müller's Fundamentals of Music Processing, published by Springer in 2015, is the standard textbook treatment of this whole chain.
Periodicity estimation. The onset signal is analysed for repeating structure, classically with autocorrelation or a bank of comb filters, which yields one or more candidate tempi.
Beat sequence selection. Given a candidate tempo, the system chooses actual beat times. Dan Ellis published a particularly clean solution in the Journal of New Music Research in 2007. He defined a score for an entire candidate beat sequence, rewarding beats that land on strong onsets and penalising intervals that stray from the target period, then observed that the highest-scoring sequence can be found exactly, and efficiently, by dynamic programming. It is a good example of a hard-looking search that becomes easy once the objective is written down properly.
Later work in this tradition improved robustness by combining evidence rather than committing early. Zapata, Davies and Gómez described a multi-feature approach in IEEE/ACM Transactions on Audio, Speech, and Language Processing in 2014, running several beat trackers over different input features and selecting the output the others most agreed with.
What neural networks changed
The modern pipeline replaces hand-designed onset detection with a network trained on annotated music. The influential design paired a recurrent network, which outputs a beat likelihood for each frame, with a dynamic Bayesian network that turned those likelihoods into a beat sequence consistent with a plausible tempo and meter. Böck and colleagues packaged this family of models into the open-source library madmom, described in a 2016 paper, which is why so much later work uses it as a baseline.
The architecture then moved on. Davies and Böck showed at EUSIPCO in 2019 that temporal convolutional networks work well for the task. Hung and colleagues presented a time-frequency transformer for beats and downbeats at ICASSP in 2022. Most recently, Foscarin, Schlüter and Widmer proposed a system in 2024 that goes after the two weaknesses of the previous generation directly: they trained on a deliberately diverse collection, including solo instrument recordings, pieces with time signature changes, and classical music with large tempo variation, and they removed the dynamic Bayesian postprocessing entirely, on the grounds that it imposes constraints on meter and tempo that the hard cases violate.
That last decision is telling. The postprocessing step existed to make outputs musically plausible. It also made the system worse on exactly the music that is least metrically rigid.
How beat tracking is scored, and why the rules look generous
Evaluation in this field forgives things that look like errors. Estimated beats count as correct if they fall within a tolerance window around the annotation rather than exactly on it, and the widely used continuity measures also accept a sequence at double or half the annotated rate, or offset onto the offbeat.
This is not laziness. McKinney, Moelants, Davies and Klapuri set out the reasoning in their evaluation study in the Journal of New Music Research in 2007: the ground truth itself is ambiguous, because the question "what is the beat here" does not have a single answer that all listeners give.
Where machines still lose the beat
| Hard case | Why it breaks a tracker | Why people cope |
|---|---|---|
| Expressive classical performance | Tempo moves continuously; a periodicity estimate is always slightly out of date | Listeners predict from phrase structure and prior knowledge of the piece |
| Rubato and fermatas | The period assumption is violated by design | The metrical expectation survives the interruption |
| Sparse or non-percussive texture | Weak onsets give the detector little to work with | Melodic and harmonic accents carry the metre instead |
| Syncopated material | The strongest onsets are deliberately off the beat | People hear syncopation against a felt beat, not as the beat |
| Time signature changes | Meter-constraining postprocessing actively fights the music | Listeners re-anchor within a bar or two |
| Very slow or very fast music | Several metrical levels are equally defensible | People also disagree here, which is the honest answer |
The last row deserves emphasis, because it is the one place where the machine is not really failing. It is reproducing a genuine human ambiguity.
What the failures say about human listening
Two things, mainly.
First, human beat finding is not purely bottom-up. A signal-driven tracker has to infer the beat from the evidence in front of it. A listener carries a metrical expectation and interprets incoming events against it, which is why a ritardando that destroys a periodicity estimate is trivially easy to follow by ear. Our guide to the neural mechanisms of rhythm covers the oscillator and prediction accounts of how that expectation is maintained.
Second, the metrical level really is undetermined by the signal. McKinney and Moelants asked what draws listeners to different metrical levels, in Music Perception in 2006, and found that the same excerpt reliably produces tapping at more than one rate across listeners, with the chosen levels clustering in a preferred region rather than converging on a single answer. Our guide to your internal clock and natural tempo looks at where that preferred region sits.
Put those together and the picture is clear enough. When a beat tracker reports half tempo, it has not necessarily made a mistake. It has made a choice that some human listeners would also make, without the ability to say so.
If you want to try this yourself
Check the octave. The single most common failure you will meet in practice is a tempo estimate at half or double the rate you wanted. Read the number, then tap along and see whether it matches.
Give the tool the easy version first. If you are extracting tempo from a live recording with an unstable intro, trim to a steady section. You are not cheating; you are removing the part the algorithm has no principled way to handle.
Convert once you have a number. Once you know the BPM, our BPM to milliseconds calculator turns it into the delay and note-duration values you actually need.
And a note on this site's games. When you play the daily tempo match, you are doing what the algorithm does: estimating a period from a short audio excerpt and committing to a number. The difference is that you get to hear yourself be wrong.
Estimate a tempo the way an algorithm has to. Chronkle plays today's tempo and gives you five tries to match it. It is the same two-part problem a beat tracker solves, minus the spectrogram.
Frequently asked questions
What does a beat tracker actually compute?
Two things at once: the period, meaning how far apart the beats are, and the phase, meaning where in time they land. Getting one right and the other wrong produces output that is completely unusable, which is why the two are almost always estimated jointly rather than in sequence.
How did beat tracking work before neural networks?
Through a pipeline. The audio was reduced to an onset strength signal that peaks when something new starts, periodicity analysis proposed a tempo, and a search picked a beat sequence. Ellis published an elegant version in the Journal of New Music Research in 2007 that scores a whole candidate sequence for both onset strength and tempo consistency, and finds the best one exactly by dynamic programming.
Why do beat trackers get the tempo wrong by a factor of two?
Because the question is genuinely ambiguous. McKinney and Moelants reported in Music Perception in 2006 that human listeners hearing the same excerpt tap at different metrical levels. Evaluation protocols therefore usually accept a doubled or halved rate as correct, which is a concession to the ambiguity rather than a loophole.
Are modern beat trackers solved?
On steady, percussive, commercially produced music they are very strong. The remaining hard cases are expressive classical performance with large tempo changes, sparse or non-percussive textures, rubato, and time signature changes. Foscarin, Schlüter and Widmer targeted exactly that range in their 2024 system by training on diverse data and dropping the meter-constraining postprocessing step.
What does machine failure tell us about human hearing?
Mostly that human beat finding is not bottom-up. People hold a metrical expectation and hear incoming events against it, which is why a listener can follow a ritardando that breaks a purely signal-driven tracker, and why the same listener can be quite unsure which level counts as the beat.
Can I use this on my own music?
Yes. Open-source libraries such as madmom implement published models, and most digital audio workstations include tempo detection built on similar ideas. Just check the result against your ears, because a confident wrong answer at half tempo looks exactly like a confident right one.
Keep reading
- Beat entrainment: what the human version of this process looks like.
- Tap tempo: the manual method, and why it is still competitive on hard material.
- Your internal clock: the preferred tempo region that shapes which level you hear as the beat.
Sources: Ellis, D. P. W. (2007), Beat tracking by dynamic programming, Journal of New Music Research; McKinney, M. F., Moelants, D., Davies, M. E. P. and Klapuri, A. (2007), Evaluation of audio beat tracking and music tempo extraction algorithms, Journal of New Music Research; McKinney, M. F. and Moelants, D. (2006), Ambiguity in tempo perception: what draws listeners to different metrical levels?, Music Perception; Zapata, J. R., Davies, M. E. P. and Gómez, E. (2014), Multi-feature beat tracking, IEEE/ACM Transactions on Audio, Speech, and Language Processing; Böck, S., Korzeniowski, F., Schlüter, J., Krebs, F. and Widmer, G. (2016), madmom: a new Python audio and music signal processing library; Davies, M. E. P. and Böck, S. (2019), Temporal convolutional networks for musical audio beat tracking, EUSIPCO; Foscarin, F., Schlüter, J. and Widmer, G. (2024), Beat this! Accurate beat tracking without DBN postprocessing; Müller, M. (2015), Fundamentals of Music Processing, Springer.