Monday, October 24, 2022

alphaHRV vs Garmin resp rate - David vs Goliath

Continuing with the theme of respiratory rate tracking, this post will explore the brand new bonus feature of alpha HRV that tracks this metric.  Since Garmin has their own methodology in doing this, we will be looking at how they do side by side.  This will be a very interesting comparison for several reasons - 

  • They are deriving resp rate from identical RR intervals (in this case the Polar H10) 
  • Use the same recording device (Garmin Epix 2 in my case)
  • The Garmin implementation is backed by their Firstbeat division, a professional group of engineers with a substantial budget and resources.
  • Alpha HRV is written by two young programmers who wrote the respiratory rate code in about 2 weeks in their spare time.

So truly, a David vs Goliath contest. 

The testing details: The subject (me) wore the Hexoskin vest as the "gold standard" resp rate device.  This measures resp rate through chest/abdominal wall motion sensors and is quite accurate.  I also used the Movesense Medical ECG given it's great track record for ECG derived resp rate.  Lastly, the Polar H10 was worn along with the Movesense belt, recorded to the Garmin Epix 2 watch.  The watch was running alphaHRV in the cycling activity mode.  This way, both alphaHRV and the Garmin algorithms would do their own computation of resp rate based on RR intervals from the same device.

The test consisted of a brief warm up then a cycling ramp to near failure (15w/min rise using Zwift) followed immediately by 90 min in zone 1.  In this way we have a handle on ramp resp rate behavior for those who are interested in using this for zone demarcation purposes as well as a steady state long interval to demonstrate how the apps handle stable intensities.  AlphaHRV was set to a smoothing of soft and a window of 45 seconds.  Kubios was set to a window of 30 seconds:


As one can see from the above, the post ramp HR (blue) was remarkably steady for the next 90 min, no real sign of drift.

Post ramp results:

Garmin (using H10 based RRs via bluetooth, artifact<1%):

  • In all the tests, the black line will be the Hexoskin displayed as a 30s moving average.  The red is HR, Green is Garmin resp rate.
  • The Garmin substantially underestimates resp rate, but is stable throughout.

AlphaHRV:


  •  Much better than Garmin, with alphaHRV (light blue) following the pattern nicely.

Kubios using RRs from the H10 (just like Garmin and alphaHRV does):

  • For the most part, good tracking with a couple of areas of "blips" where correlation is poor.

Kubios + Movesense ECG (using ECG voltage and RRs):

  • The Movesense (yellow) very closely follows the Hexoskin and continues to show why this combo is yields great ECG derived resp rate results.
 

The Ramp (Hexoskin in black):

Garmin (green):

  • Both the absolute numbers and shape (drop near the end) do not match the Hexoskin.

Alpha HRV:

  • Both the general shape and absolute values are closer to the Hexoskin than Garmin, but there are some blips along the way.
  • These "blips" will degrade the correlations and threshold curves.

Kubios using H10 RRs:

  • A good showing here for Kubios, with a reasonable shape and overall behavior.
  • There was a bit of a drop in the middle that could mess up threshold determinations.

Kubios + Movesense ECG:

  • Movesense ECG with Kubios matches the Hexoskin very well. 
  • Excellent shape and correlation.

Some comparison stats:

Correlation - here we plot Hexoskin on the X axis and each challenger on the Y.  The higher the R squared, the better the fit/matching.

Garmin:


AlphaHRV:


Kubios RR intervals:


Kubios + Movesense ECG:


Interesting observations:

  • The Movesense ECG + Kubios has a superb correlation coefficient (.81).
  • Both AlphaHRV and Garmin have moderate correlations.
  • Kubios RR derived resp rate is midway between the ECG and AlphaHRV.
  • This is an great example why a simple Pearson correlation coefficient or equivalent is not sufficient for comparison studies - you can have good correlation but a substantial deviation in the difference in values (aka, the Bland Altman analysis).  If we do a Bland Altman plot, the Garmin would have had a huge bias:
The Garmin data is shifted down about 12 breaths/min with very wide scatter - both facts attesting to it's poor agreement tot he Hexoskin.
 

Thus far we can see that the alphaHRV app does do a much better job of tracking resp rate than Garmin, but it's not up to the precision of Kubios RR methodology nor the ECG derived resp rate.

What about ventilatory thresholds?

The last type of analysis I would like to look at is how the ramp values looks when plotted as a 6th order polynomial.  Why?  A key study done by Cross et al. uses a 6th order polynomial to help calculate the first and second ventilatory thresholds using resp rate data.  Here is a figure from the study with a red arrow pointing to the 6th order curve:

  • The idea is to figure where the slope maximizes at the 2 threshold points.  See the citation for details.

How do our various recording methods stack up with 6th order plotting?  I'm not going to look at the Garmin since the values have such an enormous bias.

Hexoskin (gold standard):



AlphaHRV:

Kubios RR:

  • Unfortunately, both the AlphaHRV and Kubios RR curve shapes do not conform to the Hexoskin shape.

What about the Kubios + Movesense ECG?


  • Wow - pretty close to the Hexoskin!
  • I continue to be impressed with this combo of equipment and software.
  • If one were to try to derive VT1 or 2 from resp rate data, the Movesense ECG + Kubios would be the way to go.  With the other methods YMMV.

Some conclusions:

  • Garmin's resp rate algorithm is not very accurate.  The bias is very high and correlation modest despite high quality RR recordings.
  • AlphaHRV tracks resp rate well, especially at steady state conditions.  It potentially can do well during a ramp (from what I've seen of other examples) but in the above case, it was not good enough for threshold estimation based on published models.
  • We again see that the combo of the Movesense ECG with Kubios software is both able to track resp rate at steady state conditions and do very well during incremental ramps.  It can also reproduce the 6th order polynomial plot for threshold estimation.  However, there is a cost associated with this combo that is not trivial.
  • One wonders, what is going on with the Garmin/Firstbeat division.  These programmers/scientists are supposed to be top notch, have years of development under their belt, large financial and testing resources.  How can they release such a flawed product?  It also makes one not trust their other metrics and claims (training readiness, body battery, sleep scores etc).  Although it's impressive that a couple of young programmers working in their spare time have easily surpassed them, I think it's shameful that Garmin puts out this type of software quality.  
 
  • Bottom line - David beats Goliath.

 









4 comments:

  1. As from reading the blog text it appears as if the headings for the Corralation figures for "Garmin Resp" and "AlphaHRV Resp" are swapped. According to the correlation in the figures Garmin does a better job than AlphaHRV whereas the text sys the way around.

    ReplyDelete
    Replies
    1. The stats are correct - but as I mentioned above, this is a great example of why "correlation" alone is a poor judge of "agreement". That's why we do the Bland Altman "difference" plot which I added above. You can see the huge degree of bias and the wide "limits of agreement" which is 2x the standard deviation.

      Delete
  2. Thank you for this study.

    I got flu with high temperatures several times while using Garmin Fenix 7S. All the time Garmin shows no changes in Respiratory rate, which is strange. Same time my Oura shows a rise in respiratory rate.

    Posted it here https://forums.garmin.com/outdoor-recreation/outdoor-recreation/f/fenix-7-series/321281/shame-for-garmin-accuracy-for-respiratory-rate . Hope Garmin will fix it's algorithms

    ReplyDelete
  3. Thanks for this data. As a Garmin user I'm very disappointed. If I were an exec there I'd surely poach the two alphaHRV coders even on a contract basis to improve the underlying algorithms.

    ReplyDelete