--- title: "Alpha diversity and rarefaction" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Alpha diversity and rarefaction} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( dev.args = list(bg = "white"), collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 4 ) ``` ```{r setup} library(tidydiversity) library(dplyr) library(ggplot2) set.seed(42) ``` ## The concept **Alpha diversity** is the diversity within one sample or site. Counting species is the obvious measure, but the count depends on how hard you looked: a second day of sampling always adds species. Two ideas solve this: * **Hill numbers** put richness, Shannon and Simpson diversity on one scale, the *effective number of species*: the number of equally common species that would give the same diversity. The order `q` sets how much rare species count (q = 0 all equally, q = 1 in proportion to their frequency, q = 2 mostly the common ones). * **Sample coverage** is the estimated proportion of the community's individuals (or occurrences) that belong to species already in the sample. It is estimated from the singletons (`f1`, species seen once) and doubletons (`f2`): many singletons mean many species still unseen. Comparing sites at **equal coverage** compares them at equal completeness, which is fairer than comparing at equal sample size (Chao and Jost 2012). The examples use `png_ants`: ants recorded at nutrient baits along the Mt Wilhelm elevational gradient in Papua New Guinea, in the dry and wet seasons (Moses et al. 2023). Each bait at which a species was found counts as one occurrence. ## Hill numbers Hill numbers express diversity as an effective number of species. Order `q = 0` is richness, `q = 1` weights species by their frequency and `q = 2` emphasises the dominant species. ```{r} div_hill(png_ants, q = c(0, 1, 2), .by = season) ``` **How to read it.** `n` is the number of occurrences, `diversity` the observed Hill number. Diversity always falls from q = 0 to q = 2; the steeper the fall, the more uneven the community (a few species dominate). At 200 m in the dry season, 16 species are equivalent to 12.3 equally common ones (q = 1) and 9.9 dominant ones (q = 2): a fairly even community. ## Sample coverage Raw counts depend on sampling effort. Sample coverage estimates how complete each sample is. ```{r} div_coverage(png_ants, .by = season) ``` **How to read it.** `s_obs` is the observed richness, `f1` and `f2` the singletons and doubletons, and `coverage` the estimated completeness. Most samples are 70-95% complete, but the dry-season sample at 700 m reaches only 48%: 12 of its 15 species were found once. Its observed richness is therefore the least reliable and should not be compared directly with the others. ## Asymptotic estimates `div_estimate()` gives Chao1 richness, the Chao et al. (2013) Shannon estimator and the unbiased Simpson estimator, with bootstrap intervals. ```{r} div_estimate(png_ants, .by = season, nboot = 50) |> filter(q == 0) ``` **How to read it.** `estimate` is the estimated true richness (Chao1), a lower bound, with a bootstrap interval. Where coverage is high (200 m dry, 1700 m dry) it is close to the observed richness. Where it is low, the estimate rises far above the observation and the interval becomes very wide: at 700 m in the dry season, 15 observed species suggest 84 (interval 45-176). Such estimates show that many species are missing, but they are too uncertain to rank sites; use standardised estimates instead. ## Classic estimators (EstimateS and SpadeR) Many studies, and many reviewers, still report the estimators of EstimateS: ACE and ICE, the jackknives, the bootstrap, and the improved Chao estimators. `div_richness()` gives all of them, each with a bootstrap standard error and interval, and the values equal SpadeR's. Here, for the incidence data (baits as sampling units) at 700 m: ```{r richness} set.seed(1) rich <- div_richness(png_ants, unit = bait, n_units = baits, .by = season, nboot = 100) rich |> filter(site == "MW0700") |> select(season, estimator, observed, estimate, se, lower, upper) ``` **How to read it.** Every estimator agrees that many species were missed at 700 m, but they disagree strongly on how many: in the dry season, from `r round(rich$estimate[rich$site == "MW0700" & rich$season == "dry" & rich$estimator == "bootstrap"])` (bootstrap) to `r round(rich$estimate[rich$site == "MW0700" & rich$season == "dry" & rich$estimator == "ice1"])` (ICE-1). That spread is itself the result: with 15 species, most found at one bait, the data cannot pin down the true richness. The bootstrap and first-order jackknife are known to underestimate when sampling is this incomplete, and ACE-1 and ICE-1 to be unstable (note the huge interval). Report Chao2 (or iChao2 when rare species dominate) as the lower bound, and say that the sample was too incomplete for a precise estimate. Fisher's alpha, the log-series index common in tropical studies, is less sensitive to sample size than richness: ```{r fisher} div_fisher_alpha(png_ants, .by = season) ``` Like the coverage-standardised richness below, it is highest at 700 m and lowest at 1700 m and above in both seasons. ## How much more sampling is needed? `div_effort()` turns the estimates into a plan: how many more sampling units (here baits) each site needs to reach a target coverage, or to detect a target fraction of its estimated richness (Chao et al. 2009). ```{r effort} effort <- div_effort( png_ants, unit = bait, n_units = baits, .by = season, coverage = 0.9, fraction = 0.9 ) effort |> select(season, site, coverage_now, s_obs, s_est, target, additional) ``` **How to read it.** `coverage_now` is the completeness of the 60 baits already set, `s_est` the estimated richness (Chao2), and `additional` the further baits needed for each target. For 90% coverage, most sites need fewer than 75 more baits, and two are already there or nearly so. The exception is 700 m, the richest elevation: it needs `r effort$additional[effort$season == "wet" & effort$site == "MW0700" & effort$target == "coverage"]` more baits in the wet season and `r effort$additional[effort$season == "dry" & effort$site == "MW0700" & effort$target == "coverage"]` in the dry season, where 12 of its 15 species were found at one bait. A far larger effort, or a different method, would be needed there. Answers far beyond two to three times the current effort are order-of-magnitude guides, because extrapolation that far is uncertain. ## Rarefaction and extrapolation `div_rarefy()` interpolates each sample down to smaller sizes and extrapolates up to twice its size. The values match the iNEXT package. ```{r} curves <- png_ants |> filter(season == "wet") |> div_rarefy(q = c(0, 1, 2), nboot = 30) curves ``` ```{r, fig.alt = "Rarefaction and extrapolation curves for each site"} autoplot(curves) ``` **How to read it.** Solid lines are interpolated (rarefied) diversity for smaller samples, the point marks the actual sample, and dashed lines are extrapolated up to twice the sample size. A curve still rising steeply at its end has many species left to find; a curve that flattens is nearly complete. Curves for q = 1 and 2 flatten much sooner than for q = 0, because rare species, which are the last to be found, count little in them. Plotting against coverage shows how complete each sample is: ```{r, fig.alt = "Diversity plotted against sample coverage"} autoplot(curves, x = "coverage") ``` Read the curves at the same position on the x-axis to compare sites at equal completeness. ## Fair comparisons Compare sites at equal completeness with `div_standardise()`. With no level given, each group is standardised to the lowest coverage that its sites reach at twice their sample size. ```{r} std <- div_standardise(png_ants, .by = season, nboot = 50) std ``` **How to read it.** All sites of a season are brought to one coverage (`r round(min(std$coverage[std$season == "dry"]), 3)` in the dry season and `r round(min(std$coverage[std$season == "wet"]), 3)` in the wet season): `method` says whether a site was rarefied down or extrapolated up to reach it, and `m` the sample size at which this happens. These estimates, not the observed counts, are the ones to compare. ```{r, fig.alt = "Coverage-standardised richness by elevation"} std |> left_join(distinct(png_ants, site, elevation), by = "site") |> filter(q == 0) |> ggplot(aes(elevation, diversity, colour = season)) + geom_pointrange(aes(ymin = lower, ymax = upper)) + geom_line() + labs(x = "Elevation (m)", y = "Richness at equal coverage") + scale_colour_tidydiversity() + theme_tidydiversity() ``` **Interpretation.** At equal coverage, ant richness peaks at 700 m in both seasons and is lowest above 1700 m. Each site is richer in the wet season, when more ants forage at baits. The 700 m estimates were extrapolated and have the widest intervals, so the peak is clear but its height uncertain. The observed counts alone would have hidden the peak in the dry season, where 700 m had the least complete sample. ## How to report it > Ant richness was compared among elevations at equal sample coverage > (Chao and Jost 2012), separately for each season, with tidydiversity. > Samples were rarefied or extrapolated to the lowest coverage reached at > twice the observed sample size > (`r round(min(std$coverage[std$season == "dry"]), 2)` in the dry and > `r round(min(std$coverage[std$season == "wet"]), 2)` in the wet > season); 95% intervals are from 50 bootstrap replicates. Report the > sample coverage of each site in a supplementary table. ## References Chiu, C.-H., Wang, Y.-T., Walther, B. A. and Chao, A. (2014). An improved nonparametric lower bound of species richness via a modified Good-Turing frequency formula. *Biometrics*, 70, 671-682. Chao, A., Colwell, R. K., Lin, C.-W. and Gotelli, N. J. (2009). Sufficient sampling for asymptotic minimum species richness estimators. *Ecology*, 90, 1125-1133. Chao, A. and Jost, L. (2012). Coverage-based rarefaction and extrapolation: standardizing samples by completeness rather than size. *Ecology*, 93, 2533-2547.