# Download the Tutorial Data

Every later chapter assumes the tutorial files are already on disk under a
`data/` directory and refers to them by path alone. If a command in a later
chapter cannot find its input, the fix is here, not there.

The Atacama study behind tier 1 and the `ASV-1` … `ASV-13` labelling are covered in
[Atacama Soil Microbiome](02_datasets.md).

## Where the files go

Pick one project directory and stay in it. Every command in the tutorial is
written relative to that directory, with a `data/` prefix:

```bash
mkdir -p ~/q2-hdstats-tutorial/data
cd ~/q2-hdstats-tutorial
```

```{tip}
A few of the high-dimensional chapters show bare filenames
(`atacama-top-300-correlation.qza` rather than `data/atacama-top-300-correlation.qza`).
Either `cd data` before running those, or add the prefix. Nothing else changes.
```

## The Zenodo record

The Atacama artifacts (tiers 1 and 2) are published as a single Zenodo record,
**`q2-hdstats-tutorial-data` v1.0.0**. Publishing them rather than pointing at
the plugin repositories keeps the exact bytes citable and fixed.

```{note}
**The DOI has not been minted yet.** Wherever a base URL is needed below, the
literal placeholder `ZENODO_DOI_PENDING` stands in for it. The tier 1 and tier 2
commands therefore cannot be run as written yet — they carry one outstanding
substitution. Search the sources for `ZENODO_DOI_PENDING` to find every place
that has to change when the record goes live.
```

Set the base URL once and reuse it. Every tier 1 and tier 2 block below assumes
this variable is exported in your shell:

```bash
# Substitute the real record URL here once the DOI is minted.
export ZENODO_BASE=ZENODO_DOI_PENDING
```

## The manifest

`docs/_data/manifest.tsv` in this repository is the machine-checkable index of
everything the tutorial downloads. It has five tab-separated columns:

| Column | Meaning |
|---|---|
| `filename` | Basename as it lands on disk |
| `tier` | `1`, `2` or `3` — which part of the book needs it |
| `bytes` | Expected size |
| `sha256` | Expected checksum |
| `url` | Where it came from (Zenodo for both tiers) |

Prose can drift out of date. The manifest cannot, because it is what the
verification steps below read. Treat it as authoritative when the two disagree,
and open an issue.

Define this helper once — every tier below uses it to turn the manifest into a
checklist that `sha256sum -c` understands. It needs a checkout of this
repository; point `MANIFEST` at wherever yours is:

```bash
export MANIFEST=~/q2-hdstats-docs/docs/_data/manifest.tsv
tier_checklist () {
    awk -F'\t' -v tier="$1" 'NR > 1 && $2 == tier { print $4 "  " $1 }' "$MANIFEST"
}
```

```{note}
**Every `bytes` and `sha256` value in the manifest is real** — all 16 rows — and
each was verified against the file it describes. `tier_checklist N | sha256sum -c`
reports `OK` for every tier, so the verification steps below work today.

The `url` column for tiers 1 and 2 is the one entry still outstanding: it reads
`ZENODO_DOI_PENDING` until the record is minted. You can verify files you
already have, but the `curl` commands cannot fetch them yet.
```

## Tier 1 — Atacama, 13 ASVs

Every action in both plugins gets its one canonical demonstration on this table,
which is small enough to print in full.

```bash
cd ~/q2-hdstats-tutorial/data
curl -L -O "${ZENODO_BASE}/atacama-counts.qza"
curl -L -O "${ZENODO_BASE}/classification.qza"
curl -L -O "${ZENODO_BASE}/selected-atacama-sample-metadata.tsv"
curl -L -O "${ZENODO_BASE}/atacama-selected-covariates-veg.tsv"
```

| File | What it is |
|---|---|
| `atacama-counts.qza` | `FeatureTable[Frequency]`, 13 ASVs × 50 samples. Feature IDs are MD5 hashes; the `ASV-1` … `ASV-13` labels used throughout the tier-1 chapters are a *presentational* renaming applied in the book, not something a command emits — the mapping table is in [Atacama Soil Microbiome](02_datasets.md). The tier-1 `transform-features` call keeps the real IDs, as the default `--p-keep-original-id` implies |
| `classification.qza` | `FeatureData[Taxonomy]` for those 13 ASVs |
| `selected-atacama-sample-metadata.tsv` | Sample metadata passed to `qiime gglasso transform-features` |
| `atacama-selected-covariates-veg.tsv` | The five covariates used by q2-classo: `ph`, `elevation`, `average-soil-relative-humidity`, `average-soil-temperature`, `vegetation` |

Verify:

```bash
tier_checklist 1 > ~/q2-hdstats-tutorial/data/SHA256SUMS.tier1
cd ~/q2-hdstats-tutorial/data && sha256sum --ignore-missing -c SHA256SUMS.tier1
```

`--ignore-missing` matters: it lets you verify a partial download against a
checklist that also names files you deliberately skipped, instead of failing on
their absence.

You need `classification.qza` even for the pure network chapters:
`qiime gglasso transform-features` declares `--i-taxonomy` as a required input
although its function body never reads it. See
[Troubleshooting](../90_reference/04_troubleshooting.md) for that and the other
registration warts.

```{note}
**On the sample count.** The tier 1 table is 13 features × 50 samples, read
directly from the artifact. Earlier drafts said 49 in prose while every command
passed `--p-n-samples 50`; the commands were right. The metadata TSV has 75 rows
because it covers the full Atacama tutorial, of which 50 samples appear in this
table.
```

## Tier 2 — Atacama, 300 ASVs

The same study, scaled up: the 300 most abundant ASVs, which demonstrate model
selection and latent-rank choice in a regime where you cannot eyeball the matrix.

```bash
cd ~/q2-hdstats-tutorial/data
curl -L -O "${ZENODO_BASE}/atacama-top-300-table.qza"
curl -L -O "${ZENODO_BASE}/atacama-top-300-clr.qza"
curl -L -O "${ZENODO_BASE}/atacama-top-300-correlation.qza"
curl -L -O "${ZENODO_BASE}/atacama-taxonomy-silva138.qza"
curl -L -O "${ZENODO_BASE}/sample-metadata.tsv"
curl -L -O "${ZENODO_BASE}/top-300-asvs.tsv"
curl -L -O "${ZENODO_BASE}/atacama-classo-outcomes-mean-imputed.tsv"
```

| File | What it is |
|---|---|
| `atacama-top-300-table.qza` | `FeatureTable[Frequency]`, the top-300-ASV counts |
| `atacama-top-300-clr.qza` | The transformed table, shipped so you can skip the transform step |
| `atacama-top-300-correlation.qza` | `PairwiseFeatureData` — the direct input to `qiime gglasso solve-problem` |
| `atacama-taxonomy-silva138.qza` | `FeatureData[Taxonomy]` for the 300 ASVs |
| `sample-metadata.tsv` | Full sample metadata, including `transect-name` (Baquedano, Yungay) and `vegetation` (yes/no) — the two natural grouping variables for the multiple-graphical-lasso and PCA chapters |
| `top-300-asvs.tsv` | Total abundance per feature, keyed on `feature-id`. Useful for filtering or for reporting how abundant a hub is. Its `abundance-rank` column is not an identifier: 209 of the 300 features are tied on total abundance, so the rank order within a tie is arbitrary. Do not use it to map `ASV-k` labels back to features — see [Interpretation](../04_highdim_atacama/06_interpretation.md) |
| `atacama-classo-outcomes-mean-imputed.tsv` | The regression outcomes with missing values mean-imputed, as used by the q2-classo cross-validation chapter |

Verify:

```bash
tier_checklist 2 > ~/q2-hdstats-tutorial/data/SHA256SUMS.tier2
cd ~/q2-hdstats-tutorial/data && sha256sum --ignore-missing -c SHA256SUMS.tier2
```

Recomputing the transformed table and the correlation matrix from
`atacama-top-300-table.qza` is the first exercise of the tier 2 chapters, and
holding the reference versions on disk makes a mismatch visible immediately
rather than letting it propagate into the network.

## The cocoa appendix has no download

No feature table is published for the MOSHPIT cocoa fermentation example
(14 shotgun metagenomes, BioProject PRJNA552479), so there is nothing to fetch.
Reproducing one means running assembly and binning and holding local
Kraken2/Kaiju databases. See
[Appendix: Shotgun Metagenomics (MOSHPIT cocoa)](../99_appendix/02_moshpit_cocoa_note.md)
for what that involves before you commit compute to it.

## Verifying the download

A truncated `.qza` is still a valid ZIP prefix. QIIME 2 will often load it
without complaint, and the failure surfaces much later, inside a solver, as
something that looks like a numerical problem rather than a broken file. Ten
seconds of `sha256sum` saves an afternoon of debugging the wrong layer.

Once the checksums pass, confirm that QIIME 2 reads the artifacts as the types
the tutorial expects:

```bash
cd ~/q2-hdstats-tutorial
qiime tools peek data/atacama-counts.qza
qiime tools peek data/classification.qza
```

```{note}
The expected `qiime tools peek` output — UUID, semantic type and format — is
pending verification against QIIME 2 2026.7. Most of this book has not yet been
re-run against that release, and the UUID differs with every download, so no
captured output accompanies the commands above.
```

## The resulting tree

After both tiers:

```text
~/q2-hdstats-tutorial/
└── data/
    ├── atacama-counts.qza
    ├── classification.qza
    ├── selected-atacama-sample-metadata.tsv
    ├── atacama-selected-covariates-veg.tsv
    ├── atacama-top-300-table.qza
    ├── atacama-top-300-clr.qza
    ├── atacama-top-300-correlation.qza
    ├── atacama-taxonomy-silva138.qza
    └── sample-metadata.tsv
```

Everything else the tutorial uses is derived: each chapter writes its
transformed tables, covariance matrices, solutions and visualizations back into
`data/`, so the directory grows as you work through the book. Only the files
above have to be fetched.

```{note}
The tier 2 chapters also build a design-matrix table and a mean-imputed outcomes
TSV for the q2-classo section. Whether those ship on the Zenodo record or are
produced by the chapter commands is being settled as the record is assembled.
The manifest will say which.
```

## Next

With the data in place, continue to
[Prerequisites & Installation](../01_installation/01_prerequisites.md), then
[Verifying Your Installation](../01_installation/04_verify.md) — the last step of
which reads one of the artifacts you just downloaded. To see which chapter
demonstrates which command before you start, the
[Command Coverage Matrix](../90_reference/01_command_coverage.md) is the map.