Internals

All cycles to search by default (regular years + pre-pandemic).

_apply_labels!(df::DataFrames.DataFrame, table::AbstractString)

Apply value labels to the categorical columns of a DataFrame. Continuous columns keep their numeric values. Uses cached labels, no redundant downloads.

_build_codebook_from_data(df::DataFrames.DataFrame, variable::Symbol, entry) -> DataFrame

Build codebook DataFrame with counts computed from actual data.

Emits one row per row CDC publishes, in order. Values matching no published code are appended so the counts account for every observation.

_code_sort_key(code::AbstractString)

Sort key placing numeric codes in numeric order ahead of non-numeric ones.

Directory part of a URL path, including the trailing slash.

Build DXA table name.

Build URL for DXA data file.

_fetch_pages(f, components, cycles) -> Vector{Tuple}

Apply f(component, cycle) to every pair, keeping up to SEARCH_CONCURRENCY fetches in flight, and return (component, cycle, result) triples in components by cycles order however the fetches interleave. Pairs CDC does not publish are dropped, as in _skip_missing. A failure surfaces with its original type once the fetches in flight settle, and stops the remaining pairs from being fetched.

Format numeric code for label lookup.

_get_cached_variablelist(component::Symbol, year::Union{Int,Symbol};
                         force::Bool=false) -> Vector{Dict}

Get variable list for a component/year, using cache if available.

_get_table_labels(table::AbstractString; force::Bool=false) -> Dict

Get the parsed value table for every variable in a data table, using cache if available. Returns Dict mapping variable name -> value table, as built by _parse_all_value_labels.

_historical_cache_path(survey::Symbol, name::AbstractString, extension::AbstractString) -> String

Cache file path for a historical data file, keeping the extension CDC publishes it under.

Throws

• ArgumentError: If the name is not a plain identifier, or the extension is not one CDC publishes data under

_historical_entry(survey::Symbol, listing::DataFrames.DataFrame, name::AbstractString) -> NamedTuple

Find name in a survey listing, reporting its name, URL and file extension. Lookup ignores case because listing names are always uppercase.

Throws

• TableNotFoundError: If the listing holds no file of that name

Path of the cached copy of entry, downloading it first when needed.

_historical_tables_url(survey::Symbol) -> String

Get URL for historical survey tables list. NHANES I/II use default.aspx, NHANES III uses datafiles.aspx.

_is_blank(val) -> Bool

Report whether a data value is absent. A character column records a blank string where a numeric column records missing.

_is_historical_data_file(filename::AbstractString) -> Bool

Report whether a linked file holds survey data. Readme files sit next to the data files and share their extension, so they are excluded by name.

_is_missing_row(code::AbstractString) -> Bool

Report whether a value table row counts missing observations rather than a code.

_is_range_row(code::AbstractString, label::AbstractString) -> Bool

Report whether a value table row describes a range of measurements rather than a single code. CDC marks these rows with the description "Range of Values" and a code cell such as "3.2 to 242.6". A character variable's recorded values are also ranges: their code cell repeats the variable description.

Collapse . and .. segments in an absolute URL path.

_parse_all_value_labels(html::AbstractString) -> Dict{String,Any}

Parse the value table of every variable on a codebook page.

Returns Dict mapping variable name -> Dict with:

• "continuous": true when the variable has a range row, so its values are measurements rather than codes
• "labels": Dict(code_string -> label) for code rows only
• "rows": every published row, in order, as Dict("code" =>, "label" =>)

_parse_historical_tables(html::AbstractString, page_url::AbstractString) -> Vector{Dict}

Parse a historical survey listing page. Names are derived from the data file names, which are more reliable than the surrounding description cells. Links are resolved against page_url.

_parse_tables_from_variablelist(html::AbstractString) -> Vector{Dict}

Extract unique table information from a variable list page. Each variable on the page names the data file it belongs to.

_parse_variables_from_codebook(html::AbstractString) -> Vector{Dict}

Extract variable names and labels from a codebook page.

Reject a listing entry that historical_download cannot read.

_search_components(component::Union{Symbol,Nothing}) -> Vector{Symbol}

Components a search covers: every component, or just the one requested.

_search_cycles(years::Union{Int,AbstractRange,Nothing}) -> Vector{Union{Int,Symbol}}

Cycles a search covers, in survey order, dropping years CDC has not published.

_search_results_to_dataframe(results::Vector) -> DataFrame

Convert search results to a DataFrame.

_skip_missing(f)

Run f, returning nothing when CDC publishes no page for the request. Every other failure propagates, so a network outage or a markup change is not mistaken for an empty result.

Split a URL path from the query string or fragment that follows it.

_tables_to_dataframe(data::Vector) -> DataFrame

Convert parsed tables data to a DataFrame. Includes URL column if present in data.

_translate_column(col, labels_map) -> (Vector{Union{String,Missing}}, Set{String})

Replace codes with labels, keeping unlabelled values as strings. Returns the translated column and the codes that had no label.

_variables_to_dataframe(data::Vector) -> DataFrame

Convert parsed variables data to a DataFrame.

_warn_unmatched(table::AbstractString, column::AbstractString, codes)

Report codes present in the data that the codebook does not describe.

cache_dir() -> String

Get the root cache directory for NHANES data. Creates the directory if it doesn't exist.

codebook_url(table::AbstractString) -> String

Build the URL for a table's codebook/documentation page.

cycle_suffix(year::Int) -> String

Get the table suffix for a given survey cycle start year.

Throws

• ArgumentError: If the year is not a published cycle. 2019 gets a dedicated message pointing at the pre-pandemic cycle :P.

Examples

cycle_suffix(1999)  # ""
cycle_suffix(2017)  # "_J"

data_cache_path(table::AbstractString) -> String

Get the cache file path for a data table.

Throws

• ArgumentError: If the table name is not a plain identifier

extract_text(node) -> String

Concatenate the text content of an HTML node and its descendants.

Text nodes are joined without a separator, so inline markup inside a cell does not introduce whitespace that the rendered page does not have.

fetch_file(url::AbstractString, dest::AbstractString; retries::Int=3) -> String

Download a file from a URL to a destination path.

The transfer goes to a temporary file next to dest and is renamed into place once complete, so a failed download never leaves a truncated file at dest.

Arguments

• url::AbstractString: URL to download from
• dest::AbstractString: Local path to save the file
• retries::Int=3: Maximum number of attempts

Throws

• DownloadError: If the download fails

fetch_html(url::AbstractString; retries::Int=3) -> String

Fetch HTML content from a URL.

Arguments

• url::AbstractString: URL to fetch
• retries::Int=3: Maximum number of attempts

Throws

• DownloadError: If the fetch fails

http_status(e) -> Union{Int,Nothing}

Get the HTTP status carried by a failure, or nothing when the request failed before a response arrived.

is_cached(table::AbstractString) -> Bool

Check if a data table is cached.

is_metadata_fresh(path::AbstractString) -> Bool

Check if a metadata cache file exists and is within the TTL.

is_transient(e) -> Bool

Report whether a failure is worth retrying: connection and timeout failures, HTTP 429, and HTTP 5xx. A 4xx response means the resource will not appear on a second attempt.

load_metadata(category::AbstractString, key::AbstractString)

Load metadata from the cache.

Returns

• Parsed JSON data, or nothing if not cached or stale

metadata_cache_path(category::AbstractString, key::AbstractString) -> String

Get the cache file path for metadata.

Throws

• ArgumentError: If the category or key is not a plain identifier

normalize_component(comp::Symbol) -> Symbol

Convert a component name to its full form. Accepts full names and R-style short names (:DEMO, :DIET, :EXAM, :LAB, :Q), both case-insensitive. Returns: :Demographics, :Dietary, :Examination, :Laboratory, :Questionnaire

parse_table_name(table::AbstractString) -> Tuple{String, String}

Parse a table name into its base name and cycle marker.

Returns

• Tuple{String, String}: (basename, cyclemarker)
  • For suffix tables: ("DEMO", "_J")
  • For P_ prefix tables: ("BMX", "P_")
  • For 1999-2000 tables: ("DEMO", "")

Examples

parse_table_name("DEMO_J")    # ("DEMO", "_J")
parse_table_name("ALB_CR_J")  # ("ALB_CR", "_J")
parse_table_name("P_BMX")     # ("BMX", "P_")
parse_table_name("DEMO")      # ("DEMO", "")

parse_variablelist_html(html::AbstractString) -> Vector{Dict}

Parse a variable list HTML page.

Returns

• Vector{Dict}: One entry per variable, with name, description, table and table_description keys

resolve_url(base::AbstractString, href::AbstractString) -> String

Resolve a link against the absolute URL of the page it appeared on.

Absolute links are returned unchanged and protocol-relative links take the scheme of base. Everything else is joined onto base, with . and .. segments collapsed and .. stopping at the root.

Throws

• ArgumentError: If base is not an absolute URL

Examples

resolve_url(
    "https://wwwn.cdc.gov/nchs/nhanes/nhanes3/datafiles.aspx",
    "../../data/nhanes3/1a/adult.dat",
)
# "https://wwwn.cdc.gov/nchs/data/nhanes3/1a/adult.dat"

save_metadata(category::AbstractString, key::AbstractString, data) -> String

Save metadata to the cache as JSON.

suffix_to_year(suffix::AbstractString) -> Int

Convert a table suffix to the survey cycle start year. The leading underscore is optional.

Examples

suffix_to_year("_J")  # 2017
suffix_to_year("B")   # 2001
suffix_to_year("")    # 1999

survey_years() -> Vector{Int}

List the start years of every published survey cycle, in ascending order.

table_rows(html::AbstractString) -> Vector{Vector{String}}

Extract the text of the td cells of every table row on a page, one entry per row. Header rows, which hold th cells, come back empty.

Returns

• Vector{Vector{String}}: Cell text per row, in document order

table_url(table::AbstractString) -> String

Build the download URL for an NHANES data table.

Examples

table_url("DEMO_J")
# "https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/DEMO_J.xpt"

table_url("P_BMX")
# "https://wwwn.cdc.gov/Nchs/Data/Nhanes/Public/2017/DataFiles/P_BMX.xpt"

validate_name(name::AbstractString, kind::AbstractString) -> String

Check that a name cannot escape the directory or URL path it is joined into. Returns the name unchanged.

Throws

• ArgumentError: If the name contains anything but letters, digits and underscores

variablelist_url(component::Symbol, year::Union{Int,Symbol}) -> String

Build the URL for the variable list page.

with_cache_dir(f, dir::AbstractString)

Run f with the cache root redirected to dir, restoring the previous root afterwards.

with_retries(f, url, action, retries; backoff = RETRY_BACKOFF_SECONDS)

Run f, retrying transient failures with exponential backoff.

Arguments

• f: Zero-argument function performing the request
• url::AbstractString: URL being requested, reported in errors
• action::AbstractString: What f is doing, reported in errors
• retries::Int: Maximum number of attempts
• backoff::Real: Delay in seconds before the first retry

Throws

• DownloadError: If the request fails permanently or runs out of attempts

Cache root to use instead of the Scratch.jl space, or nothing for the default. Tests redirect the cache here so they never touch a real download cache; see with_cache_dir.

Valid NHANES data components for continuous NHANES (1999-present).

Mapping from R-style short component names to full names.

Mapping from lowercased component spellings, both full names and R-style aliases, to full names. Backs case-insensitive lookup.

Valid years for DXA data.

File extensions of the historical data files CDC publishes.

Historical survey identifiers.

Number of days before metadata cache is considered stale.

Code cell CDC uses for the missing row of a numeric value table.

Code cells CDC uses for the missing row, numeric and character.

Pre-pandemic cycle identifier (2017-March 2020). Tables use P_ prefix instead of suffix.

Description CDC uses for the row covering a variable's measured range.

Base delay in seconds before a retry, doubled on each further attempt.

Names that are safe to embed in a cache path or a CDC URL.

Number of CDC page fetches allowed in flight at once. Six matches the per-host connection limit browsers use, so a search asks no more of wwwn.cdc.gov than loading the same pages in a browser would, and measured throughput stops improving beyond it.

Mapping of table suffixes back to survey cycle start years.

Mapping of survey cycle start years to table suffixes.

Continuous NHANES uses letter suffixes starting from 1999-2000:

• 1999-2000: no suffix (or sometimes _A in older tables)
• 2001-2002: _B
• 2003-2004: _C
• etc.

The 2019-2020 cycle was cut short by COVID-19 and never published on its own. Its data was released as part of the 2017-March 2020 pre-pandemic files, and CDC skipped the letter _K, so 2021-2023 uses _L.