NOAA GHCN Daily → Metric Units

View on GitHub

What

Convert NOAA Global Historical Climatology Network daily records into a CSV with proper SI units (°C and mm) and a per-row consistency flag.

Why interesting

GHCN is the most widely cited open climate dataset in the world (10k+ stations, daily back to 1763 for some) and it ships with a unit convention that silently 10×-amplifies every reading for anyone who doesn't read the documentation: temperatures are stored as integer tenths of degrees Celsius, precipitation as tenths of millimetres, both left-padded with whitespace into a fixed-width column. A row showing TMAX=" 50" is 5.0°C, not 50°C — climate-research repos on GitHub repeatedly ship this bug.

flowchart TD
    R["**raw row** — TMAX/TMIN<br/><small>tenths °C, whitespace-padded</small>"]
    R --> Q1{"TMAX or TMIN<br/>empty?"}
    Q1 -->|yes| P["partial<br/><small>measurement gap</small>"]
    Q1 -->|no| Q2{"TMAX < TMIN?"}
    Q2 -->|yes| F["tmax_below_tmin<br/><small>instrument fault</small>"]
    Q2 -->|no| OK["ok"]

Edge cases sourced from.

• https://www.ncei.noaa.gov/pub/data/ghcn/daily/readme.txt — official GHCN daily README; documents the tenths-of-units convention, the whitespace padding, and the per-element quality flag columns
• https://github.com/pangeo-data/pangeo/issues/598 — Pangeo community thread where users were getting 50°C readings before noticing the scaling

Data source. NOAA GHCN Daily — Central Park station USW00094728 (this slice: 12 real days hand-picked from the station's 157-year history so that one short table shows every case the template handles — a dry day, a snow day, sub-zero temperatures, a 0.0 °C minimum, a genuine measurement gap, and both instrument faults).

The tricks

(See the inline comments in sample.json.)

1. TRIM + ÷10 unit conversion — IF(ISEMPTY([TMAX]), '', TRIM([TMAX]) / 10) handles both the whitespace padding and the tenths-of-°C scaling, while keeping genuinely empty cells empty (a measurement gap stays empty rather than collapsing to 0.0 °C).
2. Per-row consistency flag — flag rows where TMAX < TMIN (instrument fault), TMAX/TMIN empty (partial), or otherwise ok. Climatology averages computed across the raw column would silently swallow these.

The guard must be ISEMPTY, not [X] = ''

A day with no rain is a real measurement of 0, and in bxp '0' = '' is true — both sides coerce to the number 0. So the cheaper-looking guard reports "it didn't rain" as "we don't know", and a 0.0 °C minimum as a missing reading. ISEMPTY tests the trimmed length, which "0" survives. Compare them on show all: TRIM([TMIN]) = '' versus ISEMPTY([TMIN]) — they disagree on 2020-01-06, a real 0.0 °C day. Across the full 157-year file that one choice moves 1,559 rows out of the partial bucket.

At full scale

The committed sample.csv is a 12-row teaching slice; the real Central Park file is the complete daily history. Pull it and run the same template against the whole thing:

bash fetch-full.sh          # downloads ./full/USW00094728.csv (~17 MB)
bxp-cli --config full.json  # processes the full history

Measured on the reference machine (ReleaseFast, 8 cores):

metric	value
input	57,486 rows × 124 columns / 17 MB
date span	1869-01-01 → 2026-05-23 (157 years)
wall time	~0.17 s
peak RSS	~18 MB (flat — does not grow with the file)
ok	57,477 rows
partial	7 rows — one week in May 1869 with no temperatures
tmax_below_tmin	2 rows — 1894-10-05 (15.6 < 16.1 °C), 1897-02-07 (3.9 < 6.1 °C)

The consistency flag earns its keep here: two instrument-fault days and one missing week hidden in 157 years of records, found in a fifth of a second.

GHCN's per-station file is also 124 columns wide (every measurement element pairs with a _ATTRIBUTES quality-flag column). bxp-cli's column ceiling is 16384, so a climate file sits comfortably under it — as do day-per-column time series like the Johns Hopkins COVID-19 series at 1147 columns.

Final result

Whitespace-padded tenths become real SI values, and each row is classified:

raw TMAX  raw TMIN  raw PRCP   →  tmax_c  tmin_c  prcp_mm  consistency
"   94"   "   67"   "   38"    →  9.4     6.7     3.8      ok            2020-01-03
"   72"   "    0"   "   10"    →  7.2     0       1        ok            2020-01-06
""        ""        "  178"    →  (empty) (empty) 17.8     partial       1869-05-02
"  156"   "  161"   "    8"    →  15.6    16.1    0.8      tmax_below_tmin  1894-10-05

Without the ÷10 step a casual reader sees 94 and concludes that 3 January 2020 in Central Park hit 94 °C. And note the difference between rows two and three: 2020-01-06 really was 0.0 °C at dawn, while 1869-05-02 genuinely has no thermometer reading — the first is a number, the second is empty, and only the second is flagged partial.

Trace it in the GUI

Open sample.csvx, click any tmax_c cell: the trace pane shows the chain IF(ISEMPTY(" 94"), '', TRIM(" 94") / 10) = 9.4, one step at a time.

Sample data

Run it with bxp-cli --config ./sample.json --template noaa_daily_to_metric:

sample.json (config)sample.csvsample.csvx (result)

{
  conversion_templates: {
    noaa_daily_to_metric: {
      data_dir:           ".",
      file_pattern_in:    ".csv",
      file_pattern_out:   ".csvx",
      csv_text_quote_in:  "double",
      csv_text_quote_out: "double",

      input_schema: {
        $station: "[STATION]",
        $name:    "[NAME]",
        $date:    "[DATE]",
        $lat:     "[LATITUDE]",
        $lon:     "[LONGITUDE]",

        // TRICK 1 — GHCN stores temperatures and precipitation as integer
        // TENTHS of their base unit (°C and mm), left-padded with spaces:
        //   TMAX="   50"  →  5.0 °C
        //   PRCP="   38"  →  3.8 mm
        // A naive parser that skips the TRIM and the ÷10 reports 50°C max
        // for a winter day. Wrap in IF so empty cells stay empty rather
        // than becoming 0.
        //
        // The guard MUST be ISEMPTY, not `TRIM([X]) = ''`. A day with no
        // rain is a real measurement of `0`, and `'0' = ''` is TRUE in bxp
        // (both sides coerce to the number 0) — so the cheaper-looking
        // guard silently reports "it didn't rain" as "we don't know".
        // ISEMPTY tests the trimmed LENGTH, which "0" (length 1) survives.
        $tmax_c: "IF(ISEMPTY([TMAX]), '', TRIM([TMAX]) / 10)",
        $tmin_c: "IF(ISEMPTY([TMIN]), '', TRIM([TMIN]) / 10)",
        $tavg_c: "IF(ISEMPTY([TAVG]), '', TRIM([TAVG]) / 10)",
        $prcp_mm:"IF(ISEMPTY([PRCP]), '', TRIM([PRCP]) / 10)",
        $snow_mm:"IF(ISEMPTY([SNOW]), '', TRIM([SNOW]) / 10)",

        // TRICK 2 — sanity check column. NOAA occasionally publishes rows
        // where TMAX < TMIN (instrument fault or transcription error).
        // Flag them so they don't poison climatology averages. Same ISEMPTY
        // reasoning: a 0.0 °C minimum is a measurement, not a gap.
        $consistency: "IF(ISEMPTY([TMAX]) OR ISEMPTY([TMIN]), 'partial', IF(TRIM([TMAX]) < TRIM([TMIN]), 'tmax_below_tmin', 'ok'))"
      },

      row_rules: [ { when: "1 = 1", rows: [ {} ] } ],

      output_schema: {
        station:     "$station",
        name:        "$name",
        date:        "$date",
        lat:         "$lat",
        lon:         "$lon",
        tmax_c:      "$tmax_c",
        tmin_c:      "$tmin_c",
        tavg_c:      "$tavg_c",
        prcp_mm:     "$prcp_mm",
        snow_mm:     "$snow_mm",
        consistency: "$consistency"
      }
    }
  }
}

"STATION","DATE","LATITUDE","LONGITUDE","ELEVATION","NAME","PRCP","PRCP_ATTRIBUTES","SNOW","SNOW_ATTRIBUTES","SNWD","SNWD_ATTRIBUTES","TMAX","TMAX_ATTRIBUTES","TMIN","TMIN_ATTRIBUTES","ACMH","ACMH_ATTRIBUTES","ACSH","ACSH_ATTRIBUTES","ADPT","ADPT_ATTRIBUTES","ASLP","ASLP_ATTRIBUTES","ASTP","ASTP_ATTRIBUTES","AWBT","AWBT_ATTRIBUTES","AWND","AWND_ATTRIBUTES","DAEV","DAEV_ATTRIBUTES","DASF","DASF_ATTRIBUTES","DAWM","DAWM_ATTRIBUTES","EVAP","EVAP_ATTRIBUTES","FMTM","FMTM_ATTRIBUTES","MDEV","MDEV_ATTRIBUTES","MDSF","MDSF_ATTRIBUTES","MDWM","MDWM_ATTRIBUTES","PGTM","PGTM_ATTRIBUTES","PSUN","PSUN_ATTRIBUTES","RHAV","RHAV_ATTRIBUTES","RHMN","RHMN_ATTRIBUTES","RHMX","RHMX_ATTRIBUTES","TAVG","TAVG_ATTRIBUTES","TOBS","TOBS_ATTRIBUTES","TSUN","TSUN_ATTRIBUTES","WDF1","WDF1_ATTRIBUTES","WDF2","WDF2_ATTRIBUTES","WDF5","WDF5_ATTRIBUTES","WDFG","WDFG_ATTRIBUTES","WDFM","WDFM_ATTRIBUTES","WDMV","WDMV_ATTRIBUTES","WESD","WESD_ATTRIBUTES","WSF1","WSF1_ATTRIBUTES","WSF2","WSF2_ATTRIBUTES","WSF5","WSF5_ATTRIBUTES","WSFG","WSFG_ATTRIBUTES","WSFM","WSFM_ATTRIBUTES","WT01","WT01_ATTRIBUTES","WT02","WT02_ATTRIBUTES","WT03","WT03_ATTRIBUTES","WT04","WT04_ATTRIBUTES","WT05","WT05_ATTRIBUTES","WT06","WT06_ATTRIBUTES","WT07","WT07_ATTRIBUTES","WT08","WT08_ATTRIBUTES","WT09","WT09_ATTRIBUTES","WT11","WT11_ATTRIBUTES","WT13","WT13_ATTRIBUTES","WT14","WT14_ATTRIBUTES","WT15","WT15_ATTRIBUTES","WT16","WT16_ATTRIBUTES","WT17","WT17_ATTRIBUTES","WT18","WT18_ATTRIBUTES","WT19","WT19_ATTRIBUTES","WT21","WT21_ATTRIBUTES","WT22","WT22_ATTRIBUTES"
"USW00094728","1869-05-01","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","  147",",,Z,null","    0",",,Z,",,,"   78",",,Z","   56",",,Z",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","1869-05-02","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","  178",",,Z,null","    0",",,Z,",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","1894-10-05","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","    8",",,0,null","    0",",,Z,",,,"  156",",I,0","  161",",I,Z",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","1897-02-07","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","   43",",,0,null","    0",",,Z,",,,"   39",",,0","   61",",I,Z",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-01-01","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","    0",",,W,2400","    0",",,W,2400","    0",",,W,2400","   50",",,W","   11",",,W",,,,,"  -61",",,W","10078",",,W","10030",",,W","    0",",,W","   38",",,W",,,,,,,,,,,,,,,,,,,,,"   52",",,W","   43",",,W","   62",",,W",,,,,,,,,"  270",",,W","  260",",,W",,,,,,,,,,,"   76",",,W","  130",",,W",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-01-03","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","   38",",,W,2400","    0",",,W,2400","    0",",,W,2400","   94",",,W","   67",",,W",,,,,"   50",",,W","10095",",,W","10047",",,W","   67",",,W","   15",",,W",,,,,,,,,,,,,,,,,,,,,"   82",",,W","   58",",,W","   93",",,W",,,,,,,,,"  250",",,W","  230",",,W",,,,,,,,,,,"   45",",,W","   67",",,W",,,,,"    1",",,W",,,,,,,,,,,,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-01-06","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","   10",",,W,2400","    5",",,W,2400","    0",",,W,2400","   72",",,W","    0",",,W",,,,,"  -33",",,W","10146",",,W","10095",",,W","   11",",,W","   27",",,W",,,,,,,,,,,,,,,,,,,,,"   62",",,W","   38",",,W","   92",",,W",,,,,,,,,"  250",",,W","  230",",,W",,,,,,,,,,,"   72",",,W","  103",",,W",,,,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-01-18","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","   94",",,W,2400","   53",",,W,2400","    0",",,W,2400","   28",",,W","  -66",",,W",,,,," -106",",,W","10271",",,W","10193",",,W","  -50",",,W","   22",",,W",,,,,,,,,,,,,,,,,,,,,"   63",",,W","   36",",,W","   93",",,W",,,,,,,,,"  250",",,W","  230",",,W",,,,,,,,,,,"   76",",,W","  116",",,W",,,,,"    1",",,W","    1",",,W",,,,,,,,,,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-07-04","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","    0",",,W,2400","    0",",,W,2400","    0",",,W,2400","  306",",,W","  200",",,W",,,,,"  183",",,W","10129",",,W","10081",",,W","  206",",,W","   20",",,W",,,,,,,,,,,,,,,,,,,,,"   70",",,W","   42",",,W","   90",",,W",,,,,,,,,"   60",",,W","   40",",,W",,,,,,,,,,,"   45",",,W","   67",",,W",,,,,"    1",",,W",,,,,,,,,,,,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-08-04","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","  140",",,W,2400","    0",",,W,2400","    0",",,W,2400","  261",",,W","  211",",,W",,,,,"  211",",,W","10102",",,W","10051",",,W","  222",",,W","   32",",,W",,,,,,,,,,,,,,,,,,,,,"   84",",,W","   62",",,W","   97",",,W",,,,,,,,,"  150",",,W","  130",",,W",,,,,,,,,,,"  112",",,W","  215",",,W",,,,,"    1",",,W",,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-12-16","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","  218",",,W,2400","  165",",,W,2400","    0",",,W,2400","   -5",",,W","  -43",",,W",,,,,"  -94",",,W","10244",",,W","10169",",,W","  -44",",,W","   50",",,W",,,,,,,,,,,,,,,,,,,,,"   62",",,W","   34",",,W","   92",",,W",,,,,,,,,"   50",",,W","   40",",,W",,,,,,,,,,,"  107",",,W","  170",",,W",,,,,"    1",",,W","    1",",,W",,,,,,,,,,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,
"USW00094728","2020-12-25","40.77898","-73.96925","42.7","NY CITY CENTRAL PARK, NY US","  234",",,W,2400","    0",",,W,2400","    0",",,W,2400","  161",",,W","  -16",",,W",,,,,"   56",",,W","10010",",,W"," 9956",",,W","   78",",,W","   47",",,W",,,,,,,,,,,,,,,,,,,,,"   81",",,W","   58",",,W","   93",",,W",,,,,,,,,"  150",",,W","  140",",,W",,,,,,,,,,,"  107",",,W","  210",",,W",,,,,"    1",",,W",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

station,name,date,lat,lon,tmax_c,tmin_c,tavg_c,prcp_mm,snow_mm,consistency
USW00094728,"NY CITY CENTRAL PARK, NY US",1869-05-01,40.77898,-73.96925,7.8,5.6,,14.7,0,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",1869-05-02,40.77898,-73.96925,,,,17.8,0,partial
USW00094728,"NY CITY CENTRAL PARK, NY US",1894-10-05,40.77898,-73.96925,15.6,16.1,,0.8,0,tmax_below_tmin
USW00094728,"NY CITY CENTRAL PARK, NY US",1897-02-07,40.77898,-73.96925,3.9,6.1,,4.3,0,tmax_below_tmin
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-01-01,40.77898,-73.96925,5,1.1,,0,0,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-01-03,40.77898,-73.96925,9.4,6.7,,3.8,0,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-01-06,40.77898,-73.96925,7.2,0,,1,0.5,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-01-18,40.77898,-73.96925,2.8,-6.6,,9.4,5.3,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-07-04,40.77898,-73.96925,30.6,20,,0,0,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-08-04,40.77898,-73.96925,26.1,21.1,,14,0,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-12-16,40.77898,-73.96925,-0.5,-4.3,,21.8,16.5,ok
USW00094728,"NY CITY CENTRAL PARK, NY US",2020-12-25,40.77898,-73.96925,16.1,-1.6,,23.4,0,ok

Full-scale & binary files (run it on the complete dataset): fetch-full.sh · full.json.