Build a dataset from public sources where every row says which download it came from.
sourceledger is a small Python library and command, with no dependencies, for projects that turn other people’s published data into a dataset of their own. It keeps four promises:
datapackage.json describes every output.validate fails without them.revisions lists what changed since a copy of that log.Install and reference · PyPI · Changelog
This example follows one source through a publisher’s revision. A local file stands in for the publisher, so the revision can be made on purpose. With a real source, the steps and output are the same. The output is from an actual run, with the file path shortened.
sources.toml, in the project root:
[[source]]
id = "co2"
status = "implemented"
url = "file:///…/publisher/annual.csv"
vintage = "2025 release"
terms = "Public domain (US government work)"
terms_url = "https://gml.noaa.gov/about/disclaimer.html"
license = "LicenseRef-us-government-public-domain"
redistribute = true
$ sourceledger fetch
co2 new 37 B 38fe14e8b0ed 2025 release file:///…/publisher/annual.csv
$ sourceledger build
2 rows -> data/processed/co2.csv
package -> data/processed/datapackage.json
$ sourceledger validate
0 problems
fetch saved the file to data/raw/ with a record beside it, and logged the version in
data/fetches.jsonl. build wrote the output with its provenance columns, and the data
package that describes it.
$ sourceledger fetch
co2 unchanged 37 B 38fe14e8b0ed 2025 release file:///…/publisher/annual.csv
Over HTTP, this fetch sends the ETag back and the server answers that nothing changed, so nothing is downloaded. The log gains no line.
The publisher changes 2023’s value from 419.31 to 419.32, without saying so.
$ sourceledger fetch
co2 new 37 B 16f81da8a5c1 2025 release file:///…/publisher/annual.csv
$ sourceledger validate
data/processed/co2.csv: rows come from an earlier fetch than co2.csv; re-run build
1 problems
$ sourceledger revisions before.jsonl
- `co2`: 38fe14e8b0ed -> 16f81da8a5c1, 37 bytes, fetched 2026-10-18T03:38:33Z from <file:///…/publisher/annual.csv>
The new bytes have a new checksum, so the fetch line says new and the log gains a second
line. validate now refuses the output built from the old download. revisions, given the
log as it was before (before.jsonl), names the source that changed. The
project-templates data-pipeline
kind runs this check weekly in CI and comments on an issue when a source changes.
After sourceledger build, each row carries the new download’s provenance:
source_id,source_url,retrieved_at,vintage,raw_sha256,year,co2_ppm
co2,file:///…/publisher/annual.csv,2026-10-18T03:38:33Z,2025 release,16f81da8a5c14a281dc0d85d59275de78ee39b38d4fd9ea5a0b3e85fb3282333,2023,419.32
Commit sources.toml and data/fetches.jsonl: the log holds checksums and URLs, never
data. Keep data/raw/ and data/processed/ out of git, so the repository never
redistributes a publisher’s data. Anyone can rebuild them with fetch and build. To see
whether a source changed since your last commit, copy the log before fetching and run
sourceledger revisions against the copy.
Someone who downloads a published dataset can check it against its package, offline:
$ sourceledger verify my-dataset/
0 problems
Each file the package lists must exist, with the size and SHA-256 it gives.
To have an editor with a TOML language server (Tombi, Taplo) check the manifest, put this
line at the top of sources.toml:
#:schema https://raw.githubusercontent.com/simonvanlierde/sourceledger/main/src/sourceledger/sources.schema.json