Dear all,

I would like to share an open-source tool that may be useful to people
running simulations and model fits.

The problem. Experiment trackers such as MLflow and Weights & Biases
assume an experiment is a training loop with epochs, losses and
checkpoints. A Hodgkin–Huxley simulation, a hierarchical fit across
participants, or an EEG preprocessing pipeline has none of those.
Months later it is often impossible to say which code version,
environment and seed produced a given figure.

What it does. One decorator or context manager around existing code
records the environment, git state, every parameter, every seed, and
content hashes of inputs into a plain-JSON manifest. daftar diff then
compares two runs and separates fields that could have caused a
difference from fields that merely record one — including the case
where results differ and nothing that could have caused it does, which
is a finding rather than a glitch.

Seeds are applied rather than merely recorded, since noting that a run
was irreproducible does not help anyone. Ten adapters record
framework-specific state a generic tracker cannot infer: the
integration method Brian2 auto-selected (it is stored nowhere), which
ICA components were excluded in MNE, which confounds were regressed
out in Nilearn, whether a NEURON compiled mechanism is stale relative
to its .mod sources.

No dependencies, no server, no account, no network access. Apache 2.0.
Currently in peer review at pyOpenSci.

pip install daftar https://github.com/aslansd/daftar

One thing that emerged from building it, which may interest this list:
four independent frameworks (MNE, cpm, Brian2, sbi) all report only
one of the two reasons an optimiser stops. A fit that hit its
iteration limit is indistinguishable from one that converged.

I would welcome criticism, particularly from anyone who has previously
tried a reproducibility tool and stopped using it.

Aslan Satary Dizaji
https://turan-cybernetics.com/
https://sites.google.com/a/umich.edu/aslansdizaji