Announcement: daftar 1.0 — recording what produced each computational result
Dear all, I would like to share an open-source tool that may be useful to people running simulations and model fits. The problem. Experiment trackers such as MLflow and Weights & Biases assume an experiment is a training loop with epochs, losses and checkpoints. A Hodgkin–Huxley simulation, a hierarchical fit across participants, or an EEG preprocessing pipeline has none of those. Months later it is often impossible to say which code version, environment and seed produced a given figure. What it does. One decorator or context manager around existing code records the environment, git state, every parameter, every seed, and content hashes of inputs into a plain-JSON manifest. daftar diff then compares two runs and separates fields that could have caused a difference from fields that merely record one — including the case where results differ and nothing that could have caused it does, which is a finding rather than a glitch. Seeds are applied rather than merely recorded, since noting that a run was irreproducible does not help anyone. Ten adapters record framework-specific state a generic tracker cannot infer: the integration method Brian2 auto-selected (it is stored nowhere), which ICA components were excluded in MNE, which confounds were regressed out in Nilearn, whether a NEURON compiled mechanism is stale relative to its .mod sources. No dependencies, no server, no account, no network access. Apache 2.0. Currently in peer review at pyOpenSci. pip install daftar https://github.com/aslansd/daftar One thing that emerged from building it, which may interest this list: four independent frameworks (MNE, cpm, Brian2, sbi) all report only one of the two reasons an optimiser stops. A fit that hit its iteration limit is indistinguishable from one that converged. I would welcome criticism, particularly from anyone who has previously tried a reproducibility tool and stopped using it. Aslan Satary Dizaji https://turan-cybernetics.com/ https://sites.google.com/a/umich.edu/aslansdizaji
participants (1)
-
Aslan Satary Dizaji