Tasks

Here's a list of ideas for potential Infer.NET tasks. If you want to contribute to the framework, this is a great starting point.

• Design an evaluation framework for probabilistic inference, specifically implementations of message-passing (EP and VMP)

  • Outputs can be directly compared against another implementation, e.g. sampling. See previous work on black-box inference:
    • Kernel-Based Just-In-Time Learning for Passing Expectation Propagation Messages
    • Just-In-Time Learning for Fast and Flexible Inference
  • Alternatively, there are many ways to test without ground truth. For example:
    • Check output has correct sign
    • Check output is monotonic wrt input
    • Check output approaches a limit
    • Check that output falls within known bounds
    • Check output is symmetric
  • See the second part of The Practice of Standards Formalization (found many errors in C math libraries).
  • Instead of testing a function by executing it, one could analyze the source code. See PLSE Seminar Series Eva Darulova "Programming with Numerical Uncertainties"
  • Success is measured by
    • Number of bugs found
    • Reducing the manual work needed to test new code
  • An important question is what API should be tested. Infer.NET tests tend to either invoke message operators (the Runtime API) or invoke the inference engine (the Compiler API). Tests at the engine level could in principle be ported to other inference frameworks.
  • GPflow claims to have very good tests

• Implement papers via Infer.NET. In each case, check that inference recovers true parameters on data sampled from the model.

  • Rank aggregation model (see Prototypes/Ordering). Data is available.
  • Anomaly detection
  • Correlated Topic Model (see forum and blog)
  • "A new look at state-space models for neural data"
  • "Robust Gaussian process regression with a Student-t likelihood"
  • "Variational Inference for Heteroscedastic and Longitudinal Regression Models" (semiparametric regression)
  • Fast Approximate Inference for Longitudinal and Multilevel Data Analysis (semiparametric mixed regression)
  • "Semiparametric Regression Analysis via Infer.NET"

• Compare same model in different prob.prog systems

  • Reproduce the papers that already compared to Infer.NET
  • Implement Infer.NET Learners in other systems
    • E.g. is Edward faster at Matchbox than Infer.NET?
  • Perhaps implement a converter from Infer.NET model description to other systems
  • See http://forestdb.org/, http://mldata.io, http://openml.org
    • https://openreview.net/forum?id=HJeRm3Aqt7
    • Mlcomp (now dead) http://hunch.net/?p=1309
  • See PPAML Challenge problems: https://github.com/krlu/ppaml-cp7