This “trick” does not extend to statements, however. You can’t override if or semicolon in most languages. You can encode statements as expressions, but then you have to worry about things like variable bindings on your own.
#include <math.h>
#include <stdio.h>
struct autodiff { double value, deriv; };
autodiff just(double x) { return { x, 1 }; }
autodiff operator +(autodiff a, autodiff b) {
return { a.value + b.value, a.deriv + b.deriv };
}
autodiff operator *(autodiff a, autodiff b) {
return { a.value * b.value, a.deriv*b.value + a.value*b.deriv };
}
autodiff sin(autodiff a) {
return { sin(a.value), cos(a.value)*a.deriv };
}
int main() {
autodiff x = just(.1);
for (int ii = 0; ii<4; ii++) {
x = x*x + sin(x);
}
printf("value: %lf, deriv: %lf\n", x.value, x.deriv);
return 0;
}
There is no need to differentiate the for loop or the semicolons. This way is not doing symbolic differentiation. It's implementing the differentiation rules in parallel to calculating the values at run time.This generalizes to partial derivatives for multivariate functions too:
template<int dims>
struct autograd { double value, grad[dims}; };But this goes to question at hand, whether AD should be a library/DSL or whether it should be a primitive of a general purpose language. The thing is a general purpose language lets you present sorts of things as functions that can't be even dual numbers won't take the derivative of correctly - a dual scheme can't distinguish variable order loops from constant order loops.