Like you said, this is a variant type that represents either a value, or an error. Here's the definition of Rust's Result type:
pub enum Result<T, E> {
Ok(T),
Err(E),
}
I threw together some examples of how you can use a Result value, although I used arcsin instead of tan as the out-of-domain values are much easier to specify.
https://play.rust-lang.org/?version=stable&mode=debug&editio...
I'm not entirely sure what you're asking about regarding runtime behaviour, but here's two guesses.
As a user of a function that returns a Result type, at runtime the function returns a result that has a flag and a payload, where the flag specifies whether the payload is your value or an error. The only ways to get a value out of the result are either to check the flag and handle both possibilities, or to check it and crash the program if it's an error.
As someone writing a function that returns a result type, you write whatever logic you need to determine whether you've successfully produced a value, or which error you've produced, and then you either return(Ok(my_value)); or return(Err(my_error));. If you're doing floating point math, this might be checking an overflow flag, or looking at your inputs, or checking for NaN after operating, or whatever makes sense for your task.
Using a variant/product/discriminated union like this is orthogonal to the details of floating point operations at runtime. What it lets you do is have the compiler enforce that users of your function must check for errors, as it's a type error to assign a Result<f64,E> to a variable of type f64.
I hope that helps! Do you have any more questions about this, or things I didn't address, or topics where you'd like more explanation or examples?