>If you’re alarmed by the fact that the set of all real functions does not form a HILBERT SPACE, you’re probably not in the target audience of this post."
Video: https://youtu.be/q8gng_2gn70?t=8m3s
Thanks to
> Dr. von Neumann, ich möchte gerne wissen, was ist denn eigentlich ein Hilbertscher Raum? (Dr. von Neumann, I'd would really like to know, just what exactly is a Hilbert space?)
Asked to John von Neumann to David Hilbert at a lecture.
https://ncatlab.org/nlab/show/Hilbert+space#fn:1
I'd like to add, as a physicist by training, that anything can be a Hilbert space if you wish hard enough. You can even use results about countable vector spaces if you need them!
Conceptualizing functions as infinite-dimensional vectors lets us apply the tools of linear algebra to a vast landscape of new problemsThe observation here is that set of real value functions, combined with the set of real numbers, and the natural notion of function addition and multiplication by a real number satisfies the definition of a vector space. As a result all the results of linear algebra can be applied to real valued functions.
It is true that any vector space is isomorphic to a vector space whose vectors are functions. Linear algebra does make a lot of usage of that result, but it is different from what the article is discussing.
we’ve built a vector space of functions
Ideally, we could express an arbitrary function f as a linear combination of these basis functions. However, there are uncountably many of them—and we can’t simply write down a sum over the reals. Still, considering their linear combination is illustrative:
This is _also_ true. But the fact that finite dimensional real vectors can be viewed as a special case of set-theoretic functions or as a special case of real functions on a discrete finite space is probably less useful than the opposite: the set of real functions have a vector-space structure, and you can use all the neat theorems about (finite or infinite dimensional) vector spaces.
Well, at least the article is about the latter direction.
- How much of this structure survives if you work on "fuzzy" real numbers? Can you make it work? Where I don't necessarily mean "fuzzy" in the specific technical sense, but in any sense in which a number is defined only up to a margin of error/length scale, which in my mind is similar to "finitism", or "automatic differentiation" in ML, or a "UV cutoff" in physics. I imagine the exact definition will determine how much vectorial structure survives. The obvious answer is that it works like a regular Fourier transform but with a low-pass filter applied, but I imagine this might not be the only answer.
- Then if this is possible, can you carry it across the analogy in the other direction? What would be the equivalent of "fuzzy vectors"?
- If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"
- The argument arrives at fourier analysis by considering an orthonormal diagonalization of the Laplacian. In linear algebra, SVD applies more generally than diagonalizations—is there an "SVD" for functions?
As a result we get finite resolution and truncation of the spectrum. So "Fourier analysis with pre-applied lowpass filter" would be analysis of sampled signals, the filter determined by the sampling kernel (delta approximator) and properties of the DFT.
But so long as the sampling kernel is good (that is the actual terminology), we can form f exactly as the limit of these fuzzy interpolations.
The term "resolution of the identity" is associated with the fact that delta doesn't exist in most function spaces and instead has to be approximated. A good sampling kernel "resolves" the missing (convolutional) identity. I like thinking of the term also in the sense that these operators behave like the identity if it were only good up to some resolution.
2/3. I'm not really sure what you mean by these questions... But if you want to do "fourier analysis with a filter preapplied", you'd probably just work with within some space of bandlimited functions. If you only care around N Fourier modes, any time you do an operation which exceeds that number of modes, you need to chop the result back to down to size.
4. In this context, it's really the SVD of an operator you're interested in. In that regard, you can consider trying to extend the various definitions of the SVD to your operator, provided that you carefully think about all spaces involved. I assume at least one "operator SVD" exists and has been studied extensively... For instance, I can imagine trying to extend the variational definition of the SVD... and the algorithms for computing the SVD probably make good sense in a function space, too...
If you wanted something more quantized, you can pick some length unit, d, and replace the real numbers with {... -2d, -d, 0, d, 2d,... }. This forms a structure known as a "ring" with the standard notion of addition, subtraction, and multiplication (but no notion of division. Using this instead of R does lose the vector structure, but is still an example of a slightly more general notion of a "module". Many of the linear algebra results for vector spaces apply to modules as well.
> If it isn't possible, what similar construction on the fuzzy numbers would get you to the obvious endpoint of a "fourier analysis with a low pass filter pre-applied?"
If that is where you want to end up, you could pretty much start there. If you take all real value functions and apply a courier analysis with a low pass filter to each of them, the resulting set still forms a vector space. Although I don't see any particular way of arriving at this vector space by manipulating functions pre Fourier transform.
[1] https://proceedings.neurips.cc/paper_files/paper/1999/file/9...
Perhaps some conjugate relation could be established between finite-range in one domain and finite-resolution in another, in terms of the effect such nonlinearities have on the spectral response.
The popular lens is the porcupine concept when infinite dimensions for functions is often more effective when thought of as around 8:00 in this video.
While that video obviously is not fancy, it will help with building an intuition about fixed points.
Explaining how the dimensions are points needed to describe a functions in a plane and not as much about orthogonal dimensions.
Specifically with fixed points and non-expansive mappings.
Hopefully this helps someone build intuitions.
To me, the proper way of continuing to develop intuition is to abandon visualization entirely and start thinking about the math in a linguistic mode. Thus, continuous functions (perhaps on the closed interval [0,1] for example) are vectors precisely because this space of functions meet the criteria for a vector space:
* (+) vector addition where adding two continuous functions on a domain yields another continuous function on that domain
* (.) scalar multiplication where multiplying a continuous function by a real number yields another continuous function with the same domain
* (0) the existence of the zero vector which is simply the function that maps its entire domain of [0,1] to 0 (and we can easily verify that this function is continuous)
We can further verify the other properties of this vector space which are:
* associativity of vector addition
* commutativity of vector addition
* identity element for vector addition (just the zero vector)
* additive inverse elements (just multiply f by -1 to get -f)
* compatibility of scalar multiplication with field multiplication (i.e a(bf) = (ab)f, where a and b are real numbers and f is a function)
* identity element for scalar multiplication (just the number 1)
* distributivity of scalar multiplication over vector addition (so a(f + g) = af + ag)
* distributivity of scalar multiplication over scalar addition (so (a + b)f = af + bf)
So in other words, instead of trying to visualize an infinite-dimensional space, we’re just doing high school algebra with which we should already be familiar. We’re just manipulating symbols on paper and seeing how far the rules take us. This approach can take us much further when we continue on to the ideas of normed vector spaces (abstracting the idea of length), sequences of vectors (a sequence of functions), and Banach spaces (giving us convergence and the existence of limits of sequences of functions).
I guess it works if you look at it sideways.
(This is where I learned at least half of the math on this page: theoretical chemistry.)
Let's see: Let A be a non-empty set, N the set of positive whole numbers, and X the set of all functions f
f: A --> N
with usual notation.
Assume as is common, the scalars are the set of real numbers, but the set of complex numbers will also do.
So, is X a vector space and, thus, each f in X a vector?
No, since -f is not in X. Neither is (1/2)f.
Some references (with TeX markup):
Paul R.\ Halmos, {\it Finite-Dimensional Vector Spaces, Second Edition\/}
linear algebra treated as functional analysis.
Walter Rudin, {\it Real and Complex Analysis\/}
with Lebesgue integration and, then, Banach and Hilbert vector spaces.
Walter Rudin, {\it Functional Analysis\/}
with Fourier theory.
Jacques Neveu, {\it Mathematical Foundations of the Calculus of Probability\/}
with random variables, that is, functions from a probability space to, usually, the set of real numbers with convergence results, building on the work A. Kolmogorov building on the work of H. Lebesgue.
This is what the article is very explicitly about. I guess you can quibble that the title is imprecise, but it's just a title and the article makes it clear.
What I did was follow, as in the references, long established convention, that for a function to be a vector at least it had to be in a vector space where (1) can multiply a function by a number (e.g., reals or complex) and (2) add two functions and still get a function in the vector space. To be general, I omitted metrics, inner products, topologies, convergence, probability spaces, and more.
Or, as in the references I gave, math talks about vector spaces and vectors, and each vector is in a vector space. The references are awash in definitions of vector spaces with (1) and (2) and much more.
Computing is awash in indexes for data, e.g., B-trees, SQL (structured query language) operations on relational data bases, addressing in central processors, collection classes in Microsoft's .NET, REDIS, and calling all such also functions confuses established material, conventions, and understanding.
Well, not with the operations pulled from the codomain at least.
An infinite sequence approximates a general function, as described in the article (see the slider bar example). In signal processing applications, functions can be considered (or forced) to be bandlimited so a much lower-order representation (i.e. vector) suffices:
- The subspace of bandlimited functions is much smaller than the full L^2 space - It has a countable orthonormal basis (e.g., shifted sinc functions) - The function can be written as (with sinc functions):
x(t) = \sum_{n=-\infty}^{\infty} f(nT) \cdot \text{sinc}\left( \frac{t - nT}{T} \right)
- This is analogous to expressing a vector in a finite-dimensional subspace using a basis (e.g. sinc)
Discrete-time signal processing is useful for comp-sci applications like audio, SDR, trading data, etc.
Basically, define a finite dimensional function space V_N of dimension N, in such a way that you could grow N to be arbitrarily large. Solve not the PDE (originally defined over an infinite dimensional function space V, such as H^1), but its discretization as if it dealt only with functions in V_N rather than all functions. The PDE is then simply a linear system, easy to solve. And you can prove, for instance in the case of elliptic PDEs, that the solution to the discrete problem is the orthogonal projection of the true solution of the PDE (in V) onto V_N (Céa's Lemma). Finally, you can produce estimations of the error this projection incurs as a function of N, and thus give theoretical guarantees that the algorithm converges to the true solution as N goes to infinity.
(N in this case is the number of vertices in a mesh that is used to define the basis functions of V_N)
Given an vector space V with (+, ), you can define the vector space over functions F whose codomain is V and where F.+ and F. both take two functions as argument and return another function applying V.+ or V.* on the result. All the linear algebra properties come from the original vector space. Hence it is boring.
That's absolutely not the case.
Can you elaborate? Which vector space(s) do these vectors belong to, as drawn?
If anything, this just highlights the sloppiness of these types of illustrations. Things aren't precise enough, in my opinion, for the illustrations to do anything except confuse.
When drawing vectors, such as on a 2D space, the origin is identified with this 0 vector, and then the points in the 2D space can be associated with the vectors in the vector space.
It's not a particularly interesting proof, but the author does prove that real valued functions are vectors. The bulk of the article is less about proofs, and more about showing how the above result is useful.
how the above result is useful
Polynomials come to mind.
You don't need to go into intricacies like metrics and approximations because that's more like analysis.
You can however talk about infinite dimensional vectors spaces and talk about projections onto finite subspaces.
Functions on a countable domain are sequences.
Vector spaces can have infinite dimension, so the "only" in the first sentence does not belong there.
The second sentence is also odd. How do you define "sequence"? Are there no finite sequences?
It's fun to simulate one thing with another, but there is a deeper and more profound sense in which vectors are functions in Clifford Algebra, or Geometric Algebra. In that system, vectors (and bi-vectors...k-vectors) are themselves meaningful operators on other k-vectors. Even better, the entire system generalizes to n-dimensions, and decribes complex numbers, 2-d vectors, quaternions, and more, essentially for free. (Interestingly, the primary operation in GA is "reflection", the same operation you get in quantum computing with the Hadamard gate)