You are fighting a mathematically pure interpretation of black boxes that are making no assumptions at all. Your observations are correct. But nobody actually interprets the term "black box" the way you deem wrong.

Taken from here [1]:

White-box models: This is the case when a model is perfectly known; it has been possible to construct it entirely from prior knowledge and physical insight.

Grey-box models: This is the case when some physical insight is available, but several parameters remain to be determined from observed data. It is useful to consider two subcases.

1. Physical modeling: A model structure can be built on physical grounds, which has a certain number of parameters to be estimated from data. This could, for example, be a state-space model of given order and structure.

2. Semiphysical modeling. Physical insight is used to suggest certain nonlinear combinations of measured data signal. These new signals are then subjected to model structures of black-box character.

Black-box models: No physical insight is available or used, but the chosen model structure belongs to families that are known to have good flexibility and have been 'successful in the past'.

[1] http://www.sciencedirect.com/science/article/pii/00051098950...