First, regarding the OP: Having spent many years studying algebra, I don't find the hierarchy of axioms to be very useful in thinking about these things. Sure, you can think of a field as a "commutative ring with inverses", but rings and fields present themselves so differently that this connection doesn't end up being all that useful. Fields are not rich enough on their own to support much interest. You'll find them mostly as building blocks rather than powerful tools in and of themselves. Ditto for modules and vector spaces. Sure, a module is "like a vector space but over a ring", but vector spaces are so boring by themselves that they show up mostly as scaffolding. The study of modules, on the other hand, is its own branch of mathematics. It's much more useful to think of them in terms of what you actually do with them.
Now, on to definitions. The following few paragraphs are all very small-minded and look far more complicated than they actually are. It all encodes pretty much what you'd expect.
If you want to define algebras over commutative rings, you need to start with left- and right-algebras. A left-algebra is an abelian group A equipped with a map \phi: R -> End(A). The abelian group structure defines the addition in the algebra, and the map defines the left-multiplication: if r \in R, and a \in A, then you define a times r as \phi(r)(a), where \phi(r) is an endomorphism on A.
A right-algebra is the same, only the map is from R to the opposite ring of End(A), where the opposite ring is the one you get by just reversing the multiplication. You need to do this because associativity demands that you compute ((a)r)s, where a \in A, r,s \in R, by first acting on a with r, then by s. But with the usual conventions of composition of functions, \phi(r) \circ \phi(s) means you first "do" s, then r. So you need to flip it. Working with left- and right-algebras is a pain in the butt because you have to carry around a ton of left-right nonsense.
A bialgebra (in the literature I read) is a an abelian group that is both a left- and right-algebra. A central bialgebra is one where the left and right multiplication are the same, which is not a given. Noncentral bialgebras are especially annoying, mostly because you have to figure out how to do pre-subscripts in LaTeX so you can write nonsense like "_R M_S".
Obviously, all of these things collapse if R is commutative. Noncommutative ring theory requires a special kind of patience. And don't even get me started on noncommutative geometry.
