Groups in the topos of sets are different from groups in the topos of smooth sets. The structure of a group can be expressed as a diagram which can then be interpreted in any topos with the prerequisite mathematical structures. Toposes have products (finite limits) so every topos can potentially have group objects just like every topos can have a natural number object which is an initial algebra (colimit) for a certain diagram.

In any case, there is no royal road and if you're not willing to spend the time and effort to learn what others have written about toposes then there isn't much I can help you with here. There are no royal roads in mathematics.