OP can go to conference, in his/her "own time". OP is an adult and does not need an OK from manager to go to the conference in his/her private life.
But, OP cannot expect the manager to give the OK. OP is not "entitled" to an automatic OK. The manager does not owe OP an OK.