Well Vagrant existed as a way to take some known state and run it headless. It was good to say, "Here is the image we run on, now develop on it", where that image may not be your actual desktop machine. It helps avoid the "issue" of the problem being yours (desktop, rbenv, wrong openssl etc) and not mine (or code). This worked by running Virtual box in a headless mode and doing some nifty things to the VM (through SSH) for you.
Then it took on the optional use case of also running some Provisioning ontop of the VM. We could start from a base ubuntu, get to a point where our app could run and then deploy the app ontop. It was a little cumbersome but it worked.
Then Mitchel started adding other platforms for it to run under (VMWare Workstation) instead of just VirtualBox. I believe he is still working on these and is adding more like VMWare VSphere.
So Vagrant was always more to do with running of the VMS or taking the VMs and putting them somewhere and turning them on. Packer is more of the former tool of creating the VM in the first place I guess.
I feel from the little I do know that Packer and Vagrant seem to be sharing some of the same goals and solutions. I guess mitchel and comment further on where he plans to draw the line between the two.