Right now you basically need to either use C or OpenCL. (I'm not familiar with OpenPI and Google doesn't pull up anything that seems relevant to this conversation.) If you're not proficient in C, I would recommend the OpenCL route.

The crew at Erlang Solutions is working on running Erlang on the Epiphany chip, but I haven't seen any visible progress from them.

I can't tell you anything about programming the FPGA. You don't need to directly mess with the FPGA to use the Epiphany. From what I've seen, the jump from C to VHDL is even bigger than the jump from OO to C, so probably not a weekend project. Also, there's no FOSS toolchain for programming the FPGA--you have to use Xilinx's tools, about which I've heard mixed reviews.

I bought the cluster with the intention of, well, building a cluster, but to date haven't actually tried that--Real Life got in the way of doing that. I do have all the hardware sitting here to connect them over plain old Ethernet.