The MIT license is a license about copyrighted software, allowing people to use/modify/publish that software. But a file format isn't a piece of software.
Are you open-sourcing the specification document for the file format? (people are still free to write software that reads+writes the file format even if the specification document isn't open-sourced).
Are you open-sourcing your particular library for reading the file format? (I'm confused here, because you stressed that the file format was so simple, so I'd have expected it easy and maybe even desirable for many people to come up with libraries for reading+writing the file foramt?)