Screen readers give special treatment to tables to help users understand what is being presented. They also offer special commands to help navigating in a row or columnar fashion.
You lose this all of this if you use non semantic elements. I believe semantics can even be lost if you change the css display property of an actual <table> element. Make sure to test whatever you do!