It's not disappointing, it's quite a feat. I suppose I would say that what I was trying to say is that "parsing", while impressive, is not a large part of "understanding" in my opinion. To use an analogy: almost by definition, compilers parse languages like C++ much better than humans do (basically perfectly, unless there's literally a bug in the compiler or it doesn't follow the standard due to some error).

But that doesn't mean they understand the programs (at all.) A compiler has no idea on an algorithmic level what a program might be doing (and if you remove comments, maybe a person won't understand it either, if they're not familiar with the algorithm.)

So my basic objection is that you're really calling this reading comprehension, but I don't think anything is actually being "understood"; just parsed. A better title would be as I suggested: simple AI correctly answers reading comprehension questions.

The reason that I object to "comprehension" is that these days there really are a few "deep learning" systems, that can possibly synthesize information. (I don't know that much about them.) I don't think it's fair to elevate semantic parsing to the level of comprehension.

However the comment by tariqali34 makes a good point, that perhaps this is a criticism of reading comprehension tests. I know in multiple-choice tests from standardized exams, I've been able to correctly answer reading comprehension questions about texts that I didn't even read (by finding just the sentence that talks about it), or in other cases, texts that were too technical and that I didn't understand.

I would say that I would be able to answer a question about some biomedical excerpt that I can't understand a word of, I just can't make heads or tails of it, let's say:

In order to study the physiological roles of AGC kinases, a commonly used approach has been to over-express the active forms in cells. However, due to the overlapping substrate specificities of many AGC kinases, it is likely that the over-expression of one member of this kinase subfamily will result in the phosphorylation of substrates that are normally phosphorylated by another AGC kinase. Another strategy has been to over-express catalytically inactive ‘dominant negative’ mutants of AGC kinases in cells. However, such mutants are likely to interact with and inhibit the upstream protein kinase(s) that they are is activated by, and thus prevent the ‘upstream’ kinase(s) from phosphorylation of other cellular substrates. For example, a dominant negative RSK may interact with ERK1/ERK2 preventing the activation of MSK isoforms and hence the phosphorylation of CREB (cAMP-response-element-binding protein) [9]. Furthermore, in Saccharomyces cerevisiae, over-expression of catalytically inactive Rck2p, a kinase that binds to and is activated by the Hog1P MAPK, sequestered the substrate-docking site of the Hog1P kinase, thereby preventing Hog1P from interacting with other substrates. Thus catalytically inactive Rck2P is acting as a dominant negative mutant of Hog1P and not Rck2P.

I couldn't answer a REAL reading-comprehension test about this: I just have no idea what it's REALLY talking about, I don't actually understand it. (Obviously on a syntactic level, it's not hard to parse.) I don't know what over-expression is, I don't know what a kinase is, I don't know what phosphorylation is. I don't understand the text. But if the questions are simple, perhaps I could answer some reading comprehension questions about this by parroting back quotations from it. Syntactically, there's nothing difficult here. I just don't understand it.

So I think it's unfair to call sentence parsing real reading comprehension, even if sometimes reading comprehension tests fail to differentiate between the two. You can parse sentences perfectly while understanding nothing. For example, I could answer the question "what is wrong with studying the physiological role of AGC kinases by overexpressing the active forms in cells?" which the first sentence refers to. I can just quote the second sentence "Due to the overlapping substrate specificities of many AGC kinases, it is likely that the over-expression of one member of this kinase subfamily will result in the phosphorylation of substrates that are normally phosphorylated by another AGC kinase". I don't understand, but I think I correctly parroted.

So there are real issues in determining comprehension. The higher the level of the question that is asked, the harder it is to answer without actually understanding the text.

I did find your work very interesting, thank you.