> However, this “whistleblower” admitted to a Google investigator that he had no reason to believe fraud occurred: “he stated that he suspected that the research being conducted by Goldie and Mirhoseini was fraudulent, but also stated that he did not have evidence to support his suspicion of fraud
Er... obviously he didn't catch them on the act, literally manufacturing the data. In any case like this, before discovery, the most damning evidence you could possibly have is that you cannot reproduce the results, and in no court of law this would considered evidence of fraud. Evidence of fraud needs to show knowledge and intent.
To point this as if the whistleblower had actually recanted his accusation is misleading and clearly in bad faith. Google settled with the whistleblower (the case was for wrongful dismissal, not "fraud"), so we will never know.
> the most damning evidence you could possibly have is that you cannot reproduce the results
Seems like the "whistleblower" didn't even have that. From the paper by the AlphaChip authors: "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper."
> [whistleblower] stated he did not have evidence to support his suspicion of fraud, that he needed to cross a much larger threshold to prove his suspicion
is exactly saying what I was saying.
But in addition, this is hearsay, "quoted" only by Google's rep. It was never actually mentioned by the whistleblower. It has exactly 0 value. Using this quote at face value is intentionally misleading no matter which way you put it. They're literally the defendants - they're basically quoting themselves.
> Seems like the "whistleblower" didn't even have that.
Before he was fired?
Also, I find it funny that for all the talk of the crisis of reproducibility, anyone would trust for a second the authors of the paper more than the attempts done by a 3rd party (and literally done by one of the most important names of the entire floorplanning academic community, to begin with). At least the EDA community has used some benchmarks that have been often used by other papers, allowing some resemblance of a comparison, and a criticism that "these ancient benchmarks do not reflect our holy ways or whatever" is a criticism that maybe I also share; but it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published, unless apparently if you are called Google and publish in Nature.
Nature doesn't exactly have an stellar track record ensuring Google's results are verifiable ... https://retractionwatch.com/2024/05/14/nature-earns-ire-over...
Frankly, at this point I don't even know why would anyone bother with Google's paper. It feels as if they've managed to alienate the entire floorplanning academic community, and whenever I read one of Google's "responses" I see why.
Synopsys disavowed Markov's paper: "Regarding the CACM article that Igor Markov's comments and writings do not represent Synopsys views or opinions in any way. Synopsys is also aligned with you on the potential of Reinforcement Learning AI for chip design" (https://x.com/JeffDean/status/1859431937640665474)
Jeff Dean's post on the overall situation: https://x.com/JeffDean/status/1858540085794451906
