The report, published on Wednesday and based on more than 200 interviews with Rohingya and aid workers, says Myanmar's security forces carried out an "unprecedented, widespread and systematic" campaign of violence starting in October 2016 and continuing in August this year.
Close to one million Rohingya have been pushed out of their homes in northern Rakhine state into neighbouring Bangladesh following "coordinated" attacks on villages that included mass killings, gang-rape and arson, the report says.
"The crimes detailed in this report indicate a failure of the government of Myanmar as well as the international community to properly protect civilians from mass atrocities," it reads.
The United Nations has called the violence a "textbook example of ethnic cleansing" but stopped short of the word "genocide", a legal definition that would require global leaders to take action under the Genocide Convention.
Genocide is defined as the intentional targeting of a community for destruction in whole or in part.
"The facts laid out in this report demonstrate that state security forces targeted the Rohingya group with several of the enumerated acts in the law of genocide," the report says.
Andrea Gittleman, a program manager for the Holocaust Museum's Simon-Skjodt centre for the prevention of genocide, said: "The atrocities occurring now demand the strongest of responses in order to halt the crimes, prevent future atrocities, and hold perpetrators accountable."
A Myanmar government spokesperson could not be reached for comment on Tuesday, but the government and army have strenuously denied the allegations, saying Rohingya militants are responsible for massacres.
Matthew Smith, CEO and founder of Fortify Rights, said the Rohingya face an "existential threat", though there had not been a final determination on genocide.
"It's reasonable to be talking about the crime of genocide and genocide prevention, particularly in light of the evidence, which indicates the Rohingya may have been targeted for destruction," he said.
"We're seeing a global moral failure. The international community has failed the Rohingya. We've been warning about the indicators of mass atrocities for years. Rohingya communities have been warning about this for years. This could have been prevented."
Tens of thousands of Rohingya fled to Bangladesh last year after Rohingya militants calling themselves the Arakan Rohingya Salvation Army attacked police posts, prompting military "clearance operations" that amounted to a massive crackdown on the population.
When militants attacked again in August this year, thousands of soldiers from nearly 40 battalions were deployed, according to Fortify Rights and the Holocaust Museum. They moved from village to village carrying out a similar pattern of mass shootings and arson, the report said. More than 600,000 Rohingya have fled to Bangladesh since August.
"The large deployment of troops, as well as the use of RPGs [rocket-propelled grenades] would have required detailed planning and coordination and the strategic allocation of significant financial resources and arms," the report said.
Fortify Rights and the Holocaust Museum, whose Simon-Skjodt centre works to prevent genocide around the world, singled out three villages as sites of massacres.
In Tula Toli in Maungdaw township, Myanmar soldiers are accused of slaughtering hundreds of Rohingya, including children, who were gathered on a river bank, and then burning the bodies. "Some small children were thrown into the river," said a witness quoted in the report. "They hacked small children who were half-alive."
The allegations are consistent with reporting by the Guardian and others.
In Rathedaung township's Chut Pyin village, soldiers and armed civilians allegedly herded men and boys into a hut before setting it on fire.
At least 150 men and boys from Maung Nu village, Buthidaung township, were shot dead after sheltering in the house of a local leader, survivors told Fortify Rights.
On Monday, the Myanmar army published the results of an internal probe exonerating itself of any wrongdoing.
A similar internal investigation into allegations of mass killings last year found a Myanmar soldier guilty of stealing a bicycle.
Fortify Rights and the Holocaust Museum are calling for the international community to enforce targeted sanctions on military commanders and an arms embargo on the country, as well as for the United Nations security council to refer the situation to the international criminal court.