Sea Watch, one of several NGOs which operate search-and-rescue boats in the Mediterranean, said the lives had been lost needlessly due to heavy handed tactics of a coastguard being trained and financed by the European Union.
The German charity said its boat Sea-Watch 3 had been sent early on Monday to the aid of a sinking dinghy by Italy's coastguard and had arrived at the scene around the same time as a Libyan patrol boat.
According to the NGO, the Libyans began "beating and threatening" the migrants on the stricken boat, provoking panic that caused some to fall overboard.
The patrol boat then tried to move off with people still clinging to its sides, prompting the intervention of an Italian navy helicopter to try and calm the situation.
"Probably, nobody would have had to die today if only we had the possibility to operate reasonably in a calm environment," said Sea Watch head of mission Johannes Bayer.
"These deaths have to be blamed on the Libyan coast guards who have obstructed a safe rescue with their brutal behavior.
"European governments finally have to draw conclusions from this incident and stop the collaboration with the Libyan coastguard."
Sea Watch said the rescue had taken place 30 nautical miles north of tripoli, well outside Libyan territorial waters.
The Libyan boat took an unknown number of migrants back to the north African state while 58 survivors were on board the Sea-Watch 3.
European efforts to beef up the Libyan coastguard have been cited as one reason for a sharp fall in the number of migrants leaving Libya for Italy on traffickers' boats since mid-July, although there have been signs of a pick-up in activity off Libya since the start of this month.
Some 2 500 migrants have been rescued and brought to Italy so far this month, along with the bodies of 26 mainly Nigerian women recovered at the weekend.