The Irish navy said "significant loss of life" was feared and Medecins Sans Frontieres confirmed "many deaths".
But the Italian coast guard and the UN refugee agency say about 400 have been rescued and 25 bodies recovered so far.
More than 2,000 migrants are said to have died in 2015 trying to cross the Mediterranean to reach Europe.
Wednesday's incident occurred when the packed fishing boat ran into rough weather about 15 miles (25km) from Libya's coast.
A distress call was picked up in Sicily and one of the first ships on the scene was an Irish navy vessel. But as she launched her boats, the migrants apparently moved to one side of the fishing boat, causing it to capsize.
At least three other ships and three helicopters were reported to have arrived in the area shortly afterwards.
'Capsized very fast'
Cmdr Filippo Marini of the Italian coast guard said the rescue operation was still ongoing and now involved seven ships, .
He said survivors had indicated that between 400 and 600 people were on board the boat when it capsized.
Melissa Fleming, a UN refugee agency (UNHCR) spokeswoman, said about 100 people are thought to have been travelling in the boat's hull at the time, adding: "The boat capsized very fast".
