Police were called to Hawkeswood Metal in Nechells area of the city after receiving reports that a wall had collapsed.
A spokesman for West Midlands Police said members of the emergency services attended the scene but the men were pronounced dead.
A sixth man managed to scramble free from the rubble and was taken to Heartlands Hospital with a broken leg.
A West Midlands Ambulance Service spokesman said: "On arrival, ambulance crews found that a wall had collapsed.
The five men were all employees at the site.