PhD students do not come out of thin air. They mainly come from the CAO system. When someone applies for a physics programme. They mainly chose on the location of the university and the points of the system not on the speciality of the department.
Now if we want physics PhD students to maximise there potential we have to have them working in the field they are best at. Physics has many fields Quantum, Astronomy, Materials, Optics, Computational, Particle etc and fields within fields. Students need to be exposed to all these branches of physics. The only exposure to these various branches is via their lecturers. Once they have taken courses and been exposed to all branches can they make their decisions on what branch suits them best.
However if we try to pigeon hole topics into universities we are going to have the situation where physics courses are weighted towards certain topics. This already happens lecturers have their pet topics that they like to teach. This is their research areas. Universities balance this out by having varied research areas so having varied lectures. But when universities have specialities and are excluding from certain projects. The lecturers in the department are all going to researching in a small number of fields.
Lecturers teach in their own areas. And in these consolidation measures lectures and PhD students who are future lecturers are going to move to universities that do research in their fields. This coalescence in talent could even create a flood where for example all optics research in the country could be done in one university and all astronomy in another.
Hence in departments whose research fields are narrowed are also going to narrow the scope of the courses they teach. This is going to damage the education of students if the greatest potential cosmologist ends up in a department that does not do cosmology due to this measure they are not going to reach their potential. This will be detrimental to the future quality of graduates.