The problem isn’t with immigrants, it’s with immigration. The immigration system is the problem that allows mass un-controlled migration which impacts things like jobs, housing and schools and suppresses wages, particularly for lower skilled jobs.
The UK minimum wage is as much as 5x the basic wages paid in other parts of the EU. No wonder so many people want to come and work here. If France decided to make their minimum wage £40/h half the UK population would be looking to move. The system is at fault, not the people.
Which of these scenarios is racist?
Scenario One: An immigration system that gives preferential treatment and low (effectively no) entry requirements to people from countries with mostly white populations while at the same time putting strict conditions (age, qualifications and experience, criminal history, financial ability) on people from outside those countries.
Scenario two: An immigration system that treats everyone equally, regardless of their nationality, and judges them against the same set of criteria when deciding whether to permit them to enter the country to live and work.
Scenario a is the our current immigration system as dictated by the EU. When we leave we can change to Scenario two and give everyone a fair chance to come to the UK with the skills that we need.