The unit was announced last month by education minister Leighton Andrews as part of his 20-point action plan to drive up standards and performance in schools.
This week, it was announced that Brett Pugh, chief education officer on Newport City Council, has been seconded to lead the unit, which will operate within the Department for Education, Children, Lifelong Learning and Skills (DCELLS).
But a lack of detail on its precise role and functions and how it will carry out its work is causing concern among teachers' leaders.
David Evans, secretary of NUT Cymru, said he had reservations about its work and feared it was being developed "on the hoof".
"There's very little detail about the implementation and what it's going to be doing," he said.
"Is it just another body monitoring standards? We have enough monitoring from local authorities and Estyn.
"We need a bit more meat on the bone as to what it's going to do. We think it should be co-ordinating and bringing together the bodies responsible for the delivery of education.
"We hope it's not something to hit teachers over the head with."
Gareth Jones, secretary of ASCL Cymru, welcomed Mr Pugh's appointment, but said serving school leaders and teachers should be co-opted on to the unit.
"We have concerns about what statistics are going to be reviewed by the unit and on what basis," said Mr Jones.
"We think it should be on a three-year moving average rather than an annual basis because, as a school, you may have a very good year and then a bad year.
"We have no problem with openness and challenge but it has to be a fair challenge - it's too easy to go into league table mentality."
The Assembly government said the standards unit will develop processes for monitoring, analysing and reporting on school performance data.
It will provide "support and challenge" to the system and will be responsible for updating the minister on progress.
A spokeswoman told TES Cymru that the details of how the unit would operate were still being developed with key stakeholders.
Although the structure is still unclear, it is expected that the unit will comprise no more than 10 to 15 people drawn from within DCELLS.
She added: "It will have an initial focus on school standards until 2015, following which its remit will be reviewed.
"Transparency, openness and honesty will be core to the way the unit operates."
Education minister Leighton Andrews said: "The standards unit is just one step in my 20-point plan, but it has a vital role in driving educational improvement so that we can begin to make a real difference to education in Wales."
Brett Pugh (pictured), who has been named leader of the new school standards unit, has been chief education officer for Newport City Council since June 2007. Before that, he was director of education at Blaenau- Gwent.
Mr Pugh has wide experience working in education as a former secondary teacher, college lecturer and primary head. He is currently also chairman of the Association of Directors of Education in Wales.
Original headline: Unions fear Andrews' new standards unit is for `hitting teachers'