My Evaluation of the Evaluators

By Rebecca Thomas

Day 1 ERO

They come carrying questions.

Carefully sharpened ones.

The sort that look perfectly harmless laid neatly on a table, but you soon realise have hooks hidden beneath the surface.

And so begins the curious business of evaluation.

Except, on this occasion, I found myself becoming rather fascinated by the evaluators. Because while they were watching us, I was watching them. And it turns out there is quite an art to fishing.

One casts the line while another watches the water. A question is dropped gently into the conversation and allowed to sink.

Nothing particularly threatening.

Nothing particularly dramatic.

Just a question.

You answer.

Another follows.

Then another, perhaps from a slightly different angle.

The line tightens almost imperceptibly.

Tell us more about that.

So you do.

Because we have nothing to hide.

Our numbers are our numbers. Our data is our data. There are no strategically positioned curtains behind which inconvenient skeletons have been hurriedly shoved. We know where the gaps are. We live amongst them after all.

We know the children behind every percentage point and the whānau behind every attendance figure. We know which number represents a child who finally walked through the gate after weeks away. We know which “below” represents somebody who has moved three times, changed schools twice and nevertheless picked up a pencil today.

Numbers cannot tell those stories.

But numbers travel remarkably well.

Stories, apparently, require 'excess' baggage.

And this is where evaluation becomes interesting.

Because an onsite review is not simply people arriving to see what a school does.

It is an exercise in evidence gathering.

Questions. Conversations. Observations. Documents. Achievement information. Attendance information. Leadership decisions. Systems. Processes.

Pieces are collected and placed beside one another.

Does this match that?

If the data says this, what did you do?

How do you know?

Where is the evidence?

How did you measure it?

What changed?

How do you know it changed?

Again the line disappears beneath the water.

It is clever.

The same territory can be approached from several directions, each question sounding slightly different from the last. At times it feels less like conversation and more like someone walking slowly around a locked door, trying each handle. Not necessarily because they believe something sinister lies behind it. Simply because their job is to establish whether the door really is locked.

And leadership, I discovered, spends a great deal of time being asked to demonstrate not simply what it knows, but how it knows it.

You can know your children deeply.

You can know your community intimately.

You can describe changes you have watched unfold before your eyes.

But eventually comes the question: Where is it ‘captured’? Or, Have you got ‘evidence’ of this? And there lies perhaps the greatest philosophical divide. Some things we value are extraordinarily difficult to place inside a spreadsheet.

Belonging.

Trust.

Mana.

Connection.

The child who stayed.

The whānau member who finally came through the gate or admitted that they need some help.

The learner who stopped hiding beneath the table and joined the group.

The teenager who believed, perhaps for the first time, that someone at school actually liked them.

We can count these things. Of course we can. Human beings can turn almost anything into a number if sufficiently determined. The more interesting question is whether counting something necessarily helps us understand it.

And so the fishing continued.

Could this be measured?

Could that be tracked?

Could patterns and trends be identified?

Could trends be established?

Perfectly reasonable questions from inside an evaluation framework. But I kept wondering what happens when the things easiest to measure become the things that matter most simply because they are measurable and someone needs to see them. There is real danger in that too. Particularly in communities where context refuses to sit compliantly inside a column. Because our numbers do not lie. But neither do they tell the whole truth.

That is the uncomfortable space schools occupy.

We must own what belongs to us without accepting responsibility for everything that has ever happened around us. We must refuse the status quo while refusing the seductive simplicity of blame. We must be able to say: This isn't good enough. And, in the same breath: The explanation is bigger than the school. Those two truths can coexist.

Perhaps that is what the evaluators are really looking for.

Not perfection.

Not polished answers.

Not immaculate data.

But whether the people sitting opposite them can see what isn't working, understand why it matters and demonstrate that they have the capacity to do something about it.

So we talked.

Genuinely.

Sometimes uncomfortably.

We didn't dress the numbers in their Sunday best.

We didn't blame children.

We didn't blame whānau.

And we didn't pretend that every problem entering the school gate had been manufactured inside it.

We simply kept putting the people back into the percentages.

Perhaps that makes evaluation harder.

Perhaps context always does.

Because context is messy.

It refuses to behave itself.

It doesn't fit neatly into boxes.

And occasionally, when someone asks for another number, context has the audacity to answer with a story.

Tomorrow the fishing lines will undoubtedly be cast again.

More evidence gathered.

More stones turned over.

More questions lowered quietly into the water.

And that's okay.

Because something rather interesting happens when people arrive to evaluate you.

Eventually you begin evaluating them too.

Not their personalities.

Not their intentions.

But the machinery.

What gets counted.

What gets questioned.

What requires proof.

What is accepted as evidence.

And what remains invisible because nobody has yet invented a sufficiently tidy drop down box for it.

Perhaps every system of evaluation deserves that scrutiny.

After all, if schools must continually ask: How do we know? 

Then surely those who evaluate them should never be afraid of the same question.

Day 2 Part 2 can be found here