Category: Alternative Methodologies

Evaluation vs Hypothesis Testing

For those of us classically trained in statistics, “evaluation” is somewhat of a mystery.  We are accustomed to a thought process that proceeds through five steps.

  1. Define the variable we want to explain – the dependent variable (DV)
  2. Define the variable that we believe explains variation in the dependent variable, referred to variously as the independent variable (IV) or possibly as a factor.
  3. Our goal is to assess how much of the variation in the DV is explained by the IV.
  4. We set up an experiment or field study and collect data.
  5. Then we compute the variation in DV explained by the IV and check that it is higher than might have been observed by chance alone, i.e., we test our hypothesis and state the significance or likelihood that what we are seeing is in reality . . .well nothing.

The thinking and goals of evaluation to hypothesis testing.

  1. Above all, we are not moving towards a significance test.  We are not in the business of testing what is real and likely to be reproduced on different occasions. Put this out of your mind.  We are in the business of asking who cares and why they care!  At the end of the process we must provide an answer that is not ‘objective’.  We must be able to say whether something is good or not.
  2. When we collect our ‘data’, we assume that something happened.  Even nothing happening (e.g, lunch did not arrive) might be a relevant happening.  We also assume that those in authority, often us, set out to make something happen.  We are thus asking whether our efforts were worthwhile and above all, wanted.
  3. Those of trained in statistical methods might then want to quantify what happened. We may seek numbers but the DV is not our focus.  For example, in asking whether this child pass their examinations, the examination mark is descriptive information.  In statistical work, we are fundamentally asking what percentage of children passed their examinations and the probability that any one child in the future will pass.  Evaluation work asks whether students took an examination anyone actually cared about and what it cost us to get there. Were children required to work day and night?  Once they have completed the course, were the skills of any use?  How did this examination preparation fit into the rest of their lives?  We challenge the DV itself.  We are not asking were the goals met? We are asking whether the needs were met.  What did we learn about our values?  What have we learned about our priorities and how the programme fits into the wider scheme of things?
  4. Instead of looking at people as IV, we look at them no as the end point — are they satisfied?
  5. And instead of looking at DV as something to be explained, we learn about values as what underpins satisfaction.  I can imagine someone then defining satisfaction as the DV to be explained. My advice is ‘don’t’.  We aren’t bringing matters to a close. We are bringing voices into a conversation and enriching what we might think of as important in the world.

Resources

Davidson, E. J. (2015). Question-driven methods or method-driven questions? How we limit what we learn by limiting what we ask. Journal of MultiDisciplinary Evaluation, 11(24).
McKegg, K., & King, S. (2014). What is Evaluation? ANZEA.

 

Posted on by Jo Jordan 0

Learning to recruit better (3/3)

The project cycle in recruitment is amazingly long.  A job opens up, documents are drawn up and approved, advertisements go out.  We rarely have a chance to change the process while it is in mid-flow; and in any case, that may be illegal.

But what can we learn by thinking like a marketer?  Are we just learning to tweak our advertisements?  Are we learning more about our people, i.e., our market?

The key ideas in this phase revolve around a concept that is really difficult to grasp.  Markets are organic and we need to think about them as it they are ‘alive’ and ‘talking back to us’.  We not only learn about them but we grow through our conversations with them as they do in turn.  But they are influenced not only by us but by other people too.  Herein lies the catch and three goals.

  1. We listen to what the market is saying back. Do we get phone calls asking for more information?  And what are applicants interested in?
  2. What did applicants already know about us and who around them prompted them to call? Are we learning more about the market and personas?
  3. Do we see “welcome” in this group and what is the market pulling us to do?

How can we think about this procedurally?

Anticipate the patterns of calls and questions.  At the start of a recruitment cycle, we should write down whom we expect to call and the questions we expect them to ask.  Is our sense of who will take an interest and when they enter and leave the application process getting updated?

Anticipate the community around the job.  At the start, write down whom we think is interested in this job.  Which mentors and patrons will notice the job and pass it on?  Who is actively watching the recruitment channels and who was alerted by a colleague?  What are we learning about the community around the job?  If you asked callers a question such as do you have colleagues who are applying, what would they say?  Above all, ask if they knew the job was coming up before you advertised it?  Is the job part of wider community or are you trying to forge a relationship with the community?

Anticipate your role in this living thing called the market.  As you listen to the question people ask, then you develop a sense of whether you are part of their world or whether they are applying because you offered a job. We often look for this information, arrogantly, in selection.  But what you are trying to establish in recruitment is whether your recruitment is talking to an established need or if you are having to stimulate that need.  Are we already part of a community?  Did applicants know this job was coming up and did they know because someone told them but because “of course” the job was coming up because there is an internal logic to their community of which we are a part?  Importantly, it also follows that this community, being ‘alive’ is always changing.  We should be getting a sense of this ‘living creature’ and understanding that our relationship with it will change.   Recruitment is not just doing some set piece tasks; it is an expression of this relationship and the next time we recruit, we be going through the entire process again because the market has changed, as have we.

Will a marketing mindset be helpful to you?

Recruiters exist to bridge an organisation with the outside world but most are simply on the periphery without a good sense of what the organisation does and so focused on their commissions that they don’t think very much at all about the community behind the labour supply.  But if you want to write better advertisements and achieve good placements more quickly, that paying some attention to the essential marketing character of recruiting might help you.

 

Posted on by Jo Jordan 0

How to manage the recruitment process (2/3)

In the academic world, it is the norm to give a contact number of the future line manager who is expected to take calls and talk around the job.  It is amazing how often, though, said contact person is away.   What should applicants read into that?

In the non-academic world, I remember inviting candidates around to look over the job themselves.  Technically, this is called a realistic preview, but it made an enormous impression that they were being taken seriously.  Indeed, I also worked for a university who invited shortlisted candidates in for a week. They were put up in a hotel. Anyone and everyone who wanted to interview them could in addition to the formal interviews.  And candidates were run around the town to see everything that interested them. Sometimes it was schools; sometimes it was the beach.  Their questions were answered.

Are there formal ways to think through our interaction with applicants?

Core value proposition.  What is the core value of the job and how does it relate to the applicants’ core problem that we defined earlier?   Immediately, we see that skipping steps in our homework will cause a problem!  We do need to understand our people as well as the job!

Hook.  Having identified the core value proposition, how can we express it in the simplest terms?

Time to Value.  How quickly will the applicant experience value? What is value to the applicant in the application process?  Now here is a tough one, though probably because we spend so little time thinking about this, we can quickly identify points that would annoy them!  Where is value for them in the recruitment process?  Perhaps helping them find the information they need to make an informed decision?

Stickiness. Who begins this process and who wanders off?  And most importantly, on a 2×2 who arrives whom we sincerely believe might benefit from an employment relationship with us?  And who are we losing?  But specifically, what is that we are doing that helps applicants sift themselves?

Will a marketing mindset help you?

If I am utterly honest, I see so few people manage their entire application process that I would be hesitant to call it.  And if the previous exercises to think through the job and advertisement had not been done, it would be difficult to think clearly about the process.

Posted on by Jo Jordan 0

How to write a good job advert (1/3)

Recently, some academics complained to the big academic job board, www.jobs.ac.uk, that an advertisement for a lectureship contained exactly one sentence that was useful to the job applicants.   We know that our job advertisements are often bad.  But in truth, most of us have no idea how to fix them.

I think the trick, which I put here for you to consider, is to remember that selection is about prediction but recruitment is about securing supply.  It is marketing.  We want to catch people’s attention but we also want them to pre-select and arrive as marketers put it, pre-qualified.  There is no merit in attracting lot of applications when you only have one job.  It wastes your time and it makes a lot of enemies.  But how do we achieve the twin goals of being noticeable and being useful?

Some marketing terms

Were we to be marketing the latest £1bn FMCG product, we might think in terms of {Category, Personas, Problems, Motivations}.

Category.  In recruiting, we probably assume that Category is just “a job” but we will probably return here to answer the question – what category of product does the customer put you in?  In other words, what would be their alternative if they weren’t using your product?  Gather your competition and you will have some idea of the category, or set, in which you fall.  Sometimes you are surprised by what people would be doing if they weren’t working for you/

Persona.  Who is likely to be interested in this job?  Where have they come from?  Where are they going to?  Who are they taking with them?  Simply imagine candidates that you would very much like to apply and write down who they are – though keep this private because to do it well you must draw out of your imagination details that will be illegal if they remain part of the system. So, if you imagine yourself recruiting a young male, write it down. Then later you can challenge yourself to imagine recruiting a young woman, an older woman or an older male.  Your task here is not to finalise the system but to begin to walk in the shoes of your applicants and to learn too what assumptions you make about them.

Problems.  Now imagine the issues that your applicants have with your category – that is you and your competition (and your competition might be a gap year!).  What questions are candiates likely to ask, not just you, but other people privately?  What are they googling?  What issues have come up with previous incumbents?  I remember being asked by a very senior person where the printer was.  It really irked him that he had to waste time running around to find his stuff.  This was not central to his competence but it pointed me to factors that he was taking into consideration.

Motivations.  What is the motivation behind these problems?  Why are these problems important to the audience?  In the case of the printer, the person wanted to be efficient, didn’t want to do administration and wanted to be respected?  Maybe these are traits I don’t want and I can signal that in the advertisement.  Or maybe, these are traits that I do want and I can indicate the ‘state of the plant’ in the advertisement.  In recruitment, money and location is important.  And so too is not lying about these.  It is amazing how often recruiters do.

Will taking a marketing mindset help you?

I know a lot of recruiters who just can’t be bothered. They don’t really understand the jobs they are filling. And forgetting that they have two audiences, concentrate mainly on what hirer wants.  That is of course, their judgement.

My experience of using marketing in recruitment is that both we successfully fill jobs, and on the first pass.  Moreover, the number of applications plummets by an order of magnitude. We get better replies and we get fewer replies.   We satisfy both our audiences: the hirer and the applicants.

With this little bit of homework, which is quite enjoyable to do, particularly in a group, we write better job advertisements.

 

Posted on by Jo Jordan 0

Data, the world, me and you

A thought experiment

Look out the window.  What do you see?

Now imagine looking out the window again in ten minutes.  Do you expect to see the ground, the earth, in the same place?

Now imagine you live in ‘earthquake country’.  And imagine if you were asked the same questions there.

Just by being there, you would probably be a lot more aware of the geology beneath the earth’s crust, the moving of magma and tectonic plates; and conscious of the distinct possibility that in ten minutes, in any ten minutes, the earth may not look like it is as it is now.  How do we live in a world where the earth beneath our feet is unstable?

Paying attention to dynamics

In a dynamic world, clearly, we pay attention to dynamics, to movement, and to change.   In a dynamic world, we do our best to understand how things work, and why things work the way they do.  If we can, we even collect data and work out the probabilities of events that we fear.

But no matter how good our research is, we cannot foretell exactly when events will happen.  That is not in our gift.  Yet an interesting thing happens.

In our determination to look at what we can keep stable, we have constructed a whole new world: a world of data.  And in our desire for stability, we often act as if world is stable and if the data system is stable.  Yet, we have already established that the world is not stable and it is dynamism, good and bad, which interests us. What if our data systems are unstable too?  How can we think about data as something unstable.

Dynamic world, dynamic data

What does it mean to say the data is dynamic?  Can we think of our data not as a mirror, say as showing a car moving behind us, but as something dynamic itself?  What would it mean to think of data itself as dynamic?

If we had the power to think of the world as static, when it is not, clearly, we have the power to formulate models of data in any way we choose, or least, collectively choose.  And one choice is to interrogate the stability of our data rather than simply assert the stability of our data.

Instead of fixing the relationship between the world and our data in an formula or an algorithm, we can think about our sense-making more artistically in which we think about the unpredictability and variability of the world.

  • What happens when we bring data, nature and our actions together?
  • How does this become a collective and shared experience?
  • How do our new layered stories become important not just to us but also to nature?

In this way, through understanding how we pay attention to our world, we may understand our world better.

Acknowledgements

The inspiration and most of the content was taken from

Ballard, S., Mitew, T., Law, J. & Stirling, J.  (2016).  Data natures and aesthetics of prediction.  Proceedings og teh 22 International Symposium on Electronic Art ISEA 2016, Hong Kong.

 

 

 

 

 

Posted on by Jo Jordan 0

Six steps for a critical psychological research

What is critical research?

Critical research acknowledges power at the level of society, e.g., class in the United Kingdom.

Critical psychological research

Psychological research rarely takes a critical approach. The work of Stuart Carr at Massey in New Zealand on poverty and say, how expats justify their high salaries, is critical.

Steps for critical research

Here are six steps for a fundamentally critical approach to psychological research.

  1. We are both not fully conscious of the choices we make every day (e.g., hegemony) and, we can see conceal our interests (e.g., models of society’s conflict).
  2. Practically, we will not attend to issues unless something outside of the everyday prompts us to do so.
  3. New ideas often come from the “boy commenting on the lack of the Emperors’ clothes”.  The comments may be challenging but they often are dismissed as being naïve.
  4. Though we’re sensitized by the critical literature, and our role as academics, we are biased to
  5. The behavioural, observable data, or signal, that something interesting is going on is when people are silenced. We can detect that from texts using corpus linguistics (e.g., passive voice) but we’ll also pick it up by watching.  This is a fundamental tool for psychologists – what is making someone uncomfortable?  From their point of view, something doesn’t stack up.
  6. This method contrasts with much of the qualitative work which analyses what is said rather than what was not said.  I am not proposing thematic analysis but a way to consider counterfactuals and to consider what brings about the agenda of a group.

When to use this methodology

To use this methodology, you will need to be sufficiently trusted to be able to observe a group or the situation needs to be public.

You also need a reason to believe that taking a ‘broader angle lens’ so the group can see itself against the backdrop of what is happening in the wider world.

These methodologies are labour-intensive and the deeper investigations should begin with a compassionate intent and a reasonable belief that greater value is possible with greater sociological imagination or a historical view. And you should have time and resources to follow through to help the group digest and absorb the conversations that follow from this approach.

Posted on by Jo Jordan 0