# Category: RESEARCH

For those of us classically trained in statistics, “evaluation” is somewhat of a mystery.  We are accustomed to a thought process that proceeds through five steps.

1. Define the variable we want to explain – the dependent variable (DV)
2. Define the variable that we believe explains variation in the dependent variable, referred to variously as the independent variable (IV) or possibly as a factor.
3. Our goal is to assess how much of the variation in the DV is explained by the IV.
4. We set up an experiment or field study and collect data.
5. Then we compute the variation in DV explained by the IV and check that it is higher than might have been observed by chance alone, i.e., we test our hypothesis and state the significance or likelihood that what we are seeing is in reality . . .well nothing.

The thinking and goals of evaluation to hypothesis testing.

1. Above all, we are not moving towards a significance test.  We are not in the business of testing what is real and likely to be reproduced on different occasions. Put this out of your mind.  We are in the business of asking who cares and why they care!  At the end of the process we must provide an answer that is not ‘objective’.  We must be able to say whether something is good or not.
2. When we collect our ‘data’, we assume that something happened.  Even nothing happening (e.g, lunch did not arrive) might be a relevant happening.  We also assume that those in authority, often us, set out to make something happen.  We are thus asking whether our efforts were worthwhile and above all, wanted.
3. Those of trained in statistical methods might then want to quantify what happened. We may seek numbers but the DV is not our focus.  For example, in asking whether this child pass their examinations, the examination mark is descriptive information.  In statistical work, we are fundamentally asking what percentage of children passed their examinations and the probability that any one child in the future will pass.  Evaluation work asks whether students took an examination anyone actually cared about and what it cost us to get there. Were children required to work day and night?  Once they have completed the course, were the skills of any use?  How did this examination preparation fit into the rest of their lives?  We challenge the DV itself.  We are not asking were the goals met? We are asking whether the needs were met.  What did we learn about our values?  What have we learned about our priorities and how the programme fits into the wider scheme of things?
4. Instead of looking at people as IV, we look at them no as the end point — are they satisfied?
5. And instead of looking at DV as something to be explained, we learn about values as what underpins satisfaction.  I can imagine someone then defining satisfaction as the DV to be explained. My advice is ‘don’t’.  We aren’t bringing matters to a close. We are bringing voices into a conversation and enriching what we might think of as important in the world.

## Resources

The project cycle in recruitment is amazingly long.  A job opens up, documents are drawn up and approved, advertisements go out.  We rarely have a chance to change the process while it is in mid-flow; and in any case, that may be illegal.

But what can we learn by thinking like a marketer?  Are we just learning to tweak our advertisements?  Are we learning more about our people, i.e., our market?

The key ideas in this phase revolve around a concept that is really difficult to grasp.  Markets are organic and we need to think about them as it they are ‘alive’ and ‘talking back to us’.  We not only learn about them but we grow through our conversations with them as they do in turn.  But they are influenced not only by us but by other people too.  Herein lies the catch and three goals.

1. We listen to what the market is saying back. Do we get phone calls asking for more information?  And what are applicants interested in?
2. What did applicants already know about us and who around them prompted them to call? Are we learning more about the market and personas?
3. Do we see “welcome” in this group and what is the market pulling us to do?

Anticipate the patterns of calls and questions.  At the start of a recruitment cycle, we should write down whom we expect to call and the questions we expect them to ask.  Is our sense of who will take an interest and when they enter and leave the application process getting updated?

Anticipate the community around the job.  At the start, write down whom we think is interested in this job.  Which mentors and patrons will notice the job and pass it on?  Who is actively watching the recruitment channels and who was alerted by a colleague?  What are we learning about the community around the job?  If you asked callers a question such as do you have colleagues who are applying, what would they say?  Above all, ask if they knew the job was coming up before you advertised it?  Is the job part of wider community or are you trying to forge a relationship with the community?

Anticipate your role in this living thing called the market.  As you listen to the question people ask, then you develop a sense of whether you are part of their world or whether they are applying because you offered a job. We often look for this information, arrogantly, in selection.  But what you are trying to establish in recruitment is whether your recruitment is talking to an established need or if you are having to stimulate that need.  Are we already part of a community?  Did applicants know this job was coming up and did they know because someone told them but because “of course” the job was coming up because there is an internal logic to their community of which we are a part?  Importantly, it also follows that this community, being ‘alive’ is always changing.  We should be getting a sense of this ‘living creature’ and understanding that our relationship with it will change.   Recruitment is not just doing some set piece tasks; it is an expression of this relationship and the next time we recruit, we be going through the entire process again because the market has changed, as have we.

## Will a marketing mindset be helpful to you?

Recruiters exist to bridge an organisation with the outside world but most are simply on the periphery without a good sense of what the organisation does and so focused on their commissions that they don’t think very much at all about the community behind the labour supply.  But if you want to write better advertisements and achieve good placements more quickly, that paying some attention to the essential marketing character of recruiting might help you.

In the academic world, it is the norm to give a contact number of the future line manager who is expected to take calls and talk around the job.  It is amazing how often, though, said contact person is away.   What should applicants read into that?

In the non-academic world, I remember inviting candidates around to look over the job themselves.  Technically, this is called a realistic preview, but it made an enormous impression that they were being taken seriously.  Indeed, I also worked for a university who invited shortlisted candidates in for a week. They were put up in a hotel. Anyone and everyone who wanted to interview them could in addition to the formal interviews.  And candidates were run around the town to see everything that interested them. Sometimes it was schools; sometimes it was the beach.  Their questions were answered.

## Are there formal ways to think through our interaction with applicants?

Core value proposition.  What is the core value of the job and how does it relate to the applicants’ core problem that we defined earlier?   Immediately, we see that skipping steps in our homework will cause a problem!  We do need to understand our people as well as the job!

Hook.  Having identified the core value proposition, how can we express it in the simplest terms?

Time to Value.  How quickly will the applicant experience value? What is value to the applicant in the application process?  Now here is a tough one, though probably because we spend so little time thinking about this, we can quickly identify points that would annoy them!  Where is value for them in the recruitment process?  Perhaps helping them find the information they need to make an informed decision?

Stickiness. Who begins this process and who wanders off?  And most importantly, on a 2×2 who arrives whom we sincerely believe might benefit from an employment relationship with us?  And who are we losing?  But specifically, what is that we are doing that helps applicants sift themselves?

If I am utterly honest, I see so few people manage their entire application process that I would be hesitant to call it.  And if the previous exercises to think through the job and advertisement had not been done, it would be difficult to think clearly about the process.

Recently, some academics complained to the big academic job board, www.jobs.ac.uk, that an advertisement for a lectureship contained exactly one sentence that was useful to the job applicants.   We know that our job advertisements are often bad.  But in truth, most of us have no idea how to fix them.

I think the trick, which I put here for you to consider, is to remember that selection is about prediction but recruitment is about securing supply.  It is marketing.  We want to catch people’s attention but we also want them to pre-select and arrive as marketers put it, pre-qualified.  There is no merit in attracting lot of applications when you only have one job.  It wastes your time and it makes a lot of enemies.  But how do we achieve the twin goals of being noticeable and being useful?

## Some marketing terms

Were we to be marketing the latest £1bn FMCG product, we might think in terms of {Category, Personas, Problems, Motivations}.

Category.  In recruiting, we probably assume that Category is just “a job” but we will probably return here to answer the question – what category of product does the customer put you in?  In other words, what would be their alternative if they weren’t using your product?  Gather your competition and you will have some idea of the category, or set, in which you fall.  Sometimes you are surprised by what people would be doing if they weren’t working for you/

Persona.  Who is likely to be interested in this job?  Where have they come from?  Where are they going to?  Who are they taking with them?  Simply imagine candidates that you would very much like to apply and write down who they are – though keep this private because to do it well you must draw out of your imagination details that will be illegal if they remain part of the system. So, if you imagine yourself recruiting a young male, write it down. Then later you can challenge yourself to imagine recruiting a young woman, an older woman or an older male.  Your task here is not to finalise the system but to begin to walk in the shoes of your applicants and to learn too what assumptions you make about them.

Problems.  Now imagine the issues that your applicants have with your category – that is you and your competition (and your competition might be a gap year!).  What questions are candiates likely to ask, not just you, but other people privately?  What are they googling?  What issues have come up with previous incumbents?  I remember being asked by a very senior person where the printer was.  It really irked him that he had to waste time running around to find his stuff.  This was not central to his competence but it pointed me to factors that he was taking into consideration.

Motivations.  What is the motivation behind these problems?  Why are these problems important to the audience?  In the case of the printer, the person wanted to be efficient, didn’t want to do administration and wanted to be respected?  Maybe these are traits I don’t want and I can signal that in the advertisement.  Or maybe, these are traits that I do want and I can indicate the ‘state of the plant’ in the advertisement.  In recruitment, money and location is important.  And so too is not lying about these.  It is amazing how often recruiters do.

I know a lot of recruiters who just can’t be bothered. They don’t really understand the jobs they are filling. And forgetting that they have two audiences, concentrate mainly on what hirer wants.  That is of course, their judgement.

My experience of using marketing in recruitment is that both we successfully fill jobs, and on the first pass.  Moreover, the number of applications plummets by an order of magnitude. We get better replies and we get fewer replies.   We satisfy both our audiences: the hirer and the applicants.

With this little bit of homework, which is quite enjoyable to do, particularly in a group, we write better job advertisements.

## A thought experiment

Look out the window.  What do you see?

Now imagine looking out the window again in ten minutes.  Do you expect to see the ground, the earth, in the same place?

Now imagine you live in ‘earthquake country’.  And imagine if you were asked the same questions there.

Just by being there, you would probably be a lot more aware of the geology beneath the earth’s crust, the moving of magma and tectonic plates; and conscious of the distinct possibility that in ten minutes, in any ten minutes, the earth may not look like it is as it is now.  How do we live in a world where the earth beneath our feet is unstable?

## Paying attention to dynamics

In a dynamic world, clearly, we pay attention to dynamics, to movement, and to change.   In a dynamic world, we do our best to understand how things work, and why things work the way they do.  If we can, we even collect data and work out the probabilities of events that we fear.

But no matter how good our research is, we cannot foretell exactly when events will happen.  That is not in our gift.  Yet an interesting thing happens.

In our determination to look at what we can keep stable, we have constructed a whole new world: a world of data.  And in our desire for stability, we often act as if world is stable and if the data system is stable.  Yet, we have already established that the world is not stable and it is dynamism, good and bad, which interests us. What if our data systems are unstable too?  How can we think about data as something unstable.

## Dynamic world, dynamic data

What does it mean to say the data is dynamic?  Can we think of our data not as a mirror, say as showing a car moving behind us, but as something dynamic itself?  What would it mean to think of data itself as dynamic?

If we had the power to think of the world as static, when it is not, clearly, we have the power to formulate models of data in any way we choose, or least, collectively choose.  And one choice is to interrogate the stability of our data rather than simply assert the stability of our data.

Instead of fixing the relationship between the world and our data in an formula or an algorithm, we can think about our sense-making more artistically in which we think about the unpredictability and variability of the world.

• What happens when we bring data, nature and our actions together?
• How does this become a collective and shared experience?
• How do our new layered stories become important not just to us but also to nature?

In this way, through understanding how we pay attention to our world, we may understand our world better.

## Acknowledgements

The inspiration and most of the content was taken from

Ballard, S., Mitew, T., Law, J. & Stirling, J.  (2016).  Data natures and aesthetics of prediction.  Proceedings og teh 22 International Symposium on Electronic Art ISEA 2016, Hong Kong.

In a very useful paper Jayanti (2011) compares deductive, inductive and abductive logic.  I am using her examples here because they are couched in business language and are easier to remember than the examples used in textbooks.

I also would like to extend Jayanti’s typology to conductive logic described by Floridi (2017).

Floridi, L. (2017). The Logic of Design as a Conceptual Logic of Information. Minds and Machines, 1–25. https://doi.org/10.1007/s11023-017-9438-1

Jayanti, E. B. (2011). Toward Pragmatic Criteria for Evaluating HRD Research. Human Resource Development Review, 10(4), 431–450. https://doi.org/10.1177/1534484311412723

## Deductive reasoning

All companies have logos.

This organisation is a company.

Therefore, it has a logo.

We assert truths about the world (All companies have logos) and about the case (the organisation is a company).  If the assertion about the case is true (usually justified in the Participants section), then whether or not it has a logo ‘tests’ the truth of the first statement.

When we only want to test the idea that more companies than none have a logo, then we can take a sample of companies. If they have more logos than we can contribute to measurement error, we accept that the first statement is not wrong.

## Induction

All these organisations are companies.

All these organisations have logos.

Therefore, all companies have logos.

This methodology begins with data-in-hand and constructs a rule that is similar to the truth asserted in the first example.  In the first example, we intend to test the statement.  In this example, we arrive at the statement and we could go on to test it as before.

We can see the ‘stretch’ in inductive reasoningand most of us are uncomfortable unless this is exploratory research and the work moves on to that test.

This logic may also be useful for accounting for a case, in which case the truth value of the assertion is a methodological question that remains within the case and is not generalized beyond the case.

We are asserting information about the conclusion rather than the starting premise but in most ways it is used, this is simply a version of the positivistic paradigm as the example under deduction

## Abduction

All companies have logos.

This organisation has a logo.

Therefore, this organisation is probably a company.

The ‘stretch’ here is even greater than under induction, but it is limited to the case-in-hand.  It is when what we assumed would be true about case is evidently not so that we put a question mark around the first statement.

The purpose of abductive reasoning is to be practical. We are trying to make sense of one case and extend things we ‘know/believe’ to be true.  Surprise requires us to take action and think again.  We accept plausible, coherent accounts until they do not work.

Conduction

Floridi (2017) suggests conduction goes further than abduction.  Writing as an information theorist, a technologist takes a set of requirements and puts together a system that satisfies those requirements. The movement from requirements to the system is conduction.  The system can still be tested deductively (does it meet the requirements?).

### How could be phrased this in terms of companies and logos?

I want this to be company, meaning, it needs a legal persona of its own and must be recognised as an entity separate from its owners and have limited liability. These are requirements.  The organisation must be able to do {a, b, c}.

And the working solution could be {name, logo, registration and founding documents, directors to act for it, an official address to serve legal papers, an initial capital investment, up-to-date accounts}.  The organisation will have {x, y, z}.

We may work this out mimetically, and probably do. Imagining, the solution is conduction and has to be constructed in the local situation  . . . that is situated and embodied.

The imaginative solution also depends upon past experience, i.e.,  knowing what will work together and what the actor is able to do.

The imaginative solution anticipates meeting the requirements but it is likely that there many, but not any sets of requirements that would suffice.

## What is critical research?

Critical research acknowledges power at the level of society, e.g., class in the United Kingdom.

## Critical psychological research

Psychological research rarely takes a critical approach. The work of Stuart Carr at Massey in New Zealand on poverty and say, how expats justify their high salaries, is critical.

## Steps for critical research

Here are six steps for a fundamentally critical approach to psychological research.

1. We are both not fully conscious of the choices we make every day (e.g., hegemony) and, we can see conceal our interests (e.g., models of society’s conflict).
2. Practically, we will not attend to issues unless something outside of the everyday prompts us to do so.
3. New ideas often come from the “boy commenting on the lack of the Emperors’ clothes”.  The comments may be challenging but they often are dismissed as being naïve.
4. Though we’re sensitized by the critical literature, and our role as academics, we are biased to
5. The behavioural, observable data, or signal, that something interesting is going on is when people are silenced. We can detect that from texts using corpus linguistics (e.g., passive voice) but we’ll also pick it up by watching.  This is a fundamental tool for psychologists – what is making someone uncomfortable?  From their point of view, something doesn’t stack up.
6. This method contrasts with much of the qualitative work which analyses what is said rather than what was not said.  I am not proposing thematic analysis but a way to consider counterfactuals and to consider what brings about the agenda of a group.

## When to use this methodology

To use this methodology, you will need to be sufficiently trusted to be able to observe a group or the situation needs to be public.

You also need a reason to believe that taking a ‘broader angle lens’ so the group can see itself against the backdrop of what is happening in the wider world.

These methodologies are labour-intensive and the deeper investigations should begin with a compassionate intent and a reasonable belief that greater value is possible with greater sociological imagination or a historical view. And you should have time and resources to follow through to help the group digest and absorb the conversations that follow from this approach.

## Making ScholarWriter portable and researcher-friendly

During the last few months, I have been packaging ScholarWriter into a portable version that comes on a USB stick.  Simply, take one USB stick holding ScholarWriter, start it up, make sure the Apache and MySQL servers are running, – and you can start work on a private website in your browser.

At the end of the day, log out, shut it all down, and backup using a simple .zip file.

If something goes wrong, take your backup and unzip it.  And, you get straight back to work without any angst.

Why ScholarWriter?

So ScholarWriter is portable, but “what is the aim” as Chris Hambly of Audana  and Cornwall said last night on Twitter (@audio).

Anyone who writes long reports – dissertations, theses and papers in academia and long management consulting reports – will be familiar with something not much talked about – research is physically exhausting.

### We get relevant material

• We look for relevant material in the Libraries of the world
• We get the source material

### We read and take notes

• We track what we have read and what we haven’t read
• We take notes and carefully put the full reference on the top and paginate our pages

### We file and re-file notes (endlessly)

• We file those notes somewhere
• When we need our notes, we rely on memory to remember where they are
• Then if we need them elsewhere we re-file them

### We copy our notes again and again

• Then we start writing and that means cutting and pasting notes from our notes file to our main writing file and carefully putting in the references

Ha!  Try doing that without losing something and having to go through file after file checking details or looking for something you lost.

Now you have the reason for ScholarWriter.  Keeping meticulous track of who said what is incredibly difficult as you move things around physically and your argument evolves as you learn about the subject.  It is not only difficult, it is exhausting. I think that is what we learn in academia and why most people give up and flee to commerce.

The key software for academics at the Library end will remain Endnote, or something similar – we want to find references and import them into Word.

And at the other end, the final draft stage, the key software remains Word – we want to layout out our dissertation or paper ready to send electronically to our supervisor or publisher.

ScholarWriter sits between the two ends.

### We get relevant material

• We can import and export our bibliography as single references or a list in .xml format (don’t worry – Endnote and ScholarWriter sort that out for you)
• We can load .pdfs into the same system so they get backed up nightly with our notes and moved to other computers as one large package
• We can keep links to online references bundled with the reference in case we need them

### We read and take notes

• We write our notes into something like a “blog post” that has an extra field – type a phrase from the title of the article and ScholarWriter cross-references to the reference (and moreover keeps a list with the reference of where the notes are!)
• We can open the relevant .pdf file in another window (we can do that anyway but nothing is stopping us doing that)
• If we come back to our notes and want to make a comment, we just use the normal comments section of a blog post – there is no need to open the file even

### We file notes ONCE not endlessly

#### One large folder in date order

• We save everything – references, notes, drafts, scribbles, entries into our calendar – in one running file by date order in one folder.

#### Searching thousands of files is easy

• You can save everything in Windows too – you don’t need to make folders but this one central folder gets larger.  This is where Drupal, the CMS underlying ScholarWriter comes in.  Drupal has a powerful internal search function.  It searches the content of all your content, it searches by title, it searches by date, and it searches by tag.

#### Develop and maintain outlines of your dissertation or paper

But that is not all, as the advertisements say, the outlining feature of Drupal is very powerful.  Instead of physically moving files to a folder, you hyperlink them into the outline of a book.

1. First you set up the cover page.
2. Then you add child pages for each major section – Title Page, Introduction, Method, etc.
3. And lastly, after you have saved some notes or a reference or some scribbles that popped into your head, you drop them into the right place in an outline.

You don’t physically move the file from its position in the giant running file – you simply tell an outline which files are relevant to that section.  And you can see the outline developing on the screen in front of you. It is not buried in Windows Explorer in another file.

## Using Outlines to speed up your writing

I am always struck that US universities push outlining. This is how you use outlining in ScholarWriter.

When you want to develop a section, yourrepeat the general process.

1. You break the section up into sub-sections and then you add a child page for each subsection.
2. Then with a few clicks for each, you attach files to the sub-sections.
3. The content never moves – but the outline develops.
4. The outline develops with a few clicks – not opening and editing a file – simply because an outline is simply a “view” it is not a file that is saved anywhere.

## Commit your Outline to writing

So if an Outline is never actually saved, how do we “commit it to writing”?

When you want to see everything you have for a section, you ask for “Print Friendly”.  If you have, say five files in that section, those five files will be collated in the order you have them, into one display in another Window in your browser.  Now you can see not only the headings but everything in the files as well.

To print out everything, simple print.  It is that easy.  Five files, say, printed one after another.  A huge saving in physical work.

## How can you write up a section?

When you have all the “facts, figures and quotations” collected for a section, it is time to write.  Usually, you would open all five files and possibly physically print the notes on several articles.

### Using Scholarwriter

1. First you preview what you have using Outline and Print Friendly
2. Then you sort your notes into order – using a drag ‘n drop system
3. Then you check again with Outline and Print Friendly
4. If you are ready to write, you use CTRL A and cut ‘n paste to take everything into Word
5. And now you are ready to turn your notes into a compact paragraph, largely through deletion, and then be writing one tight, cogent, paragraph with references and page numbers.

## Building the text of your dissertation or paper

Now that you have written a powerful and complete paragraph, instead of saving in Word, you copy ‘n paste back into ScholarWriter, or to be more precise, onto the child page ‘holding’ that section.

You no longer need the links to the original notes, so you de-link them.  Each with four clicks, I believe.  You don’t lose your notes though. They sit snugly where they have always sat, in your giant running file, organized by data and fully searchable without any arduous opening and closing of files.

So at this point you have a paragraph written for your growing magnus ops saved as file and positioned correctly in your Outline. And your notes sitting where they always have been but no longer linked to the Outline because you have written that section up.

One paragraph down!  Next!

## ScholarWriter fits the advice – little and often

The best feature of ScholarWriter is that it allow you to concentrate on one task at a time.  And to complete small tasks in the time that you  have.

If you only have 30 minutes to an hour in the morning, you can realistically turn the notes on five articles into one paragraph.  A paragraph a day does not sound like a lot, but it is a lot more than no paragraphs a day and a lot quicker than wasting the time you do have on trying to get over procrastination and get down to work when you have been away from your writing for some time.

## Imagining the working day with ScholarWriter

Your working day with ScholarWriter amounts to

2. Reading an academic article and making notes which you drop into an outline
3. Structuring your outline getting down eventually to one child page per paragraph (think of an upside down tree)
4. Writing a paragraph which you save as a file and keep linked to its position in the outline.

Do any one of those and you have made progress. Do four of those and you have made a lot of progress.

## Security and ScholarWriter

We made ScholarWrite portable, partly to lower the IT knowledge needed to use it (slap it in and fire it up) but more so for security. When everything you need – your server, your WYSIWYG, your bibliography, your sources, your notes, your outline and your drafts – are in one folder, it’s dead simple to backup. Zip up the folder and send the .zip someone safe by email (start a special gmail account?).

Eveything is safe and can be recovered by unzipping the folder.   Fire up ScholarWriter and you are back in business within minutes.

## Stay oriented with ScholarWriter

Even after three decades in this business, I still find the feeling of disorientation when I shift tasks most uncomfortable.

With everything in one place and Drupal’s powerful views, I have lists refreshing themselves to help me get my bearings.

• When you add a reference, or a bunch of references, to your bibliography, your What I have yet to read list is automatically updated.
• When you take notes on an article and cross-reference a reference, the reference drops off your What I have yet to read list and joins your What I have read list.
• When you procrastinate in the morning – focus by looking at the five things you put in your To Do list the previous night using a simple a click of a flag
• At the End the day, when you are feeling exhausted yet you are asking – What did I do all day? – Click the Ta Da flag as you go and admire your list grow.
• Take off items from your To Do list and watch with pleasure as it shortens during the day!
• And ScholarWriter has a full Calendar. Put in dates up to five years’ out (fits a part-time PhD or the publication of a research paper).  Put in recurring dates such as tutorials and include times and details like room numbers

That is ScholarWriter – portable software for academics and other writers of long documents with many primary sources.  Plug ‘n play, easy to back up, and cutting down on the effort of managing your many documents.  You are still the Scholar and the Writer, but hopefully your work is not so exhausting and hopefully you cut a significant amount of time from completing your meticulously prepared document.