HIPSYS Handbook


Chapter Twelve

Questions Regarding Validity and Reliability

 

One of the first consultants to field-test HIPSYS was Dr. Peter Hubbard. Peter is an aeronautical engineer, a test pilot and a psychologist with two PhDs. After his first HIPSYS assignment, which proved to be very successful, he said to me concerning HIPSYS, "Whatever you do, keep it away from psychologists. They won’t know how to use it."

Over the years I learned that, in some cases, he had a point. Those who are used to using various instruments in personal development, team building, organisational development, psychological testing, et cetera, sometimes have a tendency to initially regard HIPSYS as just another instrument rather than as a tool supported by a consulting methodology. Consequently the first question they ask is, "What are its measures of validity and reliability?" Because every instrument must have measures of reliability and validity. Right? Yes, if it is an instrument. But once HIPSYS is understood as a tool supported by a methodology, it can be seen that the question is irrelevant.

You may not be convinced so let’s explain.

Validity

"Validity" means "Does it measure what it is supposed to measure?"

HIPSYS does not apply labels to people or organizations. It does not use clinical terms such as Manic, Epileptoid or Hysteroid; nor does it describe people in terms of their tendencies such as Intuitive or Sensate, nor their leadership style such as Developer, Executive or Bureaucrat.

It produces various displays including circles with pluses and minuses and zeros that represent what people have said. But HIPSYS deals with perceptions, which may not necessarily be facts. Perceptions can be false, based on invalid assumptions, misinformation or prejudice.

The value of HIPSYS is that it provides a visual and graphic aid to discussing specific issues and refers people back to each of the questions and the responses to them. When they are asked to give specific cases or examples to support or justify these responses, that is what validates or invalidates the profiles. The following cases may serve to illustrate: -

Case One: A Management Application

The senior management of a mining company appointed a Liaison Committee to restructure the pay and conditions of its front-line supervisors. Overtime was to be eliminated and all supervisors were to be given a salary package that would include somewhat more than the average amount of overtime earned. They were also to be called "Team Leaders" rather than "Foremen" as a step towards creating a more team oriented culture.

As an indication of management's willingness to be open, the Foremen were asked to select two of their number to sit in on the committee's discussions and observe. However, their function was purely liaison, they had no authority to vote and the decision to eliminate overtime was not negotiable.

When the decision was implemented and the new package put in place, a few Foremen who had consistently chosen to work more overtime than the others claimed they had lost over $30,000 per year with the new package. It was true that the new package limited the capacity to earn through overtime but most Foremen were financially better off and had to work no more hours than they had previously done to gain that benefit.

Nevertheless, a few disgruntled individuals made their dissatisfactions known to their colleagues at every opportunity. They blamed the two representatives on the Liaison Committee for crawling to management and not standing up for the rights and interests of the Foremen. This back-biting and malicious gossip persisted for a few months prior to their all doing a HIPSYS on one another.

The two Foremen who were on the Liaison Committee received very negative Perceived Images which, to the external consultant who had known them for several years, seemed, at least in one case, not to be valid. Indeed, he seemed to be a particularly conscientious and competent Foreman/Team Leader.

In facilitating the one-to-one feedback with each Team Leader, it transpired that there were some who were quite positive about the company and their new package and some who were very bitter and resentful. The majority were in the middle and could be easily swayed by whoever was the most vocal. In this case the negatives had the bigger following.

When it came to discussing the results of the profiles in groups, with the profiles (Circles and Histograms), the questions and their responses (Assist display) displayed on a large screen, they were asked to give specific examples to show why a particular Team Leader was a bad delegator, or not good at problem solving, or could not be trusted. Some had based their responses on what they had heard from others. Some were angry with them for having "sold out to management" and this had influenced their responses. When it was explained to them that their two representatives had no decision-making authority on that committee, not even the power to recommend, and the decision to eliminate overtime had been a fait accompli and not negotiable, some of them said they didn't know that. Others said they had been told but didn't believe it when the disgruntled ones told them not to.

As the discussion continued some men got up, walked over to the two who had been on the Liaison Committee, apologised to them and shook hands. The profile had not been valid, but the methodology enabled the issues to come out. Misperceptions were corrected and a healing of relationships took place. It also thereafter robbed the disgruntled ones of their hitherto receptive audience.

 

 

Case Two: A Personal (i.e., Non-management) Application

This application was conducted for the Commercial Services Department of a large bauxite mining company. It was applied to a General Manager and six Managers. Although all seven participants were in a management role, they decided to do a HIPSYS so they could look, not at their management skills, but rather at their inter-personal skills in working together as a departmental team. All were graduates in accounting, economics or commerce. Six were located in the Head Office based in the state's capital city and one was based at the mine site in a remote location hundreds of kilometres from Head Office. They had the latest in computer and telecommunications technology including video conferencing via a communications satellite.

The usual methodology was followed with each individual first having his or her HIPSYS profile discussed in detail with an external consultant. Often little clues can be picked up at this stage which tell the consultant what issues need to be drawn out in the group session. One individual, the manager based at the mine site, had, on many issues, two responses in the Perceived Image that were opposite to everyone else's. With four giving a positive response and two giving a negative one, an issue would show a plus overall in the Perceived Image circle.

In the group discussion it transpired that the negative responses were from the same two individuals. They were the two who, unlike the remaining four, dealt with this manager every day. They had to work particularly closely with him whereas the others did not have much to do with him at all. The two depended on this manager's output to do their jobs whereas the others did not. The manager at the mine site placed demands on these two Head Office managers that he did not have to place on any of the others.

Whose perceptions were valid in this case? It could be argued that the two who had the most to do with the site-based manager had the more valid perceptions but, although all questions and all responses to them are shown in the Assist display, the HIPSYS profile did not reflect that in the Circles which showed many pluses. It could also be argued that both perceptions are valid; after all, they are perceptions.

Here again, it is the methodology that is of benefit. When HIPSYS is used with an appropriate methodology, it validates or invalidates a profile as the case may be. The profile itself is not as important as how it is used to unearth and address the underlying issues, some of which may not have been addressed in the questionnaire.

Questions of validity that are applied to an instrument with a label are, therefore, not relevant when applied to a survey methodology such as HIPSYS.

Polarised Responses and the Validity Issue

A Polarised response is shown in the Circles as a zero. This means that at least 30% of the respondents agreed with a statement and at least 30% disagreed with the same statement and the difference between them is less than 25%. (These percentages can be changed if required. See Chapter Six.) Which response would be the more valid?

Even though they are opposite responses, both may be equally valid in that they are reflecting the experience or opinion of the responder with regard to the particular client. The HIPSYS methodology, in the hands of a competent facilitator or consultant, draws out the reasons for these different perceptions and may well in some cases validate both of them. Again, it is the methodology that validates or invalidates the data. The data itself are merely a vehicle by which to facilitate an unearthing and a discussion of the real issues.

 

Reliability

"Reliability" means "If you did the same questionnaire again, would you get the same result?" With HIPSYS the answer would in almost all cases be "No". Or even, "I sure hope not!"

The HIPSYS methodology should invariably lead to some form of development or action plan aimed at changing and improving the behaviour and performance of individuals, groups and organisations. So, here again, questions of reliability that are applied to instruments with a label are not relevant when applied to a methodology such as HIPSYS. And HIPSYS ought not to be used simply as an instrument that produces a profile.

Furthermore, a questionnaire to ascertain how an organization’s workforce perceives it during a time of massive retrenchments will likely yield very different responses than if the same questionnaire were given out at a time of assured job security.

One psychologist once said he didn’t like the way a particular question in a HIPSYS class was worded and he was told, "Then change it." He was flabbergasted. After all, you wouldn’t do that with the Humm-Wadsworth Temperament Scale or the Myers-Briggs Type Indicator or the FIRO-B. But there are other psychologists, like Peter, who are good consultants as well as good psychologists and can use HIPSYS as a very visual and effective survey feedback tool.

Use of HIPSYS

If you want something off-the-shelf that will just give you a profile or a Print Special report, then use one of the field-tested classes and HIPSYS will give you that.

But to derive the full benefits of HIPSYS for your client you have to select, adapt or create an appropriate class (structure, facets and questionnaires) that will address the issues and needs you have identified with your client in the diagnostic phase of your consulting assignment. (See Chapter Three.) You then have to assist your client, face-to-face individually or in peer groups as appropriate, to interpret the data realistically. This may mean helping the client to unearth and confront some touchy issues. You then work with the client to develop strategies and Action Plans for improvement and follow up. (See Table 1.14 on page 33 and Chapter Five.)

The foregoing examples show that HIPSYS will not make you a good consultant. But, if you are a good consultant, then in your hands it can be a powerful, useful and valuable tool that will give benefits to your client and enhance your professional image.