Paper Written For Publication in International Journal of Operations & Production Management

Development of a Methodology for Assessing the Education Requirements for the Effective Implementation of ERP / MRP II Manufacturing Control Systems

Austen K Jones

A K Kochhar

M W M Hollwey

ABSTRACT

This paper outlines the need for a production planning and control (PPC) system user education requirements specification methodology. After the need for a methodology has been outlined, the various methods of education and training requirements specification are outlined and the developed methodoloy is described. The subsequent testing and validation of the requirements specification methodology are discussed.

1 Introduction

The planning and control of production activities represent important tasks that manufacturing companies need to address competently if they are to prosper in their respective markets [Muhlemann et al 1992]. There are a variety of approaches that manufacturers can utilise in the management of the operations of their production facilities and most of these approaches are computerised.

Implementation of a computerised Production Planning and Control (PPC) system is a very complex undertaking that requires much change to job roles and to existing procedures and systems within the company implementing the PPC system. These factors combine to make implementation difficult and despite the fact that computerised PPC systems have been being implemented for a good many years a significant proportion of implementations fail to meet customer expectations [Browne et al 1996, Brown 1994]

Research has been undertaken analysing the PPC system implementation process [Ang et al 1995, Burns et al 1990, Cox & Clark 1984, Duchessi et al 1988, Turnipseed et al 1992]. Some studies have determined the requisite factors for implementation success while others have set out to elicit the barriers to implementation success. These studies highlight a number of issues and factors that require addressing if systems implementation is to be effective.

A main group of factors is staff education and training [Duchessi et al 1988, Udo & Ehie 1996]. Education & Training, as a PPC implementation success factor, is itself composed of a number of different issues, including [Wallace 1985, White 1980]:

· Timing of the education / training in the overall context of the implementation project

· Disproportionate focus on training at the expense of education

· Budgets allocated to staff education and training

· Delivery of the education and training

· Course having the appropriate and adequate content

· Course being attended by the relevant personnel

The last two points are generally the most problematic since their solution is non-trivial, i.e. it requires information to be gathered and analysed. Performing an education needs analysis is one way to ensure that the relevant personnel attend courses of the appropriate content. Figure 1 indicates the vital stage of requirements specification in the context of the education / training process [Kenney & Reid 1988].

2 Need for an Education & Training Requirements Specification Framework

The fact that there has been a dearth of research that has attempted to address the PPC implementation problem of users receiving inadequate or inappropriate education and training is surprising given that user education and training are widely accepted as being major obstacles to successful PPC implementation [Udo & Ehie 1996, Duchessi et al 1988].

The development of an effective methodology for accurately specifying a PPC system user’s education and training requirements is justifiable and necessary for the following reasons:

1. The majority of PPC system implementations are unsuccessful, i.e. they fail to meet expectations [Browne et al 1996, Burns & Turnipseed (1991), Duchessi et al (1989)]. These failures are not without cost, i.e. lost opportunity, personnel time, money and so on. As such, research which broaches a PPC system implementation problem should be of significant benefit to the manufacturing industry in the UK.

2. Specifications of PPC system users’ education and training requirements that were accurate would reduce the risk of people receiving inappropriate / inadequate education and training and, therefore, increase the chances of implementation success. An approach that can make implementation easier and less problem prone would be very useful to manufacturing companies attempting a PPC system implementation, as well as companies/ system houses responsible for implementing the systems.

3. No research work has been undertaken to attempt to develop an effective approach that can accurately establish the education and training requirements of PPC system users such that they can receive adequate and appropriate education and training.

To summarise, research work concerned with developing an effective approach to the specification of education and training requirements would be useful since it would assist the process of PPC system implementation. As such, the tool would be of importance to companies attempting to implement such systems.

3 Methodology for Specifying PPC System User Education and Training Requirements

3.1 Education and Training Needs Analysis Methods

The goal of education and training needs analysis is to determine the needs as they currently exist within the company or as they have the potential to develop, for instance, needs that may arise due to the implementation of a new production planning and control system.

The more precisely a person's education and training needs can be specified the more focused the education and training interventions can be. This ensures that the education and training will actually meet the original need and in a cost-effective way [Bee & Bee 1994]. McClelland (1994) considers there to be four major methods of gathering data for education and training needs analysis, these being:

1. Questionnaires

2. Individual interviews

3. Focus groups

4. On-site observations

These methods and their respective pros and cons are discussed below.

3.1.1 Questionnaires

The questionnaire is the most prevalent method of gathering data for the purposes of needs assessment. The reasons for this are that they can provide a cost-effective and reliable means of gathering information that can be qualitative as well as quantitative. Survey questionnaires can provide accurate and relevant data through thoughtful design, testing and administration [McClelland 1994].

Other advantages of the questionnaire are that it can easily be administered to a large population and does not require people from different places to assemble together. Questionnaires are a non-intrusive method of data collection as individuals are free to complete them in their own time and in an intimidation-free setting. Further, biasing, which is often a problem in interviews due to the nature of the sender - receiver relationship (i.e. interviewer - interviewee) is not an issue with questionnaires. On the down side, the design and piloting of questionnaires is quite an involved process and the information they yield may not be as rich as can be achieved using other methods, for instance, interviews.

3.1.2 Individual Interviews

Interviews are an important method of data collection as they allow direct interaction between the person trying to determine the training need and the person for whom the education / training is intended [Bee & Bee 1994]. However, much care needs to be taken by the interviewer that the purpose of the interview is not obscured by the interviewee or the topics of discussion. For instance, it is not uncommon for interviews to drift into gripe sessions or for the interviewee to focus on wants rather than needs [McClelland 1994].

While the information yielded from interviews can be an excellent source of qualitative data and useful for the purposes of education and training needs analysis [Boydell & Leary 1996] they do have some important drawbacks. Face-to-face interviews are an expensive method of gathering data and are very time-consuming. Another disadvantage of interviews is the need for an experienced interviewer.

3.1.3 Focus Groups

Focus groups, despite becoming popular in the marketing profession, are infrequently used as the sole method of data collection for the purposes of education and training needs assessment. McClelland (1994) suggests that focus groups should comprise of between 8 to 12 people that represent a cross-section of the training population. Advance preparation of group participants is essential if the outcome of the data collection process is to be constructive and objective.

The problem of bias is often associated with the data gathered from focus groups. This can arise for a number of different reasons, for instance, the moderator asking leading questions; the group being dominated by one or two participants; the moderator failing to keep the discussion on the relevant subject areas, or by not having well-defined goals for the group to achieve [McClelland 1994].

Focus groups are a useful method of gathering qualitative data that are best used in conjunction with another, more quantitative, data collection method. However they are relatively costly, need to be moderated by an experienced professional, and provide feedback on education and training needs that is, at best, broadly defined.

3.1.4 On-Site Observation

The value of observing work tasks has found considerable acceptance and success in the industrial engineering profession [Boydell & Leary 1996, Bee & Bee 1994]. Data provided by such observations is generally qualitative although it may produce quantitative data as well. The main strength of on-site observation as a data collection method is that the data collected is empirical in nature. Where general observations are being used to determine potential training needs, however, the data provided are generally not significant enough to draw conclusions with any degree of certainty. As such, it is normally necessary to incorporate on-site observations with another type of data collection method which yields quantitative information.

The principle behind on-site observations is to perform a job analysis (gain an understanding of the parameters of the job itself) and then a task analysis and then to determine the skill / knowledge / experience levels required such that the individual can complete the task successfully.

The different methods of collecting data for the purposes of education and training needs analysis reported in the HR literature have been outlined in the previous sections. Despite the importance of education and training needs analysis the education and training needs of personnel in manufacturing companies at the time of PPC system implementation, if done at all, are assessed in the most primitive of ways. The standard method of determining who is taught what rely on consideration of the person's job title / department and the standard package courses that are offered by PPC system seller [Jones et al 1998, Roberts 1986, Iemmolo 1994]. For instance Roberts (1986) prescriptively instructs:

"Determine the training needs required by a functional area of the organisation, as they relate to each system module"

This practice is nothing more than a rudimentary method of allocating bodies to courses and makes no attempt to determine the requirements of attendees or the suitability / relevance of course content. The following section outlines the development and validation of an innovative methodology that was developed to make the requirements specification process more credible, accurate and scientific.

3.2 Development of the Education Requirements Specification Methodology

In deciding which of the four above methods were best suited to determine the education and training requirements of staff using / implementing production planning and control systems a number of criteria were drawn up. These criteria were:

· The output must be sufficiently detailed to be useful for the intended purposes

· Data collection must be relatively quick (inexpensive)

· Data collected must be as accurate as possible

· Data must be easy to collect

· Data collected from one person must be easily comparable to that collected from another person

· Data collection method must be flexible enough to be effective in a number of different contexts (i.e. job roles and companies)

· The resulting education and training requirements specifications must be accurate

Questionnaires were selected as the best method of data collection since even though they required much up front preparation they would provide a quick, detailed and potentially accurate method of data collection that could easily be applied to different people in different companies.

A task analysis-style questionnaire and a questionnaire to determine the requisite skills and knowledge were developed. As such the questionnaires consisted of a list of tasks or Activities and associated with each task there was a particular Knowledge profile. The developed questionnaires were tested by 32 experienced PPC system users from different manufacturers and resulted in a list of over 140 generic Manufacturing Resource Planning (MRP II) focused activities with associated knowledge profiles. Figure 2 shows how a person's job role is comprised of a number of discrete tasks and how each of these tasks has its own requisite skills / knowledge profile associated with it.

The needs assessment framework is based on the following logic [Jones et al 1998, McClelland 1994]:

· Staff in manufacturing companies have specific Job roles

· Each Job role is responsible for a number of different tasks or Activities

· To undertake a particular Activity successfully an individual needs to possess certain skills and have knowledge of certain information or Concepts

The methodology has a further level of detail built into it so as to enhance the accuracy of education and training specifications that use of it yields. The methodology incorporates "Involvement levels" which have an impact on the "Knowledge levels" that are required of each Concept. There are four Involvement levels ranging from None to Detailed:

None: No involvement in the Activity

Limited: Being involved in an Activity indirectly or in a superficial sense.

Some: Being involved in an Activity which does not constitute a major part of the person's job description. The involvement may be in some sort of assisting or overseeing role.

Detailed: Being actively involved in an Activity, generally on a day-to-day basis. The Activity is likely to be a significant part of this individual's job description

Likewise there are four Knowledge levels also ranging from None to Detailed:

None: No knowledge required of the Concept.

Awareness: Conceptual background information whereby an individual has some knowledge of the topic under consideration.

Overview: The level of understanding that a person requires to effectively manage the day to day activities. (This may be higher level conceptual information as opposed to a thorough understanding of the topic.)

Detailed: The level of understanding required of a particular topic by an individual whose job activities necessitate a thorough understanding of the topic.

The Involvement levels allow detail regarding the importance and frequency of performing an Activity to be incorporated in the requirements specification process.

The Activities and Concepts that were compiled in the framework are relevant to production planning and control systems and focus on manufacturing resource planning. Determining an individual's Activity profile, with Involvement levels, implicitly defines their respective Required knowledge profile (i.e. the set of Concepts with associated required Knowledge levels that are relevant to the individual). Comparing the person's Required knowledge profile with their Actual knowledge profile yields education shortfalls which represent the education / training needs of that individual.

The Required Knowledge profiles, i.e. the benchmarks against which the existing knowledge levels of individuals are measured to determine their requirements, were developed using empirical data collected from a number of manufacturing companies. The process of compiling the Required Knowledge profiles and populating the framework with valid Activities and Concepts is discussed in the following section.

4 Population of the Framework

The framework population stage of the research consisted of two distinct phases, namely:

1. Initial population of the framework (from the available literature)

2. Empirical refinement of the framework

These different stages are discussed in turn.

4.1 Initial population of the framework

Once the structure of the education requirements specification framework was in place it was then necessary to populate the framework with relevant and meaningful information, i.e. PPC Activities and Concepts and the associated links between them. This stage involved a thorough analysis and synthesis of the available literature.

4.1.1 Activities & Involvement levels

Building the preliminary catalogue of Activities involved searching a number of textbooks and papers that described manufacturing planning and control approaches including MRP II [e.g. Vollman et al 1984, Greene 1987, Muhlemann et al 1992]. Much attention was also paid to existing work in related fields that had included the elicitation and synthesis of manufacturing activities, in particular, the work of Kochhar et al (1994) which was concerned with performance indicators and benchmarking of PPC systems and also the work of Howard et al 1998 which was concerned with functional requirements specification of PPC systems. Using these sources helped to ensure that the Activities were generic, accurate and comprehensive and, as such, were in an appropriate state to allow them to be tested and refined by industrialists. Table 1 gives examples of the Activities in the framework.

Activity Title	Explanation
Estimate lead times per order (i.e. due dates).	Carried out in situations where lead times are irregular (non-standard products etc.) and standard lead times are not used. This can be done by looking at backlogs, planning loads, stock levels, part finished goods etc.
Co-ordinate returns processing and integrate it with manufacturing.	Carried out to deal with large numbers of returns (either poor quality goods or recalls).
Establish and maintain close relationships with customers.	This should be done to improve the accuracy of information that is used to forecast demand.
Determine the resource (capacity) requirements (and identify critical resources / bottlenecks).	Carried out to provide information for production planning or master production scheduling on details about the resource requirements, and which resources are close to maximum capacity (i.e. identify bottlenecks).
Adjust production schedules in accordance with material availability.	This may be necessary if ordered materials have not been delivered by the date they are required to be used

Table 1: Example Activities

As described in the previous section a four-group classification system was developed to categorise individuals' Involvement levels with Activities. A four level scale was decided upon as this was deemed to be the optimal compromise between having sufficient granularity to make the requirements specification accurate and relevant and having too many levels such that the differences between adjacent levels were unacceptably vague.

4.1.2 Concepts & Knowledge levels

A list of over 340 Concepts relevant to the Activities was also compiled from the available literature. For instance, scanning the indices of a number of textbooks highlighted Concepts for inclusion in the framework [e.g. Technical Committee M/136 1975, Wallace 1985, Cox 1995, Greene 1987], these were then defined and added to the list.

During this stage of development it became apparent that there was a degree of terminological differences between the different information sources that were used. While this was by no means great it was significant enough to warrant an amount of redundant information to be incorporated in the framework, i.e. including commonly used synonyms in the framework, for example Safety stock - Buffer stock and ABC analysis - Pareto analysis.

As for the Involvement levels discussed in the previous sub-section, a four-group classification system was used to categorise individual's Knowledge levels of Concepts. The reasons for having four Involvement levels were equally applicable to the decision of the number of Knowledge levels of Concepts that would be most appropriate.

4.1.3 Activity - Concept Relationships

Prior to the empirical population of the framework tentative relationships were set-up linking Concepts with Activities. That is, a set of Concepts was associated with each Activity in the framework thus forming a mini requisite knowledge profile for each Activity.

Each Activity was analysed in turn and consideration was given to the:

· Activity's inputs and factors which affected it

· Activity's outputs and implications

By doing this it helped to ensure that the tentative sets of Concepts associated with Activities were comprehensive.

By way of an example of the relationships that were compiled in the framework during this phase of the research consider the Activity:

"Create a formal production plan"

In the framework this Activity is associated with a number of different Concepts, these are listed in Table 2 along with their respective definitions and reasons for being associated with the Activity. Note: that the list in Table 2 was the tentative list and was added to during the empirical phase of framework construction.

CONCEPT	DESCRIPTION	LINK JUSTIFICATION
Aggregate planning	The process of developing, reviewing, analysing, and maintaining aggregate plans. Aggregate planning usually includes total sales, total production, targeted inventory and targeted customer backlog on families of products. The production plan is the result of the aggregate planning process.	The production plan is based on the results of the aggregate planning process.
Business plan	Statement of income projections, costs and profits (financially orientated)	The business plan should agree with the production plan (although the business plan is financially oriented).
Company goals	Clear and concise objectives of the company	The production plan should support the strategic aims of the company.
Critical resources	Those resources that usually become overloaded when the demand is greater than can be met from available capacity.	The production plan may need to consider specific critical resources.
Demand Forecast	An estimate of future demand.	The production plan should meet the current planned level of sales (sales plan / forecast) while meeting the general business objectives.
Master production schedule	Anticipated build schedule (must be realistic rather than optimistic) which defines the quantities of the finished products required and the associated due dates.	The MPS must take the production plan into account as well as other issues such as backlog, material availability, capacity availability and so on.
Planning	The consideration of actions to be taken in the event of some expected scenario or occurrence happening in the future.	This Concept is important to the Activity and is mentioned explicitly in the Activity's title.
Planning horizon	The amount of time the plan extends into the future.	The production plan will be associated with a specified horizon. For example, the planning horizon of the production plan could be 6 months.
Production plan	Derived from the detailed sales forecast, it balances market demand with company resources (developed by operating management).	Explicitly important.
Sales planning	Determines the sales rates the company must maintain for its product lines to meet the strategic plan	The production plan will use the sales plan as one of its inputs.
Strategic planning	Is long range and incorporates management's goals, policies and strategies for operating the total business.	The production plan should incorporate the long term strategic aims of the organisation.

Table 2: Example of Concepts Tentatively Linked to an Activity

During this stage no attempt was made to ascribe strengths, i.e. Knowledge levels, to individual Activity - Concept relationships as this was the purpose of the empirical framework construction phase, discussed in the next section.

4.2 Industrial refinement of the framework

The principal aim of this phase of the research project was to develop the framework such that its contents, i.e. Activities, Concepts and the relationships between them, were accurate and complete for the typical PPC system used by manufacturing companies.This was to be achieved using information gathered from people in a number of different types of manufacturing companies using PPC systems. To ensure that coverage of the contents of the framework was as thorough as possible it was important that participants were from a range of different departments, holding a variety of different job roles.

An education requirements specification framework that was constructed using information and knowledge gathered from good-practice PPC companies would be more accurate and useful than a similarly developed framework that used data collected from companies less successful in the usage of their PPC systems. As such, the principal criterion for selecting companies for participation was that they were "good-practice" PPC companies.

Four good practice manufacturers were recruited to help with the research (the characteristics of these companies are shown in Table 3). Senior contacts were established in each of the companies and these contacts acted as the project co-ordinators at the respective sites. The co-ordinators assisted with the different stages of the framework population process.

Company		Axles	Image Hardware	Satellites	Electrical Connectors
Company size	Small	Y
	Med		Y		Y
	Large			Y
Product Complexity	Low				Y
	Med	Y
	High		Y	Y
Level of Technology	Low	Y
	Med				Y
	High		Y	Y
Level of automation	Low	Y		Y
	Med		Y
	High				Y
Relative Skill level	Low				Y
	Med	Y
	High		Y	Y
Production type		MTO	MTO / ATO	MTC	MTO / MTS
Category		Automotive	Electronics	Defence / aerospace	Electrical

Table 3: Company Characteristics

After initial contact was made with the companies assisting in the developmental phase of the research, the collection of the necessary data was undertaken in two stages, namely:

1. Activity validation and determination of the Activity Involvement profiles of the participants

2. Concept validation and determination of appropriate Activity - Concept relationships and associated Required Knowledge levels

4.2.1 PPC Activities

The four companies that helped to develop and populate the education requirements framework were manufacturers of:

· Off-road vehicle transmissions

· Image processing hardware

· Satellites

· Electrical connectors

The companies were good-practice PPC system users which was important as it meant that data collected from them was of a higher quality than would have been collected from less effective users of PPC systems. This also served to make the results of the development phase of the research more credible.

Having identified a project co-ordinator at each of the four participating sites, copies of the Activity Involvement Questionnaire were distributed to the companies. The questionnaires listed the tentative Activities and requested individuals to list other Activities that were relevant to PPC that were not included in the list.

A total of 32 people from the four companies contributed to the compilation of the education requirements specification framework. To ensure breadth of Activity coverage the participants were from a range of different departments in the company, for instance, there were representatives from Sales, Production, Finance, Engineering, Purchasing and Materials. As such, job titles of the individuals included:

· Production Supervisor

· Production Manager

· Production Engineer

· Shift Supervisor

· Applications Engineer

· Systems Engineer

· Business Manager

· Management Accountant

· Engineering Manager

· Materials Co-ordinator

· Buyer

· Operations Systems Manager

· Head of Manufacturing

· Business Systems Manager

· Chief Accountant

· Planning Scheduler (Manufactured)

· Purchasing Manager

At the time of taking part in the research project the people from the four companies had been in their job roles (as opposed to with the company) for an average of nearly 5 years, minimum 1 year, maximum 26 years.

The forms that were received contained many corrections to the tentative Activities and indicated that the 32 people were involved with a total of 1,820 Activities 605 of which were at a detailed level. This meant that over 98%, i.e. 141 of the 143, of the Activities in the tentative framework were valid and relevant and that 97% (139 from 143) of them (i.e. those people were involved with at a Detailed level) would have their provisional relationships with Concepts judged and added to.

Further to these results there were 59 suggested additions to the list of Activities in the framework. Before adding any Activities to the framework it was necessary to consider whether the suggested Activity was:

· relevant to PPC systems - some of the suggested Activities were important to the operation of the company but were not necessarily relevant to PPC systems, for instance, "Preparation of sales promotion literature."

· already in the framework - this duplication was due to the length of the questionnaire, i.e. by the time people got to the end of the questionnaire they may have forgotten reading a particular Activity (especially if the questionnaire was not completed in one sitting). If an Activity that was suggested for addition to the framework was found to be a duplicate then it was instead considered to be a suggestion for rewording and the appropriate action taken.

The Activities that passed both of these checks were then incorporated in the framework and tentative relationships with Concepts were outlined as per the process described in Section 0.

The information gathered using the Activity involvement questionnaire was used to improve the accuracy and comprehensiveness of the list of Activities in the framework. The information regarding the individual's level of involvement with each Activity was used to tailor a second questionnaire which was concerned with determining what Concepts a person would need to know in order to carry out an Activity competently.

4.2.2 PPC Concepts & Relationships with Activities

The Concepts and Activity - Concept relationships in the framework were validated and compiled using a second questionnaire. This tailored questionnaire listed the Concepts (and definitions) that were tentatively associated with the Activities that the individual had indicated they were involved with at a detailed level. Only people involved with an Activity at a detailed level were requested to recommend the Concepts that a person performing that Activity would need Knowledge of.

Questionnaires took the form of a list of Activities. Each Activity was followed by a list of Concepts which the person completing the form was required to indicate what level of Knowledge of the Concept would be required by a person performing that Activity at different levels, i.e. none, detailed knowledge and so on. At the end of the list of the Concepts there was a gap where the participants were asked to include any additional Concepts that they considered relevant to the Activity. All of the Concepts that had featured in the questionnaire were listed and defined at the end of it.

These tailored questionnaires were posted to the relevant people in the four companies for them to complete and 26 completed forms were returned for analysis. This represented a total of 3,772 discrete Activity - Concept relationships with associated Knowledge levels.

The collected data were entered into a database to ease the task of analysis. When in the database the responses could be analysed comparatively, i.e. the required Knowledge level that one person ascribed to a particular Activity - Concept relationship could be directly compared with that suggested for the same relationship but by a different person.

In order to conduct a meaningful analysis of the data it was necessary to convert it to numeric data. Numbers were ascribed to the four Knowledge levels as indicated:

· None 0

· Awareness 1

· Overview 2

· Detailed 3

A 0 to 3 scale for representing the Knowledge levels was considered to be more appropriate than, for instance, a 1 to 4 scale, as the Knowledge level of None was given the value of 0. This simplified processing of the data and was logically consistent.

When analysing the now numeric data consideration was given to three main factors in the process of determining the appropriate Knowledge level to accompany a particular Activity - Concept relationship:

· The size of the difference between the minimum and maximum Knowledge level that was suggested for the Activity - Concept relationship

· The number of people suggesting a Knowledge level for the relationship

· The distribution of the suggested Knowledge levels, i.e. whether the responses were predominantly one value with only one response that was different.

The process of ascribing / determining the appropriate Knowledge level to a particular Activity - Concept relationship is described below and is also represented in the form of a flow diagram in Figure 4.

If the Knowledge levels that are suggested for a specific Activity - Concept relationship are the same, i.e. spread of 0, then the choice of Knowledge level is the one specified. If, however, there is a spread of Knowledge levels suggested for a particular relationship then greater consideration is required before ascribing a level to the relationship can be done in a justifiable and reasonable way.

If two different Knowledge level suggestions are made for a particular Activity - Concept relationship neither of which are for the level "None" then both respondents are agreed that the Concept is relevant to the Activity. What they are disagreeing on is the degree of relevance / importance of the Concept to the Activity. If, however, one of the responses was for the Knowledge level of "None" then there is a fundamentally different problem that needs to be addressed. Namely, that one person considers the Concept to be relevant to the Activity and the other person considers it to be irrelevant. Hence, when there is a spread of different Knowledge levels that people have suggested for an Activity - Concept link, the most important consideration is whether or not one of the suggestions is zero.

If one of the values is zero then the relationship must be given careful consideration. This included:

· Contact the people and discuss the issue further

· Consider the respondents job role and amount of experience.

This type of detailed consideration was only required for approximately 1% of the Activity - Concept associations.

If there was a spread in Knowledge levels for a particular link and none of the suggestions were for the "None" level then the size of the spread was the next most important consideration. If the spread was between adjacent Knowledge levels the greater of the two was selected. The reason for choosing the greater level of the two was to err on the side of caution. The performance of a person educated to the recommended Knowledge level, perhaps marginally over-specified as a result of opting for the greater of the two Knowledge levels, would not suffer from the greater than required Knowledge whereas this may not have been the case if he had been educated to the lower level of Knowledge.

If the minimum and maximum suggested Knowledge levels for a particular Activity - Concept relationship are not adjacent, i.e. a gap greater than 1, then the distribution or skew of the suggestions are considered. For instance, if there were four suggested Knowledge levels for an Activity - Concept relationship three of which were "Detailed" and the other was "Awareness", then the "Detailed" level of Knowledge would be assigned to the relation. If, instead of this skew, the distribution of suggestions was more symmetrical, for instance half of the people suggested "Detailed" and the others suggested "Awareness" then this situation would require further investigation. In the described instance the difference in suggested Knowledge levels was considered to be unacceptably large for the reasons affecting the spread not to be investigated. Less than 7% of the relationships contained in the model fell into this category.

During this stage of research, namely theory building, the contents of the tentative framework changed greatly. Activities, Concepts and relations between them were added and deleted and Knowledge levels were ascribed to each confirmed Activity - Concept relationship.

After this in-depth compilation phase of the research the framework contained detailed information regarding the Knowledge profiles required of people performing particular functions. At this stage the information contained in the framework was known to be relevant and complete for the four companies that had contributed to it. However, it was not known whether or not the same could be said of the framework in the context of other manufacturing companies. The next section describes the results of testing the framework in a further seven different manufacturers.

5 Testing & Validation of the Needs Assessment Framework

This section of the paper outlines the testing of the MRP II user education needs assessment methodology. The methodology was assessed using the input from 28 people from 7 different manufacturers. The companies included were manufacturers of:

1. Aerostructures, i.e. fuselage barrels

2. Touring caravans and motorhomes

3. Diesel engines

4. Fastening systems

5. Electric motors

6. Oilfield equipment

7. Metal cabinets

Participants were sought from a variety of different departments so that the framework would be comprehensively tested. This meant that people from all of the relevant departments were used during the stage of the research, i.e. Engineering, Production, Inventory, Purchasing, Sales, Design and Quality. As such, Job titles of the people providing input to the framework included:

· Cabinet Shop Administrator

· Chief Buyer

· Commodity Buyer

· Design Manager

· Database Administrator

· Inventory Controller

· Engineering Manager

· Material Planner

· Master Production Scheduler

· Materials & Quality Assurance Manager

· Planning Leader

· Product Controller

· Project Manager

· Quality Assurance Engineer

· Sales Analyst

· Scheduling Manager

The results of the needs assessment were presented to each participant and they were requested to complete a feedback questionnaire which asked, for instance, whether the approach to requirements specification was valid, whether the education requirements specifications that it reported were accurate and so on. The questions and associated responses from the people of each company are shown in Tables 4a and 4b. The managers responsible for the co-ordination of involvement with the project at each site completed questionnaires containing the questions detailed in Table 4a. These questions were not asked of the other staff since they were generally less experienced than the co-ordinators and as such it was not considered appropriate to ask them questions concerning, for instance, the generic applicability of the Activities. All participants were asked the questions listed in Table 4b.

The numbers shown in the 3 right-hand columns of the tables are percentages of responses that, respectively:

1. Agreed with the statement

2. Neither agreed nor disagreed with the statement

3. Disagreed with the statement

Question	Agree	Unsure	Disagree
The approach is an effective way to determine education requirements of individual members of staff.	100	0	0
The approach is a useful way of determining education requirements for people in a company implementing an MRP II system.	100	0	0
The approach used to specify the education requirements of individuals is better than the approach currently used by the company / department.	100	0	0
The report contains information at an appropriate level of detail.	100	0	0
The feedback report is of value for the purposes of determining useful courses to attend / course content.	100	0	0
The list of MRP II system-related Activities is complete.	100	0	0
The Activities are relevant to individuals associated with MRP II systems in your company.	83	0	17
The Activities are applicable to companies using MRP II systems in general.	100	0	0
The Activities in the list are at the appropriate level of detail for determining a person's education requirements.	100	0	0
The list of generic MRP II system-related Concepts, as applied to the Activities was complete.	100	0	0
The Concepts are useful for the purposes of determining education requirements.	100	0	0
The MRP II system-related Concepts are relevant to companies using MRP II systems in general.	100	0	0

Table 4a: Feedback Questions & Responses From the Co-ordinators Only

Question	Agree	Unsure	Disagree
The approach used to determine the education requirements of individuals is a valid & logical one.	100	0	0
The feedback report is presented in a clear and understandable format.	96	0	4
The Required Knowledge profiles listed with each Activity are accurate.	88	0	12
Your Knowledge profiles listed with each Activity are correct (at the time you completed the questionnaire).	100	0	0
The MRP II system-related Activities, as applied to your current job profile, are correct.	93	0	7
The specified Concepts were relevant to my job.	96	0	4
The Concepts are correct.	100	0	0
The Concepts listed under each Activity are the correct ones.	92	0	8
The list of Concepts under each Activity I am involved with is comprehensive.	88	0	12
The Involvement levels and their descriptions are clear and understandable.	89	0	11
There is an adequate number of Involvement levels.	96	0	4
The exercise of assigning Involvement levels to Activities is a repeatable one.	96	4	0
The Knowledge levels and their descriptions are clear and understandable.	89	0	11
There is an adequate number of Knowledge levels.	92	4	4
The exercise of assigning Knowledge levels to Concepts is a repeatable one.	95	5	0
The report was well structured and easy to follow.	89	0	11
The radar diagrams were an effective way of presenting information.	82	0	18
The summarised information in the report was useful.	89	0	11
The detailed information ("Activity-Concept Knowledge Profiles" section) was presented in an effective way.	93	0	7

Table 4b: Feedback Questions & Responses From All Participants

6 Discussion

Considering the results of the testing of the needs assessment approach, i.e. the percentages shown in columns 2-4, there were unanimously favourable responses to approximately 45% of the questions (14 from 31). This included statements that the approach was valid and useful; the specifications were at the correct level of detail and were of value in determining course content; and that the contents of the tool were generic and complete.

The vast majority of participants, i.e. over 80%, considered the report containing the education requirements specifications to be understandable, well structured and presented in an effective way. Over 90% of people believed the Activity - Concept relationships to be correct, the Concepts to be relevant, the number of Involvement and Knowledge levels to be adequate and the exercise of assigning Involvement and Knowledge levels to Activities and Concepts respectively to be a repeatable one.

All of the site co-ordinators that offered an opinion participants believed the approach to requirements specification to be better than the approach currently in their companies. The required Knowledge profiles listed under each Activity were considered to be accurate and the lists of Activities and Concepts to be Comprehensive. Further, the Concepts were considered to be generic and the Involvement and Knowledge levels were clear and understandable.

The least favourable aspect of the framework were the radar diagrams with 18% of people suggesting them to be an ineffective method of presenting information. This was due to the fact that the radar diagram, as a means of presenting information, was new to these individuals and perhaps more explanation of the diagrams on the reports would have been beneficial. This was the most contentious question on the questionnaire - and yet still over four fifths of the participants considered the diagrams to be effective.

The test results support the view that the approach to needs assessment is valid, logical and accurate. This result justifies the use of questionnaires as the main tool for data collection and the speed with which data can be collected is another bonus associated with this method [McClelland 1994, Boydell & Leary 1996].

In the context of PPC systems the results show that the approach was an accurate and effective method of specifying the education requirements. As such it should help eliminate inadequate education and training which often hinder system implementation projects [Ang et al 1995, Brown 1994, Udo & Ehie 1996].

7 Conclusions

This paper has outlined the development of an innovative methodology for assessing the education requirements of users involved with the implementation and day to day use of production planning and control systems. It has been shown that;

· An effective framework for determining education requirements can be constructed by determining and defining job activities and the requisite knowledge profiles with which each is associated.

· An individual's education requirements can be assessed in an effective way by determining the tasks the person performs as part of his job and also the person's knowledge levels of relevant concepts.

· Questionnaires were demonstrated as being an effective method of gathering the data which was used to populate the requirements specification framework. Using questionnaires is more cost-effective than using one-to-one interviews / appraisals which are the most often used methods of determining education and training needs.

· The constructed framework is generic and comprehensive. This was achieved by collecting information from a variety of different job roles in a number of different companies.

· The results of the testing of the developed framework showed it to be valid and effective and the resulting requirements specifications to be accurate, complete and useful for the purposes of determining appropriate course content / attendances.

· The developed approach to education requirements specification represents an original development in the field of PPC system implementation research.

· Further, the knowledge profiles that are associated with each Activity represent an important synthesis of industrial understanding and experience.

The authors are grateful to the Engineering & Physical Sciences Research Council (EPSRC) for the award of a research grant that made it possible to undertake this work.

8 References

Ang et al 1995

Ang, J. S. K., Sum, Chee-Chuong & Chung, Wah-Fook, (1995), "Critical success factors in implementing MRP and government assistance: A Singapore context", Information & Management, Volume 29 Issue 2, pp. 63-70

Bee & Bee 1994

Bee, F. & Bee, R., (1994), Training Needs Analysis and Evaluation, Institute of Personnel Management, London

Boydell & Leary 1996

Boydell, T. & Leary, M., (1996), Identifying Training Needs, Institute of Personnel and Development, London

Brown 1994

Brown, Andrew D, (1994), "Transformational leadership in tackling change", Journal of General Management, Volume 19 Issue 4, pp. 1-12.

Burns et al 1990

Burns, O Maxie; Turnispeed, David; Riggs, Walter E, (1990), "Critical success factors in manufacturing resource planning implementation", International Journal of Operations & Production Management, Volume 11 Issue 4, pp. 5-19.

Burns & Turnipseed 1991

"Critical success factors in manufacturing resource planning implementation", International Journal of Operations & Production Management, Volume 11 Number 4, pp. 5-19

Cox & Clark (1984)

Cox, J.F. and Clark S.J., (1984), "Problems in implementing and operating a manufacturing resource planning information system", Journal of Management Information Systems, Volume 1 Issue 1, pp. 81-101.

Duchessi et al 1988

Duchessi, P., Schaninger, C.M., Hobbs, D.R. and Pentak, L.P., (1988), "Determinants of success in implementing material requirements planning (MRP)", Manufacturing and Operations Management, , pp. 263-304.

Duchessi et al 1989

Duchessi, P. Schaninger, C. M. and Hobbs, D. R., (1989), "Implementing a manufacturing planning and control information system", California Management Review, Volume 31 Number 3, pp. 75-90.

Greene 1987

Greene, (1987), Production And Inventory Control Handbook, McGraw-Hill Inc., New York

Howard et al

Howard, A., Kochhar, A. & Dilworth, J., (1998), "Functional requirements of manufacturing planning and control systems in medium-sized batch manufacturing companies", Integrated Manufacturing Systems, Volume 10 Number 3

Iemmolo (1994)

Iemmolo, G., (1994), "MRP II education: a team-building exercise", Hospital Material Management Quarterly, Fourth Quarter, Volume 15 Issue 4, pp. 33-39.

Jones et al 1998

Jones, A; Kochhar, A. & Hollwey, M, (1998), "Education and training requirements specification for implementation of manufacturing control systems", Strategic Management of the Manufacturing Value Chain, Edited by: Bititci, U. & Carrie, A., Kluwer Academic Publishers, Boston

Kenney & Reid 1988

Kenney, J. & Reid, M., (1988), Training Interventions, Institute of Personnel Management, London

Kochhar et al 1994

Kochhar, A., Davies, A. & Kennerley, M., (1994), "Performance indicators and benchmarking in manufacturing planning and control systems", Proceedings of the OE/IFIP/IEEE International Conference on Integrated and Sustainable Production - Re-engineering for Sustainable Industrial Production, Lisbon, Portugal, May 1997, pp. 487-494.

McClelland 1994

McClelland, S, (1994), "Training needs assessment data gathering methods: Part 1, Survey Questionnaires", Journal of European Industrial Training, Vol 18 No 1.

Muhlemann et al 1992

Muhlemann, Oakland & Lockyer, (1992), Productions and Operations Management, Pitman Publishing, London

Reid & Barrington (1997)

Reid, M. & Barrington, H., (1997), Training Interventions, Institute of Personnel and Development, London

Roberts (1986)

Roberts, B., (1986), "Education and training programs to support MRP implementations", Material Requirements Planning Reprints, APICS Inc, pp. 85-104

Technical Committee M/136 1975

Technical Committee M/136, (1975), Glossary of Production Planning and Control Terms, British Standards Institute, London

Turnipseed et al (1992)

Turnipseed, David L.; Burns, O. Maxie; Riggs, Walter E., (1992), "An implementation analysis of MRP systems: a focus on the human variable", Production & Inventory Management Journal, Volume 33 Number 1, pp. 1-6.

Udo & Ehie 1996

Udo, Godwin, J. & Ehie, Ike, C., (1996), "Critical success factors for advanced manufacturing systems", Computers & Industrial Engineering, Volume 31 Issue 1-2, pp. 91-94

Vollman et al 1992

Vollman, T., Berry, W. & Whybark, D., (1992), Manufacturing Planning And Control Systems, Irwin Inc., Homewood, IL

Wallace 1985

Wallace, (1985), MRP II: Making It Happen, Oliver Wight Limited Publications, New York

White 1980

White, E.M., (1980), "Implementing an MRP system using the Lewin-Schein theory of change", Production & Inventory Management Journal, First Quarter, pp. 1-12.

Figures

Figure 1: The Education & Training Cycle

Figure 2: Relationship Between Job Role and Required Knowledge

Figure 3: Example Relationships between Activities and Associated Concepts