The Wales-Wide Web

Hi - been quiet lately. too busy writing to post a lot. But I do have a backlog of posts for the next two weeks.

for the last few days I have been working on a draft discussion paper for the European Commission sponsored research project on the use of ICt for Learning in Small and Medium Enterprises (SMEs). It will not be a shock to most of you to learn that we are finding that few SMEs are involved in 'formal' elearning but that many, many SME employees use the internet everyday for learning at work. I am looking forward to writing a paper on this which I have been invited to present at Educa on-line Berlin. But, before i can get on to that,

I have to produce a thematic issues paper on Quality and the use of ICT for learning in SMEs. It is not finished. But I am happy to share with anyone who reads this blog the work I have done so far. As I say in the paper "This document is only intended as an unfinished public beta version. Just like unfinished beta software, I have published it in this form for feedback from project partners, friends, colleagues and anyone else interested in the subject." It is as yet unreferenced and one section, on evaluation tools, is not yet written.

So much for the health warning. But I really would appreciate feedback. If you press more at the bottom of this entry, you will get the introduction. A draft copy of the paper is provided as a downloadable rtf document. But, if you really are interested, and I realise that quality and ICT based learning is not the sexiest of subjects, I have created a wiki version. Please go to the wiki, create yourself an account and add your ten-pennyworth.

Extended text for this entry:

The question of the quality of e-learning is problematic and fraught with difficulties. These difficulties are compounded when it comes to examining the quality of the use of ICT for e-learning in SMEs.

The development of e-learning products and the provision of e-learning opportunities is one of the most rapidly expanding areas of education and training. Whether this is through an intranet, the internet, multimedia, interactive TV or computer based training, the growth of e-learning is accelerating. However, what is known about these innovative approaches to training has been limited by the shortage of scientifically credible evaluation. Is e-learning effective? In what contexts? For what groups of learners? How do different learners respond? Are there marked differences between different ICT platforms? Does the socio-cultural environment make a difference? Considering the costs of implementing ICT based training, is there a positive return on investment? What are the perceptions of VET professionals? What problems has it created for them?

However, whilst some have been desperately seeking answers to the question `What works and what doesn’t work?’ and looking for ways of improving the quality of e-learning, the response by a large sector of the community of e-learning developers and practitioners has been a growing preoccupation with software and platforms. There has been only limited attention to pedagogy and learning.
If we have limited access to research about the evaluation of elearning in general that is even more so when it comes to SMEs. Our own research has suggested the difficulties and unreliability of survey data on elearning in SMEs and whilst qualitative case study findings are interesting, they can only represent a very limited sample of enterprises. Furthermore the research we have undertaken through the Leonardo ICT and SME project has suggested that there are considerable differences in e-learning take up in SMEs according to sector and to size. If approaches to e-learning differs by sector it may be possible that approaches to quality will also differ accordingly. We are aware that Cedefop has undertaken a study in this area entitled 'Quality and e-learning in European training’ (2003) but the cover price of €195 has not only prevented us from accessing the findings, but, we suspect, has been a barrier to dissemination to SMEs. The summary which was made available for free revealed that, in a survey of 433 teachers and trainers, 61% of respondents from the public and private sectors indicated they felt the quality of e-learning was fair or poor.
One of the major findings of the ICT and SME project has been that whilst there is little evidence of widespread use of ICT for formal learning in SMEs, there is growing use of ICT for informal learning. The recently completed Leonardo Eval 3 project, looking at the evaluation of elearning, concluded that although the evaluation of elearning was more complex than traditional learning, due to the increased number of factors or variables involved, there was no fundamental difference sin the models and approaches to be deployed. If this is so then any approach to quality – or certainly to evaluation – of the use of ICT for learning in SMEs may need to consider how to evaluate the quality of informal learning. This is also an area where research is lacking.
There is widespread recognition that quality is a major issue, from academics, form the e-learning industry and from policy makers.
As early as 1995 Dick commented that “Quality and design of e-Learning courses, however, are sometimes compromised in an “ . . . effort to simply get something up and running” in response to pressing consumer demands”. Many educators and researchers have voiced concern over the lack of rigorous evaluation studies of e-Learning programs (see for example Arbaugh, 2000; Howell, Saba, Lindsay, and Williams, 2004; Lockyer, Patterson, and Harper, 1999; Robinson, 2001). McGorry (2003) adds, “although the number of courses being delivered via the Internet is increasing rapidly, our knowledge of what makes these courses effective learning experiences is limited” (p. 160). In an economic environment marked by intensive competition between educational institutions, producing and ensuring quality e-Learning programs will be a competitive advantage to attract learners to post secondary institutions (Daniel, 1996; Duderstadt, 1999).
The American Society for Training and Development said in 2001 “Although recent attention has increased e-learning evaluation, the current research base for evaluating e-learning is inadequate…Due to the initial cost of implementing e-learning programs, it is important to conduct evaluation studies.”
The literature reveals a distressing gap between the use of technology and sound pedagogical models (Khan 1997; Salmon, 2000; Willis 2000). Several researchers have written about the need for quality standards to ensure the academic integrity of e-Learning programs (Benson, 2003; Carstens and Worsfold, 2000; DeBard and Guidera, 2000; Salmon, 2000; Speck, 2000). Defining these quality standards, however, can be challenging. In a qualitative study of 13 participants from six different stakeholder groups engaged in developing an online degree program, Benson (2003) found that although everyone wanted quality courses, stakeholders brought different definitions of quality, which impacted the planning process and shaped the learning experiences.
Marquardt and Kearsley (1998) suggest “evaluation is particularly important in the context of technology use because it [technology] is highly susceptible to fads and marketplace trends” (p. 246). Robinson (2001) reports that in her three-year international study of distance learning initiatives, evaluation efforts were limited, largely due to lack of time and expertise.
From a policy makers viewpoint quality is seen as one of the four major priorities for the European Union e-learning Initiative which has funded a series of projects on the topic, The “Capitalisation report on the Leonardo da Vinci 1 programme”, one of the biggest sponsors of innovative e-learning projects in European VET, identified the lack of systematic evaluation as being the major weakness in e-learning projects.
Many within the e-learning industry, developers, content vendors and content developers would also recognise the issue of quality. However, there is little if any consensus on how the issue should be approached.
The industry group, which lobbies on behalf of the major publishers has this to say:
“Where e-learning public policy focuses on the development of low-value, low cost and poorly standardised nuggets designed by amateurs or service providers as promotional material, with limited public commitment or support, there is no incentive for publishers to heavily invest in the production of quality editorial content. However, there has been a growing demand for quality assurance for digital pedagogical content. We are firmly convinced that the best way to achieve quality assurance for content is the editorial process, which should be stimulated so that commercial success can nurture a virtuous development cycle.”
In the dark code of European speak “stimulated” is another word for subsidised. Basically the industry groups answer to quality issues is to give them public funding to edit materials and trust to market processes. As an aside, they are also deeply concerned at the role of public organisations, particularly the broadcast organisations, providing free access to high quality learning materials. In the interest of free market competition they insist this should only takle place within the confines of public private partnerships!
Opposed to the publishers are those who see quality emerging through encouraging teachers and trainers (amateurs in the publishers views) to use their expertise in developing and critically sharing learning materials. In a position paper written with Ray Elfink for a European policy seminar last year, I argued: “The lack of easy access to attractive and compelling educational content is one of the major barriers to the development and implementation of e-learning in Europe. Most educational content is pedagogically poor, consisting overwhelmingly of sequenced text based materials and exercises. Furthermore, the subject and topic range is limited. This is particularly so for vocational and occupational subjects and in lesser-used languages.
Time and cost of production are major barriers to the production of quality learning content leading to the present interest in standards based, reusable content and to the sharing of content between institutions. In many areas content developers require not only technical and pedagogic skills but also deep subject knowledge.
Publishers have an important role to play in the development of content. However, as with traditional learning materials, much content in the future will of necessity be produced by teachers. There are also intriguing possibilities for learner developed content and there is great potential from public content repositories especially from cultural heritage and media organisations. It could be argued that there is already a wealth of rich learning materials available through the web. The problem lies in how these materials can be described and accessed and pedagogically deployed.”
As the reader may guess my proposals for EU policy also differed widely from the publishers. I proposes (amongst other things) supporting the development of tools for content production, distribution, sequencing and deployment. Access to easy to use tools, I said, is more important at present than is directly subsidising the production of content itself. Furthermore, the EU should support the Creative Commons License. There seems little reason why education content produced with public funding - national or European – should not be required to be released under a Creative Commons Licence.
In a fast developing area there is no one single answer to the issue of quality of e-learning for SMEs. In this paper, I will next take a look at the (admittedly limited) empirical findings of the ICT and SME project on quality. I will go on to examine a number of different approaches to quality through evaluation. I will then explore issues around the quality of informal learning and finally will consider come of the implications for policy.

quality-1.doc