Our aim as a company is to help organisations with international teams and crews change and learn faster. As an online learning company we monitor progress and results of our students continuously to find out if we actually succeed in helping our clients and their workers in learning new things (gain knowledge, improve skills). Speed in acquiring new knowledge and skills is important because our clients operate in complex, dynamic environments. So we do not only measure if people learn new things, we also try to design effective, often pretty short programs and try to find out which of our learning interventions work best. We also personalise our programs continuously, using the data we collect. In that way we can help participants find the shortest road to reach their learning destination.
In 2017 we have done our most extensive research effort so far into the effect of our English language programs on the English language proficiency of maritime workers. So we collected data of almost 650 maritime professionals (most of them seafarers) that have been learning with us in the last two years. We have had more participants than this of course, but we tried gathered the data of students that went through our regular programs in the most ‘normal’ way, so we we would be able to compare things and draw sensible conclusions. In this article we share some of the conclusions of our research.
Digging into the data was essentially a humbling experience. Even though we collect a huge amount of data continuously, it’s certainly not easy to draw simple conclusions about our methods and actions, that have alle been developed not too long ago and have undergone a lot of changes, as we are a startup and innovator pur sang.
Diving into our data taught us that there is a lot more that we want to learn, that we must work even harder on collecting (enough of the) data in consistent ways and in accessible formats and that we want to get even better at asking the right questions and designing strategies to come up with the answers after some time.
The data we looked into were the results of almost 650 maritime professionals that were enrolled in our maritime English program. In this research we looked into the data of Russian and Filipino seafarers. We also had some groups of office workers, mostly Dutch and German, and there was a small group of seafarers that were from other - mostly Eastern European - countries. We have also collected a nice amount of data about Chinese seafarers that have studied with us but we have done a separate study into those data (you can read this article about a project to improve English proficiency of a group of Chinese seafarers). The numbers of participants from China are smaller than the those from Russia and the Philippines and there are some other differences that make it more difficult to compare the results of the Chinese seafarers with the results of the students included in this group of almost 650 mostly Russian and Filipino crew members.
All participants are tested first when they enroll in the program. Our client let’s us know what the required level of each participant is.
We call the difference between the level required from a participant and his actual level at entry of the program his ‘gap’.
The Russian seafarers were in most cases below the level required (of one group 96% was below required level, of another smaller group 90% was below the required level). Of the two Filipino groups respectively 50% and 58% of the participants tested at entry were below the level required by the principal.
The average gaps of the tested groups were (gap = required level - tested entry level in points on CEFR scale 1 - 100):
This research focusses on participants that are confronted with pretty tough required levels by the principal. Officers are required to have a CEFR level of C1, just like office workers. CEFR, short for Common European Framework of Reference for Languages: Learning, Teaching, Assessment, is a European Standard, more information can be found here..The lowest level required from the participants included in this research is A2 (required for ratings, support), for most other seafarers either B2 or in some cases B1 is required.
In the Russian group there was a relatively high percentage of officers, compared to the groups from the Philippines.
The data show that participants show measurable progress when we test them after finishing the first part of our online language program.
When seafarers participating in the program have test scores below the level required they are assigned courses that match their level and can help them improve as quickly as possible. These courses are general English courses, aimed at helping them improve their general English proficiency.
Participants are monitored closely. We collect data and in that way measure progress and identify ways to help them accelerate wherever possible. In our method (and that of our partner EnglishCentral) there are several ways to make clear where specific learning interventions can help participants improve specific skills. We will speak a bit more about this later in this article.
When a participant has finished the assigned courses and has worked on extra learning activities that are suggested during the program he will again be tested to measure progress. The test is done by Skype. Like the entry level test it is done during a live conversation with an English teacher. The progress by participants of the three groups we looked into were
The seafarers that went through a first set of courses that were assigned to them after the entry level test have in many case closed a significant part of their personal gap.
The courses are carefully chosen to match the entry level and help improve substantially in two months time. That requires regular exercise. We ask participants to exercise at least three times a week for at least three quarters of an hour to an hour. Regular exercise is necessary. The data show that it is best to take at least 7 or 8 weeks and exercise several times a week.
Sometimes participants finish their courses quickly by exercising several hours at a time for several days in a row. Those students often show less progress when tested. This was especially the case in Filipino group 2.
Students that exercise regularly during an extended period show better results at the second test (what we call the ‘level up test’).
During the entry level test (taken by skype) and the level-up test the teacher will rate the overall English language proficiency but will also specifically evaluate his:
In this research we’ve seen that the level of comprehension (listening) is in general higher than the average (overall) score. Pronunciation and fluency are often the lowest subscores of participants. This is something we see in the maritime world most of the time. Grammar and vocabulary vary a lot, we suspect that this subscore is most related to the educational background of the participant. Other skills, especially comprehension (listening) skills are more easily further developed in practice.
Our video-based methods and the speech technology and personalised feedback systems behind EnglishCentral are designed to help improve pronunciation and fluency. The test results show that this leads to measurable results.
Our most demanding clients require a C1 level for officers. People with that level have an English vocabulary of between 5,000 and 8,000 words. For A2 (the level required for ratings by most of our clients) the average vocabulary will be between 1,400 and 2,200 words.
To put this in perspective:
In Spot On Learning’s maritime English program we start by working on the General English vocabulary. More about the methodology behind building the vocabulary (and the personalised study list of words) can be read in this article on this website: Online methods to improve English communication skills of international teams and crews.
The next step is to build a specific maritime vocabulary. We start that part of the program when the participant has reached the required level of general English.
Our research into the 650 seafarers that have gone through our program that their vocabulary grows measurably during the program. The personalised study list feature with quiz functionality which is part of the EnglishCentral platform helps our participants building their vocabulary, both a more general vocabulary and a specific maritime vocabulary with words taken from IMO’s SMCP list.
Our experience and research into this group of around 650 maritime professionals gives us a lot of information about motivation of the participants. What we have learned is that it is important that the seafarers are informed about the ‘why’ of the English language program. While going we have experimented a lot with content and tone of emails and other communication. During the onboarding process we ask questions like
International laws, regulations and most shipping companies require that all seafarers speak English at a certain level. Did you know this?
We also have questions about how ‘digital and online’ they are outside their professional life. By asking questions and interacting in other ways we try to make our participants think about their own motivation, their own interests to invest in the program by their own time and efforts.
Sometimes we see that participants have problems getting started and building a habit of exercising regularly. The research shows that if participants are not informed well about the why and how of the program they often drag their feet.
In general we learned to maximise our efforts at the beginning of the program. If people start shortly after being invited, find their way quickly (with or without our support), use personalised features early they go through the program smoothly in most cases. Participants that procrastinate or get into trouble early often have more problems throughout the problem, need more support, progress slowly and show less favourable results.
We also found out that we - as a provider of the English language (and other) programs for maritime professionals - can not do all the motivating. We have to build a network of stakeholders and colleagues because they are influencers of our students. The principal can clearly communicate what they require from their crews and other workers. The crewing agent certainly has an interest (seafarers) and can influence the student by communication about the why. We see that both the ‘carrot’ and the ‘stick’ can be effective. Also we have learned that there are many cultural aspects of motivation that can make a big difference.
We share information about progress and results of our participants with the network of stakeholders. By doing this we build a supporting network around the participants.
One of the things that seems to influence motivation greatly is personalisation. The level test and level-up test are both conversations with a real teacher. The participants like this approach in most cases, though they can also be somewhat nervous about the test. After the test the participants are invited for courses that match their level. When they have spoken enough videos two personal features of EnglishCentral start working. Their personal pronunciation dashboard is generated and they can start using their personal study list to expand their vocabulary. We tell the participants how to use these features and we see that this motivates them to be more active on the system. We also see that feedback we give helps motivate them.
In the past two years we have learned a lot about how we choose interventions that help, how we can explain important workings of the system and personalised functionality. We have learned what motivates and stimulates most seafarers to work on the program and yes, we also have learned what NOT to do.
Continuous improvement is important on board ships but it is certainly also very important for us to help international crews and teams change faster. We collect enormous amounts of data and monitor as much as we can. In our team we try to take the time as often as possible to draw conclusions and improve things. This research has helped us greatly, hopefully it helps you as a reader that we shared some of it. Please contact us if you have questions about our research and our work.
Do you have questions about Spot On Learning or would you like to make an appointment? Get in touch with us!
Spot On Learning
1017 RR Amsterdam
+31 (0) 20 30 80 041