Berkeley Open Learning Technology (Bolt)

Bolt is a software infrastructure for creating web-based training courses, and is designed to meet the needs of:

• Distributed thinking projects: volunteers must be trained to perform various tasks.
• Volunteer computing projects: educating participants can increase their enthusiasm and commitment, and attainment of given knowledge levels can be shown as a web site "status symbol".

These areas have the following properties:

• Churn: constant turnover, thousands of new students per day;
• Wide geographical distribution;
• Wide age distribution;
• Motivation: most volunteers have a pre-existing interest in the topic, and are motivated by acknowledgement (e.g. being marked as an "expert" on the project web site).

What Bolt does

Using Bolt, you can

• Create exercises of various types: multiple-choice, fill-in-the-blank, graphical, etc.
• Specify a course as a sequence of lessons and exercises.

Given such a course, Bolt does the following:

• It guides students sequentially through the course;
• If the student fails an exercise, they repeat one or more lessons and retry the exercise(Bolt courses are designed to be "fail-proof");
• Each student's progress is recorded in a database, and when they return to the course later they resume at that point.
• Bolt maintain an estimate of each student's mastery of the course material.

In addition, Bolt lets you create better courses; specifically, you can

• make statistically valid comparisons of alternative lessons;
• make "adaptive" courses in which different lessons are used for different groups of students

This is done as follows:

• Bolt records the timing and results of each student interaction (viewing a lesson or completing an exercise) in a database.
• Demographics (age, sex, education level, nationality) are stored for each student.
• Course documents can have various types of "control structures". For example, they can specify that a lesson should be chosen randomly from a given set, or should be chosen based on student demographics.
• Bolt offers analytic tools that let you evaluate the effectiveness of your lessons, and that help you make your course adapt itself to different types of students.

Creating exercises

An exercise is a PHP script. Here's an example consisting of a multiple-choice question:

<?php
require_once('../inc/bolt_ex.inc');
echo 'Conifers are so named because:';
bolt_exclusive_choice(
   array(
      'They carry their seeds in cones.'
      'They are cone-shaped.',
      'They originated during the Coniceous era.',
   ),
);
?>

Each time the question is shown, the choices are shown in a random order. The correct choice is always the first element of the array.

Here's an example that shows an image; a correct answer is a click in the indicated subrectangle.

<?php
require_once('../inc/bolt_ex.inc');
echo "Click on the dog's nose:<p>";
bolt_image_rect(
   'dog.jpg',
   array(100, 60, 110, 70)
);
?>

Bolt supplies functions for other types of questions, such as inclusive multiple-choice and fill-in-the-blank. An exercise can include multiple questions.

At the implementation level, an exercise has three functions:

• When invoked with $mode_show set, it shows the exercise.
• When invoked with $mode_score set, it computes a score based on the responses stored in $_GET, and assigns the score to $score. Its text output, if any, is ignored.
• When invoked with $mode_answer, it shows and "answer sheet" based on the responses stored in $_GET. If the response is correct and no answer sheet is to be shown, it sets $no_answer_sheet.

Bolt's exercise primitives perform all these functions for you; however, you're free to implement your own exercises.

Course documents

The structure of a Bolt course is defined by a ​JSON document. Here's an example of a course with two lessons followed by an exercise:

{
   "name": "Identifying Sierra Conifers",
   "description: "Learn to identify the major conifers of California's Sierra Nevada",
   "items": [
      {
          "type": "lesson",
          "name": "Introduction",
          "file": "intro.html"
      },
      {
          "type": "lesson",
          "name": "The Linnaean hierarchy",
          "file": "linnaean.html"
      },
      {
          "type": "exercise",
          "file": "linnaean.php"
      }
   ]
}

Course items can be grouped into sets; for example:

{
   "type": "set",
   "show_n": 1,
   "order": "random",
   "items": {
      {
      ...
      }
   }
}

The attributes of a set include:

• show_n: the number of items in the set to show
• order: whether to show the items sequentially or randomly

Items (lessons, exercises, and sets) can include properties, e.g.:

{
   "type": "lesson",
   "name": "The Linnaean hierarchy",
   "file": "linnaean.html"
   "properties": {
      "verbal_level": 12,
      "detail_level": 0.8
   }
},

When Bolt has a choice of items (e.g. when it encounters a set from which a single item is to be shown) it calls, for each item, a course-supplied matchmaking function, passing to it the student object (which includes demographics such as age) and the item's properties (represented as a PHP object). The matchmaking function returns a number representing the estimated effectiveness of that item for that student, and Bolt chooses the item with the highest value.

Memory refresh

Bolt offers a memory refresh system that periodically repeats exercises and, if necessary, lessons. Memory research suggests that this is necessary for students to shift learning to long-term memory. This mechanism works as follows:

• A sequence of inter-refresh intervals is defined. For example, (7, 28) means that an exercise should be repeated 7 days after it is first taken, and then every 28 days thereafter.
• Bolt provides a function that returns the set of items, for a given student, for which refresh is due. Your course can use this function to implement a "Review now" button on web pages.
• Bolt provides a "review mode" in which the student is presented with exercises due for review.

Analytics

Bolt offers two web-based analytic tools, course maps and lesson comparer. You can use these tools to iteratively refine your course:

1. Develop an initial course
2. Operate the course until a statistically significant sample size of interactions exists
3. Use the course map tool to find problem spots
4. Develop alternative lessons
5. Operate the course some more
6. Use the lesson comparer to find better lessons or to do demographic adaptation
7. go to 1.

Course maps

A course map shows you the overall flow of students through your course (in the style of Charles Minard's map of Napoleon's march to Moscow in the war of 1812), revealing the points where they are getting bored or discouraged.

A course map shows you graphically how many students enter each step of the course, how many seconds they spend there, and their average performance on exercises. You can get a color-coded breakdown by any student attribute, and you can select a subpopulation based on attributes.

Lesson comparer

You can develop several alternative lessons for the same concept and, using the "set" construct, arrange for them to be selected randomly, followed by a single exercise. You can then use Bolt's lesson comparer tool to study the results. The tool will tell you, for a given statistical confidence level:

• whether one lesson is worse than another, e.g. students viewing lesson A score worse than students viewing lesson B
• whether a given lesson is better for a particular demographic subgroup, e.g. a lesson is highly effective for females under 18.

Other features of Bolt

By default, every Bolt page includes an "ask a question" link. These questions are delivered as private messages to the course developer.

Identification of course elements

A course can change over time. Elements (lessons, exercises, sets) may be added, deleted, or reordered.

A student has a "context" in a course - a sequential position, and a set of review items. What if the course changes, and these items change or disappear?

Each element may be given a "logical name". For items, this defaults to the file name.

A user's sequential position is stored in the DB as the logical name or last item viewed. On continuation, Bolt looks up that item.

The bigger picture

Bolt's primary goal is to serve the needs of volunteer computing and skill aggregation projects. However, we believe that it can also become a tool for research in education and cognitive science, for a variety of reasons:

• Experiments can be deployed with large sample sizes.
• Experiments can be deployed immediately, with no dependence on the academic calendar. Significant results are available in days rather than months or years.
• The student population varies widely in age, language, and education level.
• The student population is self-selecting for interest in the topic area, and has diverse learning goals.
• Experiments are not limited by standards or syllabi.
• Experiments can be conducted without dependence on educational institutions or teachers.