Automatic generation of figural analogies with the R package 'IMak'¶

Short instruction about figural analogy making and test administration¶

Introduction¶

Even though very concise instructions for item generation are available inside the package, some remarks are made here in order to help developers construct their own items efficiently. Additional instructions can be found in Blum and Holling (2018). The package provides two functions at the moment, namely 'build_fa' and 'plot_fa'. The first of these functions can be the most challenging, while the second one is mainly used for plotting. The functions will be explained in a very general way to help developers acquire basic knowledge leading to the creation of items. After that, it is recommended that a thorough understanding of the package is made by studying and using the parameters of each function (the so-called learning by doing). Two important steps to consider for automatic item generation are the following:

1. Use the parameters of the 'build_fa' function to specify the rules, and assign its output to an object. There are five general rules that can be manipulated with function 'build_fa': main shape rotation, main shape reflection, trapezium rotation, line segment subtraction, and dot movement. Each general rule can be converted into a specific rule when the user gives precise arguments to their respective parameters. These parameters are the following according to the order in which the general rules were just mentioned: 'main.rot', 'mirror', 'trap.rot', 'subtract', and 'dot.mov'. The guidelines concerning how to apply each rule are available inside the package. It should be remembered that 'main.rot' and 'mirror' are not allowed to be altered together inside the function. When applying a rotation rule, it is strongly recommended to work with two numerical values that are not so distant to one another (see the examples inside the package).
2. Once the object is created with the 'build_fa' function, use the 'plot_fa' function to plot the information contained inside the object: plot_fa(object). If, at any time, you wish to save your items in a specific directory, then provide it between quotation marks in the following way: dir <- "print here your directory"; plot_fa(object, directory = dir). Please do NOT save the plot in a directory by using the options provided by the R interface buttons; instead, the 'directory' parameter inside 'plot_fa' can do this effectively and straightforwardly. If 'info' is not altered, then an additional CSV file is saved in the same folder where you save the items. The CSV file contains information about right answers to every item (given the order from left to right and top down) as well as the name of every general rule used.

Suggested instructions¶

For the administration of items, easy examples of each general rule should be provided to individuals, and it must be at least stated that a rule might be presented alone or together with other rule(s) in the same item. It is convenient to avoid technical terms like “rules” or “analogy” by replacing them with commonly understandable sentences like “changes between the above-seen shapes” or “the same change seen before should also occur here to complete the missing shape”. Finally, we recommend that sociodemographic data is recollected, as well as to inform about the anonymity of responses and the participants’ voluntary collaboration if appropriate. The following draft can be useful to prepare the written instructions. Please replace the sentences within square brackets with the appropriate contents.

The exercises you will solve consist of two parts. In the [upper / left] part, a matrix with three shapes is presented. A fourth shape is missing, and the space that it is supposed to occupy has been filled with a question mark. In the [lower / right] part, eight options of shapes are shown, and you will have to select the missing shape out of these options by [clicking on / marking] it. If you think that the missing shape is not among the options, choose the option “No answer is correct”. If the exercise is so difficult that you cannot find a proper answer, then choose the option “I don’t know”. Throughout the exercises, you will find different types of changes between the shapes of the matrix, either from left to right (thus in the rows) or top-down (thus in the columns). It is your task to recognize these changes and to choose, out of the given options, the one which fits in the place with the question mark. Please do not take notes while doing the test. For this task it is particularly important that you work alone and concentrate. Avoid distractions, for example from other people or media. On the next pages you will be able to work through a series of practice items.
Practice Item [n]. Focus on the [upper / left] matrix. If you see the first row, you will find that [a specific, quantifiable change is made from the left shape to the right shape], while both shapes are similar in the rest of their properties. As for the second row, the same part of the left shape should also [perform the same change] to obtain the missing shape, while these two shapes share the rest of their properties as well. So, what would be the right answer here?

Scoring¶

Before scoring, it must be noted that two important pieces of information about the items should have been saved in advance, namely the right answers and the rules applied. This information is provided in the CSV file described for the 'plot_fa' function, and it is also available as output inside R when running 'plot_fa'. Once the data has been recollected, responses should be converted to binary values for the case of right (1) and wrong (0) answers respectively. Then the total score of a give test taker is simply calculated as the sum of all binary values belonging to that person.

Validity and reliability measures¶

'IMak'-generated items have already been proofed and validated. According to the main results presented by Blum and Holling (2018), test validity and reliability can be regarded as acceptable, as shown by assessments of unidimensionality, Rasch model item and test fit and convergent validity. Several indicators reveal that item difficulty can be predicted by a set of cognitive operations (i.e., rules) to a satisfactory extent, and specific rule-based contributions to item difficulty can be interpreted. For instance, all main shape rotation rules had a strong impact on item difficulty, followed by the reflection rule.
Research with the 'IMak' package is at its very beginning. Therefore, the author strongly recommends other researchers to construct items with 'IMak' and to establish further psychometric assessments, with the purpose of understanding such package in its full complexity.

Name and email¶

Diego Blum, blumworx@gmail.com

References¶

Blum, D., & Holling, H. (2018). Automatic generation of figural analogies with the IMak package. Frontiers in psychology, 9(1286), 1-13