The CSL is currently using three different type of analysis techniques for anomalous cognition data.
In the rank-order method, an analyst, who is blind to all the experimental details, is presented with a single response from an AC trial and a target pack which contains the intended target and a number of decoy targets. Our target packs usually contain five targets that are carefully selected to be as different from one another as possible.
Regardless of the quality of the match, the analyst must chose which of the five targets best match the response, second best match, and so on for all five targets. Under the null hypothesis (i.e., no anomalous cognition) on the average the analyst will assign the intended target a rank of three. That is some times she/he will pick the intended target better than a third place match and some of the time worse than a third place match.
Over a number of trials, we compute the average observed rank and compare it statistically to the expected average rank of 3. Typically, we see an average rank of 2 in our AC studies.
The fuzzy set method allows the analyst to express the degree to which a concept is present in the target and response rather than being forced into a "yes" or "no" decision.
For example a target concept might be city . A photograph of Manhattan would receive a one on a 0-1 scale; whereas, a village on the banks of a river might only receive a 0.3 on the same scale. Likewise a response that showed buildings and streets and was labeled "city" would receiver a one for city, while another response with only a cross-hatched drawing would only be assigned a value of 0.2.
Currently our fuzzy set descriptor list is 130 elements long ranging from simple geometric factors to high-level concepts such as fort.
Three quantities are defined for analysis:
To obtain a large figure of merit, the sizable fraction of the target must be described with very little incorrect material in the response. A probability statistic is obtained by computing an FM for all targets in the pool and using a rank-order analysis for the FM of the intended target.
An assessment rating differs from a rank-order in that an attempt is made to determine the degree to which a response matches a target (i.e., decoy or intended). We currently use an 8-point scale where 8 implies a near-perfect match and a zero means that nothing in the response matches the target. Each level is defined by a declarative statement that must be satisfied before that level can be assigned as a rating.
In an experiment, no statistical analysis can be used with the raw ratings, but they can be used appropriately either to rank-order targets based upon the rating or as a variable in a correlation study. For example, how does an AC response depend upon a physical property of a target?