Hi
I used Sawtooth Designer to generate 10 MaxDiff module versions for fieldwork, allocated randomly to each of 600 respondents. MaxDiff module was composed of 18 tasks, 6 items in each, 35 items in total. Recently, client asked to analyze a Subgroup of respondents, n=393.
I noticed, that not all MaxDiff versions were displayed evenly in the Subgroup. One of the 10 versions was displayed only 32 times (8.1%), while one other - 45 times (11.5%). Going further, I checked display frequency of each of 35 items. The mean frequency varied from 2.96 to 3.14 - some items were displayed up to 6% more often than the others.
I used t-test to compare mean display frequency between the Subgroup and the rest of sample - it showed a few significant differences, but eta2 were rather low (typically between 0.01 and 0.10). Friedman test, comparing the display frequencies of 35 items inside the Subgroup, was statistically significant - the 6% bonus seems to be relevant.
So my first question is how to determine if I can safely analyze results for a subgroup for which quotas on MaxDiff versions were not controlled?
Moreover, I also noticed some intriguing patterns for the Overall Sample as well. The imbalance of version sage was rather small, close to perfection - from 59 (9.8%) to 61 (10.2%) per version. Nonetheless, mean frequency of display of 35 items was quite varied - from 3.00 to 3.11 - some items were displayed up to 4% more often than the others. Again, Friedman test was statistically significant, which - by theory - would lead to a conclusion that items were not displayed completely randomly. And that is stated only for the aggregated level of entire module, I haven't checked e.g. average display position in exercise (as 1st/2nd.../6th), nor average excercise number in the module (1st/2nd/...18th), where most likely I would observe additional significant discrepancies.
So my second question is how Sawtooth CBC-HB is addressing the issue that some items are displayed more frequently than the others?
Regards
Piotr