Dear all,
I have collected importance and satisfaction data for 20 attributes (using 10 choice sets each with 4 attributes so that each attribute is repeated 2 times per respondent) by using the best-worst scaling technique. I have collected a sample of about 650 including complete and incomplete responses. To start with the analysis, I have a few doubts:
1. To identify bad respondents using RLH cut-off value at some confidence level, say, 95%, obtained from HB estimation of say 1000 random respondents, usually, it is specified that the maxdiff experiment should be well powered (at least each item is repeated 3 times i.e., 3x). But my experiment is sparse with only 2x repetition of each item for a respondent which causes more loss of real respondents. Is this method of using the RLH cut-off value at 95% would be good enough or reliable to identify bad respondents for my sparse maxdiff experiment (2x repetition)? For my data set, what would be the best possible confidence level (80, 90, or 95%) for removing bad data?
2. After identifying RLH cut-off value, should all the respondents with RLH below that value be removed at a time or each one is removed and the fit statistic for the whole data in HB be checked for any improvement? When to stop cleaning data and understand that all bad respondents are removed? Is it based on the fit statistic? Fit statistic for my raw data before cleaning using HB analysis is 0.388. What could be the fit statistic for a good HB model?
3. In the case of Logit model, what are the goodness of fit measures, and how to read and understand the model? What could be the possible values for percent certainty and chi-square (for my raw data, percent certainty is 4.69, chi-square is 1652.06, and relative chi-square is 86.95)? What is the meaning and significance of percent certainty?
4. Is the HB model or aggregate logit model a better method to rank my 20 attributes in case of the present sparse maxdiff experiment (2x repetition)?
Kind regards,
P. Vaishnavi.