Dear Sawtooth professionals,
I have a question about "In sample" and "out of sample" validation.
I have collected the initial of 80 customer panels, using CVA scenarios without holdout sets.
Upon reading the technical paper "how many holdouts for model validation?", something came to me and wondered:
I plan to collect 450 customer panels more using CVA scenarios, this time by adding three holdout choice tasks.
If I use the sample of 80 for "out of sample" validation and plan to use the second phase of data collection, 450 customer panels using exact same CVA scenarios used for initial 80 samples along with 3 holdout sets as "in sample" validation of my model, am I misunderstanding these concepts?
Or, if I had 3 holdout sets from the beginning for both data collection of 80 and 450, would this example make sense?
Other technical paper stated that a different set of questions needs to be asked for "out of sample" holdouts.
Would including 3 fixed holdout sets to the original CVA questions would be sufficient to be considered as a different set of questions?
I really appreciate the response. Thank you.