Reliability, Validity and Generalisability
Reliability
According to Wiener et al. (2017), the reliability of a measure is the degree to which a measurement technique can be depended upon to secure consistent results upon repeated application. Reliability also measures the internal consistency of a measuring instrument by the amount of intercorrelation between a set of items. When there are statements measured on the Likert scale in a survey questionnaire, the Cronbach alpha coefficient is the most appropriate measure of internal consistency (Laerd Statistics, 2018).
Validity
Validity determines whether an instrument measures what it is supposed to measure (Wiener et al., 2017). During the piloting stage of the questionnaire, the face and content validity are normally verified by respondents' feedback. A more scientific and objective way of validating a measuring instrument is via construct validity which, determines whether the data contains underlying constructs, i.e., exhibits dimensionality (Ahmad and Sabri, 2013). One way of verifying construct validity is by way of factor validity (Nako and Barnard, 2012), which consists of subjecting the data to factor analysis and then observing the results for Bartlett's test of Sphericity. The instrument, and hence the data, is deemed to be valid if the p-value does not exceed 0.05 (Field, 2016), the usual default level of significance.
Generalisability
You need to discuss about the extent to which your results and findings are also true of other populations. Remember that not all studies are as generalisable as others, namely case studies. Bearing in mind the delimitations of your research and its external validity, you need to discuss how generalisable your results are likely to be, and why.
Data Analysis
Specify and justify the use of the techniques and tests that you intend to analyse your data with. For example, if you have sampled texts, or have a lot of qualitative data, will you be using semiotics analysis, discourse analysis and so on? You may also mention which software (if any) that you have used, e.g. NVivo and so on and why you chose to use these particular software.
With regards to quantitative analysis, mention (with justification) which software you used, e.g. SPSS, MS Excel, Stata and so on. With hindsight, you will realise the importance of the chosen measurement scales in your survey questionnaire, as they are crucial for the implementation of statistical tests and advanced techniques. This is why it is important to have an idea of how you will analyse your data, prior to designing your questionnaire.
In quantitative data analysis, tests and techniques should be judiciously chosen in order to maximise the accuracy and reliability of your findings. Ensure that you check all the necessary assumptions before you use any test or apply any technique. As an example, normality testing via the Shapiro-Wilk test or the Kolmogorov-Smirnov test is essential before deciding whether to use parametric or non-parametric tests. There is a wide variety of techniques that may be used, just to name a few well-known ones:
- Descriptive analysis: charts and tables, measures of central tendency, dispersion, skewness and kurtosis, method of weighted means (including clustering), gap analysis (SERVQUAL)
- Parametric tests: paired t-test, independent-samples t-test, one-way ANOVA, two-way ANOVA
- Non-parametric tests: Chi-Squared test of independence, Mann-Whitney U test, Kruskal-Wallis H test
- Multivariate techniques: Correlation analysis, exploratory factor analysis, multiple regression analysis, logistic regression, structural equation modelling (SEM)
Ethical Considerations
According to Cooper and Schindler (2014, p.108), "the goal of ethics in research is to ensure that no one is harmed or suffers adverse consequences from the research activities." Therefore, you need to explain how you have adhered to ethics in your research, particularly if it includes human subjects. The usual aspects covered in this section are:
- Informed consent and right of withdrawal
- No harm to participants
- Anonymity and confidentiality
- Authorisation for data collection
- Violation of intellectual property rights (plagiarism)
References
Ahmad, NS and Sabri, A (2013) "Assessing the unidimensionality, reliability, validity and fitness of influential factors of 8th grades student's Mathematics achievement in Malaysia", International Journal of Advance Research, Vol. 1, No. 2, pp. 1-7.
Cooper, DR and Schindler, PS (2014) Business Research Methods (12th edn), McGraw-Hill, New York.
Gill, J and Johnson, P (2010) Research Methods for Managers (4th edn), Sage, London.
Laerd Statistics (2018) "Cronbach's Alpha (α) using SPSS Statistics" [online] Available from: https://statistics.laerd.com/spss-tutorials/cronbachs-alpha-using-spss-statistics.php
Malhotra, NK (2019) Marketing research: An applied orientation (7th edn), Pearson/Prentice Hall, Upper Saddle River, NJ.
Nako, Z and Barnard, A (2012) "Construct validity of competency dimensions in a leadership assessment and development centre", African Journal of Business Management, Vol. 6, No. 34, pp. 9730-9737.
Saunders, M, Lewis, P and Thornhill, A (2016) Research Methods for Business Students (7th edn), Pearson Education Limited, England.
Wiener, BJ, Lewis, CC, Stanick, C, Powell, BJ, Dorsey, CN, Clary, AS, Boynton, MH and Halko, H (2017) "Psychometric assessment of three newly developed implementation outcome measures", Implementation Science, Vol. 12, No. 1, pp. 1-12.