For a very long time, publishers could easily move on after publishing an instrument and establishing its basic psychometric properties. Times have changed now. Just publishing, ensuring that certified professionals use the instruments and taking care of operations are not enough. Recent debates on consequential validity are taking the field by storm.
A critical question that some psychometricians are asking is what should matter the most to the user when it comes to validity. There are no easy answers. But it is obvious that the real test of validity of an instrument is not just that it measures what it purports to measure but also that it has the impact on end users that it purports to have.
Succinctly put, this concept for many is an issue of ethical concern and not an issue of validity. Many others have called it the soft validity of an instrument. The key component of the debate however is that whatever we may call it, can we really escape it? Can a publisher or practitioner take a hands off approach and claim that while an instrument was proved to be valid over a tested sample, its real impact need not be evaluated?
This allows us to ask other questions too. Where does a publisher's or a practitioner's responsibility end when it comes to assessments and its impact. Would it be fair to hold a publisher responsible if environmental factors dilute the utility of an instrument?
It is definitely not one of the easier debates where numbers could hold the verdict. A more comprehensive view and probably a more paradigm shifting view of the field and its applications is required.