ModFit LT Rule-Based Training^™

Proficiency

Question: 

What criteria from the proficiency exam should a laboratory use to determine when a technician is suitably trained to analyze DNA histograms?

Answer: 

Before we give you our answer, let us present a little background to you so you can better understand our position.  Before we published our paper on "Optimizing Flow Cytometric DNA Ploidy and S-Phase..." (see Cytometry 46:121-135, 2001), we needed to make sure that operator variability was essentially eliminated.  The bottom line message of this paper was that if one makes a number of adjustments to both DNA ploidy and S-phase, highly prognostic node-negative patient categories can be established from a single prognostic model.  Since this study incorporated many laboratories from here in the US and abroad, the inescapable conclusion is that it looks like the test and its interpretation can be standardized worldwide.  Obviously, the paper cannot be considered proof because of the relatively limited number of laboratories tested, but it is certainly suggestive and represents the first good news concerning this test for over a decade.

Since our intent was to demonstrate that this test could be standardized, we also had to tackle the detail of how to eliminate operator variability and bias in DNA histogram analysis.  In other words, it was not good enough for one person to analyze all the DNA histograms and show the highly significant prognostic categories.  We had to show that anyone, if trained properly, could obtain equivalent results.

This problem was initially intractable since there were so many areas in the modeling decision-making that resulted in similar results.  It wasn't until we changed the problem from obtaining similar results to identical results that we started making good headway solving this problem.  The process started with two operators independently analyzing a single DNA histogram.  If there were any difference at all, we hunted down the reasons behind this difference and created rules that if followed would eliminate the difference.  In other words, by tolerating no difference (<0.01% S-phase difference) we were able to start creating a concrete rule set.

This process took us far along to an eventual solution, but not all the way.  There were still some model decisions that could not be eliminated by simple rules.  One was the input of a linearity factor and the other involved the relationship between the G2M and G1 peaks.  Within any decent modeling program there is a factor that is used to control the aggregate model component and the relationship between G2M and G1, if they happen to be dependent on each other multiplicatively.  The user has traditionally entered this linearity factor.  We found there were biases in its selection, which ultimately changed the results.  We finally eliminated this source of error by creating an AutoLinearity adjustment in ModFitLT 3.1 that automatically finds the best linearity factor for a given DNA histogram.

The other intractable problem we had was the determination of whether a G2M should be dependent on G1 or should be allowed to float.  Traditionally, this decision was based on a rather subjective decision of whether the G2M was clearly visible or not.  This decision was confounded by the fact that other neighboring model components such as an aggregate doublet or another ploidy population could destabilize the position of the G2M.  We initially tried to fix this problem by creating criteria rules regarding the position of the G2M, but they kept getting more complicated and were impossible to standardize between operators.  We finally solved this problem by making all default models in ModFitLT 3.1 and above use dependent G2M's.  In other words, the position of the G2M's is dependent on the position of the G1 and the calculated optimal linearity factor.

This approach resulted in operators obtaining nearly identical results (<0.1% difference in S-phase).  We've successfully used this training method in many flow cytometry courses without much difficulty.

Given all this discussion, here is our answer.  If a prospective operator takes the proficiency test and there are any red marks on any of the pages, they must go through the training of the relevant sections again.  A red score is made if any S-phase result is off by 0.1%.  Even though this criteria is far less than what is clinically significant, it is still logical to hold the prospective operator to this high standard.  The reason is that if they are off by 0.1% or more, they did not follow one or more of the rules and are likely to not follow them for future clinical samples or they are using the wrong version of ModFitLT (version <3.1).  It may be that no matter how many times they go through the training, they just can't get a few files.  Under this condition they should go back to our reference reports and understand how we analyzed the files differently.  It's possible that they will still disagree with our analysis.  Under that condition they can challenge the analysis.  Initially, Verity will need to respond to that challenge, but eventually someone there at your laboratory can do it as well.  There are three results possible from a challenge(s): 1) they didn't see one of the rules in operation and will learn from the experience, 2) we made a mistake and didn't follow one of our rules or 3) the rules result in an ambiguous situation resulting in two or more correct approaches.

The first possibility is just part of the learning process.  The second possibility would be corrected immediately by Verity and their analysis stands.  The third possibility might necessitate a rule change for the next version of the training system.

We believe the above operating procedure is the only logical way to proceed.  It sounds initially like it can't work, but we think it can.  It sounds like it might be too expensive in technician time, but we think the reverse will be true.  Creating a really tough proficiency exam at the outset will have long-term benefits that will outweigh the multiple passes through the training system. 

There are likely to be problems with already "DNA histogram analysis savy" technicians.  We must expect this problem and attribute it to a process of breaking down biases.

ModFit LT Rule-Based Training
Available now!

Click here to request a free download.