What’s the value of predictive models for drug discovery? Most of the undergraduate students I used to teach would almost certainly say something like “to get the right answer”. They’d probably also roll their eyes in impatience with such an obvious question. There’s an important assumption that underlies this response that’s worth making explicit: we want a model to get the right answer so that it can replace doing the experiment. In drug discovery, relevant and important data need to be collected experimentally, not modelled (though there is interesting work in toxicology to be able to use some modelling to minimise animal testing).
While we certainly do our best at Optibrium to make our models as accurate as possible, in this blog I’d like to argue that the answer to the purpose of a predictive model is more nuanced. What’s missing in the simplistic answer is the idea of confidence, error, or uncertainty, and how important it can be to leverage this information to help your project succeed. Rather than trying to replace experimental measurements, good models should guide experimental design and help teams prioritise the most important and meaningful measurements to make.
Caveat: There are many ways to build predictive models, and many reasons to do so. Each type of model has its pros and cons. There are lots of good resources about how, when, and why to build QSAR or kNN or DNN or LLM models. The aim of this blog is to demonstrate how to incorporate information about the uncertainty of the prediction to get the most value from your models.
Let’s start by considering an example from our CEO Matt’s review paper on multi-parameter optimisation:
Suppose that the true measured value is 4.5. This is below the threshold for success on the project of Y <= 5 (we want to use our model to identify compounds which will have a measured value of property Y that is less than or equal to 5). Which model was most accurate?
Most of my students would say that model D was the most accurate. This is because its predicted value is closest to the measured value. However, if I’m using the model to identify compounds with a value lower than 5 , the results of the models look like this*:
In other words, Model E says there is more than a 27% probability that the measured value for this compound will satisfy the success criteria. For that reason, Model E gives us the most accurate information since the other model more confidently (and incorrectly) excludes this compound from consideration for success on my project.
In general, most QSAR models provide an uncertainty estimate. This is based on their performance on validation and test sets. Some models provide different uncertainties based on the domain of applicability (that’s a topic for another blog). It’s relatively simple (though not always easy) to include these uncertainties when evaluating the model’s predictions. This explicit incorporation of uncertainty is at the heart of Optibrium’s StarDrop, where values and uncertainties are combined and used to evaluate a compound’s chance of success against specific project objectives.
In our on-demand webinar, you can learn more about Optibrium’s approach to compound prioritisation. ‘Finding balance in drug discovery through multi-parameter optimisation‘.
*to calculate this, draw a (Gaussian) distribution with the prediction as the mean and the uncertainty as the standard deviation. Calculate the cumulative distance function for the threshold value. In this case, it is a value of 5. This gives the confidence with which the model predicts the measured value for this compound will be below 5.
Let’s look at another example. Here, it is less obvious how to incorporate information about the confidence of the model. Optibrium provides models to predict the likelihood of a compound’s metabolism by a variety of enzyme families. Our models also identify the sites on the compound that are most likely to be metabolized.
Shown below are the results for the prediction of metabolism for a PDE10A inhibitor (Chino et al., J. Bioorg. Med. Chem., 2014). In this case, we can see that our WhichEnzyme™ models predict that this compound is about as likely to be metabolized by aldehyde oxidase (AOX) as by P450s. However, when we consider the reactivity of the possible AOX sites, the regioselectivity model concludes that there are no AOX-labile sites on this compound. (NB: for clarity, only the regioselectivity predictions from the AOX models are shown in this diagram; there are predicted sites for other enzymes). It is of special interest to note that one AOX site has a 47% probability of being metabolized by AOX. In this case, we would say that the model does not necessarily rule in or rule out metabolism by AOX.
So now we conclude with the heart of the question we aimed to address in this blog. What is the value of a predictive model that doesn’t make a specific prediction? What value is there when a model has a level of uncertainty that precludes the easy identification of a “right” or “wrong” answer?
There is considerable value in highlighting areas where more experiments are needed. The cost of an additional experiment or two to narrow the uncertainty around a prediction is negligible compared to the cost of a missed opportunity or a late-stage failure.
At Optibrium, we aim to give you the most accurate predictions possible. Since that’s not always possible, we also aim to give you transparent information about the uncertainty in the prediction. We want to help you incorporate those predictions and uncertainties into a clear evaluation of your compound’s chance to succeed on your project objectives.
Want more examples where we leveraged confidence to succeed on project objectives? You can access our case study with Open Source Malaria. In this example, Optibrium was the only team to correctly propose a novel active compound. Rather than choosing the compound predicted to be most active, we chose to submit the compound that was most confidently predicted to be active.
Daniel is a Senior Application Support Scientist at Optibrium, where he applies his expertise in computational chemistry and statistics to support advancements in medicinal chemistry. With a passion for utilising data effectively and accounting for measurement uncertainty, Daniel fosters meaningful insights to drive innovation in drug discovery.
He holds a Ph.D. in Chemistry from Arizona State University and a B.S. in Biochemistry from the Barrett Honors College, graduating with honors and Phi Beta Kappa recognition.
Discover which metabolite prediction software is best for your needs in this comprehensive guide from Optibrium. Compare top tools like Meteor Nexus, MetaSite, and StarDrop to make informed decisions for drug metabolism prediction
Summary The main use of ADMET models, whether in silico or in vitro,tends to be molecule ‘profiling’; identifying compounds which are expected to…
