Clinical Trial Search Engine
Many patients have medical issues that don't get better with the usual treatments. They suffer because their condition doesn't respond to the existing methods. This is where clinical trials come in - they test new ways of treating diseases that could change how we take care of patients. But, it's really hard for someone who isn't a doctor to find clinical trials that might help them.
Right now, doctors check for applicable clinical trials because the systems are so complex. But, if your doctor isn't checking for new trials every day, you might miss out on treatments that could save your life.
If you tried to search for these trials on your own, it would take over **400 days**!
Our client, ClinicalNet, is innovating the clinical trial landscape by helping patients find the best clinical trials for their needs. They want to create a search tool that gives people accurate and quick results.
However, their old search tool used outdated text matching methods. These worked sometimes, but often missed trials because they didn't consider different words that mean the same thing. We needed to look at over 65,000 active clinical trials, each with more than 20 rules for who can join, and create a search engine that was accurate and fast. This like a job for a Natural Language Processing AI!
Challenges
Understanding How Medical Conditions are Related
Medical conditions are related to each other like a family tree. There are broad conditions that include more specific ones, and so on. We call these relationships "parent" and "child."
It's easy to see that "lung cancer" is a child of "cancer." But, for more specific conditions, it gets tricky. For example, "oat cell carcinoma" and "small cell lung cancer" what’s the parent, and what’s the child? (Trick question - they're siblings!)
After a lot of thought, we came up with a method that solved this problem nicely and gave us a complete map of how conditions are related. This method is fully automated, and doesn’t require any human effort to construct this mapping! Plus, we figured out how to solve this problem ahead of time for many conditions, so we didn't have to do as much math for every search.
Making it Fast
This system needed to give fast results while looking at a huge number of clinical trials and medical terms. We had to rank over 65,000 clinical trials for each search, while also checking against millions of medical terms to make sure our results were accurate. And we needed to do all this in just a few seconds, even for searches the system had never seen before!
To make this possible, we:
1. Used statistics to pre-calculate key condition families and match those to common searches.
2. Made different levels of search, so we could give the most obvious results right away while doing a deeper search in the background.
3. Finetuned our AI model to work best for our specific needs.
End Result
ClinicalNet now has a search engine that matches patients with highly relevant clinical trials across over a million unique conditions in 5 seconds or less.
Our new search system gives much more relevant results than the old one, without taking much more time in most cases.
Plus, our method for building the condition relationships saved over 1000 hours of manual work, and it automatically includes any new conditions.
By creating custom AI models and using smart engineering solutions, we were able to solve this tough problem in a clever way!