I tried "deep search" in GROK 3 for medical research. Is it any good?

1. Introduction:

Grok 3 was recently released, and I was particularly interested in testing its "deep search" capability. As a medical resident, I wanted to evaluate its potential as a research assistant, especially since they've made it temporarily accessible to free users.

2. Testing Methodology

A fundamental challenge with AI models is their tendency to hallucinate—generating false or non-existent information. This is particularly concerning in medicine and other fields where accuracy is crucial.

There are typically two ways to identify AI hallucinations, in my opinion:

1. Having deep expertise in the subject matter (which is somewhat counterintuitive since we often use AI to learn about unfamiliar topics)
2. Verifying the provided sources, which is particularly relevant for Grok's "deep search" feature

For my evaluation, I chose to focus on a specific medical topic: the use of corticosteroids in Acute Respiratory Distress Syndrome (ARDS). Specifically, I wanted to examine published Randomized Controlled Trials (RCTs) regarding their efficacy and safety.

3.Results

The model conducted its search across 35 webpages over 2 minutes and 16 seconds before providing the following results:

Yes it crashed!

So I tried again and it took 1min 54s for the results, they are listed below:

I wanted to test it more but unfortunately, I reached my limit as a free user and I have to pay 30$/month to use the "deep search" feature in what they call supergrok. In my opinion it's quite expensive but it's less expensive than the 200$/month for the pro version of openai's chatGPT.

4.Evaluating Grok's Search Accuracy:

Search Performance

The model completed its search in 1 minute and 54 seconds, yielding good but mixed results. Let's break down this performance into its strengths and limitations:

Strengths

The model demonstrated impressive accuracy in identifying core bibliographic elements:

• Successfully captured 13 out of 14 study titles with perfect accuracy
• Nailed all 14 first-author names (100% accuracy)
• Correctly identified publication details (journal and date) for 13 out of 14 studies

Limitations

However, some notable inconsistencies emerged:

• The final study entry showed partial inaccuracies:
  • Title was partially misrepresented
  • Journal of publication was incorrectly cited
  • Though interestingly, both first author name and publication date remained accurate
• PMIDs showed significant discrepancies:
  • Only 4 papers had correct PMIDs
  • The remaining identifiers were inaccurate

This mixed performance highlights both the promising potential and current limitations of AI-powered research tools. While the high accuracy in capturing basic bibliographic information is encouraging, the inconsistencies in unique identifiers like PMIDs suggest there's still room for improvement in data verification and cross-referencing capabilities.

5.Comparing AI Performance with Human Systematic Reviews:

Benchmarking Against Published Research

I've found something truly remarkable in comparing Grok 3's performance with traditional human-conducted systematic reviews. Let me break it down:

Key Reference Points

The comparison was made against two recent meta-analyses:

• Chang et al. (2022)^[1]: A comprehensive systematic review and meta-analysis
• Li et al. (2024)^[2]: A more recent meta-analysis

Remarkable Alignment

What's particularly fascinating is the perfect alignment between Grok 3's results and the Chang et al. study. All 14 studies identified by the human researchers were also captured by the AI model. This isn't just coincidence - it reflects the model's ability to access and process open-access content effectively.

Here you can see the paper in question being included in the research:

and it was explicitly stated in the final result:

The alignment with the Li et al. (2024)^[2] meta-analysis further reinforces our findings - all 9 RCTs from their study were accurately captured in Grok's results. Again, the open-access nature of this paper likely contributed to this performance.

Understanding the Context

This alignment makes perfect sense when we consider that Chang's and Li's papers are openly accessible. It's a brilliant example of how AI can leverage existing high-quality systematic reviews to rapidly synthesize information. The model's ability to identify and extract this information demonstrates both its potential and its current limitations - it's excellent at finding and processing accessible, well-structured academic content.

Note that Grok-3 didn't stop there, it searched for RCTs published after the Chang et al. paper. as illustrated below:

I wanted to highlight the rapid evolution of AI models and their capacity to understand, reason, and search for specific information. Believe me, I tried the same task with other models, and they were unable to give such a detailed response—they would tell me that they could not provide an exhaustive list and that I had to use PubMed for a manual search.

6.Implications for Research

This comparison opens up exciting possibilities for how AI could transform systematic review processes and research in general:

• Rapid initial screening of literature
• Validation of reference lists
• Cross-checking of study inclusion criteria
• Time-saving in the preliminary stages of review work

The technology isn't replacing the need for human expertise but rather amplifying our ability to conduct thorough, comprehensive reviews more efficiently.

7.Looking to the Future: Beyond Traditional Research Methods

Here's where things get really exciting! While some might ask, "Why use AI when we can directly consult systematic reviews?", this perspective misses the revolutionary potential unfolding before us. Let me explain why this test reveals something truly remarkable about the evolution of AI in medical research:

Current Breakthroughs

What sets this performance apart is the model's ability to:

• Understand complex medical queries with remarkable precision
• Execute targeted searches across multiple studies
• Synthesize information from various sources
• Provide detailed, specific responses where other models simply defer to manual searches

Imagining the Future of Medical Research

The possibilities are absolutely thrilling! Consider these potential developments:

• A PubMed integration with advanced AI search capabilities
• Specialized models fine-tuned for medical research analysis
• AI-powered statistical analysis tools for instant meta-analysis
• Real-time research assistants that can process thousands of papers simultaneously

For medical professionals and researchers, this isn't just about saving time - it's about transforming how we interact with medical literature. Imagine having a research assistant that could instantly synthesize decades of clinical trials, while maintaining the crucial element of human oversight and expertise.

8.The Bigger Picture

While we must remain vigilant about AI hallucinations and maintain rigorous verification processes, these results hint at an approaching revolution in medical research methodology. We're witnessing the early stages of tools that could dramatically accelerate the pace of medical discovery while making research more accessible and efficient than ever before.

What excites me most is not just what these tools can do today, but how they're laying the groundwork for even more sophisticated research assistance tomorrow. We're not looking to replace traditional research methods - we're enhancing them in ways that could fundamentally transform how we advance medical knowledge.

If you want me to create more content about this topic like comparing this model with other models, more detailed analysis, and other use cases in our field, let me know in the comments below and don't forget to subscribe so you won't miss any upcoming content.

References:

1. Safety and efficacy of corticosteroids in ARDS patients: a systematic review and meta-analysis of RCT data.
2. Efficacy of corticosteroids in patients with acute respiratory distress syndrome: a meta-analysis.

Keywords:

Grok 3, Deep search capability, Medical research, AI in healthcare, AI research assistant, Corticosteroids in ARDS, Acute Respiratory Distress Syndrome, ARDS randomized controlled trials, RCT evaluation, Systematic review automation, AI hallucinations, Bibliographic search automation, Evidence-based medicine, Clinical trial analysis, PubMed.