New Mimecast Threat Intelligence: How ChatGPT Upended Email 

Across most cybersecurity media, references to generative AI are not only increasing exponentially, but publications are stating its use for malicious activities which has a potential impact upon every organization. When interviewing Mimecast threat researchers for our recent Threat Intelligence report, questions were asked about the pervasiveness of AI in phishing emails - but no metrics could be quantified. This left unanswered questions, including how prevalent is this and can it be measured? Our data science team took on the challenge to help, by building a detection engine to determine if a message is human- or AI-generated based on a mixture of current and historical emails, and synthetic AI-generated emails.  

The research indicates a point in time when we start observing an increasing trend in AI-generated emails correlating with the release of ChatGPT. We also observed malicious AI-generated BEC, fraud and phishing emails. The net effect of this is a need for understanding by analysts/security teams and end users of the indicators of AI generated content which could help them spot these attacks.   

Telltale signs of AI-generated emails 

ChatGPT made AI-assisted email writing accessible to everyone, even malicious actors but this is not the only set of tools available to them. In a previous blog post we outline some of their generative AI tools Previously, such tools were mainly for businesses. Now, anyone can use AI to write well-crafted emails suited to various situations. As AI-generated content becomes more prevalent, the ability to discern between human-written and machine-generated text has become increasingly difficult. One of the most notable characteristics of AI language models is the use of complex words and sentence structures, which can reveal their involvement in writing. Researchers found AI language models favor certain words in scientific writing. “Analyzing 14 million papers from 2010-2024, they noticed a sharp increase in specific ‘style words’ after late 2022, when AI tools became widely available. For example, ‘delves’ appeared 25 times more often in 2024 than before. Other AI-favored words include ‘showcasing,’ ‘underscores,’ and ‘crucial.’” 

How we know ChatGPT changed email 

Mimecast’s data science team started with the intention to train a model the differences between human- and AI written emails. In total over 20,000 emails were utilized from Mimecast’s data coupled with LLM generated - OpenAI’s GPT4o, Anthropic’s Claude 3.5 Sonnet, Cohere’s Command R+, AI21’s Jamba Instruct and Meta’s Llama3 - synthetic data. The deep learning model created determined what characteristics make each data point related to the language utilized to either be human or AI written. For testing, to ensure that our model did not overfit to our training set, but could generalize well, we used four datasets: 

• 4,000 emails from Mimecast 
• 2,600 LLM generated synthetic data  
• Human and LLM dataset from Kaggle (link) 
• Fraud dataset from Kaggle (link). All emails are assumed to be human written, as they were collected before the rise of LLMs 

Once training was complete, our model was shown one email after the other and asked to determine whether that example is written by a human or AI. We repeated this exercise hundreds of times on different sets of emails. We were able to use it to analyze a subset of emails to predict whether it was written by a human or AI. The results from this exercise can be found in figure 1 which also highlights the increase of AI-written emails. It is important to note that the model was not looking to identify malicious AI-written emails, but rather to estimate the pervasiveness of AI. Prior to undertaking this study it was known that AI-written messages were being seen but we did not know the scale.  

Figure 1 – Human vs AI-written emails 

We sampled 1000 emails per month from January 2022 to June 2024. These statistics show that out of 30, 000 emails analyzed it was found that 2330 were AI-written representing 7.8% of all emails in the dataset. But importantly the line chart is showing not only a marked increase in the use of AI to write emails but the reduction in human writing which continues to fit with what is being seen in publications. Whether this is attributed to non-English language speakers or the use of AI to aid in writing to try and make them better is unknown at present. 

The accuracy details for each data set can be found at the bottom of this notification. 

Examples of AI-Generated Emails 

During the process of reviewing the submissions a few malicious examples were found containing distinctive language. 

Example #1 of Gen AI spam message 

Indicators:  

• "delves into the intricacies of", "navigating through the complexities of" 
• Overuse of bullets 

Example #2 of Gen AI BEC message 

Indicators:  

• ‘I hope this message finds you well'  
• Repetition of the words ‘gift cards’ and ‘surprise’

Example #3 of Gen AI BEC message 

Indicators:  

• ‘Hello!’ 

Example #4 of Gen AI phishing message

Indicators:  

• ‘delve deeper into this’ 
• ‘stumbled’ or ‘stumbled upon’ 
• Long ‘-’ utilized across ChatGPT 

Recommendations 

These findings indicate that manual phishing investigations should remain a crucial layer of defense, especially when flagged by end users. It's vital that threat researchers scrutinize the language for specific markers that align with our findings; by cross-referencing indicators such as “delve deeper into this” or “hello!”, particularly among end users who commonly don’t use such language with known threat patterns, you can identify phishing threats more effectively, reducing remediation time and mitigating organizational risk.  

As always, security teams should ensure their indicators evolve alongside large language models and new data sets.  

Technical details 

Our model is based on the transformer architecture, which has been key to the improvements we have seen in natural language processing and generative AI over the last few years. (https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)) It enables the model to learn a holistic understanding of the entire email, by taking the entire context of the email into account.  

DistilBERT base model was used and fine-tuned for LLM generated text detection. 

Anything more we could say about distilbert? Since it is in the screenshot below 

The accuracy when testing the data sets is outlined below, and please use the key to understand each value 

Acc (Accuracy) - proportion of correct predictions over all predictions 

Prec (precision) - how many were actually AI-written? Proportion of predicted AI generated emails that were actually AI written 

Recall - How often did the model correctly identify AI-written emails 

F1 - Harmonic mean of precision and recall 

Raw data for % of human and % of LLM written emails over time 

month 

['2022-01', '2022-02', '2022-03', '2022-04', '2022-05', '2022-06', '2022-07', '2022-08', '2022-09', '2022-10', '2022-11', '2022-12', '2023-01', '2023-02', '2023-03', '2023-04', '2023-05', '2023-06', '2023-07', '2023-08', '2023-09', '2023-10', '2023-11', '2023-12', '2024-01', '2024-02', '2024-03', '2024-04', '2024-05', '2024-06'] 

% Human 

 [0.949, 0.928, 0.946, 0.947, 0.946, 0.913, 0.945, 0.935, 0.948, 0.925, 0.916, 0.946, 0.933, 0.932, 0.931, 0.925, 0.922, 0.924, 0.905, 0.925, 0.919, 0.918, 0.906, 0.917, 0.906, 0.89, 0.896, 0.886, 0.909, 0.882]  

% LLM 

[0.051, 0.072, 0.054, 0.053, 0.054, 0.087, 0.055, 0.065, 0.052, 0.075, 0.084, 0.054, 0.067, 0.068, 0.069, 0.075, 0.078, 0.076, 0.095, 0.075, 0.081, 0.082, 0.094, 0.083, 0.094, 0.11, 0.104, 0.114, 0.091, 0.118]