Experiment: Does AI Write Better Twitter Posts Than Humans?

Everyone is talking and tweeting about AI. You’ve probably heard it all at this point: AI can help you write better. AI can help you tweet faster. AI is coming for your job (let’s just put that theory to rest — AI is a tool, not a replacement).

But can AI help your social captions perform better?

As a busy social media marketer, anything that promises to make the content creation process easier and faster is appealing. But we wanted to find out if using AI actually impacts engagement and reach, specifically on Twitter.

Do human-written captions have better engagement and reach than AI-written ones? Does the Twitter algorithm penalize tweets written by AI? I ran an experiment to find out.

Hypothesis: Human-written Twitter captions will get more engagement and reach than AI-written captions

Our educated guess is that human-written Twitter captions will get more engagement and reach than ones written by AI.

Twitter is a conversational platform. People love tweets with a clear voice, and Twitter users love engaging with other humans. We suspect people won’t be drawn to a tweet that doesn’t sound like it came from a human.

But would people really know if a tweet was written by AI or not? And if so, how would that impact reach and engagement?

Let’s find out.

Methodology

To test whether human-written Twitter captions generate more engagement and reach than AI-written ones, I shared three sets of tweets on my personal Twitter account. Each set of tweets used the same format so they could be fairly compared.

For this experiment, I posted two tweets that were text only, two tweets that included a link, and two tweets with a visual.

To compose the AI-generated tweets, I used Lately, a tool that repurposes long-form content into bite-sized social captions.

(Bonus: You can connect the Lately app with Hootsuite to finalize, schedule, and auto-publish your posts right inside your Hootsuite dashboard)

Source

To get started, you submit a piece of content like a blog post or a podcast into the generator.

Source

It’ll then analyze the content and generate captions for you to review and edit. After you edit, you can send them directly to Hootsuite to be scheduled.

Some things to remember about Lately and any other AI-powered content creation tools: First, you have to train it. For AI tools to generate content that sounds the most like you or your brand, they have to understand your voice, content, and audience. This is why Lately asks you to submit a long-form piece before it can generate captions.

Another thing to note about AI-generated content is that it still needs to be edited by a human.

After Lately comes up with tweets based on the content you submitted, you can edit them for voice, clarity, and context. Lately then uses these insights to create better content the next time. The more you use it, the more it’ll start to sound like something that your brand would actually post.

Once I had my human-written captions created, I used Hootsuite’s Lately integration to generate similar tweets using the same long-form pieces of content. I then drafted them in Hootsuite and used the recommended posting time feature to get them scheduled.

Results

I checked the performance of each tweet about 24 hours after I posted (the Twitter lifecycle moves fast). Here are the results:

I compiled the engagement rate for each tweet as an overview of how they performed. I’ll dig into specific metrics for each one below.

Tweet #1: Text-only tweet

Since I’m using Lately’s AI tool for this experiment, I had to focus on paraphrasing long-form pieces of content for these captions.

Here is the first tweet I wrote for this experiment. For this tweet, I paraphrased a quote I contributed to an article about freelance work routines.

And here are the results of this tweet, according to Hootsuite Analytics. It generated an engagement rate of 14.14%, 28 total engagements, and 198 impressions.

As far as the style goes, this is a normal tweet for me. I like to add voice through asterisks, and I’m a sucker for using parentheses in my writing. Parenthetical statements (statements that qualify or clarify something and usually have a set of parentheses around them) are a great way to communicate extra information and provide more context (see what I did there?). Plus, sentences with parentheses just add that extra human element if you ask me.

All that to say, when I had the AI generate a similar tweet from the same article, I didn’t edit it to add my usual human touches because I wanted to see how it’d perform on its own.

Here are the results of the AI-generated caption. It generated an engagement rate of 4.06%, 5 total engagements, and 123 impressions.

For this first tweet comparison, the human-written caption performed better. Is that because of the lack of parentheses? Maybe, but probably not.

There are a few factors that could’ve influenced this tweet’s performance, so let’s try again with another type of tweet.

Tweet #2: Tweet with link

For my next comparison, I wanted to see how a tweet with a link would perform. I paraphrased an article I wrote about types of career paths for people who work in content marketing (specifically freelance vs. full-time roles).

Here is the tweet I wrote:

With all the job uncertainty many are facing lately, this article I wrote for @superpathco feels like a timely reminder.

Freelancing doesn't have to be an all-or-nothing move! Many folks I spoke to for this piece do it on the side or in between jobs. https://t.co/EI8rePLBuS

— Sam Lauron (@Sam_Lauron) February 8, 2023

And here are the results of this tweet. It generated an engagement rate of 8.47%, 32 total engagements, and 378 impressions.

This tweet included a link to the article and a mention of the brand I wrote it for. This tweet had better reach than my last, and I suspect this is because the brand I mentioned quote retweeted it. I checked Twitter’s native analytics for additional metrics, and this tweet got 21 detail expands and five link clicks. Interesting!

For the AI-written caption, I had Lately paraphrase the same article and generate a tweet for it. It also included the link and a mention (which I had to add), but I didn’t change anything about the text that it generated.

"It's nice to know freelancing is there in case my career ever takes me down that path again!" It's not uncommon for FT content marketers to take on freelance work on the side or transition between FT and freelance throughout their careers. @superpathco

https://t.co/dEeb63G7IB

— Sam Lauron (@Sam_Lauron) February 16, 2023

Here are the final results from that tweet. It generated a 7.5% engagement rate, 6 total engagements, and 80 impressions. I also checked out Twitter’s analytics for this tweet and it received one detail expand and two link clicks.

My human-written tweet outperformed the AI-generated one for this round. While the engagement rate was about the same, the tweet I wrote received over four times as many impressions, likely because it got retweeted.

Would the AI-generated caption have performed better if it had also been retweeted? Maybe. Would it have received more link clicks if it had more reach? Perhaps. But you could also argue that it wasn’t retweeted or clicked on as much because the caption itself wasn’t as engaging.

Let’s do one more test.

Tweet #3: Tweet with visual link

For my final tweet comparison, I wanted to include something visual to see how that made an impact on engagement and reach. I used two articles I’ve written for Hootsuite for comparison. (Again, since I’m using Lately for this experiment, the tweets had to be based on a long-form piece of content).

Each of them has a similar thumbnail image and would appear similarly in the feed. Plus, both articles cover a similar topic — social accessibility — so the content could be fairly compared.

Here’s the first tweet I shared with a caption that I wrote.

If you’ve worked in social media in the last few years, you know that digital accessibility and inclusivity are top priorities.

In my latest for @hootsuite I share why Facebook alt text is one way to make your content more accessible and discoverable. https://t.co/huQ5AksuCg

— Sam Lauron (@Sam_Lauron) February 9, 2023

And here’s how it performed:

This tweet generated a 5.71% engagement rate, 8 total engagements, and 140 impressions. It got a couple of likes and one comment. According to Twitter’s native analytics, this tweet also got one link click and five detail expands.

I had Lately generate a tweet for another article I wrote on a similar topic about TikTok auto captions. I should note that the AI paraphrased the article for the first part of the tweet, but I had to add the second line for some context and to include the mention as my first tweet did.

Want more people to consume your TikTok videos? Make the viewing experience as accessible and enjoyable as possible by adding auto captions.

Read more about auto captions in my first article for @hootsuite: https://t.co/JhSjH0TQ0g

— Sam Lauron (@Sam_Lauron) February 13, 2023

Here are the results from this tweet:

Overall, this tweet didn’t perform great. It hardly had any reach or engagements — no likes, comments, retweets, or link clicks. But it did get a couple of detail expands.

Once again, the human-written tweet outperformed the AI-generated text.

What do the results mean?

Ultimately, all of my human-written tweets performed better than the AI-written captions. I admittedly don’t have a large following on my personal Twitter account, so none of these metrics are too exciting. But I do think they demonstrate (on a very small scale) that more human-sounding tweets resonate better on the platform.

Here are some of my takeaways from this experiment:

AI-written captions can be a time saver

I’ll be honest: I may be a writer, but it takes me an embarrassing amount of time to write a tweet — especially one that paraphrases a long article. I was amazed at how quickly the AI tool was able to come up with captions, even if they weren’t publish-ready.

Whether writing comes naturally to you or not, coming up with creative captions every week takes time and energy away from your other tasks, like strategizing or engaging with followers. So when it’s your job to write dozens of social captions on any given day or week, using an AI tool can save you a lot of time.

A tool like Lately is especially helpful if you use Twitter to share long-form content like blog posts. This tool can create dozens of captions from one piece, which means you can have a month’s work of Twitter captions created in a matter of minutes.

AI-written captions still need a human touch

Since Lately generated dozens of captions for each article I gave it, there were a ton of captions that didn’t make the cut. Some of them were decent, but others were just plain confusing and would’ve needed a lot of editing.

If you use a tool to generate captions, you’ll still need to review and edit them, at least in the beginning — especially to add voice.

AI tools usually don’t understand the full context of a tweet and don’t know your brand voice right off the bat. Most AI tools need to be trained to better understand what you want them to do and to get your brand’s tone right. It can take time for these tools to catch on, but the more you use them and tweak what they write, the more they will adapt.

As far as whether or not AI-written captions directly impact reach? The jury’s still out. In my experiment, all of my human-written captions had better reach than the AI-written ones, but that could’ve been because the content itself resonated more. It’s hard to say if the platform knew the AI-written tweets were, in fact, written by AI. But if I had to guess, I’d say that Twitter doesn’t penalize you for using AI to write your captions.

Ultimately, if you’re going to use AI to write Twitter captions, think of them more as a starting point. You’ll have to use your social expertise to make them engaging and publish-ready, but you can knock them out much faster than if you were starting from scratch.

Save time by using Hootsuite to manage your Twitter presence alongside your other social channels. You can run contests, share videos, schedule posts, and monitor your efforts — all from one convenient dashboard! Try it for free today.

Get Started

The post Experiment: Does AI Write Better Twitter Posts Than Humans? appeared first on Social Media Marketing & Management Dashboard.