Claude 2 Vs ChatGPT-4

Exciting news! We've just released a brand-new video that's generating quite a buzz in the AI community. Are you ready to witness a head-to-head battle between two cutting-edge AI chat models?

Check out the latest video or stay here and read on.

In order to compare the two, we ran a series of tests.

Test #1: Content generation

The Prompt

“I am writing a 1,500 word article structure for writing about how to do SEO competitor analysis for your digital marketing agency’s website.”

Claude 2

We were impressed with the results. Because Claude 2 has a big token context window (75,000 words), we were able to give it our full book on digital marketing for reference. The result? A strong structure and an impressive, well thought out piece on SEO competitor analysis that matched our tone of voice.

ChatGPT-4

We couldn’t upload our book due to the limited token context window, so we simply copied and pasted the section we needed from the book. While we were happy with the structure, when we asked for the article, things got very STRANGE. It decided to produce this:

We have no words 🤯. That’s one point for Claude 2.

Test #2: Analyse data from GA4

The Prompt

“This is some data from GA4 that shows the top locations of conversions on my website. I’d like to know how to get more conversions, and spot any areas where my website is underperforming. What can you identify?”

We also uploaded the CSV file of data from GA4.

Claude 2

We received a number of very broad insights, none of which were particularly groundbreaking or that insightful.

ChatGPT-4

We were able to upload the CSV file to their ‘Code Interpreter’ and wow, what a difference. Firstly, it identified there was an issue reading the CSV file but managed to fix the issue without any further input from ourselves.

Then, it laid out the structure of the data to make sure it was reading the data correctly, which it was.

Finally, it produced four really insightful areas we could explore further.

The insights are spot on and we think anyone who was new to GA4 and used ChatGPT in this way, these types of insights would be helpful to make some key marketing decisions.

It’s glaringly obvious that ChatGPT’s Code Interpreter is one of it’s biggest strengths. One point to ChatGPT-4.

Test #3: Comprehension and maths

The Prompt

Using the same CSV file, we asked “What is the average conversion rate on mobile across all locations and landing pages?“

All the tool needed to do here is find the total number of conversions and total number of visitors and divide to come up with an answer. The answer is 7%.

Claude 2

Despite using a much smaller file because it wasn’t able to process the whole data (another advantage of Code Interpreter), Claude was able to give us a near correct answer of 6.8% and even showed us it’s calculations.

However, the answer is still wrong.

ChatGPT-4

Not a great start. The answer was wrong, suggesting it is “infinity” and it didn’t show us any of it’s calculations. However, it acknowledged that this could be due to anomalies so it took those anomalies out and redid the test without any further input form ourselves.

Now, it gives us the answer of 16%. Much higher than the actual answer, 7%, and still no calculations. We were pretty shocked that Code Interpreter got this one wrong.

So neither tool got this one right. No one gets a point.

Test #4: Create a marketing strategy

The Prompt

“I’m the marketing manager for a regional commercial law firm and have been tasked with increasing the inbound leads through the website.

The website needs redesigning to focus on conversion rate.

Your objective is to help me create a plain-english digital marketing strategy that maximises return on investment, Use a mixture If digital marketing channels that are likely to attract commercial law customers, and suggest a prioritised action plan for the next 12 months, allowing 4 months for the website build and suggesting marketing activity that can go in parallel to the build.

I would like you to prioritise SEO. Content marketing, search ads, email marketing (to our existing list) and LinkedIn.

I want these channels to work together, so please suggest some creative content ideas that could work well across them. We have a moderate budget of £250k per year.”

Claude 2

To be honest, the answer we got from Claude wasn’t that great. there was nothing technically wrong with the plan but it was far too broad and not enough details for it to be genuinely useful.

In order to get this plan to a point where we could use it, it would require extensive iterations going through each bullet one-by-one.

ChatGPT-4

We were much more impressed with ChatGPT on this test. It gave a detailed plan and in our experience, there was not much it missed in it’s quest to create a decent marketing strategy. It even acknowledged that the Search Ads couldn’t start until the website was built.

Some of the content ideas suggested were not very creative but we were happy to get at least some inspiration. We were also impressed with the suggestions for how to allocate our budget. Nice.

Another point to ChatGPT.

The Winner

Despite the buzz around Claude 2, the clear winner in this fight is ChatGPT-4.

However, Claude 2 is still an impressive tool and has some big advantages over ChatGPT that we really like. It’s has a much larger token context window, it’s superior in content creation, it’s information is more up to date (2023 vs 2021) and it’s FREE.

Conclusion

Neither one of these tools is universally the best. Each has it’s own unique set of strengths and ultimately, the one you use will depend on the task at hand. If you’re after top quality content generation, we suggest Claude 2 is the way to go. If need data processing, ChatGPT-4 is the best bet (Code Interpreter is a beast!).

We hope you enjoyed this thorough comparison of each tool. Feel free to share your own experiences of using each with us over on Twitter (excuse us, we meant 𝕏).

If you have any feedback, just hit reply and let us know what you think of the newsletter! If you like the newsletter, be sure to forward it on so others can sign up too.