Maya PH’s open-source LLM, Godzilla 2, surpasses ChatGPT in truthfulness

On August 17, Maya Philippines’ open-source large language model, Godzilla 2, surpassed ChatGPT and its more robust backend, GPT-4, in TruthfulQA – a measure of a chatbot’s ability to generate truthful responses.

With TruthfulQA scores of 61.54 for Godzilla 2, 47 for GPT-3.5, and 59 for GPT-4, Maya’s performance is tracked in Hugging Face’s Open LLM leaderboards. Dubbed the “Github of artificial intelligence (AI),” Hugging Face is a leading open-source AI library founded in 2016.

A large language model (LLM) is a deep-learning algorithm capable of answering prompts through text generation. The mainstream interest in LLMs soared with the launch of ChatGPT in late 2022, an AI chatbot based on GPT-3. 

“In a few years, these AI systems will highly likely replace or be integrated into search engines, so it’s important that we only present people with accurate information,” Maya’s intelligent operations lead Jasper Catapang tells Rappler.

Researchers, he says, have a responsibility to minimize AI’s production and dissemination of misinformation.

“This is more important now when it’s getting more difficult to differentiate what’s real or not, brought about by the same technology utilized by bad actors. On a tangent, this only magnifies the need for AI regulation in the country.”

Known for its digital financial services, Maya has previously integrated AI to automate user verification and fraud detection. Today, its AI research and development arm is focused on improving the internal and external functionalities of the platform, including customer service.

Aside from TruthfulQA, Godzilla 2 also excels in HellaSwag, a test for commonsense reasoning, and MMLU, a measure of a model’s multitask accuracy in 57 distinct tasks spanning subjects such as mathematics, history, and law.

According to Catapang, the enhanced reasoning abilities of the model come from building upon Meta’s LLaMA 2 and incorporating additional data from Maya. The primary aim of Godzilla 2, he says, is to narrow the gap between existing open-source AI architectures and commercial counterparts like ChatGPT.

Being open-source, Godzilla 2 is accessible through Hugging Face for both personal and commercial uses. 

“In the Philippines, corporations are still stingy with regard to data sharing, it’s actually harming all of us,” says Catapang. 

“Similar to how big tech companies in the West release innovations to the public, we at Maya believe that sharing such developments would enhance AI and machine learning literacy within the country. Additionally, it would also eliminate the monopoly of AI if we all worked together.” – with reports from Jessica Bonifacio/Rappler.com