A government test reveals that AI performs

remarkably worse than human workers.

 

 

 

An absolute turd.

Sums It Up Australian news outlet Crikey discovered that the results of an experiment conducted for the Australian Securities and Investment Commission (ASIC) show that generative AI is dreadfully bad at summarizing information when compared to humans.

The government authority commissioned Amazon Web Services to perform the trial as a proof of concept for the possible applications of generative AI, particularly in corporate settings.

The trial concluded that there is not much hope for such possibility.

In a series of blind evaluations, the trial’s rubric yielded an appalling 47 percent overall score for the generative AI summaries of actual government papers, greatly underperforming the human-generated summaries, which received 81 percent.

The results confirm a prevalent trend in discussions around the current wave of generative AI technology: AI models are not only a poor substitute for human labor, but their terrible dependability makes it doubtful that most firms would find any real use for them in the workplace.

Typical Shoddiness

Although it’s not the newest model available, the evaluation made use of Meta’s open source Llama2-70B, which has up to 70 billion parameters.

The AI model was given instructions to compile a summary of documents that were turned in for a parliamentary investigation.

Page numbers and references were to be included, and special emphasis was to be paid to anything that mentioned or had anything to do with ASIC. Employees of ASIC were asked to produce their own summaries in addition to the AIs.

After reading the source materials, five evaluators were asked to compare and contrast the summaries produced by AI and humans. These were finished carelessly; the summaries merely received the letters A and B, and the scorers had no idea that AI had ever been used.

Stated differently, that was not their goal.

After the trial completed and they discovered the true purpose of the exercise, three assessors expressed doubt about seeing results from artificial intelligence. Even by itself, this is a somewhat terrible statement.

Horrible in Every Way

Overall, the AI summary outperformed the human summary in every category, according to the analysis.

First of all, the page numbers referring to the information’s sources could not be produced by the AI model.

According to the paper, this can be fixed by slightly changing the AI model.

However, a more basic problem was that it frequently made confusing decisions about what to highlight or emphasize and frequently failed to recognize context or subtleties.

Beyond that, the AI summaries were typically “waffly” and “wordy,” with a tendency to incorporate repetitive and irrelevant material.
In conclusion, the assessors concluded that the amount of fact-checking these AI summaries required made them so awful that employing them would eventually need additional labor. In that scenario, the ostensible benefits of employing technology to reduce costs and save time are put into serious doubt.