this post was submitted on 15 May 2024

656 points (100.0% liked)

TechTakes

1491 readers

32 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago

MODERATORS

dgerard@awful.systems

656

Remember how ChatGPT totally aced the bar exam? Wow! yeah, turns out that was just a lie (www.nytimes.com)

submitted 7 months ago by dgerard@awful.systems to c/techtakes@awful.systems

252 comments fedilink hide all child comments

archive https://archive.ph/is57b

top 50 comments

sorted by: hot top controversial new old

[–] Amoeba_Girl@awful.systems 171 points 7 months ago (3 children)

From Re-evaluating GPT-4’s bar exam performance (linked in the article):

First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population.

Ohhh, that is sneaky!

[–] Amoeba_Girl@awful.systems 125 points 7 months ago (63 children)

What I find delightful about this is that I already wasn't impressed! Because, as the paper goes on to say

Moreover, although the UBE is a closed-book exam for humans, GPT-4’s huge training corpus largely distilled in its parameters means that it can effectively take the UBE “open-book”

And here I was thinking it not getting a perfect score on multiple-choice questions was already damning. But apparently it doesn't even get a particularly good score!

[–] ebu@awful.systems 36 points 7 months ago (26 children)

[...W]hen examining only those who passed the exam (i.e. licensed or license-pending attorneys), GPT-4’s performance is estimated to drop to 48th percentile overall, and 15th percentile on essays.

officially Not The Worst™, so clearly AI is going to take over law and governments any day now

also. what the hell is going on in that other reply thread. just a parade of people incorrecting each other going "LLM's don't work like [bad analogy], they work like [even worse analogy]". did we hit too many buzzwords?

[–] Amoeba_Girl@awful.systems 18 points 7 months ago* (last edited 7 months ago)

"Nooo you don't get it, LLMs are supposed to be shit"

[–] genuineparts@infosec.pub 18 points 7 months ago (1 children)

But LLM’s don’t work like Typewriters, they work like Microwaves!

load more comments (1 replies)

load more comments (24 replies)

[–] Deceptichum@sh.itjust.works 33 points 7 months ago (3 children)

That’s like saying a person reading a book before a quiz is doing it open book because they have the memory of reading that book.

[–] Sterile_Technique@lemmy.world 64 points 7 months ago (2 children)

It's more like taking a digital copy into the test room with you and Ctrl+F'ing every question/answer.

[–] Deceptichum@sh.itjust.works 26 points 7 months ago (7 children)

Except it’s not, because they can’t perfectly recall everything.

It’s more like reading every book in the world, and someone asking you what comes next after “And I…”.

[–] realbadat@programming.dev 36 points 7 months ago

"will alwaaays love you...."

Easy. No other answer.

[–] DessertStorms@kbin.social 21 points 7 months ago* (last edited 7 months ago) (12 children)

But the AI isn't "recalling" in the same way you do, it doesn't "remember" what it "read", it "reads" on demand and has instant access to essentially all of the information ~~available online~~ it was trained on (E: though it's becoming more or less the same thing, and is definitely the same when it comes to law books for example), from which it collects the necessary details if and when it needs it.

So yes, it is literally "sat" there with all the books open in front of it, and the ability to pinpoint a bit of information in any one of all the books in milliseconds.

load more comments (12 replies)

load more comments (5 replies)

[–] bdonvr@thelemmy.club 20 points 7 months ago* (last edited 7 months ago) (7 children)

I'm not a big AI guy but it's really not quite like that, models do NOT contain all the data they were trained on.

Edit: I have no idea what's going on down below this comment

[–] self@awful.systems 27 points 7 months ago (11 children)

I’m not a big AI guy

we can tell

load more comments (11 replies)

load more comments (6 replies)

[–] V0ldek@awful.systems 21 points 7 months ago* (last edited 7 months ago) (14 children)

I'm not even going to engage in this thread cause it's a tar pit, but I do think I have the appropriate analogy.

When taking certain exams in my CS programme you were allowed to have notes but with two restrictions:

Have to be handwritten;
have to fit on a single A4 page.

The idea was that you needed to actually put a lot of work into making it, since the entire material was obviously the size of a fucking book and not an A4 page, and you couldn't just print/copy it from somewhere. So you really needed to distill the information and make a thought map or an index for yourself.

Compare that to an ML model that is allowed to train on data however long it wants, as long as the result is a fixed-dimension matrix with parameters that helps it answer questions with high reliability.

It's not the same as an open book, but it's definitely not closed book either. And the LLMs have billions of parameters in the matrix, literal gigabytes of data on their notes. The entire text of War and Peace is ~3MB for comparison. An LLM is a library of trained notes.

load more comments (14 replies)

load more comments (1 replies)

load more comments (61 replies)

load more comments (2 replies)

[–] MajorHavoc@programming.dev 88 points 7 months ago (1 children)

AI being pushed by scam artists...Gee. Who could have guessed?

[–] FiniteBanjo@lemmy.today 17 points 7 months ago

I did. I guessed. I expressed skepticism when that headline first appeared.

[–] vin@lemmynsfw.com 42 points 7 months ago (1 children)

Though making an unreliable intern is amazing and was impossible 5 years ago...

[–] self@awful.systems 27 points 7 months ago (43 children)

thank fuck sama invented the concept of doing a shit job

load more comments (43 replies)

[–] skillissuer@discuss.tchncs.de 36 points 7 months ago (1 children)

the perils of hitting /all

[–] dgerard@awful.systems 23 points 7 months ago (2 children)

416 updoots, what on earth

load more comments (2 replies)

[–] Tb0n3@sh.itjust.works 34 points 7 months ago (12 children)

AI = Actually Indians

load more comments (12 replies)

[–] walter_wiggles@lemmy.nz 30 points 7 months ago (1 children)

I asked AI to summarize the article since it's paywalled. It didn't say anything about lying, should I trust it?

[–] dgerard@awful.systems 38 points 7 months ago

As a large language model, absolutely

load more comments