this post was submitted on 15 May 2024
656 points (100.0% liked)

TechTakes

1550 readers
119 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] Amoeba_Girl@awful.systems 171 points 8 months ago (149 children)

From Re-evaluating GPT-4’s bar exam performance (linked in the article):

First, although GPT-4’s UBE score nears the 90th percentile when examining approximate conversions from February administrations of the Illinois Bar Exam, these estimates are heavily skewed towards repeat test-takers who failed the July administration and score significantly lower than the general test-taking population.

Ohhh, that is sneaky!

[–] Amoeba_Girl@awful.systems 125 points 8 months ago (146 children)

What I find delightful about this is that I already wasn't impressed! Because, as the paper goes on to say

Moreover, although the UBE is a closed-book exam for humans, GPT-4’s huge training corpus largely distilled in its parameters means that it can effectively take the UBE “open-book”

And here I was thinking it not getting a perfect score on multiple-choice questions was already damning. But apparently it doesn't even get a particularly good score!

[–] booly@sh.itjust.works -4 points 8 months ago (3 children)

I don't think you understand the type of multiple choice questions involved. Here's a real question:

A father lived with his son, who was an alcoholic. When drunk, the son often became violent and physically abused his father. As a result, the father always lived in fear. One night, the father heard his son on the front stoop making loud obscene remarks. The father was certain that his son was drunk and was terrified that he would be physically beaten again. In his fear, he bolted the front door and took out a revolver. When the son discovered that the door was bolted, he kicked it down. As the son burst through the front door, his father shot him four times in the chest, killing him. In fact, the son was not under the influence of alcohol or any drug and did not intend to harm his father.

At trial, the father presented the above facts and asked the judge to instruct the jury on self-defense.

How should the judge instruct the jury with respect to self-defense?

(A) Give the self-defense instruction, because it expresses the defense’s theory of the case.

(B) Give the self-defense instruction, because the evidence is sufficient to raise the defense.

(C) Deny the self-defense instruction, because the father was not in imminent danger from his son.

(D) Deny the self-defense instruction, because the father used excessive force.

Memorizing the book itself doesn't teach how to answer this type of question. It requires actual application of concepts to the new facts being given.

[–] self@awful.systems 11 points 8 months ago

thanks for posting this long, pointless shit twice in this thread for no discernible reason

load more comments (2 replies)
load more comments (144 replies)
load more comments (146 replies)