Don’t use AI to summarize documents — it’s worse than humans in every way : techtakes

[–] RagnarokOnline@programming.dev 9 points 3 months ago (12 children)

I had GPT 3.5 break down 6x 45-minute verbatim interviews into bulleted summaries and it did great. I even asked it to anonymize people’s names and it did that too. I did re-read the summaries to make sure no duplicate info or hallucinations existed and it only needed a couple of corrections.

Beats manually summarizing that info myself.

Maybe their prompt sucks?

[–] sxan@midwest.social 8 points 3 months ago

How did you make sure no hallucinations existed without reading the source material; and if you read the source material, what did using an LLM save you?

[+] TexasDrunk@lemmy.world -8 points 3 months ago (1 children)

I also use it for that pretty often. I always double check and usually it's pretty good. Once in a great while it turns the summary into a complete shitshow but I always catch it on a reread, ask a second time, and it fixes things up. My biggest problem is that I'm dragged into too many useless meetings every week and this saves a ton of time over rereading entire transcripts and doing a poor job of summarizing because I have real work to get back to.

I also use it as a rubber duck. It works pretty well if you tell it what it's doing and tell it to ask questions.

[–] YourNetworkIsHaunted@awful.systems 9 points 3 months ago (2 children)

Isn't the whole point of rubber duck debugging that the method works when talking to a literal rubber duck?

[–] self@awful.systems 8 points 3 months ago

what if your rubber duck released just an entire fuckton of CO2 into the environment constantly, even when you weren’t talking to it? surely that means it’s better

load more comments (10 replies)

[–] beefbot@lemmy.blahaj.zone 4 points 3 months ago (2 children)

Is it only me, or is the linked article not super long on details & is reaching a conclusion from 2 examples? This is important & I need to hear more, & I’m generally biased against AI at this point— but the article isn’t doing enough to convince me

load more comments (2 replies)

[–] lvxferre@mander.xyz -2 points 3 months ago* (last edited 3 months ago) (17 children)

You could use them to know what the text is about, and if it's worth your reading time. In this situation, it's fine if the AI makes shit up, as you aren't reading its output for the information itself anyway; and the distinction between summary and shortened version becomes moot.

However, here's the catch. If the text is long enough to warrant the question "should I spend my time reading this?", it should contain an introduction for that very purpose. In other words if the text is well-written you don't need this sort of "Gemini/ChatGPT, tell me what this text is about" on first place.

EDIT: I'm not addressing documents in this. My bad, I know. [In my defence I'm reading shit in a screen the size of an ant.]

[–] pikesley@mastodon.me.uk 6 points 3 months ago (1 children)

@lvxferre @dgerard have you bumped your head?

[–] lvxferre@mander.xyz 2 points 3 months ago (1 children)

No, it's just rambling. My bad.

I focused too much on using AI to summarise and ended not talking about it summarising documents, even if the text is about the later.

And... well, the later is such a dumb idea that I don't feel like telling people "the text is right, don't do that", it's obvious.

[–] dgerard@awful.systems 9 points 3 months ago

You'd think so, but guess what precise use case LLMs are being pushed hard for.

[–] V0ldek@awful.systems 6 points 3 months ago

if the text is well-written you don’t need this sort of “Gemini/ChatGPT, tell me what this text is about” on first place.

And if it's badly written then the LLM will shit itself.

Now let's ask ourselves how much of the text in the world is "well-written"?

Or even better, you could apply this to Copilot. How much code in the world is good code? The answer is fucking none, mate.

load more comments (15 replies)

[+] z00s@lemmy.world -6 points 3 months ago* (last edited 3 months ago) (19 children)

The problem is not the LLMs, but what people are trying to do with them.

They are currently spoons, but people are desperately wishing they were katanas.

They work really well for soup, but they can't cut steak. But they're being hyped as super ninja steak knives, and people are getting pissed when they can't cut steak.

If you give them watery, soupy tasks they can do successfully, they can lighten your workload, as long as you're aware of what they are and aren't good at.

What people want LLMs to be able to do, ie. "Steak" tasks:

write complex documents
apply complex knowledge/rules to a situation
Write complex code and create entire programs based on vague description

What LLMs can currently do ie. "Soup" tasks:

check this document and fix all spelling, punctuation and grammatical errors
summarise this paragraph as dot points
write a python program that sorts my photographs into folders based on the year they were taken

Half of Lemmy is hyping katanas, the other half is yelling "Why won't my spoon cut this steak?!! AI is so dumb!!!"

Update: wow, the pure vitriol pouring out of the replies is just stunning. Seems there are a lot of you out there who have, in one way or another, tied your ego very strongly to either the success or failure of AI.

Take a step back, friends, and go outside for a while.

[–] froztbyte@awful.systems 10 points 3 months ago

good god this entire post is the most tortured believer whataboutism I've encountered this month and there's extremely strong competition here

are currently spoons, but people are desperately wishing they were katanas

ie. “Steak” tasks

you should make a youtube channel, The Katana Steak-Eater. I'd watch the shit out of that at least one saturday afternoon

[–] FredFig@awful.systems 9 points 3 months ago* (last edited 3 months ago) (2 children)

Food analogy

This level of discourse wouldn't fly on 4chan, how is it so popular with LLM fans?

[–] dgerard@awful.systems 9 points 3 months ago (1 children)

needs to be a car analogy

What people want LLMs to do, i.e. Corvette tasks
What LLMs actually do, i.e. Trabant tasks

[–] self@awful.systems 6 points 3 months ago

What LLMs actually do, i.e. Trabant tasks

more of a Power Wheels Barbie Jeep whose battery got left out in the sun too long, but I’ll allow it

[–] froztbyte@awful.systems 5 points 3 months ago

don't diss the course, this steak's great

[–] swlabr@awful.systems 7 points 3 months ago

Actually, LLMs are syringes filled with brain-parasite-infested poop

load more comments (16 replies)

[+] Eheran@lemmy.world -8 points 3 months ago (2 children)

Hahaha what a load of nonsense.

load more comments (2 replies)

[+] lightnsfw@reddthat.com -9 points 3 months ago (12 children)

Ok? I don't have another human available to skim a shitload of documents for me to find answers I need and I don't have time to do ot myself. AI is my best option.

load more comments (12 replies)

TechTakes