the_dunk_tank

Never heard of Devin, but ChatGPT (the current leader) can barely follow instructions well, but is usually faster to get the information I need than Googling questions/topics for a wide variety of tasks. It's pretty much only good at things that had solid stackoverflow answers, including combining 2 questions or answering it for any language. Anytime you attempt to add complexity or details, previous details just fall off at random eventually. Sometimes it just can't combine 2 key details. Sometimes you get a lot of dead-ends. This mostly makes up for Google not being as good as it was 10 years ago, but my productivity boost is probably less than 10%. With Google, you probably click into 2-10 websites which almost all them missing a key keyword (you know a keyword that was key...). Some of the Google quick information boxes copy information from a website and phrase it in the form of a response to your question, but sometimes that specific data point had nothing to do with your question. An example would be a random date in the webpage was picked by Google's LLM to answer my question, but the date I was looking for was not even in the webpage. ChatGPT usually does well if you ask it 101 stuff or questions about well documented non-obscure facts or starting-from-scratch stuff or "write a function that does X". Anything else, you can almost safely assume a miss on first attempt, and a dead-end (no further conversing reaches viable solution) very likely.