An Amazon chatbot that’s supposed to surface useful information from customer reviews of specific products will also recommend a variety of racist books, lie about working conditions at Amazon, and write a cover letter for a job application with entirely made up work experience when asked, 404 Media has found.
I don’t see these stories as about what the chat ai outputs, but more about questioning whether or not amazon should be held liable for what their AI outputs. Traditional customer support chatbots are often less than useless, but they wouldn’t go about suggesting the product they’re selling are defective or recommending offensive products. I’m of the opinion that Amazon’s review search AI thing should be held up to the same standard that a human would be. And if a person started acting like this they would surely be quickly fired.
They are a black box, and for now trying to restrain the black box has sever impact on the usefulness of the output even in easier and legit situations.