NYC’s business chatbot under fire for providing ‘dangerously inaccurate’ information

The New York City government's AI-powered chatbot, designed to assist business owners with accessing accurate information, has come under scrutiny for providing "dangerously inaccurate" responses, an investigation by The Markup has revealed.

Launched as part of the MyCity portal in October, the chatbot aims to offer reliable guidance directly from city government sources but has been found to misinform users on various city policies.

Among the examples cited in the report are instances where the chatbot incorrectly confirmed the legality of cashless stores, despite New York City's ban on such establishments starting in 2020. It also provided incorrect information regarding workers' rights, landlords' obligations, and business practices, leading to concerns about the potential harm caused by its inaccuracies.

While acknowledging the chatbot as a pilot program, the city emphasizes its commitment to transparency and reliability in AI applications. Leslie Brown, a spokesperson for the NYC Office of Technology and Innovation, noted that users are cautioned about the beta nature of the tool and encouraged to verify responses independently.

Despite these assurances, critics argue that the chatbot's shortcomings undermine its utility and highlight the need for ongoing improvements to ensure accurate and helpful support for small businesses citywide.