I worked on this project at my former employer for nearly 8 months. We worked as a good faith partner with McDonald’s, only to find out eight months in that McDonald’s had never had any intention of working with us and we’re only using us as a negotiating tactic for the acquisition they eventually made for this technology.
I learned two things from this project. Our team built a complete English language model for every possible permutation of ordering every menu item. No LLMs, no gpt required.
The hilarious part is that McDonald’s stopped responding to our team and abandoned the project only about a month before we were going to tell them that it was not technically feasible to do it scale. The problem is not bad AI nor a lack of data. The problem is all of the real world interface challenges, keeping speakers and microphones working outdoors across many different climates and weather conditions, and temperatures and humidity is incredibly expensive. And on top of that, the hardware needed per store at the edge and the permanent network infrastructure required to keep everything running makes the whole system substantially more expensive than just having a single human being running drive-through even have much higher wages than minimum wage.
So McDonald’s never got this conclusion from us and instead spent several billion more dollars and another six years of R&D to come to the same conclusion.
My other take away from this project was that I will never give McDonald’s another dollar of my money. Working with over 200 fortune 500 companies in my career McDonald’s is far away the most evil heartless and ruthless company I’ve ever dealt with.
They don’t care about their customers, their users, their franchisees or their employees. The only thing they care about at all is their stock price.
Doesn't surprise me in the least. McDonald's is such a bad company on many levels, even from its founding roots.
They make fast food (an industry notorious for craping all over their employees) and it was usurped by a greedy business man (Ray Kroc) from its original founders.
Their food is okay, but I would never, ever do work for them.
> usurped by a greedy business man (Ray Kroc) from its original founders
It's a weird aside, but maybe worth noting that Ray Croc was probably not just greedy. Ray Kroc was probably also racist against the Irish. A tell-tale sign is using a clown as the company mascot. The US strain of clowns were heavily influenced by "pale white face" racism jokes about Irish immigrants [0] from some of the same minstrel shows notorious for "black face" and "yellow face" and "red face". Ray Kroc was from a generation that would have easily been aware of that and would have been "entertained" by it. Ray Kroc's behavior to the actual McDonald's founders is rather easier to explain assuming it included quite a bit of Old Fashioned Racism than assuming just pure greed. Sometimes it is useful to remind ourselves that past isn't as clean as corporate memos want to paper over it.
[0] Notably, among other things: red hair, big feet, freckles, red drunken noses, loutish drunken behavior. Even the "clown car joke" is the exact same "joke" as "Mexican pickup truck" transposed across a couple of decades and about a different working class immigrant population. (Racists seem pretty lazy in how they reuse old material.) So yeah, if you ever wondered why clowns don't seem all that funny in the modern era, congratulations you probably aren't a racist. Also, now that this past horror is in your head I'm sorry for ruining Disney's Dumbo which uses all the worst of clown stereotypes and "jokes" all in the same place and eats up a lot of runtime with it, if you weren't already concerned about the "black face" crows in the movie or thought you could dismiss them as not central characters or contributing that much to the runtime.
Why is this grey? Does that mean people are reporting it? If you report it or want to, then why?
I haven’t verified these claims, nor do I have the background knowledge to form an opinion. I don’t care. Seems plausible and I think Irish were within the definition of “black” a century ago.
Is it because you don’t believe in this? Or do you think it is astroturfing with bad intentions similar to the anti women memes plastering social media (where it is some wrong committed by a woman with 100s of bot comments saying all women are bad)? Kinda like repeating long ago marginalizations of white people to stir up a feeling of wronghood among older more conservatives white men?
I don't even consider the food okay... I liked a few breakfast options, but the prices of even that have gone up so much the past 3 years, I won't even go then. It wasn't that long ago that the breakfast burrito was $1 (then 2 for $2/3) and they had some sandwiches 2 for $2/3/3.33/4/5 ... When I'm paying over $10 for a couple breakfast sandwiches and a drink, I'm out.
They definitely seem to be squeezing more out of their franchises than ever at this point. They will push technology optimizations, but they're at a point where service and price just isn't there. If I'm spending $15+ for lunch, I may as well go to Applebees/Chili's, etc.
Thanks for sharing, everything makes sense except this:
> The problem is all of the real world interface challenges, keeping speakers and microphones working outdoors across many different climates and weather conditions, and temperatures and humidity is incredibly expensive.
I thought this was a solved problem already, by McDonald’s in fact?
I am not the poster, but my guess here is that a point to point or radio setup which just needs to be good enough for humans to understand it (and even then just barely) is likely way worse quality than what would be needed for recording and feeding to a program
Treat each drive through ordering system as an edge compute location, run the software right there and then transmit only the order instead of streaming audio from the ordering system.
Edge compute requires field IT for servicing. You can hire 3-4 employees for the price of a single field service specialist.
Servicing industrial edge compute is insanely expensive for B&M businesses. This is a big part of why so many companies have spent the last decade offloading everything humanly possible to public cloud workloads.
With background noise? My wife and I had an interesting unintentional experiment regarding this a few weeks ago at a Slowdive concert. She doesn't know the band well and asked me the name of each song as it began and then also asked Google's song identifier service. I knew all of them and Google couldn't even produce an answer, incorrect or not. Either because of the crowd noise or a live version is sufficiently different from any recorded example it has heard before.
> The problem is all of the real world interface challenges, keeping speakers and microphones working outdoors across many different climates and weather conditions, and temperatures and humidity is incredibly expensive.
This sounds like a problem for humans too - are humans just better equipped to deal with bad audio?
Yes, but not in the ways you think. Humans can do about as good a job as a computer in understanding poor audio quality in context, but the compute needed for the latter, in realtime, is pretty substantial.
As the commenter says below, a human can intervene in many more ways when equipment malfunctions or customers have special needs that an AI just gets blocked by.
There are literally hundreds of edge cases where a voice powered drive through just stops working, from high winds to pouring rain, to thick accents, broken equipment, out of stock or seasonal items not available. Just a few of the ones I encountered personally in the wild tagging 15,000+ orders.
Humans have had to evolve to be able to communicate with each other effectively in this and far worse conditions - on battlefields, in driving hurricanes, while being stalked by other humans (and animals), yelling across vast distances, over raging rivers, while wounded, sick, etc.
Yeah this sounds strange to me - every drive thru has had an intercom system like this working outside for decades. Doesn't seem like an insurmountable challenge to me.
You're not entirely wrong, but often these AI systems need some pretty clear audio to work. It's kinda shocking how good we are at working around bad audio when it comes to conversation, and I'm certain most people know how bad these intercom systems get. The issue isn't that they need to be fixed at all, it's how far they can go before they must be fixed. And the one thing we can do that AI can't is have face-to-face conversations. If the speaker simply doesn't work, it's a bit of a drag, but you can just pull up to the window directly and skip the entire audio system. Or just walk inside. Both options eliminate the problem hardware, where as AI would need additional hardware to do those jobs.
> The problem is not bad AI nor a lack of data. The problem is all of the real world interface challenges, keeping speakers and microphones working outdoors across many different climates and weather conditions, and temperatures and humidity is incredibly expensive.
That's a great point. Shazam, which has a "simpler" challenge, still trips up with identifying music in challenging environmental conditions.
Having worked for a variety of corporate structures, I'd agree that franchise models tend to have... a bit more ruthless approach to everything. (Even unintentionally)
When "the business" is literally outside your company, you get the same dynamics as when it's in another org structure, except even worse.
I can answer that. Because I'm sick of installing a hundred apps for every single company, setting up accounts, verification, so I can spend 10$, and never use it again.
And, when you are traveling, and just taking the highway exit for a quick order, what, now I have to stop, park, download an app then order.
Or do you mean, like mount Ipads in kiosk mode? I think that gets back to the expense and weather.
I think it gets better over time. It is like with cars having no knobs or no physical buttons anymore. We have to figure it out. We have big screens in cars now, why is the menu not even integrated, when you get close to your McDrive? Why not allowing each shop a virtual space in your car, when you drive nearby?
And by the way, we all wanted employees to have a better pay. It is just pure survival mode now by mcd. Since paying humans is expensive now, one has to find ways to make them expendable even more, so that the company remains profitable for the franchise people and keeping the product price low.
(And after all, was McD not a real estate company?)
I hate the self-serve kiosks inside the places... I'm relatively tall, and trying to use them is often an exercise in frustration. Especially when there's no mechanism to adjust the height or angle.
This is why McDonalds literally spends tens of millions of dollars per quarter giving away free food to attempt to get more people to just use the app.
Their plan longterm is exactly this - have a robot box with a window that only accepts orders through your phone and removes all humans from the process.
You might be surprised how many people absolutely hate using apps, and do not want to interact with a business through a mobile phone. Older folks especially, but also there is a large portion of the population that only use technology when necessary.
We build a heuristic model from scratch. We used BART for some NLP bits and Azure speech to text for the basic mic -> raw text. My memory is very muddy on this bit as I didn’t work on the algo portion of the project, I was working on the UX workflows and interaction / conversational workflow design and validation.
Fun fact, McDonalds menu is a graph database that tracks every single ingredient as purchasable entity. You can order damn near any combination of elements and they will sell it to you - want 32 pickles and the bottom piece of a bun with a chocolate chip cookie on top? No problem.
And every promotional item and name ever put on sale is persisted in the menu forever.
> You might be surprised how many people absolutely hate using apps, and do not want to interact with a business through a mobile phone. Older folks especially
It is not just older folks. It is also anyone mildly or more security/privacy aware. App's offer far more opportunities for spying/tracking and as well "advertising" (push messages), none of which I am interested in giving the business.
In my case, if an app is required, I move on to the next business where an app is not required.
> You might be surprised how many people absolutely hate using apps, and do not want to interact with a business through a mobile phone. Older folks especially
Yep, but not because I'm older but because:
1. I do not want McDonald's (or anyone else) to link my orders together. It's none of their business.
2. App fatigue. I don't even eat at McDs, but I don't want an app for every grocery chain i buy from either.
3. Any app, even if not "AI" based, is going to fail in more annoying ways than a human.
McDonald's has screwed up my order less than any other restaurant, provided better, warmer food than any other restaurant, is cheaper for the quality than any other restaurant, and their Ronald McDonald House charity rescued my family during our darkest hour.
>> There are over 380 Ronald McDonald Houses in 64 countries. These accommodate families with hospitalized children under 21 years of age (or 18 or 26, depending on the House), who are being treated at nearby hospitals and medical facilities. [...] Ronald McDonald Houses allow families to stay free of charge.
That hasn't been my experience at all. Not sure where you are, but in the Phoenix area, I have gone several multi-year periods not spending anything at McDonald's. They had messed up my order more times than getting it right.
Worst was going ketovore... how the hell hard is it to put two pieces of meat in a box without a bun and nothing on it? OR getting 3x "2 strips" of bacon and counting to 6.
I learned two things from this project. Our team built a complete English language model for every possible permutation of ordering every menu item. No LLMs, no gpt required.
The hilarious part is that McDonald’s stopped responding to our team and abandoned the project only about a month before we were going to tell them that it was not technically feasible to do it scale. The problem is not bad AI nor a lack of data. The problem is all of the real world interface challenges, keeping speakers and microphones working outdoors across many different climates and weather conditions, and temperatures and humidity is incredibly expensive. And on top of that, the hardware needed per store at the edge and the permanent network infrastructure required to keep everything running makes the whole system substantially more expensive than just having a single human being running drive-through even have much higher wages than minimum wage.
So McDonald’s never got this conclusion from us and instead spent several billion more dollars and another six years of R&D to come to the same conclusion.
My other take away from this project was that I will never give McDonald’s another dollar of my money. Working with over 200 fortune 500 companies in my career McDonald’s is far away the most evil heartless and ruthless company I’ve ever dealt with.
They don’t care about their customers, their users, their franchisees or their employees. The only thing they care about at all is their stock price.