> A fact of life: Nearly always people with money, power, and optimization problems don't understand optimization, fear and resent those who do, and choose just to avoid the subject.
Food for thought: the solution to real-world optimization problems is often dictated by constraints instead of optimal values.
This means that if you fail to understand the constraints, or even fail to identify them, then whatever your solution to the problem is, it will be wrong. And it will be obvious to those who are aware of the constraints.
Now, you're complaining that those presenting you with problems "don't understand optimization".
From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping.
Have you ever considered the possibility that you failed to understand the actual problems presented to you and even failed to clarify why your output was aligned with anyone's best interests?
Naw, you list some mistakes, but I didn't make any of those.
> From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping.
First, for any application, there has to be some practical interest. My view, there isn't much. The schools of math, engineering, and business have given optimization a big push, back at least to Dantzig, but from my long experience the interest was and is still just too low for "applied" optimization to have much in applications.
Cases: Sure, there has been a professor at Princeton who applied optimization to oil refining: What mixture of crude oil to mix into the refinery and what mixture of refined products to take out. Maybe a few, large livestock operations actually do run some diet problem solutions. And can use 0-1 optimization for Sudoku problems? What path for picking orders in a big warehouse, for Amazon or Walmart? A simple traveling salesman problem, and for a good enough solution build a minimum spanning tree and walk around that -- maybe they are doing that already. Assembly line balancing: Assign workers to positions to maximize the speed of the slowest worker assignment. Is anyone actually doing that? Even if they are, the solution is quite simple. Yes, a start on P vs NP was at Bell Labs designing networks. So, maybe with the Internet there are still valuable applications? Considered that. Got an interview at a company trying that. They were impressed by what I'd done at FedEx, but they were nearly dead and, I suspect, soon died. Maybe with big logistics, ocean, rail, trucks, warehouses, there are some big logistics problems where optimization could save a lot -- applications enough for careers? Better than grass mowing? When I got my Ph.D., the Chair of my dissertation orals committee was a big name in logistics -- saw no evidence
of significant interest in applications. No ones in the halls. Phone not ringing. No suggestions of contacts for me.
Look, when there is a big need, ESPECIALLY when there is big money involved, it soon gets obvious, and the US economy gets to it right away. In that, "applied" optimization is not hot, warm, or much above freezing.
Right, you are mentioning formulation:
(1) They had already formulated a 0-1 optimization problem. It had 40,000 constraints and 600,000 variables. They had tried the then popular simulated annealing, ran for days, and quit. So, the formulation was done and not mine.
I worked hard, with the IBM OSL (optimization subroutine library), did 900 primal-dual iterations, Lagrangian relaxation, got a feasible solution within 0.025% of optimality, within two weeks, for free, a free sample, and never heard from them again. They resented and were afraid of my success.
(2) Another company was working a little more generally in optimization. Had a crude heuristic running. On some of their problems, 0-1, linear, again was successful with the OSL, and got only insults and resentment. Continued on, gave them a nice formulation, better than their heuristic, and path through optimization, and got fired. They'd hired me and wanted to fire me before 6 months was up. They were not very good with linear programming at all, and I was a LOT better at what they were doing in the formulation, math, and computing, and their reaction was they didn't want me for competition.
(3) In a military group, did well with some non-linear optimization (their formulation). Then they had a challenging strategic problem. I did a formulation of a Monte-Carlo solution and wrote and ran the code (used an Oak Ridge random number generator I'd programmed in assembler). They called in a famous probability professor for a review. His remark was that there was no way the Monte-Carlo could "fathom" the tree. He was right; the tree was huge. But each trial of my Monte-Carlo yielded at each point in time a random variable on 0-15, and the law of large numbers applied right away. It wasn't D-day, but suppose it was: The tree of possibilities was enormous, but the, say, number of Allied soldiers killed was, what, 0-200,000. So, each trial give a random variable value at, say, each second, for, say, 48 hours -- the law of large numbers applies and could tell Ike the distribution of number of deaths, the expected values, the median, the variance for each second of the 48 hours. Passed the review. One guy there used my random number generator on one of his old problems, got significantly different results, was afraid, said "I don't want you in the center of all my projects", and I got ignored on the way to being fired.
(4) At FedEx, had written a program that showed the BOD that the program made the fleet scheduling easy enough and saved the company. So, to do better, formulated a set covering direction. Savings? 1% would have been $millions a year. The founder, COB, CEO wrote a memo making that my project, but my boss, a Senior VP, said that there was no money in the budget for me; I'd been commuting between Memphis and Maryland where my wife was in her Ph.D. program; the stock promised in three weeks was very late; and I went for a Ph.D.
Actually another student at another school ran with my set covering formulation for his dissertation.
The high level, overview, simple fact of life, is as I described: There just is no real career in "applied" optimization. That horse is nearly dead and should not be further flogged. Millions of US families have a house, stable marriage, and healthy children, and I'd believe that fewer than 20 of those families are supported by careers in "applied" optimization -- maybe 0 families.
I don't think I got my point across. The constraints I've referred to aren't a reference to how problems are formulated, but what leads decision-makers to make decisions. The goal of any number-cruncher in a corporate environment, whether they are data scientists, machine learning engineers, operations research specialist, etc., is to advise decision-makers on what are their options. If they stop collaborating and start to lift barriers and create problems, instead of adding value they turn themselves into a bigger problem. And I'm not even touching on the problem of gains.
Adding to that, decision-making is all about tradeoffs. All problems have sensitivity to input parameters. This means that there are always choices that can be made to have different solutions if decision-makers are willing to accept the tradeoffs. They always do, because not all constraints and requirements are expressed or expressable in a problem statement. More to the point, the output of an optimization problem is not reaching the optimal point, but to improve on the current performance.
Not everything in life can be limited or summarized in crisp values. Moreso in the business world. Do you understand what I'm saying?
> Savings? 1% would have been $millions a year.
That's your projection. And 1% of anything is completely irrevelant, I might add. No wonder the project was killed.
I worked in projects that we could advise cost improvements of around 4% and the project was slashed as well. What's the year on year variance though? 1% is a fraction of inflation. How many meetings would they need to meet a ceiling of 1%? Is 1% the value-added of a PhD? Do you get what I'm saying?
At one point, we need a correction: The optimization was to schedule the fleet for FedEx. They were spending big time money on the airplane operations, and a better schedule, saving even 1% of that, would amount to maybe $millions a year, at any rate, money worth saving and where, in comparison, my work cost peanuts. At FedEx at the time, the 1%, the automation of the fleet scheduling, the business planning potential, made the project worthwhile.
The "decision maker" was F. Smith, founder, COB, CEO. My office was next to his. On paper I reported to a Senior VP, but in reality reported directly to Smith. Smith wrote a memo giving the optimization problem to me -- I still have a copy. The project was approved, and by passing all the executive considerations you mentioned. The project was not "killed". Instead, due to (a) commuting between Memphis and Maryland where my wife was in her Ph.D. program, (b) wanting to stay in Maryland, where I'd done the best work for FedEx and saved it the first time (access to consultants and time sharing computing), (c) the Senior VP I was reporting to telling me there "was no money in the budget for me", and (d) the promised stock very late, I left for the Ph.D.
I got Smith to approve the project by explaining "Integer linear programming set covering" to him for this application: (i) Take all or a reasonably large subset of all the reasonable tours from Memphis and back. All the planes were the same, French Falcons, .... (ii) For each tour, program standard means of evaluating flight times and costs and find the cost of the tour. Throw out some tours for however goofy reasons, e.g., can't fly over this city at that time of day. Note, some of the costs and constraints were really goofy, way beyond what could be converted to software even for non-linear integer programming. Such is the magic of set covering enumeration.
(iii) Intuitively regard each tour as a piece in a jigsaw puzzle where have lots of extra pieces, each piece has a cost, and want to cover. "set covering", the board exactly at minimum total cost. That's when Smith understood the work well enough to approve it as my project. Considering what airplanes cost, my project was not very expensive.
(iv) Now have a 0-1 integer linear program with one row for each city and one column for each tour. In a column, a row i = 1 to 90 is 1 if the tour serves city i and a 0 otherwise. The cost of that column is just the cost of the tour. The right side is all 1s. The constraints are all >=. With 90 US cities, there were only 90 rows in the linear program. So, use some LP (linear programming) software to get solutions feasible and optimal or nearly so. At some point if necessary just use old branch and bound. Worth a try. My computing was VM/CMS time sharing on relatively large IBM mainframes.
"1% is a fraction of inflation." Maybe you are saying that CEOs should have optimization involved only for super big aspects of the company, like Ike's work on D-day, and ignoring a 1% reduction in the cost of M1 rifle ammunition. Well, not then at FedEx. Fleet operating costs were by far the biggest expense of the company and .... Even for D-day, such a 1% reduction might have been worthwhile but to be handled by some Colonel and not Ike!!
Your other points seem to be correct. One response is when working for US national security, some of the problems were
"strategic" for the US and where ... would be as you describe "advising" "options" at no higher than some mid-level uniform in the Pentagon and not for the POTUS. But for my work in the commercial economy, I never was trying to advize the CEO of some $800 billion company on some crucial decision. And I didn't work on enough problems to encounter many of the considerations you listed.
Again, my main point here was just the "applied" optimization in the book title in the OP. My experience, academic, military, commercial, student, professor, research and applications, indicates that outside of the US military there is nearly nothing real about "applied" optimization -- optimization applications are nearly like hen's teeth.
Maybe Amazon, Walmart, Google, Microsoft, and a few of the largest companies have an office of planning and analysis on the organization chart reporting to some VP for something or other and there occasionally develop/run some optimization models, but I never saw any evidence of such. Sure, such an "office" would have to do the TLC of the CEO you mentioned. I'd guess that such an "office" would get their optimization done with a lot of contact with some professors, but the only professors I ever saw doing any such work were the few I contacted. I just didn't see any credible evidence of "applied" optimization in the US commercial economy.
Now, in the last 10 years or so, everything about such optimization -- data collection, data manipulation, word processing for the math, the basic desktop computing, statistical tools, and, sure, Gurobi, etc. are just MUCH better. Soooo, maybe now the US commercial world is ready to exploit optimization like Dantzig, Kuhn, Tucker, Arrow, Hurwicz, Nemhauser, etc. intended. Maybe.
If I got an offer, now, for such work I'd turn it down and stay with my startup. (a) You're talking too much in office politics; that's not one of my specialties; and I have none of that in my startup. (b) I know the math for my startup, all nicely written up with TeX, but I'm rusty on a lot of the rest of math. E.g., for Lagrangian relaxation I'd look at my notes for the last time I did that. For some of the classic integer programming on graphs, I'd go for my grad school notes. For anything in probability, I'd want to review the Radon-Nikodym theorem and conditional expectation, all the standard theorems on convergence of random variables, etc. (c) From the time I got the offer, I'd have to start spending my money, and I might get fired before I even got that back -- such a job would be a gamble where I could lose money significant for me.
Again, my post here was just to object to "applied" for optimization. No way am I looking for a job; not now; not any more; not again. Instead I'm staying with my startup.
Food for thought: the solution to real-world optimization problems is often dictated by constraints instead of optimal values.
This means that if you fail to understand the constraints, or even fail to identify them, then whatever your solution to the problem is, it will be wrong. And it will be obvious to those who are aware of the constraints.
Now, you're complaining that those presenting you with problems "don't understand optimization". From your anecdotes you were the one tasked with clarifying things to them. From the sound of it, you didn't accomplished that, and it was unclear to stakeholders whether your output even provided any value worth keeping.
Have you ever considered the possibility that you failed to understand the actual problems presented to you and even failed to clarify why your output was aligned with anyone's best interests?