This makes me sad. I hoped to see less use of language models in the wild; looks like we're heading the other direction.
I was recently able to articulate to myself why I dislike language models like GPT-3 so much.
Language is fundamentally about communication. I have a mental state, and language lets me share that some of that mental state with another human being. It's quite remarkable, and the cornerstone for pretty much every aspect of our lives as social creatures.
GPT-3 can generate text that's convincingly human-like; it's an amazing technical achievement. But (so far, anyway!) it is just that; a generator of plausible text. To borrow a philosophical term, there's no "interiority" there.
You can see this clearly in GPT-3's output; it is at its weakest when writing about the relationships between a set of concepts, or when trying to carry a line of thought or argument through a longer text. This is because there is no underlying _thing_ that is being communicated; no mental model or grounding concept. Just text, generated solely based on plausibility in a given context.
So while GPT-3 is truly remarkable in its ability to generate text, it cannot, by definition, communicate (except on the meta level via prompt selection and output curation.)
Does the world really need more words but with less communication?
I thought this was interesting so I literally pasted the entire thing into OpenAI's playground and asked DaVinci to write a rebuttal of your argument. Here is what it said:
---
There are a few problems with LukeV's argument. First, he conflates the ability to generate human-like text with the ability to communicate. Just because GPT-3 can generate text that sounds like it was written by a human does not mean that it cannot communicate. In fact, GPT-3 is quite good at carrying a line of thought or argument through a longer text, as evidenced by the fact that it is often used by writers to generate ideas and outlines for their articles.
Second, LukeV claims that GPT-3 is at its weakest when writing about the relationships between a set of concepts. However, this is not necessarily true. GPT-3 may not be able to generate text that is perfectly grammatical or that makes perfect sense, but it is often able to generate text that is nonetheless meaningful and insightful. For example, GPT-3 has been used to generate essays on complex topics such as the philosophy of mind, and these essays have been well-received by readers.
Third, LukeV argues that the world does not need more words, but this is clearly not the case. The world needs more communication, and GPT-3 can help to facilitate this by generating text that is meaningful and insightful. In fact, GPT-3 is already being used by writers to generate ideas and outlines for their articles, and it is likely that this use will only increase in the future.
In conclusion, LukeV's argument against GPT-3 is flawed. GPT-3 is a valuable tool that can help to facilitate communication, and the world needs more communication, not less.
----
I'm not sure who I agree with, but that's GPT-1's response to you regarding your critique of itself. Have at it!
> For example, GPT-3 has been used to generate essays on complex topics such as the philosophy of mind, and these essays have been well-received by readers.
That's the other problem with GPT-3... it will happily say things that are absolutely false :)
I think this response actually illustrates my point quite nicely; it is a plausible sounding rebuttal but does not actually engage on a logical level.
That said, I don't mean to diminish the impressiveness here. It's quite evident that it could pass a Turing test in a wide variety of circumstances. After all, it's not like humans are consistently great at concept-informed writing either. I'm pretty sure the majority of undergrad college papers have more in common with GPT than with real critical thought... generating something that looks plausibly like an argument rather than actually having one.
Lol, that last point about undergraduate papers is exactly what I was thinking reading your initial statement. As a current student, it feels that sometimes we're forced to just generate something that looks plausibly like an argument as opposed to actually generating one.
> First, he conflates the ability to generate human-like text with the ability to communicate.
This is incorrect. In fact, lukev does the opposite. That is one of the fundamental parts of his argument -- that those two things are different.
> Second, LukeV claims that GPT-3 is at its weakest when writing about the relationships between a set of concepts. However, this is not necessarily true. GPT-3 may not be able to generate text that is perfectly grammatical or that makes perfect sense, but it is often able to generate text that is nonetheless meaningful and insightful. For example, GPT-3 has been used to generate essays on complex topics such as the philosophy of mind, and these essays have been well-received by readers.
I believe this is the best paragraph of the argument, but it is also the most difficult to rebut because it is rather subjective. Whether a text is meaningful or insightful is up the individual reading it. I would be surprised, however, if experts on the philosophy of mind found the essays mentioned by GPT-3 to be meaningful or insightful -- as opposed to regurgitations of concepts that are in the texts that have been fed to GPT-3's model.
> Third, LukeV argues that the world does not need more words, but this is clearly not the case. The world needs more communication...
The presence of more words in the world does not necessarily entail more communication. The two concepts are fairly closely related but are definitely not synonymous.
This doesn't necessarily mean that I agree with lukev, of course. I do think that GPT-3 as it currently exists should not be used to generate text that is meant for human consumption without being edited first. I can certainly see how it would be useful as a tool for generating an outline of an essay, especially with well-crafted prompts.
I am of course open to more arguments and data on the subject =)
I noticed that when GPT-3 has trouble continuing a sentence or create a difficult argument it relies on filler words to stretch out a sentence until it has no choice but to write something meaningful.
Wow, pretty much every single word of that rebuttal is complete nonsense. This is like 7th-grade level writing, filled up almost entirely of non-sequiturs, filler words, and regurgitating the prompt back at you. I may not have agreed with lukev before reading this response, but I certainly do now.
> So while GPT-3 is truly remarkable in its ability to generate text, it cannot, by definition, communicate
GPT-3 does nothing without a human inputting what it wants it to output. GPT-3 is as much a tool for communication as are sounds, words, symbols, etc.
Whether it was carved in stone by a human hand or generated on the cloud by an AI, the intent to communicate is always coming from a human.
If you think the text generated by GPT-3 does not carry the right meaning or has "no underlying _thing_ that is being communicated", it just means that the human behind is not using the tool properly. Just like sometimes you use a word that doesn't mean what you think it means.
So, I actually agree with you there. Crafting a prompt, curating the output, and then distributing it certainly itself an act of communication.
I don't particularly like it as an act of communication because the information density is going to be quite low. If you want to tell me something, I'd rather you just told me the prompt! It will have all the relevant "real" communication; anything added by GPT-3 is basically just filler.
I’m not sure I follow. If we’re talking about turning a few sentences into a blog post then sure, but there are many use cases for these models.
E.g., distilling large amounts of text into short summaries while retaining substance is effectively the opposite of this, and is a use case GPT3 has been applied to successfully/commercially.
This is actually a big deal since this includes the davinci model everyone uses and was the biggest obstacle to cost-affordable deployment of GPT-3.
The exemption for finetuned models is interesting: since they were 2x the price of the old prices, they're now 4x-6x more expensive than the base models. I bet that's how OpenAI really makes their money with enterprise.
I'm really surprised they don't charge a very high price with a generous free tier, and then have an army of salespeople who will offer you whatever discount is necessary to make your business model work.
If you want to use GPT-3 to build an AI-lawyer, it'll be very expensive... but to build a tool for customer support for walmart will be cheap, etc.
The OpenAI CEO Sama is primarily an investor, not an operator, so the model is tuned towards giving small companies an opportunity versus enticing enterprise customers. Sama want somebody to figure out how to run & productive his business versus actual customers. In particular because of efforts like Stable Diffusion—- it’s going to be a lot easier for enterprise to build off tech that’s actually Open.
If your looking for a free teir that allways is there checkout https://text-generator.io which has 100 free requests a month. The free their really helps people get started with it or run demos etc without needing to commit to anything
hence the generous free tier. You want it to be big enough for a team of developers to do all the experimentation and development work, and also to launch a product to a small audience.
As soon as your product is a success, you run out of free tier, and you will have to negotiate a price with the sales team.
If I were OpenAI, then while I had a market leading product, the price would be 'I want half the profit margin of whatever you're using it for'.
> We have been looking forward to reducing pricing for a long time. Our teams have made incredible progress in making our models more efficient to run, which has reduced the cost it takes to serve them, and we are now passing these savings along to our customers.
I have used BLOOM a little, and I struggle to get it to produce the same kind of output as GPT-3. Anyone have tips there? It often seems to not keep the right context of the previously generated text.
GPT-3 has been fine-tuned after release to better interpret prompts (see InstructGPT). Perhaps Bloom is more like the original GPT-3; a little more raw and requiring better prompt engineering?
In my small amount of testing of Bloom so far it seems capable of advanced behaviour but it can indeed be trickier to coax that out. Playing with temperature and sampling matters for sure.
Yes this move is I think forced hand by competitors like https://text-generator.io etc which is already 10x cheaper.
If the open source community then other company's didn't follow along with them then we would still be being price gauged in a monopoly situation, hopefully same thing happens to dalle soon
I lack imagination. What is GPT-3 (and generative text models in general) useful for, other than toys? It can write essays, can we expect it to replace writers for newspapers and magazines? Can it write coherent technical documentation? Replace speechwriters?
GPT-3 is useful for a lot of things. It can write essays, it can write technical documentation, it can write speeches, it can write fiction, it can write poetry, it can write jokes, it can write jokes about jokes, it can write jokes about jokes about jokes, it can write jokes about jokes about jokes about jokes, it can write jokes about jokes about jokes about jokes about jokes, it can write jokes about jokes about jokes about jokes about jokes about jokes, it can write jokes about jokes about jokes about jokes about jokes about jokes about jokes.
This answer was written by GPT-J when I gave it the following prompt:
question: I lack imagination. What is GPT-3 (and generative text models in general) useful for, other than toys? It can write essays, can we expect it to replace writers for newspapers and magazines? Can it write coherent technical documentation? Replace speechwriters?
I'm not sure about GPT-3 specifically, but often these models are trained with an explicit "stop token", which the model can output when generating text to indicate that no more text should be produced.
That said, there's no reason it couldn't go into an infinite loop, just that the models are pretty sophisticated and therefore are less likely to do this than earlier generations of models (like predictive text on smart phones in the past)
As someone who's used the models a fair bit (I work at OpenAI), I mentally sort applications into four buckets:
- Generative use cases, where you give the model the kernel of an idea and then you curate its output (e.g., blog writing, code completion, etc.)
- Extractive use cases, where you give the model some big piece of text, and then process it in some way (e.g., extract names and addresses, classify it, ask a question about the text)
- Transformational use cases, where you need to fix/adjust a piece of text, or translate from one domain to another (e.g., sometimes I'll use GPT-3 for little tasks like copying and pasting a table from a presentation and then asking the model to translate it to markdown; saves me a visit to Google and finding some table generator website)
- Comparisons, where you use embeddings to do search/clustering/recommendations over any set of strings (e.g., can combo nicely with the Q&A use case above, where you search over a knowledge base)
This is a really nice breakdown -- I made a little side project that falls into your "Extractive" and "Comparisons" bucket: https://www.tripwise.me/
My tools uses GPT-3 to dig through Airbnb reviews to find the "cons" of staying at a particular place and summarize them neatly. You can also ask any question at all in natural language about all the listings you're evaluating.
I'm bookmarking your repo for future reference, this is very useful
It’s incredibly good at generating scholarly text. Put in a title and abstract and it will come up with a strong intro that is clear and compelling. It helps me write academic papers more clearly, as a result. Also, great for early stages of grant writing or proposals. It came up with a fantastic conference proposal for me that I spent 20 minutes editing and then just sent, unedited.
It will aid speechwriters or help more people write their own speeches, I guess.
Assuming that AI tech is a person replacement is, IMO, the wrong mental model. It helps talented people be more efficient. And will help a lot of hacks “fake it till they make it.” Power to the people, if they can handle it.
I also asked it to help me write a letter to my homesick daughter at summer camp. It was really helpful. I feel that GPT3 has an enviable degree of emotional intelligence, nuance and conceptual understanding. Writing and communicating effectively with people is very hard. Our own emotions get in the way. Now we have technology that can help people be better communicators with each other. It’s a big deal.
When you say you're an empath, do you mean to claim that you have psychic powers? An empath is a being from science fiction with the psionic ability to sense emotions, like Counselor Troi on Star Trek. The term originates from the short story "Empath" by J. T. McIntosh, published in New Worlds Science Fiction #50, August 1956.
Thanks for clarification! I was a bit unsure if that was what you intended. As you mentioned, writing and communicating effectively with people is very hard.
It has always seemed natural (to me) to be able to directly feel what other people are feeling. Most people can. Sometimes, however, it can feel shocking when I realize I’m talking to someone who doesn’t feel what I’m feeling. Or, that doesn’t feel that I’m feeling what they are feeling.
I've started asking GPT-3 for relationship advice. It's often... insightful.
There is a bit of prompt engineering. If you start your question with things like "As a qualified therapist, what would you recommend to" .... There is also a bit of engineering with the question. I often ask about an analogous situation, rather than my own.
I'm not batting 100%, but I often get strategies and things to think about I wouldn't have come up with myself.
You can use GPT-3 for relationship advice, but should you?
Could you elaborate a little more on what you're asking? Perhaps an example?
I feel as if any use case like this, — particularly if it crosses into interpersonal relationships and disagreement — risks being incredibly dangerous as far as reinforcing your own biases.
It should go without saying that to even consider this, you should have a solid grasp of the fundamentals of these LLMs, their limitations and inherent biases, and their tendency to unconditionally agree with you regardless of your position – particularly in a dialog-style format.
One clearly problematic use case would be asking about an interpersonal conflict where there exists an established societal and cultural power imbalance.
E.g., describing a disagreement between a man and a woman and asking for insight into that conversation would be incredibly flawed, no matter how objectively it were depicted.
This isn't an attack on you nor am I claiming you're doing anything at this level, but taken to the extreme it’s something I can easily imagine someone doing after reading your post.
I’d really just like people to be cautious. After all, this is the same language model that told me a social media post contains hate speech due to “negative sentiment towards a protected class (racists)”.
With GPT-3, that bias is pretty explicit, and not hard to manage. I also don't treat advice as necessarily /good/ advice. A good way to think about it is ideation.
A good example is to ask how to communicate something. I sometimes miscommunicate what I'm thinking, and it often takes a lot of effort to figure out how to say something. With different prompts, GPT-3 will give different ways to say something, and I can pick one which works well and meets what I want to say. Let's say you promised someone something, and need to change plans, and you don't want them to think you're blowing them off. GPT-3 will often give me good ways to communicate something like that.
Or another example is: "I am struggling with ____. What are good strategies I can use to ____."
GPT-3 will often come up with things which I wouldn't have come up with myself, or ones which would have taken me a lot of time. Just as often, it will give a useless, generic list of suggestions. At the end of the day, though, it provides helpful ideas often enough to be valuable.
GPT-3 definitely doesn't (and shouldn't) act as an arbiter in a conflict, but even in conflicts, there's often a solution which works for both sides. My major problem is that I tend to think slowly, and come up with those solutions too late. GPT-3 thinks a lot faster than I do, and having those ideas before they're moot is sometimes helpful.
As a footnote, human therapists tend to reinforce biases too. They only have one side of the story. I've seen people really damaged in the way you describe. In one case, both the person and the therapist were living in a (plausible-sounding) fantasy world.
Concrete example: I had someone reaching out to me, quite a bit, wanting to interact. On the other hand, whenever we interacted, I had the distinct impression they didn't like me. It was not a person I knew well. I'm limiting a lot of personal information from the story, but that's a fair summary.
I asked GPT-3 for advice for how to interpret that person's behavior (with more context).
GPT-3 gave several plausible explanations, must of which I hadn't thought of.
Here's one that I find personally really interesting:
Imagine you have a disability, and the government agencies responsible for helping you have denied your claim. You need to write them a letter. You don't have much (if any) experience writing letters to government agencies, so you're finding that task challenging.
GPT-3 can write that letter for you, and it will de a really good job of it.
Of course, you need access to GPT-3 and you need to learn how to drive it with prompts, so this isn't necessarily an easy thing. But I think it helps lean in the direction of the kind of benefits people can get from this class of technology outside of just playing with the AI.
Because providing a template requires someone with knowledge of that particular problem space (in this case "writing letters to government agencies to inquire about disability benefits") to put in the work of designing the template, coming up with likely placeholders and predicting how people might need to use it.
GPT-3 can do a good job of this all today, without any additional training.
An expert could help out by providing a few example prompts, but the result would be a lot more powerful and require a lot less development work to put together than trying to build some kind of placeholder wizard from scratch.
A madlibs is more predictable, requires only one example, easy to understand for people with language issues or disabilities, more regulation-safe as it doesn't spit out unpredictable things.
And the madlibs is free to use, for everyone.
All things considered, GPT-3 is a more of a shibboleth of AI revolution than an actual one. Much better to give models like GPT-NeoX to talented development teams who can really optimize and fine tune it, attach great sampling strategies to it, and create great products from there. The API is just too limited for me to want to build a company atop it.
If no-one has built the madlib template for "a letter to the council asking about my disability benefits" then madlibs being more predictable isn't going to help you.
Still requires you to have an understanding of the concept, an understanding of how to prompt GPT-3, and how to verify the accuracy/veracity of the claims the AI model makes.
In a few years, when a generative text model that's actually open is made, erotic text will 100% be a main use. But of course people don't like thinking about that...
My friend used it to "explain like I'm 5" a medical report he had. I sent that to my dad (a surgeon in that specialty) and he said it was all accurate. I thought that was interesting.
I've used it a bit to help with creativity. For example, I wanted to brainstorm some unit designs and gameplay mechanics for a RTS game I wanted to build. I prompted GPT-3 with an interview with a game designer where unit designs and mechanics were discussed. For a prompt I used a few questions and answers and then a single question I wanted GPT-3 to answer for me. After a few attempts with slight variations it spat out a bunch of ideas for me. While few of the designs were fully worked out and immediately usable, a couple of suggestions by GPT-3 were able to spark enough creativity to create some cool things.
My favorite reply was:
"A: I think the best example of this is the ability to swap positions with a friendly unit. This ability can be used in so many different ways. You can use it to support an ally, or you can use it to escape a tricky situation. You can use it to surprise an enemy by swapping with a unit they didn't expect, or you can use it to put yourself in a better tactical position. It's a simple idea, but it actually turns the whole game on its head."
Besides all the ideas of well meaning people in HN, I fear GPT-3 (and its competitors) will just bring almost impossible to spot fake Amazon reviews, SEO garbage sites, fake blogs, fake podcasts, etc.
And even worse, now corporations and governments don't even need to have troll farms. Just rent some AI, give it basic directions, and let it loose.
Until Google starts to expect better garbage from SEO providers. The bot can simply ask a language model if the page looks well written and truthful. So SEOs will have to spam us with well written useful articles about topics we like.
Maybe Google should require webmasters to mark the automated content with a special HTML tag or attribute. Failure to comply leading to deranking. This would be necessary for the future - to know what data was human generated when training the next model, and to have official reason for penalising spammers who try to pass artificial text as human written.
Most of these seem preventable.
1. Stop ordering things from Amazon, support local stores instead of reading reviews.
2. Stop reading SEO’d sites, go to the source when possible
3. Only read your friends blogs and listen to your friends podcasts.
i think this is a bit pessimistic. You can also use GPT-3 or similar technologies to much more easily train a fine-tuned "real vs fake" review classifier with fewer training examples than was previously necessary.
GPT-3 like all tech is double-edged I think. Has good & bad things about it
I doubt this will happen - at some point the generated content should be indistinguishable from human written text - and AI can produce it in droves and learn to exploit existing algorithms.
Is it just me, or are phrases like "3x cheaper" hard for other people to grok too? For some reason they always require me to pause and try to sort it out. Like, is the price being reduced by 66% or 75% or something else? What would "1x cheaper" mean?
For the opposite word, "expensive", it all depends on the word before it: "as expensive" or "more expensive".
"100 widgets cost $100. Unit price is $1."
They are now twice _as_ expensive -> they now cost $200. Unit price is now $2.
"100 widgets cost $100. Unit price is $1."
They are now two times _more_ expensive -> they now cost $300. Unit price is now $3.
But How do you do this with the word "cheap"? Does "cheaper" clearly mean "3x as cheap" or "3x more cheap" (which even sounds a bit wrong)? I guess it means "more cheap" means "cheaper".
Better to avoid the problem and say "They are now two times the cost, or half the cost". Which is much clearer.
> For the opposite word, "expensive", it all depends on the word before it: "as expensive" or "more expensive".
Frustratingly, I find this isn't always true in practice. Lots of people use "x times as y" and "x times more y" interchangeably. To avoid ambiguity I try to only use the former in any context where precision is useful.
That reminds of the sign "Buy one, get one".
Does that mean you pay for an item and you get that item, or you pay for an item and you get another item for free? I expect the meaning to the be the second one, but my brain parses it to be the first one.
That’s because it they have left off the word “free” - it’s just an odd shortening of the less ambiguous “buy one get one free”, probably made by someone who is very familiar with seeing “buy one get one free”
It not really all that odd in retail. “BOGO” (pronounced bo-go) is quick to say on the floor and is less awkward than “BOGOF” which sounds like either a franken-portmanteau of “beef” and “stroganof [sic]” or an amateur spy’s attempt at a cover identity.
It’s also used more generally for buy one get one X% off sales and you’ll see lots of “BOGO50” promo/coupon codes. Then marketing jumped on the bandwagon and started plastering “BOGO” all over marketing material so consumers are used to the lingo now
I know what you mean. 1x cheaper would be same price. 3x cheaper means “the new price times 3 equals the old price.” However this is easier than percents or fractions. 100% cheaper means free. Do you want something 67% cheaper, 2/3s cheaper or 3x cheaper?
Let's say you have a product for sale at $32. Then you have a sale to boost name recognition, and have a coupon for $32 dollars off (limited time offer). Or equivalently, that would be $32 cheaper. So the net price is free ($0). $32 cheaper = 100% cheaper = 1x cheaper. Likewise, 25% cheaper = 25% less expensive, or doing the math $32 - 0.25*$32 = 0.75*$32 = $24.
That’s precisely why it’s confusing, yes. You can’t equate 100% cheaper with 1x cheaper. 99% cheaper is sensible. 200% cheaper is not. 2x cheaper is. I’d love a cogent reason to explain this convention.
> I’d love a cogent reason to explain this convention.
"cheaper" isn't a defined operation except for "a is cheaper than b".
So the statements "6% cheaper" and "2 times cheaper" are shorts for "it is cheaper, it costs 6% less" and "it is cheaper, you get 2 times more for the same amount of money".
No, it isn't sensible, and is certainly confusing, and that is why you should commit to never using that phrasing from this day forward. It isn't like there is a hardship you have to overcome when using precise language. There are plenty of alternative ways to state the desired objective that are clear and unambiguous:
- The product is now half-priced!
- The item costs half as much!
- The unit comes with a 50% discount!
No, it's not just you. When you "times" something it is multiplication, not division. The phrasing is a logical impossibility, where the reader has to guess the intent of the author - of which there are several interpretations. The key here is that this is a language "thing" only and certainly not a construct of logic or math.
In the headline of this page, it wouldn't have taken but a moment to write a phrasing that would be understandable to all readers of the page. It could read, "GPT-3 will be Two Thirds Less," or, "GPT-3 will cost One Third the Price."
This example from a non-native English speaker shows the difficulties of having several possible interpretations, at least three! [1]
To open.ai's credit, I don't see that text anywhere on the page or in the page title. The only info I see on the page about the price change is.
> We’re making our API more affordable on September 1, thanks to progress in making our models run more efficiently.
If you visit the page the price change is pretty clear, they show the before and after. 1k tokens will be anywhere from 1/2 to 1/3 the current price depending on which model you're using.
It probably means that it has 3X better value. If I spent $1 previously to do X, I can now do 3X for the same $1.
I think this is more intuitive for most people than using percent difference for the same reason using a raw ratio (speed up) is better for expressing performance improvement.
I usually read that as "new price = old price / X" where X = "3" in this scenario. In my head, "1x cheaper" would mean "new price = old price / 1", so no reduction, I guess.
For me, I tend to think of "3x" and "cheaper" as separate operations. So "3x cheaper" sounds like you're saying new_price = price - 3*price.
That (usually) makes no sense, so I assume people mean something else when they say it, but I never have any clue what. Is it a third of the original price? Is it two thirds (new_price = price - price/3)?
Percentages also work this way. If I say "10% cheaper", most people understand that I mean the price is 90% of what it was previously. Although percentages get weird when increasing the price.
e.g. if it was a flat doubling, I would say that as "200% of the price" (new_price = price*(200/100)).
However, I take "200% more expensive" to mean new_price = price + price*(200/100). To me, the "more" implies addition specifically.
One of my former colleagues who was always in charge of publishing our team's performance testing results was a complete hardass on the rest of us for using proper language around multipliers and superlatives. It's well worth the clarity to avoid precisely this situation.
It's hard to understand because "cheaper" only makes sense as a relative comparison.
Assuming that if, say, the service was 25% cheaper than the nominal price last month, it is now 0.25 * 3 = 75% cheaper than the nominal price. However, it would make more sense to advertise that as a 75% discount over nominal price, so probably that's NOT what they meant.
That's the only explanation that makes sense arithmetically, I believe. There cannot be another interpretation of "3x cheaper" unless you reinterpret the word "cheaper."
10x is literally the difference of an order of magnitude.
An order of magnitude is an exponential change of plus or minus 1 in the
value of a quantity or unit. The term is generally used in conjunction with
power-of-10 scientific notation.
Order of magnitude is used to make the size of numbers and measurements of
things more intuitive and understandable. It is generally used to provide
approximate comparisons between two numbers. For example, if the
circumference of the Sun is compared with the circumference of the Earth,
the Sun's circumference would be described as many orders of magnitude
larger than the Earth's. [1]
An order of magnitude is an approximation of the logarithm of a value relative to some
contextually understood reference value, usually 10, interpreted as the base of the
logarithm and the representative of values of magnitude one. [2]
Yes, an order of magnitude literally means 10x. But in my experience, in common speech it's often used to convey an approximation. i.e. 'Changing this will decrease performance of that endpoint by an order of magnitude', when it's really somewhere around 10x. If someone said 'decrease performance by 10x' that seems much more concrete to me.
From my copy editing days, it's drilled into me to never write "n times less" -- I get that it means "1/n" and that's how I'd rewrite it, but yes, it's notoriously confusing, or at least imprecise.
Consider “cheapness” to be the reciprocal of price; thus, something that costs $30 has a cheapness score of 1/30 antidollars.
For this item to become 3x cheaper would mean for its cheapness score to be multiplied by 3, thus 1/10 antidollars. Thus “3x cheaper” means the same thing as “price multipled by one third”.
I like this hypothesis; however, if that was indeed the original meaning, it would have more sense to advertise it as a 33% discount, or as slashing the price by 1/3. "3x cheaper" seems to be aimed to the average Joe shopping for shoes, not to a developer who knows arithmetic.
I'm not sure about OpenAI's internal policies, but I've worked for several SaaS companies that complied with Export Administration Regulations (EAR) to limit access to "Embargoed and Sanctioned Countries" which has meant CRIMEA - REGION OF UKRAINE, CUBA, IRAN, NORTH KOREA, and SYRIA. ITAR (defense related) and OFAC (finance related) both have further restrictions. For us, it was easier to block all of Ukraine rather than limit to access to Crimea in particular.
Trade compliance requirements may require embargoing occupied Ukrainian territories however it isn’t necessarily possible to distinguish between occupied and unoccupied regions. Export controls might be applied based on geoIP with granularity at the country level. Unfortunately the safest position from a compliance perspective is to block the entire country than risk getting it wrong.
I've been experimenting with using it to build a human-language-to-SQL tool, so that people can ask questions of their data "what country had the highest GDP in 2019" and it will turn them into the correct SQL query, given a table schema. I'm still iterating on this but it's shown some very promising initial results.
I use it a lot when I need to get something small working in a language that I don't have day-to-day familiarity with. "Write a bash script that loops through every MOV file in this folder and extracts the audio as MP3" is a good example of that kind of prompt.
That is the type of application that I am also interested in.
But how does one "train" GPT-3 on your business schema? How does one train it on any custom domain?
Maybe not for a developed, but for an AI based startup:
1. Generate synthetic data that is well aligned to your needs. With careful prompting + ensembling + after-fact human filtering you can generate a lot of very particular human-like data that you can then used to train/etc your product.
2. Generate labels. gpt-3 can give pretty good NLU results through appropriate prompting. You can do multiple prompts + ensembling to get very good labels on free text (sentiment, entity linking, intent, etc).
In both above use cases you can actually avoid deploying gpt-3 as part of client facing product, but instead leverage gpt-3 to train smaller "on-rails" models/rules/etc.
I wonder if anyone has successfully used it to create library documentation. Obviously you'd have to tweak whatever output you get but can GPT-3 provide a substantial starting point?
I really wonder about the economics and business models of large models APIs, mainly because of:
* large running costs due to the expensiveness of the inference
* low barriers to entry: the tech behind Dalle appeared 1.5 years ago if I recall right and there are already a few competitors (Midjourney, Imagen, Stable Diffusion)
BLOOM and OPT are both GPT-3 sized models that are open source and similarily sized (175B parameters) but a) text generation is not as good as GPT-3 for whatever reason and b) getting them to run in a cost-effective manner is not easy and with this new price decrease, it may be more economical to use GPT-3 for deployment despite that.
The reason the similarly sized open source models aren’t as good is because gpt-3 has had a 2 year head start of additional development and improvement. The gpt-3 that is available today is Much better than what was initially produced in 2020.
Not entirely. There are many ways to optimize models for deployment both at the software and hardware levels: some are public such as quantization/fusing, others are not.
The so-called "open" in their name is an oxymoron. The whole thing is a trap. We need to work on our own, possibly publicly founded but really open solutions rather than depend on proprietary ones.
We really gotta come up with another thing to repeat as the go-to diss than this. I'm all for putting pressure on them to release weights etc but they are a lot more open than e.g. google or meta, and the whole "ClosedAI" is so over-repeated at this point.
They brought it on themselves. They chose the name OpenAI. It was a terrible choice on their part, and the repeated criticism is a direct result of that choice.
I definitely agree, "open" in the name is pretty annoying and wrong. I'm a bit torn though - I tend to think GPT-3 is approaching a level where ML is becoming dangerous. Spam bots + GPT-3 is not my idea of a fun time. So the restrictions they put on usage do seem important.
Regardless of that moral question, I don't think a fully open model on the level of GPT-3 is even possible. Given the required cost to train and the expertise involved, big tech will always be a few years ahead. And it's unlikely they would give it away with how much they invest in creating it. Unless capitalism suddenly ends, I don't see any of the major tech companies parting with state of the art ML.
Or how about opening everything up and releasing all the models as open source?
Since even with these price decreases, lots of competitors are out there with more parameters and are open source, and are already matched or just as good as GPT-3 and are generally running OpenAI's pricing to the ground.
Might as well open everything up since you can't compete with free anyway.
The free alternatives to OpenAI are not as good as OpenAI. For example, Craiyon (formally DALL-E Mini), is good, but it's nothing compared to the real DALL-E.
Also checkout https://text-generator.io which is already 10x cheaper... I think they know text generation is no longer a monopoly any more hence having to move pricing, the openai margins are still really high though honestly.
It is not prohibitively expensive for me but it is a bit more than I would like to pay for such a tool. I hope that prices come down over time. Also, it is free for open source project developers and students but not for teachers and researchers, that is a little bit surprising to me.
I'm sure those are factors, but it's mainly (1) GPU prices have dropped significantly, and (2) a lot of progress is happening in optimizing large language models to be more efficient while maintaining performance.
Is any of that actually going to be a massive industry-defining feature? Marketing blogs are already negative value to me, relying on AI to write something as important as my resume is a non-starter, and marketing emails already go straight to the trash.
I'm very impressed with the AI image generation, but again, it replaces... stock art for blog posts?
If I was a creative writer then I think the AI could help me with plot outlines or get over writing blocks. But otherwise I'm not grasping why this is going to automate some huge number of jobs.
> "If I was a creative writer then I think the AI could help me with plot outlines or get over writing blocks. But otherwise I'm not grasping why this is going to automate some huge number of jobs."
Writing definitely looks like the key use case. The New Yorker used GPT-3 to write the concluding sentence for one of their articles [0], and I believe The Economist did the same (but for a slightly greater length; though I couldn't find the source just now). It's actually a bit hard to come up with a good conclusion, so this could save some effort and trouble. Separately, GPT-3 can also likely be used for articles about sports, especially if the purpose is to track changes on a scoreboard and present the changes with an article written in prose, versus displayed in a table.
It's quite fair that marketing blogs may not be interesting to many readers, but they can be useful for SEO/boosting the search engine rankings of a website. So, a cash-strapped early or small company could save money from freelance copywriting to GPT-3 if they're not looking for anything fancy, and a lot of people work as freelance copywriters.
I agree that this may not displace anyone in the high-end/high quality segment of copywriting, but it could reduce demand for content farms in the future (i.e. any application where low-quality copywriting written on a short notice is in-demand).
I am using it for a little project to make deciding between Airbnbs easier: https://www.tripwise.me/
You basically just enter in the Airbnb URLs you're considering, and it'll use GPT-3 to scrape through the reviews in order to find anything negative or positive people are saying about the listings.
You can also ask any question you want at all, and it'll return answers (if people have mentioned it in their reviews). For example "Is the air conditioning loud?"
I would love to hear what others are using GPT-3 for!
I can comment that a friend teaching college philosophy said his department is eliminating papers at the undergraduate level,
because it is now too easy to generate-then-polish papers at that level.
They're too good in other words: better than baseline even. And when touched up by a student who's putting in a modest effort, are disruptively better.
Probably has more to do with how slowly the field evolves (esp. wrt what undergrads are learning)... so I imagine there's a huge training corpus for what is ultimately a very limited number of meaningful assignments to give.
I was recently able to articulate to myself why I dislike language models like GPT-3 so much.
Language is fundamentally about communication. I have a mental state, and language lets me share that some of that mental state with another human being. It's quite remarkable, and the cornerstone for pretty much every aspect of our lives as social creatures.
GPT-3 can generate text that's convincingly human-like; it's an amazing technical achievement. But (so far, anyway!) it is just that; a generator of plausible text. To borrow a philosophical term, there's no "interiority" there.
You can see this clearly in GPT-3's output; it is at its weakest when writing about the relationships between a set of concepts, or when trying to carry a line of thought or argument through a longer text. This is because there is no underlying _thing_ that is being communicated; no mental model or grounding concept. Just text, generated solely based on plausibility in a given context.
So while GPT-3 is truly remarkable in its ability to generate text, it cannot, by definition, communicate (except on the meta level via prompt selection and output curation.)
Does the world really need more words but with less communication?