We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.
You said you praise the American Fair Use model. I said I don’t like it to work in that way. And most of all not grant exceptions to certain business models. And I agreed that there are some issues in the underlying copyright model, which might change the entire picture if addressed. I mean the interesting question is: How should copyright work in conjunction with AI and in general? And who needs to be compensated how?
I’m fairly sure the term “Fair Use” by definition means unauthorized and unpaid use. I mean we can try to twist the meaning of these words. Or maybe I misunderstood it. But paying would be kind of contradictory to the entire concept. It’d be (forced) licensing or something within the realm of copyright, depending on what you mean. But I think we need a new/different word for it.
In the US, it almost certainly wouldn’t be fair use if it meant that the author doesn’t get paid. Of course, you don’t get paid for the fair use, but there are a lot of things you don’t get money for.
You’re talking about authors not being paid at all. What’s that about?
That was about abolishing copyright altogether. Since we discussed that as an option. We’re now discussing what I called “subsidies” earlier. Authors do get paid, but for certain “uses” and not for others. And authors get financed by a different group of people.
In your example with the farmers, they’re not paid by me buying the product in the supermarket and that money gets handed down the chain to every supplier… But Nestle got the cocoa beans for free and society now gets to pay the farmer by a different method. Unless you have a specific proposal here, that’d be likely the definition of a subsidy to help Nestle and make their products look cheaper on a supermarket shelf.
A cocoa farmer is paid for some uses of their cocoa beans but not others. For example, Nestle has to pay to turn their beans into chocolate and sell it in supermarkets. On the other hand, no one has to pay to take a photo of their beans and sell it to Nestle for ads. Right? I’m with you so far.
I don’t get the next step. Because some uses are free, all uses should be free? Then Nestle gets a subsidy and we pay the farmers some other way?
Pretty much. The AI companies are Nestle in that analogy. They get their supplies for free?! While I and everyone else had to pay for the very same supplies, when I needed the textbook to study CS to become a computer programmer. The professor gets to brush up their salary, and I think it’s a bit unfair to me that I’m asked to take out 60€ from my mediocre turnover of a few hundred bucks a month as a student. I think I should have been asked to pay 30€ and a company with a billion dollar budget should be asked to pay something like 100€ since they make use of it multiple times. And they should hand that cost down to their customers. And my use was transformative as well. The information from the textbook is now modeled in my brain.
I think the analogy with the picture is kind of alright as well. I mean analogies are hard here, since it’s a labour intensive task to duplicate crops and coffee beans, while duplication is pretty much for free in case of information. And it doesn’t take away the original.
Now what is a picture? It’s kind of a summary, a depiction of the outer appearance. And snapping a picture of a book cover would make sense for Fair Use. That’s kind if what it’s made for. If you now snap a picture of each and every one of the 400 pages inside, that’s where law says Fair Use stops. And what do AI companies use for training? A picture/summary of the book? Or the content within?
Your professor abused their monopoly. That’s the sort of thing I’ve been condemning. You are basically fine with that. You just think they should adjust their price policy to income. Well, yes, that would be the profit maximizing move. You make everyone pay as much as they are able to. That’s what the copyright lobby wants. But I have to point out: There is no reason why they should lower the price for you. After all, you were able to pay. Rather, there seems to be room to raise the price.
Do you actually think this kind of monopoly abuse is a good thing?
Now what is a picture? It’s kind of a summary, a depiction of the outer appearance. And snapping a picture of a book cover would make sense for Fair Use. That’s kind if what it’s made for. If you now snap a picture of each and every one of the 400 pages inside, that’s where law says Fair Use stops.
No, that’s not what the law says. I think, the problem is that we have different ideas over how Fair Use in the US actually works. I’ll have to think about that.
Why do you want fair use to work that way?
You said you praise the American Fair Use model. I said I don’t like it to work in that way. And most of all not grant exceptions to certain business models. And I agreed that there are some issues in the underlying copyright model, which might change the entire picture if addressed. I mean the interesting question is: How should copyright work in conjunction with AI and in general? And who needs to be compensated how?
I understand, But why do you want a fair use model that means that authors don’t get paid at all?
I’m fairly sure the term “Fair Use” by definition means unauthorized and unpaid use. I mean we can try to twist the meaning of these words. Or maybe I misunderstood it. But paying would be kind of contradictory to the entire concept. It’d be (forced) licensing or something within the realm of copyright, depending on what you mean. But I think we need a new/different word for it.
In the US, it almost certainly wouldn’t be fair use if it meant that the author doesn’t get paid. Of course, you don’t get paid for the fair use, but there are a lot of things you don’t get money for.
You’re talking about authors not being paid at all. What’s that about?
That was about abolishing copyright altogether. Since we discussed that as an option. We’re now discussing what I called “subsidies” earlier. Authors do get paid, but for certain “uses” and not for others. And authors get financed by a different group of people.
In your example with the farmers, they’re not paid by me buying the product in the supermarket and that money gets handed down the chain to every supplier… But Nestle got the cocoa beans for free and society now gets to pay the farmer by a different method. Unless you have a specific proposal here, that’d be likely the definition of a subsidy to help Nestle and make their products look cheaper on a supermarket shelf.
Let me try to follow this.
A cocoa farmer is paid for some uses of their cocoa beans but not others. For example, Nestle has to pay to turn their beans into chocolate and sell it in supermarkets. On the other hand, no one has to pay to take a photo of their beans and sell it to Nestle for ads. Right? I’m with you so far.
I don’t get the next step. Because some uses are free, all uses should be free? Then Nestle gets a subsidy and we pay the farmers some other way?
Pretty much. The AI companies are Nestle in that analogy. They get their supplies for free?! While I and everyone else had to pay for the very same supplies, when I needed the textbook to study CS to become a computer programmer. The professor gets to brush up their salary, and I think it’s a bit unfair to me that I’m asked to take out 60€ from my mediocre turnover of a few hundred bucks a month as a student. I think I should have been asked to pay 30€ and a company with a billion dollar budget should be asked to pay something like 100€ since they make use of it multiple times. And they should hand that cost down to their customers. And my use was transformative as well. The information from the textbook is now modeled in my brain.
I think the analogy with the picture is kind of alright as well. I mean analogies are hard here, since it’s a labour intensive task to duplicate crops and coffee beans, while duplication is pretty much for free in case of information. And it doesn’t take away the original.
Now what is a picture? It’s kind of a summary, a depiction of the outer appearance. And snapping a picture of a book cover would make sense for Fair Use. That’s kind if what it’s made for. If you now snap a picture of each and every one of the 400 pages inside, that’s where law says Fair Use stops. And what do AI companies use for training? A picture/summary of the book? Or the content within?
Now I’m even more confused.
Your professor abused their monopoly. That’s the sort of thing I’ve been condemning. You are basically fine with that. You just think they should adjust their price policy to income. Well, yes, that would be the profit maximizing move. You make everyone pay as much as they are able to. That’s what the copyright lobby wants. But I have to point out: There is no reason why they should lower the price for you. After all, you were able to pay. Rather, there seems to be room to raise the price.
Do you actually think this kind of monopoly abuse is a good thing?
No, that’s not what the law says. I think, the problem is that we have different ideas over how Fair Use in the US actually works. I’ll have to think about that.