I think this makes 100% sense because GPUs are no built for inference-specific purposes-- just co-opted because it was at hand. The idea of bespoke hardware for a given model is intriguing. These optimized chips will be much faster/energy than the more generalized GPUs. Also, much cheaper and smaller I expect. Eventually, I think Local Inference will dominate almost all usages. Only extreme, edge-case's will require a cloudy data center.

I think just having tokens that fast opens up the doors to massively recursive chain of thought. You can get an even greater level of accuracy if you let the models refine they're thoughts rather than just get it to write an essay of the spot. That's what i'm most excited for.

*There were no transformers (Attention abstraction) before 2017* Etched is incorrectly assuming that Transformers (unlike RNN/LSTM etc.) will *NEVER* be replaced by an entirely new innovation (using an abstraction that's way more effective than *Attention* ). From that moment on, all LLMs including Vision Transformers based models will migrate to the new (better-than-Attention) mechanism. Now Sohu needs to be replaced by another special purpose ASIC. Meanwhile, nVidia is incorporating version-3 transformer engine in their post-Blackwell ( *Rubin* ) architecture.

Betting everything to transformers is a risky thing, their main drawback is the context size, memory needs goes with context size squared, the result is small contexts and memory bandwidth starvation, apart from needing zillion of ultrabytes to operate. Even the most advocate companies are looking for solutions with other architectures. We will be in a transition period for a few years.

We need this to make Open Source models viable for the average consumer.

Who’s going to buy them? Not the chips, but the company. I could see a major acquisition to pass the bag and build the moat.

Frick....shut up and take my money! This will make local hosting much more common.

Inference costs will start dropping based on how quickly these guys can deliver their product. Assuming same price as Nvidia cards and 20x inference improvement. Inference cost should drop by 20x. I bet however that their accelerators will be cheaper than Nvidia's.

🐴We should assume that other chip manufacturers will jump on the bandwagon of dedicated LLM chips as well.

So you cant use them to train models? That is where we need the biggest breakthrough at this point.

something will replace transformers. Will they be able to adapt and change fast enough when transformers architecture evolves to something else? Things are moving fast and research shows there's lots of room for improvement.

Very interesting, keep us informed. Thank you so much.

I went to there website and there's no place to place any orders for chips or software there not even on KZfaq yet is this company really there 🤔

Hold on a second. We're all getting worked up over this, but has anyone actually seen a real card yet? I can't help but feel we should be a lot more skeptical here. The whole thing just reeks of 'too good to be true'. Look, I'd love to be wrong, but let's face it - this wouldn't be the first time someone's tried to pull a fast one in this industry. These guys aren't blind; they know the market's on fire right now, with money being thrown around like there's no tomorrow. Maybe we should take a step back and think twice before buying into all this hype.

interesting to see a real demo of this, that is what my robot needed!

If Gen AI settle on transformers already, I expect mobile SoC and laptops to add a Transformers ASIC block on each chip Just like video encoder and decoder blocks for each individual codecs.

It's like any other kind of technology. Those who enter first can get superseded by those who follow as technology, techniques and thinking changes. it's great because it keeps the pressure on to have companies innovate and improve. With the resources at NVIDIA's disposal they can too observe improvements by others and include them with additions in their next iterations of product.

I think this is less interesting as a specific product than it appears on the surface. It's interesting certainly for the implications on the industry, but in terms of its direct relevance to "ordinary" people who like to buy a piece of hardware and use it (as opposed to being locked behind APIs) it's way less useful. Adding onto that, I think the most interesting thing in inference workloads is ultra low precision; forget FP8, why aren't they doing Int8? Int4? Int2 (as per Bitnet 1.58 I think 1.5 bit should be possible with a dedicated ASIC) could be incredible. Floating point numbers aren't actually very easy to work with on a hardware level, so it seems like a really weird choice for an inference only chip, as I'm sure they could have squeezed even more performance out of just using the same number of bits as integers, let alone using low precision integers. (I think Int 2 could potentially be ~4.5x faster off the top of my head). More to the point, given I'm primarily interested in hardware I can actually buy, if I wanted something like this Tenstorrent's accelerators seem way more interesting, and affordable. With all of the nay-saying said, the one thing about these that seems really interesting is that Moore's Law isn't actually dead. We're still getting more transistors. The issue is that it becomes harder and harder to control them all together in a centralized manner (like on a CPU), hence, CPU performance has declined, and even parallelized you see the same leveling off of improvements over time due to things like accessing cache and so on in GPUs. I'm no hardware engineer, but it seems to me, that in addition to gaining more performance from removing roadblocks (ie: only using hardware needed to calculate a Transformer network, so no CPU or GPU specific elements are included), Transformer specific hardware should still continue scaling with performance node improvements to a greater degree than existing legacy architectures, very similarly to how bitcoin ASICs were able to do so.

most amount of compute will be needed for inference to produce synthetic data so this type of chips might be more important than training chips

Like groq LPU, won't they provide us any platform to try it out ourselves?

Nemotron was released a couple days ago, definitely see a race going on 🔥🔥🔥🔥💡💡

Nvidia traps itself. It's enough to remember how 3d graphics has started - 3dfx creating the gpu to do specific load - meshs calculations making quake run fast. gpu were the tool to do certain task faster, not "everything in the world should be ever done on gpu", as it was on cpu before..

Inference is not the biggest capital outlay for the big players and I thought the Google, Microsoft's, Meta's & Amazon's of the world are already making their own GPU's. its making the foundation models that requires big CapEx spends, this is of little threat to Nvidia or am I missing something?

Excerpt from a Nvidia Press release: "DRIVE Thor is the first AV platform to incorporate an inference transformer engine, a new component of the Tensor Cores within NVIDIA GPUs. With this engine, DRIVE Thor can accelerate inference performance of transformer deep neural networks by up to 9x, which is paramount for supporting the massive and complex AI workloads associated with self driving."

More than wanting to buy one.. How do you buy stock in this company? Are they on the market yet?

Dead on arrival if data center + inference only is their strat. The initial sales will be great but nvidia, AMD, Intel, and Google are doubling down on merging inference and training. When you’re taking data center, the model that Sohu is taking will not make it. Their positioning will make or break them.

Wen consumer home ASIC LLM inference machines for under $4,000? 😊

The challenge today is still training, not inference.

Of course when we have a model with "average human intelligence," it's not going to have the _knowledge_ of an average human -- it will be vastly more knowledgeable than any human. It'll be like an average human only in specific sorts of reasoning, where they are rather poor presently. The top models are already quite good at verbal analogical reasoning.

Majority of AI compute over the last year is being used for inference. If these guys can deliver these accelerators and servers quickly with no issues, Nvidia won't know what hit them.

Arent transformers going to be replaced by mamba/S4 models? And even if that isnt true, is it really a good bet to assume the transformer will be the dominant type of model forever?

500k T/s is genuinely the most unbelievable absolutely insane thing I've ever heard in my life so until we actually see it irl I will continue to not believe it but I hope it's true so badly

self-driving cars could use this

When a new ai model appears, this startup will tank

Where is the John Coogan video? I looked and couldn't find it.

4000 € ? I have money ! I wish to buy one Sohu (and more after) ! Where can we buy it ?!

Brainchips Corps revolutionary neural spiking Akida2 Chip has its own revolutionary new algorithm called TENN's creating ultra low power consumption and possibility similar productivity Etch whilst being highly adaptable And it doesn't require a cpu or memory to operate.

the Ai hardware accelerators race has been started !

What’s the price tag for one of these though 👀

Watch Microsoft as they have gone the complete binary AI route and they have their core of users that they want to service, gaming and business. the one thing that is blatantly obvious it that the trend is for server centralization when that may not be in the best interests of the user or get the best results.

this is actually huge news..

Can it run a diffuser like SDXL? Or a diffusion-transformer like SD3 / Sora?

One day those type of chips will be on computer or movile device and work with the largest text or video models witout even internet

Isnt this just theorethical so far, if not, where is the demo?

500,000 tokens/s???? 😂 Groq is TOAST

Good luck, hope they succeed, gpu/tpu/npu/apu manufacturers and clouds just burning the power with their not optimal solutions for ML and charging too much for it...

What of training. Inference is only one part, training is really, really, really important.

Nothing about Power usage / price

Nahhh…nvidia just would integrate more specialized circuits into its chips

We need a maba acsci lpu.

without making chips that can combine memory and processing on wafer, you can't create a human-like intelligence

wow. designing and building hardware that can only run transformers, not knowing if something better would come along. But what an amazing payoff if they can be cost effective on release.

Not sure it's wise to invest in making an AI chip only for transformers when transformers are not what is going to take us to the next level. Transformers are limited regardless of context size or how many tokens you can generate per second.

This is real

human years to #AI is like dog years to humans.

There is an issue of the shortsightedness of designing chips to do inference alone, especially that of transformer based models. Transformer based models are probabilistic in nature, and businesses and individuals can't bet on them alone. Other forms of software and hardware needs to pair with this before it really does have long term value. It's a good start, but NVIDIA is still ahead. After all, NVIDIA use their GPUS to not just simply generate probabilities that are novel and interesting, they use them to design chips and model physics. Reliability and consistency is the real magic here. And no one, not Meta or Google or OpenAI have been able to hone down AI to the point where it is reliable and consistent. And they will never get there with simple inference chips like these.

This clown just said no one is using AMD MI300X. 😂 Microsoft/Open AI, Oracle, IBM and Amazon AWS and more have announced they are using MI300x. Reportedly it is one of the best selling AI accelerators right now.

Ai overhype. Can we …cease Ai? Or else be… laid off by Ai, then human extinction? Or suffer an… Ai new world order? With swell robotics everywhere, …Ai jobloss is the top worry. Anyone else feel the same?

ouh

Finally someone showing how Jensen's marketing BS was just that!

And I can make a chip 10000X faster than them

This is fake news dude...

So far nothing but claims and nothing to show for it.