Transcript 0:00 The memory is the biggest problem. There is literally-- The LLMs are a memory-heavy, uh, method, and memory is everything. 0:10 So one of the biggest problems that people should solve is, like, how do you get the same performance without using as much memory? You know, once you solve that problem, a lot of things fall into place, right? 0:22 Optics was just like for long-haul links, you know, metro links, uh, undersea cables, some applications, whatever. Um, but never has it taken a prominence like it has today. 0:35 Today, it is the technology that is required for AI and the future of the build-out, and the entire world is scrambling for it because they're like: "Look, we were never prepared for this demand." 0:47 Like, this industry has never been built for this level of demand. You are asking for too much. So everybody's in short supply because of this. What is the useful amount of work done per token? 0:58 Like, do I always need 2,000 tokens per second? I don't. Like, you know, I'm talking to you right now, but my AI could be crunching some PDFs that I wanna read later or something like that. 1:08 I'm-- I don't have any immediate need for it. [on-hold music] Awesome. Cool. Well, Vik, thank you so much for joining me. Uh, very excited. 1:23 Um, I appreciate you coming on the podcast. Um, I've been following your work for some time and very impressed, and helped me personally kind of come up to zero to one as fast as possible. 1:35 So wanted to reach out and see if I could get you on the podcast, because I think you do a very good job of explaining technical concepts in ways that people can understand. And 1:49 I, having known lots of engineers, it's a tough skill. So I appreciate- Yeah... uh, all the work that you've put in to be able to teach others. Oh, thank you. Thank you for having me on. 1:59 Always happy to s- sit down and talk tech, and how things work is basically where my head usually is at. Uh, but yeah, like always happy to talk about this stuff. 2:11 Very-- Even more happy to hear that, you know, my explanations and writings and talking on Semidoped and my newsletter on Substack, Vik's Newsletter, uh, has all been resonating with people who are consuming my content. 2:27 So like, very happy to hear that. Yes. And for the people that aren't aware, I'll link all of Vik's, uh, Substack, podcast, everything that he's been working on, including his institutional research. 2:39 Um, please reach out to him because I think he does a great job. Um, and I'm gonna have- Appreciate it... to reach out to you afterwards for better podcast studio setup. Maybe I can- [chuckles]... 2:48 uh, get my own frictionless, uh, um [chuckles] logo in the background. Yeah, why not? Maybe- It's nice... 2:55 maybe a great place to start is obviously the industry has really gone, it feels like zero to one, or maybe b- better term, zero to 100 since Opus 4.5, um, really Q4, um, of last year. 3:12 Uh, we start- started moving off kind of OAuth subscriptions to API base, and that kind of kickstarted the whole race, it seemed like. 3:20 Um, and now everybody is really talking about agents, which I think is very fascinating, just computer use, uh, being able to do things for more normal people outside of software engineering and coding. 3:33 Um, and we kind of have a new race that gets kicked off. So maybe to like open a new question to start off the podcast is just how are you seeing the agent landscape, and what are your broad thoughts? 3:46 I really like the, uh, current landscape of agents a lot, more so than I've ever liked it since the beginning of this year, which is when like agents really took off. In the beginning of the year, it was 4:01 more like: "Oh, how do we step away from the chat interface and maybe get it to work from the phone for the first time?" 4:08 Like, literally when OpenClaw came out, this is literally when you could like text from your phone, and you could get responses, intelligent things happening, notifications, all of that stuff. 4:22 But the problem with that approach, at least earlier this year, was that it was very difficult for, quote-unquote, "regular people" to go and set this up. 4:31 And I'm not just saying like the, the, the regular non-technical person. I tried setting up OpenClaw. I still have a- It's tough... home server that's running it. Yeah. It's not, it's not straightforward, you know. 4:41 It's like it's pretty, pretty hard. I had to s-- I spun up my own VM. There were all these security concerns about giving OpenClaw access to your whole computer, right? 4:52 Like, so people are like: "No, no, no, don't, don't install it on your laptop. It's way too intrusive. Go get a different computer." So everybody went and bought a Mac Mini. You remember that whole thing? Yeah. Yep. Yep. 5:03 Uh, to me, that was super funny, um, just given that most of the models that people would run, in my opinion, were not going to be local models, um, even though people went out and bought specced out MacBooks Minis trying to get as much unified RAM as possible, but then they still do API calls to data centers. 5:23 Yeah. The whole idea was that you can run some kind of a local model that at least handles your personal tasks or something like that. 5:29 But again, it's like a very expensive proposition, uh, because models are getting like cheaper to use over time, but you have to make this upfront investment of like several thousand dollars hoping that the local model you will deploy on your allocation of RAM, uh, would actually do the job. 5:44 It's not a very long-lasting bet. You'd rather bet on token pricing going down, being able to use frontier models for cheaper, uh, over time, right? Yeah, fully. Um, maybe as-- M- maybe an interesting place to start 6:02 pulling on that thread is obviously as- Agents are starting to get in, in popularity. We had OpenClaw kick this all off, and then you kind of saw Hermes model come out. Mm-hmm. 6:15 And both were-- I think each iteration is probably a little bit more user-friendly than the next. Um, then GroqBot came out, then you had Muse. What is kind of your take on what it actually takes to run these things? 6:30 Because it has been interesting that people went out, tried to buy a bunch of Mac Minis, and to your point, the CapEx for a normal, um, person was high. Mm-hmm. 6:40 Um, and that doesn't normally seem like the path forward, where you ask the kind of everyday person, so to speak, to go out and spend $5,000 to $10,000 to have your personal assistant. Um- Yeah. 6:52 But a-as these things start to scale, um, obviously that affects data centers and kind of the data center build-outs. 7:00 What are you kind of broadly seeing under the hood when you kind of peel this back, if these agents really start scaling to tens of millions, hundreds of millions, and billions of people? [lip smack] Yeah. 7:11 So the evolution is very interesting because when people went and bought out, uh, these MacBook-- uh, these Mac Minis, it was important to ask the question like why were they buying them? 7:22 The reason was not always because you could run local models, it's just that they wanted a separate environment where they don't have to have their personal information. 7:31 So on that Mac Mini, you could just keep the stuff that the agent can access. Even if you're using the cloud models, you could just keep stuff. Like never log into your email in that computer, for example, right? 7:42 Never log into your bank accounts. So because you don't know what this agent is gonna do, nobody was speaking about guardrails like earlier this year. [chuckles] So- Yeah... 7:49 you just like give it its own and sandboxed environment, and say, "You know, go crazy within this. Your damage is gonna be limited. You're not gonna blow up my life. 7:58 You know, you're gonna keep it contained within this box. If stuff goes really south, I just like wipe this box and everything is normal again." But then the immediate thing people said like, "Look, look, look. 8:08 Like you don't have to spend like hundreds of dollars to buy a Mac Mini. Just go get a cloud virtual machine." So people were like, "Okay, let me go just like spin these up." 8:17 You know, so all these talks about like all these like DigitalOcean and all these kinds of companies went up because everybody was like spinning up virtual machines in the cloud. 8:25 Still, like all of this is difficult to do. Like first you have to buy a machine, then you have to get OpenClaw installed in. 8:30 Now you have to get a virtual machine installed, and then you get OpenClaw on that virtual machine. So the, uh, next, uh, step of the evolution these companies said is like, "Wait, why don't I do all of that for you? 8:41 Like I'll give you the virtual machine, I'll install the agent, but I don't wanna install OpenClaw. I'll install something that my company is developing." So GroqBot was like the first one I tried. 8:52 So as soon as it came up, I, I signed up for $200 and said like, "I'm gonna use it right away." So- Sure... it was fun because you could literally spin up agents by talking to it. 9:01 It had these cute icons and this GUI was nice. I'm like, "This is sweet. Like I don't have to do any of that stuff I tried to do with like OpenClaw. This is amazing." So in minutes, like I just downloaded it. 9:12 My credit card was like already on file [chuckles] so it's like bye-bye $200, and then like my, my agents are running. It's a, it's awesome, right? 9:20 So it's become accessible now, uh, way more than it ever was, and that's where we are now. And that has its implications on what actual hardware, um, we would need to run all these things, right? 9:31 Because now the-- what, what people were going and buying, uh, the, these companies like Groq, now Muse too. R- Muse runs two CPUs and like eight GB of RAM, supposedly. 9:42 This is what people are saying is running behind every, every, um, agent that you sign up to use. But all of these are like kept somewhere. There's a computer running on your behalf somewhere in the world, right? 9:56 Doing your stuff for you. 9:58 So all this translates into is into like demand for CPUs, demand for RAM, uh, demand for more inference because all these machines are now, now going to be running inference without you asking for anything. 10:10 That's the whole point of an agent. It's gonna run its own inference and tell you what matters when it matters. So all of these things are going to go up. All the real question is relative amounts, right? 10:21 Like what is more, what is less, yeah. But all of them should go up. [chuckles] Yeah. It, it is interesting. 10:27 I-- when Groq kind of got his own virtual machine and was able to kind of start doing computer use on the back end instead of tapping into your own, it did kind of feel like a magical moment, even though that you could set this up, uh, by yourself. 10:42 It's just, to your point, a bunch of extra steps that were a little bit more cumbersome. Um- Yeah... 10:48 yeah, and I think to me the interesting thing, and I think you once talked about this quite a bit, is on the computer u- side is there's a bunch of software that has been invented in the world that does not have APIs or m- MCP connections. 11:03 Um, and if you can use it like normal human, uh, that unlocks a lot of, uh, functionality. Um, and it's been amazing even more recently just with Astra and kind of the advances there on the computer use side. 11:18 I've been pleasantly surprised. It's still not obviously at the human level, but it seems like the agents in each iteration of the model is getting better and better at just doing things on your behalf. It's amazing. 11:32 I have a story around this. So when there's another agent thing that, uh, we haven't mentioned yet called Instinct, which is like ga-gathering a lot of attention right now. Uh, a lot of funding too. 11:42 Like even, like Code2 actually on their X account posted about Instinct today. 11:48 It's very exciting, um, technology and, uh, I was-- I ha-ma-managed to get my hands on an invite pretty early, so I was like, "Okay, you know, I'm, I'm gonna try it. Like I have to try it. 11:59 For the, for the sake of science, let me log in." [chuckles] "You know, if my stuff is gonna be hacked, I'll deal with it later. I mean, like YOLO this whole thing now." 12:08 [chuckles] So I like signed up, I gave it my access, and then I got this, uh, registration email for a conference. "Hey, you know, Open Compute Project Global Summit is, you know, open and you can, you can register here. 12:21 You know, here is the code," and all of that stuff. So I'm like, I was doing something else and I was like I told Instinct, "Yeah, yeah, fine. Do it. You do it. Like sign me up, you know, just do it." 12:31 And then it-- I don't know, I forgot about it, went back to doing what I was doing, and then came back to it. 12:35 I got a message on my phone because I was running it through my, uh, iPhone, and it said, uh, "Oh yeah, you've been signed up. Here's the QR code, and, uh, oh, by the way, I ordered a T-shirt for you, medium size." 12:47 [chuckles] "That should be okay, right?" I'm like, "How the hell did you know my T-shirt size?" Like that worked out great. Like it did everything and even got my T-shirt size right. So, uh- That's crazy... 12:56 that might have been a lucky guess, but I'm like, this is-- I'm in. If something can go and do all that for me, I'm in. [chuckles] Yeah. It's-- I, 13:06 I think obviously the zero to one with the software engineering side, I mean, that was kind of a slow iteration over time, but- Yeah... 13:14 I am excited now that more people can interact with AI outside of just the chatbots and LLMs. 13:23 And it does seem like agents and, uh, kind of thinking about it and the limit, um, hopefully in the good sense of like Her, the movie, um, where it's like- Yeah... the agent just like knows everything about you. 13:35 You can do things. Maybe you have a personal relationship with your agent. But- Yeah... definitely seems like that is the direction that we're heading. Um- Yeah... 13:43 now how to actually make that work under the hood from both a kind of how many gigawatts and how large of a model do you need versus like how much, uh, memory do you need, and do you need [chuckles] kind of one CPU? 13:58 Do you need many? Uh, I think is also an interesting question for people kind of following the space and trying to- Yeah... go deeper on some of the hardware side of things. Yeah. 14:08 I have a, I have a take on this at least to the-- at the very least. I don't know if I'm right, but I'll, I'll, I'll say it. So for most people's tasks, I think day to day, right, we don't need frontier intelligence. 14:18 We don't-- we really don't need to run, you know, the, the heavy models with like extra reasoning set and all that. 14:27 So in general, I think a lot of these models that run personal assistants can do with, I don't know, Sonnet level, uh, models. Or even if you go open source, I don't know. 14:38 It, it could be the cost could be even lower to run these things. Um, and from the-- So the inference-wise, I think the token count is going to go up, but it's not like everybody's gonna be running the frontier model. 14:50 So that has its own consequences, right? So maybe we can accommodate more users running, uh, like Sonnet level all the time compared to everybody trying to run frontier models all the time. 15:00 There's a big difference on the infrastructure needs between these two cases, and I think personal agents mostly don't need frontier model. 15:07 I mean, you're not gonna ask your personal agent, like solve this gigantic math problem. I mean, that's not like a personal agent kind of a job, right? Uh, those kinds of things will continue to happen on this side. 15:17 So it's going to be like mid-level models who-- which will do just fine. 15:22 So the other thing is that the whole CPU allocation story, it would be wrong to assume that everybody who, uh, runs an agent is going to have their own CPU all the time, right? 15:32 It's not like buying a Mac Mini, but in the cloud. It's, it's not that way because, um, whenever y-your agent is not using it, that piece of hardware will be used serving other people. 15:42 So it's a shared resource across everybody. So the real question is how much sharing will happen, and, you know, how often do you need it? 15:52 Like, do you need to-- your agents to be working all the time for you and doing stuff all the time for you? Um, or, you know, maybe in every hour or so it could come to you and tell you, "Hey, this is what has happened." 16:03 I already have my ChatGPT doing that for me. Every hour I get a notification on ChatGPT that says, "Hey, you've missed these important emails. 16:11 This one needs your attention immediately, but the rest of it you can do later in the evening." So I already have that set up. So, so it's about an hour every day. 16:18 Like, I mean, I have other stuff to do, which I don't always need the agent running for me. So when it comes to hardware demand, right, like 16:26 it's re-- I think there are a lot of people who will get on this train now because it's easy to use. You can text. You-- Everybody knows how to text somebody, right? So you just have to text a, like an agent. 16:37 That's all you have to do. A lot of people can do this, but I think the hardware demand itself is not going to be like cutting edge frontier stuff all the time, you know. 16:46 It's going to be shared CPUs, lower level of inference. Memory will be like shared as well. 16:52 Like maybe they, it will-- they will decommission your memory when you're not using it and spin up your computer just in time to use it, you know. So a lot of this is going to be shared. 17:00 So a lot of the sentiment that everything is going to explode because of agents may not actually happen. And is that because of y- in your belief, it's more shared resources and that the 17:15 frontier intelligence is not necessarily required for every day-to-day task, so that you could stay constant and just onboard additional users? Yeah. You c- No, you don't have to stay constant. I think it'll go up still. 17:29 I think it will-- we will need more compute. We always need more compute. Even for training, we, you know, the scaling laws haven't ended, so we can make bigger and bigger clusters, put in more and more GPUs. 17:39 These things keep getting smarter, which is amazing. So we're gonna keep building bigger, uh, GPUs and CP- That, that path has never changed. 17:47 Uh, but the incremental demand for like agents, um, from a much broader population, uh, who would like to use these things, of course, that depends on cost. 18:00 Because right now the ChatGPT Dots thing is gonna be only for the prosumer, right? So I mean, the adoption- Yeah... is a questionable thing at that point. I think Muse is much, much more inex-inexpensive to use. 18:11 So adoption is a question, but assuming a lot of people adopt it, um, they're going to use more compute than we did in the past. But 18:21 I-- it-- we have to be careful to like limit our expectations that it's not going to like completely blow up and be like we are, we are in a whole new wave. 18:32 This is good because now this AI has become useful to the regular person doing regular things like, you know, you could keep your kids' school calendar in check and find out, you know, when their trips are, when the soccer lessons are, any rescheduling is happening, that kind of stuff. 18:47 Everybody can use this thing. So I think it'll be much, much more useful than it has been in the past. Uh, and that's a good thing for the AI build out, right? 100%. 18:58 Um, yeah, to me, what has been really interesting, um, is really trying to understand what are the things that 19:09 I would say the broad adoption curve, because I think everybody is trying to underwrite, at least in my point of view, like how many new gigawatts are coming online? Um, is that 10 this year? Is that 20? 19:23 Is it 30 next year? Is it 40? Um, 19:27 just so whether that's pre-training to build larger and larger models or, uh, just for inference as kind of m- potentially more of the workload shifts to the inference side, maybe that's more memory bound than, uh, GPU bound, um, and trying to put all the pieces together. 19:46 Um, one thing that I personally have been trying to track at a high level is just the amount of gigawatts, just because I feel like that's kind of downstream of everything else, or upstream, so to speak, and everything else kind of falls, falls back from that. 20:01 Um, what are-- maybe if you feel like, uh, the space on the agent side is obviously increasing general adoption, but maybe not, um, [lip smack] it's not as, uh, straight up [chuckles] line, so to speak. 20:18 Um, w-what are the broad things that you are really following as like a broad trend? Because I, I think people are either in like one of two camps right now. 20:26 It's like AI's bubble, uh, the build out is insane, like this is not sustainable. Or on the flip side, it's like you're completely AGI pilled. You're like- Yeah... of course, all these gigawatts are getting built out. 20:39 Inference me- demand is insatiable. We'll never be able to get it, and then everything else in between is [chuckles] a little bit tough. Um, so I guess- Yeah... 20:49 within this maybe not exact spectrum, but, um, within kind of how you're seeing the next year or even six months, where are you kind of in line there? Yeah, that's a good question. 21:03 Uh, I think I have a long term view and a shorter term view. The long term view for me is much clearer. I think that I would not consider myself completely AGI pilled or anything like that. Um, 21:16 but I do believe AI is a genuinely useful technology that has useful implications for the long run. So in a 10 year time span, regardless of what happens, uh, it's going to be up and to the right, you know. 21:31 Um, you know, when the internet and the dot-dot-com era showed up, um, yeah, we had a crash, of course, and a lot of stuff happened. 21:39 But, you know, it was fundamental foundational technology that exists to this day and is, is a durable technology that's gonna last. Now, we could argue whether this is what LLMs are actually capable of AGI. 21:54 I don't know. People say like, "No, this, this, like, this probabilistic statistical approach of intelligence really c- is not true intelligence and something else must come by." Maybe, I don't know. 22:05 There are like far smarter people than me who know that stuff better. But given what AI can do today, I think it's a useful tool. 22:12 Like, I use it for a lot of day-to-day stuff that has unlocked so many things that is not possible otherwise, right? Um, 22:21 a- otherwise, I would-- even for what I do now between the Substack and the podcast and the institutional stuff I'm getting off the ground really, is, is too much for like one person to do. 22:32 So I do have ways in which I can streamline so many things, meetings, re- you know, schedulings. All the administrative work is going away. 22:41 So long term, I see that this is like a very useful technology that's here to stay. Shorter term, do I believe that we are overdoing it? 22:50 [chuckles] Uh, that's a difficult question to answer because every six months or so, something new comes up. 22:57 Uh, and you know, last year, if you had asked me this question like December or November last year, I'd be like, like what really... Like I-- coding agents were a good thing, right? Like, you know. 23:09 And so yeah, coding, is that all we're gonna do with this thing? Okay. What about this chatbot? This chat box gives you-- was giving you like half answers some time ago. 23:19 It wasn't-- You never had any faith in whether it was saying the right thing or hallucinating stuff. I think a lot of those concerns have gone away today. So 23:28 like it's-- So even in a six-month period, it's very hard to pr-predict where the trajectory of AI is going. Then we had agents come up. Now we have agents in our phones, in our pockets, right? 23:38 And now the next question is, what, what do those phones have to do? Like is this a game for like is edge AI finally going to be useful? 23:48 Because although your ph- device is, uh, used-- is doing some of this inference, like you're texting Instinct or Muse with your phone, but why do you have to go to the cloud every time? 23:58 What if you can do some stuff on your phone, right? That's the whole next, next step of like, what do we do with edge AI? So in the short term, if I look at the financial side, it look makes me very nervous. 24:08 I'm like, "Oh my God, look at the number of like the amount of build out." Look at Anthropic's S1 and see how much, uh, commitment they have to capital deployment and to build out compute. It's incredible. 24:20 The number is like above 500 billion, I believe. It's like insane. Like They have, they have commitments to build out that much. It's, it's scary on the financial side. 24:30 But pure- purely from a technical side, I expect that, you know, I think in a year we'll be in a much better place compared to now, even in with this technology, because we haven't even started on the adopt, like the curve of like amazing yet. 24:43 I think there's a lot more good stuff to come. So I'm optimistic on the technology side, but don't ask me the, the financial stuff is like a little too much. Are people over-ordering memory? 24:53 Are, is everybody like, you know, buying too much and we're gonna overstock and get into a glut and crash? All those things might happen, okay? I'm not ruling it out at all. 25:03 But in the meantime, I think we're, we're still going-- we still have a long way to go. So, so that's a long answer to your short question. [chuckles] No, that, that's a great answer. 25:12 In terms of kind of the, the build-out, so to speak, what things have kind of been interesting most to you? 25:21 Because it does feel like obviously the leading labs are doing or investing the most, um, in kind of the broad hyperscalers into CapEx- Yeah... I think north of a trillion dollars at this point. 25:36 Um, and obviously they're building smarter and smarter models. 25:39 Um, at least from my standpoint, it looks like as those models scale up, the clusters need to be coherent in the sense that they're co-located and, um- Yeah... 25:51 uh, we've seen, uh, some attempts, uh, to do decentralized training, and in large part, I don't think that has played out. 25:57 And so continuing to focus on kind of coherent clusters that have extremely high throughput and going, I think Elon has even talked about building a Memphis data center end of this year. 26:08 I think they're targeting 2.5 gigawatts- Mm-hmm... like 8 to 10 next year. But it does seem like as the models generally grow, um, in ter-terms of like total parameter counts, 26:22 that over time, um, and I think, uh, Jensen has said this on, uh, Brad Gerstner's podcast, he was like, "Inference is going to like a million X or a billion X." 26:32 Um, and that does seem true just given that, uh, that is the source of generally all the revenue and, uh, training is really a loss leader for inference. Um, so how are you kind of like seeing 26:47 like things on the ground floor? 26:48 'Cause I know you, you're very deep on the technical side of the different build-outs, both on, um, inference and in training, um, and kind of how that shifts the workload a little bit from more tokens or not tokens, but more maybe GPU heavy to more memory side, which obviously the me- the memory stocks have done fairly well this year. 27:11 Yeah. So the difference in training and infer-- I think training is going to continue. Um, we are still trying to build larger and larger world sizes. We want to put, you know, 144 GPUs together and scale up. 27:26 We wanna put 576 and scale up and fine men, what, you know, 1,152 GPUs and scale up. All of that will continue. What's the max right now? 72? Yeah. 27:37 Today it's 72 in a rack, but we wanna go beyond that, and we wanna put 144 in a rack, but it gets really, really difficult because even a 72 GPU rack today, the Blackwell rack is like a 200 kilowatt rack. 27:51 Now, you wanna double that, you're gonna increase a lot of the networking as well. 27:56 So even in e- realistically, if you just double the GPUs and you say you have north of 400 kilowatts in a very simple calculation, uh, that's a lot of power for a single rack. 28:08 And cooling is a concern, interconnects is a concern, you know. So a lot of things happen when you go into that, uh, number of GPUs in a single rack. Now, imagine putting 576 in a single rack or 1152 in a single rack. 28:25 It's just not possible, right? So now what you wanna do is you wanna put them in multiple racks, right? And now that's opening up so many other problems like, how do you hook them up? 28:34 Because it's not one or two fee- one or two meters anymore. Uh, to cover like eight racks, you need nearly 10 meters of reach, and copper can't do that at the current, like generation of speed at 200 gigabits per second. 28:49 Copper can't do 10 meters of reach at all, like no way. So there, that, that's the problem we are faced with now. 28:56 So we will continue to go to larger and larger domains, uh, and scale up because training is going to continue and is very important, um, as we push the frontier because we have still not seen, uh, any signs of scaling or slowing down. 29:10 But yeah, inference is the real big, I think, money maker because everybody needs inference, right? [chuckles] Like we all need a model, but do we all need a frontier model? Maybe. 29:21 You know, in the, in the large course of things, it's a good thing to have. Um, it's a longer project, like everybody needs to run research projects that give you the next best thing, right? 29:31 Like otherwise we are-- what's the progress here? Progress has to continue. So that will happen. But inference is, is gonna unlock like so much more useful stuff, right? Um, agents is just one, one use case. 29:44 Um, and the chips and the architectures are continuously diverging from what chips used to be used for training. Like now, even in the Google TPU version eight, you have a, an inference chip and a training chip. 29:57 Now, they've been doing this since, uh, version four actually, uh, but this is the first time they, I think they released them together, and they have different chip architectures. 30:06 So you could even look at the data movements in inference and the data movements in training. They're completely different. So in training, you have all these weights. 30:16 You know, I like to think of them as like waves, you know, in, in an ocean. Like you, they come in sets and they go in sets, you know. That's how data flows. 30:24 You know, you have all these operations that complete, and then there's this thing called all reduce. 30:29 So the entire wave of data comes back and that, and then the next iteration completes, and then it's a wave of data comes back. You know, that the-- it's a very known data flow movement. 30:39 But in inference it's like completely chaotic. Like it's, it's chaos. It's like looking at instead of the beach, you're looking at like, uh, the rapids, you know, the Grand Rapids or something. 30:50 You're looking at the water and going, "What the hell is happening here?" 30:52 Because- You have the mixture of experts model, and what you're doing is you're not pulling all those trillion parameters of 5 trillion parameters, whatever. 31:01 You're just pulling a few, few hu- few tens of billions and saying, "That's my expert." Now, you're going to use a different expert from me, so you think about it from a data center perspective, right? 31:11 All of these things are, like, random. It's chaotic, like, weights are being pulled in all kinds of places. 31:16 The, the, the, the right weights you need might live in the cluster over there, but then your next query might require the weights all the way in this, this end of the room, you know? Literally, that's what it is. 31:27 Like, the weights can be stored in different ends of the room that you are asking for. So when you look at the data movement patterns, it's chaotic, right? It's so-- We will see all these things, like, diverging. 31:37 So inference now unlocks, because of the nature of the problem, it unlocks so many more, um, architectures in chips that can be better suited to that, right? It's not for training. 31:51 That's why you don't see too many companies out there trying to make training chips, because those big GPU chips are great for training, right? Like, from AMD and Nvidia and all that. 32:01 Uh, but you know, when, when you wanna go to even Trainium, obviously. But you know, when you go to the inference world, it's the Wild West. You've got Cerebras, you've got Groq, you've got, like, so many approaches. 32:13 Like, if you go to Hot Chips, for example, you'll see, like, so many varieties of stuff, like Maya's inference accelerator, Microsoft Maya is totally different. Meta's MTIA is a different, like, uh, way of doing it. 32:25 Of course, OpenAI's Jalapeño itself was, like, completely different from all of these other things. So, uh, so many different things are happening in inference, it's, like, amazing. 32:35 Uh, so clearly there's a reason why everybody's focused on that [chuckles] because there are a lot of dollar va- value to be unlocked there as we go out into the future. Yeah. It's super fun. 32:47 Um, I would say in kind of my world currently within blockchains, uh, really h- as if you boil it down to kind of one key thing, it's historically been pretty throughput bound. 33:03 Um, [chuckles] and I think, uh, the interesting thing within the da- AI data centers, you're generally talking about, as you put it, at like the w- the weakest link across racks as, like, 200 gigabytes. 33:17 Um, w- when you look at some of the things on the crypto side, you, you'll laugh, it's, like, measured in megabytes. Um- [chuckles]... and I'm like, "This is so cool." 33:26 Like, we're pushing the frontier of really what's possible across, like, the physical limits of these different materials. 33:34 I mean, then obviously, like, they're scaling to terabytes and beyond, which is super interesting. 33:39 And I'm hoping crypto, um, and kind of blockchains on the back end will get to gigabytes and will keep scaling, but thus far it's taken a bit w- a little bit longer than I would like. Um, but one thing- Yeah. 33:51 It's, it's insane, right? The, the, the r- the speeds are insane. Like, most people don't even have gigabit fiber to their home, you know? Like, your whole internet, most people don't have gigabit fiber. 33:59 Like, some, you know, you could get it, I guess. It's about it. Like, think about the speeds in the data center, it's, like, insane. [chuckles] It, it is crazy. Um, yeah. The, the amount of throughput is very high. 34:11 I mean, I, I think even, like, normal data centers, um, they have, like, 10 gigabyte lines, um, and then they scale to 100. Um, I know e- like, uh, maybe a slight tangent. 34:25 Uh, I think blockchains are m- mostly the next evolution of finance, um- Mm-hmm... 34:30 a- as, like, a fintech, and it's really just connecting all these heterogeneous databases, um, and syncing information at, like, a bare level. 34:39 Um, and so the, the why it's a throughput issue is because you have these different, uh, 34:44 call 'em banks or exchanges, um, that have relatively low throughput, and the, the thing that you need to do is synchronize the data between them. Mm-hmm. 34:52 And so, uh, uh, we've been scaling from [chuckles] really k- kilobytes, which was terrible, to megabytes, and now hopefully gigabytes, and I, I think if you push it to the extreme, it kind of looks more like high-frequency trading where you get, like, 100 gigabit internet connect lines. 35:08 Yeah. You take a bunch of data and you do some, like, data parsing and then, uh, [chuckles] some, like, interesting stuff with the models on what you want to do. 35:16 But the reason why I've been super interested in the data center side of things, and why I feel like it, at least trying to, uh, learn as fast as I can on zero to one, and I, again, appreciate your help here, has been because a lot of it's, like, different throughput issues. 35:32 Um- Mm-hmm... 35:32 and I, I think o- one thing that I wanted to touch upon in terms of what you mentioned, uh, just, like, going to the scale up, and then also wanna touch upon the inference side of the things is I think, uh, y- from a rack perspective, it's also very interesting. 35:47 It's like one server in a rack has a certain amount of throughput on the chip itself. Um, and then as you go to the rack, the rack has a certain amount of throughput with things like NVLink. 35:59 Um, and then as you go from rack to rack, the throughput is much different kind of across those racks. 36:05 And so maybe in more layman's terms, can you kind of explain, like, the scale up there and why, uh, it's difficult from, like, a throughput issue, and even scaling up past the 72 GPUs, so 144, et cetera- Mm-hmm... 36:21 um, and why that has kind of been more difficult? Yeah. So it's always easy to push bits, uh, really fast if the distance is really close. 36:35 Like, if you look at the bandwidth, uh, between HBM and the GPU, right, you are at something like 22 terabytes per second, okay? That's, like, extraordinary. Uh, very fast because you c- 36:51 whenever you push, um, bits through copper- You know, the longer it stays in that copper line, the, the worse it becomes. 37:00 You wanna get from point A to point B and get rid of it because the longer s- you stay in copper, the worse the signal gets. So 37:08 the same speed will not work, um, once you go from, you know, that level of GPU to, you know, HBM, which is like very, very, very close by, right? 37:17 Like, but then when you go, uh, between two GPUs in a rack, which are like spaced, I don't know, let's say one, uh, a few feet apart, now you have to travel, you know, instead of, uh, few s- few millimeters, you have to travel a few feet. 37:33 You know? [chuckles] Uh, that's pretty bad. Uh, that's, you know, that's a big, big dimension jump. So you can't push the same speeds. There's no way, you know? So you have to drop the speed. 37:44 And so it does, it gets slower there compared to like the die-to-die link. 37:49 But, uh, the, and y- but to push the speeds anyway, you have to do all kinds of like crazy, you know, design tricks, uh, where you are like, "Oh, I'll, I'll make sure I compensate for what happens in the copper on either end so that, you know, sometimes I know like, oh, the copper's going to lose this much energy by the time it reaches there, the signal. 38:12 So what if I, if I boost the energy to begin with so that, you know, by the time it reaches there, you know, I compensate?" All these kinds of tricks, you know. 38:19 Or they'll say on the other side, like, "I'm just gonna receive the bits and then I'm gonna count them and see, like make sure I can see if some kind of error calculation, like how many of these 10 bits are wrong, and can I tell which of these 10 bits are wrong and how to correct them," right? 38:33 So that's, that's why you need a digital signal processor, a DSP, 38:38 on the other end sometimes to-- but, you know, uh, that, just so that you can com- you know, compensate for all the crappy stuff that happens on copper lines. 38:48 Uh, all this is a problem because it takes time to, for the DSP to count the bits and be like, "Oh, bit seven is wrong. Let's change this from zero to one." 38:56 Or it takes energy because you have to like blow, uh, you know, like silicon and energy on that chip that is doing all these functions. So it becomes a problem. Now, copper, 39:07 whatever you can do, like now we're reaching 200 gigabit per second. Maybe next generation is 400 gigabit per second. Um, it's run out of reach. Like the physics has reached its limits. 39:18 Like the loss of that bit, uh, b- when, when you put a, a, a single electrical signal on one end of a copper line, it's almost i- un- indistinguishable, uh, you know, uh, like five meters away at this speed. It's no use. 39:35 You can't even tell what's happening. So that's why now the industry is like, "Okay, look, copper has reached its limit at 400 gigabit, absolutely." 200 gigabit is still running. 39:45 Like the, the Rubin racks still have 200 gigabit, um, and copper. It's fine. But at the 400 gigabit, it's, it's totally dead. Even, even at 200 gigabit per lane, it's within one rack, it's okay. 39:57 But if you've got to go to the next rack, which is like two to three, two, you know, I said two meters away this side, it doesn't have enough reach. So now everybody's like, "Wait, so I can't put more GPUs in a rack? 40:11 There's no space or, you know, or I have to put 400 kilowatts in a rack?" Like that's ridiculous. It's already... By the way, in the cloud era, the data center per rack power consumption was like 20, okay? 20 kilowatts. 40:23 Now an AI rack consumes 10 times that, and now we wanna double that just like, it's, it's very hard, right? Cooling gets hard. Uh, power delivery gets hard. 40:32 So they're like, "Okay, the best interim solution while we figure out all this crazy stuff is let's just put separate racks at each other, next to each other and then hook 'em up." 40:41 Think of it like a giant rack, but like with made up of eight smaller racks, right? Um, so that's, that requires interconnects that reach eight racks worth of distance. Copper's not gonna do it. 40:56 So that is why the entire industry is like optics, optics, optics. We need to go faster. How do we hook up racks? How do we get the reach copper can't get? 41:05 And so they're like, "Wait, I can't use traditional copper, uh, like optical pluggable modules because those are just like op plugs. You can see them online. They're just like plugs that hook in. 41:16 They are energy inefficient. Like Nvidia never wanted those things or they would've gone long ago to optics. Why would you struggle with copper when the solution exists? There's too much power. 41:26 So the, the solution, the industry is like, let's move this trans- optical converter module right next to the GPU or the switch, and that's what is like co-packaged optics, right? 41:38 Like that saves the energy problem, but also solves the reach problem, but introduces other problems [chuckles] like reliability. Like when it fails, how do you change it? 41:48 It's next to the GPU, you know, so many other concerns, and it's never been done before. So can we deploy it at scale? Can we put 100,000 GPUs hooked up with this technology? These are all real questions, right? Yeah. 42:02 It is, it is super interesting. 42:04 Um, and I, I really love the physics of, of it all because again, you're pushing what is possible from these different materials and e- even as you mentioned with carpo- copper, uh, having kind of a physical limit with just how, how far you can actually, uh, send the signal to further distances and potentially needing repeaters or other things there, but that it's also much more energy efficient than something like fiber and the optical side. 42:37 Um, but optical now has much higher throughput and potentially uses or does use more energy. It's like, okay, how do we, uh, figure out the optical side? 42:48 Because we want throughput to be high, but we don't want [chuckles] the increased u- energy usages. We don't want usage, we want reach. 42:55 Uh, copper was essentially free, like you just like put energy through it, it goes through the copper wire simple. Uh, optics is like so much harder, right? 43:02 Like you got to convert electrical to optical and do all of this stuff. So it is harder, but it's become a necessity now and that's why you see so much optical content in data centers. 43:11 It's only going to get more Like optics is not-- The industry has never seen anything like this in the optics world. 43:19 In the past, optics was just like for long haul links, you know, metro links, uh, undersea cables, some applications, whatever. Um, but never has it taken a prominence like it has today. Today it is- So you-... 43:34 the technology that is required for AI and the future of the build-out, and the entire world is scrambling for it because they're like: Look, we were never prepared for this demand. 43:45 Like, this industry has never been built for this level of demand. You are asking for too much. So everybody's in short supply because of this. 43:52 [chuckles] So you're telling me, Vic, I'm gonna soon have, uh, one terabyte internet connections to my house, uh, thanks to data centers? No, they'll never give it to you. You don't need that. 44:01 [laughs] Only the data centers need it. You don't need it. [laughs] Yeah. [laughs] We're gonna upload this podcast and- Unless you're running a training, unless you're running training at home. [laughs] Maybe. Maybe. 44:10 Yeah. Um, that's funny. Um, yeah. It, it, it is super interesting. I... Yeah, the, 44:16 it, I again, I, I think it-- the coolest thing that I have found is that because the demand is so high and people's willingness to push these has really gotten to the physical limits of the material, it's very exciting because then you have to come up with clever, new, different engineering solutions to make these things all work. 44:38 Yeah. Um, so maybe n- touching upon the training thing a little bit more on the maybe interconnects, and then we can jump gears to the inference side of things, because I do think that's super interesting as well, is 44:51 as you're starting to scale up these interconnects, um, and, uh, it seems like people want to almost have, like, one giant GPU, if possible, and that- Yes... 45:04 obviously, if you couldn't have, um, the throughput issues, uh, that would be- Yeah... ideal, but there's- [chuckles] Correct... kind of a you're, you're only as fast as your slowest link, um- Yes... 45:14 within the data center. But we are starting to scale, um, and have to one gigawatt clusters and now two gigawatts. And again, I, Elon, I think, uh, is pushing 10. 45:26 Now, TBD, how, how much of that's going to be interconnected or not versus kind of the old H100s versus new V300s, et cetera. But the direction seems clear. 45:37 So as you start to scale up, um, more and more GPUs and more racks, what is the thing that you've kind of been following there? Is it the interconnects on, um, across racks, um, things like Lumentum, Coherent, et cetera? 45:55 Or are there other things that you think are more interesting to kind of follow along, um, as the scale-up continues? Um, 46:06 no, I think the biggest, most pressing problem today is the interconnect problem, and that's why you see these other interconnect technologies also show up, uh, called wide and slow, which means, you know, instead of running eight links of 200 gigs each to get 1.6 terabit per second, you can run, I don't know, 50 gig, but, you know, uh, many more of them. 46:30 How many is that? Yeah, like 35 of them, right? Something like that. 46:36 Um, so you can run a, a fewer number of, um, fewer data rates, but more number of lanes, like putting more lanes on a highway rather than making the- Is that how they're doing it today? No. 46:46 So that's the big- Instead of, uh, like a one- That's the big debate now. So we are like- Okay. Interesting... we are in this phase where, like, you know, laser technology and optics, when you're doing 200 gigs per lane, 46:58 uh, normally they, they bundle like eight lanes together, and you get 1.6 terabit per second, uh, per, per link. Um, so- So it's not a 1.6 terabyte link. They're sharding it across those eight. Yeah. 47:11 It, there are like eight of them that run one point six-- uh, 200 gig, gigabit, and that's one point six terabit per second. So yeah. So essentially, what happens is that, um, you need specific kind of optics for that. 47:26 You need lasers specifically made out of this indium phosphide material, which is perfect for lasers, and that's been the mainstay of the industry forever. 47:35 And the problem is, like I mentioned, nobody has this kind of capacity. So people are, like, asking: Look, lasers, those kinds of lasers can connect continents, okay? 47:45 Like, that's how continents under subsea cables work on this technology. This kind of laser can hook up data centers, uh, you know, kilometers apart, tens of kilometers, or even under the o- ocean. 47:58 Why do we need that to hook up like a neighboring rack? Like, why? Don't you think it's overkill? We don't have supply. Why don't we do something else that is not copper? Yeah, we know copper doesn't work. 48:09 Why do we need to go to this, like, Ferrari technology that, like, is useful for something else completely? And why don't we just use something else? 48:17 Like, you know, so that's where, in a sense, we are kind of going back in time because we are going back to these specific kinds of lasers called vertical cavity surface emitting lasers, VCSEL. Right? 48:32 And you'll hear most people, like, shorten that to VCSEL. And those kinds of lasers are, like, kind of before, you know, doing what, like, we've been doing these days. 48:43 And they-- You can make those kinds of lasers even out of like gallium arsenide, which is actually far more available than indium phosphide. So you could make a gas VCSEL, right? 48:54 And so the supply chain is available for something like that. And so people are like: Why don't we use that material instead? Problem is that it doesn't run at 200 gigabits per second. 49:03 It only runs at like, I don't know, 20 or 50, right? And then people are like: Okay, just, just put more of them together. Like, why, why are you fighting this? 49:11 Like, just, yeah, put, put more bundle, more cables together and you're gonna get the speed back. Why don't we just use that technology because we are not restricted by supply chains. 49:20 We know this technology works And then there's another contender, like LEDs. So people are like, "Okay, LED is another light source that can do optics, so why don't you use that instead?" 49:31 But the problem with m- uh, micro LEDs is that, uh, they're even slower. You can't run them past, like, five gigabit per second or three gigabit per second, something like that, right? That's the limit. 49:42 And so it's like you have to bundle even more to get the speed now. So people are like, "No, no, that's a, that's a terrible approach because look-" [chuckles] "... you have to put hundreds together to get the speed. 49:49 Like, why would you do hundreds if you can put, like, tens," right? That, that's what VCSELs give you. You can just put 50 gig times, like, 35, 36, and then you're good, you know? 50:00 Uh, so that's, that's where the industry's like, there's a lot of debate now on what is the best way to hook up, you know, optics in the scale-up domain. 50:09 Do we really need this Ferrari technology or is there something else we can use? Yeah. So everybody's fighting about this. I don't know the right answer. That's very interesting. Um- [chuckles]... 50:18 I, I thought they had a 1.6, uh, uh, gigabit line. I didn't know that they're doing the sharding across the eight or so lines. Um, but that's very interesting. 50:29 Um- Yeah, because, uh, LEDs are also, like, based on gallium nitride- Mm-hmm... which is, like, another nicely available material because you can even build, uh, you know, gallium nitride on silicon substrates. 50:40 You can make, like- Mm... really big wafers, um, of this stuff like silicon, and you can make a lot of them. It's kind of nice to make. It doesn't require this special material, indium phosphide. 50:51 Problem with indium phosphide is that, you know, it's got, it, it's made on a f- three or four-inch wafer. I don't know if you know this, like, this, this pastry called, like, a stroopwafel. 51:00 Like, it's just, like, a small pancake, you know? [chuckles] That's the size of these wafers. It's like a really tiny pancake. And you've got to make lasers out of that. 51:08 Like, you know, so how many-- You're supply constrained because you can't even make these Ferrari lasers on big honking wafers. [chuckles] So it's a problem. So we need to be more bullish pancakes. Huh? 51:21 We need to be more bullish pancakes. Yes, exactly. It's not easy to make these pancakes bigger too, because they don't provide enough yield. You know, you can't make enough- Right... of them that work. 51:30 You know, so many problems. So they're like, "You know what? Forget it. We'll just go do something else," you know? Mm-hmm. 51:34 So the whole industry is in, like, a big turmoil of what are we gonna do here for this interconnect problem? So yeah, that's like- Very interesting... a big thing right now. 51:42 So to answer your original question of s- what happening in scale-up domain, this, this interconnective connectivity problem is, like, the biggest, uh, biggest thing going on. 51:54 The next thing I would believe is, like, the power aspect. Like, like how are we going to- Yeah... deliver more and more power to these things? That's a long-term problem, and that's a massive- Yeah... 52:03 and super interesting problem because it starts all the way from power generation. 52:07 You could talk all the way from nuclear power plants to how it's delivered to the GPU, and there are, like, infinite stuff in between, like from inverters and UPSes to s- energy storage and capacitors and oh, it's insane. 52:20 Like, the power chain is just insane. [chuckles] I, I wanna come back to power, um- Yes... even talk about, uh, space data centers. But I, I wanna- Yeah... 52:29 talk real quickly a- about the inference side of things, just given that that does seem like it is an area of focus, as you mentioned, and more kind of like the Wild, Wild West. 52:42 And even if you just look at the equities markets, I think there's been a lot of questions about people trying to go zero to one on the memory side, just because they have been some of the best performing, um, stocks this year. 52:56 Mm-hmm. 52:57 Um, and so I would love to kind of pick your brain on kind of the memory hierarchy that's starting to form with SRAM, with Groq and Cerebras, and then obviously high bandwidth memory, potential new tiers with high bandwidth flash and- Mm. 53:13 NAND, um, uh- So interesting... 53:16 and just how you're kind of seeing that memory hierarchy, um, kind of unfold as you have extremely high throughput and hundreds of terabytes r- at the top with SRAM versus the limited capacity that it has as you kind of go down that hierarchy. 53:35 You have much more capacity, but limited throughput and kind of where things live across that stack. Yeah. Um, this memory hierarchy used to be simple. It used to be SRAM- [chuckles]... in the CPU- Yep... 53:49 [chuckles] in the CPU times. It used to be SRAM, where you hold your L1, L2 cache. Then you have DRAM, pretty simple. Like, everybody bought DRAM sticks for their gaming PCs and put it in there. 54:01 It was pretty s- straightforward. And then you had storage, you know. And then SSD storage is one kind. Then there's, like, hard disk drive storage is another kind. 54:10 And you could talk about an even, like, s- uh, bigger archival type called tape, you know. So you could store everything to tape, you know. 54:18 That is actually something that is very little talked about, very little in the AI data center build-out. But if everybody wants to store everything all the time, the only way to do it is go to tape. 54:30 You know, the arc- It's very slow to pull everything from there. It's like the sl- lowest bandwidth you can ever imagine, but capacity is near infinite, right? And cost is very low. 54:40 So tape storage is the o- other extreme. It used to be simple. But now you have SRAM. Uh, then, you know, Cerebras wants to put DRAM on top of SRAM, which is what, um, you know... 54:53 And then Qualcomm wants to put DRAM on top of logic. And then you've got, like, even in that memory, you've now, you've got, like, high bandwidth memory, 3D stacked memory, all of this stuff. 55:04 Then in DRAM, um, that's the DRAM tier, right? Like, you know, where you put DRAM and HBM or not. 55:11 Then when you go into flash storage, it's like another explosion now because you've got all kinds of high performance flash where the performance is tweaked towards for, you know, higher speed vers- and by sacrificing capacity, like single-layer cell NAND. 55:26 Or you can go to, like, more capacity side, which is like quad-level cell, QLC NAND. All of these things exist. 55:33 And even in this, there's like a whole lot of continuum, because you could have a single level cell cache with a triple level cell like storage. 55:41 So you could have, you know, various kinds of mix and match, and the performance is optimized all over the place in all kinds of fashions just within the NAND tier, right? 55:51 And then you've got like the hard disk and the tape still existing. Now the question is, okay, how do you use all this stuff? Like what do you do in inference? Like how do you decide? And, um, that's where I think lot of 56:07 work is still to be done in deciding how to match the s- memory hierarchy to the workload that you're running. Like do you always need the highest throughput? 56:18 Like right now, I don't think we-- everybody is like bragging throughputs, just like everybody used to brag about flops at some point in time some time ago. Now you don't see anybody bragging about flops. 56:27 Everybody's talking about like, "Oh, we can get 2,000 tokens per second," or, "We can get like, uh, 1,000 tokens per second." So it's all about speed. 56:35 But then at the end of the day, it's always like, um, what is the useful amount of work done per token? Like do I always need 2,000 tokens per second? I don't. 56:46 Like, you know, I'm talking to you right now, but my AI could be crunching some PDFs that I wanna read later or something like that. I'm, I don't have any immediate need for it. 56:54 So eventually it should use the memory hierarchy in a way that is best suited for the job it's going to do, because it'll control my cost, therefore my usage of the system. 57:06 I'll use it more if I can keep the costs controlled, right? As a enterprise or a consumer, um, I want more done, more work done per token. So run the tokens, run many tokens, but run them slow for the right workloads. 57:20 Run them really fast if I'm doing certain kinds of workloads. Um, if, if I want something coding done really quickly for an enterprise, time is money. 57:27 They wanna make a product faster than their competitor, and so they're willing to burn more money in tokens because they can recover that in being first to market, so it makes business sense for them. 57:37 So that use case requires faster speeds. So I think we will see the memory hierarchy like, uh, be utilized throughout to match the token usage to the best useful work it can do. 57:47 I think there's a lot of optimization done, to be done there. I fully agree. And one thing, uh, w- with my recent podcast with Bubbleboy, I was like, how does potentially the agent's workload change things? 58:03 Because it seems like the industry has very much optimized for throughput, which I guess makes sense if you're charging per token. Uh, the more tokens per second, [chuckles] the more they can charge you. Yeah. 58:15 But on the flip side, going back to the agent point and the point that you made earlier about us chit-chatting and potentially your agent can go out and do things, it doesn't seem like they need the max throughput. 58:27 Um, and to me, what would be interesting is if they could have more contacts, um, which seems like it'd be like you want a larger KB cache, you forget less things over time. 58:38 Um, maybe that goes from 1 million to 10 million and I was-- I've been, been waiting for kind of a [chuckles] context windows to increase, but, uh- No, I don't think it remembers-... at this point, which I think- 58:50 Yeah, my point about that is that I don't think it needs to remember everything about me all the time. Like, you know, when the context comes up, it can go look up what, what that part of what it knows about me, right? 59:02 Like if I'm working out and I want some AI thing to tell me like, "Hey, um, you know, how much, how many steps do I have this week?" 59:10 Or whatever, it can go and look up my call re- my step records from my fitness tracker or whatever it is. That much is enough for it to store in memory. It doesn't have to remember anything about my work. 59:20 [chuckles] It doesn't have to reference like optics and power and data centers when it comes to like fitness stuff. So I think they can be broken up, like context even for a person has sections, right? 59:30 Like you don't remember everything all the time. So I think it can be broken up as a problem. So h- how do you think that looks under the hood? 59:39 Because one of my assumptions, and this could be extremely wrong, was that context windows should grow over time, um, as you want your agent to remember more, just more specific workloads. 59:51 Um, I saw this podcast with Dario, who was like, "We'll eventually get to like maybe 100 million context windows, and at that point you can do like r- recursive, um, learning on the context window itself and maybe bake the most important things back into the weights." 1:00:06 I was like- Hmm... "Okay, cool. Let, let's, uh, push towards that direction." But- Yeah, sure. [chuckles]... it seems, it, it seems that- I'm up for it, actually... 1:00:15 it, it seems that a lot has been really optimized, to your point, on [chuckles] 1:00:21 if one user needed a certain amount of hardware, um, it would take even, uh, or maybe like a 10 million context window and I'm just kind of making up these numbers. 1:00:31 You could do 10 people on a lesser model that could serve potentially more inference than having like one super user. 1:00:39 And so people, and I think rightfully so, are starting to optimize m- much more on the number of total users and getting the hands into kind of technology than like the extreme power user, um, at least for the moment. 1:00:53 Yeah, exactly. I, I, I don't think we still know the answer to how much context is enough context. Can we-- We-- I don't think we can answer that. 1:01:03 It seems like more is always better, but at what point does it stop being useful? Like everything I naturally in nature seems to have a diminishing return. 1:01:15 So at what point do we hit a diminishing context window, um, you know, return, return on the context window? I don't think we know yet, I think. Um, but we'll know. We'll, we'll find this out soon enough. 1:01:25 That's why I say we are still early in this thing. There's so much still unknown about this that we're still quite early. What's your guess? I think about how I think, right? Like, what is my context window? 1:01:39 Uh, i- for some things it is a lot, right? Um, I, I remember, uh, you know, my, my kids' entire lives, like, not, I mean, not every moment, but I can date back to when they were born, right? That's still in my brain. 1:01:54 But I can't remember what I had for breakfast yesterday. So that's, [chuckles] you know. But- [chuckles] So what context is useful context? Not all context is useful. 1:02:02 So what is the best kind of context that you should retain? Not everything, right? So that remains, that puts in, uh, what is the useful limit of context that is, uh, diminishing value here. 1:02:13 Because I don't wanna forget, like, my kids' younger days and all. That's, like, joyful to me, so I remember it. But I really don't care, like, what I had for dinner two nights ago. It's fine. 1:02:23 Like, if it was a nice meal, I'd probably remember it, but otherwise, I don't know. [chuckles] So it's- Yeah... it's totally fine, right? Uh, that, that kind of information- Very true... need not be stored. 1:02:31 So it's all about what context do you want to retain? What is important? 1:02:35 And I, I think, uh, you wrote a great article about this as well with, uh, uh, DeepSeek, um, and some of the unique techniques that they've been doing with kind of KB caching. As you hit the cache, the, um, 1:02:50 essentially s- mount that they're spending because you're hitting the cache instead of doing, like, more prefill is dramatically less. 1:02:58 Um, which to me was super interesting as trying to optimize as many cache hits as possible, so you can either potentially pass on the savings to the consumer or just increase your margins, um, because [chuckles] it costs a lot less than doing expensive prefills. 1:03:15 Yeah. That's the equivalent of writing it down in a notebook. You know, just like, "This is, this is my NAND storage." You know? 1:03:22 It's, it takes longer to retrieve from my notebook, but you know, it, it has much more context than I can hold in my brain. So [chuckles] it's, it's- Yeah... it's a useful technique. 1:03:31 So yeah, DeepSeek is actually writing stuff to a notebook. Uh- So s- s- specifically there, and maybe last thing on the memory hierarchy, and, um, I appreciate your time, sir. 1:03:42 If we have to jump as well, let me know, and we can wrap it up. Um, on, on the kind of memory hierarchy, is there one part, part of the hierarchy that you're more or less excited? 1:03:52 Is, are you really interested in SRAM and kind of the things that they're doing there? Is it the networking within, like, high bandwidth memory to have larger weights? 1:04:03 Um, is it NAND and kind of the notebook, so to speak, um, as you kind of reference? Um, is there any part of the stack that you're most interested in, or m- maybe that the field is really doubling down on? Uh, DRAM. 1:04:19 DRAM is the, the best, uh, area to focus on because HBM is required. You can't not do without HBM at the current moment. 1:04:28 Like, the technology-wise, that's the most important layer because it gives you the best balance between bandwidth and, uh, capacity. Um, but it, you know, stacking is becoming a problem. It's getting too expensive, 1:04:41 and it's becoming of questionable value going forward. Um, so how, how high can you stack? We're already at 16. Are we gonna go to 20? What about that? What's next, 24? It's kind of getting illogical at some point. 1:04:52 Maybe we keep stacking stuff. So I think one useful way to do this is to put memory on top of logic, and you can see many companies doing this. 1:05:02 So Cerebras is going to bond an entire DRAM wafer on top of their, uh, s- wafer scale engine. So you'll have DRAM on top of SRAM, uh, completely bonded wafer to wafer. 1:05:15 And then you will have, um, I think Groq is going to have a version of this as well. And then, um, you will see Dmatrix is already having their Raptor engine, which has a logic on top of memory. 1:05:28 And then Qualcomm is also announcing, like, high bandwidth compute, which is putting, uh, I think two or four levels of stacks of DRAM on top of logic. 1:05:38 So now what you do is suddenly you expose the entire area of the chip for interconnects, not just the edge of the chip, right? 1:05:46 So now you have so much more bandwidth, you get, like, SRAM-like bandwidth, but with DRAM-like capacity. So it's a, that's a big, important area to work on. That's super interesting. Um, if you were, uh, 1:06:00 interested on maybe the investing side, on the private side of things, um, are there any companies, um, as you mentioned, on the inference side, it, it is kind of a wild, wild west. 1:06:13 People are experimenting with lots of different things as the memory hierarchy. We're trying to see where things best fit. Uh, we're trying to figure out the different stacking of things, um, the interconnects. 1:06:27 And a- as you continue to scale these things up, are there any companies that you have found uniquely interesting on that side of the things? I know companies like Etched just raised a big round. 1:06:39 Um, are there any others? Weka is also- Yeah... I know you guys have had them on the podcast. They're super interesting in how they do the data sharding. 1:06:48 Um, any companies that you have kind of found interesting that you think are worthwhile to, uh, double-click on? Uh, there are so many of them. Um, that's the whole-- 1:07:01 The tracking all these, like, private companies doing, like, startups and stuff, doing inference is really fun. Like, um, I like the 3D DRAM approach, which is why I mentioned Dmatrix is one of them. 1:07:13 Um, I have actually looked at, uh, SambaNova in the past. I think their architecture gives you a, a different kind of optimization. 1:07:20 It's more of a TCO-based optimization, uh, where they can deliver for a different end user, so it's always not about max performance. I think they have a different, uh, customer profile in mind for their hardware. 1:07:32 Etched is very interesting. Um, although I don't really Have too much understanding about their low voltage inference architecture, but it seems super fun. I saw their hardware at Hot Chips. It was really cool. 1:07:43 Um, I'd be really excited to see what, you know, comes out of Etched. 1:07:48 And then we have, like, Madex, we have Fractile, we have so many, like, inference companies all doing slightly different stuff, which is why I was like, "This is the Wild West of inference chips." It's really fun. 1:07:57 On the laser side, you've got companies doing, like, fancy, like, lasers so that we can now hook up chip-to-chip interconnects with optics instead, because why use copper ever, right? 1:08:08 So there's so much interesting stuff across the space. Uh, the company doing that, uh, one of them is called Encoden, um, which is like, has these lasers they call nano lasers for scale-in. 1:08:18 So I find that really interesting. Yeah, so there are, like, lot of different things that are going on. So maybe if I could like, uh, push for an answer in terms of like what things you find the most uniquely inciting. 1:08:34 Is, is it the interconnects? Do you think it's like continuous innovations on the memory side in terms of stacking? Or I guess, like, if you could like, 1:08:44 uh, close your eyes and, like, fast-forward five years, like, what do you think are the most pressing engineering issues kind of across the data center stack as we continue to scale up? Um, 1:08:58 the memory is the biggest problem. There is literally-- The LLMs are a memory-heavy, uh, method, and memory is everything. 1:09:09 So one of the biggest problems that people should solve is like, how do you get the same performance without using as much memory? You know, once you solve that problem, a lot of things fall into place, right? 1:09:20 Because, uh, if you can-- if you don't have the memory limitation and the memory, then probably that if, if you don't have to move between memory so much, then your interconnect problem gets easier. 1:09:32 Like, you know, maybe you don't have memory bandwidth issues because of the way it computes everything. 'Cause right now we are leaving so much compute on the table. 1:09:41 GPUs have flops that we are not using because [chuckles] networking is not getting data there fast enough. 1:09:47 But if we can come up with a way to do LLMs that aren't memory intensive, that would be very, very interesting because that would remove the memory bottleneck and as a result, the memory bandwidth bottleneck, as a result, the networking bottleneck, you know. 1:10:01 So the real transformational change happens at the top of the stack because it's the-- it's this bunch of like esoteric LLM equations that drive this entire AI industry, right? 1:10:12 The, the attention is all you need paper from like 2017. Those equations are what we are implementing in hardware. Now, five years from now, let's say some bunch of really, really smart people at some of these 1:10:24 universities or like companies, uh, research labs come up with saying, "Hey, why don't we do these equations instead? We can do the same thing, but without all of this craziness." You know, that would change everything. 1:10:36 [chuckles] That- Yeah.... that, that is the real, like, game changer that we should really be looking out for. 1:10:43 And every time something pops up like Turbo Quant or something like this, everybody's like, "Oh my God, memory stocks are hit," right? Everybody's afraid of memory. So yeah, that's something to watch for. 1:10:54 [chuckles] And then, uh, we'll just run back Jevons paradox again, increase the total, uh, usage, and then we'll be back to square one. Exactly, right. I don't know when this Jevons paradox will stop being a thing. 1:11:07 Like, so far every- Yes... anytime something improvement happens, they're like, "We are back to square one. Sorry, we still have a shortage of compute. We still have this." So 1:11:15 I think it's an useful, uh, exercise for anybody listening to this. I would like to do it myself. Uh, but, uh, the, the exercise is like, go find a time in history where Jevons paradox did not work as a challenge. 1:11:31 You know, um, your listeners can write back to you and let you know, like, "Hey, I found one." You know, "This is where Jevons paradox- Sure... did not work, and this is why it didn't work." 1:11:40 Like, I would like that case because that'll tell us when this thing stops being a, uh, you know, consuming all our compute all the time. 1:11:48 So I guess, uh, moral of the story of this podcast is the show goes on, uh, and deep dive into memory. 1:11:55 Uh, try to, uh, figure out, uh, ways that we can use memory more efficient, but, uh, it seems like memory is still the bottleneck. Memory is at the heart of all AI inference and training. 1:12:07 [chuckles] There's no way to change that, so. Yeah. Perfect. Well, I, I want to, uh, be respectful of your time, Vivek. Um, so thank you so much for coming on the pod. 1:12:17 I feel like we could keep chatting, uh, another hour about all these things. 1:12:21 And a- again, I really appreciate your clarity of thought here because I, I think that is really the hallmark of an expert is to explain complex things simply, and you do that beautifully. 1:12:32 So, uh, I know the listeners are definitely going to enjoy this podcast. Again, uh, I'll li- link all your info in the show notes, and please reach out to Vivek if you are interested in diving deeper into any of these. 1:12:44 He has great research reports, so thank you again. Awesome. Thanks, Logan. Really fun being on the pod. Thank you. Thank you for having me.