We're sunsetting PodQuest on 2025-07-28. Thank you for your support!
Export Podcast Subscriptions
cover of episode Why Your GPUs Only Run at 10%! - CentML CEO Explains

Why Your GPUs Only Run at 10%! - CentML CEO Explains

2024/11/13
logo of podcast Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)
AI Deep Dive AI Chapters Transcript
People
G
Gennady Pekhimenko
Topics
Gennady Pekhimenko在访谈中主要讨论了AI系统优化和企业实施中的挑战，特别是GPU利用率低下的问题。他指出，许多公司只能达到10%的GPU效率，这其中存在许多原因，例如软件栈的低效、模型与硬件的不匹配以及对硬件资源的浪费等。他介绍了CentML公司致力于解决这些问题，通过优化机器学习工作负载（训练和推理），使其易于使用、廉价且高效。他认为，开源模型的快速发展正在缩小与专有模型之间的差距，这对于企业级AI应用的广泛采用至关重要。他还讨论了团队建设、组织结构、AI模型的推理能力、以及AI系统可靠性等问题。他认为，未来的发展方向是构建基于基础模型的应用，而非仅仅是新的基础模型，并且需要构建能够可靠地运行复杂系统的基础设施。他强调了成本效益的重要性，认为企业应该选择最经济高效的模型和解决方案。他还谈到了与云提供商的合作，以及如何利用机器学习编译器等技术来提高计算效率。最后，他还谈到了MLPerf基准测试的重要性，以及学术界和工业界在AI研究中的合作。 Gennady Pekhimenko还详细解释了“暗硅”的概念，即由于功耗限制而无法充分利用的芯片计算资源。他指出，现代芯片拥有大量的晶体管，但由于功耗限制，无法同时运行所有晶体管。这需要更智能的硬件利用方式，例如降低频率或动态分配资源。他认为，充分利用GPU的计算能力需要考虑功耗和散热限制，并介绍了CentML公司在该领域的一些技术突破，例如同时运行训练和推理工作负载。他认为，Python等高级语言虽然易于使用，但效率低下，而C++等低级语言虽然效率高，但难以使用。因此，需要开发更自动化、更智能的编译器来优化模型和硬件之间的匹配。他认为，未来的AI系统将更加复杂，需要能够可靠地运行复杂的系统，例如由多个Agent和模型组成的系统。他还强调了监控和调试的重要性，认为需要构建能够监控和调试复杂AI系统的工具。

Deep Dive

Chapters
Open-source models are rapidly improving, closing the gap with proprietary models. This wider access benefits developers and the broader AI community, fostering innovation and value creation. Building sophisticated systems on top of these models is now a critical focus.
  • Open-source models are rapidly improving and closing the gap with proprietary models.
  • Wider access to models benefits the AI community and fosters innovation.
  • Focus is shifting towards building applications and sophisticated systems on top of existing models.

Shownotes Transcript

Prof. Gennady Pekhimenko (CEO of CentML, UofT) joins us in this sponsored episode to dive deep into AI system optimization and enterprise implementation. From NVIDIA's technical leadership model to the rise of open-source AI, Pekhimenko shares insights on bridging the gap between academic research and industrial applications. Learn about "dark silicon," GPU utilization challenges in ML workloads, and how modern enterprises can optimize their AI infrastructure. The conversation explores why some companies achieve only 10% GPU efficiency and practical solutions for improving AI system performance. A must-watch for anyone interested in the technical foundations of enterprise AI and hardware optimization.

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. Cheaper, faster, no commitments, pay as you go, scale massively, simple to setup. Check it out!

https://centml.ai/pricing/

SPONSOR MESSAGES:

MLST is also sponsored by Tufa AI Labs - https://tufalabs.ai/

They are hiring cracked ML engineers/researchers to work on ARC and build AGI!

SHOWNOTES (diarised transcript, TOC, references, summary, best quotes etc)

https://www.dropbox.com/scl/fi/w9kbpso7fawtm286kkp6j/Gennady.pdf?rlkey=aqjqmncx3kjnatk2il1gbgknk&st=2a9mccj8&dl=0

TOC:

  1. AI Strategy and Leadership

[00:00:00] 1.1 Technical Leadership and Corporate Structure

[00:09:55] 1.2 Open Source vs Proprietary AI Models

[00:16:04] 1.3 Hardware and System Architecture Challenges

[00:23:37] 1.4 Enterprise AI Implementation and Optimization

[00:35:30] 1.5 AI Reasoning Capabilities and Limitations

  1. AI System Development

[00:38:45] 2.1 Computational and Cognitive Limitations of AI Systems

[00:42:40] 2.2 Human-LLM Communication Adaptation and Patterns

[00:46:18] 2.3 AI-Assisted Software Development Challenges

[00:47:55] 2.4 Future of Software Engineering Careers in AI Era

[00:49:49] 2.5 Enterprise AI Adoption Challenges and Implementation

  1. ML Infrastructure Optimization

[00:54:41] 3.1 MLOps Evolution and Platform Centralization

[00:55:43] 3.2 Hardware Optimization and Performance Constraints

[01:05:24] 3.3 ML Compiler Optimization and Python Performance

[01:15:57] 3.4 Enterprise ML Deployment and Cloud Provider Partnerships

  1. Distributed AI Architecture

[01:27:05] 4.1 Multi-Cloud ML Infrastructure and Optimization

[01:29:45] 4.2 AI Agent Systems and Production Readiness

[01:32:00] 4.3 RAG Implementation and Fine-Tuning Considerations

[01:33:45] 4.4 Distributed AI Systems Architecture and Ray Framework

  1. AI Industry Standards and Research

[01:37:55] 5.1 Origins and Evolution of MLPerf Benchmarking

[01:43:15] 5.2 MLPerf Methodology and Industry Impact

[01:50:17] 5.3 Academic Research vs Industry Implementation in AI

[01:58:59] 5.4 AI Research History and Safety Concerns