Hot Chips 2020 Live Blog: Cerebras WSE Programming (3:00pm PT)
by Dr. Ian Cutress on August 18, 2020 6:00 PM EST- Posted in
- AI
- Live Blog
- Cerebras
- Wafer Scale
- CS-1
- Hot Chips 32
- WSE
06:03PM EDT - Cerebras did the wafer scale - a single chip the size of a wafer
06:03PM EDT - Here's WSE1
06:04PM EDT - Uses standard tools like TensorFlow and pyTorch with Cerebras compiler
06:04PM EDT - CS-1 fits in standard 15U rack
06:04PM EDT - 400k cores
06:04PM EDT - (Costs a few $mil each)
06:05PM EDT - No DRAM, full on-chip SRAM
06:05PM EDT - 3D mesh network
06:06PM EDT - Allows all 400k cores to work on the same problem
06:06PM EDT - Linear perf scaling
06:07PM EDT - Cerebras graph compiler
06:07PM EDT - extract compute graph, create a graph in the WSE format, route kernals on the fabric, then create executable
06:09PM EDT - Graph matching for matrix multiply loops
06:09PM EDT - Supports hand optimized kernels
06:10PM EDT - Model-parallel and data-parallel optimization with the compiler
06:11PM EDT - Trade-off as resources vs compute for each kernel
06:12PM EDT - All kernels can be resized as needed
06:12PM EDT - All functionally identical
06:13PM EDT - Global optimization function to maximize throughput and utilization
06:14PM EDT - 3 key benefits
06:15PM EDT - Flexible parallelism
06:16PM EDT - Enough Fabric performance to connect everything at scale
06:16PM EDT - Otherwise slow across GPUs or a cluster
06:16PM EDT - Small batch size has super high utilization
06:16PM EDT - no weight sync overhead
06:19PM EDT - Core is designed for sparsity
06:20PM EDT - Intrinsic sparsity harvesting
06:20PM EDT - filters out all zeros
06:23PM EDT - ML user has full control over full range of sparse techniques
06:23PM EDT - WSE is MIMD, each core can be independent
06:24PM EDT - True variable sequence length support
06:24PM EDT - No padding required
06:24PM EDT - Higher utilization for irregular models
06:25PM EDT - Dynamic depth networks
06:26PM EDT - only need to process exact lengths of sequences
06:26PM EDT - World's most powerful AI computer
06:27PM EDT - Complete flexibility due to the size of the wafer scale engine
06:28PM EDT - Working in the lab today
06:29PM EDT - More info later this year
06:29PM EDT - Q&A team
06:29PM EDT - time*
06:31PM EDT - Q: What's the main benefit of WSE? A: Bypassing the issues with workloads that need multiple GPU/TPU/DPU. Opens up novel techniques that wouldn't run on traditional hardware at any sense of speed
06:32PM EDT - Q: How to feed the beast? A: Traditional server might be the GPU/TPU, so the next one is IO. We are a system company, our product is the full system, as we control all aspects of the system. We have a 1.2 Tb/s ethernet interconnect to feed the engine to keep up with the compute
06:34PM EDT - Q: How long does it take a compile a model over 400k units? A: It's an algorithmically complex search space problem. Annealing and heuristics bring that down - we are borrowing many ideas from the EDA industry. Our problem is simpler than billions of LEs on FPGAs, so we're in the minutes.
06:34PM EDT - That's a wrap. Next talk is 4096 RISC-V chip
Facebook
Twitter
RSS
36 Comments
View All Comments
jamessw - Monday, September 21, 2020 - link
I will definitely share your post on my https://www.bestessays.com/edu_college.php website. My audience will be happy to find this info. Thanks! ReplyHomework Help - Monday, September 28, 2020 - link
If you have any problem with writing your assignment, you don’t have to worry because we understand the importance of assignments in your academics. We understand how important it is for you to get good grades/marks in your assignments. Friends, you are just a click away from getting better grades in your assignment. Our assignment help service with its experienced and expert provide all sorts of assignment help to the students. Our assignment help services are affordable and you can get the maximum value for your money.Our services are available 24/7. Talk to us any time. We guarantee you quality and 100% satisfaction.
Visit us Now !!!******************
<a href="http://helpinhomework.org/blog/how-to-write-an-ess... to write a good essay</a> Reply
Alex2131 - Thursday, November 26, 2020 - link
Abdulla Thani Techical Services L.L.C ranks high amongst < a href=''http://technicalservice.ae/services/fitout-work/ "> list of interior fitout companies in Dubai </a>. Our well-experienced team of professionals has the skills to give you the best Fit-out services. By making use of high-quality material and available resources we are guaranteed to provide you the most satisfactory results. ReplyIntegral23 - Thursday, November 26, 2020 - link
A leading private investigation company, Lipstick Investigations famous for its investigation services. We have a wealth of experience in matters including cheating partners and infidelity investigations. We specialize in family law investigations, find my logical mother, and other missing person investigations.https://lipstickinvestigations.com.au/company-prof... Reply
Integral23 - Thursday, November 26, 2020 - link
Integral Investigations offers a range of different investigation services. Our operators can provide comprehensive services for: Missing Persons, Surveillance, De-bugging, Infidelity, <a href="https://integralinvestigations.com.au/services/&qu... Follow my partner</a> , Background and Asset Checks, Family Law and Custody Investigations, Forensic Telephone and Computer Investigations, Undercover, Fraud, Liability, Criminal, and Unsolved Cases. ReplyIntegral23 - Thursday, November 26, 2020 - link
Integral Investigations offers a range of different investigation services. Our operators can provide comprehensive services for: Missing Persons, Surveillance, De-bugging, Infidelity, Follow my partner, Background and Asset Checks, Family Law and Custody Investigations, Forensic Telephone and Computer Investigations, Undercover, Fraud, Liability, Criminal, and Unsolved Cases. They also have a website https://integralinvestigations.com.au/services/ ReplyGoodasnew - Saturday, December 12, 2020 - link
Good As New Pressure Washing is a family owned business that specializes in Exterior Surface Cleaning. We are based in Harris County GA, but our service area includes Harris County and the entire surrounding area. We virtually cleans and remove rust stains from vinyl siding. We offer free estimate.Website: https://www.goodasnewpressurewashingllc.com/reside... Reply
Integral investigations - Wednesday, December 30, 2020 - link
Integral investigation Sydney offers different services for your case that are performed by private detectives.<a href="https://integralinvestigations.com.au/private-dete... detective Sydney's cost </a> is carrying out all types of private also business investigations charge you a flat rate per hour with everything else included. Our industry offers you private investigation throughout Australia. The thing to be proud of is we are a discreet and affordable private investigation industry in Australia. ReplyTechnical Service - Wednesday, December 30, 2020 - link
How to decor and renovate the house through <a href="https://technicalservice.ae/carpentry-work-service... carpentry work services in Dubai </a>?This desire can now be easily fulfilled by ABDULLA THANI’S GROUP. ABDULLA THANI’S GROUP are providing carpentry work services in Dubai by name of TECHNICAL SERVICES BY ABDULLA THANI’S GROUP. ReplyTechnical investigation - Thursday, December 31, 2020 - link
Dubai is one of the most attractive cities in the world due to its catchy infrastructure and well-known buildings. We also have a <a href="https://technicalservice.ae/wall-partition-and-fal... False Ceiling Contractor in Dubai</a> for commercial and residential spaces as Beautiful and ethical Commercial and residential layout of houses is highly demanded in Dubai. In this regard, Abdullah Thani’s group of technical works are providing high-quality services to their clients. Abdullah Thani group provides many services. Reply