Presto @ Netflix: Interactive
Queries at Petabyte Scale
Nezih Yigitbasi and Zhenxiao Luo
Outline
Our big data platform
Presto @ Netflix
Netflix integration
Our contributions
What’s next?
Cloud
Apps
S3
Suro/Kafka Ursula
SSTable
s
Cassandra Aegisthus
Event Data
500 bn/day, 15m
Daily
Dimension Data
Netflix Data...
Data
Warehouse
Service
Tool
s
Gateways
Our Big Data Platform
Prod
Clients
Clusters
Query Prod TestProd
Big Data API/Portal...
Our Use Cases
Batch jobs (Pig, Hive)
ETL jobs
reporting and other analysis
Interactive jobs
Presto @ Netflix
What is Presto?
An open source distributed SQL engine
for running interactive queries against
large datasets
Why we love Presto?
Fast
0
200
400
600
800
Group By Join + Group By Needle in
Haystack
Presto Hive
QueryCompletionTime[s]
Why we Love Presto?
Fast
Scalable
ANSI SQL
Open source
Works well on AWS
Hadoop friendly
presto-cli, Python, R, BI tools (ODBC/JDBC), etc.
Our Deployment
v 0.100
Java 8™
1 coordinator (r3.4xlarge)
~220 workers (...
4 s
Median runtime
15+ PB
Total data size
2.5K
Queries/day
300+
Presto users
99 MB
Median data size
Netflix Integration
S3
Atlas
Sidecar
PrestoAmazon EMR
Amazon
RDS
HCat
Server
Coordinator
Worker
S3
Atlas
Sidecar
PrestoAmazon EMR
Data Lineage
query
completion
events
S3
Atlas
Sidecar
PrestoAmazon EMR
Monitoring
metrics
S3
Suro
Atlas
Sidecar
PrestoAmazon EMR
BI Tools
Our Contributions
S3 Filesystem
Query Optimizer
Parquet File Format
Complex Types
Multipart upload
Instance credentials
Role support
Reliabi...
presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
Worker
Worker
Parser Optimizer
Scheduler
Distributed
Planner
Coordinator
Fun...
Single Distinct => Group By
select
count(distinct c)
from t
select count(*)
from (select c
from t
group by c) q
Output
Cou...
Joins with Similar Subqueries
select *
from (select k,
agg1,
agg2
from t
group by k) a
join (select k,
agg3,
agg4
from t
g...
Output
Table Scan
table = t
Group By
Aggregation
key= k
agg1, agg2, agg3, agg4
select k, agg1,
agg2, agg3,
agg4
from t
gro...
presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
Worker
Worker
Parser Optimizer
Scheduler
Distributed
Planner
Coordinator
Fun...
Complex Type Support
map_agg()
map_keys()
map_values()
map<K,V> row(F T)
=, !=
bug fixes
array<T>
array_join()
sort_array(...
presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
S3
Filesystem
Worker
Worker
S3
Filesystem
Parser Optimizer
Scheduler
Distrib...
Presto S3 FileSystem
(multipart upload, instance/static credentials,
assume role, reliability, etc.)
S3
open() seek() list...
presto-cli
other
clients
Odbc/Jd
bc
S3
Worker
S3
Filesystem
Worker
Worker
S3
Filesystem
Parser Optimizer
Scheduler
Distrib...
RowGroup Metadata
codec, encoding, etc.
Column Chunk
Page
Page
Page
Column Chunk
Page
Page
Page
Column Chunk
Page
Page
Pag...
What’s next?
Parquet optimizations
vectorized reader
predicate pushdown
lazy load
lazy decompression/decoding
Better resou...
THANK YOU
of 31

presto-at-netflix-hadoop-summit-15

Presto @ Netflix
Published on: Mar 4, 2016
Published in: Technology      
Source: www.slideshare.net


Transcripts - presto-at-netflix-hadoop-summit-15

  • 1. Presto @ Netflix: Interactive Queries at Petabyte Scale Nezih Yigitbasi and Zhenxiao Luo
  • 2. Outline Our big data platform Presto @ Netflix Netflix integration Our contributions What’s next?
  • 3. Cloud Apps S3 Suro/Kafka Ursula SSTable s Cassandra Aegisthus Event Data 500 bn/day, 15m Daily Dimension Data Netflix Data Pipeline
  • 4. Data Warehouse Service Tool s Gateways Our Big Data Platform Prod Clients Clusters Query Prod TestProd Big Data API/Portal Metacat
  • 5. Our Use Cases Batch jobs (Pig, Hive) ETL jobs reporting and other analysis Interactive jobs
  • 6. Presto @ Netflix
  • 7. What is Presto? An open source distributed SQL engine for running interactive queries against large datasets
  • 8. Why we love Presto? Fast
  • 9. 0 200 400 600 800 Group By Join + Group By Needle in Haystack Presto Hive QueryCompletionTime[s]
  • 10. Why we Love Presto? Fast Scalable ANSI SQL Open source Works well on AWS Hadoop friendly
  • 11. presto-cli, Python, R, BI tools (ODBC/JDBC), etc. Our Deployment v 0.100 Java 8™ 1 coordinator (r3.4xlarge) ~220 workers (r3.4xlarge) Clients
  • 12. 4 s Median runtime 15+ PB Total data size 2.5K Queries/day 300+ Presto users 99 MB Median data size
  • 13. Netflix Integration
  • 14. S3 Atlas Sidecar PrestoAmazon EMR Amazon RDS HCat Server Coordinator Worker
  • 15. S3 Atlas Sidecar PrestoAmazon EMR Data Lineage query completion events
  • 16. S3 Atlas Sidecar PrestoAmazon EMR Monitoring metrics
  • 17. S3 Suro Atlas Sidecar PrestoAmazon EMR BI Tools
  • 18. Our Contributions
  • 19. S3 Filesystem Query Optimizer Parquet File Format Complex Types Multipart upload Instance credentials Role support Reliability Single distinct => Group By Joins with similar subqueries Schema evolution Parquet 1.6 Various new functions Comparability
  • 20. presto-cli other clients Odbc/Jd bc S3 Worker Worker Worker Parser Optimizer Scheduler Distributed Planner Coordinator Functions Type System 1 2 3 4 5 6 7
  • 21. Single Distinct => Group By select count(distinct c) from t select count(*) from (select c from t group by c) q Output Count Aggregation masks = {column$distinct} Distinct marker = column$distinct Table Scan Output Count Aggregation masks = {} Group By Aggregation count Table Scan
  • 22. Joins with Similar Subqueries select * from (select k, agg1, agg2 from t group by k) a join (select k, agg3, agg4 from t group by k) b on ( a.k = b.k ) Output Table Scan table = t Join key= k Group By Aggregation key= k agg1, agg2 Group By Aggregation key= k agg3, agg4 Table Scan table = t
  • 23. Output Table Scan table = t Group By Aggregation key= k agg1, agg2, agg3, agg4 select k, agg1, agg2, agg3, agg4 from t group by k Joins with Similar Subqueries
  • 24. presto-cli other clients Odbc/Jd bc S3 Worker Worker Worker Parser Optimizer Scheduler Distributed Planner Coordinator Functions Type System 1 2 3 4 5 6 7
  • 25. Complex Type Support map_agg() map_keys() map_values() map<K,V> row(F T) =, != bug fixes array<T> array_join() sort_array() concat() =, !=, <, >
  • 26. presto-cli other clients Odbc/Jd bc S3 Worker S3 Filesystem Worker Worker S3 Filesystem Parser Optimizer Scheduler Distributed Planner Coordinator Functions Type System 1 2 3 4 5 6 7
  • 27. Presto S3 FileSystem (multipart upload, instance/static credentials, assume role, reliability, etc.) S3 open() seek() list() Get Object Get Object Metadata List Objects
  • 28. presto-cli other clients Odbc/Jd bc S3 Worker S3 Filesystem Worker Worker S3 Filesystem Parser Optimizer Scheduler Distributed Planner Coordinator Functions Type System 1 2 3 4 5 6 7 Parquet Cursor Parquet Cursor
  • 29. RowGroup Metadata codec, encoding, etc. Column Chunk Page Page Page Column Chunk Page Page Page Column Chunk Page Page Page RowGroup Footer schema, version, etc. Column Metadata value count size, min, max Column Metadata value count size, min, max Column Metadata value count size, min, max
  • 30. What’s next? Parquet optimizations vectorized reader predicate pushdown lazy load lazy decompression/decoding Better resource management Better BI tool integration
  • 31. THANK YOU

Related Documents