Polyglot
Persistence
Majid Fatemian
R E D V E N T U R E S
@majidfn
Relational Non-Relational Key-Value Big Data
Application
Relational
Application
Relational Relational Non-Relational
Relational
BIG
DATA
Capture
Search
Storage
Q u e r y
Privacy
Visualize
Analysis
Non-Relational
1 2 3
Non-Relational Cluster Schemaless
2 x
1 x
1 x
2011
2012
2013
2014
2011
2012
2013
2014
Application
ACID
Normalized
{Consistency
Relational
Player ID Map ID Difficulty Score
1000 223 E 5
2000 673 H 4.5
1000 451 M 4
2000 980 H 3
Activity
Player ID Name Location
100...
Replication
Fault Tolerance
Availability
{Relational
OPs / DBAs
Tools
Devs{DevOps
Relational
2011
2012
2013
2014
Scalability
Relational
Distribution
Relational
Data C
Data B
Data A
Data Access Layer
Schema
Relational
{

player_id: 1234,

* level: 40,

* duration: 75,
* activity: activity,

game_id: 4001,

platform: WiiU...
Total
Detailed
stats
stats
Players Profiles
Individual Activities
Aggregated
stats
Categorized
ACID
Transaction
Partitioning
Replication
Tools
OPs
Data Volume
Fixed Schema
Scale Up
Manual Sharding
+ -
NoSQL
...
mongoDB
1 2 3
OPs Devs Community
Schema-less
Non-Relational
player_id: 1234,

* level: 40,

* duration: 75,
* activity: activity,
game_id: 4001,

platform:...
Scalability
Non-Relational
Scalability
Non-Relational
+
Sharding
Non-Relational
Data A
{key}
Sharding
Non-Relational
Data A
A
Data B
B
Data C
C
{key}
Map/Reduce
Non-Relational
Query
K, V
K,V1 K,V2 K,V3
Map/Reduce
Non-Relational
Map
K,V1 K,V2 K,V3
Map/Reduce
Non-Relational
Result
ReduceMap
K,V1 K,V2 K,V3
Denormalized
Non-Relationalread /
}write
Disk Space
Player ID Map ID Difficulty Score
1000 223 E 5
2000 673 H 4.5
1000 451 M 4
2000 980 H 3
Activity
Player ID Name Location
100...
{Name: John, game:1, activity: Mission01, Score: 1245}
{Name: John, game:3, activity: Mission04, score: 5431}
{Name: John,...
Data Model
Non-RelationalQuery
Aggregation
}
Eventual-
Consistency
Non-Relational
X2
W
X1 X1
X2
Eventual-
Consistency
Non-Relational
X2
W
X1 X1
R
X1
X2
Eventual-
Consistency
Non-Relational
X2
W
X2
X2
R
X2
X2
Balancer
Non-Relational
Balancer
Non-Relational
+
Balancer
Non-Relational
Non-Relational
Resilience
A B C
Non-Relational
Resilience
A C
Locking
Non-RelationalDB Level (2.2)
}Document Level (3.x)
Backup
Non-Relational
Point-in-time
Denormalized data
Disk Space
Expertise
Complex Querying

Eventual Consistency
Resource Usage
DB-Level Locking
-
Schema-Les...
Random Profile

Search

(1M)
Full History
Retrieval

(6M)
Insert

(6M)
Aggregation

Map/Reduce

(6M)
0
0.003
0.005
0.008
0....
Persistence
Polyglot
Players Profile
Leaderboards
Activity Details
Relational Data
Schema-less
Sharding
Total Sum
Activity 1
Activity n
Activity m
……
Shard1
Shard2
Config
Data Access Layer
Application
The
Challenge
Inconsistency1
Data Access Layer
W
Pending
Commit
2-Phase
Data Access Layer
Pending
W
Commit
2-Phase
Data Access Layer
Done
Commit
2-Phase
2 Latency
vs. Consistency
Operations3 Exponential complexity
Availability4 Plan for failure
Data Access Layer
Data Access Layer
Data Access Layer
Data Integrity5 Backward compatibility
Unit
{TEST Integration
Functional
Data Migration6
Profile
History }
{
profile:
{
GUID:…,
platform:
{
name:’xbox’,
xuid:…
}
},
activity:…,
duration:…,
datetime:…
}
3rd Party
1 One-Shot Migration
4 hours !!
Live !!
2 Profile versioning
V
+1
1
V
+1
1
V
+1
2
Final
Results
Scalability
2011 2012 2013 2014
9x response time
3 years uptime
Go Hybrid!
Thoughtfully
>
Thank
YOU!
@majidfn
References
Polyglot Persistence
NoSQL Distilled
Polyglot Persistence - Confoo 2016
Polyglot Persistence - Confoo 2016
Polyglot Persistence - Confoo 2016
Polyglot Persistence - Confoo 2016
Polyglot Persistence - Confoo 2016
Polyglot Persistence - Confoo 2016
of 87

Polyglot Persistence - Confoo 2016

NoSQL databases are every where and they are here to stay. For a successful scalable web application it is vital to understand capabilities of NoSQL databases. Also relational databases are not dead and they will not be. Understanding abilities of each, would help making better architectural decisions. Hybrid database solutions could use bests of the both worlds. We would go in to the important decision making factors for each.
Published on: Mar 4, 2016
Published in: Internet      
Source: www.slideshare.net


Transcripts - Polyglot Persistence - Confoo 2016

  • 1. Polyglot Persistence
  • 2. Majid Fatemian R E D V E N T U R E S @majidfn
  • 3. Relational Non-Relational Key-Value Big Data
  • 4. Application Relational
  • 5. Application Relational Relational Non-Relational
  • 6. Relational
  • 7. BIG DATA
  • 8. Capture Search Storage Q u e r y Privacy Visualize Analysis
  • 9. Non-Relational
  • 10. 1 2 3 Non-Relational Cluster Schemaless
  • 11. 2 x 1 x 1 x
  • 12. 2011 2012 2013 2014
  • 13. 2011 2012 2013 2014
  • 14. Application
  • 15. ACID Normalized {Consistency Relational
  • 16. Player ID Map ID Difficulty Score 1000 223 E 5 2000 673 H 4.5 1000 451 M 4 2000 980 H 3 Activity Player ID Name Location 1000 John US 2000 Catherine CA 3000 Paul CA 4000 Kylee US Profile
  • 17. Replication Fault Tolerance Availability {Relational
  • 18. OPs / DBAs Tools Devs{DevOps Relational
  • 19. 2011 2012 2013 2014
  • 20. Scalability Relational
  • 21. Distribution Relational Data C Data B Data A Data Access Layer
  • 22. Schema Relational {
 player_id: 1234,
 * level: 40,
 * duration: 75, * activity: activity,
 game_id: 4001,
 platform: WiiU, difficulty_level: medium,
 repetition: 178,
 score: 450,
 endurance:3
 } {
 player_id: 1234,
 level: 12,
 duration: 120, activity: activity
 }
  • 23. Total Detailed stats stats Players Profiles Individual Activities Aggregated stats Categorized
  • 24. ACID Transaction Partitioning Replication Tools OPs Data Volume Fixed Schema Scale Up Manual Sharding + -
  • 25. NoSQL ...
  • 26. mongoDB 1 2 3 OPs Devs Community
  • 27. Schema-less Non-Relational player_id: 1234,
 * level: 40,
 * duration: 75, * activity: activity, game_id: 4001,
 platform: WiiU, difficulty_level: 2,
 repetition: 178,
 score: 450,
 endurance:3 player_id: 1234,
 level: 75,
 duration: 120, activity: activity
  • 28. Scalability Non-Relational
  • 29. Scalability Non-Relational +
  • 30. Sharding Non-Relational Data A {key}
  • 31. Sharding Non-Relational Data A A Data B B Data C C {key}
  • 32. Map/Reduce Non-Relational Query K, V K,V1 K,V2 K,V3
  • 33. Map/Reduce Non-Relational Map K,V1 K,V2 K,V3
  • 34. Map/Reduce Non-Relational Result ReduceMap K,V1 K,V2 K,V3
  • 35. Denormalized Non-Relationalread / }write Disk Space
  • 36. Player ID Map ID Difficulty Score 1000 223 E 5 2000 673 H 4.5 1000 451 M 4 2000 980 H 3 Activity Player ID Name Location 1000 John US 2000 Catherine CA 3000 Paul CA 4000 Kylee US Profile
  • 37. {Name: John, game:1, activity: Mission01, Score: 1245} {Name: John, game:3, activity: Mission04, score: 5431} {Name: John, game:1, activity: Mission01, Score: 1245}
  • 38. Data Model Non-RelationalQuery Aggregation }
  • 39. Eventual- Consistency Non-Relational X2 W X1 X1 X2
  • 40. Eventual- Consistency Non-Relational X2 W X1 X1 R X1 X2
  • 41. Eventual- Consistency Non-Relational X2 W X2 X2 R X2 X2
  • 42. Balancer Non-Relational
  • 43. Balancer Non-Relational +
  • 44. Balancer Non-Relational
  • 45. Non-Relational Resilience A B C
  • 46. Non-Relational Resilience A C
  • 47. Locking Non-RelationalDB Level (2.2) }Document Level (3.x)
  • 48. Backup Non-Relational Point-in-time
  • 49. Denormalized data Disk Space Expertise Complex Querying
 Eventual Consistency Resource Usage DB-Level Locking - Schema-Less Aggregated Data Large Scale Data Sharding Map/Reduce Memory Storage Journaling +
  • 50. Random Profile
 Search
 (1M) Full History Retrieval
 (6M) Insert
 (6M) Aggregation
 Map/Reduce
 (6M) 0 0.003 0.005 0.008 0.01 MySQL MongoDB
  • 51. Persistence Polyglot
  • 52. Players Profile Leaderboards Activity Details Relational Data Schema-less Sharding
  • 53. Total Sum Activity 1 Activity n Activity m ……
  • 54. Shard1 Shard2 Config
  • 55. Data Access Layer Application
  • 56. The Challenge
  • 57. Inconsistency1
  • 58. Data Access Layer W Pending Commit 2-Phase
  • 59. Data Access Layer Pending W Commit 2-Phase
  • 60. Data Access Layer Done Commit 2-Phase
  • 61. 2 Latency vs. Consistency
  • 62. Operations3 Exponential complexity
  • 63. Availability4 Plan for failure
  • 64. Data Access Layer
  • 65. Data Access Layer
  • 66. Data Access Layer
  • 67. Data Integrity5 Backward compatibility
  • 68. Unit {TEST Integration Functional
  • 69. Data Migration6
  • 70. Profile History } { profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… } 3rd Party
  • 71. 1 One-Shot Migration 4 hours !! Live !! 2 Profile versioning
  • 72. V +1 1
  • 73. V +1 1
  • 74. V +1 2
  • 75. Final Results
  • 76. Scalability 2011 2012 2013 2014
  • 77. 9x response time
  • 78. 3 years uptime
  • 79. Go Hybrid! Thoughtfully >
  • 80. Thank YOU! @majidfn
  • 81. References Polyglot Persistence NoSQL Distilled

Related Documents