distributed_systems_MIT/lec10

lec10

dz / distributed_systems_MIT / lec10

Summary

Cloud replicated databases, Aurora

Node Tree

Nodes

aurora
content Aurora
children amazon (Created by), append_only_log_writes, aurora_description (description), backhistory (Backhistory before Aurora), big_db_handling, db_server, db_tutorial (Fundamentals to understand Aurora architecture), quorum, storage_server

amazon
content Amazon
parents aurora

aurora_description
content Reliable, high performance, cloud databse
parents aurora

backhistory
content Backhistory
children ec2
parents aurora

ec2
content EC2
children ebs (works with), good_for_webservers, not_good_database, physical_storage
parents backhistory

good_for_webservers
content Good for webservers
children not_good_database (not as good for)
parents ec2

not_good_database
content Not as good for databses
children data_loss_on_hardware_crash (why ec2 wasn't good for datases)
parents good_for_webservers, ec2

physical_storage
content Storage: physical, slice to guest
children data_loss_on_hardware_crash
parents ec2

data_loss_on_hardware_crash
content If hardware crashes, data is lost
children ebs (fixes), s3 (intermediate solution)
parents not_good_database, physical_storage

s3
content S3
parents data_loss_on_hardware_crash

ebs
content EBS: elastic block store
children ebs_problems, looks_like_regular_drive, not_sharing_system, replicated_pair_storage_servers (description)
parents ec2, data_loss_on_hardware_crash

replicated_pair_storage_servers
content Replicated pair of storage servers
parents ebs

not_sharing_system
content Not a system for sharing
parents ebs

looks_like_regular_drive
content Looks like a regular drive to client
parents ebs

ebs_problems
content problems with EBS
children large_volumes_network_traffic, not_fault_tolerant
parents ebs

not_fault_tolerant
content Not very fault tolerant
parents ebs_problems

large_volumes_network_traffic
content Produces large volumes of network trafic
children database (I think this has to do with how databases send whole,pages around the network)
parents ebs_problems

database
content database
children db_tutorial
parents large_volumes_network_traffic

availability_zone
content Availability zone
children data_configuration, read_2_dead_az_1_other_server, write_one_dead_az

db_tutorial
content Database Tutorial
children crash_recovery, data_pages, locks, transaction, write_ahead_log
parents aurora, database
remarks A brief overview of how databases work. Just enough to understand Aurora

transaction
content Transaction
children locks, multi_ops_grouped_atomic
parents db_tutorial

multi_ops_grouped_atomic
content Multiple operations grouped and declared atomic
children locks, transactions_all_or_none
parents transaction

locks
content Locks
parents db_tutorial, multi_ops_grouped_atomic, transaction

crash_recovery
content Crash Recovery
children transactions_all_or_none
parents db_tutorial

data_pages
content Data pages
parents db_tutorial

transactions_all_or_none
content After crash: all transactions commited or none
parents crash_recovery, multi_ops_grouped_atomic

write_ahead_log
content Write-Ahead log (WAL)
children redo
parents db_tutorial

redo
content redo: do all transactions again using log
parents write_ahead_log

RDS
content RDS
children RDS_def

RDS_def
content Replicated database in multiple availability zones
parents RDS

db_server
content Database server
children custom_software, database_setup, storage_understand_db_entries
parents aurora

custom_software
content Custom software
parents db_server

database_setup
content Setup
children log_records
parents db_server

data_configuration
content Data: 6 replicants in 3 availability zones
children super_ft
parents availability_zone

log_records
content Log records anything being sent to replicas
children immediately_appended_log_records, log_smaller_than_pages
parents database_setup, storage_server

log_smaller_than_pages
content Log entries smaller than database pages
parents log_records

storage_understand_db_entries
content Underlying storage system understands database entries
parents db_server
remarks An important fact. Aurora went with a specialized filesystem rather than a generalized filesystem and got significant performance gains as a result of this.

super_ft
content Super fault-tolerant
parents data_configuration

quorum
content Quorum
children ft_goals, replication
parents aurora

ft_goals
content Fault-Tolerant Goals
children fast_replication, read_2_dead_az_1_other_server, ride_out_transient_slowness, storage_only, write_one_dead_az
parents quorum

write_one_dead_az
content Write with one dead availability zone (AZ)
parents availability_zone, ft_goals

read_2_dead_az_1_other_server
content Read with 2 dead availability zones plus one other server
parents availability_zone, ft_goals

ride_out_transient_slowness
content Ride out ransient slowness in replicas
parents ft_goals

fast_replication
content Fast replication
children parallel_copy
parents ft_goals

storage_only
content Storage Only
parents ft_goals

replication
content Replication
children N_rep, R_replicas, W_writes
parents quorum

N_rep
content N rep
children R_plus_W_equals_N_plus_one
parents replication

W_writes
content W Writes
children R_plus_W_equals_N_plus_one
parents replication

R_replicas
content R replicas
children R_plus_W_equals_N_plus_one
parents replication

R_plus_W_equals_N_plus_one
content R + W = N + 1
children guaranteed_overlap, variable_defaults
parents N_rep, R_replicas, W_writes

guaranteed_overlap
content Guaranteed to overlap
parents R_plus_W_equals_N_plus_one

variable_defaults
content N=6,W=4,R=3
parents R_plus_W_equals_N_plus_one
remarks Defaults used by the Aurora setup

append_only_log_writes
content Append-only log writes
children Appendtoatleast4replicastobecompleted
parents aurora

app_4_reps_complete

Appendtoatleast4replicastobecompleted
parents append_only_log_writes

parallel_copy
content Parallel Copy
children chunks_of_log_protection_groups
parents fast_replication

chunks_of_log_protection_groups
content Chunks of log protection groups
parents parallel_copy

storage_server
content Storage Server
children db_tracks_server_log_indexes, log_records
parents aurora

apply_log_record_to_page
content Apply log record to page when ready req
parents old_page_log_records
remarks not sure what I meant by "req" also, added that this is "quorum logic" not here

old_page_log_records
content Old page and log records
children apply_log_record_to_page
parents immediately_appended_log_records

immediately_appended_log_records
content Immediately appened to list of log records
children old_page_log_records
parents log_records

db_tracks_server_log_indexes
content DB tracks server log indexes (how far along) for each server
children db_read_updated_ss
parents storage_server

db_read_updated_ss
content On read, find up-to-date storage server and read from that
children when_quorum_reads_used
parents db_tracks_server_log_indexes

when_quorum_reads_used
content Quorum reads used during crash recovery of DB server
parents db_read_updated_ss

big_db_handling
content Big Database Handling
children sharding
parents aurora

sharding
content sharding
children protection_servers
parents big_db_handling

protection_servers
content Protection Servers
children groups_of_six_servers, how_to_split_up_logs, logs, pg_stores_logs_relevant_to_pages
parents sharding

logs
content Logs
parents protection_servers

how_to_split_up_logs
content How to split up logs?
children pg_stores_logs_relevant_to_pages
parents protection_servers

pg_stores_logs_relevant_to_pages
content Protection Group stores logs relevant to pages
children find_pg_with_data
parents how_to_split_up_logs, protection_servers

groups_of_six_servers
content Groups of six servers
parents protection_servers

find_pg_with_data
content Find protection group with data
parents pg_stores_logs_relevant_to_pages