Saturday, March 23, 13
Brought to You By Marc Campbell Martin Hald Thanks Fernanado!Saturd...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 2013-­‐03-­‐20  15:49:06.243  [info]  <0.2780.0>@tt_account:came_...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Live DemoSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 ConceptsSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Whats in a Log Entry? Lager ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Whats in a Log Entry? Lager ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Lager Log Messages  88  2013-­‐03-­‐16  08:29:4...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Popcorn Log Message  88  2013-­‐03-­‐16  08:29:...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 From Log Entry to Alert Log ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Alerts Groups Critical Log Entries #lo...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Address Scheme Role Version ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Crashes have no line number http_hander:404   http_h...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Data Stream Structured ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Acme CalculatorsSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Build Popcorn git  clone  git@github.com:marccampbe...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Configure Popcorn app.config               ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Hook in Popcorn Calculator app rebar.config ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Hook in Popcorn Calcul...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Demo Using the Calculator HTTP API curl ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Customers ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Customers ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Customers ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Popcorn DesignSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Ingesting Storag...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 UDP or TCP or ... Connecti...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Ingesting Architecture ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Storage Architecture ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Storage gen_server behaviour init...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 handle_call {counter_value,  Count} {...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 handle_cast {expire_logs_matching,  Timestamp} ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Server Sent Events ...
SSE Events in JS $(function() { var streamSource = new EventSource("/dashboard/stream/{{stream_id}}...
SSE Erlang Design http_stream_handler.erl -behavior(cowboy_http_handler). ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 server  {   listen      80;   server_name  popcorn.logwithpopcorn...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Wire Protocol Standards & Effic...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Protobuffs        erlang:iolist_to_binary([  ...
JSON encode_json_packet(Node)  -­‐>        jiffy:encode({[          ...
Bits and Bytes ^H^A^Ropcorn@anduin.tigertext.me^Z calculator"^E0.0.1(^A2^XCalculator  request  5  *   4:^Ohtt...
Processing Time Over 1,000,000 Requests on Apple Air with R16B ...
Memory 3x Me Le ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Rendering Architecture ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 countsByPe...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Scaling PopcornSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Scalability Goal Handle n lo...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Let’s Get Some ToolsSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 statsd & rps (mh...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Chrome ...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Mnesia UDP Receiving...
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Issue: MnesiaSaturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
#POPCORN ERLANG_FACTORY @SF_BAY - 2013 Issue: The Browser Executing on t...
of 55

#Popcorn Erlang Factory @SF_Bay 2013

Published on: Mar 4, 2016
Published in: Technology      
Source: www.slideshare.net


Transcripts - #Popcorn Erlang Factory @SF_Bay 2013

  • 1. Saturday, March 23, 13
  • 2. Brought to You By Marc Campbell Martin Hald Thanks Fernanado!Saturday, March 23, 13
  • 3. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 2013-­‐03-­‐20  15:49:06.243  [info]  <0.2780.0>@tt_account:came_online:457        Account  came  online:  c2a8c653-­‐d4c2-­‐4d3d-­‐ad2s-­‐bd488cd9ec81/                e5e752ef-­‐79ae-­‐4e26-­‐9b6b-­‐654e9926c56eSaturday, March 23, 13
  • 4. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Live DemoSaturday, March 23, 13
  • 5. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 ConceptsSaturday, March 23, 13
  • 6. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Whats in a Log Entry? Lager • Timestamp • Severity • Log Text • Module • Function • Line Number • Process IDSaturday, March 23, 13
  • 7. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Whats in a Log Entry? Lager Popcorn • Timestamp • Role • Severity • Node • Log Text • Version • Module Coming Soon • Function • Caller Information • Line Number • App Metrics • Process ID • Env MetricsSaturday, March 23, 13
  • 8. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Lager Log Messages  88  2013-­‐03-­‐16  08:29:46.781  [info]   <0.362.0>@avatar:req_handler:123  Starting   avatar  renderingSaturday, March 23, 13
  • 9. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Popcorn Log Message  88  2013-­‐03-­‐16  08:29:46.781  [info]   <0.362.0>@avatar:req_handler:123  Starting   #customer_request  (avatar  rendering)  for   @mhaldSaturday, March 23, 13
  • 10. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 From Log Entry to Alert Log Alert Message • Error • Warning • Critical • Crash • #identified_topicSaturday, March 23, 13
  • 11. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Alerts Groups Critical Log Entries #login_failed for @fearing, returning #404 #login_failed for @ga, returning #404 Alert #login_success for @bhatia, returning #200Saturday, March 23, 13
  • 12. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Address Scheme Role Version Module Line Number XMPP v2.0.15 user_auth : 54Saturday, March 23, 13
  • 13. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Crashes have no line number http_hander:404   http_hander:404   Bad  request   Bad  request   <<“error”>> <<“api/v2”>> http_hander:404   Bad  request   to ive u at ******** A ci y o or ss m A e 0fc4cf26168b2a4b959d914682c10bc3 MSaturday, March 23, 13
  • 14. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Data Stream Structured Unstructured • Role • Identity • Node • Topic • Version • SeveritySaturday, March 23, 13
  • 15. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Acme CalculatorsSaturday, March 23, 13
  • 16. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Build Popcorn git  clone  git@github.com:marccampbell/ popcorn.git cd  popcorn ./rebar  get-­‐deps ./rebar  compile lessc  apps/popcorn/priv/css/popcorn.less  >          apps/popcorn/priv/css/popcorn.css ./rebar  generate  -­‐fSaturday, March 23, 13
  • 17. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Configure Popcorn app.config                {http_auth_users,      [{<<"admin">>,  <<"password">>}]},                {outbound_notifiers,  [                    {new_alert,  outbound_mailer,                        [{options,          [{retries,  3},  {relay,  "smtp.gmail.com"},                            {username,  "acme.popcorn@gmail.com"},                            {password,  "PASSWORD"}]},                        {from,                  "acme@logwithpopcorn.com"},                        {recipients,      ["your.email@domain.com"]},                        {subject,            "New  Alert  at  {{product}}  {{version}}"}                ]},Saturday, March 23, 13
  • 18. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Hook in Popcorn Calculator app rebar.config {deps,  [    ...    {lager_popcorn_backend,  ".*",  {git,   "git@github.com:marccampbell/ lager_popcorn_backend.git",   "master"  }}    ... ]}.Saturday, March 23, 13
  • 19. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Hook in Popcorn Calculator application sys.config  %%  Lager  config  {lager,  [  {handlers,  [                        {lager_console_backend,  none},                        {lager_file_backend,  [  {"log/error.log",   error,  104857600,  "$D0",  5}  ]},                        {lager_popcorn_backend,  [                                                                          {level,                debug},                                                                          {popcorn_host,  "127.0.0.1"},                                                                          {popcorn_port,  9125},                                                                          {node_role,        "Calculator"},                                                                          {node_version,  "1.0.7"}                                                                        ]}      ]}Saturday, March 23, 13
  • 20. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Demo Using the Calculator HTTP API curl -v -X POST --data-urlencode "2 4 /" http://calculator.logwithpopcorn.com/api/calculator 15+ 93/ 5 2 - (will cause crash) 22* See and try Popcorn at http://demo.logwithpopcorn.com/Saturday, March 23, 13
  • 21. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Customers SupportSaturday, March 23, 13
  • 22. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Customers SupportSaturday, March 23, 13
  • 23. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Customers • Identify edge case bugs • Real-time alerting • Logs as data streams SupportSaturday, March 23, 13
  • 24. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Popcorn DesignSaturday, March 23, 13
  • 25. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Ingesting Storage RenderingSaturday, March 23, 13
  • 26. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 UDP or TCP or ... Connectionless Out of order Dropped packets No Quick acknowledgementsSaturday, March 23, 13
  • 27. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Ingesting Architecture Client (lager) node_fsm Client (winston) node_fsm udp_listener (gen_server) Client (log4j) node_fsm Client (...) node_fsm (...)Saturday, March 23, 13
  • 28. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Storage Architecture storage_monitor storage_worker (gen_server) (gen_server) popcorn_app (application) storage_sup storage_worker (supervisor) (gen_server) storage_worker (gen_server)Saturday, March 23, 13
  • 29. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Storage gen_server behaviour init([])  /  init([worker]) gen_server:handle_call/3 gen_server:handle_cast/2Saturday, March 23, 13
  • 30. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 handle_call {counter_value,  Count} {is_known_node,  Node_Name} {get_alert,  Key} {get_alerts,  Severity,  Sort} {get_alert_keys,  Type} {get_release_module_link,  Role,  Version,  Module} {search_messages,  {...}} {get_alert_timestamps,  Severities}Saturday, March 23, 13
  • 31. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 handle_cast {expire_logs_matching,  Timestamp} {send_recent_log_lines_matching,  Pid,  Count,  Filters} {new_log_message,  #log_message} {new_release_scm,  Record} {new_alert,  Key,  #alert} {new_alert_timestamp,  Key,  Severity,  #alert} {new_alert_key,  Type,  Key} {new_release_scm_mapping,  Record} {delete_counter,  Counter} {increment_counter,  Counter,  Increment_By} {increment_counters,  Counters} {add_node,  #popcorn_node}Saturday, March 23, 13
  • 32. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Server Sent Events SSE Browser Server HTTP Post alert: {key: “001A”, seen:10} node: {key: “xmpp@con8”, log_msgs:324}Saturday, March 23, 13
  • 33. SSE Events in JS $(function() { var streamSource = new EventSource("/dashboard/stream/{{stream_id}}"); streamSource.addEventListener( update_counters, function(event) { var counters = JSON.parse(event.data) // ... }); streamSource.addEventListener( new_node, function(event) { var counters = JSON.parse(event.data) // ... });Saturday, March 23, 13
  • 34. SSE Erlang Design http_stream_handler.erl -behavior(cowboy_http_handler). Wait in New Request Process init receive loop handle handle path Re gis te Ev r en ts fsm node One FSM per SSE clientSaturday, March 23, 13
  • 35. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 server  {   listen      80;   server_name  popcorn.logwithpopcorn.com;   access_log  /var/log/nginx/popcorn_access.log;   error_log  /var/log/nginx/popcorn_error.log;   location  /  {     proxy_pass    http://localhost:9125;     proxy_redirect  off;     proxy_buffering  off;     proxy_cache  off;     proxy_set_header  Host  $host;     proxy_set_header  X-­‐Real-­‐IP  $remote_addr;     proxy_set_header  X-­‐Forwarded-­‐For  $proxy_add_x_forwarded_for;     proxy_read_timeout  420;   } }Saturday, March 23, 13
  • 36. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Wire Protocol Standards & EfficiencySaturday, March 23, 13
  • 37. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Protobuffs        erlang:iolist_to_binary([                protobuffs:encode(1,  1,  uint32),      %%  Packet  version                protobuffs:encode(2,  atom_to_list(Node),  string),                protobuffs:encode(3,  Node_Role,  string),                protobuffs:encode(4,  Node_Version,  string),                protobuffs:encode(5,  lager_util:level_to_num(Severity),  uint32),                protobuffs:encode(6,  Message,  string),                protobuffs:encode(7,  opt(Module,  <<"">>),  string),                protobuffs:encode(8,  opt(Function,  <<"">>),  string),                protobuffs:encode(9,  opt(Line,  <<"">>),  string),                protobuffs:encode(10,  opt(Pid,  <<"">>),  string)        ]).Saturday, March 23, 13
  • 38. JSON encode_json_packet(Node)  -­‐>        jiffy:encode({[              {packet_version,  1},              {node,  list_to_binary(Node)},              {role,  <<"calculator">>},              {version,  <<"0.0.1">>},              {severity,  128},              {message,  <<"Calculator  request  5  *  4">>},              {module,  <<"http_calculator">>},              {function,  <<"rpn">>},              {line,  <<"58">>},              {pid,  <<"">>}        ]}).Saturday, March 23, 13
  • 39. Bits and Bytes ^H^A^Ropcorn@anduin.tigertext.me^Z calculator"^E0.0.1(^A2^XCalculator  request  5  *   4:^Ohttp_calculatorB^CrpnJ^B58R^@ {"packet_version":1,   "node":"popcorn@anduin.tigertext.me",   "role":"calculator",  "version":"0.0.1",  "severity":128,   "message":"Calculator  request  5  *  4",   "module":"http_calculator",   "function":"rpn","line":"58","pid":""}Saturday, March 23, 13
  • 40. Processing Time Over 1,000,000 Requests on Apple Air with R16B JSON Protobuffs 12 us 9 us 6 us 3 us 0 us Encoding DecodingSaturday, March 23, 13
  • 41. Memory 3x Me Le JSON Protobuffs m ss or y 400 bytes 300 bytes 200 bytes 100 bytes 0 bytes MemorySaturday, March 23, 13
  • 42. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Rendering Architecture Plain Old Crossfilter D3 Javascript Used only for grouping Store the grouped streaming data, Rendering data from destroyed every few secondsSaturday, March 23, 13
  • 43. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 countsByPeriod severitiesByPeriod rolesByPeriod nodesByPeriodSaturday, March 23, 13
  • 44. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Scaling PopcornSaturday, March 23, 13
  • 45. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Scalability Goal Handle n log messages per second 10 50 500 5000Saturday, March 23, 13
  • 46. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Let’s Get Some ToolsSaturday, March 23, 13
  • 47. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 statsd & rps (mhald) graphiteSaturday, March 23, 13
  • 48. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Chrome dtop redbug statsd graphite Activity MonitorSaturday, March 23, 13
  • 49. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Mnesia UDP Receiving Client (Browser)Saturday, March 23, 13
  • 50. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Issue: MnesiaSaturday, March 23, 13
  • 51. #POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
  • 52. #POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
  • 53. #POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
  • 54. #POPCORN ERLANG_FACTORY @SF_BAY - 2013Saturday, March 23, 13
  • 55. #POPCORN ERLANG_FACTORY @SF_BAY - 2013 Issue: The Browser Executing on the server Executing in the browser vs (Erlang) (Javascript)Saturday, March 23, 13

Related Documents