AWS re:invent Live Stream – Werner Keynote 11/30/2017
Werner Vogels is VP and CTO of Amazon (AWS).
Egalitarian Platform – everyone has access to the same storage, analytics, algorithms, etc…
What sets a company aside is the quality of the data they have.
G.E. – You go to bed as a manufacturing company, and you wake up as a data and analytics company.
Every device that draws a current has the possibility of connecting (to the internet).
Can now build neural networks that we can execute in real time (used to be just offline).
Example of a Philippine application where machine learning systems tells farmer how much fertilizer to use and when to apply it. Saved 90% on fertilzer cross and doubled the produce.
Voice Apps – natural way of working with computers.
Announcing: Alexa for Business – fully managed
Get access to your home private skills at work.
Imagine Saying the following:
Alexa, 1) cancel my staff meeting, 2) what meetings do I have today? 3) How many laptops do we have in stock?
Integration with Sales Force, flight information, reimbursement software, HR skills, Splunk skills (error logs),
Wynn Hotel is putting Amazon Echos in every room of their hotel, with private skills to lower blinds, control TV, ask about your room bill, etc… No longer need five different remote controls!
1) Proving and manage shared services
2) Configure conference rooms
3) Enrollers and assign skills
4) Build Custom Skills
11:05 Voice represents the next biggest disruption in computing.
I made a little picture to emphasize this quote:
11:08 Three Operational Planes
1) Admin Plane
2) Control Plane
3) Data Plane
Koala Lumpur Video Streaming Business
11:13 AWS Well-Architected frameowrk – 5 Pillars
1) Oper Excellent
2) Security
3) Reliability
4) Performance
5) Cost Optimization
Lens
1) HPC – High Performance Computing
2) Serverless
Certification
Bootcamps – train others to do well-architected systems
The system you build now is not going to be the the same as it is six months from no2.
Principles:
1) Stop guessing capacity needs
2) Test systems at production scale
3) Automate to make architectural experimentation easier
4) Allow for evolutionary architectures
5) Drive your architecture using data
6) Improve Tough Game Days
1) Identity
2) Detective ontrols
3) Infrastuructre protection
4) Data Protectioin
5) Incident Response
1) Implement a strong identity foundation
2) Enable traceability
3) Apply security at all layers
4) Automate security best practices
5) Protect data in transit and rest
6) Prepare for security events
11:19 Dance like no one is watching, encrypt like everyone is.
Eliminate IAM users, get down to least needed minimal security. Don’t run anything under root account.
Ubiquitous Encryption – use https/TOS (see toolkit)
No excuse anymore to not use encryption
Security is your job, not just the security team. It’s all of our jobs to protect the customers.
The new security team: Operations, Developers,
Pace of Innovation,
11:22 Protection in a CI/CD world – Build servers have CloudTrail enabled, security in the pipeline,
every update to source code is validated, …
Amazon Inspector can scan the software you are deploying.
AWS Config Rules – great tool – monitor, compare how system looks 2 weeks ago with now.
Tracks all changes in your environment.
CloudTrail will log every API call to every service, and put in S3.
11:26 How has development changed?
1) Has to be more security aware
2) More collaborative
3) More Languages
4) More services
5) More mobile
6) Q&A and Operations are deeply integrated
11:28 If you develop in this fast changing environment, you need help.
There is something extra we need to do for you.
Every great platform has a great Integrated Development Environment (IDE).
Announcing: AWS Cloud 9 – a cloud IDE for writing, running, and debugging code. Generally available today.
Speaker: Clare Liguori – Sr Software Engineer
Cloude 9 – can pull up in any browser.
Can change theme, key-binding.
Several languges and syntax highlighters installed.
Can run the code directly in the IDE.
You can see all the Lambda blueprints in the IDE.
Has breakpoints and debugger.
One user can share his Cloud 9 environment to another person, so you can pair program and review code.
You can chat inside the IDE. As one person types, his changes show up on the partners screen.
After testing, you can deploy to Lambda, then run it from Lambda.
11:37 Back to Werner speaking
Deep integration with CodeStar tools (Pipeline, Deploy, CodeCommit, CodeBuild).
Availability, reliability, and resilience in 21st century architectures.
Quote from years ago: Everything will fall all the time. You don’t know when, but it might.
11:53 Nora Jones – Chaos Engineering – from NetFlix
Chaos Experiments – look a lot like unit tests, but can add latency and time between calls.
We call it “experiments” instead of testing; we assume we are resilient from the failures.
Chaos engineering doesn’t replace, but works with unit testing
Worked on Chaos Engineering book (OReilly publishers).
You may have heard of their “chaos monkey”.
1) Graceful Restarts + Degradation
2) Targeted Chaos
3) Cascading Failure
4) Failure Injection
5) ChAP – Chaos Automation Performance?
Not if it fails, but what to do when it fails.
Key Metric for Netflix is: Metric is whether or not you can press “play”;
total is called: SPS – Stream Starts per Second
Smallest fraction of traffic possible to know if chaos experiment is working properly.
They needed 2% signal of actual live traffic/transactions, route 1% into control cluster, and 1% into experiment cluster.
One is the control, and in the other one they add in their points of failure.
Automated Canary Analysis – If things go wrong, they shorten the experiment early; before it renders NetFlix unusable for customers. (then developer can go debug it).
They automated the experiments, and the criticality of them. They run the critical ones more often.
http://www.PrinciplesOfChaos.org
Chaos doesn’t cause problems, it reveals them.
12:05 Back to Werner
Gall’s Law – a complex system that works is invariable found to have evolved from a simple system that worked.
Werner gives example of which “planes” are removed off your plate when you go with various cloud services like hosted RDS.
You should use as many “managed services” as possible to achieve reliability and high availability.
Launched Amazon MQ and Time Sync yesterday.
You can focus just on the business functions you want to write.
The rise of microservices.
Scaling components down to the minimum business logic that has scaling and reliability requirements.
Example, login/security service is used on every page, but business logic differs on each page; so one needs to be scaled at a different level. Decomposing into smaller components… Container Technology is helping with this. Has become the default if you want to build microservices.
12:12 Abby Fuller – Senior Technical Evangelist – expert on containers.
Tons of options (a good thing): ECS, Fargate for ECS, EKS, Fargate for EKS
The power is in the choices.
Monzo – is a mobile only bank in England. Have about 350 microservices.
The idea was that “Highly available Kubernetes was not for the faint of heart.”, so “manage Kubernetes for me”, and let me focus on the application. Containers in production can be hard work; lot’s of moving pieces.
Which brings us to “Fargate” and the future, not a service, but technology to help…
12:20 Start Demo of Fargate in action.
It’s not about how, but how well.
Don’t worry me about setting, tune up, images; just run it. Let’s AWS handle the heavy lifting.
Create systems that will support you in 2020.
12:22 Back to Werner –
Question: So what does your future look like?
Answer: All the code you ever write is business logic.
iRobot – strict costs, they only pay for the robot, they don’t pay for cloud services that come with it.
All their code is serverless, they never have to pay for idle time.
Agero – a company that has devices/software that can detect if vehicle is in an accident – Crash Detection on Lambda
HomeAway – established business – takes in 6 million photos/month. Entirely serverless!
Architectural Principles:
1) State machines
2) Tables != Databases
3) Events as Interface s
4) Encrypted parameter store
What customers are asking for:
1) Language Support
2) Peformance
3) Function concurrency
4) VPC Integration
4 new poerful Featurs for AWS Lambda:
1) API Gateway/VPC Integratoin
2) Concurrency Controls
3) 3GB memory support
4) .NET Core 2.0 support
5) and Go (language)
Making Serverless even more “Less”
1) Getting started
2)
Announcing: AWS Serverless Application Repository – discover, deploy, publish, reuse
12:32 Dr Walter Scott – CTO of and Founder, DigitalGlobe
Deals with images and maps and satellite photos of earth.
A lot of information in these pictures, but it’s a really big plante, i.e. BIG DATA!
80 Terabytes collected every day from their satellites.
Before AWS, had their own data center with 100 Petabytes of images, but it was “stuck in a jail”.
Used the Snowmobile (data center on wheels) to transfer data.
Moved 17 years worth of data in a single cost effective-operations, to two regions in Glacier.
Second challenge – provide on-demand access while still managing cost.
Maybe take last 180 days and do aged-based caching.
But two problems:
1) Was still a big cache
2) Still missed a lot – Only 40% cache hit rate.
We have gone from viewing images, to analyzing images.
Data access is highly variable and diverse, included current and historical data.
They turned into machine learning service (SageMaker) to the caching problem.
Used pictures of villages in Africa to know how many vaccines to prepare and send.
Can we predict where the next access is likely to be, and preload from glacier before it is needed.
Got cache ratio up to 80% and trending up to 90%.
Third problem solved: Analysis – extracting information from 100 Petabytes at scale.
GBDx – Geo Spatial Big Data Platform – Geo Data as a service
takes unstructured imagery and makes it structured and pulls out various features.
GBDx Notebooks, built on the “jupyter” notebook framework.
Geoscape – continuously updated service of all buildings, roads, roof heights, tree canopies, etc… in Australia.
Used for Telecom – as it moves to 5G, which is blocked by things like trees.
Wildfires in 2014, how to evacuate people… Time = Risk to Life.
Geoscape takes out the guess work from the first response workers.
GBDX for Sustainability – challenge – an experiment with purpose – ideas for solutions for UN sustainable development goals, like good health and well-being. DigitalGlobe.com/revinent
12:43 Back to Werner
Machine learning impacts: developer tools, operations, security
Multi-Lingual Social Analytics
737 Flight Simulation – all machine learning, Alexa talks to pilot, gives advice about best actions to take.
trainline – price prediction
Guest speaker – Sensors, machine learnings and real humans – Mati Kochavi – Founder, AGT International and Heed
Tribe of storytellers – endless search for telling stories in a new ways, and today we all join this journey.
Two UFC MMA boxer/fighters (Diakiese vs Barboza) come on page.
Uses terms like: 70 new insights were introduced such as resiliency, aggression index, etc…
in real time with real data.
How to tell the world with the Internet of Things, the story of sports and entertainment.
Starts with data collected form sensors. The floor mat is smart can sense pressure, movement.
The glove is a smart glove with sensor, creates 12 stories… stength, impact, …
World Graph is the representation/model of the world.
It’s not about the statistics of the fight, but the story of the fight.
AI Agent sits on top of the World Graph. One mission, to provide information what you want to know about the sport event, even if you are not there. What is interesting and fun to know how to provide information in a “cool way” to my audience. And different people want to know different things, so can be customized to the relevant people.
