Another year and yet another re:Invent. Once again our Cybercom team landed on McCarran International super excited and ready for everything re:Invent has to offer. Majority of the team was really waiting for the keynotes and especially the one held by AWS CTO - Dr. Werner Vogels.
As expected, a huge bunch of new launches and features saw daylight. This year the emphasis of the releases was on Machine Learning, Data, Governance and Serverless. There’re way too many new features to cover in detail, but let’s go through some of them.
Data Lakes, Storage and Databases
This year we saw quite many new launches and feature enhancements in the data domain, where majority of them was related to S3, databases or to the hot-topic-of-the-day data lakes. Being one of the first ever launched AWS services S3 may serve as the first topic to cover in this domain.
As to our surprise, S3 really received many feature additions. Glacier, the cheap cold storage S3 service, got even cheaper, with the new S3 Glacier Deep Archive feature (GA in 2019). It’s a really nice option for objects requiring long-term retention and digital preservation. As the standard Glacier is suitable for more active archiving, the new Deep Archive Glacier is the most suitable solution for replacing tape libraries used for really long term archival (e.g. regulatory archival), where retrieval is not necessary happening at all. Classic Glacier also received an overhaul with a couple of new features targeted to reduce overall storage costs. New S3 cross-region-replication policies allow copying data directly to a different region into a low-cost Glacier cold storage bucket, restoring data from a Glacier archive can now trigger Lambda functions with new ‘restoration started’ and ‘restoration completed’ notifications and finally for those impatient ones, you’re actually able to request an S3 Restore Speed Upgrade to get your data faster. S3 also received a couple of non-Glacier related features for intelligent tiering and batch operations, targeted for cost optimization and managing objects at scale respectively.
Continuing chronologically, in AWS releases order, let’s focus on databases. This year we saw many new feature additions to existing databases as well as completely new databases. Let’s focus on the new ones.
One database type that’s clearly been missing from AWS services offering is definitely a time series database. Up until now you’ve been obligated to run an opensource or commercial options like InfluxDB or Informix TimeSeries. This is where AWS Timestream (in preview) comes into play. Timestream is a fully managed, serverless, scalable and fast time series database with a really attractive pricing compered to more traditional relational databases. Timestream is capable of handling trillions of events daily and is doing really well on query performance, compared to more generic databases.
Other completely new database is the new Quantum Ledger DataBase or QLDB for short. Last year AWS was not hopping into the blockchain bandwagon as they didn’t quite see the customer use cases for it. This year this all changed as in addition to the QLDB they also released Amazon Managed Blockchain with Hyperledger Fabric and Ethereum (coming soon) support. Really interesting to see how they will be adopted and all the application that are built on top of them.
On data streaming and stream analysis side, AWS Kinesis has been the AWS managed solution until now. To complement the offering and to allow customers to leverage their existing solutions, Amazon Managed Streaming for Kafka, MSK for short, was released. It’s a fully managed service, which is a full drop-in replacement to a self-managed version of Apache Kafka. The managed nature frees you from the burden of running your own Kafka cluster and allows you to easily deploy highly available clusters. At the launch MSK is supporting Kafka version 1.1.1.
Last but certainly not least to mention in this category is the upcoming AWS Lake Formation (in preview). During his keynote, Andy Jassy mentioned that there’s already 10000+ data lakes built on top of S3. Most likely all built a bit differently and maybe not too securely either. And with time required to build one also varying from weeks to months. This is where Lake Formation is set to help, reducing the time needed to set things up, in a secure fashion. Data is collected and cataloged with ML algorithms and made available for further processing with other analytics and ML services.
This was certainly the area which received the majority of new features and updates. And why not? It’s really fast-growing part of technology. The spectrum of features ranges from ML training dataset annotation to various inference enhancements and even real-world end user products.
Sagemaker Ground Truth is certainly an interesting feature. The main focus for it is to make high quality and highly accurate datasets for machine learning algorithm training. The way Amazon is addressing the usually slow and costly operation really fascinating. With Ground Truth and with the help of Amazon Mechanical Turk, you can harness the power of 500 000+ workers, or your own workers to start annotating the data while Ground Truth is learning in real-time and starts to apply labeling and annotations to the remaining dataset. Can’t wait to test this one out with some custom dataset.
One of the coolest, yet weirdest, release in re:Invent this year was definitely AWS DeepRacer. Serving as a technology demo for Reinforced Learning and their Sagemaker RL offering, the idea is really, really cool. DeepRacer is basically an RC-car without the controller. Instead of using the controller, DeepRacer is controlled using ML algorithm and the training is done using RL and a simulator. Re:Invent hosted a series of sessions where participants had the possibility to tune their algorithms, train them using simulator and finally race against each other. Want to hear the best thing? DeepRacer League! Yeah, you read it right. Starting early 2019 and open for anyone. Now how cool is that? Start your engines!
Compliance and Security
Some new launches on AWS Security services as well. Nothing ground-breaking in general, but new ways to implement security and compliance to account structures. During this year, trend on AWS services has been to integrate to organizations model and aggregating results from multiple accounts to one master account. AWS Security Hub follows the trend by aggregating security alerts and compliance statuses across AWS accounts and services. Security Hub ingests data from AWS services like GuardDuty, Macie, Inspector and generates aggregated view of security findings. It also integrates to AWS Partner solutions and collects data from 3rd party services like Alert Logic and Dome9.
Another new service AWS Control Tower is more on abstracting Organizations level security, making it easier to deploy preset security configurations, or blueprints if you will. Talking about Organizations, we're of course meaning AWS multi-account structure, which is combining single accounts to set of master account and Organizational Units. This is another main paradigm of many AWS compliance and security services. So, if you're handling your company's services in a one fully-packed AWS account, maybe it's time to rethink and maybe restructure.
Back to AWS Control Tower (still in preview), it doesn't seem to bring that much of new features to the security - table. Guardrails are defined as 'high-level rule for preventing deployment of resources that don't conform to policies'. This can go along lines of readymade IAM Policy / Organizational SCP / permission boundary, easily deployed to account structure.
If it's more custom, fine-grained, high class security you're after, we have a treat for you also. I think this was the best session of re:Invent 2018, FSV325, Nubank CTO Edward Wible presenting some parts of their AWS security structure.
Cloudwatch Logs Insights will drop to this category. A long-waited service for running queries and generating analysis from Cloudwatch Logs. 'Well, didn't we already had this in Cloudwatch Logs?' you may wonder, but no, this one is totally different from filtering events from a log stream. Cloudwatch Logs Insights include a sophisticated ad-hoc query language, time series aggregation, you can use regular expressions to extract data from an event field etc. It also has set of preset queries to get you started. For many development teams out there, this may be an alternative to pushing logs to Elasticsearch cluster.
There are so many services to cover this year, but we have to finish with Lambda related launches. My favorite one being Lambda Layers, meaning possibility to use libraries without need to include them in your deployment package. Layers can also be referenced by many functions at the same time and have specific layer-policy by which you can grant layer access to other accounts, or even whole Organization. Centrally managing common libraries across multiple functions is now possible.
Application Load Balancer added Lambda function as a target, this seems somehow overlapping feature with API Gateway but I'm sure serverless community will find this as useful addition to a current featureset. API Gateway may be a bit too complex for some usecases where you just want to invoke Lambda to handle http/s requests. Talking about API Gateway, widely-requested feature for WebSockets was added, enabling whole new subclass of applications (think of realtime chatrooms and similar) in the serverless ecosystem.
Lambda support for Ruby was added, as well as the new Lambda Runtime API, which AWS blog post refers as a simple interface to use any programming language, or a specific language version, for developing your functions. As an example of using Lambda Runtime API, reference implementation of C++ runtime was released. So, get your helloworld.cpp and make it run serverless!
Once again re:Invent delivered countless amounts of experiences and fun. Although the week was super exhausting due to early mornings and late nights, our team is feeling super energized and enthusiastic to try all newly released features and updates soon. Looking forward to next year and yet another re:Invent. Stay tuned!