Totango Engineering

AWS re:Invent 2015 - Summary for the busy software engineer

Totango proudly runs on Amazon Web Services. Our technology stack deeply leverages what AWS has to offer - from basic EC2 instances to services such as RDS, DynamoDB and Kinesis that provide higher level abstractions.

While it is not without flaws, AWS is a key ingredient in our ability to develop fast and provide a service at high quality and uptime.

That is why we're excited about every AWS re:Invent. Much like Google I/O, Apple WWDC or Dreamforce is for teams developing on those platform, re:Invent is the most important vendor day in the software industry for Totango.

It can get overwhelming sometimes... so here's our quick summary of the announcements we found most interesting from the latest re:Invent 2015 from early October.

Software developers building over the AWS platform and looking for a quick way to catch up on the news may find it useful. Keep in mind that it is opinionated to our particular needs and technology choices, so your mileage may vary .

Most exciting

These new products got us most excited. We'll definitely be evaluating and working with them in 2016.

1. Amazon Elasticsearch:

Like RDS for MySQL, get access to a fully managed Elasticsearch cluster. Pay by cluster size (machine type and amount of data).
Elasticsearch is one of the core components in our data-architecture here at Totango, so this is very important news for us. Hosted Elasticsearch offerings have been in the market for some time (including Elastic's own Found), but we want to keep all data secure and controlled in our AWS private network, so have been hesitant to consider them.

AWS Elasticsearch changes that and allow us to provision new cluster with a few clicks. Maintenance becomes easier because Amazon takes care of upgrades and backups while we focus on our data and application.

We have many terabytes of data in multiple Elasticsearch clusters. We're not sure yet if and how we will migrate them all to this new offering, but we use ES all the time so will certainly be looking at this for new data projects we work on.

2. Amazon Lambda and the serverless architecture

We are also very excited about Amazon's push towards a serverless, event-based architecture. Lambda itself isn't new, but we were happy that Amazon is continuing to improve it, including new support for Python and scheduled, long-running Lambda functions. Most importantly, Lambda functions can now run within our VPC, so they can access our internal infrastructure. This expands what can be done in Lambda and gives us more ways to leverage it.

Amazon is developing an interesting architecture blueprint over Lambda, Kinesis, SNS and other tools. This event-driven architecture can be used to deliver end-to-end solution without any servers, virtual or physical along the way.
Just write idempotent, stateless functions and AWS will deal with running them at the right time, with scalability and high-availability built in.

It doesn't fit every use-case, but is very enticing for the cases where it does.

So far we've done some experimentation with Lambda and really like it. We definitely plan to introduce it to our production environment in more than one way during 2016.

3. MariaDB and Aurora on RDS

MariaDB support brings another high-performance SQL database engine to RDS. This announcement follows up on Aurora (Amazon's own proprietary db engine) and opens the door for using SQL at high throughput without having to pay for, or master the intricacies of a high-end commercial database.

As big as we are on NoSQL technologies at Totango, we also love SQL and use it all the time. Sometimes we even need it to run fast (-: . Honestly, before Aurora and MariaDB this wasn't really an option for us -- We would never consider implementing a large scale commercial Oracle database for many reasons. Having open-source, cheap, AWS hosted alternatives makes it more accessible.

MariaDB in particular is compelling because of its compatibility with MySQL. In many ways it is the next generation of MySQL, and is something we've been wanting to use, but had not had the bandwidth to transition to. Having it on RDS takes away at least some of this effort and we are very excited to try it out really soon.

4. Amazon WAF

AWS WAF is a web application firewall that helps protect your web applications from common web exploits that could affect application availability, compromise security, or consume excessive resources.

We always feel we could do more on security infrastructure. It's an area that requires significant time to master and an ongoing investment to keep up with the latest. Anything we can use to help us on this path is welcomed.

Application-firewall provide a good first line of defense by sitting in-front of your API or web gateway and filtering out common exploits 'on the wire'. Having this as part of AWS makes such a technology much more accessible to us. We are curious to evaluate it's capabilities and how it can help us keep our applications and data safe.

Interesting

We found these announcements interesting and will be taking a closer look at them in the coming months

  • Kinesis Firehose: An interesting service which allows you to setup a Kinesis stream that persists data into a S3 bucket. An interesting way to collect data from customers without having to build and maintain a collection gateway

  • Amazon Inspector: Amazon's version of a web vulnerability scanner. We use 3rd party tools for this, but it would be nice to consolidate this into AWS assuming it good

  • EC2 Spot Fleets and Spot Blocks: You can now bid for Spot instances for a duration of up to 6 hours. Lots of potential cost-saving for batch compute tasks.

  • S3 Infrequent access: Somewhere between S3 and Glacier, it allows you to tag buckets as "infrequently used" and pay less in exchange for a lower latency and availability SLA. It's great for data that is, well, infrequently accessed, but can't be archived in Glacier from an application perspective. It's another option to tune our storage needs. S3 is AWS's first service and AWS continues to add interesting features to it. If only Amazon makes its functionality consistent across regions and eliminates the occasional downtime we would happily declare it as perfect.

  • Amazon EC2 Container Service (ECS) Updates: Including support for CLI and availability-zone support. We're not yet on the Docker bandwagon, but when we jump on we'd love to use ECS to streamline work.

Noteworthy, but not for us

These are some of the announcements we found interesting in general, though not as applicable to our needs.

  • Snowball: Probably the coolest announcement this year. We don't really need it, but may order one just for the fun of playing with such a large, sturdy and automatic storage solution

  • Quicksight: The technology looks absolutely amazing. The main use-case is BI, primarily for folks that have their business data already stored somewhere on AWS since Quicksight automatically detects it. It's not that relevant for our application, but we are crossing fingers for our BI colleagues that this promising tool will deliver. It sounds mind blowing.

  • AWS IoT: We'll sit out the hype at this point (-:


Oren Raboy