Data Engineering and Analytics on AWS

This is a major question covering several subjects. Start with your business outcome, what is it you are trying to achieve for your business through the use of data.  After you determine the outcome, you need to think about data ingestion and ETL, along with how you plan to query and visualize. Services like AWS Glue and Amazon S3 can be critical, with tiered storage based on access frequency for cost, and you’ll want to consider how you will query, with services like Amazon Athena, and how you plan to visualize, like with Amazon QuickSight. You may want to segregate data by use case, to help your stakeholders stay organized, but as you can see it is key not to think of the data lake in isolation—understand where it stands in relation to your users and data sources.

For any compliant solution, you need to consider how you’re handling encryption at rest in services like Amazon S3, Amazon Redshift, and Amazon RDS, as well as encryption in transit. This is also about instituting correct IAM policies for protecting sensitive data access and governance. If you’re storing any personally-identifiable data, you may also want to consider adopting Amazon Macie to ensure it stays appropriately secured or scrubbed where necessary. Services like Amazon Comprehend can also be valuable for a redaction stage, if you need to hide sensitive data as part of an analytics workflow.

You need to design your architecture to leverage services like AWS Lambda while considering issues of state management and design for processing logic. Amazon S3 obviously provides serverless benefits, as does AWS Glue, so we often recommend these in conjunction. We may also suggest you consider serverless options for data querying like Amazon Athena and Amazon Redshift.

AWS SageMaker can be a great starting point for working with ML models on your data. With SageMaker Studio, you can do exploratory work in shared notebooks, giving your data science team the ability to collaborate and iteratively work out a solution, while only using resources during experimentation. You’ll also want to develop an architecture that lets you seamlessly retrain as new data becomes available.

Centralizing metadata and having a unified cataloging strategy is crucial to ensure that data is discoverable, its lineage is traceable, and stakeholders across the organization can find and utilize what they need in an efficient manner. AWS Glue is powerful for data cataloging and discovery, and for querying across these datasets you may want to consider tools like Amazon Athena or Amazon Redshift Spectrum.