[{"title":"Using AWS Serverless Architectures with legacy data processing","seq":3,"msg":"Recently we had the pleasure of implementing a serverless architecture on Amazon Web Services infrastructure to deliver a solution that is both durable &amp; robust, while also being scalable &amp; cost effective.&nbsp;<br><br>To my surprise, all this was possible even with a requirement to import and process legacy data.&nbsp; Using Docker containers on ECS FARGATE makes it easy to run tasks that otherwise aren't suitable for computation with Lambda, either because of it's execution time limit when processing lots of data or library and environment requirements.<br><br>For the purposes of this Business Intelligence Dashboard App, keeping the architecture serverless has several advantages:<br><br><b>Resilience</b>.&nbsp; Using robust services that are well tested saves you from reinventing the wheel.&nbsp; Leveraging existing AWS services means focusing more time on your application.<br><br><b>Scale</b>.&nbsp; The size and power of AWS means you don't ever have to worry about provisioning and migrating to larger resources as you grow without ever compromising performance.<br><br><b>Cost effective</b>.&nbsp; Start tiny and only pay for what you use.&nbsp; Growing costs scale with usage and growing revenue.<br><br>Those features gave us enough confidence and reason to setup an architecture without a traditional dedicated cloud virtual machine server to power the website.<br><br>You might be able to get a jump-start on your application with scaffolding from either the <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws-amplify.github.io/\">Amplify</a> project or <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/mobile/\">AWS Mobile</a>, but using services off-the-self is easy too if you want to power an kind of app, like ours in Vue.js.&nbsp; Writing <a rel=\"nofollow\" target=\"_blank\" href=\"https://en.wikipedia.org/wiki/Infrastructure_as_Code\">Infrastructure as Code</a> with CI/CD tooling makes maintenance a breeze.<br><br>Using <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/s3/\">S3</a>, <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/route53/\">Route 53</a>, and <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/cloudfront/\">CloudFront</a> is great for hosting your website's static assets like html, javascript, stylesheets and images.&nbsp; You'll be able to download them quickly, from anywhere in the world thanks to Amazon's Global CDN.<br><br>Use <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/api-gateway/\">API Gateway</a>&nbsp;and <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/lambda/\">Lambda</a> functions to access data from <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/dynamodb/\">DynanoDB</a> and <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/s3/\">S3</a> based on <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/cognito/\">Cognito</a> user authentication.<br><br>Upload legacy data files into an S3 bucket to trigger execution of a Lambda function that adds the even to an <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/sqs/\">SQS</a> queue and launches an <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/ecs/\">ECS</a> <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/fargate/\">Fargate</a>&nbsp;task with our docker container to process the queue, importing the file data into DynamoDB and S3.&nbsp; There is a <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/\">nice guide</a> available from Amazon too.<br><br>Bringing it all together also requires <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/iam/\">IAM</a> roles for permissions and <a rel=\"nofollow\" target=\"_blank\" href=\"https://aws.amazon.com/cloudwatch/\">CloudWatch</a>&nbsp;for monitoring logs of course.","slug":"using-aws-serverless-architectures-with-legacy-data-processing","wistia":{"id":""},"wistiaAudio":{"id":""},"youtube":{"id":""},"headerTitle":null,"headerCaption":{"color":"#000000","html":""},"tweet":{"id":""},"at":"2018-10-05T08:24:00.000Z","owner":{"id":"533f1d7b84872f30431febfa","name":"jeff"},"feed":{"id":"5bb71fad21d296001a90c46b","at":"2018-10-05T08:24:13.027Z"},"avatar":{"filename":"uploads/AWS Serverless Mobile Dashboard App Cloud Architecture_1.png","ref":{"col":"files.files","id":"5bb7d04e2c61ce001a9fb2bb"},"exif":{"ExifTool Version Number":"10.20","File Name":"upload_c0ee6e985bcf1dd3c7735afa69c6644f.png","Directory":"/tmp/app","File Size":"85 kB","File Modification Date/Time":"2018:10:05 13:57:50-07:00","File Access Date/Time":"2018:10:05 13:57:50-07:00","File Inode Change Date/Time":"2018:10:05 13:57:50-07:00","File Permissions":"rw-r--r--","File Type":"PNG","File Type Extension":"png","MIME Type":"image/png","Image Width":"928","Image Height":"719","Bit Depth":"8","Color Type":"RGB","Compression":"Deflate/Inflate","Filter":"Adaptive","Interlace":"Noninterlaced","Image Size":"928x719","Megapixels":"0.667","":null},"groups":["public"],"owner":{"id":"533f1d7b84872f30431febfa","name":"jeff"},"at":"2018-10-05T20:57:50.000Z","height":719,"width":928,"mimeType":"image/png","sizes":{"full":{"id":"5bb7d04e2c61ce001a9fb2c1","filename":"images/full/uploads/AWS Serverless Mobile Dashboard App Cloud Architecture_1.png"},"medium":{"id":"5bb7d04f2c61ce001a9fb2c6","filename":"images/medium/uploads/AWS Serverless Mobile Dashboard App Cloud Architecture_1.png"},"small":{"id":"5bb7d04f2c61ce001a9fb2cb","filename":"images/small/uploads/AWS Serverless Mobile Dashboard App Cloud Architecture_1.png"},"square":{"id":"5bb7d04f2c61ce001a9fb2d0","filename":"images/square/uploads/AWS Serverless Mobile Dashboard App Cloud Architecture_1.png"},"thumb":{"id":"5bb7d04f2c61ce001a9fb2d5","filename":"images/thumb/uploads/AWS Serverless Mobile Dashboard App Cloud Architecture_1.png"}},"id":"5bb7d04e2c61ce001a9fb2bf"},"groups":["public"],"cachedAt":"2020-07-13T01:44:12.321Z","id":"5bb71fad21d296001a90c46a"},{"groups":["public"],"title":"Manipulate complex data using friendly spreadsheet interfaces in the browser","seq":4,"msg":"Managing big complex data sets isn't always easy.&nbsp; It's even more difficult when you need to be able to make changes quickly too.<br><br>Using what was once an offline only tool like spreadsheets in a browser, you can navigate large sets of data, using your trusty compass of rows and columns.&nbsp; Making edits is fast as you would expect in a desktop application yet seamlessly synchronizes with the cloud.<br><br><h2>More than just a spreadsheet</h2><br>In one case, edit's input on the spreadsheet don't change the source data, but are adjustment % values managed by administrators.&nbsp; Cells with an adjustment display along side the source value in either red (-) or green(+).<br><br>Using the industry standard segmentation and dimensions, users are able to easily manage the 3,600+ values, grouped into relevant tabs.<br><br>Harnessing the power of the spreadsheet interface, coupled with the custom needs of your application can make for an efficient data management experience.","slug":"manipulate-complex-data-using-friendly-spreadsheet-interfaces-in-the-browser","wistia":{"id":""},"wistiaAudio":{"id":""},"youtube":{"id":""},"headerTitle":null,"headerCaption":{"color":"#000000","html":""},"tweet":{"id":""},"at":"2018-09-02T17:58:00.000Z","owner":{"id":"533f1d7b84872f30431febfa","name":"jeff"},"feed":{"id":"5bb94bfb2c61ce001a9fcbad","at":"2018-10-06T23:57:47.517Z"},"cachedAt":"2018-10-09T18:50:09.714Z","avatar":{"filename":"uploads/Screenshot 2018-10-09 at 11.42.18 AM.png","ref":{"col":"files.files","id":"5bbcf6c84ab3a000172f5ed2"},"exif":{"ExifTool Version Number":"10.20","File Name":"upload_eb55b50c6fe870b5816b9d13dd1fec80.png","Directory":"/tmp/app","File Size":"65 kB","File Modification Date/Time":"2018:10:09 11:43:20-07:00","File Access Date/Time":"2018:10:09 11:43:20-07:00","File Inode Change Date/Time":"2018:10:09 11:43:20-07:00","File Permissions":"rw-r--r--","File Type":"PNG","File Type Extension":"png","MIME Type":"image/png","Image Width":"1130","Image Height":"569","Bit Depth":"8","Color Type":"RGB","Compression":"Deflate/Inflate","Filter":"Adaptive","Interlace":"Noninterlaced","Image Size":"1130x569","Megapixels":"0.643","":null},"groups":["public"],"owner":{"id":"533f1d7b84872f30431febfa","name":"jeff"},"at":"2018-10-09T18:43:21.000Z","height":569,"width":1130,"mimeType":"image/png","sizes":{"full":{"id":"5bbcf6c94ab3a000172f5ed8","filename":"images/full/uploads/Screenshot 2018-10-09 at 11.42.18 AM.png"},"medium":{"id":"5bbcf6ca4ab3a000172f5edd","filename":"images/medium/uploads/Screenshot 2018-10-09 at 11.42.18 AM.png"},"small":{"id":"5bbcf6ca4ab3a000172f5ee2","filename":"images/small/uploads/Screenshot 2018-10-09 at 11.42.18 AM.png"},"square":{"id":"5bbcf6ca4ab3a000172f5ee7","filename":"images/square/uploads/Screenshot 2018-10-09 at 11.42.18 AM.png"},"thumb":{"id":"5bbcf6cb4ab3a000172f5eec","filename":"images/thumb/uploads/Screenshot 2018-10-09 at 11.42.18 AM.png"}},"id":"5bbcf6c84ab3a000172f5ed6"},"id":"5bb94bfb2c61ce001a9fcbac"},{"at":"2014-04-09T05:17:00.000Z","feed":{"id":"5344d7ed9cef652e4e74d929","at":"2014-04-09T05:17:33.150Z"},"groups":["public"],"headerCaption":{"color":"#000000","html":""},"headerTitle":null,"msg":"<blockquote>We follow the latest in <i>web development</i>, <i>technology</i>, <i>security</i>, <i>design</i>, <i>user experience</i>, <i>interfaces</i> and everything that's part of our digital lives.</blockquote>Subscribe to updates via <a rel=\"nofollow\" target=\"_blank\">email</a> and <a rel=\"nofollow\" target=\"_blank\" href=\"http://alternatingbits.com/api/posts?_format=rss\">rss</a>. &nbsp;Email us your thoughts directly to info@alternatingbits.com!","owner":{"id":"533f1d7b84872f30431febfa","name":"j"},"seq":1,"slug":"a-bit-about-this-blog","title":"A Bit About This Blog","tweet":{"id":""},"wistia":{"id":""},"wistiaAudio":{"id":""},"youtube":{"id":""},"cachedAt":"2018-10-10T03:09:54.564Z","id":"5344d7ed9cef652e4e74d928"}]