Fun with the Twitter Stream API - ELK Stack, RabbitMQ, Redis and High Performance SQL

Dave Mateer Profile Picture Dave Mateer
Taking an existing system which consumes 4.3m Tweets per day then making it more Scalable and Testable. What happens if we need 10x the scale? Will look at: Signing up to the Twitter API Simplest possible C# app working consuming the Twitter stream: Making more scalable with RabbitMQ ELK (Elasticsearch, Logstash, Kibana) stack Caching with Redis SqlBulkCopy ETL patterns This application has been in development for 6 months. It is born out of 7 years of writing similar enterprise software - can my system handle 2x or 10x the load if I need it? Can I prove my system is working ie Auditable/Testable and Dashboard overviews.