BigQuery differentiation for Teradata Migration
A decade ago, Marc Andreessen famously said in an article in the Wall Street Journal that Software is eating the world. Today we know that if software is the engine then data is the fuel. Today, data is feeding the world. The world is generating more data than ever before. But how do we make sense of this data? Traditional data warehouses were not designed to handle the explosive growth in data.
Teradata has been a leader in Data Warehousing technology for the past 20-30 years and has a huge installed customer base - Apple, Walmart, eBay, AT&T, Verizon, Citi, Bank of America, Coca-Cola, and Ford to name just a few. Today, the exponentially increased variety, volume, and velocity of data calls for a data warehouse solution that is more agile, global, and cost-effective. Those who have previously relied on Teradata’s solutions are faced with a question: continue to make investments in rigid, on-premises solutions, or face the one-time cost of migrating to an agile, cloud-based enterprise data warehouse (EDW) solution.
We are seeing that today clients are broadly looking at three possible options of upgrading from the on-premise Teradata implementation:
Option 1: Migrating to Teradata Vantage over a public cloud platform e.g. Google Cloud Platform (GCP)
Option 2: Use Cloud Native PaaS e.g. Google BigQuery
Option 3: Use 3rd party PaaS such as Snowflake, Cloudera, Databrick etc. deployed over a public cloud platform.
Of these three options, the Cloud Native PaaS provide the best value-for-money. A 2019 study1 conducted by ESG (Enterprise Strategy Group) shows that organizations can save up to 52% by using BigQuery over an on-premises EDW and up to 41% over an EDW deployed in the cloud. BigQuery is Google Cloud’s highly scalable, enterprise data warehouse solution. It is fully managed and serverless, offers real-time insights from streaming data, has built in ML ‘out of the box’ and has a high-speed in memory BI engine for faster reporting and analysis. BigQuery is designed to make data analysts and data scientists more productive and it is a secure offering that has data encryption by default, both in transit and at rest, which ensures customer data is protected.
The value proposition of BigQuery is based on 5 pillars:
Pillar 1: Jumpstart your data analysis
Accelerate time-to-value with a serverless, self-tuning and highly scalable modern data warehouse, that is easy to set up and manage, and doesn’t require a database administrator. Get up and running in seconds, and start querying from gigabytes to petabytes of data with standard SQL. Automatically move data from 100s of popular business SaaS applications into BigQuery with Data Transfer Service (DTS) or use data integration tools like Cloud Data Fusion, Informatica, Talend, and more, to ETL data at any scale from hybrid and multi-cloud applications. Get access to multiple data sources with federated queries and the Storage API and join public or commercial datasets with your data for richer insights. Enjoy flexible pricing models that offer predictability with flat rate and bundle pricing options.
E.g. Home Depot CIO is able to empower business units to create their own BigQuery projects and reduced dependency on IT budgeting for additional data warehousing capacity. This has resulted in wider adoption of the platform and faster time to value.
Pillar 2: Instant insight with real time analytics
Unlock real-time insights with BigQuery's high-performance streaming ingestion service that makes streaming data immediately available in the data warehouse for analysis. Query data on the fly and know what is happening right now. Stream IoT data from globally dispersed devices, at the edge or in the cloud, and make important data-driven decisions with ease. Build comprehensive batch and streaming data pipelines with Cloud Pub/Sub and Cloud Dataflow integrations and transform data with equal reliability and expressiveness.
E.g. Zulily is an online retail company in USA and they launch more than 9k SKUs every day on their online store. Zulily is able to analyze the product performance and pricing data in real-time and make changes to their online merchandizing and pricing in a much more agile manner. This has improved Zulily’s online conversion significantly.
Pillar 3: The power of advanced & predictive analytics
Operationalize machine learning and predict business outcomes easily without the need to move your data from the data warehouse. Build machine learning (ML) models using standard SQL with BigQuery ML or automatically create ML models by simply pointing Auto ML tables to BigQuery. Unlock advanced machine learning use cases with Cloud ML Engine and TensorFlow integrations, allowing you to train ML models on structured data in minutes. Run advanced geospatial analytics with BigQuery GIS to gain location-based insights.
E.g. 20th Century Fox, a global movie studio is able to operationalize ML using simple SQL and create customer segmentation and targeting strategy for their new movie releases in matter of days. Their marketing analysts are able to execute on this project with ease and reduced dependencies on their lean data science team.
Pillar 4: Share data insights at scale
Seamlessly share analytical insights within your organization as datasets, queries, spread sheets and reports with a few clicks. Create public and external read-only datasets to scale knowledge sharing and collaborate with your external business stakeholders. Analyze large and complex datasets interactively with sub-second query response time by combining BigQuery BI Engine, an in-memory, column-oriented analysis service, with your favorite BI tools. Easily create stunning reports and dashboards using popular BI tools like Looker, Tableau, Google Data Studio, Google Sheets and more, out of the box and securely share insights at scale.
E.g. AccuWeather is a global leader in the weather data analysis. AccuWeather offers weather data to thousands of businesses. BigQuery allowed AccuWeather to create datasets and easy share the read-only datasets to their customer worldwide in a secure and transparent manner. This eliminated the need for costly FTPs and file transfers. AccuWeather is updating their data in one place and their customers are able to get the latest data. We are seeing similar patterns from Government agencies, world organizations, retailers, banks and other businesses.
Pillar 5: Protect your data and operate with trust
BigQuery offers robust security, governance and reliability controls that offer high availability and a 99.9% SLA so you can have peace of mind. Data is automatically replicated, restored and backed-up to ensure business continuity. Classify and redact sensitive data in BigQuery with Cloud Data Loss Prevention and leverage fine-grained identity and access management with Cloud IAM. Data is encrypted at rest and in transit by default, and customer-managed encryption keys provide control over your data. Seamlessly monitor your log data and events with native Stackdriver integration and eliminate surface-level attacks with BigQuery’s vertical integration within Google’s hardware and software security stack.
E.g. HSBC is a global bank and security is a top priority for HSBC. HSBC is using BigQuery’s customer managed encryption key, Data Loss prevention technologies and Cloud IAM to secure every bit of their data.
Accenture has developed a number of tools and accelerators to deliver the value from the migration to BigQuery. Through these assets, efforts savings of up to ~20-40% can be achieved as part of the end-end data movement. Some of these assets are listed here:
Pulse On-prem data landscape discovery, performance bottleneck identification & optimization facilitating strategic decision making before migration
Smart Schema Optimizer - Schema migration supported to facilitate validation by DBA prior to data movement. ETL code analyzer to provision replication of ETL mappings in target system
Smart Query Convertor - 50-60% automated ANSI compatible conversion of SQL views & queries
Smart DWH Mover - Historical data movement facilitated with basic sanity reconciliation. Additional flexibility to simultaneously move data to multiple targets in the same platform
Smart Data Lake Mover - Hadoop discovery and data migration for seamless movement to GCP
Smart Data Validator - Elaborate row-level & column level verification with potential to plug-in in multiple migration use cases
When all of these are taken into consideration, the opportunity for savings and benefits for organizations are astounding. Through the newfound ability to provide more comprehensive intelligence to the organization in a faster and more agile manner, large enterprise organizations could easily expect overall savings and benefits in the millions to tens of millions of dollars by migrating from on-premises EDW solutions to Google BigQuery.
1 The Economic Advantages of Migrating Enterprise Data Warehouse Workloads to Google BigQuery