5 keys to a modern data architecture

Andrew Stewart and Will Vranderic

In recent years, modern data architecture has been an increasingly common topic when we meet with clients. Companies across all industries are realizing the value of analytics and want to make sure they’re able to fully leverage their data.

To best address this subject, we find it important to focus on the desired business outcomes instead of focusing solely on the architecture itself. Based on our experience, we've found the following principles to be critical to the success of an enterprise data program.

1. Get the right data to the right people at the right time

Data is only useful if people can act on the data in a timely manner. It must be accurate and actionable. However, “people” in this case means several different things. No matter if we’re talking about external people (e.g., customers, etc.) or internal teams (e.g., marketing, operations, etc.), data must be easily consumed, visual, and tailored to the targeted use case(s). Therefore, data needs to be delivered with the desired context and format of the persona and relevant for the individual.

When starting with a persona-based approach, it is critical to build your next generation data platform by focusing on the people you’re looking to serve. “Build it and they will come” isn’t a good strategy when it comes to data platforms, unless you’re highly in-tune with the end-user’s needs, and you have a way to mandate the tools they use. But, if you’re like most companies (and statistically speaking you are), we’ve found that people resist change and there is a science to tackling adoption challenges. An easy way to foster success in this area is including end-users in your project team early on and seeking their input often.

2. Develop a flexible architecture

Businesses are always evolving, and traditional data architectures can struggle to keep up, particularly those built on highly relational data models. Simply put, data can be rigid. However, you can design your architecture to allow for flexibility in the types of data you ingest and the ways you deliver information to various personas.

With the advent of data lakehouses, which combine the best features of data lakes and data warehouses, it's now possible to handle both structured and unstructured data more effectively. Newer AI technologies and techniques also make it easier than ever to enrich all data products with appropriate technical and business metadata.

Here are some best practices to consider for a modern, flexible data architecture:

Minimize data transformations: Simplify the layers within your data processing pipeline to reduce complexity and improve agility.
Virtualize data views: Look for opportunities to virtualize the business’ view of the data, minimizing the need to frequently physicalize data models.
Utilize high-performance storage: Leverage high-performance storage tiers to eliminate the need for legacy performance techniques such as “Cubes” or summary tables.
Empower experimentation and rapid prototyping: Establish data sandboxes or refineries to provide business analysts with access to explore ideas and experiment with large datasets. Quickly build real data-driven prototypes to identify and prioritize the highest value use-cases and “fail-fast” to avoid wasted effort.
Adopt event-driven architecture: Use event-based data streams (e.g., Kafka) to quickly and efficiently move data to the right locations, ensuring timely access to up-to-date information.

Credera’s Data Lakehouse Modern Data Reference Architecture with sample tool selection

3. Scale up (and down) quickly

In today's cloud-driven data environment, leveraging the cloud's inherent scalability is crucial. Here’s how you can make the most of it:

Auto-scaling capabilities: Utilize auto-scaling features in cloud platforms and data services to dynamically adjust resources, ensuring optimal performance without over-provisioning.
Cost monitoring & efficiency: Take advantage of cloud cost management tools to monitor and optimize spend, ensuring you only pay for the resources you need.
Rapid experimentation: Use cloud technologies to quickly spin up environments for testing and validating new data sources and ideas. This approach helps prioritize valuable data for your architecture, avoiding unnecessary data ingestion or data product development.
Managed services: Focus on your data strategy and analytics and let managed cloud services handle the infrastructure. As an example, services like AWS Glue, Google Dataproc Serverless, and Azure Databricks enable you to run Spark code without provisioning or managing servers.
Security and compliance: Leverage the advanced security features and compliance certifications offered by major cloud providers to protect your data and meet regulatory requirements.

4. Commit to security from the beginning

Security remains a critical concern and should be a primary focus throughout your data project. Designing for security from the outset helps avoid issues down the road and ensures that your data remains protected.

Shift-left security: Incorporate security measures early in the development lifecycle. This approach, known as "shift-left security," ensures that potential vulnerabilities are addressed before they become critical issues.
Persona-based security: Use a persona-based approach to define security requirements that meet the needs of all users. This method helps tailor security measures to different roles and access levels within your organization.
Data encryption: Apply encryption to data at rest and in transit to safeguard sensitive information. Cloud providers offer robust encryption services that can be easily integrated into your data architecture.
Access controls: Implement fine-grained access controls through one of the common approaches – RBAC (role-based access controls), TBAC (tag-based access controls), or ABAC (attribute-based access controls) – to ensure that users only have access to the data necessary for their roles. This minimizes the risk of unauthorized or unintended access.
Unified security policies: Instead of relying on platform, technology, or data store-specific security measures, establish unified security policies that can be applied consistently across all data sources and environments. This approach enhances overall security and simplifies management.

5. Account for machine learning and artificial intelligence

With the rapid advancement of AI and machine learning (ML) technologies, incorporating ML into your data strategy is more accessible and impactful than ever. Tools like AWS SageMaker, Google Cloud AI Platform, and Azure Machine Learning provide robust platforms for developing and deploying ML models.

Plan for ML readiness: Even if ML feels daunting now, it's crucial to design your data architecture with future ML capabilities in mind. This foresight will enable your business to leverage advanced analytics and predictive solutions when the time is right.
Data quality and organization: Ensure your data is clean, well-organized, and accessible. High-quality data is the foundation of successful ML models, and maintaining data hygiene will streamline future ML initiatives.
Evaluate ML tools: With all cloud and software providers leaning into this area heavily over recent years, regularly assess the latest ML tools and platforms to understand their requirements and capabilities. Consider factors such as scalability, integration with existing systems, and ease of use when selecting tools.
Automated ML (AutoML): Leverage AutoML tools to automate the process of training and tuning ML models. Platforms like Google AutoML, H2O.ai, and DataRobot can help democratize ML by enabling non-experts to build high-quality models.
Model deployment and monitoring: Plan for seamless deployment and continuous monitoring of ML models. Use MLOps practices to manage the lifecycle of ML models, ensuring they remain accurate and effective over time.

Modern data architecture for your organization

Given the importance of data in today’s market, it is critical to make smart decisions when investing in a modern data architecture. While implementations may vary from business to business, we have found these principles to be consistent for successful projects.

If your company is wondering how to put these principles into practice, reach out to us to start a conversation. We’d love to help as you think through how your company can build an enterprise data program that sets you up for long-term success.