In recent years, we've witnessed a significant transformation in data management, moving from the traditional ETL (Extract, Transform, Load) framework to the more agile ELT (Extract, Load, Transform) methodology.

In this blogpost I describe 5 building blocks that are crucial for the success of a data team.

In this research, we go deeper into the essential capabilities of a URP and how it differentiates from event stream processing (ESP) platforms, streaming DBMSs, and stream-enabled analytical DBMSs.

From answering questions to being a strategic partner.

In this reference article, author talks about data quality and list its seven dimensions, giving examples to make them palatable and some fun facts.

Since Dehghani (2019) articulated the principles of Data Mesh four years ago, the framework has been heralded as a new paradigm that overcomes many of the challenges data teams and organizations face. The cornerstone – Treating Data as a Product, more commonly referred to as "Data Products" – has received widespread attention (see What Exactly is a Data Product? by Sanjeev Mohan 2023).

In this new Product Mastery series, we’ll be presenting live powerhouse conversations with the world’s most influential product leaders.

ChatGPT has taken the world by a storm because of how eloquently it answers our questions and the outstanding experience it provides for users.

Design and ship valuable data products your company will actually use.

The data-as-a-product approach has recently gained widespread attention, as companies seek to maximize data value.

Today I’m joined by Vishal Singh, Head of Data Products at Starburst and co-author of the newly published e-book, Data Products for Dummies.

Over the past two months, I have been running a survey to get an understanding of what is going on in the data world.

Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Now that AI has reached the level of sophistication seen in the various generative models it is being used to build new ETL workflows.

Data structures and algorithms interviews make some data engineers scream and others cry. Although there are a rare few who smile when barraged with a list of difficult questions pulled from Leetcode.

Data engineering provides the foundation for data science and analytics, and forms an important part of all businesses. This book will help you to explore various tools and methods that are used for understanding the data engineering process using Python.

A data flywheel isn’t that different from a mechanical flywheel in the sense that it takes tremendous effort to put it in motion, but once it picks up speed it almost becomes self-sustaining.

One of the practices at the core of data engineering is ETL, which stands for Extract Transform Load. From the name, it is a 3-stage process that involves extracting data from one or multiple sources, processing (transforming/cleaning) the data, and finally loading (or storing) the transformed data in a data store.

Data fabric, data lakehouse, and data mesh have recently appeared as viable alternatives to the modern data warehouse. These new architectures have solid benefits, but they're also surrounded by a lot of hyperbole and confusion. This practical book provides a guided tour of each architecture to help data professionals understand its pros and cons.

In today’s work, the goal of every organization is to ensure that data is managed properly which meets their business needs for information and more.

Let’s explore the Medallion Architecture. An approach to organizing data that many businesses are using.

It’s not easy for data teams working with batch workflows to keep up with today’s real-time requirements. Why? Because the batch workflow — from data delivery and processing to analytics — involves a lot of waiting.

Becoming a data-driven organization remains one of the top strategic goals of many companies. Data-driven means placing data at the center of all the decisions and processes that are made in the organization.

A solid data architecture is critical to the success of any data initiative. But what is meant by “data architecture”? Throughout the industry, there are many different “flavors” of data architecture, each with its own unique value and use cases for describing key aspects of the data landscape.

There has never been a better time to create impact with data & analytics. More and more data is available, computing power is increasing fast and analytical techniques are getting mature.

In the digital era, data has truly become the lifeblood of organizations across various industries.

The promise of using a chatbot to derive data insights threatens the decades-old dominance of business intelligence reports and dashboards.

Understand with examples the similarities and differences between a “data product” and “data as a product”

Many organizations have invested in a central data lake and a data team with the expectation to drive their business based on data.

Microsoft Fabric has clearly been influenced by Data Mesh. If you are seeking a Data Mesh inspired vision for your organisation, Microsoft Fabric is a solid choice to help you drive that strategy forward.

A Data Product is a way to make a data asset reusable and consumable. Data assets can be datasets, dashboards, machine-learning models and much more!

There’s a disconnect between data science courses and the reality of working with data in the real world.

Learn how customers can build a hybrid, modern cloud data architecture on AWS with IBM Db2 for z/OS Data Gate on IBM Cloud Pak for Data, to access their mainframe data in IBM Db2 for z/OS in real-time or near-real-time.

There are millions of data products out there, some successful and others…less so. But the truly standout data products are the ones that change users’ behavior.

This blog post reviews mechanisms to ingest data into Snowflake and then covers best practices to optimize costs incurred for data ingestion.

This blog post is the last part of ByteHouse's 5-part series titled Data Modelling: Unlocking Insights, One Model at a Time

A quick comparison between these two solutions, Databricks and Synapse Analytics, exploring as well Synapse Link: the hybrid transactional and analytical processing (HTAP) option, which allows direct and efficient connection to OLTP data in Cosmos and/or Azure SQL Databases.

Data is a mess at most companies. Data teams face multiple issues, such as handling human-edited Excels, duplicated metric definitions, incorrect source data, no data governance, lack of infrastructure, etc. If you

In our rapidly evolving digital age, data engineering has emerged as the backbone of the modern data-driven world.

How to develop the right data product for the right need, with the right technology, with the right data, for the right users?

Artificial intelligence is undoubtedly one of the hottest topics in the big data space today, and its importance will only increase in the coming years.

Transforming Data Management: Embracing a Spotify-Inspired Streaming Model for Seamless Access, Control, and Collaboration in Enterprises.

Tech's Complication Crisis: Simplifying Life by Embracing Simplicity Over Feature Overload

Embracing Data-Centric Architecture: Unlocking Operational Efficiencies and Solving Persistent Enterprise Challenges

Unveiling the Importance of Data Profiling: Streamlining Data Preparation and Enhancing Data Quality for Data-Related Roles

Automated tests play a critical role in ensuring the integrity and reliability of data pipelines, with advanced pytest features facilitating efficient testing for modern data practitioners

Master schema validation and become a data validation hero with msgspec and ChatGPT, ensuring robustness in data pipelines

Join Amy Raygada as she delves into the transformative power of Data Mesh, kicking off a mentality change within organizations. 🌐💡 Discover how Data Mesh goes beyond infrastructure, empowering teams to perceive data's value, break down silos, and achieve buy-in for successful implementation, all while minimizing resistance to change.

Exploring Unified Real-Time Platforms: A comprehensive guide to handling demanding streaming data and data at rest, integrating elements of ESP platforms and real-time data management for complex analytics and low-latency applications.

This article describes the genesis of Bitol, the open-source data contract standard and solutions that are an essential part of modern data engineering.

Like every new technology or concept, Open Data Contract Standard (ODCS) may require a quick getting started page. In this article, I detail how you can get started and contribute.

Open roundtable & Q&A around Data Mesh. Rain or shine, meet Scott Hirleman, Jean-Georges “jgp” Perrin, and many of their friends for an open discussion around Data Mesh. There are no bad questions.

As enterprises become more agile, centralization appears to be a thing of the past world – the waterfall world. The same appears to be true for data platforms. This is why Data Mesh, a new generation of data platforms, is being constructed.

Data engineering is a critical function in all industries. However, data engineering grows exponentially as the company grows, acquires, or merges with others.

All Things Open is a major open-source conference. It has been around for 12 years, has grown significantly, and survived the pandemic. It occurs at the Raleigh Convention Center in the middle of the North Carolinian capital.

Centralised data platforms have been around since the invention of the data warehouse (created last century by Ralph Kimble amongst others) to provide businesses the analytics capability they need.

There has been a lot talk about Data Products, especially with the rise of approaches like Data Mesh. This is important, as I do believe if done correctly, creating Analytics using Data Products can drive real business value and significantly speed up analytics deliveries.

With the hype of Generative AI, how do we keep focused on the goal of delivering the right valuable and business facing analytics and data use-cases with as low friction and maximum agility as possible?

The sharing economy has exploded in growth of the last 10 years. The sharing economy business model formalizes this desire and opens up new opportunities for engagement.

Have you ever been asked to show the impact of your data team? I have. It can be a pretty nerve-wracking question to receive as a data leader, especially when it's coming from your senior executive team (and the quality of your answer will determine whether you’ll be getting those resources you asked for).

ChatGPT is the talk of the town, with everyone asking - are the robots finally coming for our jobs? Join this panel of data leaders to hear about why they believe that ChatGPT and other AI tools are actually an analytical engineer's new best friend.

dbt - what it is, why you need it, and how to build a modern data stack, as well as data leadership and increasing representation of women at the top of the data pyramid.

Many data teams deal with “alert fatigue”–an overload of data test failures that make it difficult to separate signal from noise.

This talk is for data team leads and members facing hiring freezes, lay-offs, and budget cuts. There is increasing pressure on data teams to “do more with less”.

This post details the evolution of data platforms, highlights their problems, and why we decided to build a Data Mesh. I will detail the four principles of the Data Mesh, how to get started, look at the architecture, and describe some of the challenges.

Key Components, Recommended Tools, and Fundamental Development Concepts

In this episode, Scott interviews Manisha Jain, data engineer at thoughtworks. She shares her views on how to get going with data mesh in detail.

In the data mesh journey, the platform team plays a critical part: they build the self-serve platform. In this blog, we cover everything the platform team needs to consider when building it.

A deep dive into the four key principles in the Data Mesh platform.

This is the second blog in a two part series exploring how we created a streamlined developer experience in data mesh. You can read part one here, or, if you need more background, learn more about the Data Mesh architecture from a high-level perspective here. Everything here is based on our experience working with a large health care provider.

The deployment of data products is the crux of Stage 3 in the Data Product Lifecycle series.

The Data Product Lifecycle is not a linear but a continuous progression, and Stage 4, or the Evolution Phase, creates a concrete bridge to the initial Design Stage, contributing to the expansion and stabilisation of data products over time so domains can reap long-term value instead of kickstarting heavy and costly data initiatives every few months.

Focusing on the What, the Why, the Who, and the How, of data contracts, these data practitioners will bring to life both the theory and the practice of data contracts. Attendees will discover how data contracts are a force for good for any organisation that is building a domain-oriented, decentralised data platform, and how to drive data culture change with data contracts; and how to design and how to implement a data architecture based on data contracts.

A panel discussion on data mesh & data contracts with Andrew Jones and Amy Raygada.

The medallion architecture, championed for its focus on data quality and flexibility, might be losing its shine.

This article is the second in a series about cleaning data using Large Language Models (LLMs), with a focus on identifying errors in tabular data sets.

To minimize hallucinations in large language models (LLMs) post-training, or post-fine-tuning it is crucial to attend to the quality of the dataset, its geometry, its content and to curate the pre-training data carefully.

Hear how Chad’s team is building out one of the most advanced Experimentation and Machine Learning Platforms in the world from the ground up, servicing thousands of carriers every day to ship freight more efficiently.

Dive deep into the fascinating world of data contracts and their ever-growing importance in modern data management. Prepare to explore the role of contract-driven data governance and its profound impact on businesses in the digital age.

This episode contains in-depth discussion of data contracts. Are they a solve for the collaboration challenges between producers + consumers that impact data quality?

While data contracts are not new, our modern federated implementation leveraging schema registries and CI/CD checks has much less organizational overhead than hand-written data-SLA documents.

Chad dives into everything data platform product management.

Chad discusses the complexities of managing the data supply chain, emphasizing the importance of data quality, feedback loops, and aligning incentives within organizations.

We are starting to see a sheer abundance of event data. Events can be collected from frontends, backends, databases, streams, webhooks. But events are technical and usually don’t explain important business aspects. The concept of an activity layer makes this translation while it is defined from a business perspective and events are mapped towards business activities.

We’re in the midst of an economic shift in the tech industry, which has put increased pressure on data teams to demonstrate impact/ROI while reducing costs.

Timo discusses the challenges faced by product managers, his journey into product management, and the technical skills he deploys. He spoke about event data and marketing data, as well as the people, processes, and design involved in event tracking.

A framework for designing event data: The double three-layer framework.

Many enterprises are investing in their next generation data lake, with the hope of democratizing data at scale to provide business insights and ultimately make automated intelligent decisions.

Listen to learn more about the challenges common in many organizations, pillars, benefits, and vision of the Data Mesh concept, common points of confusion in understanding Data Mesh, data observability, data quality, and many more topics.

The true purpose of analytics.

Ergest Xheblati is the author of Minimum Viable SQL Patterns, and he's spent the last 15+ years mastering data modeling.

In this episode, Tim, Juan, and Ergest Xheblati, will live discuss why data modeling should be at the core of an enterprise.

A data catalog user interface should look like a search engine - a search bar and a company logo, thats it.

I have often found that high level data stakeholders have different understandings of the word "classification".

Ole Olesen-Bagneux joins the show to chat about the future of data catalogs and much more!

In this blog post, we’ll explore how to build a simple AI chatbot, enhance it with memory capabilities, and finally, implement a feature to download conversations for further fine-tuning.

Retrieval-augmented generation (RAG) has been empowering Conversational AI by allowing it to access and leverage external knowledge bases.

Event sourcing is a software design pattern that involves storing a log of all changes made to an application’s data as a series of events.

This is the first of a two-part series exploring the scope of data products in a dynamic landscape. This article delves into managing the evolving scope of individual data products in a Data Mesh architecture.

Metrics are one of the most important and durable primitives in a company. And if data teams are looking to create value FAST… they have to learn how their businesses work.

Metrics are the most important primitive in the data world and driving the use of powerful and reliable metrics is the best way data teams can add value to their enterprises.