Data Engineering

Data Engineering | News, how-tos, features, reviews, and videos

I've built data pipelines on AWS, Azure and Snowflake. Here's what Palantir Foundry did that surprised me

A senior data engineer's honest first impressions after a Palantir Foundry bootcamp, including what it does better, what it does not and whether the cost makes sense for your organization.

By Sashank Siwakoti

Aug 17, 20268 mins

Amazon Web ServicesData EngineeringMicrosoft Azure

Snowflake extends Cortex Code CLI to dbt and Airflow to streamline data engineering workflows

By Anirban Ghoshal

Feb 24, 20263 mins

AnalyticsData EngineeringDeveloper

Articles

Microsoft acquires Osmos to ease data engineering bottlenecks in Fabric

The acquisition could help enterprises push analytics and AI projects into production faster while acting as the missing autonomy layer that connects Fabric’s recent enhancements into a coherent system.

By Anirban Ghoshal

Jan 7, 2026 4 mins

AnalyticsArtificial IntelligenceData Engineering

Four important lessons about context engineering

Four big lessons, seven practical tips, three useful patterns, and five common antipatterns we learned from building an AI CRM.

By Md Abdul Halim Rafi

Nov 27, 2025 10 mins

Artificial IntelligenceData EngineeringGenerative AI

Improving annotation quality with machine learning

Treating annotation as a data understanding problem, rather than a labeling workflow challenge, can systematically drive down error rates and reduce the time and cost of producing high-quality data sets.

By Nick Lotz

Nov 20, 2025 9 mins

Data EngineeringGenerative AIMachine Learning

3 data engineering trends riding Kafka, Flink, and Iceberg

The Apache Kafka, Apache Flink, and Apache Iceberg communities are developing new ways for engineers to manage data and meet application needs.

By Adi Polak

Nov 25, 2024 5 mins

Data EngineeringData ManagementSoftware Development

Microsoft gives enterprises new reasons to adopt Fabric

At its European Fabric Community Conference last week, Microsoft released a volley of incremental updates to woo enterprises to its data platform.

By Anirban Ghoshal

Sep 30, 2024 7 mins

Business IntelligenceData EngineeringData Warehousing

Google adds Gemini to BigQuery, Looker to help with data engineering

Other updates include support for open source Apache Spark and Kafka integrations to BigQuery.

By Anirban Ghoshal

Aug 1, 2024 4 mins

Data EngineeringGenerative AI

I've built data pipelines on AWS, Azure and Snowflake. Here's what Palantir Foundry did that surprised me

Five ways to evaluate AI agent orchestration platforms

Five tips for developing data products

Don’t waste your next cloud outage

What can you do with quantum computing today?

Mastering the dull reality of sexy AI

Migrating from Apache Airflow v2 to v3

Anthropic alleges large-scale distillation campaigns targeting Claude

Snowflake extends Cortex Code CLI to dbt and Airflow to streamline data engineering workflows

Articles

Microsoft acquires Osmos to ease data engineering bottlenecks in Fabric

Four important lessons about context engineering

Improving annotation quality with machine learning

3 data engineering trends riding Kafka, Flink, and Iceberg

Microsoft gives enterprises new reasons to adopt Fabric

Google adds Gemini to BigQuery, Looker to help with data engineering

Resources

From alert noise to action: How 24 Hour Fitness modernized IT Ops with Jira Service Management

Atlassian: Shatter the service quo

Rovo Agents in action: Accelerating self-service deflection and support productivity