Dagster Labs

I used Dagster Labs  in a previous assignment in 2024, where I was working on a POC that required DBT and Dagster Labs  integration for ETL pipeline orchestration. In that project, I worked mainly on DBT, and we selected Dagster Labs for the orchestration purpose so that we could create an ETL pipeline with orchestration and scheduling jobs.

The main use case I worked on for Dagster Labs was the DBT with Dagster Labs integration, where all our ETL jobs were built into DBT. There was a need to create a lineage and also the orchestration, and for the orchestration purpose, we used Dagster Labs.

Dagster Labs helped with lineage and orchestration in my project by allowing us to see what the impacted upstream and downstream systems were, which was a limitation in DBT. It enabled us to create dependencies in DBT, and I saw the flexibility of Dagster Labs where we could directly integrate it seamlessly with the DBT tool. All the ETL jobs could be imported directly into Dagster Cloud, and we could create the dependencies and schedule the jobs based on the frequency decided by the business, and it was quite seamless.

The best features Dagster Labs offers mainly include error handling, job scheduling, and lineage visualization, with all three being quite easy to manage. After fixing the code, we do not have to rerun the pipeline again; it automatically restarts. There is also an option to put data validation and data contracts within Dagster Labs. If there is any data quality issue that happens from the source, we can put the data validation checks over there, and Dagster Labs can easily check that and based on the results or the outcome of it, Dagster Labs can run the pipeline. The data lineage part is quite visible, making it easy to see what is the source, what is the target, and what are the in-between intermediate layers we have put in place for the pipeline.

The data contract or data validation part is a great feature because it has enabled us to do our quality checks at the runtime and that has saved a lot of manual efforts from the engineering team.

Dagster Labs has impacted my organization positively by enabling faster project delivery since it has reduced a lot of manual efforts, especially the manual scheduling part. With Dagster Labs, we have scheduled all our jobs and orchestration automatically, and we did not have any data lineage tool or orchestration tool before. After onboarding Dagster Labs, we have achieved some good results.

In terms of project delivery, it has reduced the effort significantly because a lot of manual things were automated, resulting in our timelines being met on time. Earlier, it was getting delayed, but we have made our deliveries right on time, so that was the really outstanding outcome.

There are multiple products from Dagster Labs: Dagster Cloud, Dagster Labs, and sometimes it is quite confusing to choose which one for what purposes. I know there are some licensing buckets for small organizations, medium organizations, and bigger organizations, so the modeling and the cost estimation part could be more intuitive, making it easier for beginners to understand how much time and money they would save by onboarding Dagster Labs in their project.

I deducted two points because I faced some challenges working with the Git  repository integration with Dagster Labs due to some security keys or some weird issue. I had to get in a call with Dagster Labs subject matter expert, and we really struggled for two to three weeks because we could not establish the secure connection through that.

I have been working in the Data and AI industry for almost 14 years now.

Dagster Labs is quite stable.

Dagster Labs is quite scalable, allowing us to scale it to enormous sizes.

The customer support was good; I had a weekly call with them when I was running into some issues, and they helped me out promptly.

I would rate the customer support nine out of 10.

We were not using any previous solution, so that was the first experience, having onboarded Dagster Labs.

The initial setup is quite good. I am really impressed.

The documentation is quite clear, and the integration is seamless as I mentioned, such as with DBT, which was good. The user experience was also good.

I do not have the numbers with me, but we have definitely reduced the manual efforts, resulting in one or two fewer employees needed and some productive tasks being accomplished. We have saved some time over there.

The pricing, setup cost, and licensing were quite intuitive but not as clear for the data architect to understand what would be the cost estimation when onboarding Dagster Labs.

We were thinking of using Airflow , but after comparing it with Dagster Labs, Dagster Labs came out as the winner, leading us to decide to go ahead with it.

One of the user experiences was that earlier DBT was a back-end tool, and Dagster Labs is something we can actually show to stakeholders with all the job run logs. It is easy to monitor all your running jobs and also what has been completed and what has errored out, and also the error status messages are quite descriptive. If there is any error that occurs, you can look into the error log details and then easily debug that and identify the error and fix it. If there is any failure that happens at one checkpoint, we do not need to restart the job from the start; from that checkpoint itself, we can resume the job after fixing the code, and it was quite seamless.

I would advise others to start using Dagster Labs and explore their functionalities and features because it is easy to onboard, and their cloud-native features are really awesome. It has a great lineage with all the ETL tools and also with GCP  and other cloud platforms, so I would definitely recommend others to start using it.

From the security perspective, everything was good, and I do not have any feedback on that, to be honest.

It met all the organization's requirements. I would rate this review an overall 8 out of 10.

The main use case for using Dagster Labs  is to utilize ETL processes such as extract, transform, and load because we already have raw data, and we perform all the necessary transformations, including feature engineering and mathematical calculations, and then we load it to the front end so that customers can see how their data has evolved into something useful, allowing them to make actionable decisions.

Let me consider a scenario where a customer has raw and unreadable data. We need to identify features first. For example, a customer may have IP addresses with last seen timestamps along with raw security logs. During the initial feature engineering phase, we check for use cases such as detecting if a user on the customer side is using unauthorized resources. To accomplish this, we need to perform feature engineering on several attributes, such as the user's authorization level and the objects they are accessing. For a second example, we can detect if a user is authorizing company access after work hours by using the last seen attribute. We can perform feature engineering on such scenarios.

That covers my main use case, but we are also currently trying to use the DLT tool by using Dagster Labs  so that we can extract the data as well.

The best features Dagster Labs offers are that it is very beginner-friendly, the UI is very nice, and we can backfill most of the processes. The backfill feature is the best feature I would consider.

When any new user comes in, they can really see the UI and do all sorts of things rather than going into code-level work. When it comes to backfill, if an error occurred in the past and we want to retrieve the data from that point, we can do that using backfill, which makes it good for our workflow.

Dagster Labs has positively impacted our organization by making it much easier to create data pipelines, helping us create new use cases, and allowing us to transform all of the data we have into features, which Dagster Labs has greatly assisted with.

I can say that since using Dagster Labs, we have saved time, increased productivity, and also cut down on some third-party vendor applications, which saves us money in infrastructure costs and other expenses.

Dagster Labs is currently providing a Dagster University course, but those offerings are somewhat high-level. When a user comes in and tries to learn Dagster Labs, there need to be different kinds of projects that they can understand. I do not think that Dagster University course has that.

Also, when we are doing something in Dagster Labs, we need to know that when I make a change in the UI, it needs to be reflected in the code, so as a developer, I can check on the code and understand the workflow. The UI needs to be connected to the code in real-time.

I have been using Dagster Labs for the past four to five months.

Dagster Labs is stable.

Dagster Labs's scalability is good because you can easily create different pipelines for different purposes, which is beneficial.

I would advise others looking into using Dagster Labs to utilize all of the features provided by Dagster Labs, as it has many great offerings. I would rate this product a seven out of ten.

I primarily use Dagster Labs  for asset-based orchestration, scheduling and automating pipelines, and visualizing dependencies between datasets, and testing it. I test unit test cases in the pipeline.

A practical example of how I use Dagster Labs  for asset-based orchestration is when I take the raw orders, clean orders, sales fact tables, daily revenue summary, and VBA dashboard. I use Dagster Labs to track the dependency between these assets. When a raw table arrives, the dependent assets are recomputed. If the sales table fails due to a schema change, Dagster Labs prevents downstream assets from running, highlighting the failed assets in the lineage graph. This allows me to check the logic behind the data, and it also allows me to rematerialize only the affected assets after the fix. This asset-centric approach makes it more reliable and easier to debug when orchestrating individual assets.

I also use Dagster Labs to create scheduled pipeline jobs for running the pipeline.

One of the best features that Dagster Labs offers is the asset-aware data orchestration.

Asset-aware data orchestration is valuable for me because it becomes easy to find out when an asset fails and how the asset failed when the entire pipeline runs. It becomes easy for me to fix the problem, rerun the asset, materialize it, and see it in the lineage graph.

Dagster Labs has positively impacted my organization because whenever there is a change in the data, it becomes easy for me to schedule the job, run the changes, and make changes to be imported to the slowly changing dimension tables. It makes my work easier.

I think it has been great for me to use Dagster Labs, and I would not suggest any areas where it could be better because I have not fully utilized Dagster Labs to its fullest potential. It has been great for me.

I do not have any improvements needed for Dagster Labs that I have not mentioned, and there is nothing else I would like to see changed or added.

I have been using Dagster Labs for quite some time, approximately one and a half years.

The specific outcomes I have noticed since using Dagster Labs include saving time and reducing errors. It becomes easy when I run the pipeline in everyday cases instead of someone manually running it. Dagster Labs makes it easy to run scheduled pipelines more often when a large chunk of data comes into the data table or the database. It reduces dependencies on manual work and personnel, and is also efficient in consuming the data that has changed in the SCD data tables.

I have not used Dagster Labs's AI capabilities, so I have no idea about its governance and security.

Regarding Dagster Labs's AI capabilities, I have not used it, so I would not give any metrics on its accuracy or reliability of output because I have not relied more on the AI capabilities of Dagster Labs.

Dagster Labs is deployed in my organization in a private cloud.

This interview was great, and I would not recommend any changes for the future. I gave this review a rating of eight.

I perform orchestration around data by connecting raw app data using Fivetran  inside BigQuery  warehouse to DBT. I do this entire process using Dagster Labs .

My workflow is that Fivetran  reads data from the raw data source and sends the data to BigQuery . From BigQuery, the data is read inside the BI dashboards. The entire pipeline from raw to Fivetran to BigQuery to BI dashboard is monitored inside Dagster Labs .

What I find beneficial with Dagster Labs is that it excels at pipeline monitoring. It provides good notification of schema changes and indicates whether the pipeline is stable or not.

I appreciate the visual lineage graph that Dagster Labs offers. It shows exactly which upstream table or API pull failed, timed out, or produced an anomaly.

Dagster Labs has given me better monitoring ability. I know whether data can be trusted based on what I see on the dashboards. This gives me more confidence in my data sources.

Dagster Labs has provided a good sense of the data pipeline I have been using. I have started to trust it even more because I have full confidence in whether it has been updated correctly and if it has failed. When there are any anomalies, I end up trusting my data even more. I have full confidence and control over the data pipeline. Overall, it has created a very positive impact.

One of the features I appreciate in Dagster Labs is resources, which is a cool capability to have.

There are many ways Dagster Labs can be improved. I believe the UI is very slow and it prevents loading state.

Regarding needed improvements, I think there should be very good support available for migration if someone is migrating from a legacy system to Dagster Labs. Maybe that support needs to be better in a way that there should be customer service available for it, with more documentation or GitHub  documentation that details what you should do and should not do, including do's and don'ts and how it will be beneficial or not beneficial. Something approach may be helpful during migration.

I have been using Dagster Labs for around a year.

It has definitely improved my downtime by twenty percent in relation to Dagster Labs.

Based on how Dagster Labs works, I have more confidence in them. They would surely have a very good model to support all of the features they are providing.

My advice to others looking into using Dagster Labs is to take it slow and give time for things to develop. Things will get better.

I would rate this product an eight out of ten.