I have been using this structure for over a year now and it has been working well. I use the term prep instead of base, however, it works the same. The real advantage is that all of the code for a model lives in the same directory of the parent model with as much depth as needed, without having to hop around the entire project hunting down fragments spread over base/intermediate/staging paths. This treats sub-models of a parent model somewhat like private methods or properties of a programming module, with the parent model being the main interface where it is all brought together and used in the database or downstream models.
Love it! I'm glad you find this approach sensible. And yes, I got the idea from traditional software development. It's rather non-traditional in data, but actually makes sense.
I have a question from this structure: so if we have a lineage on a specific domain with staging, intermediate and marts models how would that fit into this?
Yep, it makes sense. I'm just wondering if you break it down into additional folders to visually isolate staging / intermediate / marts models or just using like naming convention to address those in the main domain folder (with only the exception of base folder)?
Oh yes, I do create subfolders. Without those, more complicated marts are very hard to trace. I only leave "user-facing" assets in the main domain directory.
I have been using this structure for over a year now and it has been working well. I use the term prep instead of base, however, it works the same. The real advantage is that all of the code for a model lives in the same directory of the parent model with as much depth as needed, without having to hop around the entire project hunting down fragments spread over base/intermediate/staging paths. This treats sub-models of a parent model somewhat like private methods or properties of a programming module, with the parent model being the main interface where it is all brought together and used in the database or downstream models.
Love it! I'm glad you find this approach sensible. And yes, I got the idea from traditional software development. It's rather non-traditional in data, but actually makes sense.
I have a question from this structure: so if we have a lineage on a specific domain with staging, intermediate and marts models how would that fit into this?
Every domain is a top drirectory in your "models" dir, and everything domain-related goes into that respective directory. Does that make sense?
Yep, it makes sense. I'm just wondering if you break it down into additional folders to visually isolate staging / intermediate / marts models or just using like naming convention to address those in the main domain folder (with only the exception of base folder)?
Oh yes, I do create subfolders. Without those, more complicated marts are very hard to trace. I only leave "user-facing" assets in the main domain directory.
Ok, that's what I thought but just wanted to make sure I was not making wrong assumptions. Thanks for clarifying that.