# The Ops Compendium
The central hub for learning all things Ops.
The Ops Compendium is your central hub for learning all things Ops—covering 80 topics across MLOps, DataOps, DevOps, DevSecOps, Architecture, and is continuously being updated. It’s similar to the [Deep Learning & Machine Learning Compendium](https://www.mlcompendium.com/) and is designed as an educational resource. Through it, I aim to help people learn and connect with the amazing authors whose work I’ve summarized, quoted, and referenced.
The Ops Compendium is a fully open project on [GitHub](https://github.com/orico/www.opscompendium.com) (please star it!).
{% embed url="" %}
The Ops Compendium Official GitHub Repo
{% endembed %}
I am committed to education and knowledge sharing, ensuring that this compendium remains not-for-profit and freely accessible. I envision it as a go-to resource for individuals at all skill levels—industry professionals, data engineers, machine learning engineers, DevOps practitioners, data scientists, and academics alike. This compendium is designed to save you countless hours of searching and filtering through articles of uncertain value, while also connecting you with exceptional authors whose work you can further support.
Please note that this is an ongoing project covering a wide range of topics. If you think something needs improvement or updates, you can easily contribute via [GitHub](https://github.com/orico/www.opscompendium.com) or [reach out to](https://www.linkedin.com/in/cohenori/) me directly.
Many Thanks, \
Dr. Ori Cohen
[My Website](https://www.oricohen.com/) |[ Medium](https://medium.com/@cohenori) |[ LinkedIn](https://www.linkedin.com/in/cohenori/) | [ML Compendium](http://www.mlcompendium.com/) | [State of GenAI](https://stateofgenai.com/) | [State Of MLOps](https://stateofmlops.com/) |
# Definitions
I asked ChatGPT "what are DataOps, MLOps, GitOps , DevOps and the DevSecOps professions?
The answer:
DataOps, MLOps, DevOps and GitOps are specialized areas within the field of operations management. DevSecOps is a combination of these areas with a focus on security.
DataOps is a set of practices that combines data management (data) and information technology operations (Ops). It aims to improve the speed and reliability of data analytics by automating data pipelines and promoting collaboration between data engineers, data scientists, and IT operations professionals.
MLOps, or machine learning operations, is a set of practices that combines machine learning (ML) and information technology operations (Ops). It aims to improve the speed and reliability of deploying machine learning models into production environments by automating the build, test, and release process, and promoting collaboration between data scientists, machine learning engineers, and IT operations professionals.
GitOps is a set of practices that uses Git as a single source of truth for declarative infrastructure and application code. It aims to improve the speed and reliability of deploying applications by using Git as the central source of truth for the entire delivery process, from development to production.
DevOps is a set of practices that combines software development (Dev) and information technology operations (Ops). It aims to shorten the development lifecycle and provide continuous delivery and integration of software.
DevSecOps is a set of practices that combines DevOps with a focus on security (Sec). It aims to improve the security of the software development and delivery process by integrating security practices into the development lifecycle and promoting collaboration between development, security, and operations teams.
Generally, individuals working in these specialized areas of operations management may be referred to as DevOps professionals, DataOps professionals, MLOps professionals, GitOps professionals, or DevSecOps professionals, depending on their specific area of expertise.
# Ops Definition Comparisons
* [Dev vs ops, vs devops vs sre - history and details by google.](https://www.youtube.com/watch?v=tEylFyxbDLE\&list=PLIivdWyY5sqJrKl7D2u-gmis8h9K66qoj\&index=2)
* [Sre vs devops](https://medium.com/hackernoon/sre-vs-devops-the-dilemma-f7054714525c)
* [Cloudops vs devops](https://victorops.com/blog/what-is-cloudops-vs-devops)
* [Itops vs devops](https://www.graylog.org/post/itops-vs-devops-what-is-the-difference)
* [AIOps](https://www.appdynamics.com/what-is-ai-ops/) - “AIOps platforms utilize big data, modern machine learning and other advanced analytics technologies to directly and indirectly enhance IT operations (monitoring, automation and service desk) functions with proactive, personal and dynamic insight. AIOps platforms enable the concurrent use of multiple data sources, data collection methods, analytical (real-time and deep) technologies, and presentation technologies.
* [Definition](https://dzone.com/articles/dev-vs-ops-and-devops)
# MLOps Intro
1. A [great intro about MLOPs](https://betterprogramming.pub/mlops-and-mlflops-795781d17989), what are the basic building blocks that you need to understand, in comparison to Data Engineering & DevOps - by Andrew Blance.
2. [Awesome MLOPs on github](https://github.com/visenger/awesome-mlops)
3. Analytics Vidhya
* [A list of MLOps articles base on the "MLOps" label](https://medium.com/analytics-vidhya/tagged/mlops)
* [A beginner guide](https://www.analyticsvidhya.com/blog/2021/06/mlops-a-beginners-guide-to-machine-learning-operations/)
* [A comprehensive guide](https://www.analyticsvidhya.com/blog/2021/06/mlops-a-beginners-guide-to-machine-learning-operations/) - there are quite a lot of details in this article that you should know only if you truely know the basics of ML lifecycle, feature engineering, deployment strategies etc.
* A two part series about MLOPs - Part 1: [connecting agile, data, devops, and technology.](https://www.analyticsvidhya.com/blog/2022/02/mlops-part-1-revealing-the-approach-behind-mlops/) Part 2: [going deeper into architecture, deployment, training](https://www.analyticsvidhya.com/blog/2022/02/workflow-of-mlops-part-2-model-building/).
* MLOps vs DevOps
* [The tip of the iceberg](https://www.analyticsvidhya.com/blog/2022/09/how-is-mlops-different-from-devops/) i.e., adding model + data to DevOps methodologies.
* [Another comparison, has some more details](https://www.analyticsvidhya.com/blog/2020/11/mlops-the-why-and-the-what/) - ML challenges, components
* [MLOps & Kubernetes ](https://www.analyticsvidhya.com/blog/2022/09/mlops-and-use-of-kubernetes/) - a very sparse intro
* [High level E2E architecture and explanations](https://www.analyticsvidhya.com/blog/2023/02/mlops-end-to-end-mlops-architecture-and-workflow/)
* [High level E2E concepts](https://www.analyticsvidhya.com/blog/2021/07/deepdive-into-the-emerging-concpet-of-machine-learning-operations-or-mlops/)
4. MLOps without Ops series [Part 1](https://towardsdatascience.com/mlops-without-much-ops-d17f502f76e8), [Part 2](https://towardsdatascience.com/ml-and-mlops-at-a-reasonable-scale-31d2c0782d9c), [Part 3](https://towardsdatascience.com/hagakure-for-mlops-the-four-pillars-of-ml-at-reasonable-scale-5a09bd073da), [Part 4](https://towardsdatascience.com/the-modern-data-pattern-d34d42216c81)
5. [As an engineering discipline](https://towardsdatascience.com/ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f)
# MLOps Teams
1. [Organizational structure for MLOps](https://www.newsletter.swirlai.com/p/sai-28-organisational-structure-for) by AURIMAS GRICIŪNAS - stages in a DS team production development, when do you need ML engineers and when do you need to split it to MLOps Engineers and MLEs.
2.
# MLOps Literature
1. [12 Factors](https://12factor.net/) - The Twelve-Factor App is a methodology for building scalable and portable SaaS applications. It emphasizes best practices like maintaining a single codebase, isolating dependencies, storing configuration in the environment, treating services as replaceable resources, and enabling fast, stateless, and resilient deployments across consistent environments.
2. [**Designing data-intensive applications reliable maintainable**](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321)
1. [**medium**](https://medium.com/@m_mcclarty/tech-book-talk-designing-data-intensive-applications-eb4908f2f6d6)
3. [**Google SRE**](https://landing.google.com/sre/books/)
1. [**Site Reliability Engineering**](https://www.amazon.com/dp/149192912X?psc=1\&pf_rd_p=0c07d3ef-dd9a-4ce4-8daa-7b9b90db3048\&pf_rd_r=F4QGSBSPA6DJJCX10WXK\&pd_rd_wg=RfKM8\&pd_rd_i=149192912X\&pd_rd_w=khBYM\&pd_rd_r=5bd80e38-7a30-41bb-ae0e-25a91dd1cb3d\&ref_=pd_luc_rh_crh_rh_sbs_sem_01_03_t_ttl_lh)
2. [**Site Reliability Workbook**](https://www.amazon.com/Site-Reliability-Workbook-Practical-Implement/dp/1492029505/ref=sr_1_1?dchild=1\&keywords=The+Site+Reliability+Workbook\&link_code=qs\&qid=1598257953\&sr=8-1\&tag=amznsearchff-20)
4. [**Designing Distributed systems patterns paradigms**](https://www.amazon.com/Designing-Distributed-Systems-Patterns-Paradigms/dp/1491983647)
5. [**Building Microservices: Designing Fine-Grained Systems** ](https://www.amazon.com/dp/1491950358/?coliid=I1H3OSVXC7XRBL\&colid=300M9JC4311P3\&psc=0\&ref_=lv_ov_lig_dp_it)
6. [**Jez Humble**](https://www.amazon.com/Jez-Humble/e/B003SNGS8E/ref=dp_byline_cont_pop_book_2)
1. [**Accelerate software performing organizations** ](https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339/ref=tmm_pap_swatch_0?_encoding=UTF8\&qid=\&sr=)
2. [**DevOps Handbook.**](https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002/ref=tmm_pap_swatch_0?_encoding=UTF8\&qid=\&sr=)
3. [**Continuous delivery deployment** ](https://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley-dp-0321601912/dp/0321601912/ref=mt_other?_encoding=UTF8\&me=\&qid=)
4. **Lean Enterprise**
7. [**Clean code**](https://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882) **(java) first chapters are good**
8. [**Clean code in Python**](https://www.packtpub.com/product/clean-code-in-python/9781788835831)
1. [**Medium**](https://medium.com/@m_mcclarty/tech-book-talk-clean-code-in-python-aa2c92c6564f)
2. [**git**](https://github.com/zedr/clean-code-python)
9. [**Domain-Driven Design**](https://www.amazon.com/Domain-Driven-Design-Reference-Definitions-Summaries/dp/1457501198/ref=pd_cart_crc_cko_mrai_1_1/146-7136232-7217867?_encoding=UTF8\&pd_rd_i=1457501198\&pd_rd_r=e0b19e54-c0c3-4ad1-abd0-eff99f815aee\&pd_rd_w=TP8KJ\&pd_rd_wg=ompBT\&pf_rd_p=77f3805b-bff9-40ee-9688-bcdb2cd9e197\&pf_rd_r=1WF281MJHVQYG506PTRX\&psc=1\&refRID=1WF281MJHVQYG506PTRX)
10. [**The complete python course** ](https://www.packtpub.com/product/the-complete-python-course-video/9781839217289)
11. [**pyVideo**](https://pyvideo.org/)
# MLOps Course
1. Weights & Biases
1. [Effective MLOPS for data scientists](https://www.wandb.courses/courses/effective-mlops-model-development)
2. [CI/CD using GitOps & W\&B](https://www.wandb.courses/courses/ci-cd-for-machine-learning)
# MLOps Patterns
## General Patterns
1. [ML product lifecycle patterns ](https://towardsdatascience.com/understanding-ml-product-lifecycle-patterns-a39c18302452)
2. [ML design patterns book repo](https://github.com/GoogleCloudPlatform/ml-design-patterns)
4. [MLOps Roadmap](https://github.com/cdfoundation/sig-mlops/blob/main/roadmap/2022/MLOpsRoadmap2022.md)
5. [Google’s Practitioners Guide to MLOps: A framework for continuous delivery and automation of machine learning](https://cloud.google.com/resources/mlops-whitepaper)
6. [State of MLOps](https://ml-ops.org/content/state-of-mlops)
Template
7. [Easy mlops with pycaret and mlflow ](https://towardsdatascience.com/easy-mlops-with-pycaret-mlflow-7fbcbf1e38c6)
8. [Challenges and solutions by iguazio](https://towardsdatascience.com/ml-ops-challenges-solutions-and-future-trends-d2e59b74dc6b)
## 
9. [Stanford CS329](https://stanford-cs329s.github.io/syllabus.html) - CS 329S: Machine Learning Systems Design - the course goes in-depth about how ML systems are built, and how to debug, root cause, monitor, etc.
10. Metaflow, medium [1](https://medium.com/bigdatarepublic/a-review-of-netflixs-metaflow-65c6956e168d) (high level review), [2](https://medium.com/acing-ai/decoding-netflix-metaflow-2ad84b36199e) (schema), [3](https://medium.com/analytics-vidhya/metaflow-by-netflix-the-good-the-bad-and-the-ugly-b7fc6a833484), [\*\*4](https://towardsdatascience.com/learn-metaflow-in-10-mins-netflixs-python-r-framework-for-data-scientists-2ef124c716e4) (amazing), [5](https://towardsdatascience.com/be-more-efficient-to-produce-machine-learning-pipeline-with-metaflow-db5f943ebbe7) (extra), [6](https://docs.metaflow.org/metaflow/data) (loading and storing data docs!)
11. HyperparameterHunter, [Hyperopt, mlflow, unit test, concept drifts, using python and kafka](https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e)
## Patterns in Practice
1. An MLOps End-to-End system, i.e., "[You dont need a bigger boat](https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat)", using MetaFlow, Snowflake, DBT, Prefect, Great Expectations, Weights & Biases, Sagemaker, Lambda
2. [A simplistic E2E system](https://github.com/jacopotagliabue/post-modern-stack) - Snowflake, DBT, S3, CometML, Reclist, SageMaker.
# ML Experiment Management
1. [All the alternatives](https://blog.valohai.com/top-machine-learning-platforms)
2. Cnvrg.io -
1. Manage - Easily navigate machine learning with dashboards, reproducible data science, dataset organization, experiment tracking and visualization, a model repository and more
2. Build - Run and track experiments in hyperspeed with the freedom to use any compute environment, framework, programming language or tool - no configuration required
3. Automate - Build more models and automate your machine learning from research to production using reusable components and drag-n-drop interface
3. Comet.ml - Comet lets you track code, experiments, and results on ML projects. It’s fast, simple, and free for open source projects.
4. Floyd - notebooks on the cloud, similar to colab / kaggle, etc. gpu costs 4$/h
5. [Trains - open source](https://heartbeat.fritz.ai/trains-all-aboard-ba92a728eb6d)
6. Missing link - RIP
7. Spark
1. [Rdds vs datasets vs dataframes](https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html)
2. [What are Rdds?](https://www.quora.com/What-are-resilient-distributed-datasets-RDDs-How-do-they-help-Spark-with-its-awesome-speed)
3. [keras , tf, spark](https://medium.com/qubida-analytics-blog/build-a-deep-learning-image-classification-pipeline-with-spark-keras-and-tensorflow-3bf26fda15e6)
4. [Repartition vs coalesce ](https://medium.com/@mrpowers/managing-spark-partitions-with-coalesce-and-repartition-4050c57ad5c4)
5. [Best practices](https://www.bi4all.pt/en/news/en-blog/apache-spark-best-practices/)
8. Databricks
1. [Koalas](https://github.com/databricks/koalas) - pandas API on Apache Spark
2. [Intro to DB on spark](https://www.youtube.com/watch?v=DqihOzZl5jM\&list=PLTPXxbhUt-YV-CwJTiE36C-0le8wlFJ5G\&index=5), has some basic sklearn-like tool and other custom operations such as single-vector-based aggregator for using features as an input to a model
3. [Pyspark.ml](https://spark.apache.org/docs/latest/api/python/pyspark.ml.html)
4. [Keras as a single node (no spark)](https://docs.databricks.com/applications/deep-learning/single-node-training/keras.html)
5. [Horovod for distributed keras (and more)](https://docs.databricks.com/applications/deep-learning/distributed-training/mnist-tensorflow-keras.html)
6. [Documentations](https://docs.databricks.com/index.html) (read me, has all libraries)
7. [Medium tutorial](https://towardsdatascience.com/how-to-train-your-neural-networks-in-parallel-with-keras-and-apache-spark-ea8a3f48cae6), explains the 3 pros of DB with examples of using with native and non native algos
1. Spark sql
2. Mlflow
3. Streaming
4. SystemML DML using keras models.
8. [systemML notebooks (didnt read)](http://systemml.apache.org/get-started.html#sample-notebook)
9. [Sklearn notebook example](https://docs.databricks.com/_static/notebooks/scikit-learn.html)
10. [Utilizing spark nodes](https://databricks.com/blog/2016/02/08/auto-scaling-scikit-learn-with-apache-spark.html) for grid searching with sklearn
1. from spark\_sklearn import GridSearchCV
11. [How can we leverage](https://databricks-prod-cloudfront.cloud.databricks.com/public/13fe59d17777de29f8a2ffdf85f52925/5638528096339357/1867405/6918044996430578/latest.html) our existing experience with modeling libraries like [scikit-learn](http://scikit-learn.org/stable/index.html)? We'll explore three approaches that make use of existing libraries, but still benefit from the parallelism provided by Spark.
These approaches are:
* Grid Search
* Cross Validation
* Sampling (random, chronological subsets of data across clusters)
1. Github [spark-sklearn](https://github.com/databricks/spark-sklearn) (needs to be compared to what spark has internally)
1. [Ref:](https://mapr.com/blog/predicting-airbnb-listing-prices-scikit-learn-and-apache-spark/) It's worth pausing here to note that the architecture of this approach is different than that used by MLlib in Spark. Using spark-sklearn, we're simply distributing the cross-validation run of each model (with a specific combination of hyperparameters) across each Spark executor. Spark MLlib, on the other hand, will distribute the internals of the actual learning algorithms across the cluster.
2. The main advantage of spark-sklearn is that it enables leveraging the very rich set of [machine learning](https://mapr.com/ebook/machine-learning-logistics/) algorithms in scikit-learn. These algorithms do not run natively on a cluster (although they can be parallelized on a single machine) and by adding Spark, we can unlock a lot more horsepower than could ordinarily be used.
3. Using [spark-sklearn](https://github.com/databricks/spark-sklearn) is a straightforward way to throw more CPU at any machine learning problem you might have. We used the package to reduce the time spent searching and reduce the error for our estimator
2. [Airbnb example using spark and sklearn,cross\_val& grid search comparison vs joblib](https://mapr.com/blog/predicting-airbnb-listing-prices-scikit-learn-and-apache-spark/)
3. [Sklearn example 2, tfidf, ](http://cdn2.hubspot.net/hubfs/438089/notebooks/ML/scikit-learn/demo_-_1_-_sklearn.html)
4. [Tracking experiments](https://docs.databricks.com/applications/mlflow/tracking.html)
1. [example](https://docs.databricks.com/applications/mlflow/tracking-examples.html#train-a-scikit-learn-model-and-save-in-scikit-learn-format)
5. [Saving loading deployment](https://docs.databricks.com/applications/mlflow/models.html#examples)
1. [Aws sagemaker](https://docs.databricks.com/applications/mlflow/model-examples.html#scikit-learn-model-deployment-on-sagemaker)
2. [Medium](https://towardsdatascience.com/a-different-way-to-deploy-a-python-model-over-spark-2da4d625f73e) and sklearn random trees
6. [How to productionalize your model using db spark 2.0 on youtube](https://databricks.com/session/how-to-productionize-your-machine-learning-models-using-apache-spark-mllib-2-x)
# ML Model Monitoring & Alerts
## **MONITORING & ALERTS**
* [Monitor! Stop being a blind DS](https://towardsdatascience.com/monitor-stop-being-a-blind-data-scientist-ac915286075f)
* [Monitor your dependencies! Stop being a blind DS](https://towardsdatascience.com/monitor-your-dependencies-stop-being-a-blind-data-scientist-a3150bd64594)
* [Data science observability for executives](https://towardsdatascience.com/data-science-observability-for-executives-a054411faecc)
* [Production Machine Learning Monitoring: Outliers, Drift, Explainers & Statistical Performance](https://towardsdatascience.com/production-machine-learning-monitoring-outliers-drift-explainers-statistical-performance-d9b1d02ac158), [youtube](https://www.youtube.com/watch?v=QcevzK9ZuDg), uses [alibi-explain](https://docs.google.com/document/d/1dXELAcJn9KCPSRMDvZoumUyHx8K8Yn7wfFxesSpbNCM/edit#heading=h.xs1o8m3ro5iy) (see compendium) and Ali-detect (see compendium)
* [Mlflow, Hyperparameterhunter,hyperopt, concept drift, unit tests.](https://towardsdatascience.com/putting-ml-in-production-ii-logging-and-monitoring-algorithms-91f174044e4e)
* [meta anomaly over multiple models, aggregate.](https://www.anodot.com/blog/monitoring-machine-learning/)
* [Vidhya on monitoring data & models](https://www.analyticsvidhya.com/blog/2019/10/deployed-machine-learning-model-post-production-monitoring/)
* [Monitor ML features using Amazon SageMaker Feature Store and AWS Glue DataBrew](https://towardsdatascience.com/monitor-ml-features-using-amazon-sagemaker-feature-store-and-aws-glue-databrew-c530abcc479a)
### **Drift**
1. [Data & concept drifts](https://deepchecks.com/how-to-monitor-ml-models-in-production/), [2](https://www.explorium.ai/blog/understanding-and-handling-data-and-concept-drift/)
2. (good) [Inferring Concept Drift Without Labeled Data](https://concept-drift.fastforwardlabs.com/). also talks about stream-based drift by Cloudera - fast forward labs.
3. Arize.ai
1. Data, concept, [feature drifts](https://towardsdatascience.com/using-statistical-distance-metrics-for-machine-learning-observability-4c874cded78) - various comparisons between train/prod/validation time windows, diff models, a/b testing etc.., and how to measure drifts
2. [Model store, Feature store, evaluation store](https://towardsdatascience.com/the-only-3-ml-tools-you-need-1aa750778d33)
3. [Monitor model performance in production](https://towardsdatascience.com/the-playbook-to-monitor-your-models-performance-in-production-ec06c1cc3245) - real- time, biased, delayed, and no ground truth.
4. [use cases - i.e., how to use statistical differences/distances](https://towardsdatascience.com/using-statistical-distance-metrics-for-machine-learning-observability-4c874cded78)
4. [Some advice on medium](https://towardsdatascience.com/concept-drift-and-model-decay-in-machine-learning-a98a809ea8d4), relabel using latest model (can we even trust it?) retrain after.
5. [Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber](https://arxiv.org/abs/2004.03045) - Previous research on concept drift mostly proposed model retraining after observing performance decreases. However, this approach is suboptimal because the system fixes the problem only after suffering from poor performance on new data. Here, we introduce an adversarial validation approach to concept drift problems in user targeting automation systems. With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data.
6. Drift estimator between data sets using random forest, the formula is in the medium article above, code here at [mlBOX](https://github.com/AxeldeRomblay/MLBox/blob/811dbcb04fc7f5501e82f3e78aa6c119f426ee78/python-package/mlbox/preprocessing/drift/drift_estimator.py)
7. [Alibi-detect](https://docs.google.com/document/d/1dXELAcJn9KCPSRMDvZoumUyHx8K8Yn7wfFxesSpbNCM/edit#heading=h.y6mpsp4co5t9) - is an open-source Python library focused on outlier, adversarial, and drift detection, by Seldon.
8. [What is concept drift and why does it go undetected](https://censius.ai/blogs/what-is-concept-drift-and-why-does-it-go-undetected) Breaking down concept drit and explaining the best methods to avoid it
9. [\*\*How does data drift hamper AI performance \*\*](https://censius.ai/blogs/data-drift-barrier-to-ai-performance) Understand how data drift affect peak AI performance and how you can detect it
### Tool Comparisons
1. [State of MLOps](https://www.stateofmlops.com) (by me), [medium](https://towardsdatascience.com/mlops-monitoring-market-review-66904f0863bb) article, open-source [AirTable](https://airtable.com/shr4rfiuOIVjMhvhL).
2. [MLOps.toys](https://mlops.toys/) - A curated list of MLOps projects by [Aporia](https://aporia.com)
3. [Neptune.AI](https://mlops.neptune.ai/) MLOPS tools landscape
4. [Twimlai](https://twimlai.com/solutions/) ML AI solutions
5. [Ambiata](https://www.ambiata.com/blog/2020-12-07-mlops-tools/) how to choose the best MLOps tools
6. [Lakefs](https://lakefs.io/the-state-of-data-engineering-in-2021/) on the state of data engineering - has monitoring and observability inside
7. [The NLP Pandec](https://github.com/ivan-bilan/The-NLP-Pandect#mlops-for-nlp) - MLOps for NLP
8. [ml-ops.org](https://ml-ops.org/)
9. [Awesome production ML](https://github.com/EthicalML/awesome-production-machine-learning/)
# MLOps Tools
### CONTINUOUS INTEGRATION
1. [Travis](https://travis-ci.org/)
2. [Circle CI](https://circleci.com/)
3. Github Actions
1. [poetry black pytest](https://medium.com/@vanflymen/blazing-fast-ci-with-github-actions-poetry-black-and-pytest-9e74299dd4a5)
### PACKAGE REPOSITORIES
1. Pypi - public
2. [Gemfury](https://gemfury.com/) - private
### Docker for DS
* [What are docker layers](https://medium.com/@jessgreb01/digging-into-docker-layers-c22f948ed612)?
* [Install on ubuntu](https://linuxconfig.org/how-to-install-docker-on-ubuntu-18-04-bionic-beaver)
* [Many jupyter docker images (spark too)](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html)
* [How to run jupyter docker 1](https://medium.com/@rahulvaish/jupyter-docker-badd38fd6b51), [2](https://medium.com/fundbox-engineering/overview-d3759e83969c)
* [Tell docker to run on a mounted disk](https://stackoverflow.com/questions/32070113/how-do-i-change-the-default-docker-container-location)
* [Docker, keras, k8s, flask serving](https://medium.com/analytics-vidhya/deploy-your-first-deep-learning-model-on-kubernetes-with-python-keras-flask-and-docker-575dc07d9e76)
* [Compose](https://docs.docker.com/compose/) - run multi coker applications.
* [Docker on ubuntu, tutorial](https://medium.com/fundbox-engineering/overview-d3759e83969c)
* [Containerize your ds environment using docker compose](https://towardsdatascience.com/containerize-your-whole-data-science-environment-or-anything-you-want-with-docker-compose-e962b8ce8ce5) - Docker-Compose is simply a tool that allows you to describe a collection of multiple containers that can interact via their own network in a very straight forward way,
* [docker for data science](https://aoyilmaz.medium.com/docker-in-data-science-and-a-friendly-beginner-to-docker-186fafdfbdeb)
* [using vscode to debug containers](https://nirradi.medium.com/vsc-vs-pycharm-developing-inside-docker-containers-4892c83d30e4)
### Kubeflow for DS
* [Youtube - the easy way,](https://www.youtube.com/watch?v=P5wcE4IwKgQ) [intro](https://medium.com/@amina.alsherif/how-to-get-started-with-kubeflow-187792f3e99)\*, [intro2\*](https://kubernetes.io/blog/2017/12/introducing-kubeflow-composable/), [intro3](https://medium.com/better-programming/kubeflow-pipelines-with-gpus-1af6a74ec2a),
* [Really good detailed article, for example it supports many serving options such as seldon](https://ubuntu.com/blog/ml-serving-models-with-kubeflow-on-ubuntu-part-1)
* [presentation](https://www.oliverwyman.com/content/dam/oliver-wyman/v2/events/2018/March/Google_London_Event/Public%20Introduction%20to%20Kubeflow.pdf)
* Tutorials:
* [Official example](https://github.com/kubeflow/example-seldon)
* [Step by step tut](https://codelabs.developers.google.com/codelabs/cloud-kubeflow-e2e-gis/index.html?index=..%2F..index#0)\*
* [endtoend tut](https://journal.arrikto.com/an-end-to-end-ml-pipeline-on-prem-notebooks-kubeflow-pipelines-on-the-new-minikf-ee618b7dc7de),
* [really detailed tut](https://towardsdatascience.com/how-to-create-and-deploy-a-kubeflow-machine-learning-pipeline-part-1-efea7a4b650f)
* KF + [Seldon on ec2](https://docs.seldon.io/projects/seldon-core/en/latest/examples/kubeflow_seldon_e2e_pipeline.html)
### AirFlow
* [Airflow](https://airflow.apache.org/) is a platform created by the community to programmatically author, schedule and monitor workflows.
* [Airflow in 5 minutes](https://medium.com/swlh/apache-airflow-in-5-minutes-c005b4b11b26) by Ashish Kumar
* [Airflow 2.0 tutorial](https://medium.com/apache-airflow/apache-airflow-2-0-tutorial-41329bbf7211) by Tomasz Urbaszek
* [Simple ETL](https://adenilsoncastro.medium.com/apache-airflow-the-etl-02-f4ac25f4d9b4) by Adnilson Castro
* [Airflow Scheduler & Webserver](https://medium.com/analytics-vidhya/manage-your-workflows-with-apache-airflow-e7b0e45544a8) by Shritam Kumar Mund
* [Airflow for DS](https://towardsdatascience.com/apache-airflow-for-data-science-how-to-write-your-first-dag-in-10-minutes-9d6e884def72)
### Prefect
* [A better airflow ](http://airflow)?
* [ml workflows with prefect](https://www.youtube.com/watch?v=SP6WqCRUkNc)
### Seldon
* Runs in k8s
* Seldon-core seldon-deploy (what are the differences?)
* [Serving graph, recipe file](https://becominghuman.ai/seldon-inference-graph-pipelined-model-serving-211c6b095f62)
* [Descriptive intro ](https://medium.com/seldon-open-source-machine-learning/introducing-seldon-core-machine-learning-deployment-for-kubernetes-e10e94c19fd8)
* [Sales pitch intro](https://medium.com/seldon-open-source-machine-learning/introducing-seldon-deploy-c390d11af20c)
### Tutorials
* [Kubernetes, sklearn, s2i, gcloud, seldon random serving for ab testing](https://medium.com/analytics-vidhya/manage-ml-deployments-like-a-boss-deploy-your-first-ab-test-with-sklearn-kubernetes-and-b10ae0819dfe)
* [Polyaxon - training, argo-package/deployment , seldin -serving](https://medium.com/analytics-vidhya/polyaxon-argo-and-seldon-for-model-training-package-and-deployment-in-kubernetes-fa089ba7d60b)
### Sentry
* [For python,](https://sentry.io/for/python/) Your code is telling you more than what your logs let on. Sentry’s full stack monitoring gives you full visibility into your code, so you can catch issues before they become downtime.
### Kafka for DS
1. [What is, terminology, use cases](https://sookocheff.com/post/kafka/kafka-in-a-nutshell/#:~:text=Kafka%20topics%20are%20divided%20into,from%20a%20topic%20in%20parallel.)
### Redis for DS
1. What is, vs [memcached](https://medium.com/@pankaj.itdeveloper/memcached-vs-redis-which-one-to-choose-d5177482dc42)
2. [Redis cluster](https://medium.com/@inthujan/introduction-to-redis-redis-cluster-6c7760c8ebbc)
3. [Redis plus spacy](https://towardsdatascience.com/spacy-redis-magic-60f25c21303d)
4. Note: redis is a managed dictionary its strength lies when you have a lot of data that needs to be queries and managed and you don’t want to hard code it, for example.
5. [Long tutorial](https://realpython.com/python-redis/)
### Statsd
1. [Statistics server, with gauges/buckets and flushing/sending ability](https://github.com/statsd/statsd/blob/master/examples/python_example.py)
### FastAPI
1. [Flask on steroids with variable parameters](https://fastapi.tiangolo.com/alternatives/)
### Visualization
1. [How to use plotly in python](https://plot.ly/python/ipython-notebook-tutorial/)
Plotly for jupyter lab “jupyter labextension install @jupyterlab/plotly-extension”
1. [Venn for python](http://ow-to-create-and-customize-venn-diagrams-in-python-263555527305)
### Serving Models
1. ML SYSTEM DESIGN PATTERNS, [res](https://docs.google.com/presentation/d/1pSkklHkBySMnJNODshW8NZVpBSqOsbJBWeEq8RrS0M4/edit#slide=id.g81f938aa2b_0_47), [git](https://github.com/mercari/ml-system-design-pattern)
2. Seldon
3. [Medium on DL as a service by Nir Orman](https://towardsdatascience.com/serving-deep-learning-algorithms-as-a-service-6aa610368fde)
4. [Scaling ML on the cloud](https://towardsdatascience.com/scalable-efficient-big-data-analytics-machine-learning-pipeline-architecture-on-cloud-4d59efc092b5)
5. [Dapr](https://github.com/dapr/dapr) is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.
Dapr codifies the best practices for building microservice applications into open, independent, building blocks that enable you to build portable applications with the language and framework of your choice. Each building block is independent and you can use one, some, or all of them in your application.
## MetaFlow
1. [Intro](https://www.youtube.com/watch?v=JCbOI_1ZA5E), and [what is](https://www.youtube.com/watch?v=bxVAniteuQs)?
# MLOps Deployment
1. [Lightning](https://lightning.ai/) - A Framework to train and deploy PyTorch models. Focus on the science, not the engineering
1. Lightning Apps - Framework to build composable, reactive ML workflows. Bye YAML + k8s, hello Python.
2. [Examples](https://lightning.ai/docs/app/stable/#build-self-contained-components)
# Feature Stores & Feature Pipelines
1. [feature store and feature pipelines](https://medium.com/data-for-ai/feature-pipelines-and-feature-stores-deep-dive-into-system-engineering-and-analytical-tradeoffs-3c208af5e05f) - assaf pinhasi
# Model Formats
1. [stop using pickle, use JSON/PMML](https://faoel.medium.com/stop-using-pickle-to-save-machine-learning-models-eaa46e8e561a)
### [PMML ](http://www.kdnuggets.com/faq/pmml.html)
\- an XML file that describes a ML model that is transferable between applications.
* [PMML](http://dmg.org/pmml/v4-3/GeneralStructure.html) uses XML to represent mining models.
* The structure of the models is described by an XML Schema.
* One or more mining models can be contained in a PMML document
# AI As Data
1. [MindsDB](https://mindsdb.com/?utm_medium=community\&utm_source=github\&utm_campaign=mindsdb%20repo) ML-SQL Server enables machine learning workflows for the most powerful databases and data warehouses using SQL. [git](https://github.com/mindsdb/mindsdb)[hub](https://github.com/mindsdb/mindsdb).
* Developers can quickly add AI capabilities to your applications.
* Data Scientists can streamline MLOps by deploying ML models as AI Tables.
* Data Analysts can easily make forecasts on complex data (like multivariate time-series with high cardinality) and visualize them in BI tools like Tableau.
* [Medium article](https://towardsdatascience.com/introduction-to-ai-tables-a719251e1a58)
# MLOps Interview Questions
1. [10 Qs](https://360digitmg.com/mlops-interview-questions-answers) by 360dgtmg
2. [15 Qs ](https://www.analyticsvidhya.com/blog/2023/02/15-must-read-interview-questions-on-mlops-for-2023/)by Analytics Vidhaya
3. [15 Qs](https://hashdork.com/top-mlops-interview-questions/) by Hashdork
4. [Mock Interview](https://www.youtube.com/watch?v=uZlYXHm8EPg) (youtube)
5. [MLOps at Scale](https://www.youtube.com/watch?v=7AOgPspCOaQ)
6. (good) [MLOps Tips](https://medium.com/nlplanet/mlops-tips-from-an-interview-study-952cc3b65801)
7. [Interview Qs](https://medium.datadriveninvestor.com/data-scientist-mlops-interview-questions-list-from-interviewer-2022-2023-f680c1e90e9f)
# ML Architecture
1. [Lambda architecture for ML serving / Training](https://www.youtube.com/watch?v=fPlgoTLJh38)
2. A PaaS [End-to-End ML Setup](https://github.com/jacopotagliabue/no-ops-machine-learning) with Metaflow, Serverless and SageMaker. [No Ops ML](https://towardsdatascience.com/noops-machine-learning-3893a42e32a4) Article - by Jacopo Tagliabue
3. An [end-to-end implementation](https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat) of intent prediction with Metaflow and other cool tools
4. [A Post modern stack ](https://github.com/jacopotagliabue/post-modern-stack)with an [Article](https://towardsdatascience.com/the-post-modern-stack-993ec3b044c1) - by Jacopo Tagliabue, speaks about DBT, Snowflake, S3, Comet, Sagemaker.
# SQL
1. (very good) [DDL, DQL, DML, DCL and TCL](https://www.geeksforgeeks.org/sql-ddl-dql-dml-dcl-tcl-commands/) - by geek for geeks
1. DDL – Data Definition Language (create, drop, alter, truncate)
2. DQl – Data Query Language (select)
3. DML – Data Manipulation Language (update, insert, delete, lock)
4. DCL – Data Control Language (grant, revoke)
5. TCL - Transaction Control Language (commit, rollback, savepoint, set transaction)
2. [Introduction, index, keys, joins, aliases etc.](https://www.youtube.com/watch?v=nWeW3sCmD2k), [newer](https://www.youtube.com/watch?v=9ylj9NR0Lcg)
3. [Sql cheat sheet](https://gist.github.com/bradtraversy/c831baaad44343cc945e76c2e30927b3)
4. [Primary key](https://www.eukhost.com/blog/webhosting/whats-the-purpose-use-primary-foreign-keys/)
5. [Foreign key, a key constraint that is included in the primary key allowed values](https://www.1keydata.com/sql/sql-foreign-key.html)
6. [Index, i.e., book index for fast reading](https://www.tutorialspoint.com/sql/sql-indexes.htm)
7. [Sharding vs Partioning](https://planetscale.com/learn/articles/sharding-vs-partitioning-whats-the-difference)
# Tools
1. [Debezium](https://debezium.io/) - an open source distributed platform for change data capture
2. [Hudi](https://hudi.apache.org/) - "Hudi is a rich platform to build streaming data lakes with incremental data pipelines on a self-managing database layer, while being optimized for lake engines and regular batch processing."
3. [Upsolver](https://www.upsolver.com/) - "Continuous SQL Pipelines for Cloud Data Lakes. No custom coding. No orchestration. No infrastructure maintenance."
4. [DBT](https://www.getdbt.com/) - "dbt helps data teams work like software engineers—to ship trusted data, faster. collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Now anyone who knows SQL can build production-grade data pipelines."
1. [intro](https://www.youtube.com/watch?v=R2nr1uZ8ffc)
2. [in depth intro](https://www.youtube.com/watch?v=MHnJDqEKyUY)
3. [dbt in one hour](https://www.youtube.com/watch?v=na6eu9WSXGY)
4. [CI/CD with dbt](https://www.youtube.com/watch?v=snp2hxxWgqk)
5. [snowflake terraform and dbt](https://www.youtube.com/watch?v=r4tAyiTgwRw)
6. [hubspot snowflake and dbt](https://www.youtube.com/watch?v=qDHgknWW_oo)
5. [Metorikku](https://github.com/YotpoLtd/metorikku) - A simplified, lightweight ETL Framework based on Apache Spark
6. BI tools that directly connect to a DB.
1. [redash](https://redash.io/) - Connect and query your data sources, build dashboards to visualize data and share them with your company.
2. [Metabase](https://www.metabase.com/) - "[is an easy-to-use, open source business intelligence tool that lets you analyze data from a variety of data destinations and sources. It also follows a simple and fast setup process. Its data visualization capabilities are exceptional and can be showcased in a user-friendly way, without using SQL. With Metabase, you can easily share live dashboards, automated reports, and questions with the rest of your team.](https://growthfullstack.com/analyse/metabase-bi-tool/)" - by fullstackgrowth.com
3. [Superset](https://superset.apache.org/) - Apache Superset is a modern data exploration and visualization platform
7. [Stitch](https://www.stitchdata.com/) - Stitch rapidly moves data from 130+ sources into a data warehouse so you can get to answers faster, no coding required.
8. [SnowPlow](https://snowplowanalytics.com/) - Generate [complete, accurate and well-structured event data](https://snowplowanalytics.com/web-and-mobile-data/) across all platforms and channels in a common format, with the Snowplow Behavioral Data Platform.
9. [Workato](https://www.workato.com/) - A SINGLE PLATFORM FOR INTEGRATION & WORKFLOW AUTOMATION ACROSS YOUR ORGANIZATION
10. [AWS Deequ](https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/) - Test data quality at scale
# Databases
## Lakes & Warehouses
1. [What is a DWH? a comprehensive guide](https://www.oracle.com/database/what-is-a-data-warehouse/)
2. [DataLakeHouse](https://www.firebolt.io/blog/snowflake-vs-databricks-vs-firebolt) - Firebolt comparison with Snowflake vs Databricks.
1. Delta lake is a data lake that can store raw unstructured, semi-structured, and structured data. When combined with Delta Engine it becomes a data lakehouse.
3. [What is SnowFlake](https://www.stitchdata.com/resources/snowflake/), [2](https://www.slalom.com/insights/snowflake-implementation-success) - Snowflake decouples the storage and compute functions, which means organizations that have high storage demands but less need for CPU cycles, or vice versa, don’t have to pay for an integrated bundle that requires them to pay for both. Users can scale up or down as needed and pay for only the resources they use.
1. [get started with SF](https://www.phdata.io/blog/getting-started-with-snowflake/)
4. data mart
1. [talend on data marts](https://www.talend.com/resources/what-is-data-mart/) + 3 types (dependent, independent, hybrid)
2. (good) [netsuite on data marts](https://www.netsuite.com/portal/resource/articles/data-warehouse/data-mart.shtml) - the three types ^ + structures (star, snowflake, denormalized) + comparisons
3. [basic intro](https://study.com/academy/lesson/what-is-a-data-mart-design-types-example.html)
5. Data Lake
1. [monitoring health status at scale](https://towardsdatascience.com/how-to-monitor-data-lake-health-status-at-scale-d0eb058c85aa) using great expectations and spark
## Comparisons
1. [Data Lake vs Data Warehouse](https://www.talend.com/resources/data-lake-vs-data-warehouse/)
2. [Top 5 differences between DL & DWH](https://www.bluegranite.com/blog/bid/402596/top-five-differences-between-data-lakes-and-data-warehouses)
3. [Amazon on DL vs DWH](https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/)
4. [Snowflake vs Delta Lake vs Fire Bolt](https://www.firebolt.io/blog/snowflake-vs-databricks-vs-firebolt) - "Databricks Delta Lake and Delta Engine is a lakehouse. You choose it as a data lake, and for data lakehouse-based workloads including ELT for data warehouses, data science and machine learning, even static reporting and dashboards if you don’t mind the performance difference and don’t have a data warehouse.
Most companies still choose a data warehouse like Snowflake, BigQuery, Redshift or Firebolt for general-purpose analytics over a data lakehouse like Delta Lake and Delta Engine because they need performance.
But it doesn’t matter. You need more than one engine. Don’t fight it. You will end up with multiple engines for very good reasons. It’s just a matter of when. "
5. [Snowflake vs Amazon Redshift](https://www.sphereinc.com/blogs/snowflake-vs-aws-redshift-which-should-you-use-for-your-data-warehouse/)
6. Snowflake [Intro and demo](https://www.youtube.com/watch?v=dUL8GO4ZK9s)
7. [The three pillars - snowflake](https://towardsdatascience.com/why-you-need-to-know-snowflake-as-a-data-scientist-d4e5a87c2f3d)
8. [Snowflake vs redshift on medium](https://towardsdatascience.com/redshift-or-snowflake-e0e3ea427dbc)
9. [SF vs RS](https://www.xplenty.com/blog/redshift-vs-snowflake/)
10. [RS vs BQ](https://www.xplenty.com/blog/redshift-vs-bigquery-comprehensive-guide/)
11. [SF vs BQ](https://www.xplenty.com/blog/snowflake-vs-bigquery/)
## Use Cases
1. [Hunters on their architecture](https://www.youtube.com/watch?v=S78gCJ3tdc4), airflow, snowflake, snowpipe, flink, rockdb, cluster optimization during ingestion, monitoring metrics, cost.
## Snowflake
1. [(good) Guides ](https://www.snowflake.com/guides/)
2. [getting started with SF tasks](https://medium.com/snowflake/getting-started-with-snowflake-tasks-945ecd54c77b) - sql or procedures, schedules, B-tree tasks.
3. [cost](https://www.phdata.io/blog/what-is-the-snowflake-data-cloud/)
## ClickHouse
1. [open source database for real time apps and analytics](https://clickhouse.com/) - think snowflake open source
## Feature engineering
1. [Feature engineering in snowflake](https://towardsdatascience.com/feature-engineering-in-snowflake-1730a1b84e5b)
## Data lake Table Formats
### Apache Iceberg
1. [A short Intro](https://medium.com/expedia-group-tech/a-short-introduction-to-apache-iceberg-d34f628b6799)
2. [A Primer](https://thedatafreak.medium.com/apache-iceberg-a-primer-75a63470bfa2)
3. [Benchmarking Delta vs Iceberg vs Hudi](https://databeans-blogs.medium.com/delta-vs-iceberg-vs-hudi-reassessing-performance-cb8157005eb0)
4. [How we migrated our production data lake to iceberg](https://medium.com/insiderengineering/how-we-migrated-our-production-data-lake-to-apache-iceberg-4d6892eca6e6)
5. [How we reduced our cost by 90%](https://medium.com/insiderengineering/apache-iceberg-reduced-our-amazon-s3-cost-by-90-997cde5ce931)
6. [Top 5 Features](https://dipankar-tnt.medium.com/apache-iceberg-features-101-331a254a7ada)
### Databricks Delta Lake
1. [integrating delta lake into other platforms](https://www.databricks.com/blog/integrating-delta-lakehouse-other-platforms)
## Vector Databases
1. Gartner - [Innovation Insight: Vector Databases](https://www.gartner.com/doc/reprints?id=1-2HBZK5EN\&ct=240418\&st=sb)
2. Chroma - AI native open source embedding database, [github](https://github.com/chroma-core/chroma)
# Database Modeling
1. [Types of DBs](https://simonatta.medium.com/database-types-2dac81461709)
2. Data Vault Modeling
1. [**Data vault modeling**](https://en.wikipedia.org/wiki/Data_vault_modeling) is a [database](https://en.wikipedia.org/wiki/Database) modeling method that is designed to provide long-term historical storage of [data](https://en.wikipedia.org/wiki/Data) coming in from multiple operational systems. It is also a method of looking at historical data that deals with issues such as auditing, tracing of data, loading speed and resilience to change as well as emphasizing the need to trace where all the data in the database came from. This means that every [row](https://en.wikipedia.org/wiki/Row_\(database\)) in a data vault must be accompanied by record source and load date attributes, enabling an auditor to trace values back to the source. It was developed by [Daniel (Dan) Linstedt](https://en.wikipedia.org/w/index.php?title=Daniel_Linstedt\&action=edit\&redlink=1) in 2000. - wikipedia
2. its a design pattern to build dwh for enterprise analytics. it has hubs (core business concepts) links (relationshipts between hubs) satellites store info about these two. good for lakehouse paradigm. [link has a good image.](https://www.databricks.com/glossary/data-vault#:~:text=What%20is%20a%20data%20vault,hubs%2C%20links%2C%20and%20satellites.) - databricks
3. Data Fabric
1. [Data Fabric](https://preetihemant.medium.com/modern-data-architecture-models-69e90b725a05) - is a data architecture, that follows a set of steps that determine its flow. The first step takes data through an integration phase. In the integration phase, data is ingested and then cleaned, transformed and loaded into storage. Then, there is the data quality phase where quality assessment is performed on the stored data. This data is then made available for different use cases through a combination of a data lake and a data warehouse, Typical use cases are BI, analytics and machine learning. Data governance policies are defined for the ingested data and a data catalog is used for discoverability - by preeti hemant.
2. (short) [Netapp](https://www.netapp.com/data-fabric/what-is-data-fabric/#:~:text=A%20data%20fabric%20is%20an,on%20premises%2C%20and%20edge%20devices.)
3. (good) [IBM](https://www.ibm.com/topics/data-fabric) - Data Management layer, Data Ingestion Layer, Data Processing, Data Orchestration, Data Discovery, Data Access.
4. (good) [Gartner](https://www.gartner.com/smarterwithgartner/data-fabric-architecture-is-key-to-modernizing-data-management-and-integration) - the pillars of data fabric
5. [talend](https://www.talend.com/resources/what-is-data-fabric/)
6. (good but lengthy) [spiceworks](https://www.spiceworks.com/tech/big-data/articles/what-is-data-fabric/) - architectural components, best practices
7. (good) [k2view](https://www.k2view.com/what-is-data-fabric) - has a great figure of integration storage catalog, cleansing & masking, transformation & enrichment, governance, webservices.
8. (good) [tibco](https://www.tibco.com/reference-center/what-is-data-fabric) - application & services, dev & integration, security, storage management, transport, endpoints.
9. [mesh vs fabric](https://www.datanami.com/2021/10/25/data-mesh-vs-data-fabric-understanding-the-differences/)
4. [data virtualization](https://www.ibm.com/analytics/data-virtualization) - single view without movement of data.
5. [tools for deploying data models in prod](https://www.superdatascience.com/podcast/tools-for-deploying-data-models-into-production)
## Data Warehouse
1. (good) [a guide from strategy to implementation](https://www.analytics8.com/blog/what-is-a-data-warehouse/)
# Data Analytics
## Course
* [What Is Data Analytics? - An Introduction (Full Guide)](https://www.youtube.com/watch?v=yZvFH7B6gKI)
* Intro [FREE Data Analyst Bootcamp!!](https://www.youtube.com/watch?v=rGx1QNdYzvs\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=1)
* How [How to Become a Data Analyst in 2023 (Completely FREE!)](https://www.youtube.com/watch?v=CUBfrdDwznQ\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=2)
* Excel 21-27 [Pivot Tables in Excel | Excel Tutorials for Beginners](https://www.youtube.com/watch?v=lH7HfwUFnYA\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=22)
* PowerBI 33-41 [How to Install Power BI | Building First Visualization | Microsoft Power BI for Beginners](https://www.youtube.com/watch?v=g0m5sEHPU-s\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=34)
* Tableau 23-31 [Tableau Training For Beginners | Tableau Tutorial | Tableau Tutorial For Beginners | Simplilearn](https://youtube.com/watch?v=UWT14hhjsL0\&list=PLEiEAq2VkUUKgEFXH1tBbHwq38oWYDScU\&index=24)
* SQL 3-20 [Installing MySQL and Creating Databases | MySQL for Beginners](https://www.youtube.com/watch?v=wgRwITQHszU\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=4)
* SQL questions [Solving Easy SQL Interview Questions on Analyst Builder](https://www.youtube.com/watch?v=ZHaYOC0H5KE\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=70)
* Python 42-56 [Installing Jupyter Notebooks/Anaconda | Python for Beginners](https://www.youtube.com/watch?v=WUeBzT43JyY\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=43)
* Pandas 57-64 [Reading in Files in Pandas | Python Pandas Tutorials](https://www.youtube.com/watch?v=dUpyC40cF6Q\&list=PLUaB-1hjhk8FE_XZ87vPPSfHqb6OcM0cF\&index=58)
* Course [Data Analysis with Python Course - Numpy, Pandas, Data Visualization](https://www.youtube.com/watch?v=GPVsHOlRBBI)
* Not sure if needed [Data Analytics Full Course In 5 Hours | Data Analytics For Beginners | Data Analytics | Simplilearn](https://www.youtube.com/watch?v=oQmxQh92Nh8\&list=PLEiEAq2VkUUKgEFXH1tBbHwq38oWYDScU\&index=2)
# Data Engineering
1. [Data engineers are not software developers - a comparison](https://betterprogramming.pub/data-engineering-is-not-software-engineering-af81eb8d3949)
2. [Different Types of data engineers](https://medium.com/coriers/different-types-of-data-engineering-teams-6a1056986d3)
1. data infra
2. data platform
3. data engineers
4. data governance
5. BI & analytics
### [Cap theorem](https://towardsdatascience.com/cap-theorem-and-distributed-database-management-systems-5c2be977950e)
* [Cap](https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/) 2015
* [Cap is changing](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/)
# Data Pipelines
1. [Decision guide for data integration tools](https://www.metaplane.dev/blog/decision-guide-to-choosing-a-data-integration-tool)
2. [ETL vs ELT](https://www.qlik.com/us/etl/etl-vs-elt), [2](https://www.guru99.com/etl-vs-elt.html)
3. [What is reverse ETL (ELT)](https://torbjornzetterlund.com/what-is-reverse-etl/)?
4. [Airbyte](https://airbyte.com/) Open-source ELT data integration for modern data teams
5. [Stitch vs Airflow vs Rivery](https://www.stitchdata.com/vs/airflow/rivery/)
6. Rivery
7. Fivertran
1. [Operators for AirFLow](https://www.fivetran.com/blog/announcing-the-fivetran-airflow-provider)
8. [Astronomer AirFlow as an orchestrator scheduler for FT, AB, etc](https://www.astronomer.io/blog/best-etl-tools-airflow/)
9. [The 7 Principles of reliable data pipelines](https://medium.com/bigeye/seven-principles-for-reliable-data-pipelines-e82a82810e4f) - bigeye medium article
10. [Hevo](https://hevodata.com/integrations/pipeline/) - Hevo is a Fully Automated, No-code Data Pipeline Platform that supports 150+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services.
# Data Strategy
1. (excellent) [how to get started](https://towardsdatascience.com/data-strategy-716059d5c066)
2. Analytics 8(excellent)
1. [7 elements of data strategy](https://www.analytics8.com/blog/7-elements-of-a-data-strategy/) - you can deep dive into other articles
2. [7 tips for strong data infra](https://www.analytics8.com/blog/7-tips-for-strong-data-infrastructure/)
3. [step 2/5 how to develop it](https://www.analytics8.com/blog/how-to-develop-data-strategy-roadmap/)
3. (very good) [research data strategy - for research orgs](https://towardsdatascience.com/research-data-strategy-2fd447f83719)
4. [how to create an effective DS](https://medium.com/method-perspectives/how-to-create-an-effective-data-strategy-ffedb1bdc71a) - helps to ask the right questions on several topics
5. (needed) [RACI matrix](https://en.wikipedia.org/wiki/Responsibility_assignment_matrix) by wikipedia
6. [How to build an actionable data strategy phdata.io](https://www.phdata.io/blog/how-to-build-an-actionable-data-strategy-framework/)
7. two part article:
1. [how to create a data strategy for your org](https://towardsdatascience.com/how-to-create-a-data-strategy-for-your-organization-e0493110b2e7)
2. [how to create a perfect data strategy](https://towardsdatascience.com/how-to-create-a-perfect-data-strategy-7e8fd9bbfad0)
8. [a step by step guide](https://towardsdatascience.com/data-strategy-a-step-by-step-guide-91529a75c72b)
9. [why is your data strategy failing](https://medium.com/geekculture/why-your-data-strategy-is-failing-19e569b6b082)
10. [Become Data-Driven or Perish: Why your company needs a Data Strategy and not just more Data People](https://towardsdatascience.com/become-data-driven-or-perish-why-your-company-needs-a-data-strategy-and-not-just-more-data-people-aa5d435c2f9)
11. [create a data strategy](https://towardsdatascience.com/data-strategy-make-data-count-fa42b42d409) - a meta discussion
12. [data strategy for AI](https://medium.com/mmc-writes/the-ai-playbook-how-to-develop-a-data-strategy-for-ai-d74df9486c0e)
13. [how to create a successful data strategy](https://towardsdatascience.com/how-to-create-a-successful-data-strategy-1293bacf463c)
14. [Definition taken from DAMA](https://www.cleverrepublic.com/what-is/data-strategy/), what is [data management](https://www.cleverrepublic.com/what-is/data-management/), and [intelligence](https://www.cleverrepublic.com/what-is/data-intelligence/)
15. [Data Strategy: Good Data vs. Bad Data](https://towardsdatascience.com/data-strategy-good-data-vs-bad-data-d40f85d7ba4e)
16. [Data strategy in progress](https://dasbarrett.medium.com/data-strategy-in-progress-53e5e341ff7)
17. Series: Big data strategy, [Part 1](https://francesco-ai.medium.com/big-data-strategy-i-tips-for-analyzing-your-data-963292a43105), [Part 2](https://francesco-ai.medium.com/big-data-strategy-part-ii-a-data-maturity-map-8e41db568625), [Part 3](https://francesco-ai.medium.com/big-data-strategy-part-iii-is-your-company-data-driven-acf871c38001)
18. [Data before ML Strategy](https://medium.com/machine-learning-in-practice/first-things-first-you-need-a-data-strategy-before-you-can-have-a-machine-learning-strategy-366f8439aedf)
19. [from a product POV](https://towardsdatascience.com/connect-the-dots-in-data-strategy-56f65a1e63a5)
20. [Data Advantage Matrix](https://towardsdatascience.com/data-advantage-matrix-a-new-way-to-think-about-data-strategy-4178cd2f520a)
21. [create an effective strategy](https://medium.com/method-perspectives/how-to-create-an-effective-data-strategy-ffedb1bdc71a)
22. [how to get started](https://towardsdatascience.com/data-strategy-716059d5c066)
# Data Vision
1. [vision for a data team](https://medium.com/alan/vision-for-a-data-team-2eae845b8052)
# Data Teams
1. [how to build an effective analytics team](https://www.analytics8.com/blog/how-to-build-an-effective-data-analytics-team/)
# Data Catalogs
Tools
1. [Alation](https://www.alation.com/) - "A [Data Catalog](https://www.alation.com/product/data-catalog/) is a collection of metadata, combined with data management and search tools, that helps analysts and other data users to find the data that they need, serves as an inventory of available data, and provides information to evaluate fitness of data for intended uses."
# Data Governance
1. [DAMA](https://www.dama.org/cpages/dmbok-2-image-download)
2. How to build [part 1](https://www.phdata.io/blog/snowflake-data-governance/), [part 2](https://www.phdata.io/blog/snowflake-data-governance-2/)
3. [Definition](https://www.cleverrepublic.com/what-is/data-governance/), taken from DAMA
4. [What is ](https://www.imperva.com/learn/data-security/data-governance/)- by Imperva
5. [a very thorough article](https://bi-survey.com/data-governance) \
6. [action items](https://data.wisc.edu/data-governance/)
7. [8 steps to start your data governanace program](https://www.analytics8.com/blog/8-steps-to-start-your-data-governance-program/)
# Data Quality
1. [why cant data quality be fixed with tech](https://www.analytics8.com/blog/why-data-quality-cannot-be-fixed-with-technology/)
2. [CloverDX - data quality](https://www.cloverdx.com/explore/data-quality)
3. [CloverDX - 6 Data Quality Metrics You Can't Afford To Ignore](https://www.cloverdx.com/blog/6-data-quality-metrics-you-cant-ignore)
4. [FirstEigan - data quality metrics](https://firsteigen.com/blog/6-key-data-quality-metrics-you-should-be-tracking/)
5. [Dataladder - data quality metrics](https://dataladder.com/10-data-quality-metrics-you-should-measure/)
6. [Datacademia - data quality metrics](https://datacadamia.com/data/quality/metric)
7. [Open sistemas - big data challenges - veracity](https://opensistemas.com/en/the-four-vs-of-big-data/#:~:text=of%20a%20video\).-,Veracity,have%20implications%20in%20terms%20of%20the%20data%20sources%20we%20may%20have.,-Velocity)
## DBT
1. [DBT builtin tests](https://docs.getdbt.com/docs/building-a-dbt-project/tests)
2. [DBT expectations](https://hub.getdbt.com/calogica/dbt_expectations/0.1.2/)
3. [DBT custom generic tests](https://docs.getdbt.com/guides/legacy/writing-custom-generic-tests)
4. [Jinja](https://www.youtube.com/watch?v=OraYXEr0Irg) (used for DBT templates)
# Data Observability
## Tools
1. [Monte Carlo](https://www.montecarlodata.com/)
2. [BigEye](https://www.bigeye.com/)
3. [Databand](https://databand.ai)
1. [What is data observability](https://databand.ai/data-observability/) and what is the dataOps cycle, i.e., "The DataOps cycle outlines the fundamental activities needed to improve how data is managed within the DataOps workflow. This cycle consists of three distinct stages: Detection, Awareness, and Iteration."\\
4. [DQOps](https://dqops.com)
# Data Program Management
1. [What is a data program management? and a manager.](https://www.ddmcd.com/dpm-intro) by Dennis D McDonald.
1.
2.
3.
4.
5.
6. [program manager responsibilities](https://www.projectmanager.com/training/program-manager-responsibilities)
# Data KPIs
1. [Measuring ROI of data and analytics](https://santiagotacoronte.com/5-ways-to-measure-the-roi-of-data-analytics-the-simple-way-a61790eff7d1)
2. [How to measure Roi of data analytics](https://medium.com/be-data-driven/how-to-measure-the-roi-of-data-analytics-2d2ac172754a)
# Data Mesh
1. what is a [data mesh](https://databricks.com/session_na20/data-mesh-in-practice-how-europes-leading-online-platform-for-fashion-goes-beyond-the-data-lake)? and in practice.
2. Zhamak Dehghani
1. [Introduction to Data Mesh](https://www.youtube.com/watch?v=_bmYXWCxF_Q)
2. [how to move from data lake to distributed data mesh](https://martinfowler.com/articles/data-monolith-to-mesh.html)
3. [principles and logical architecture](https://martinfowler.com/articles/data-mesh-principles.html)
4. (good) [Keynote - Data Mesh by Zhamak Dehghani](https://www.youtube.com/watch?v=L_-fHo0ZkAo) (oltp->etl->olap is broken)
5. [lessons from the trenches](https://www.youtube.com/watch?v=Nw_bxIyR1L0)
3. [lake vs mesh, he probably means fabric vs mesh](https://medium.com/codex/data-lakehouse-vs-data-mesh-bfa1132f94b)
4. [data mesh 101](https://www.youtube.com/watch?v=hgKOpEQaqdY\&list=PLa7VYi0yPIH0L8ahQYbyBFkGc6a949-Lj\&index=10) - by confluent
5. (good) [mesh topologies](https://towardsdatascience.com/data-mesh-topologies-and-domain-granularity-65290a4ebb90) - by Piethein Strengholt
6. [Data Mesh & MLOps using a data platform.](https://medium.com/swlh/building-a-data-platform-to-enable-analytics-and-ai-driven-innovation-1bd95e37efb9)
7. [Explained](https://medium.com/@david.c.dupuis/data-mesh-explained-a95b6ae50878)
8. [The next gen of architecture](https://datagrad.medium.com/data-mesh-transition-to-next-generation-of-data-architecture-832c4bc27e9f)
9. [how not to mesh it - monte carlo](https://www.montecarlodata.com/blog-what-is-a-data-mesh-and-how-not-to-mesh-it-up/)
10. [building data mesh using lake house approach](https://www.youtube.com/watch?v=YPYODx4Pfdc) by aws
11. [enterprise data mesh by oracle](https://www.oracle.com/a/ocom/docs/datamesh-ebook.pdf)
12. [deconstructing data mesh principles](https://medium.com/slalom-data-ai/data-mesh-232e50f42e66)
# Data Contract
1. A data contract is **a formal agreement between a service and a client that abstractly describes the data to be exchanged**. That is, to communicate, the client and the service do not have to share the same types, only the same data contracts. - [Microsoft](https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/using-data-contracts)
2. The following is a set of articles around the topics of data product / contract etc. however the focus is about data contracts and expectations IMO, therefore I place it here. - by Chad sanderson
1. [the existential threat of data quality](https://dataproducts.substack.com/p/the-existential-threat-of-data-quality)
2. [the death of data modeling part 1](https://dataproducts.substack.com/p/the-death-of-data-modeling-pt-1)
3. [data's collaboration problem](https://dataproducts.substack.com/p/datas-collaboration-problem)
4. the rise of [data contracts](https://dataproducts.substack.com/p/the-rise-of-data-contracts)
5. [production grade data products](https://dataproducts.substack.com/p/the-production-grade-data-pipeline)
6. (finally!) [a guide to data contracts p1](https://dataproducts.substack.com/p/an-engineers-guide-to-data-contracts)
7. (good) c[ontracts, robustness in datamesh (but not only)](https://towardsdatascience.com/data-contracts-ensure-robustness-in-your-data-mesh-architecture-69a3c38f07db), also has about data-sharing agreements.
8. (good) [from zero to hero](https://towardsdatascience.com/data-contracts-from-zero-to-hero-343717ac4d5e)
9. [data contract experimentation](https://www.newsletter.swirlai.com/p/sai-04-data-contracts-experimentation) - swirlai
## Implementation
1. [**JSON Schema**](https://json-schema.org/) is a vocabulary that allows you to **annotate** and **validate** JSON documents, [example](https://json-schema.org/learn/miscellaneous-examples.html).
2. protobuf & gRPC
1. [Protobuf on git](https://github.com/protocolbuffers/protobuf/tree/main/python), [google dev](https://developers.google.com/protocol-buffers)
2. [what is gRPC?](https://grpc.io/docs/what-is-grpc/introduction/)
3. [what are proto buffers, what do they solve and what are the benefits?](https://developers.google.com/protocol-buffers/docs/overview)
4. gRPC - [a basic pythonic tutorial](https://grpc.io/docs/languages/python/basics/)
5. Protobuf - [a pythonic tutorial](https://developers.google.com/protocol-buffers/docs/pythontutorial)
6. (good) Intro to gRPC & Protobuf by [Trevor Kendrick](https://medium.com/@trevor.kendrick?source=post_page-----c21054ef579c--------------------------------)
7. (good) What are Protocol Buffers and why they are widely used? by [Dineshchandgr](https://medium.com/@dineshchandgr?source=post_page-----cbcb04d378b6--------------------------------)
8. [Understanding protobufs ](https://medium.com/danielpadua/understanding-protocol-buffers-protobuf-a466d8943df8)by [Daniel Padua Ferreira](https://medium.com/@danielpadua?source=post_page-----a466d8943df8--------------------------------)\
"Protocol Buffers (protobuf) is a method of serializing structured data which is particulary useful to communication between services or storing data.\
It was designed by Google early 2001 (but only publicly released in 2008) to be smaller and faster than XML. Protobuf messages are serialized into a [binary wire](https://developers.google.com/protocol-buffers/docs/encoding) format which is very compact and boosts performance."
9. protobuf what and why? by [Swaminathan Muthuveerappan](https://medium.com/@swamim?source=post_page-----fcb324a64564--------------------------------)
10. off topic - [how to choose between grpc, graphql, rest](https://ashish-bania.medium.com/the-exhaustive-guide-to-choosing-between-grpc-graphql-and-rest-b7e4fd6d547e)
3. managing proto files and other schema types such as avro or json schema, can be done in [kafka's schema registry](https://docs.confluent.io/platform/current/schema-registry/index.html).
# Data Product
1. [definition phdata.io](https://offers.phdata.io/hubfs/Checklists/Data%20Product%20Definition%20Checklist_D2.pdf)
2. (good) [a framework for data product](https://www.datasciencecentral.com/data-product-framework/) - accountability, boundaries, contracts, expectations, consumers, knowledge.
3. [a meta discussion on data platform as a product](https://medium.com/stuart-engineering/how-were-building-our-data-platform-as-a-product-f89142b6547f)
4. [build a data platform as a product](https://productcoalition.com/how-to-design-and-build-a-data-platform-as-a-product-d22329ff5a3f)
# Data Engineering Questions & Training
#### Data Engineering Questions / Training
1. General
1. [What scales of data have you worked with in the past?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
2. [How do you generally work with the departments that make use of your data?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
3. [Tell me about a time you had performance issues with an ETL. How did you identify this as a performance issue and how did you fix it?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
4. [Describe a time when you found a new use case for an existing database.](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
5. [Describe the most challenging project you’ve worked on. What was your role?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
6. [Think back to a project you’re proud of. What was it that gave you that sense of pride and accomplishment?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
7. [What do you consider to be one of the biggest mistakes you’ve ever made in your previous job?](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
8. [Explain the differences between stream processing and data processing, with one caveat: pretend that I’m not familiar with data at all.](https://business.linkedin.com/talent-solutions/resources/interviewing-talent/data-engineer)
9. What are the considerations when choosing methods of ingesting data to bigquery?
10. Have you worked with data science teams? What were your responsibilities?
11. What are the considerations of choosing spark vs bigquery?
12. What are the differences between ETL and ELT? [1](https://www.guru99.com/etl-vs-elt.html), [2](https://www.xplenty.com/blog/etl-vs-elt/), [3](https://blog.panoply.io/etl-vs-elt-the-difference-is-in-the-how)\
\
By guru99/david taylor
By mark smallcombe
1. CAP Theorem
1. [What Is CAP Theorem?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
2. [Can you 'get around' or 'beat' the CAP Theorem?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
3. [Name some types of Consistency patterns](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
4. [What Do You Mean By High Availability (HA)?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
5. [What are A and P in CAP and the difference between them?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
6. [What does the CAP Theorem actually say?](https://www.fullstack.cafe/blog/cap-theorem-interview-questions)
7. how it effect real world application (latency is availability in real world)
8. [Another great resource for CAP questions](https://github.com/henryr/cap-faq)
9. 
10. 
11. 
12. 
13. 
2. Explain the difference and the reason to choose using NoSQL {mongoDB | DynamoDB | .. } over Relational database {Postgress |MySQL} and vice versa. Give an example for a project where you had to make this choice, and walk through your reasoning.
(This question can be modified for the relevant technologies.. )
1. Streaming vs Batch “Explain the difference and the reason to choose using Streaming over Batch and vice versa. Give an example for a project where you had to make this choice, and walk through your reasoning.”
2. Job vs Service “Explain the difference and the reason to choose using Job over Service and vice versa. Give an example for a project where you had to make this choice, in the context of ML pipelines and walk through your reasoning.”
3. Athena
1. What is the engine behind athena
2. How is presto different from Spark? How does it affect your query planning?
3. [Performance tuning](https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/) - Top 10: partitioning, bucketing, compression, optimize file sizes, optimize columnar data store generation, query tuning, optimize order by, optimize group by, use approx functions, column selection. What are the tradeoffs (time vs cost)?
4. What is the cost composed of?
5. How can you calculate cost?
6. How can you optimize your queries (partitions, join order, limit tricks, etc)
7. What options do you have to limit the cost of athena?
8. when would u use athena vs spark
4. Spark -
1. [Several spark articles that can be used as candidate questions](https://medium.com/@sivaprasad-mandapati) by sivaprasad mandapati.
2. [Join strategies #](https://medium.com/datakaresolutions/optimize-spark-sql-joins-c81b4e3ed7da)1, [Join strategies](https://towardsdatascience.com/strategies-of-spark-join-c0e7b4572bcf) #2 - how? Pros and cons. (broadcast hash, shuffle hash, shuffle sort merge, cartesian).
3. What’s the difference between a data frame and a dataset?
4. [Sort merge vs broadcast](https://medium.com/swlh/spark-joins-tuning-part-1-sort-merge-vs-broadcast-a98d82610cf0)
1. broadcast join is 4 times faster if one of the table is small and enough to fit in memory
2. Is broadcasting always a good solution ? Absolutely no. If you are joining two data sets both are very large broad casting any table would kill your spark cluster and fails your job.
5. [Shuffle & AQE](https://medium.com/@sivaprasad-mandapati/spark-joins-tuning-part-2-shuffle-partitions-aqe-8688cb23317b)
1. Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that makes use of the runtime statistics to choose the most efficient query execution plan.
2. Dynamically coalescing shuffle partitions
3. Dynamically switching join strategies
4. Dynamically optimizing skew joins
5. BigQuery
1. What is the difference in the implementation between partitions and clustering in BQ?
2. What ways do you know to reduce query cost in BigQuery?
3. What is the BigQuery cost composed of? How can you reduce storage cost?
4. Did you ever encounter a memory error when running BigQuery? Why does it happen and how is it related to the Dremel implementations
5. How can you control the access to sensitive data in BigQuery?
6. What options do you have to limit the cost of BigQuery?
7. When using BigQuery ML to train TF models - what happens in the background?
6. Airflow
1. What is airflow?
2. How do you transfer information between tasks in airflow?
3. Please give me a real-world example of using spark and airflow together
7. Data Validation
1. How can you protect yourself from bad data? Data validation, TDDA, monitoring.
2. Tools:
1. Type validation: [typeguard](https://github.com/agronholm/typeguard)
2. Data validation [pydantic](https://pydantic-docs.helpmanual.io/usage/dataclasses/)
3. Test driven: [tdda](https://github.com/tdda/tdda)
4. Data quality: [great expectations](https://greatexpectations.io/)
5. Saas: [SuperConductive by GE](https://superconductive.ai/)
8. File formats
1. Can you explain the parquet file format?
2. How is this leveraged by Spark?
3. What are the shortcomings of parquet and how is it solved by file formats like hudi, delta, iceberg?
9. Julien simon on [AWS glue data brew vs data wrangler](https://julsimon.medium.com/data-preparation-aws-glue-data-brew-or-amazon-sagemaker-data-wrangler-d8e76d1510cb)
10. [What is a CDC and why do you need it, or how do you use it?](https://rockset.com/blog/change-data-capture-what-it-is-and-how-to-use-it/) - Change data capture (CDC) is the process of recognising when data has been changed in a source system so a downstream process or system can action. A common use case is to reflect (replication) the change in a different target system so that the data in the systems stay in sync.
11. Outage handling and the differences between stream-based processing vs concurrent isolated worker-based processing using
Q: you have a real time stream - what is better? A stream-based processing system, or a worker-based, that can be triggered on different time ranges, in the context of recovery from outage.\
By nielsen Ilai Malka
1. [How to design a](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions) .
* How would you design and implement an API rate limiter?
1. [The twitter question](https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/twitter/README.md)
1. [More than 2000 questions for data engineers](https://github.com/OBenner/data-engineering-interview-questions)
1. [More data engineering questions](https://realpython.com/data-engineer-interview-questions-python/)
1. [Even more qs](https://www.softwaretestinghelp.com/data-engineer-interview-questions/)
References:
1. [DS leads](https://docs.google.com/document/d/1gdfJce0p7jx0ptHJt3NE3NIhvzZCXJ5O0V-dvJse0HI/edit)
2. [System design interview q’s with solutions](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions)
3. [Cap theorem](https://github.com/donnemartin/system-design-primer#system-design-interview-questions-with-solutions), [2](https://github.com/henryr/cap-faq) (which is great), [3](https://github.com/kislerdm/data-engineering-interviews) (isn't complete)
4. [ACID](https://bardoloi.com/blog/2017/02/26/db-deep-dive/), [CAP](https://bardoloi.com/blog/2017/03/06/cap-theorem/), [PACLEC](https://bardoloi.com/blog/2017/03/06/pacelc-theorem/)
5. [Why do we need Data engineering?](https://podcastaddict.com/episode/116229803) (podcast)
# Data Patterns
1. [Slowly Changing Dimensions (SCD)](https://adatis.co.uk/introduction-to-slowly-changing-dimensions-scd-types/) by adatis - what can you do when the information in your table changes, types 0-6. Image by Adatis.co.uk.\
# Data Architecture
1. [how to build modular data stack](https://medium.com/the-prefect-blog/how-to-build-a-modular-data-stack-data-platform-with-prefect-dbt-and-snowflake-89f928974e85)
2. Nielsen [multi petabyte data platform using EMR](https://www.youtube.com/watch?v=9hUTkBsuIZA)
3. [Evolving and scaling the data platform at Yotpo](https://www.dataengineeringpodcast.com/yotpo-data-platform-architecture-episode-285/)
4. [Intro to domain driven design (DDD)](https://medium.com/latentview-data-services/introduction-to-domain-driven-design-part-1-of-2-6630f72f2853)
# Data Platforms
## Databricks
1. [Databricks is ACID](https://databricks.com/glossary/acid-transactions#:~:text=ACID%20is%20an%20acronym%20that,operations%20are%20called%20transactional%20systems.)
2. DB Learning Library
1. [Free courses](https://www.databricks.com/training/catalog?costs=free)
2. [Docs Optimization recommendations](https://docs.databricks.com/en/optimizations/index.html)
3. [Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads](https://www.databricks.com/discover/pages/optimize-data-workloads-guide)
4. [Vector Search](https://www.databricks.com/training/catalog/new-capability-overview-vector-search-2535)
5. DB [for ML](https://www.databricks.com/training/catalog/get-started-with-databricks-for-machine-learning-2460)
6. [DB for Data Engineering](https://www.databricks.com/training/catalog/get-started-with-databricks-for-data-engineering-1511)
3. (good) [Introduction & Tutorial](https://medium.com/@chuck.connell.3/databricks-a-history-and-introduction-438ce827227) - cluster / notebook / table / SQL / DataFrame / connections
4. [must know 7 concepts](https://www.datacamp.com/tutorial/introduction-to-databricks)
5. RDD vs Dataframe vs Dataset
1. [2016 official blog post](https://www.databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html)
2. [linkedin blog post](https://www.linkedin.com/pulse/rdd-vs-dataframe-dataset-sanyam-jain-iwsfe/)
3. [comparison on youtube](https://www.youtube.com/watch?v=aBUqIAGxeg8)
4. [RDDs vs. Dataframes vs. Datasets – What is the Difference and Why Should Data Engineers Care?](https://www.analyticsvidhya.com/blog/2020/11/what-is-the-difference-between-rdds-dataframes-and-datasets/)
6. Optimizations
1. [Optimization recommendations on Databricks](https://docs.databricks.com/en/optimizations/index.html)
2. [Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads](https://www.databricks.com/discover/pages/optimize-data-workloads-guide)
3. [How I Use Caching in Databricks to Increase Performance and Save Costs](https://blog.det.life/caching-in-databricks-explained-68c07bf1f76b)
4. [Why and How: Partitioning in Databricks](https://medium.com/@eduard2popa/why-and-how-partitioning-in-databricks-e9e6f960db43)
7. Best Practices
1. [official docs](https://docs.databricks.com/en/delta/best-practices.html)
# Data Lineage
Data lineage refers to the detailed history of data as it moves through various stages and transformations in an information system. It's essentially the life cycle of data, from its origins to its endpoint, including how it is modified and processed over time. Understanding data lineage is crucial for several reasons:
* Traceability - It helps track where data comes from, which is vital for debugging issues, understanding dependencies, and ensuring data quality.
* Compliance - Many regulatory requirements, such as GDPR and HIPAA, require knowing the flow of data to ensure it's handled securely and within legal parameters.
* Data Governance - It aids in managing data, understanding its utility, and ensuring that data usage is consistent with organizational policies.
* Impact Analysis - It allows organizations to assess the potential impact of changes in the data environment. This is crucial for risk management and strategic planning.
* Audit and Reporting - Data lineage provides transparency for audits, ensuring that all data used in financial reporting, for instance, is accurate and verifiable.
Tools and systems that manage data lineage collect metadata from various parts of data handling systems, providing a visual or documented trail of how data flows through software and systems, which transformations it undergoes, and how it's used in different analyses and decisions. This capability is particularly important in complex systems where data is handled across various platforms and services.
## Good Articles
* [What is Data Lineage?](https://www.octopai.com/what-is-data-lineage/)
* [Data Lineage](https://www.ardoq.com/knowledge-hub/data-lineage)
* [The Complete Guide to Data Lineage: Benefits, Techniques, and Best Practices](https://www.selectstar.com/resources/the-complete-guide-to-data-lineage-benefits-techniques-and-best-practices)
## Data lineage vendors
* [Octopai](https://octopai.com/)
* [Collibra](https://www.collibra.com/)
* [Azure Purview](https://learn.microsoft.com/en-us/purview/purview)
* [Cloudera](https://www.cloudera.com/)
* [Alation](https://www.alation.com/)
* [Apache Atlas](https://atlas.apache.org/)
# DevOps Strategy
1. (good) [8 Devops Strategies and how to implement them](https://www.spiceworks.com/tech/devops/articles/devops-strategies-to-test/)
# DevOps Tools
# Tutorials
* [Kubernetes, sklearn, s2i, gcloud, seldon random serving for ab testing](https://medium.com/analytics-vidhya/manage-ml-deployments-like-a-boss-deploy-your-first-ab-test-with-sklearn-kubernetes-and-b10ae0819dfe)
* [Polyaxon - training, argo-package/deployment , seldin -serving](https://medium.com/analytics-vidhya/polyaxon-argo-and-seldon-for-model-training-package-and-deployment-in-kubernetes-fa089ba7d60b)
# Continuous Integration
1. [Travis](https://travis-ci.org/)
2. [Circle CI](https://circleci.com/)
3. [TeamCity](https://www.jetbrains.com/teamcity/)
4. [Jenkins](https://www.jenkins.io/)
5. Github Actions
1. [poetry black pytest](https://medium.com/@vanflymen/blazing-fast-ci-with-github-actions-poetry-black-and-pytest-9e74299dd4a5)
# Docker
* [What are docker layers](https://medium.com/@jessgreb01/digging-into-docker-layers-c22f948ed612)?
* [Install on ubuntu](https://linuxconfig.org/how-to-install-docker-on-ubuntu-18-04-bionic-beaver)
* [Many jupyter docker images (spark too)](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html)
* [How to run jupyter docker 1](https://medium.com/@rahulvaish/jupyter-docker-badd38fd6b51), [2](https://medium.com/fundbox-engineering/overview-d3759e83969c)
* [Tell docker to run on a mounted disk](https://stackoverflow.com/questions/32070113/how-do-i-change-the-default-docker-container-location)
* [Docker, keras, k8s, flask serving](https://medium.com/analytics-vidhya/deploy-your-first-deep-learning-model-on-kubernetes-with-python-keras-flask-and-docker-575dc07d9e76)
* [Compose](https://docs.docker.com/compose/) - run multi coker applications.
* [Docker on ubuntu, tutorial](https://medium.com/fundbox-engineering/overview-d3759e83969c)
* [Containerize your ds environment using docker compose](https://towardsdatascience.com/containerize-your-whole-data-science-environment-or-anything-you-want-with-docker-compose-e962b8ce8ce5) - Docker-Compose is simply a tool that allows you to describe a collection of multiple containers that can interact via their own network in a very straight forward way,
* [docker for data science](https://aoyilmaz.medium.com/docker-in-data-science-and-a-friendly-beginner-to-docker-186fafdfbdeb)
* [using vscode to debug containers](https://nirradi.medium.com/vsc-vs-pycharm-developing-inside-docker-containers-4892c83d30e4)
### S2i
* Builds docker images out of gits
# Kubernetes
### Kubernetes
* For beginners:
* [1](https://medium.com/containermind/a-beginners-guide-to-kubernetes-7e8ca56420b6), [2](https://medium.com/faun/kubernetes-basics-for-new-users-d57fdf85adba)\*, [3](https://medium.com/google-cloud/kubernetes-101-pods-nodes-containers-and-clusters-c1509e409e16)\*, [4](https://medium.com/swlh/kubernetes-in-a-nutshell-tutorial-for-beginners-caa442dfd6c0), [5](https://kubernetes.io/docs/tutorials/kubernetes-basics/)\*, 6,
* Advanced
* [1](https://www.freecodecamp.org/news/learn-kubernetes-in-under-3-hours-a-detailed-guide-to-orchestrating-containers-114ff420e882/), 2, 3,
* [A list with all the tools ](https://collabnix.github.io/kubetools/)
### Helm
* [Package manager for kubernetes](https://helm.sh/)
### Kubeflow
* [Youtube - the easy way,](https://www.youtube.com/watch?v=P5wcE4IwKgQ) [intro](https://medium.com/@amina.alsherif/how-to-get-started-with-kubeflow-187792f3e99)\*, [intro2\*](https://kubernetes.io/blog/2017/12/introducing-kubeflow-composable/), [intro3](https://medium.com/better-programming/kubeflow-pipelines-with-gpus-1af6a74ec2a),
* [Really good detailed article, for example it supports many serving options such as seldon](https://ubuntu.com/blog/ml-serving-models-with-kubeflow-on-ubuntu-part-1)
* [presentation](https://www.oliverwyman.com/content/dam/oliver-wyman/v2/events/2018/March/Google_London_Event/Public%20Introduction%20to%20Kubeflow.pdf)
* Tutorials:
* [Official example](https://github.com/kubeflow/example-seldon)
* [Step by step tut](https://codelabs.developers.google.com/codelabs/cloud-kubeflow-e2e-gis/index.html?index=..%2F..index#0)\*
* [endtoend tut](https://journal.arrikto.com/an-end-to-end-ml-pipeline-on-prem-notebooks-kubeflow-pipelines-on-the-new-minikf-ee618b7dc7de),
* [really detailed tut](https://towardsdatascience.com/how-to-create-and-deploy-a-kubeflow-machine-learning-pipeline-part-1-efea7a4b650f)
* KF + [Seldon on ec2](https://docs.seldon.io/projects/seldon-core/en/latest/examples/kubeflow_seldon_e2e_pipeline.html)
### MiniKF
* [Tutorial](https://journal.arrikto.com/an-end-to-end-ml-pipeline-on-prem-notebooks-kubeflow-pipelines-on-the-new-minikf-ee618b7dc7de), [youtube](https://www.youtube.com/watch?v=XZGHFktDSE0)
* [ROK - save snapshot of your env](https://journal.arrikto.com/arrikto-launches-rok-and-rok-registry-93d76eb0c3a2)
# Cloud Objects
### AWS Lambda
* [Comparison](https://www.bluematador.com/blog/serverless-in-aws-lambda-vs-fargate) against aws fargate
# Key Value DB
### RabbitMQ
"RabbitMQ is an open-source message-broker software that originally implemented the Advanced Message Queuing Protocol and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol, MQ Telemetry Transport, and other protocols."
* [Producer broker, consumer - a tutorial on what is RMQ](https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html)
* [Part2.3](https://www.cloudamqp.com/blog/2015-05-21-part2-3-rabbitmq-for-beginners_example-and-sample-code-python.html) - python code
* [Part 3](https://www.cloudamqp.com/blog/2015-05-27-part3-rabbitmq-for-beginners_the-management-interface.html) - managing
* [Part 4](https://www.cloudamqp.com/blog/2015-09-03-part4-rabbitmq-for-beginners-exchanges-routing-keys-bindings.html)
### [ActiveMQ ](http://activemq.apache.org/)
Apache ActiveMQ™ is the most popular open source, multi-protocol, Java-based messaging serve
### Kafka
Apache Kafka is a distributed event store and stream-processing platform.
* [Kafka 101 youtube](https://www.youtube.com/watch?v=j4bqyAMMb7o\&list=PLa7VYi0yPIH0KbnJQcMv5N9iW8HkZHztH), [event sourcing & event storage](https://www.youtube.com/watch?v=p_sSRwpBkgs\&list=PLa7VYi0yPIH1TXGUoSUqXgPMD2SQXEXxj\&index=1) with Kafka - confluent
* [Web](https://kafka.apache.org/)
* [Medium - really good short intro](https://medium.com/hacking-talent/kafka-all-you-need-to-know-8c7251b49ad0)
* [Intro](https://medium.com/@jcbaey/what-is-apache-kafka-e9e73884e367), [Intro 2](https://medium.com/@patelharshali136/apache-kafka-tutorial-kafka-for-beginners-a58140cef84f),
* [Kafka in a nutshell](https://medium.com/@aiven_io/apache-kafka-in-a-nutshell-df10dfcc7dc) - But even these solutions came up short in some cases. For example, RabbitMQ stores messages in DRAM until the DRAM is completely consumed, at which point messages are written to disk, [severely impacting performance](https://blog.mavenhive.in/which-one-to-use-and-when-rabbitmq-vs-apache-kafka-7d5423301b58).
Also, the routing logic of AMQP can be fairly complicated as opposed to Apache Kafka. For instance, each consumer simply decides which messages to read in Kafka.
In addition to message routing simplicity, there are places where developers and DevOps staff prefer Apache Kafka for its high throughput, scalability, performance, and durability; although, developers still swear by all three systems for various reasons.
* [Apache Kafka Kafka](https://medium.com/develbyte/introduction-to-zookeeper-bcda7ef136cd) is a pub-sub messaging system. It uses Zookeeper to detect crashes, to implement topic discovery, and to maintain production and consumption state for topics.
* [Tutorial on putting a model in Kafka and using zoo keeper](https://towardsdatascience.com/putting-ml-in-production-i-using-apache-kafka-in-python-ce06b3a395c8) with code.
### KSQLDB
* KSQLDB 101 uses Kafka streams to run queries over kafka, [youtube](https://www.youtube.com/watch?v=oThQzCXjuk4\&list=PLa7VYi0yPIH3ulxsOf5g43_QiB-HOg5_Y)
### Zoo keeper
* Intro [1](https://medium.com/@rinu.gour123/role-of-apache-zookeeper-in-kafka-monitoring-configuration-c5bd1a7e4226), [2 - use cases](https://medium.com/@bikas.katwal10/zookeeper-introduction-designing-a-distributed-system-using-zookeeper-and-java-7f1b108e236e), [3\*](https://medium.com/@ben2460/about-apache-zookeeper-distributed-lock-1a990315e05c), [4](https://www.tutorialspoint.com/zookeeper/zookeeper_overview.htm)
* What is [1](https://medium.com/@gavindya/what-is-zookeeper-db8dfc30fc9b), [2](https://medium.com/rahasak/apache-zookeeper-31b2091657a8)
* It is a [service discovery in a nutshell, Kafka is using it to allow discovery, registration etc of services. So that customers can subscribe and get their publication. ](https://www.quora.com/What-is-ZooKeeper)
# API Gateway
[what is](https://www.javatpoint.com/introduction-to-api-gateways)?
### NGINX
1. [NGINX](https://www.nginx.com/resources/glossary/nginx/) is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. It started out as a web server designed for maximum performance and stability. In addition to its HTTP server capabilities, NGINX can also function as a proxy server for email (IMAP, POP3, and SMTP) and a reverse proxy and load balancer for HTTP, TCP, and UDP servers.
2. Cloudflare on [what is a reverse proxy](https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/)
3. "[One advantage of using NGINX as an API gateway ](https://www.nginx.com/blog/deploying-nginx-plus-as-an-api-gateway-part-1/#:~:text=One%20advantage%20of%20using%20NGINX,deploy%20a%20separate%20API%20gateway.)is that it can perform that role while simultaneously acting as a reverse proxy, load balancer, and web server for existing HTTP traffic. If NGINX is already part of your application delivery stack then it is generally unnecessary to deploy a separate API gateway"
# Infrastructure As code
### Terraform
1. [youtube course](https://www.youtube.com/watch?v=SLB_c_ayRMo)
2. [Terraform vs Ansible](https://www.youtube.com/watch?v=n1zW_QHg8nY)
# Logs
### Logz.io
* [Intro](https://www.youtube.com/watch?v=LqJYeeTss9Q)
* [What is](https://www.youtube.com/watch?v=6VVig5tnTJE)
# ELK
* [Elastic on ELK](https://www.elastic.co/what-is/elk-stack) - "**So, what is the ELK Stack?** "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch. The Elastic Stack is the next evolution of the ELK Stack."
* [Logz.io on ELK](https://logz.io/learn/complete-guide-elk-stack)
* [Hackernoon intro](https://hackernoon.com/elastic-stack-a-brief-introduction-794bc7ff7d4f)
# SLO
1. [Sloth](https://itnext.io/slos-should-be-easy-say-hi-to-sloth-9c8a225df0d4), [Github](https://github.com/slok/sloth) - "Easy and simple Prometheus SLO (service level objectives) generator"
# DevOps Courses
1. [Docker in 2 hours](https://www.youtube.com/watch?v=fqMOX6JJhGo\&list=RDQMjGhJ6Dhkx4Q\&start_radio=1)
2. [Kubernetes crash course](https://www.youtube.com/watch?v=XuSQU5Grv1g)
3. Helm
1. [What is Helm](https://www.youtube.com/watch?v=kJscDZfHXrQ)
2. [Helm Charts](https://www.youtube.com/watch?v=jUYNS90nq8U)
4. [What is minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Fx86-64%2Fstable%2Fbinary+download)
5. [Terraform for beginners](https://www.youtube.com/watch?v=YcJ9IeukJL8)
6. Argo
1. [Events](https://www.youtube.com/watch?v=sUPkGChvD54\&list=PLyicRj904Z9_dGuNs6AN5Khljjn9ssbQ6)
2. [Workflows & Pipelines](https://www.youtube.com/watch?v=UMaivwrAyTA)
3. [Argo CD](https://www.youtube.com/watch?v=vpWQeoaiRM4)
4. [Rollouts](https://www.youtube.com/watch?v=84Ky0aPbHvY)
5. [How to harnass all of the above together](https://www.youtube.com/watch?v=XNXJtxkUKeY)
7. [Bitnami](https://www.youtube.com/watch?v=xd2QoV6GJlc) - K8S secrets in Github
# Definitions
1. [What is by Redhat](https://www.redhat.com/en/topics/devops/what-is-devsecops)
2. [What is?](https://medium.com/@xenonstack/what-is-devsecops-and-benefits-of-adopting-devsecops-1c47e7616722) by xenonstack
3. [Explained - vc perspective](https://medium.com/inside-inovo/devsecops-explained-venture-capital-perspective-cb5593c85b4e) by Kris Przybylak
# Tools
* [Tools by Atlassian](https://www.atlassian.com/devops/devops-tools/devsecops-tools)
# Concepts
1. (cool) [simplified concepts of infosec ](https://securityzines.com/)by Rohit & Anshu
# Problems
1. [6 symptoms for a distributed monolith](https://torvo.com.au/articles/6-symptoms-of-a-distributed-monolith) - in general, coupling of services, deployments, versions, dependencies, sharing the same queues, databases, storage, messaging systems. etc - by Ben Brazier
# Development Concepts
## PATTERNS
1. [**Software and architectural design patterns**](https://github.com/DovAmir/awesome-design-patterns)
## Programming Concepts
[Dependency injection](https://www.freecodecamp.org/news/a-quick-intro-to-dependency-injection-what-it-is-and-when-to-use-it-7578c84fa88f/) - based on [SOLID](https://scotch.io/bar-talk/s-o-l-i-d-the-first-five-principles-of-object-oriented-design#toc-single-responsibility-principle) the class should do one thing, so we are letting other classes create 3rd party/class objects for us instead of doing it internally, either by init passing or by injecting in runtime.
[SOLID](https://scotch.io/bar-talk/s-o-l-i-d-the-first-five-principles-of-object-oriented-design#toc-single-responsibility-principle) - the five principles of object oriented.
# System Design
1. (Amazing)[ System Design in a Hurry](https://www.hellointerview.com/learn/system-design/in-a-hurry/introduction)
2. (Amazing) [**System Design and architecture**](https://github.com/puncsky/system-design-and-architecture)\\
3. [Hired in tech](https://www.hiredintech.com/system-design/) - high level
4. [Episode 06: Intro to Architecture and Systems Design Interviews](https://www.youtube.com/watch?v=ZgdS0EUmn70) - an approach
5. [System design template](https://leetcode.com/discuss/career/229177/My-System-Design-Template)
6. [RAG SD](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/)
7. Jordan
1. [Distributed Job Scheduler](https://www.youtube.com/watch?v=pzDwYHRzEnk)
2. [Distributed Locking](https://www.youtube.com/watch?v=Lp8oITg0MiI)
3. [Rate Limiter](https://www.youtube.com/watch?v=VzW41m4USGs)
8. [(great) System Design Primer](https://github.com/donnemartin/system-design-primer/tree/master) -Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
9. [CS75 (Summer 2012) Lecture 9 Scalability Harvard Web Development David Malan](https://www.youtube.com/watch?v=-W9F__D3oY4)
10. Scalability
1. [a word about](https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html)
2. [scalability for dummies](https://web.archive.org/web/20221030091841/http://www.lecloud.net/tagged/scalability/chrono) [1](https://web.archive.org/web/20220530193911/https://www.lecloud.net/post/7295452622/scalability-for-dummies-part-1-clones) [2](https://web.archive.org/web/20220602114024/https://www.lecloud.net/post/7994751381/scalability-for-dummies-part-2-database) [3](https://web.archive.org/web/20230126233752/https://www.lecloud.net/post/9246290032/scalability-for-dummies-part-3-cache) [4](https://web.archive.org/web/20220926171507/https://www.lecloud.net/post/9699762917/scalability-for-dummies-part-4-asynchronism)
3. (good for topics) [Scalability, Availability & Stability Patterns](https://www.slideshare.net/slideshow/scalability-availability-stability-patterns/4062682)
4. (good) [Use Back-of-the-envelope-calculations to Choose the Best Design](https://highscalability.com/google-pro-tip-use-back-of-the-envelope-calculations-to-choo/)
11. [System Design Guided Practice](https://bugfree.ai/) - AI-Powered platform provided guided practice on system design problem and behavior questions like the way you do at Leetcode.