Data Engineer · Step 8 of 8 · 10 min
Build a portfolio pipeline
Plan and ship one small, end-to-end pipeline that shows an employer how you work.
One finished project beats five tutorials
Hiring managers want to see that you can take data from a real source to a trusted table, and explain your choices. One small project, done properly and documented, does this better than a list of course certificates.
A shape that works
Pick a free public source that updates, such as weather, public transport, open government data or exchange rates. Extract it daily with Python. Land the raw data as files or tables. Transform it with SQL, ideally with dbt, into a small star schema. Add tests. Schedule it with an orchestrator or a scheduled GitHub Actions workflow.
DuckDB keeps it free and local. If you use a cloud warehouse, set a budget alert first and stay inside the free tier.
What to show
Put the code in a public Git repository. The README should state the question the data answers, a diagram of the pipeline, the grain of each table, the tests you run, and how to run it. Add one short section on what you would change at larger scale.
Show one failure you hit and how you fixed it. That tells an employer more than a perfect run.
Keep secrets out
Never commit API keys or passwords. Use environment variables or your platform's secret store, and add a .gitignore before the first commit. If a key does slip into Git history, revoke it at once; deleting the file is not enough.
Try it
Write your project plan
- 1.Choose a data source and the question it will answer.
- 2.Ask a chatbot the prompt below.
- 3.Cut the plan until you can finish a first version in two weekends.
- 4.Create the empty repository and README today.
I am building a portfolio data engineering project. Source: [source]. Question: [question]. Tools I want to use for free: Python, DuckDB, dbt, GitHub Actions. Give me a two-weekend plan with milestones, the tables I should build with their grain, five data tests, and a README outline.