Thursday 9 AUGUST 2018
National Convention Centre, Canberra


Jorg Raap
Director, Data Warehouse
Data Warehouse, Intelligence, Identity & Biometrics Systems Branch, Information & Communications Technology Division Department of Home Affairs

Steve O'Neill
Director EDW / Hadoop Data Warehouse, Intelligence, Identity & Biometrics Systems Branch, Information & Communications Technology Division
Department of Home Affairs


About This Masterclass

This workshop will present an overview of the Department of Home Affairs’ Big Data journey.

We’ll commence with an overview of the Department, the types and volumes of data we hold, and why a move into the Big Data space was both necessary and beneficial. We’ll also discuss the opportunities and challenges that arose along the way, and will conclude with a group session where you can apply our experiences to your organisation.


About Jorg

Experienced Director of the Enterprise Data Warehouse (EDW) of a large Government Department. Jorg is responsible for the oversight of, and the end-to-end development/maintenance of the EDW environment including: data acquisition, data integration and data provisioning across multiple technologies in a complex environment. He has recently championed and overseen the introduction of a Big Data environment to prepare the Department for the future as transactions and data volumes increase.

Jorg has successfully balanced managing a 24/7 mission critical environment while introducing the latest capabilities and tools to support new analytical techniques within the Department.

About Steven

Steven O'Neill began his career as a DBA and has about 20 years' experience specialising in Oracle and SQL Server. He started working with Hadoop, using Hortonworks HDP, three years ago and in that time has overseen the implementation of multiple projects involving data archiving, image storage and text indexing.

Steven's current projects focus on analytic workloads, graph storage and analysis, and improvements to streaming ingestion of data into both Hadoop and the Department's Teradata Enterprise Data Warehouse.

Who Should Attend?

  • Executives who need to start or continue with a Big Data implementation.
  • Technicians interested in real-world examples of a Big Data implementation in the public sector.
  • Anyone interested in Data Warehousing and how it can be augmented with a Big Data platform.

Why Attend?

  • Take a step-by-step walk through of the implementation of a Big Data platform at the Department of Home Affairs.
  • Understand how to successfully implement a Big Data platform, secured and operationalised from start to finish.
  • Explore the challenges and opportunities that the Department of Home Affairs faced along the way.

What to bring

We have no particular requirements, but feel free to take notes and ask as many questions as you can.


Wednesday 9 AUGUST 2018 from 9am-4pm


9:00| Arrival


9:30| Commencement and Group Introductions


9:45| Session 1: The Department, Big Data Drivers and Baby Steps
This session will cover:

  • An overview of the Department of Home Affairs including who we are, what we do and the type of data we hold.
  • Big Data in the department – The drivers for Hadoop and our initial use cases
  • Baby Steps – How we started small and delivered real benefits.
  • Early learnings – What learned as we started out and started delivering.

Learning Outcomes: During this session you’ll gain an understanding of the Department and why we implemented a Big Data platform. You’ll find out how we got started, our initial use cases, and why we chose to start small rather than go all out with a complex data lake or deep learning on Day 1.


11:00| Morning Tea & Networking


12:00| Session 2: Challenges and Maturity
This session will cover:

  • The initial challenges we faced, from security to skillsets.
  • How we met those challenges and developed some technical and process maturity along the way.

Learning Outcomes: During this session you’ll gain an understanding of the biggest challenges the Department faced when implementing a Big Data platform, and how meeting those challenges led to the development of maturity in our approach and our processes.  


13:00| Lunch


13:45| Session 3: Opportunities, what’s next, and what about you?
This session will cover:

  • An exploration of the opportunities we saw once we had our Big Data platform established.
  • Where we’re heading next.
  • A group session where you and your colleagues can map out your own journey, whether from the start or from partway through.
Learning Outcomes: This session will start with an overview of the opportunities we saw once our Big Data platform was in place, stable, and well understood by technicians and clients. We’ll also cover some of the work that is on our roadmap.
Drawing upon the learnings and observations from the day, we’ll finish with a group session focusing on how your organisation could make a start, or continue with, Big Data.


16:00| Conclusion