Mike Allen Systems

Doing Smarter Things With IT

Analysis and the Design of Computer Systems

Good computer systems don’t just spring up of themselves, they require a lot of work from end users, consumers, managers, analysts, designers, documenters,  developers and testers. These are all roles in the development process, it is not meant to be definitive, a single person may have multiple roles, a single role will often have multiple persons. For small systems a few people may occupy most of the roles, there may be a whole class of roles, such as stakeholders, who intersect other roles. In order to disentangle these roles and relationships and create a good computer system we use analysis.

Analysis is sometimes seen as an optional extra, on small projects where there is a temptation to start first and repent at leisure, on large projects it can get lost in a maze of priorities, lists and milestones, then repentance is less leisurely. Analysis starts with some introspection, and a review of where a project is at and where it is going. A good project is linked to a business initiative, but there can be cases where the computer system itself is the trigger for a remediation of replacement project. A useful ingredient is cost benefit analysis and risk assessment. These activities need to be conducted rigorously to ensure outcomes satisfy user expectations and that risks are acknowledged and ameliorated.

Having attended to the basic prerequisites basic prerequisites of cost and risk, the analyst’s attention must turn to the business domain. The analyst, or the team of analysts, will attempt to decompose the business problem into it’s component parts and interact with subject matter experts and stakeholders. One of the interesting things about subject matters experts is that they exist at all levels of an organization and may rarely get together to share their views, so discovery will be multifaceted at all of, and between organizational levels.

At this point many stakeholders will, no doubt,  keen to see lists of requirements, process models and specifications. The Analysts will be trying to internalize a large quantity of information in order to create a broad brush design of a workable solution to the business problem. These goals and aspirations are not necessarily incompatible with each other, but they may not be naturally compatible.

The initial analysis and design process can be thought of as antithesis, synthesis and thesis. The decomposition of the problem domain is the antithesis, the design process is the synthesis and the design is the thesis. Ensuing iterations of the development process repeat this cycle, which can also be viewed as a continuous process, as synthesis continues until the goals are achieved.

Power BI, the new desktop multi-tool

I’ve been creating short videos for Power BI and Excel users for around a year now, publishing one a month to my You Tube channel. I’m starting to get a feel for what people are looking for in Power BI, and I am surprised. My most recent video Power BI - scraping the web is a real hit! This says to me that not only is Power BI a superb business intelligence tool, it’s also handy for data extraction and manipulation. The other real lesson is that observation of how people approach a new app can indicate areas of potential growth.

Offering more cloud solutions

I’ve been ramping up my BI offering, centered around PowerBI. In order to offer my clients a more complete solution I have become a cloud solution provider (CSP), partnered with rhipe.com.au. This enables me to offer a range of Microsoft products including Office 365, Power Bi Pro and Azure services. I’m still adjusting my offering, but, if it’s Microsoft, I can supply it to you at a reasonable price and offer some excellent support.

Focusing on Business Intelligence and Power BI

Much of my work over the last few years has focused on Data,Reporting and ETL, all components of Business Intelligence, BI. Now that Power BI is establishing itself as a BI market leader I find that I can re-focus my business more easily on BI and I intend to promote and market Power BI, and my abilities and experience in Power BI. I have been pursuing this in two areas, I am creating BI how-to videos on YouTube , and I have a new Mike Allen Systems web site to again focus purely on BI. Check out my videos and new web site, but most importantly check out Power BI , it’s a free download!

Microsoft Professional Program for Data Science

I recently completed the new Microsoft Professional Program Certificate in Data Science. I've a background in software and data migration, and I knew how to use the tools. The course was intended to formalise my skills. Well, was I in for a surprise.

I'd already completed the Power BI Course, and first up was Querying Data with SQL, what could possibly go wrong? The courses were through edX and were mainly USD $50, they have all gone up to $100 since then.

The first speed bump was choosing a statistical language, I flipped a coin between R and Python a few times, and the gods of statistics seemed to favour R. There was also a new plug-in for R in Visual Studio, which made life easier. I kind of fell in love with R, anyone who likes to manipulate a whole dataset at a time would be at home with this language. My favourite bit of code just extracts the year from a date column and adds it as a new column on the end, cool! The density graphs are an absolute delight, as are all the graphical functions. There are R packages for everything and a vibrant user community, this is all good. I also found R useful for generating test data.

Next up was learning some real Statistical Thinking with ColumbiaX. I had done my stats training years ago while studying Applied Science. This course cleaned out all the cobwebs and got me enthused, there are more stats courses from ColumbiaX which I may get into in the future. Great stuff to get one Statistically Thinking. They've taken this off the Data Science track now, I can see why, but it needs a suitable replacement.

Then things got a little more technical with the Principles of Machine Learning. I'll never look at a regression graph in the same way. the introduction to Azure Machine Learning was a real epiphany of drag and drop Data Science. This is amazing stuff and right at the leading edge of cloud technology. I've has some exposure to Azure but this was really something else. Although calling an Azure Web Service from Excel did give me a touch of cognitive dissonance. Another great feature I found in Azure was Jupyter Notebook, which is a lovely open source intelligent notebook that can execute code. Wow! Executable notes!

I think every design team could benefit from this, what a way to prototype data transformations.

My decision to do the Developing Intelligent Application was a little off track, but I bought a book on C# to help me. This is part of the Applied Data Science section and there are some more data oriented courses available. Developing a bot was fun, and there was lot in this course. Analysing Twitter Feeds was great and bought in the Azure IoT hub and other Azure elements. This is obviously very popular and you can do all this with Microsoft Flow now :-(. Lot's more technical stuff in this one, I've even made friends with JSON. I did discover that SQL is useful for data analytics, the old is made new again!

The final Capstone project was a return to data analysis and Machine Learning. The hardest bit was writing the Data Analysis report and reviewing other students’ efforts.

This Certification took me over six months to complete and I learned a lot along the way. I paid for all the courses to get certification, however casual auditing of the course is available. The Professional Program offers a good conceptual learning alternative for those not seeking Microsoft technical certification. I suppose the down side is that you are expected to do a lot of technical things, like C#, and get no real certification for it.

I can't speak more highly of edX. Their courses are so well organised, the platform is easy to use and the content is excellent. Microsoft have made a great choice in partnering with them.

Was it all worth it? Absolutely! I'll be back to edX as soon as I've finished my book on Bayesian Analysis :-).

Bell and Whistle V2.0 – bigger and better

I was reasonably happy with my first Bell and Whistle data set, but I felt it needed more data, a lot more. I’ve just done a course on using R for simulations, so I thought I would generate some test transactions with ‘days’ stretching over 10 years and a variety of customer and product numbers. I settled on 113 Customers and 50 Products and used a randomized section of a normal distribution to give me around 50,0000 transactions that became denser over time. That’s a complicated way of saying sales increased over time.

What’s it for? Primarily for demonstrating data visualizations in products like Excel, PowerBI and whatever takes you fancy. I’ve been using the previous version to produce some training videos https://www.youtube.com/channel/UCtvirMVp8r8j6jGbWE8SqIA

I’ve decided to distribute Bell and Whistle V2.0 as an Access Database and as a Excel workbook that uses the data in a Data Model. You’ll need Excel Pro to fully realize the work book. The Access database is just data, with the table relationships. The Excel data model has some DAX and Measures.

These file are available on Google Drive try, https://drive.google.com/open?id=0B9nJpvPnppwldm9tWU9GbV9CT2s for the Access Database and https://drive.google.com/open?id=0B9nJpvPnppwlcy1WRW9hRDZRUG8 for the Excel Workbook.

Have a play, let me know if you like it.

Project 21–A more efficient IT methodology

For quite some time now I’ve been working on an IT development methodology called ‘Project 21’. This has emerged from a number of medium sized projects where no methodology was in place and a lightweight pragmatic approach was required. I’ve just migrated the web site to a new host and the latest version of WordPress. This year I’m going to do some more work on it, particularly to address the development phase and the implications of Agile on this sort of methodology. Project 21 is designed to be used and does not attempt to comply with any particular standard. Unfortunately standards compliance would be self-defeating as this methodology is designed to be minimalistic and something that can be tailored to individual project requirements. This is the Project 21 web site. Please feel free to use this content, any attribution or feedback would be more than welcome.

Bell & Whistle Dataset

In the past I’ve often demonstrated concepts to clients using their own data, recently I have been working on a dataset using my own data so the I can share demonstration and training videos and other materials more readily.

My first effort was a training video for pivot tables, although it ended up being more about how to import CSV data into Excel!


The current state of the data is here in a Access Database BellAndWhistleV1.2

This data is supplied free and with no guarantee, download it, use it, attribute if appropriate.

Analysing Data Ownership

Finding data ownership can be a really worthwhile task. There you are with the task of documenting a database, someone suggests that you google a few definitions and and do a bit of cut and paste. Then you decide to do something worthwhile, and determine who owns the data, this may cause a few sharp intakes of breath from those that think the IT department owns the data. If you want to attach a bit of methodology, think of applying a Zachman Framework to the data structure as a business model (don’t get too hung up on Zachman). Once you’ve started identifying ownership you can start thinking about dependencies, permissions, life-cycle’s and all sorts of other data goodness. That list of data owners will be useful for a number of roles, managers, analysts, testers etc. The structure may not be simple, you may find records with multiple owners and have to negotiate who is the master, and it’s a good idea to have a single master, for transactional data it may be you!

This is where the a governance model comes into play, IT can be the keeper of the metadata, assuming you have the luxury of an IT department else the responsibility falls to the data owner. The owner will have contributors and consumers and needs to be aware of who they are and what their rights are. Individual data items may be owned by customers, please consider confidentiality and privacy considerations. Useful information can be collected down to the individual data item, you may need to define some domain information for some data items, alternate names and scope.

Once all this information is is collected, or maybe even sooner, it is important to publish, and publish so that all stakeholders can access the information. Then when someone says “We need a new keyword for this!”, they can find the ownership and responsibilities of “keyword” and act accordingly. Database documentation, if you build it well, it will be adopted and maintained by the business as an asset.

A New Blog, a Fresh Start

In the beginning I was in love with WordPress and how simple it was. When I wanted to host my own blog, it got more complicated but it was bearable. I started using ASPX and C# a little, but I stuck with WordPress as there was no equivalent C# available. We used ScrewTurn an ASPX, C# product at work, I selected it, but then it was discontinued. Then I had a few issues with WordPress and I thought I’d look around and BlogEngine caught my eye, ASPX, C# Open Source, fresh and fast, what wasn’t there to love. Even better Open Live Writer became available on the Windows store, a match made in code,

So here we are with a fresh blog and a new blogging client, I’ve cleared out some old content and I’m ready to blog again.