tag:blogger.com,1999:blog-33917763379738606682024-03-09T18:46:24.138-08:00ZAN KAVTASKINMusings about Systems Architecture, Decision Analysis and Software Engineering
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.comBlogger41125tag:blogger.com,1999:blog-3391776337973860668.post-14933731676252497762021-12-21T15:14:00.003-08:002022-01-07T14:53:22.577-08:00Decision Analysis - Pros and cons framework is not best for decision making<p>Pros and cons (for and against list) seems to be used universally. While it might be better than nothing, it has some flaws. Typically users resort to pros and cons when they are stuck and have to make a decision, should they choose A or B or maybe between A, B, C and D.</p>
<p>What typically happens is that pros and cons user will write a pro for A, then that pro will be used as a con for B, pros for B will be used as cons for A and so on. Users tend to look at the problem from a subjective vantage point which makes perspective narrow and from experience it can invite emotive points on to the list. Emotions might have a place on the list given the right context, however they need to be framed correctly. Once the user has written everything down, the user can review the pros, cons and the relevant weights against these points and then decide on the “right” outcome.</p>
<p>I do not know about you, I was frequently underwhelmed by the effectiveness of this method. As by the end of it I was not in a much better place than when I had started. I think this is because at the <u>core pros and cons do not add any new perspective or value</u> to the decision making. What you write down is probably what you already know, and you are writing stuff down because you want to know more as you are stuck. Writing down what you already know is not going to tell you anything new, and that is the problem.</p>
<p>By now you probably want to hear the alternative approach to the pros and cons. Instead <u>write down possible options / solutions and in what situation they would work best</u>. I will now provide a hypothetical example:</p>
<p><b>Requirements:</b> My existing laptop does not have enough storage space and it has become noticeably slower. Get the latest Mac that will meet my requirements.</p>
<p><b>MacBook Air</b>, best choice when:</p>
<ul>
<li>Want to fit it into my backpack as I will need to do work on it outside of my house </li>
<li>Want to keep price of the purchase as low as possible</li>
<li>Type of work that I will be doing does not require large screen </li>
</ul>
<p><b>MacBook Pro 13”</b>, best choice when:</p>
<ul>
<li>Want to fit it into my backpack as I will need to do work on it outside of my house </li>
<li>Type of work that I will be doing does not require large screen </li>
<li>Touch bar aids productivity and is worth the extra cost</li>
</ul>
<p><b>MacBook Pro 16”</b>, best choice when:</p>
<ul>
<li>Touch bar aids productivity and is worth the extra cost</li>
<li>Need a large screen for the “split view” with good resolution to avoid the need of getting a standalone monitor</li>
<li>Need extra compute to run intensive tasks such as naive brute force deep learning algorithms</li>
<li>Price is no longer a constraint </li>
</ul>
<p><b>Mac</b>, best choice when:</p>
<ul>
<li>There is space for a desktop computer in the office room</li>
<li>Family members will be allowed to use it, this might be economical long term as will not need to buy as many computers</li>
<li>Time-sharing with the family members will not be a problem</li>
<li>Need a large screen for the “split view”</li>
<li>Need a high-quality camera for online meetings</li>
</ul>
<p>This analysis is not complete, in the real scenario I would expand on the requirements, provide price for each option and add technical specifications. However, this should give you a feel for this alternative method. What I get out of this method is the value that it adds by <u>making you think through under what conditions certain solutions would work and what makes them stand out in the certain context.</u></p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-10552457791562961912020-09-26T03:23:00.012-07:002022-01-15T12:53:39.409-08:00Lead Time Driven Delivery - Part 5, Practical and closing thoughts<p>It is time to close this mini-series with some practical and personal ideas.</p>
<h2>Work in Progress - Reducing lead time through queue control</h2>
<p>If your team has a backlog than applying Little’s Law will not make much sense to the backlog itself. Little’s Law can be applied only to a queue and in queue systems something needs to come in and then come out. If something stays there and never comes out, then Little’s Law will not hold. From my experience a backlog is not a queue as work enters the backlog and can stay there indefinitely. When someone enters a queue, they are committed and slowly or quickly make their way through it and eventually leave the queue. If you are using Sprints then you can apply Little’s Law to the Sprint, under a Sprint condition team commits to the work, work enters the Sprint and then leaves the Sprint. If you have number of projects that are going through the system that are committed, then you can use Little’s Law for that as well. </p>
<p>Little’s Law is very significant to the LTDD as average Lead Time = Work in Progress / Throughout. This means the more things we commit to the queue the higher average lead time grows. So, it makes sense to pick a strategy where it is possible to quickly make space for new high priority work. This way customers don’t experience long lead times while software teams deliver their long-term commitments. This also tells us something very interesting. Average delivery lead time can be brought under control if your work scheduling is respected, this stabilises average delivery lead time which makes it more predictable and it enables planning and forecasting. However, for an organisation to benefit from low lead time they need to understand what work is important to them and give that work priority.</p>
<h3>Understanding what work is important</h3>
<p>Most of us heard the famous thought experiment: “If a tree falls in a forest and no one is around to hear it, does it make a sound?”. If your software team delivers a brand-new “feature A” that customers don’t care about (as they don’t use it right now), but you don’t fix their bugs, customers cases and answer their questions. Will they perceive your organisation to be responsive (low lead time) or not? In this scenario bugs, customers cases and questions are visible, that is what customer cares about, and this new “feature A” is that tree that has fallen in forest.</p>
<p>Prioritisation will depend on what is the most important thing for your company at that point in time. Maybe other customers have been told that “feature A” is coming and it is in the contract. Maybe “feature A” is amazing and most customers will upgrade to a new tier to get that feature. So, revenue might play a critical role. Maybe your company believes in quality above everything else. This means company might be OK with delivering new functionality a little bit slower while they prioritise customer cases, bugs and answering questions. There are two personas here: internal stakeholders (investors and sponsors) and external stakeholders (customers and partners). Personally, I don’t think internal and external stakeholders are at all mutually exclusive. However, your organisation needs to make a decision, which stakeholders lead time needs to be minimised, internal or external. Of course, there are more levers then that and it is more nuanced, but priority needs to be established. </p>
<p>I think that there is something that we can all learn from the Manufacturing here. Manufacturers have found that health and safety has correlation with productivity, in fact <a href="https://myosh.com/blog/2020/02/12/is-a-safe-workplace-more-productive/" rel="nofollow">Lockheed Martin stated that by focusing on Health and Safety they have experienced 24% productivity increase and 20% reduction in factory costs</a>. Think about that for a second. Does that mean we can focus on delivering quality service and be even more productive? There is no trade off? The answer seems to be yes. Now when it comes to software products some companies choose to sweep bugs under the rug, accept security risks and not deal with quality problems in their products in order to get more features out. This creates more escalations, customer cases, late night calls, bugs, endless cycle of firefighting and hardship. Everyone has to work harder just to keep the lights on. What if software companies on average constantly prioritised "Operational Quality" over new features first. Is it possible that software companies would provide get ~24% productivity increase by working this way? </p>
<h3>Minimising customer facing lead time</h3>
<p>More work there is in progress the higher lead time grows. Let’s say your team is working on important strategic project that will increase company revenue. You are about to commit to a deadline. Before you do it is important to remember that if you commit without leaving any space for customer requests, bugs, or small features customers will have to wait until you have completed your important strategic project. The only way for you minimise customer or contractual (security, SLA, GDPR) lead time is by factoring in some “operational slack” to address customer or contractual concerns. Also please don’t confuse your contingency with “operational slack”. Project contingency deals with project-based risk (discovering unknowns, someone needs few unexpected days off) and “operational slack” is space to deal with day to day operational concerns. This does mean that important strategic project will experienced longer lead time overall, but not at the cost of customer facing lead time. </p>
<h2>Throughput - Ideas to reduce your cycle time</h2>
<p>As you look to improve throughput you will want some explicit examples on what techniques can be used to actually achieve this. Here is a cheatsheet on what you can do per each factor:</p>
<h3>Wait Time</h3>
<p>Minimise knowledge, decisions and work dependencies. Try the following:</p>
<ul>
<li>Reducing handovers</li>
<li>Prioritised Backlog</li>
<li>Planning and Scheduling</li>
<li>Prioritised Product Portfolios</li>
<li>Project Management</li>
<li>Removing dependencies on others</li>
<li>Training and knowledge sharing</li>
<li>Empowering to make local decisions and create knowledge</li>
<li>Single piece flow</li>
<li>Self-service</li>
<li>Just-In-Time</li>
<li>Minimise supporting teams WIP</li>
</ul>
<br />
<h3>Disruption Time</h3>
<p>Minimise expedite (reactive work), rework, interruptions and mental health impact. Try the following:</p>
<ul>
<li>Building quality in to the development process and product</li>
<li>Prioritised Backlog</li>
<li>Planning and Scheduling</li>
<li>Prioritised Product Portfolios</li>
<li>Empowering to make local decisions and create knowledge</li>
<li>Aligning work to an individual and company objectives</li>
<li>Protecting team from disruptions and being proactive</li>
</ul>
<br />
<h3>Task Time</h3>
<p>Minimise / Maximise volume of work, unknowns, complexity, experience, attitude, aptitude and risk. Try the following:</p>
<ul>
<li>Training and knowledge sharing</li>
<li>Components re-use</li>
<li>Involve subject experts</li>
<li>Increase talent retention </li>
<li>Aligning work to an individual and company objectives</li>
<li>Maximise team’s strengths a minimise weaknesses</li>
<li>Continual learning and experimentation</li>
</ul>
<p>Please remember that you can make some quick wins in throughput, however really important breakthrough improvements will take time. These improvements will never really stop either, as you make some improvements, you will find new improvements that were hiding. Then you will make those and find new ones, and so on. It is important to stress that process improvements don’t have to be expensive to implement and most likely you don’t even need to build any additional software to make these improvements happen. From my experience, process changes mostly require a lot of thinking and communication. </p>
<p>I strongly recommend that you log all of the Wait, Disruption and Task Time during a Sprint so that your team can discuss this during the retrospective. Your team needs to take just one improvement away (ideally the one that will make the biggest impact) and actually make the change, that is the key. If after each retrospective team actually reviews issues and implements just one improvement, then after a while there will be no stopping this team. </p>
<h2>Constraints - Remove constraints, reduce wait time</h2>
<p>I am a big fan of theory of constraints and Eliyahu M. Goldratt’s work. After years of use, I have realised that theory of constraints model does not translate literally into knowledge work (this debate is outside of the scope of this blog post). However, I believe that there are few useful mental shortcuts (heuristic) that can be applied to get the benefit from theory of constraints in knowledge work. </p>
<h3>Dependency constraint heuristic</h3>
<p>If you would like to know if there is an operational constraint in the system just listen to the people say: “We are constantly waiting for X”, “How can X be so slow?”, “I can never get hold of X”, “They are just so busy, but we really need X”, “They keep promising that they will get it done by it never happens”, “Quality from X is never good enough”, “X is constantly down, this really slows us down”, etc. These constraints slow down the whole system as it can’t perform to the optimal levels. This means these individuals (can also be technology .e.g. build servers) are not delegating enough (usually managers), not saying no to things enough (anyone who over commits) or there are not enough of people to do the work (hands on people i.e. not managers) to remove this constraint. </p>
<h3>Change constraint heuristic</h3>
<p>If you don’t see enough change in the process it normally means that people who are supposed to be implementing the change are not prioritising it as a top priority (they are either over committed, can’t delegate or prioritise). This creates a big problem with opportunity cost. By this individual being a constraint to a process change you cannot enjoy the benefits of the change and you will not get to the desired destination sooner. Assuming that these individuals need to make the change, you need to implement a "change circuit breaker", which individual just prioritises the changes in over everything else for a short while (gracefully without impacting the customer lead time of course). If this does not happen then opportunity cost will just keep growing.</p>
<h2>Conclusion</h2>
<p>Lead time consists of number of items in the queue and your team’s throughput speed. It is possible to provide low lead time to your customers by leaving space in the queue for customer requests (Little’s Law). However, your investors and sponsors will most likely want you to also focus on getting your throughput improved so that they get more for their investment. Throughput is made up from three factors: wait, disruption and task time, by eliminating wait and disruption and minimising task time you can finally increase throughput speed. Once you start to eliminate wait, disruption and minimise wait time it might force you to go beyond your existing agile framework methods. As you focus on results and not methods you might end up questioning your long-term beliefs about what actually makes your team and your organisation productive. </p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-73928049707031372522020-09-21T15:00:00.009-07:002022-01-15T12:52:56.213-08:00Lead Time Driven Delivery - Part 0, Introduction<strong>Contents:</strong>
<ul>
<li><a href="http://www.zankavtaskin.com/2020/09/lead-time-driven-delivery-introduction.html">Part 0 - Introduction</a></li>
<li><a href="http://www.zankavtaskin.com/2020/01/applied-software-delivery-system.html">Part 1 - Learning to see</a></li>
<li><a href="http://www.zankavtaskin.com/2020/03/lead-time-driven-delivery-metrics.html">Part 2 - Learning from data</a></li>
<li><a href="http://www.zankavtaskin.com/2020/08/lead-time-driven-delivery-focus-on.html">Part 3 - Focus on results, not methods</a></li>
<li><a href="http://www.zankavtaskin.com/2020/09/lead-time-driven-delivery-stabilise-and.html">Part 4 - Stabilise through embedded testing</a></li>
<li><a href="http://www.zankavtaskin.com/2020/09/lead-time-driven-delivery-practical-and.html">Part 5 - Practical and closing thoughts</a></li>
<li><a href="http://www.zankavtaskin.com/2018/10/applied-software-delivery-optimal.html" target="_blank">Applied Software Delivery - Optimal Backlog</a></li>
<li><a href="http://www.zankavtaskin.com/2018/09/applied-software-delivery-full-stack.html" target="_blank">Applied Software Delivery - Full Stack Developer vs Partial Stack Developer</a></li>
<li><a href="http://www.zankavtaskin.com/2018/08/applied-software-delivery-hidden-impact.html" target="_blank">Applied Software Delivery - Hidden Impact Of Team Leaders</a></li>
</ul>
<p>Lead Time Driven Delivery (LTDD) approach has emerged from personal need to improve software delivery teams speed, LTDD is an extension of your Agile framework and it attempts to fix the following problems:</p>
<ul>
<li><strong>Agile frameworks tend to be collection of methods from industry practitioners.</strong> Most of these methods do not have any real evidence behind them that they actually work. Agile frameworks don’t necessarily have clear focus on what result they are trying to achieve, that is apart from vague "delivering value to customer” which is hard to measure.</li>
<li><strong>Once organisations roll out Agile framework, it is not happily ever after.</strong> Some organisations start to deliver software slower, some speed up. However, no matter what happens, organisation's sponsors expect continuous improvement, so what’s the next improvement? How do you know what you can and can’t change? Are you bound to the Agile framework methods? </li>
<li><strong>New practitioners and managers starting in the industry should not need years of experience to learn (often arbitrary) methods </strong> to be able to understand the main delivery concepts of why they are following some method, how it is applied and how they can make further improvements. </li>
<li><strong>Certain scientific manufacturing management paradigms and models such as Theory of Constraints, 8 Wastes, etc don’t translate well into knowledge work.</strong> In fact, some aspects are hurtful and damaging to the knowledge work. </li>
<li><strong>Software engineering department is not the only department in your company</strong>, how do you integrate your Agile Framework with Sales? Customer support? Implementations? </li>
</ul>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPfGquAMCFo3PK-OVp1jOcNgJxaQ_IM9jnWI00MhgAB_I6B20Cfiz_9ZuiZ89kGYe5ZyBP0Yp2hk-DIiaiWc_tU9fEv630YOxPJoWepi_V8tcDf4B1ueZUtOlVmBi8AraGS5wH26qylW0/s620/asian_garden.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="620" data-original-width="620" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPfGquAMCFo3PK-OVp1jOcNgJxaQ_IM9jnWI00MhgAB_I6B20Cfiz_9ZuiZ89kGYe5ZyBP0Yp2hk-DIiaiWc_tU9fEv630YOxPJoWepi_V8tcDf4B1ueZUtOlVmBi8AraGS5wH26qylW0/w400-h400/asian_garden.jpg" width="400" /></a></div>
<p>
As a practitioner if you have identified similar problems then you might be happy to know that you are not alone, maybe this short series will give you some ideas on how you can further improve your team and your organisation overall. LTDD is not a collection of specific delivery methods such as pair programming, sitting together, using story points etc, this is already covered in abundance. LTDD is a framework and a way of thinking, it frees you from the Agile method and it allows you and your organisation to choose the methods that minimise your organisation’s lead time.
</p>
<p>The name "Lead time driven delivery" name comes from research book called “ACCELERATE Building and Scaling High Performing Technology Organisations", this book identified KPIs that seem to correlate with profitability of organisations, and lead time is one of them. This is hardly surprising, our sponsors and customers don’t care that you have taken 5 minutes to make a software change but have taken 3 months to ship this change to production, all your customers see is 3 months elapsed time and not that 5 minutes. So, if lead time makes your organisation respond to market changes faster and provide better customer experience than why is this not your number #1 KPI?</p>
<p>This short series will attempt to give you some tools to make a change, and a really great thing is that it does not matter where you are and it does not matter how long it will take you to reduce that elapsed time from 3 months to 5 minutes, what matters is that you make a start and work with your peers through these problems, that collaboration is the real transformation. </p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-29286463924399688132020-09-03T14:37:00.011-07:002022-01-15T12:53:26.862-08:00Lead Time Driven Delivery - Part 4, Stabilise through embedded testing <p>Before you read this please read prerequisite <a href="http://www.zankavtaskin.com/2020/08/lead-time-driven-delivery-focus-on.html">Focus on results, not methods</a> blog post as it briefly explains the basic scientific thinking that will be used here.</p>
<p>How do you know if a piece of process, software, hardware, concept or idea will behave in a correct way? Also, how do you know if this thing will meet the required quality, performance and reliability levels? Well, it is all about knowing how this thing will behave under certain conditions and more specifically it is about knowing when something will work and when it will fail. </p>
<p>To make this a bit more concrete let’s imagine that a customer with a lot of money went to two different software houses, one is called Henry’s Software and the other one called Adam’s Apps. Customer asked them to develop an identical software. To keep things simple, we will focus on a specific requirement. Here is what these two companies have written down for an identical requirement. </p>
<p>
<strong>Henry’s Software:</strong>
As a user when I open the mobile app for the first time, I would like to be able to quickly and easily connect to sign into my companies account.
</p>
<strong>Acceptance Criteria: </strong>
<ul>
<li>User enters information that he/she knows</li>
<li>User is directed to the relevant login screen</li>
<li>User puts in username and password</li>
<li>User successfully is directed to company account. </li>
</ul>
<p>
<strong>Adam’s Apps:</strong>
As a user when I open the mobile app for the first time, I would like to be able to quickly and easily sign into my companies account.
</p>
<strong>Acceptance Criteria: </strong>
<ul>
<li>User knows the name of the (1) company that he/she works for or their (2) corporate email address. </li>
<li>Story needs to deliver an experience that facilitates login with just one piece of information, there is no need for 2. </li>
<li>If email address is used, full valid email address needs to be provided before the company is looked up.</li>
<li>If company name is used at least 3 letters need to be entered before company is looked up, this is done to slow down the enumeration attack. </li>
<li>If email address or company name detects more than 1 company, then list of companies is given so that user can select which one they will be given a login screen for.</li>
<li>Once user clicks on the company user gets redirected to companies configured authentication provider for login.</li>
<li>If user cancels out of the login screen, then they will get redirected back to the company selection screen.</li>
<li>After 6 attempts to provide company information or email user will be asked to wait for 30 seconds, then 1 minute then 2 minutes, following [30 seconds * num of attempts], all the way to 24 attempts. </li>
<li>Company look up approach must be discussed with senior customer support team member(s) to ensure that it will result in least amount of customer support calls. Their feedback needs to be documented.</li>
<li>All web requests must not take more than 2 seconds. </li>
</ul>
<p>Remember, it is exactly the same requirement. Henry’s Software requirement documentation is vague, it does not provide specific information that can be used to verify and test what was delivered. While Adam’s Apps does capture user behaviour, expectation, delivery options, prerequisites and system performance. Adam’s Apps can establish specific test criteria for this feature and verify it when it is delivered. </p>
<p>Henry’s Software might say that Adam’s Apps documentation is heavy and too specific, their argument might be:</p>
<ol>
<li>The customer is always available, requirements will iteratively emerge or will be discussed (see below).</li>
<li>Conversation over documentation, team knows what was discussed so we don’t need to be explicit also remember that customer is always available, it is possible to reconfirm.</li>
<li>Team is trusted to make a right decision, there is no need to be so specific. </li>
</ol>
<p>Of course, we do want customers or business analysts (who represent the customer) to be always available, but they are not due to holidays, meetings and competing corporate priorities. Individuals might have discussed this requirement with customer, however, these individuals might leave, go on a holiday, get sick, have to do other work, which means story might be picked up by someone who has no context. This means this individual will not be able to fill in the assumptions and in turn make mistakes which will cause rework. If business analysts or whoever writes the story knows the criteria, they should write it and not assume assumptions as known. Finally, number 3, trust does not mean that team should not take the vague requirement and refine it to be testable, at the end of the day they will still need to test it.</p>
<p>The main argument of this section is not about requirements documentation, it is about testing. Regardless of the format of how we write requirements down, I hope we can agree that when you know under what conditions something will work and fail then it is possible to test the piece of process, feature, concept, idea or hardware under specific test criteria.</p>
<p>As software people we all want to deliver great software experiences on time to our users. However, so many projects overrun, things go wrong for millions of reasons. Normally something goes wrong early in the process and then it cascades issues downstream, these issues are normally systemic i.e. they are bugs in your delivery process. They normally emerge because process is: non existent, not followed, opaque, regressed, out of date or poorly designed.</p>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtuxGdIA31z25W7NWbTTx-q5ySalKCgYZwwlzo2YV46E661eOwqbHrkQW9UcVDSipCMb8lcb13uQEz6zEvRDiDU1jU-5bxF-qInMFGL-BG64UTHZO5GQBhscyG3auikCNGqFIyJtwF7MI/s1200/1200px-Stock_market_crash_%25282020%2529.svg.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="808" data-original-width="1200" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtuxGdIA31z25W7NWbTTx-q5ySalKCgYZwwlzo2YV46E661eOwqbHrkQW9UcVDSipCMb8lcb13uQEz6zEvRDiDU1jU-5bxF-qInMFGL-BG64UTHZO5GQBhscyG3auikCNGqFIyJtwF7MI/w625-h420/1200px-Stock_market_crash_%25282020%2529.svg.png" title="volatility : liability to change rapidly and unpredictably, especially for the worse." width="625" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Volatility : liability to change rapidly and unpredictably, especially for the worse.</td></tr></tbody></table>
<p>It can also be hard for managers to see where issue has actually stemmed from as people are looking at the whole thing and not the individual parts. One of the ways to increase predictability and transparency in your process is to break it down into component parts and then exposing each component part to a test criteria. Why would you want to do this? Remember the reason why we are doing any of this is because we are trying to reduce the lead time. Additionally, further the faulty feature travels through your development lifecycle more costly it is to fix it, it adds more lead time to the faulty feature, and it adds additional lead time to other features in the queue! This means we want to catch bugs as soon as possible and release work into the next stage only if it has passed all of the relevant test criteria. This way you will stabilise your delivery and reduce lead time across your entire development lifecycle.</p>
<p>You might be thinking, but we have documentation for all of this stuff. We know how things should work. We have definition of done, ready, test plans and so on. Yes, the problem is that normally this documentation is outside of your development lifecycle. It is a separate piece of document that you need to read and let’s face it, you probably don’t read it often enough. You probably refresh yourself once in a while just before the auditors knock on your door. The real challenge is to embed this process documentation into your development lifecycle so that system becomes self-checking, testing and auditing. To make what I am saying more concrete, here are few examples of what you can do to enable this:</p>
<ul>
<li><strong>Embedded checklist</strong> - If you are using some sort of Application Lifecycle Management software then consider embedding a version of your definition of done / ready into the Epic, Feature, Story or Task as a checklist or validation rules. When individual fills in the Epic in they have to confirm that they have done X, Y and Z, or they can’t complete the work until something is done. </li>
<li><strong>Public self-accountability</strong> - I don’t know about you, but when I have to email large group of people update on the project, I 100% want to get my facts right. When we publicly report something, we tend to be more transparent, accountable and self-governing. Normally, we don’t want to lose face. This means it can be a good idea to get teams to send out fortnightly project updates to senior stakeholders. There are many ways that this can be used.</li>
<li><strong>KPIs</strong> - If you know how you and your peers are being evaluated then you will change your behaviour around that evaluation. In this case you and your team should be evaluated against Lead Time. </li>
<li><strong>Automation</strong> - It should be no surprise that when processes are automated correctly than reliability and speed of these processes drastically improve. Conceptually what you want to do is have your entire "Development Lifecycle As Code", this means that if process does not need human creativity it should be standardised and automated away (where appropriate).</li>
</ul>
<p>
Testing in development lifecycle is not just about writing down testable requirements like we did for Henry’s Software and Adam’s Apps above. It is about embedding quality controls throughout the process, and these quality controls might look nothing like you would expect. You might not even think of them as quality controls. Do you think of automation as a quality control? Routine email being sent out by an individual? Your weekly stakeholder update meeting? What about your morning stand-up? These are all form of quality controls designed to catch faults and problems in your process. </p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-31151075863236654052020-08-26T14:54:00.009-07:002022-01-15T12:53:10.613-08:00Lead Time Driven Delivery - Part 3, Focus on results, not methods<p>Imagine you come up with a process change in your company, and you believe this process change is going to make things better. How do you verify that this process change is “Agile” compatible? Does change just need to sound and look good? What is Agile? If you really think about it, it is not immediately clear.</p><p>
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS62Mqm0_JFEWljSimE40KoRAH33jSin88f4avBeBWUhm8sVYcZEFD1b1KntkU_aQxO_Ih4erPb6n1mZqhHaVoJvqVGDEI-vkhl_2nh6KF_4Xqa1dRd6xv1TLhvmXrtzLE9U-e-pu07T4/s750/fb-stopwatch2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="394" data-original-width="750" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiS62Mqm0_JFEWljSimE40KoRAH33jSin88f4avBeBWUhm8sVYcZEFD1b1KntkU_aQxO_Ih4erPb6n1mZqhHaVoJvqVGDEI-vkhl_2nh6KF_4Xqa1dRd6xv1TLhvmXrtzLE9U-e-pu07T4/s640/fb-stopwatch2.png" width="640" /></a></div><p>Scientist have created a simple framework to test plausibility of what is being said and proposed. All claims must be testable. Let’s say I tell you that I can jump 3 meters high, test is simple, either I can, or I can’t. Once test is conducted you will quickly find out that it was a lie, I can’t jump that high. If you have a COVID-19 cure and you are running clinical trials, you will measure average recovery time of patients without your amazing drug (control) and compare it against the group that is taking the actual drug, and you will look for a very significant average improvement. Things are not that simple when it comes to business, sociology, psychology and many other fields that have people element in them. For example, in the 20th century Freud ideas were treated as scientific. Freud was able to come up with a theory about someone’s behaviour and then fit it in to the situation so that it seemed to makes sense. If his theory worked it was because it worked, if it did not work it was because something was wrong with patient or some other theory of his worked better. Either way he was right. Fortunately for us, <a href="https://www.youtube.com/watch?v=ztmvtKLuR7I" rel="nofollow">Karl Popper</a> was around to introduce falsificationism to bring some sense in to the world. In over simplified terms, he said that a theory must have a clear test criteria for when it works and it does not work. If it does not have a clear test criteria than it is pseudoscience. Pseudoscience leave things vague enough so that it sounds like they might be true, and it leaves things open to interpretation, just think of Astrology, Hypnosis, Polygraph, Psychoanalysis ... (the list is long, I strongly recommend you Google it). At this point you might be thinking, OK, I get it, how is any of this relevant to software delivery?</p>
<p>Now let's come back to that Agile Framework process change. How do you test for Agile Framework compatibility? I can take a look at the Agile manifesto tenets. However, some of the statements there are not testable, some of them are falsifiable and the rest is list of working practices. However, it is just that, it is a list of working practices i.e. methods of doing something and not the results, this makes it hard to test. I don’t know if this is true for all Agile Frameworks, however most of them seem to suffer from same problem that they focus on method (how something should be done) and not the result (when an action is performed it produces a consistent result):</p>
<p>“<i>The increasing adoption of agile practices has also been criticized as being a management fad that simply describes existing good practices under new jargon, promotes a one size fits all mindset towards development strategies, and wrongly emphasizes method over results</i>” - <a href="https://en.wikipedia.org/wiki/Agile_software_development#Criticism" rel="nofollow">Wiki</a></p>
<p>It seems that Scrum, XP and other Agile frameworks are collection of working practices that seemed to have achieve good result in the organisation that it was rolled out in at the time. This can be a good thing as companies can take process of the shelf and just get started. In reality I have not worked for a company that has rolled out pure form of Agile Framework, there are always variants. Personally, I have seen some really good results by rolling out Agile and Lean working practices (even with the variants). However, I have heard of companies that have never seen their Agile process change investment pay back dividends or worse it has grinded their operations to a halt. One might say that in that case issue is with leadership of the organisation, leaders are not bought into the Agile framework and this is the cause of all of the problems. I don’t know if that is true, at some point you have to take a step back and just wonder why so many Agile Framework deployments fail all over the place. When you take paracetamol, it works (for most people), pharmaceutical companies do not say that patients that are not getting better are wrong and that their drug works in all circumstances. Surely the problem is that the solution that is being offered is wrong or is wrong for that company? Industry experts seem to be prescribing the same solution to everyone and then when there are no results, they point the finger at the leadership and never at the panacea. So why are we so fixated with the ceremonies, shrines and the dogma of how we work and not focus more on what actually produces consistent results?</p>
<p>Even if things don’t go wrong during the Agile Framework roll out, once the roll out is complete, then what? You want to instil culture of continuous improvement, but you are out of book, there are no more agile tricks are left up your sleeve. How do you achieve further improvement? If you start to implement changes how do you know that they are compatible with your existing Agile framework? Are your internal Agile purists (years ago I was one of them) going to become dissatisfied that you are changing this pure roll out? How do you know that your changes will improve the process and not make it worse? Also what about the rest of the company, how are they going to get better and integrate with your department Agile process?</p>
<p>It seems that there are three problems with Agile Frameworks: </p>
<ul>
<li>They rarely get rolled out in pure form as organisation fail to fully embrace them </li>
<li>They don’t tend to consider the whole organisation, just a part of it</li>
<li>Eventually you run out of the book and you are on your own. Your investors will expect you to improve your operations further, further changes are hard to make as there is nothing to guide you</li>
</ul>
<p>Companies should pick an Agile Framework (after all you don’t need to reinvent the wheel) that suits their organisation and once they hit a wall (and you have genuinely rolled out as much of the framework as possible) they should refocus on to the results. This might feel scary, this is because as humans we derive comfort from being told what to do by industry experts, conformity and sometimes superstitions. This is where that scientific and critical thinking can help you cut through the noise and help you to make your own decisions. Decisions that will help your company and your unique circumstances. </p>
<p>When you say we are going to change process from A to B. How do you know of this change will enable faster delivery? Proposed process change should have some of the following characteristics: </p>
<ul>
<li>Removes number of handovers required to get something done</li>
<li>Shifts left work by enabling people upstream to get work done themselves earlier in the process </li>
<li>Reduces wait time to get something done </li>
<li>Reduces amount of disruption </li>
<li>Reduces the constraints</li>
<li>Improves individuals domain knowledge, moral or skill </li>
<li>Reduces unknowns, complexity or risk</li>
</ul>
<p>Ultimately if you are not sure you can try the change and test it quantitatively by seeing if cycle time and lead time has reduced / increased. The main thing is that you can verify or refute process change as ultimate test is reduced average lead time.</p>
<p>Please take note, when it comes to results over methods, it is important to establish organisational values that will create boundaries of how far you will take the results over methods approach. Last thing you want is behaviour in your organisation that does not align with your organisational moral compass as "anything goes" to get the results, this is where your company culture needs to step in. </p>
<p></p><p></p>Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-11421569679962498042020-03-07T05:08:00.010-08:002022-01-15T12:52:42.122-08:00Lead Time Driven Delivery - Part 2, Learning from data<p>
In <a href="http://www.zankavtaskin.com/2020/01/applied-software-delivery-system.html">part 1</a> of this series we have explored the basic idea of Lead Time Driven Delivery. The main idea is to minimise Wait Time, Disruption Time, Task Time and thus Lead Time for the work that is flowing through the organisation. In this blog post we are going to explore metrics and processes that will help you to:
</p>
<ul>
<li>Verify if process changes that you are implementing are minimising Lead Time</li>
<li>Identify how much Lead Time can be further removed</li>
</ul>
<p>Before we explore the core idea, let me introduce you to the Little's Law.</p>
<h2>Little's Law</h2>
<p>Imagine a busy, but a small web agency in Edinburgh building web products for it's clients. This web agency employees UX Designers, Web Developers, Testers and Cloud Developers. They work as one team. This web agency has made commitments to deliver number of features for a very important client, this work is committed to the queue, which means this work is "work in progress" (WIP). All work that is being done will need some time from all of the team members (UX, Web Devs, Testers, etc). As soon as the team takes the work of the queue the timer starts and the timer stops when team has stopped working on the task and the work is done. This is called Cycle Time. Finally there is Lead Time, Lead Time timer starts from the moment that the work is committed to the queue and timer stops when work is done. </p>
<p>Remember the Hot Feature A from last blog post? Well it has spent 1 week in the queue, then finally it was picked up by the UX, however Web Devs, Testers, etc were all busy working on other work. So it has slowly made its way from one person to another until it was completed 3 weeks later. So it has taken 1 month to complete overall, but it required only 12 hours worth of work. This is one poorly managed web agency!</p>
<p>Relationship in the above diagram can be described with <a rel="nofollow" href="https://www.process.st/littles-law/"> Little’s Law</a>:</p>
<p style="text-align:center;">Lead Time = WIP / Throughput</p>
<p>Web agency team on average completes 0.3 of a task per day. Team on average has 9 commitments in the backlog that they need to get through. That means (9 / 0.3) = 30 days lead time. To improve this delivery situation team has two options:</p>
<p>1. Reduce the amount of committed work in the queue. If team reduced committed queue size from 9 to arbitrary lower number such as 3, this would mean that lead time would go to (3 / 0.3) = 10 days. </p>
<p><strong>OR</strong></p>
<p>2. Team needs to improve the Throughput (Cycle time).</p>
<p>For more information around Little’s Law do check out this <a rel="nofollow" href="">awesome article</a>. This entire blog series focuses on improving Cycle Time and not reducing the WIP. </p>
<h2>Core idea</h2>
<a rel="nofollow" href="https://medium.com/slashdeploy/book-review-accelerate-92ebc00f4354">Accelerate book</a> suggests that one of the important metrics that should be tracked is Lead Time. This totally makes sense as this is what customer experience's and it impacts recovery time, experimentation speed, etc. This has very much inspired this entire blog post series. Accelerate book does recommend to track other additional metrics, if you are interested in knowing what they are then check out this <a rel="nofollow" href="https://medium.com/slashdeploy/book-review-accelerate-92ebc00f4354">summary / review of Accelerate book</a>.
<p>As a software delivery practitioner I find that Lead Time is a start, however it is very high level and it does not provide much detail for me to make the improvements or scheduling decisions.</p>
<p>Lead Time Driven Delivery is suppose to help by exposing Wait, Disruption and Task time. If you know how much Wait and Disruption there is in the system and where it is coming from then you can do something about it. By this point you might be wondering, how can I extract this information from my Application Lifecycle Management System (ALM)? Is it even possible to automatically get metrics for Wait Time, Disruption Time and Task Time variables? Answer is that you will need to use quantitative and qualitative techniques to extract data. </p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqQLjspK5eProsxaNTnem3CNjcGhXugEToDCpUdqCjQjvMTcY6Rxmammbr8hg14RAwu95oeQjW8inW9VV-Wyvm6LaU7XQ0giZEFU5a05BjoUo-shAZ57BrNPcE12txEkKRfXS6CPcqc2M/s1600/qvsq.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqQLjspK5eProsxaNTnem3CNjcGhXugEToDCpUdqCjQjvMTcY6Rxmammbr8hg14RAwu95oeQjW8inW9VV-Wyvm6LaU7XQ0giZEFU5a05BjoUo-shAZ57BrNPcE12txEkKRfXS6CPcqc2M/s640/qvsq.png" width="auto" height="auto" data-original-width="1600" data-original-height="1203" /></a>https://www.slideshare.net/Intellspot/qualitative-vs-quantitative-data-infographic</div>
<br>
<h2>Quantitative analysis</h2>
This is the easy part. If your team is storing dev data in some ALM system then you can get this data in many different ways. You just need to make sure:
<ul>
<li>Only actual development time against the work is logged. In this web agency they have terrible internet speed, they love meetings and build server takes forever to run tests. So developer has taken 2 hours to do the actual work, but between all of the waiting, meetings and random requests the whole day passes (8 hours). In this instance developer should log only 2 hours of actual dev time and not 8 hours. </li>
<li>All work that needs to be done is grouped in a logical way so that it is possible to identify wait states between tasks for the deliverable. </li>
</ul>
<p>Now you can create a two metrics Lead Time Resistance and Lead Time Spent Idle.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6H6O7X1i-uUxIi5SBGuHD04I1yKk8kezdbmMKMtlUDRoNuYrs5gcGJu_qT3pefM_osiFw4xXnUVAmcNBF4IRNhL3HtPXE_-_sVwFEzKiLl3sLWPzHLpFiS5IwT1g-dQBKCLZkLgunmuM/s1600/algo.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6H6O7X1i-uUxIi5SBGuHD04I1yKk8kezdbmMKMtlUDRoNuYrs5gcGJu_qT3pefM_osiFw4xXnUVAmcNBF4IRNhL3HtPXE_-_sVwFEzKiLl3sLWPzHLpFiS5IwT1g-dQBKCLZkLgunmuM/s640/algo.jpg" width="auto" height="auto" data-original-width="1600" data-original-height="517" /></a></div>
<br>
<h3>Lead Time Resistance</h3>
<p>Lead Time Resistance measures how difficult it is to get work done. In the above digram Sam (UX) might take only 2 hours to do the design work (blue), however for the rest the day he is disrupted (orange) and before he knows two days have gone by. Lead Time Resistance calculation takes total actual time for the work and divides it by the total elapsed time for the work.</p>
<p>Feature in the diagram has taken 12 hours actual time, but it took 5 days total elapsed time. Lead Time Resistance for this is 1-[12 (hours) / [5 (days) * 8 (hours per day)]] = 70%, 70% was spent on Disruption and Wait Time i.e. stuff getting in the way, creating resistance.</p>
<h3>Lead Time Spent Idle</h3>
<p>Lead Time Spent Idle measures how well the work was planned. Sam (UX) has completed his work in 2 days. Work waited for ~1.5 days before it was picked up by John (Dev). After John was done, it has waited for another ~3.5 days before Dan (Test) picked it up. Total idle time of no activity is divided by the total elapsed time. Feature in the diagram has taken 20 (5 days * 4 weeks) business days to complete. Out of 20 business days it has spent 8.5 business days (1.5 days + 3.5 days + 3.5 days) in idle state, this means 8.5/20 = 42.5%, 42.5% was spent in idle state.</p>
<p>
Now let's bring it all together. The feature in the diagram has encountered 70% of resistance and it has spent 42.5% of the time in the idle state. I don't know about you, but this is really useful information. Now that this is known, team can move on to the qualitative analysis and do some deeper analysis on what can they do to improve this situation.
</p>
<h2>Qualitative analysis</h2>
<p>This is the hard part, this is where team needs to actually continuously question existing working practices and get creative about improvements. Quantitative analysis will expose a lot of variables, just take a look at this:</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhE90eyhyphenhyphenOlCF3SNRd7Uj6m1eFJzRXabZKzyDzqKimue93V2-ztaP1pr47M_Zn7Tu52x0FCSE_6c5gXxr-oy0UEQTPzS-1CJmD5ir1EiIK990XP_r83Xk6h4WOem4TG-Zu_Cxwo_9Y5dgg/s1600/littlelaw.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhE90eyhyphenhyphenOlCF3SNRd7Uj6m1eFJzRXabZKzyDzqKimue93V2-ztaP1pr47M_Zn7Tu52x0FCSE_6c5gXxr-oy0UEQTPzS-1CJmD5ir1EiIK990XP_r83Xk6h4WOem4TG-Zu_Cxwo_9Y5dgg/s640/littlelaw.jpg" width="auto" height="auto" data-original-width="1126" data-original-height="1600" /></a></div>
<br>
<p>However, numbers alone will not tell you if Lead Time Resistance is large due to waiting around or chronic disruption culture. Additionally, it will not tell you how much time is lost due to poor work distribution, poor designs, lack of standards / components, staff turnover, etc. This is where teams should keep a daily log of all of the Wait Time, Disruption Time and Task Time. They should then use this information during retrospectives to review their current workflow setup and figure out how the future workflow should look like to improve the Lead Time.</p>
<h3>Side Note</h3>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqeisQbR9W0-CVB9-AOBSBi-uC6_XJzygxUkfWfS_LXcxzUbOS4Zk_2i2AFTB9deEfE644j7I_FeeirAFLnVtFXO1ctMy4uUogID7HDhe-k30_XeVYKx4O6p8li4-n1a8zBAXw8Ck0FG8/s1600/pablo_posit_on_postit_1024x1024.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqeisQbR9W0-CVB9-AOBSBi-uC6_XJzygxUkfWfS_LXcxzUbOS4Zk_2i2AFTB9deEfE644j7I_FeeirAFLnVtFXO1ctMy4uUogID7HDhe-k30_XeVYKx4O6p8li4-n1a8zBAXw8Ck0FG8/s640/pablo_posit_on_postit_1024x1024.png" width="auto" height="auto" data-original-width="1024" data-original-height="512" /></a></div>
<p>In Lean there is strong focus on waste elimination, just Google 8 wastes or check out my older <a href="https://www.sitepoint.com/we-simulated-waterfall-kanban-scrum-which-works-best/" rel="nofollow">blog post</a>. Problem is that you need to look up what waste means in order to understand it. Then you need to translate it to knowledge work so that it is relevant. Personally I don’t think it is that relatable to knowledge work and once translated it does not stay in your mind for that long. Software pactioners have introduced "Waste Snake” while I like the concept, problem is still the same. "Waste" is a vague name. From my personal experience, I have seen teams use it for a while showing mainly disruptions. I don't know why but they have not focused on other more hidden wastes such as Wait and Task Time. It might sound less cool, but instead of "Waste Snake" create a "Lead Time Wall" and just stick on to it anything that impacts Wait, Disruption and Task Time with the amount of time lost.</p>
<h2>Conclusion</h2>
<p>Your team should automate the following metrics:</p>
<ul>
<li>Lead Time</li>
<li>Cycle Time</li>
<li>Work in progress (WIP)</li>
<li>Lead Time Resistance</li>
<li>Lead Time Spent Idle</li>
</ul>
<p>These automatic metrics are useful as they will expose Wait Time and they will tell you if you are going in the right direction with your process changes. To actually figure out what needs to change to improve Lead Time, your team will need to conduct constant qualitative analysis where you manually review Disruption and Task Time.</p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-70061300564484299472020-01-28T14:55:00.008-08:002022-01-15T12:52:30.361-08:00Lead Time Driven Delivery - Part 1, Learning to see <p style="text-align:center;"> <strong>TL;DR</strong> Your organisation needs to minimise <strong>Wait, Disruption and Task Time</strong> so that work gets delivered quickly all the way to the customer. This means you need to minimise <strong>Lead Time</strong>.</p>
<p>I am very luck as I have been an Agile and Lean practitioner for a while. I have spent time reading about different delivery methodologies such as XP, Scrum, Kanban, etc and I had an opportunity to work in organisations where they have used these different methodologies. One thing that constantly stood out to me was how they all prescribe best practice with a very shallow reasoning behind these prescriptions. It looks like they are based on experience and environment where they were created. This complicates things as approaches end up being open to interpretation on what is Agile and what is someone's subjective opinion on the matter. For example, if someone has came up to you and said, I want to change our existing process from A to B. How would you test/verify that this new process is more "Agile"? Would you refer to Agile Manifesto? Use your experience/training? As far as I know there is no testable/verifiable way to measure Agile, this creates a communication and expectation problem with new and seasoned practitioners. If you promote someone to be a team lead or if you onboard a new member of staff you need to explain to them why you are doing something in a certain way. Saying to them "please read this Agile book and follow it" is not going to work. If you tell your new members of staff just do "what we do”, well that’s flawed because they don’t understand the reasoning behind the intents. Also, why should they follow it? This means that the moment you need to change how your company works you end up with organisation that starts to make inconsistent decisions between different teams and departments as no one really understands what behaviour and KPIs they are trying to minimise or maximise. </p>
<p>
In this blog post (and eventually series) I am going to attempt to breakdown the core reasoning behind the Agile practice so that it is more verifiable. Hopefully this will mean that core Agile ideas can be explained quicker to people around you and that you and your team can confidently mature your own delivery practice. Let’s get started.
</p>
<h2>1. Anatomy of you sitting down and trying to do some work</h2>
<br>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAWqxZb5AgUd6GSPq_mcQzYgCwwh748XDmgZseWF9ynSGC6AVEj199cg0pkt44l2iz7osyI5O8hWMANpawvOx5Uz_TeyODRy9GiYOHu8QNNd0ByPdkZ3cRpyRpXxOFktzKDpedgYDhgcA/s1600/SystemThinking.jpg" imageanchor="1"><img border="0" data-original-height="119" data-original-width="478" height="auto" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAWqxZb5AgUd6GSPq_mcQzYgCwwh748XDmgZseWF9ynSGC6AVEj199cg0pkt44l2iz7osyI5O8hWMANpawvOx5Uz_TeyODRy9GiYOHu8QNNd0ByPdkZ3cRpyRpXxOFktzKDpedgYDhgcA/s640/SystemThinking.jpg" width="auto" /></a></div>
<br>
<p>Three factors that make up your work:</p>
<ul>
<li><strong>Wait Time</strong> - This is when you are waiting around for some knowledge that you don’t have, decisions that you can’t make and finally you are waiting around for someone else to complete some work before you can start yours. </li>
<li><strong>Disruption Time</strong> - This is when you have to expedite some work, rework some work, corporate interruptions and mental health impact.</li>
<li><strong>Task Time</strong> - Finally, this is the actual work that you are doing, pure sitting down and getting things done. </li>
</ul>
<p>
Imagine you are working on your own on your own start-up. You will have very little wait and disruption time. You are on our own, you can make all of the decisions. Also if you are lucky enough to work in a quiet environment you should experience very little or no disruptions. You get things done fast, your users are impressed with your company, new features just come out all the time. However, this changes the moment you hire your first employee in your start-up. The moment you do that, you create an organisation, that means you have created a system. In the system work no longer gets done by a single individual, it gets done by many individuals. You as the founder are unlikely to feel much impact by hiring this new person (apart from knowledge transfer burden), but if you are not careful your new employee will have to wait for your decisions, knowledge and task allocation. Their Wait Time will grow as they wait for you and they will probably be disrupted by you. You will wonder why they are not as a productive as you, it might be because they have not got enough autonomy to make decisions (maybe they don’t know your values so they don’t know what decisions to make on your behalf) also they might not be getting enough clarity about the desired outcomes. Most people are not founders, they are the employees and they struggle to do their best as they just don’t understand the reasoning framework and don’t get enough autonomy. </p>
<h2>2. Anatomy of your Waterfall company trying to deliver some value to the customer </h2>
<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHgUDd37GHIA5tsJImiN_HRUsbUtuMlBbeH_8dedPUI4sa8Oaj3W8SwztaqIf48Ndz-lVDzyySxXGnw2gXgQ47JZfoFpEuHnurPG0OjptXBJVx1KN7H2gJINLisrVsaUo28IsshfbNjAk/s1600/waterfallteam.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHgUDd37GHIA5tsJImiN_HRUsbUtuMlBbeH_8dedPUI4sa8Oaj3W8SwztaqIf48Ndz-lVDzyySxXGnw2gXgQ47JZfoFpEuHnurPG0OjptXBJVx1KN7H2gJINLisrVsaUo28IsshfbNjAk/s640/waterfallteam.jpg" width="auto" height="auto" data-original-width="1600" data-original-height="493" /></a></div>
<br>
<p>Image a company that does not follow any Agile process and instead they have departments of people per discipline. So Web devs in one department, API Devs are in another department, you get the point. Each department will have their own backlog, which means everyone has their own Lead Time, on top of that all individuals will experience disruptions (team meetings, urgent requests you know the drill) and there will be many handovers from one department to another. Work will also end up traveling backwards due to misunderstandings. So if a customer has requested a “Hot Feature A” they will have to wait for a long time for this work to travel through this type of organisation (system). Actual Task Time for "Hot Feature A” might be 12 hours of work in total, however given all of the Wait Time (handovers and lead times) and disruptions it might take up to 1 month before it gets shipped. So there is a big difference between 1 month Lead Time and 12 hours Task Time. However your customer will not care about the 12 hours of Task Time, they will just care that you took 1 month Lead Time. Overall in this type of organisation Lead Time for most work will be very high, fewer projects will be shipped, projects will very rarely go out on time and individuals will feel frustrated as there will be a lot of firefighting. </p>
<h2>3. Anatomy of your Agile company trying to deliver some value to the customer</h2>
<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4gCFjJM2UtIF-vFO5klx-W1Q6LoQS5NijsaIEwviTl6HHQ8KhiiHRvXE5k6bvULXtNvdQwgy5cv8zRJzUt_LgpY4p-xDYpwWGF2sU5ywWmJH3k8WltBY9hIibCBzy0mvE0BQm9aOywFE/s1600/agileteam.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4gCFjJM2UtIF-vFO5klx-W1Q6LoQS5NijsaIEwviTl6HHQ8KhiiHRvXE5k6bvULXtNvdQwgy5cv8zRJzUt_LgpY4p-xDYpwWGF2sU5ywWmJH3k8WltBY9hIibCBzy0mvE0BQm9aOywFE/s640/agileteam.jpg" width="auto" height="auto" data-original-width="1600" data-original-height="773" /></a></div>
<br>
<p>Now imagine another company that understands importance of Lead Time and works to remove as much Wait, Disruption and Task Time (more on Task Time later) from overall delivery process. They have decided to sit people together for a limited amount of time to deliver certain features and projects. They have done this as they want to remove handovers, the amount of project management is required, competing agendas, waiting for decisions, knowledge and organisational dependencies. They work as a team on one story at time and their main job is to push that one story through the system as fast as possible. Now, that story that took 1 month to deliver, in this new system will take 12 hours or even less. This is because you have removed all of the waiting around, disruptions (team lead and product owners act as defenders) and because this team is sitting together they can actually expose the unknowns faster, tame complexity, share their experience and share the burden of the work so they can actually deliver the work faster.</p>
<h2>4. Anatomy of Task Time </h2>
<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAFw7NZPuSpUK6S162LUCW8L88AD4Yl3JH059ak6wQXpn8ZoAkXW0Nk2jr6pZEkrXYdPE2fDt_-SWJRy12wV_9104HxF_rJwP9naZRv7MmUmwiCAppBraefknRL6mHBBOnRWo46P43Pqg/s1600/tasktime.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAFw7NZPuSpUK6S162LUCW8L88AD4Yl3JH059ak6wQXpn8ZoAkXW0Nk2jr6pZEkrXYdPE2fDt_-SWJRy12wV_9104HxF_rJwP9naZRv7MmUmwiCAppBraefknRL6mHBBOnRWo46P43Pqg/s640/tasktime.jpg" width="auto" height="auto" data-original-width="1200" data-original-height="1000" /></a></div>
<br>
<p>I have left this till last for a reason. Management team needs to improve the overall system before they look at improving Task Time. Why? It is much healthier to focus on fixing the overall organisation before looking at how they can improve individual's performance. In companies where Lead Time is high, good talent might become disengaged. As you fix the systemic problems, you might find that people who were not performing that well start to really surprise you and that Task Time reduces naturally. </p>
<p>The actual Task Time is made up from eight factors which are dynamic:</p>
<ul>
<li><strong>Volume of work</strong> - This is just you sitting and typing, copy and pasting.</li>
<li><strong>Unknowns</strong> - This is you identifying stuff that you did not consider when you were estimating the work. </li>
<li><strong>Complexity</strong> - This is you figuring out an algorithm to solve a problem, the main thinking part. </li>
<li><strong>Risk</strong> - This is how much testing you have to do given the risk level that is acceptable for the task at hand.</li>
<li><strong>Skill</strong> - This is you improving your hard/soft transferable skills (programming, math, architecture, algorithm design, management, etc) or using your existing skills to get work done quicker. </li>
<li><strong>Domain</strong> - This is you gaining new domain knowledge (HR, Logistics, Financial Trading, etc) or using your existing domain knowledge to get work done faster. </li>
<li><strong>Attitude</strong> - This is how you perceive your work environment and tasks.</li>
<li><strong>Aptitude</strong> - This is you having developed or have predisposed skills towards the work that you are doing. </li>
</ul>
<p>I know this is obvious but I would like to stress one point. Most people will take different amount of time to get a task done. Why? They are different people, with different mindsets, skills, domain knowledge, aptitude and attitude. All of these things impact overall Task Time. </p>
<p> It will not surprise anyone that experienced teams (high skill and domain knowledge), are more likely to identify unknowns, reduce volume of work through some automation, tame complexity and as a result deliver high quality work quickly. If your organisation wants to improve Task Time then it needs to ensure that people stick around.</p>
<h2>What does it all mean?</h2>
<p>In priority order, everyone in your organisation should be working hard to minimise Wait, Disruption and Task Time and thus minimise Lead Time. Organisations will never achieve perfect Lead Time, however they need to constantly work towards it. To me this is what DevOps, Slack, XP, Scrum, Kanban, Lean, etc is all about.</p>
<p>If you take anything away from this blog post, then it would simply be this, start to measure Lead Time for work that is traveling through your organisation and find ways to minimise it.</p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-2816280796214327872019-07-11T13:55:00.000-07:002019-09-29T06:59:51.792-07:00Applied Software Delivery : Optimal Backlog<p>
Backlog management should be simple and efficient, what are the key things you can do to make it so?
</p>
<br/>
<h3>Give teams their own backlogs</h3>
<p>Each cross functional team of around (-+3)+7 members must have their own backlog. </p>
<p>This is one of the biggest productivity improvements you can make. If teams have to work in a global backlog then team members have to scan the whole backlog, sort stories against each other, discuss and understand stories, allocate bugs, etc. This is highly inefficient and puts backlog management on the exponential waste curve, when team has its own backlog it puts them on to the linear waste curve. </p>
<br/>
<h3>Keep it short</h3>
<p>Large backlogs are wasteful because refined stories will get dropped as business priorities change and they create lots of unnecessary maintenance and conversations which intern creates confusion and misunderstandings. </p>
<p>You can prevent backlog waste by ensuring that each cross-functional team has their own backlog and that they have only 1-2 sprints or just-in-time worth of work refined overall.</p>
<br/>
<h3>Prepare for the story refinement meeting</h3>
<p>You can reduce further time waste by creating draft stories with clear INVEST acceptance criteria before the team refinement meeting. This will give story refinement session overall focus and a strong starting point for a discussion. I don’t agree that all stories need to be created together with the whole team from scratch. Story refinement meetings should be used to get everyone to ask questions, think, create shared understanding and figure out “how” they are going deliver “what” is specified in the user story.</p>
<br/>
<p><span style="font-style:italic;font-size:120%;">“Organizing is only necessary when you have too many things.
Think about it: when we organize a collection of books, it’s because when they’re not organized, we can’t find the books we want. But if we had, say, five books, we wouldn’t need to organize.”</span> <a href="http://mnmlist.com/minimalism-is-the-end-of-organizing" rel="nofollow">By Leo Babauta</a> </p>Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-46609902014276771462018-09-02T15:09:00.000-07:002018-09-02T15:13:16.313-07:00Applied Software Delivery : Full Stack Developer vs Partial Stack Developer<p>Agile team consists of cross-functional team members, some of them work on the backend, frontend, infrastructure, persistence, etc. Typically developers specialise in one of these areas. However what happens if you identify a constraint in your team? What if a developer leaves? This means you have to hire or borrow a developer from another team. What is the alternative? </p>
<h3>Full Stack</h3>
<p>
This developer is a Swiss army knife developer. Full stack developers have large amount of shallow (and maybe in depth) knowledge and can take on large amount of technologies, such as .NET, Angular, MSSQL, MongoDB, Azure Hosting, Selenium, etc. Finding these people is near impossible, why is explored <a href=“http://andyshora.com/full-stack-developers.html” rel="nofollow">here by Andy Shora</a>.</p>
<h3>Partial Stack</h3>
<p>Much more realistic alternative would to be to move away from developers with single speciality towards hybrid developers. These developers would not know the entire stack but they would have a primary (core) and secondary skill. How does this compare?</p>
<h4>Team with single skill</h4>
<style>
table.tbl tr th, table.tbl tr td {
border: 1px solid #cccccc;
}
</style>
<table class="tbl">
<tr>
<th>Name</th>
<th>Skill</th>
</tr>
<tr>
<td>John Austin</td>
<td>Angular</td>
</tr>
<tr>
<td>Vince Perk</td>
<td>BDD</td>
</tr>
<tr>
<td>Sarah Wood</td>
<td>.NET Developer</td>
</tr>
<tr>
<td>Ed Skim</td>
<td>Test Analyst</td>
</tr>
<tr>
<td>Martin Lee</td>
<td>Angular</td>
</tr>
<tr>
<td>Jason Dmit</td>
<td>.NET Developer</td>
</tr>
<tr>
</table>
<p>Single skill model creates fragile teams. If single skill is not available then developers need to be borrowed from other teams or new additional people need to be hired, chances are that inventory will start to pile up. This slows down overall delivery.</p>
<h4>Team with hybrid skills</h4>
<table class="tbl">
<tr>
<th>Name</th>
<th>Primary Skill (Core)</th>
<th>Secondary Skill</th>
</tr>
<tr>
<td>John Austin</td>
<td>Angular</td>
<td>.NET Developer</td>
</tr>
<tr>
<td>Vince Perk</td>
<td>BDD</td>
<td>.NET Developer</td>
</tr>
<tr>
<td>Sarah Wood</td>
<td>.NET Developer</td>
<td>Angular</td>
</tr>
<tr>
<td>Ed Skim</td>
<td>Test Analyst</td>
<td>BDD</td>
</tr>
<tr>
<td>Martin Lee</td>
<td>Angular</td>
<td>Test Analyst</td>
</tr>
<tr>
<td>Jason Dmit</td>
<td>.NET Developer</td>
<td>BDD</td>
</tr>
<tr>
</table>
<p>Teams with hybrid developers are less fragile, they can elevate constraints, resolve issues within a team, don’t have to wait for anyone, which means they are more empowered and productive.</p>
<p>To create teams with hybrid developers your company will need to cross-skill existing staff and start hiring people with relevant skills. All of this will take time and patience. However, I do believe that this will not just benefit the organisations, but also the developers.</p>Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-90330344381251006672018-08-25T13:29:00.000-07:002018-08-25T13:29:35.663-07:00Applied Software Delivery : Hidden Impact Of Team Leaders <p>Upon reading software management books, one of the things that I don’t recall reading much about is the impact of team leaders on the overall team's performance. Most software management books talk about lean production, theory of constraints, etc. These concepts are important, however these books do skip the very important factor in software engineering, people. Good team leaders can make a software team fly or crawl. </p>
<p>No matter what you call your team leaders, managers, scrum masters, supervisors, etc. These people have huge impact. Good team leaders resolve issues, bring people together and ironically enable the team to be more self organising. Team leaders are the ceiling of your teams performance.</p>
<p>When it comes to software development things are not easy, especially at scale. Your typical software team will be facing issues with the following:</p>
<ul>
<li>Infrastructure
<ul>
<li>Build servers </li>
<li>Release pipelines </li>
<li>Laptops </li>
<li>Environments </li>
<li>Access </li>
</ul>
</li>
<li>Timely, clear and prioritised:
<ul>
<li>Functional Requirements </li>
<li>Non functional requirements </li>
<li>Business priorities </li>
</ul>
</li>
<li>Bugs </li>
<li>Customer requests / feedback </li>
<li>Ambiguous or bad decisions </li>
<li>People
<ul>
<li>Absence </li>
<li>Behaviour </li>
<li>Conflict </li>
<li>Lack of staff </li>
</ul>
</li>
<li>… the list goes on</li>
</ul>
<p>
At scale there are hundreds of things that can go wrong during a sprint. Good team leaders work with their teams to implement permanent solutions to these problems. They assign work to the right people in the team, they provide right level of support and coach for optimal performance. This is not easy stuff, especially when you have to do it everyday. It is unlikely that team leaders in your organisation will be able to do this out of the box, which means you will need to listen, train, mentor, coach, and generally invest in them.</p>
<p>
Managers of all levels often overlook this and focus on the wrong things. Personally, I have fixated more on the lean production concepts such as single piece flow, theory of constraints then team leaders. Why? Well, process is easier to implement and change. People are hard. They are not code, they are not servers, they can’t be changed with few lines of code and they can’t be reconfigured within few seconds. It takes time and patience. In software engineering this is the hardest and the most important thing you can do, invest in your people, especially in your team leaders, before you fixate on agile concepts and technology. </p>Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-9216149991699216712018-06-21T14:21:00.000-07:002018-06-21T14:24:02.544-07:00Azure Active Directory (AD) Inviting Members and Guests (B2B)<p>
Recently Azure has switched off the old portal and along with it the old Azure AD client. In the old portal there was a way to invite guests and members. In the new portal you can currently only invite guests, if you would like to invite members you need to do this through PowerShell. In this blog post we are going to take a look at how B2B actually works and how you can invite members.
</p>
<p>
What is a difference between a member and a guest user type? User type is used to enforce different security polices, <a href="https://docs.microsoft.com/en-us/azure/active-directory/b2b/user-properties">Microsoft has written some great documentation about this</a>. I found this image extremely helpful in their documentation:
</p>
<h2>How B2B invite works</h2>
<img border="0" data-original-height="220" data-original-width="403" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdHcEGp_YkoM3pK7gVHY3m15CRxrGOjRDr3x3Dq5Zv0yQ1THiZzXw5glnXm6xlTcFTLJDG-vDQGELp5P8IHkUebmTNVyDsp3BvbXhXnRsWamAH5LPyeC7Q-oXC_ugiH3KQv8uS-h_8T8E/s1600/InviteFw.jpg" alt="How B2B invite works" />
<p>
Let's unpack what is happening in the image above. You can invite users from external Azure ADs and Microsoft Accounts. You can't invite users from Google Identity, etc. The whole B2B collaboration feature works in Microsoft ecosystem only. This means if your company decides to invite 3rd party company users that does not have Azure AD then these users will end up creating Microsoft Accounts to login in to your service.
</p>
<h2>How to invite members from external Azure AD</h2>
<p>
Let's say that Elliot Alderson works for ECorp, his original account is residing in ecrophq.onmicrosoft.com Azure AD. Elliot Alderson works for Ecorp Fraud department. ECorp fraud department has decided to get their own Azure AD as they want to enforce their own security rules on their subset of users. Also, they do not want to call ECorp HQ IT department every time they want to setup a brand-new application for authentication (client id, audience, etc).
</p>
<p>
In this case ECorp Fraud admin will need to run the following script. To keep things simple when this admin calls Connect-AzureAD he will just use adprincipal@ecorpfraud.onmicrosoft.com account to authenticate and invite Elliot.
</p>
<pre class="brush: csharp">
Connect-AzureAD
New-AzureADMSInvitation -InvitedUserEmailAddress "elliot.alderson@ecorphq.onmicrosoft.com" -InviteRedirectUrl https://www.ecorp.com -SendInvitationMessage $true -InvitedUserType Member
</pre>
<p>
Once Elliot receives the email and accepts invitation this is how this setup will look like:
</p>
<img border="0" data-original-height="219" data-original-width="383" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrko6z_FEtujq4aeZf0i0jnuFe1Nd8Jt8Lg5mMxKiIaSgOreMirY7_O-Mej6v2MGC0dqD-OGBKxHxDVc2YZpb9ihyphenhyphen8uRtQ8hVQyijRtZP4HEeG1SbObL_OeeIOcgCJSIEuGOuM2WvsXTc/s1600/InviteEx.jpg" alt="How Azure AD references users from external Azure AD" />
<p>
I do not know about the Azure AD internals, so I am going to speculate now. It seems that when you add a user from external Azure AD in to your Azure AD it creates this user record as an extension. When Elliot gets the invite or is given the consent URL and approves permissions, referenced link gets created so Azure AD knows where to redirect user to authenticate.
</p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-69394690702565481842018-06-05T05:20:00.005-07:002018-06-05T05:20:55.183-07:00Azure AD - Relationships between Azure EA Accounts and Subscriptions<p>
When it comes to relationship between Azure Portal, Azure AD and Azure EA Portal it can get very confusing and frustrating. It can be hard to understand what you have to do to enable authentication in to the Azure Portal and why you have so many Azure ADs flying around. This brief blog post will try and explain relationship between Azure EA Portal account, Azure AD that gets created automatically and Azure subscriptions that can be accessed in the Azure Portal.
</p>
<p>
When you login to the Azure Enterprise Agreement Portal (Azure EA Portal) you can create "accounts" and under accounts you can create "subscriptions". Subscriptions is where you place your services (this is done in Azure Portal). Accounts is where you place your subscriptions (this is done in Azure EA Portal). When you create "account" Azure AD is automatically created for that account, and all subscriptions under that account link to that Azure AD. Here is the diagram that shows this relationship:
</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiChete_Z0mbW-tXFKjHL1WyCnrJwqDBsbX23XQ47R_lpfFKs2YuoQidD5kllx9tU4iMn0yXupO1OrmZGcmp4tWsCyzp7_wWJCFMEABYHPS2ywfIrHNSp6knJ4tRD0JPglzF7-DT6DUCwI/s1600/AzureADSubAcc.png" alt="Azure AD, Subscription and Account Relationship Diagram" width="640" height="274">
<p>
Let's say you have created ECorpFraud account in the Azure EA Portal and it belongs to ECorp Fraud department. That account will automatically get ECorpFraud Azure AD. This AD will be used to authenticate your users in to Azure Portal and let them see relevant Azure Subscriptions.
</p>
<p>
Imagine that Alice works in ECorp Fraud department, someone will need to add her to the ECorpFraud AD, once she is in there you will be able to give her permissions to see resources in Subscription A or B. Azure Portal itself uses Azure AD to enable authentication.
</p>
<p>Now, if I create ECorp Finance account, same thing will happen. Azure AD for ECorp Finance will be created. I will then be able to add relevant users to that AD and then give them access in to the relevant subscriptions, in this case Alice and John.</p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-83595589503228352632018-05-26T14:44:00.006-07:002020-08-27T05:21:54.361-07:00ASP.NET MVC Multitenancy - Introduction & Architecture Overview<p>
Over the past 5 years I had an opportunity to architect and develop multi tenanted applications. However, when it came to ASP.NET C# implementation I have struggled to find examples that solve common problems. In this blog series I will try to provide these examples:
</p>
<ul>
<li>ASP.NET MVC Multitenancy - Introduction & Architecture Overview</li>
<li><a href="http://www.zankavtaskin.com/2018/05/aspnet-mvc-multitenancy-part-0.html">ASP.NET MVC Multitenancy, Part 0 - Tenant Name Subdomain VS URL Path</a></li>
<li><a href="http://www.zankavtaskin.com/2017/08/aspnet-mvc-multitenant-routing-with-owin.html">ASP.NET MVC Multitenancy, Part 1 - Tenant Name Routing with OWIN</a></li>
<li><a href="http://www.zankavtaskin.com/2017/11/aspnet-mvc-multitenancy-part-2-openid.html">ASP.NET MVC Multitenancy, Part 2 - OpenID Connect Authentication per Tenant</a></li>
<li><a href="http://www.zankavtaskin.com/2017/12/aspnet-mvc-multitenancy-nhinbernate-shared-database-tenant-data-filtering.html">ASP.NET MVC Multitenancy, Part 3 - NHibernate Tenant Data Filtering and Shared Database</a></li>
<li><a href="http://www.zankavtaskin.com/2018/05/aspnet-mvc-multitenancy-part-4.html">ASP.NET MVC Multitenancy, Part 4 - Authorisation with Federated Identity</a></li>
<li><a href="http://www.zankavtaskin.com/2016/12/basic-saas-conversion-guide-migrate.html">Basic SaaS conversion guide, migrate your application to the Azure Cloud</a></li>
</ul>
<p>
This series focuses on a simple multitenancy architecture (more accessible to wider audience). Here is a high-level overview:
</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgutmOEDRVm0mFKa2tUyPvnWE5wZMc0oQUSdp5ZHaSNu8x4MneOr-rRw55-8IGykriXVCLTjw2zIZFRJy6ve5IzCQLBObZdMryMHiVZQm8ccJ2Zqd_G2P8A_ky-GYgII2dUz0YiA3rosb0/s1600/Architecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="662" data-original-width="1600" height="234" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgutmOEDRVm0mFKa2tUyPvnWE5wZMc0oQUSdp5ZHaSNu8x4MneOr-rRw55-8IGykriXVCLTjw2zIZFRJy6ve5IzCQLBObZdMryMHiVZQm8ccJ2Zqd_G2P8A_ky-GYgII2dUz0YiA3rosb0/s640/Architecture.png" width="640" /></a></div>
<p>
If you would like to get an architectural overview of multitenancy in general and see how you can deploy your application on to Azure then check out my "Developing multi-tenant SaaS business applications on Azure" <a href="https://www.wrocsharp.com/">WROC# conference</a> talk. This talk is very much architectural extension of this blog post series. I hope you will find it useful:</p>
<div style="width:100%;text-align:center;">
<iframe width="560" height="315" src="https://www.youtube.com/embed/zhQW74o_RcI" frameborder="0" style="border:1px solid black;" allow="accelerometer; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div>
<br/>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/" rel="nofollow">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Browse "Multitenancy-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br/>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-2871028286988968172018-05-20T14:22:00.001-07:002019-09-29T06:38:57.603-07:00ASP.NET MVC Multitenancy, Part 4 - Authorisation and Claims<p> Once users have authenticated in to your application you will need to enable authorisation. If you are building an application that is using federated identity pattern chances are that you are going to be working with claims. ASP.NET comes with Role Based Access Control (RBAC). In this blog post we are going to explore how you can use this with OpenID Connect to enable authorisation.</p>
<h2>Identity Provider Token </h2>
<p>
With OpenID Connect upon authentication users get redirected back to the application with JSON Web Token, this token contains claims about the authenticated user.
</p>
<h3>JSON Web Token (JWT)</h3>
<p>JSON web token looks like this:</p>
<pre class="brush: csharp">
eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6ImlCakwxUmNxemhpeTRmcHhJeGRacW9oTTJZayIsImtpZCI6ImlCakwxUmNxemhpeTRmcHhJeGRacW9oTTJZayJ9.eyJhdWQiOiJkYTdhNzQ1Yi05NzUyLTRjNGMtYjI0Zi05YWEyZmVjNGM3N2QiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC83NWIwMmUwZC05MGQxLTQzZTUtYjVkYi0yMGVhYWRkYmZhYzYvIiwiaWF0IjoxNTI2ODE5MTM5LCJuYmYiOjE1MjY4MTkxMzksImV4cCI6MTUyNjgyMzAzOSwiYWlvIjoiQVNRQTIvOEhBQUFBREpYd0VWRnJ2WndBcS9WMlBHcjYzeVQybVk5UVYyS0F1MHZqeExGVXp5ST0iLCJhbXIiOlsiSHdkIl0sImZhbWlseV9uYW1lIjoiS2F2dGFza2luIiwiZ2l2ZW5fbmFtZSI6IlphbiIsImlwYWekciI6IjE4NS42OS4xNDQuMTA0IiwibmFtZSI6IlphbiBLYXZ0YXNraW4iLCJub25jZSI6Il9I609zaENvIiwib2lkIjoiODY4OGZlNWYtYjAxZS00MmVkLTk3MWYtN2E0NDkxZWZjMGY1Iiwib25wcmVtX6NpZCI6IlMtMS01LTIxLTE4NzEzNjg5OTctMjU3NzcyMDU0NS02ODMwNjc4MzMtMTA2OTIiLCJzdWIiOiJ4d3JydG5DY2xfS2lrbngxYlJTTzRPem9aTVEwV0V4bmozRmtqXzY2TEVRIiwidGlkIjoiNzViMDJlMGQtOTBkMS00M2U1LWI1ZGItMjBlYWFkZGJmYWM2IiwidW5pcXVlX25hbWU2OiJaYW4uS2F2dGFza2luQG1oci5jby51ayIsInVwbiI6Ilphbi5LYXZ0YXNraW5AbWhyLmNvLnfrIiwidXRpIjoiRnVqeFZFMFAya3lyUEhDTkFJRVdBQSIsInZlciI6IjEuMCJ9.PYi8_x_NJjhPYflU156dy-5XkmTJc_RAu_Gq0C06QHWuoDbBcMAh1TdkulZ7tVq7lF24EY19W965riA0pruz74GWo_9Ny8Jk1MkJi3oBaZQSMrTz_7rc2Km30UJrKKulmM9em5BfT43fMndCDS4fSVMrp8w0ijNgdCzSvTrK9xid0BsJw8hWiMxDdPPYTI_cEzfx0d587knjoamxaizrNsYwkuHuVJvZwITN94j_hV0KW1LBAW-GRtbbNpB7NDi_-GD6dZH3YW6IsipPRBXNxTNMKZ6zjtdQwDCZ8lui1-Gf77tC0cMUGHX_eIgzIAdEIHr5RfZgik2HT1tBJzzq8g
</pre>
<p>I recommend that you read more about JWT tokens <a href="https://jwt.io/introduction/">here</a>. When this token is base 64 decoded it will look like this (above token is not valid, so it will not decode):</p>
<pre class="brush: csharp">
{
"aud": "fa74745b-8g52-cc4c-bv4f-0aa2fec4casd",
"iss": "https://sts.windows.net/fa74745b-8g52-cc4c-bv4f-0aa2fec4casd/",
"iat": 1526819139,
"nbf": 1526819139,
"exp": 1526823039,
"aio": "ADQA3/8HAAZADJXwEVFrvZwAq/V4Pgr63sT2mY9Q42KAu0fjxLFUzyI=",
"amr": [
"pwd"
],
"family_name": "Kavtaskin",
"given_name": "Zan",
"ipaddr": "182.67.124.103",
"name": "Zan Kavtaskin",
"nonce": "_091823kl",
"sub": "xwrrtnCcl_Kiknx1bRSO4OzoZMQ0WExnj3Fkj_66LEQ",
"unique_name": "Zan.Kavtaskin@someemail.com,
"ver": "1.0"
}
</pre>
<p> Token is essentially a key value property bag, these properties are called claims, these claims are about the authenticated user.</p>
<h3>Claims</h3>
<p>Claims are simply properties that are trusted (<a href="https://msdn.microsoft.com/en-us/library/ff359101.aspx">read more about claims here</a>). Now your web app will mainly use these claims for authentication i.e. who is the user, their unique id, etc. It is unlikely that claims will contain user role information (it can happen, however I have not seen this in practice). This is because 3rd party authentication provider normally does not know anything about your application. You can argue that role data is application data and does not belong with the authentication provider. However, ASP.NET needs role claims to perform authorisation checks, what do you do? Well, you append roles to your claims in your application.</p>
<h2>ASP.NET MVC Implementation</h2>
</br>
<h3>Appending roles to the claims</h3>
<pre class="brush: csharp">
public class AuthenticationClaimsAppenderMiddleware : OwinMiddleware
{
public AuthenticationClaimsAppenderMiddleware(OwinMiddleware next)
: base(next)
{
}
public override Task Invoke(IOwinContext context)
{
IUserService userService = ServiceLocator.Resolve<IUserService>();
ClaimsPrincipal claimsPrincipal = context.Authentication.User;
string idpID = claimsPrincipal.FindFirst(ClaimTypes.NameIdentifier).Value;
UserDto userDto = userService.GetUserByIdpID(idpID);
if (userDto == null)
return this.Next.Invoke(context);
ClaimsIdentity claimsIdentity = claimsPrincipal.Identity as ClaimsIdentity;
if (userDto.IsAdmin)
{
claimsIdentity.AddClaim(new Claim(ClaimTypes.Role, "Admin"));
}
return this.Next.Invoke(context);
}
}
</pre>
<h3>Checking the role claim</h3>
<p>Now all you have to do is check the role claim using the authorisation attribute this can be done at the controller and action level:</p>
<pre class="brush: csharp">
[Authorize(Roles = "Admin")]
public class TenantSettingsController : Controller
{
//...
}
</pre>
<pre class="brush: csharp">
[Authorize]
public class GroupsController : Controller
{
//...
[Authorize(Roles = "Admin")]
[HttpPost]
public ActionResult RemoveUser(UserModel model)
{
//...
}
}
</pre>
To enable unauthenticated access use AllowAnonymous attribute:
<pre class="brush: csharp">
[AllowAnonymous]
public class TenantController : Controller
{
//...
}
</pre>
To ensure that user is authenticated without specific role check, use authorize attribute:
<pre class="brush: csharp">
[Authorize]
public class UserSetupController : Controller
{
//...
}
</pre>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Browse "Multitenancy-Microservice-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br />
*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment.
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-18303940494382347592018-05-12T03:59:00.000-07:002018-05-26T15:05:45.779-07:00ASP.NET MVC Multitenancy, Part 0 - Subdomain VS URL Path<p> With multitenancy you will need to decide how you are going to identify your tenants. There are two user friendly approaches, subdomain fictionaltenant.multitenantedapp.com or URL Path multitenantedapp.com/fictionaltenant/. In this post we are going to explore disadvantages and advantages of these two approaches. </p>
<h2>Subdomain</h2>
<br/>
<h3>Configuration</h3>
<p>It is surprisingly easy to set this up for a simple deployment model. This is what you have to do:</p>
<h4>Setup DNS record</h4>
<p>I went to my domain registrar site and configured all my traffic on *.multitenantedapp.com to point to my Azure Web App domain e.g. *.multitenantedapp.com -> zkwildcardtest.azurewebsites.net.</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh25UpR5tNB-ZJ0QJr_o4Ge9qvlmCUtflNENYMIJguhkYVyWcAo6PKiZskbWUvQv1xMIai-98BLEBIqLSqNfG4g69KQLxl4EP9X39VyWUIkroAlXbZr2hDDx2IREWXaR18CddpyjoFRdfE/s1600/cname.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="128" data-original-width="863" height="94" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh25UpR5tNB-ZJ0QJr_o4Ge9qvlmCUtflNENYMIJguhkYVyWcAo6PKiZskbWUvQv1xMIai-98BLEBIqLSqNfG4g69KQLxl4EP9X39VyWUIkroAlXbZr2hDDx2IREWXaR18CddpyjoFRdfE/s640/cname.png" width="640" /></a></div>
<br/>
<h4>Setup wildcard hostname</h4>
<p>After that all I had to do was configure my Azure Web App to accept traffic for the hostname *.multitenantedapp.com. If you are using a web server such as IIS you will just need to change the IIS hostname binding. </p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh59P4hUZWclxgs2PKdLzPQSOKqMNZ-kJvFpJa1P-uHN18g0K4mmCf0kF68LIF7u71l9cDhxGqbFrZxd4pIQB9gnbwPNU_upVeSfA_d9apYzAU_6zkcPl_7GueLP_NQp8BEsk5ekTatzBo/s1600/hostname.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="274" data-original-width="798" height="218" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh59P4hUZWclxgs2PKdLzPQSOKqMNZ-kJvFpJa1P-uHN18g0K4mmCf0kF68LIF7u71l9cDhxGqbFrZxd4pIQB9gnbwPNU_upVeSfA_d9apYzAU_6zkcPl_7GueLP_NQp8BEsk5ekTatzBo/s640/hostname.png" width="640" /></a></div>
<br/>
<h4>Deploy your app</h4>
<p>Just to make sure everything works as it should, I have deployed a simple test app that reads the URI and extracts subdomain from it. I went to tenanta.multitenantedapp.com and this is what I saw:</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRzrZMr-fEXfyXYdyqLHWlRlIAR5wQx0_XpbPpsnNy01EuGAZ2TPDOtAH1SHgkCQYbKsN_PCl5lgxXQu2NbHYGGgMa_ixMjgtQfOY1G2CTim6r8qEYtCZFt8_mgAzhM7oMVijmHHtlrGw/s1600/example.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="472" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRzrZMr-fEXfyXYdyqLHWlRlIAR5wQx0_XpbPpsnNy01EuGAZ2TPDOtAH1SHgkCQYbKsN_PCl5lgxXQu2NbHYGGgMa_ixMjgtQfOY1G2CTim6r8qEYtCZFt8_mgAzhM7oMVijmHHtlrGw/s1600/example.png" /></a></div>
<br/>
<h3>Why should you use this approach?</h3>
<br/>
<h4>Advance estate management</h4>
<p> For me advance estate management is the main reason why you would go for this approach. As your application scales you will want to split up your deployment zones. For example, you might want to deploy 50 tenants to zone A and other 50 tenants to zone b. What if some tenants in zone A are being noise neighbours (using your application a lot)? How will you migrate them to the zone C? With subdomain you can use DNS to redirect traffic, subdomain will remain the same, just the DNS mapping will need to change.</p>
<h4>Customisation</h4>
<p> You can enable customisation, so for example if you own fictionalcompany.com domain you can point files.fictionalcompany.com to fictionalcompany.multitenantedapp.com. This sounds simple but there is more to it. You need register this hostname on your web server. Also if you would like your connection to be secure you will need to get fictionalcompany to give you the subdomain SSL certificate.</p>
<h4>Log quality</h4>
<p>
Here is how the subdomain IIS log looks like:
</p>
<pre class="brush: csharp">
date: 2018-01-30
time: 22:25:30
s-sitename: ZKWILDCARDTEST
cs-method: GET
cs-uri-stem: /
cs-uri-query: -
s-port: 80
cs-username: -
c-ip: 102.152.31.182
cs(User-Agent): -
cs(Cookie): -
cs(Referer): -
cs-host: fictionalcompany.multitenantedapp.com TENANT'S NAME APPEARS HERE
sc-status: 200
sc-substatus: 0
sc-win32-status: 0
sc-bytes: 1958
cs-bytes: 1066
time-taken: 0
</pre>
<p>cs-host (client-server host) actually shows fully qualified subdomain. This is great, as it is very easy to parse and hard to spoof. This can be used for business and security analytics. </p>
<h2>URL Path</h2>
<br/>
<h3>Configuration</h3>
<p>There is not much to URL path setup. You just need to configure your web application's framework mapping. If you are using ASP.NET MVC it would look something like this:</p>
<pre class="brush: csharp">
namespace Web
{
public class RouteConfig
{
public static void RegisterRoutes(RouteCollection routes)
{
//...
routes.MapRoute(
"multi",
"{tenant}/{controller}/{action}/{id}",
new {controller = "Dashboard", action = "Index", id = UrlParameter.Optional}
).DataTokens.Add("name", "webclient_multitenancy");
}
}
}
</pre>
<p>For full mapping explanation take a look at <a href="http://www.zankavtaskin.com/2017/08/aspnet-mvc-multitenant-routing-with-owin.html">this article.</a></p>
<p>With this approach you can introduce proxy layer, that proxy layer can then direct your traffic to different deployment zones. So in a way you can achieve same thing. However you are introducing additional infrastructure, which will require implementation, deployment, hosting and maintenance. If you can avoid it, you should.</p>
<h3>Why should you use this approach?</h3>
<p>
The main benefit is that it is simple to implement and requires no DNS work of any kind. However with this approach you do compromise on advance estate management, customisation and log quality. </p>
<p>Here is how the IIS log would look like with URL path:</p>
<pre class="brush: csharp">
date: 2018-01-30
time: 22:25:30
s-sitename: ZKWILDCARDTEST
cs-method: GET
cs-uri-stem: /fictionalcompany/somepage/ TENANT'S NAME APPEARS HERE
cs-uri-query: -
s-port: 80
cs-username: -
c-ip: 102.152.31.182
cs(User-Agent): -
cs(Cookie): -
cs(Referer): -
cs-host: multitenantedapp.com THIS IS NOT SHOWING TENANT'S NAME ANYMORE
sc-status: 200
sc-substatus: 0
sc-win32-status: 0
sc-bytes: 1958
cs-bytes: 1066
time-taken: 0
</pre>
<p>
The main thing to notice is that client server host is now not showing the tenant's name. Now you need to parse cs-uri-stem which is more difficult. Also if your application is a single page application that performs all operations via API, chances are that tenant's name will be in the header which means you will not know which tenant is making the API requests.
</p>
<h2>Conclusion</h2>
<p>If you are starting to work on a large-scale project, I strongly recommend that you give it a bit extra time and invest in the subdomain implementation. It's just so versatile, it will help you scale long term. However if you are working for a small start-up, you don't have the DNS skills in your team and you are not sure where project is going you can just use the URL path to get you off the ground.</p>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Browse "Multitenancy-Microservice-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br />
*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment.Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-60448931456832901662017-12-03T14:38:00.000-08:002017-12-03T14:38:37.083-08:00ASP.NET MVC Multitenancy, Part 3 - NHibernate Tenant Data Filtering and Shared Database
<p>In <a href="http://www.zankavtaskin.com/2017/08/aspnet-mvc-multitenant-routing-with-owin.html">part 1</a> we have established tenant context. In <a href="http://www.zankavtaskin.com/2017/11/aspnet-mvc-multitenancy-part-2-openid.html">part 2</a> we have configured authentication. Now we will be getting tenant data out of our RDBMS. Before we start storing and retrieving data it's important that you figure out what kind of multitenancy data segregation model you are going to go for. This is an important point and it really depends on your business requirements. To help you choose check out this Microsoft article: <a href="https://docs.microsoft.com/en-us/azure/sql-database/saas-tenancy-app-design-patterns">Multi-tenant SaaS database tenancy patterns</a>.
<p>My sample app is called “Stop The Line”, it is very basic. It holds very little data, and I have no idea if it's going to take off or not. I have decided to go with "Multi-tenant app with a single multi-tenant database" pattern. Put simply, this means that all my tenants are going to share the same database and I am going to use foreign key to discriminate between them. This is one the cheapest and fastest ways to implement tenancy.</p>
<p>When it comes to development what I don't want to do is to specify tenant id in each query and I don't want to manually specify tenant id when I create and update my domain objects. Remember the <a href="http://jeffreypalermo.com/blog/the-onion-architecture-part-1/">onion architecture</a>, as far as I am concerned tenancy is an infrastructure concern and it should stay out of my domain. After few minutes of searching I have found this great blog post that meets these requirements: <a href="http://www.mikevalenty.com/bolt-on-multi-tenancy-in-asp-net-mvc-with-unity-and-nhibernate-part-ii-commingled-data/">Bolt-on Multi-Tenancy in ASP.NET MVC With Unity and NHibernate: Part II – Commingled Data</a>. Let's incorporate this in to my sample app.
<h2>
Implementation</h2>
<p>In my sample app it's possible to query in multitenancy context and without. For example, if a new tenant is signing up for an account then we would not apply the multitenancy context. This is because TenantContext will not exist until tenant was created in the first place. </p>
<p>Also in this implementation I have tried to keep infrastructure decoupled so that it would not be too difficult to move to another database tenancy pattern.</p>
<h3>Base Configuration</h3>
<p>As there are going to be two different contexts, non-multitenant and multitenant. I have decided to create NHConfiguration base class.</p>
NHConfiguration.cs<br />
<pre class="brush: csharp">
public abstract class NHConfiguration
{
public abstract global::NHibernate.Cfg.Configuration Config { get; }
}
</pre>
<p>Below is the standard configuration that is going to be used for non-multitenant read/write.</p>
Configuration.cs<br />
<pre class="brush: csharp">
public class Configuration : NHConfiguration
{
public override NHibernate.Cfg.Configuration Config => config;
private NHibernate.Cfg.Configuration config;
public Configuration()
{
this.config = Fluently.Configure()
.Database(MsSqlConfiguration.MsSql2012
.ConnectionString(c => c.FromConnectionStringWithKey("default"))
#if DEBUG
.ShowSql()
.FormatSql()
#endif
.AdoNetBatchSize(50)
).Mappings(m =>
m.FluentMappings.AddFromAssembly(Assembly.GetExecutingAssembly())
)
.ExposeConfiguration(c => SchemaMetadataUpdater.QuoteTableAndColumns(c))
.BuildConfiguration();
}
}
</pre>
<p>This configuration will get injected into the Unit of Work. If you are new to this, then you should read <a href="http://www.zankavtaskin.com/2013/12/unit-of-work-for-nhibernate-and-entity.html">Unit Of Work Abstraction For NHibernate or Entity Framework C# Example</a>. I have omitted the class implementation.</p>
<pre class="brush: csharp">
public class NHUnitOfWork : IUnitOfWork
{
//..
private NHConfiguration nhConfiguration;
public NHUnitOfWork(NHConfiguration nhConfiguration)
{
this.nhConfiguration = nhConfiguration;
//..
}
//..
}
</pre>
<p>Unfortunately with this solution you will still need to add the private "tenantId" field to your domain classes.</p>
<pre class="brush: csharp">
public class Some
{
//..
private Guid tenantId;
}
</pre>
<p>The good news is that you don't have to do with it anything inside your domain. It will be handled by infrastructure. The bad news is that it still there. I have spent few hours trying to remove it, I have used some reflection and proxy class generation techniques, unfortunately to no avail. If you find an elegant solution please do share it.</p>
<h3>Multitenant Writes</h3>
<p>This is where standard configuration is expanded and interceptor is set. Interceptor will be used when some object is updated or saved.</p>
ConfigurationMultiTenancy.cs<br />
<pre class="brush: csharp">
public class ConfigurationMultiTenancy : Configuration
{
public ConfigurationMultiTenancy(NHSharedDatabaseMultiTenancyInterceptor interceptor)
{
this.Config.SetInterceptor(interceptor);
}
}
</pre>
<p>Above you will notice that NHSharedDatabaseMultiTenancyInterceptor is injected in to ConfigurationMultiTenancy. Here is the actual interceptor.</p>
NHSharedDatabaseMultiTenancyInterceptor.cs<br />
<pre class="brush: csharp">
public class NHSharedDatabaseMultiTenancyInterceptor : EmptyInterceptor
{
readonly TenantContext tenantContext;
public NHSharedDatabaseMultiTenancyInterceptor(TenantContext tenantContext)
{
this.tenantContext = tenantContext;
}
public override bool OnSave(object entity, object id, object[] state, string[] propertyNames, IType[] types)
{
int index = Array.IndexOf(propertyNames, "tenantId");
if (index == -1)
return false;
state[index] = this.tenantContext.ID;
entity.GetType()
.GetField("tenantId", BindingFlags.Instance | BindingFlags.NonPublic)
.SetValue(entity, tenantContext.ID);
return base.OnSave(entity, id, state, propertyNames, types);
}
}
</pre>
<h3>Multitenant Reads</h3>
<p>Configuring reads is much simpler. You just need to enable a filter and provide the parameter. First you need to define the actual filter.</p>
MultitenantFilter.cs<br />
<pre class="brush: csharp">
public class MultitenantFilter : FilterDefinition
{
public MultitenantFilter()
{
WithName("MultitenantFilter").AddParameter("TenantId", NHibernateUtil.Guid);
}
}
</pre>
<p>Then you need to enable it.</p>
NHUnitOfWorkMultitenancy.cs<br />
<pre class="brush: csharp">
public class NHUnitOfWorkMultitenancy : NHUnitOfWork
{
public NHUnitOfWorkMultitenancy(NHConfiguration nhConfiguration, TenantContext tenantContext)
: base(nhConfiguration)
{
this.Session.EnableFilter("MultitenantFilter").SetParameter("TenantId", tenantContext.ID);
}
}
</pre>
<p>Finally you just need to apply it.</p>
<pre class="brush: csharp">
public class SomeMap : ClassMap<Some>
{
public SomeMap()
{
//..
//This is used for writes
Map(Reveal.Member<Some>("tenantId")).Not.Nullable();
//This is used for reads
ApplyFilter<MultitenantFilter>("TenantId = :TenantId");
}
}
</pre>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Want to see this in action? Check out <br /> "Multitenancy-Microservice-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br />
<p>*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment.</p>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-57646615261804424972017-11-30T15:33:00.000-08:002018-05-26T15:08:03.933-07:00ASP.NET MVC Multitenancy, Part 2 - OpenID Connect In <a href="http://www.zankavtaskin.com/2017/08/aspnet-mvc-multitenant-routing-with-owin.html">part 1</a> we have established tenant context. Most business applications need authentication. I have decided to use OpenID Connect, luckily for me it comes with awesome middleware and it's very easy to configure. Here is how you would do it:
<br />
<pre class="brush: csharp">public class Startup
{
public void Configuration(IAppBuilder app)
{
//...
app.UseOpenIdConnectAuthentication(new OpenIdConnectAuthenticationOptions
{
ClientId = "clientid",
Authority = 'authority",
RedirectUri = "http://mysite.com/"
});
}
}
</pre>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx2Nf2vWttEBxSiAljPPSs6Zrjt0KcVY6_M2l2gPJ2xUczyEGyk1rjhx_68voqM5CI6xflXghlE_rmTCKjPZP1TPTcsl4xek7GHiMDybAW04gkY3b0WlM7SqyNKV8sI8dH3A1N15FFO8s/s1600/openid-r-logo-900x360.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="128" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx2Nf2vWttEBxSiAljPPSs6Zrjt0KcVY6_M2l2gPJ2xUczyEGyk1rjhx_68voqM5CI6xflXghlE_rmTCKjPZP1TPTcsl4xek7GHiMDybAW04gkY3b0WlM7SqyNKV8sI8dH3A1N15FFO8s/s320/openid-r-logo-900x360.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
However, not so fast, for each tenant you will need to configure client id, authority, url, etc. This also needs to be done at runtime as you don't want to ship new code every time new tenant registers to use your software. I have looked around for a solution and it turns out that this is a known problem, just take a look at the <a href="https://github.com/aspnet/Security/issues/1179">The Grand Auth Redesign</a>. After a surprising amount of searching I have stumbled upon this amazing <a href="http://benfoster.io/blog/aspnet-core-multi-tenant-middleware-pipelines">Multi-tenant middleware pipelines in ASP.NET Core</a> blog post by Ben Foster. This solution is so damn good. Please read it. Unfortunately it did not work for me out of the box I had to port it over from ASP.NET Core to ASP.NET. <br />
<br />
Hopefully by blogging and linking to Ben's article it will be easier for others to find it.
<br />
<br />
<h2>
Example Usage</h2>
<pre class="brush: csharp">public class Startup
{
public void Configuration(IAppBuilder app)
{
//..
app.UsePerTenant((tenantContext, appBranch) =>
{
appBranch.SetDefaultSignInAsAuthenticationType(CookieAuthenticationDefaults.AuthenticationType);
appBranch.UseCookieAuthentication(new CookieAuthenticationOptions
{
CookieName = $"OAuthCookie.{tenantContext.FriendlyName}"
});
appBranch.UseOpenIdConnectAuthentication(new OpenIdConnectAuthenticationOptions
{
ClientId = tenantContext.AuthClientId,
Authority = tenantContext.AuthAuthority,
RedirectUri = $"http://localhost:2295/{tenantContext.FriendlyName}/"
});
});
}
}
</pre>
<br />
<h2>
ASP.NET Implementation</h2>
<br />
TenantPipelineMiddleware.cs<br />
<pre class="brush: csharp">
/// <summary>
/// Workaround for the known OpenID multitenancy issue https://github.com/aspnet/Security/issues/1179
/// based on http://benfoster.io/blog/aspnet-core-multi-tenant-middleware-pipelines and https://weblogs.asp.net/imranbaloch/conditional-middleware-in-aspnet-core designs
/// </summary>
public class TenantPipelineMiddleware : OwinMiddleware
{
readonly IAppBuilder rootApp;
readonly Action<TenantContext, IAppBuilder> newBranchAppConfig;
readonly ConcurrentDictionary<TenantContext, Lazy<Func<IDictionary<string, object>, Task>>> branches;
public TenantPipelineMiddleware(OwinMiddleware next, IAppBuilder rootApp, Action<TenantContext, IAppBuilder> newBranchAppConfig)
: base(next)
{
this.rootApp = rootApp;
this.newBranchAppConfig = newBranchAppConfig;
this.branches = new ConcurrentDictionary<TenantContext, Lazy<Func<IDictionary<string, object>, Task>>>();
}
public override async Task Invoke(IOwinContext context)
{
TenantContext tenantContext = context.GetTenantContext();
if (tenantContext == null || tenantContext.IsEmpty())
{
await this.Next.Invoke(context);
return;
}
Lazy<Func<IDictionary<string, object>, Task>> branch =
branches.GetOrAdd(tenantContext, new Lazy<Func<IDictionary<string, object>, Task>>(() =>
{
IAppBuilder newAppBuilderBranch = rootApp.New();
newBranchAppConfig(tenantContext, newAppBuilderBranch);
newAppBuilderBranch.Run((oc) => this.Next.Invoke(oc));
return newAppBuilderBranch.Build();
}));
await branch.Value(context.Environment);
}
}
</pre>
<br />
OwinContextExtensions.cs<br />
<pre class="brush: csharp">
public static class OwinContextExtensions
{
static string tenantContextKey = "tenantcontext";
public static TenantContext GetTenantContext(this IOwinContext context)
{
if (context.Environment.ContainsKey(tenantContextKey))
{
return (TenantContext)context.Environment[tenantContextKey];
}
return null;
}
}
</pre>
<br />
AppBuilderExtensions.cs<br />
<pre class="brush: csharp">
public static class AppBuilderExtensions
{
public static IAppBuilder UsePerTenant(this IAppBuilder app, Action<TenantContext, IAppBuilder> newBranchAppConfig)
{
return app.Use<TenantPipelineMiddleware>(app, newBranchAppConfig);
}
}
</pre>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Browse "Multitenancy-Microservice-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br />
*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment.
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-42661906689859924622017-08-20T03:10:00.002-07:002018-05-26T15:08:36.966-07:00ASP.NET MVC Multitenancy, Part 1 - Routing with OWIN<span style="font-style: italic;">This blog post was rewritten on 29/11/2017.</span><br/><br/>
Recently I have been migrating one of my multitenant ASP.NET MVC application's to the OWIN middleware. This has presented me with an opportunity to change my initial tenant resolution and pipeline registration implementation. <br />
<br/>
I've started by doing some research to see if I could find something I could just download and use. However, I could not find anything that would meet the following requirements:
<ul>
<li>ASP.NET Core and ASP.NET compatibility</li>
<li>Optional multitenant processing i.e. all requests don't have be multitenant</li>
<li>Lifecycle delegates for missing tenant identifier and tenant record, and tenant context creation</li>
<li>Runtime tenant context dependency injection registration</li>
<li>Compatible with multitenant URL paths mysite.com/sometenant/ and could work with subdomains sometenant.mysite.com</li>
</ul>
<br/>
After much deliberation, I have decided to go with my own implementation. However this implementation was massively influenced by <a href="http://benfoster.io/blog/asp-net-5-multitenancy">Ben Foster's Building multi-tenant applications with ASP.NET Core (ASP.NET 5)</a> and <a href="http://sourcebrowser.io/Browse/jchannon/katanaproject/src/Microsoft.Owin.Security.OpenIdConnect/OpenIdConnectAuthenticationMiddleware.cs">OpenID Connect Authentication Middleware</a> designs.
<br/><br/>
<h2>Example Usage</h2>
<pre class="brush: csharp">
public class Startup
{
public void Configuration(IAppBuilder app)
{
//..
app.UseMultitenancy(new MultitenancyNotifications<YourTenantRecord>
{
TenantIdentifierNotFound = context =>
{
throw new HttpException(404, "Tenant identifier must be provided");
},
TenantRecordNotFound = context =>
{
context.Response.Redirect("/signup/tenant/");
return Task.FromResult(0);
},
CreateTenantContext = (context, tenantRecord) =>
{
ITenantContextFactory tenantContextFactory = ServiceLocator.Resolve<ITenantContextFactory>();
TenantContext tenantContext = tenantContextFactory.Create(tenantRecord.Id, tenantRecord.NameFriendly, tenantRecord.AuthClientId, tenantRecord.AuthAuthority);
return Task.FromResult(tenantContext);
}
});
}
}
</pre>
Please note that "CreateTenantContext" is a factory delegate. It allows you to control tenant record to tenant context mapping. It also allows you to perform other actions such as runtime tenant context dependency injection registration. By doing this, downstream objects such as controllers will get tenant context injected in to them.
<pre class="brush: csharp">
public class SomeController : Controller
{
public SomeController(TenantContext tenantContext)
{
//..
}
}
</pre>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/tree/master/Web/Middleware/TenantContext">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Browse "Multitenancy-Microservice-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br/>
<h2>
Implementation</h2>
<br/>
<h3>Tenant Identifier Extraction</h3>
In my implementation have decided to use MVC routing with data tokens for tenant discrimination. <br />
<br/>
RouteConfig.cs<br />
<pre class="brush: csharp">
public class RouteConfig
{
public static void RegisterRoutes(RouteCollection routes)
{
//...
routes.MapRoute(
"multi",
"{tenant}/{controller}/{action}/{id}",
new {controller = "Home", action = "Index", id = UrlParameter.Optional}
).DataTokens.Add("name", "webclient_multitenancy");
}
}
</pre>
By using data tokens, I am able to check the request route. In this case if request route has data token of name "webclient_multitenancy" I will know that I will be able to extract tenant identifier from the route. As you have probably gathered, in this case I am getting multitenancy identifier from the URL path mysite.com/sometenant/. <br/>
<br/>
ITenantIdentifierExtractor.cs<br />
<pre class="brush: csharp">
public interface ITenantIdentifierExtractor
{
bool CanExtract(IOwinContext context);
string GetIdentifier(IOwinContext context);
}
</pre>
<br/>
RouteDataTokensTenantIdentifierExtractor.cs<br />
<pre class="brush: csharp">
public class RouteDataTokensTenantIdentifierExtractor : ITenantIdentifierExtractor
{
public bool CanExtract(IOwinContext context)
{
RouteData routeData = this.getRouteData(context);
return routeData != null && routeData.DataTokens.ContainsValue("webclient_multitenancy");
}
public string GetIdentifier(IOwinContext context)
{
if (!this.CanExtract(context))
return null;
RouteData routeData = this.getRouteData(context);
return routeData.GetRequiredString("tenant");
}
private RouteData getRouteData(IOwinContext context)
{
HttpContextBase httpContext = (HttpContextBase)context.Environment["System.Web.HttpContextBase"];
return RouteTable.Routes.GetRouteData(httpContext);
}
}
</pre>
<br />
You don’t have to use data token approach in your application, you can get multitenancy data out of the HTTP request manually.<br />
<br />
<h3>Tenant Record Resolution</h3>
Resolution is where you call service or repository to obtain tenant record, this tenant record will be used later on to create tenant context. <br /><br />
ITenantRecordResolver.cs<br />
<pre class="brush: csharp">
public interface ITenantRecordResolver<TTenantRecord>
{
TTenantRecord GetTenant(string tenantIdentifier);
}
</pre><br />
In this implementation, I have decided to call the ITenantService to get the TenantDto object.
<br/>
<br/>
TenantRecordResolver.cs<br />
<pre class="brush: csharp">
public class TenantRecordResolver : ITenantRecordResolver<TenantDto>
{
readonly ITenantService tenantService;
public TenantRecordResolver(ITenantService tenantService)
{
this.tenantService = tenantService;
}
public TenantDto GetTenant(string tenantIdentifier)
{
return this.tenantService.Get(tenantIdentifier);
}
}
</pre><br />
Most of the time on each request you will be resolving tenant record. At scale this will become very inefficient so it is highly likely that you will want to cache tenant record, to do this you can use decorator pattern.<br /><br />
TenantRecordResolverCacheDecorator.cs<br />
<pre class="brush: csharp">
public class TenantRecordResolverCacheDecorator : ITenantRecordResolver<TenantDto>
{
readonly TenantRecordResolver tenantRecordResolver;
public TenantRecordResolverCacheDecorator(TenantRecordResolver tenantRecordResolver)
{
this.tenantRecordResolver = tenantRecordResolver;
}
public TenantDto GetTenant(string tenantIdentifier)
{
string cacheKey = $"tenantIdentifier:{tenantIdentifier}";
TenantDto tenant = (TenantDto)MemoryCache.Default[cacheKey];
if (tenant != null)
{
return tenant;
}
tenant = this.tenantRecordResolver.GetTenant(tenantIdentifier);
if (tenant == null)
{
return null;
}
MemoryCache.Default.Set(cacheKey, tenant, new CacheItemPolicy
{
AbsoluteExpiration = DateTimeOffset.Now.AddMinutes(10)
});
return tenant;
}
}
</pre>
<br />
<h4>
Middleware & Extensions</h4>
We now have all the key components in place and it is time to implement the actual middleware.<br /><br />
MultitenancyMiddleware.cs<br />
<pre class="brush: csharp">
public class MultitenancyMiddleware<TTenantRecord>
where TTenantRecord : class
{
readonly ITenantIdentifierExtractor tenantIdentifierExtractor;
readonly ITenantRecordResolver<TTenantRecord> tenantRecordResolver;
readonly MultitenancyNotifications<TTenantRecord> notifications;
readonly Func<Task> next;
public MultitenancyMiddleware(Func<Task> next, ITenantIdentifierExtractor tenantIdentifierExtractor,
ITenantRecordResolver<TTenantRecord> tenantRecordResolver, MultitenancyNotifications<TTenantRecord> notifications)
{
this.next = next;
this.tenantIdentifierExtractor = tenantIdentifierExtractor;
this.tenantRecordResolver = tenantRecordResolver;
this.notifications = notifications;
}
public async Task Invoke(IOwinContext context)
{
if (!this.tenantIdentifierExtractor.CanExtract(context))
{
await this.next();
return;
}
string identifier = this.tenantIdentifierExtractor.GetIdentifier(context);
if (string.IsNullOrEmpty(identifier))
{
await this.notifications.TenantIdentifierNotFound(context);
return;
}
TTenantRecord tenantRecord = this.tenantRecordResolver.GetTenant(identifier);
if (tenantRecord == null)
{
await this.notifications.TenantRecordNotFound(context);
return;
}
TenantContext tenantContext = await this.notifications.CreateTenantContext(context, tenantRecord);
context.SetTenantContext(tenantContext);
await this.next();
}
}
</pre>
<br />
MultitenancyNotifications.cs<br />
<pre class="brush: csharp">
public class MultitenancyNotifications<TTenantRecord>
where TTenantRecord : class
{
public Func<IOwinContext, Task> TenantIdentifierNotFound { get; set; }
public Func<IOwinContext, Task> TenantRecordNotFound { get; set; }
public Func<IOwinContext, TTenantRecord, Task<TenantContext>> CreateTenantContext { get; set; }
public MultitenancyNotifications()
{
this.TenantIdentifierNotFound = context => Task.FromResult(0);
this.TenantRecordNotFound = context => Task.FromResult(0);
this.CreateTenantContext = (context, tenantRecord) => Task.FromResult<TenantContext>(null);
}
}
</pre>
<br />
OwinContextExtensions.cs<br />
<pre class="brush: csharp">
public static class OwinContextExtensions
{
static string tenantContextKey = "tenantcontext";
public static void SetTenantContext(this IOwinContext context, TenantContext tenantContext)
{
context.Environment.Add(tenantContextKey, tenantContext);
}
}
</pre>
AppBuilderExtensions.cs<br />
<pre class="brush: csharp">
public static class AppBuilderExtensions
{
public static IAppBuilder UseMultitenancy<TTenantRecord>(this IAppBuilder app,
MultitenancyNotifications<TTenantRecord> notifications)
where TTenantRecord : class
{
return app.Use((context, next) =>
{
MultitenancyMiddleware<TTenantRecord> multitenancyMiddleware =
ServiceLocator.Resolve<MultitenancyMiddleware<TTenantRecord>>(new { next, notifications });
return multitenancyMiddleware.Invoke(context);
});
}
}
</pre>
</br>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Multitenancy-Microservice-FederatedIdentity-Example/">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Browse "Multitenancy-Microservice-FederatedIdentity-Example" Repository On Github.</span>
</a>
</div>
<br/>
*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment.
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-22240572192869213532017-03-09T13:18:00.000-08:002017-11-12T09:25:15.033-08:00Cloud Architecture & Engineering, B2B Revolution and B2C Evolution<style>
.b2brevo tr th {
border: 1px solid #cccccc;
}
.b2brevo tr td {
border: 1px solid #cccccc;
}
.b2brevo tr:nth-child(even) {
background: #F9F9F9;
}
</style><a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br /></a>
Few years ago I was having a conversation with one of my friends about cloud, SaaS and multitenancy architecture. I was telling him how different it was to normal “hosted” app architecture. That we can setup new business tenant with custom url in seconds. He has said, well, instant tenant provisioning and custom URLs are not new. So what’s so special about it? Blogger and WordPress have been doing it for years. At that point in time I have struggled to answer his question, there was too much to say, ideas in my head were too raw. In this article I am going to explain why cloud, SaaS and multitenancy is revolutionary.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhumrxkDzlnmZnHXCygZ_Qmghus65DhZzrS7RAZL0g1PzIX5KvO1ZSzuzIYqmLZgDuKiNGHhIaXfkXKgLJQfy65bi4utifyV4F8SRkgtQIaQNZxlSmCYLtXD83CQLFkTnRGUKZyR-vLjTE/s1600/revolution.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="505" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhumrxkDzlnmZnHXCygZ_Qmghus65DhZzrS7RAZL0g1PzIX5KvO1ZSzuzIYqmLZgDuKiNGHhIaXfkXKgLJQfy65bi4utifyV4F8SRkgtQIaQNZxlSmCYLtXD83CQLFkTnRGUKZyR-vLjTE/s640/revolution.jpg" width="640" /></a></div>
<br />
<div class="" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Before we start, let’s get few definitions out of the way:<br /><br />
<table border="0" cellpadding="5" cellspacing="0" class="table table-striped table-bordered b2brevo">
<thead>
<tr>
<th><div style="text-align: left;">
Model</div>
</th>
<th><div style="text-align: left;">
Description</div>
</th>
<th><div style="text-align: left;">
Example</div>
</th>
</tr>
</thead>
<tbody>
<tr>
<td>B2B, Business Applications</td>
<td>Business sells to businesses</td>
<td>Oracle, Visual Studio Team Services, Sage & Salesforce</td>
</tr>
<tr>
<td>B2C, Consumer Applications</td>
<td>Business sells to consumers</td>
<td>Spotify, Netflix & Blogger.</td>
</tr>
</tbody>
</table><br />
Your solution’s technical requirements will differ a lot depending on what business model you follow, we will be compare typical B2C and B2B requirements shortly.<br />
<br />
<table border="0" cellpadding="5" cellspacing="0" class="table table-striped table-bordered b2brevo">
<thead>
<tr>
<th><div style="text-align: left;">
Term</div>
</th>
<th><div style="text-align: left;">
Somewhat My Definition</div>
</th>
</tr>
</thead>
<tbody>
<tr>
<td>Software as a Service (SaaS)</td>
<td>This is where you don’t care about software versioning, upgrades, deployments, where it sits, etc. You just pay for it and use it. If you are using Spotify, you are using SaaS.</td>
</tr>
<tr>
<td>Cloud</td>
<td>Before cloud we had hosting companies, where you could rent a VM and host your site. Now we have Cloud, which is the same thing but it’s just much smarter and comes with lots of tools. Companies such as Microsoft offer prepackaged services that seamlessly scale your apps horizontally and all you have to do is just upload your binaries via FTP to a folder.</td>
</tr>
<tr>
<td>Multitenancy</td>
<td>Traditionally each company would get their own “installation”, with multitenancy there is only one installation and everyone is seamlessly using it.</td>
</tr>
</tbody>
</table>
<br />
These concepts are very important as they will feed of each other, as I will illustrate this later on.<br />
People use cloud, multitenancy and SaaS as synonyms and this is a mistake (I make this mistake as well).<br />
<br />
<h2>B2C, Consumer Applications</h2>
<br />
When it comes to selling directly to customers things are easier, there are a lot less hoops to jump through. Here are some technical requirements that you might need to follow when you build B2C app:<br />
<br />
<table border="0" cellpadding="5" cellspacing="0" class="table table-striped table-bordered b2brevo" style="text-align: left;">
<thead>
<tr>
<th>Requirement</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>Availability</td>
<td>Consumer facing sites, need to be highly available, if they are not, this can cause reputation damage and loss in revenue.</td>
</tr>
<tr>
<td>Authentication</td>
<td>Consumers need to be able to login using some form of login, Google, Facebook, etc.</td>
</tr>
<tr>
<td>PCI DSS</td>
<td>When customers make payments it’s important that card data is used & stored securely.</td>
</tr>
<tr>
<td>Recovery Point & Time Objectives</td>
<td>Similiar to availability it’s critical that if data was corrupted or upgrade has gone wrong that app can recover quickly with as much data as possible.<br />
<br /></td>
</tr>
</tbody>
</table>
<br />
<div style="text-align: left;">
<b>B2C architecture before cloud:</b></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqVtQ9NMqYsVh8fG_aeSWDCo8KcR5RrAobUFXrxPJYasI6nLe-IeyXunsy0XPmSu7m89t7bdpWCemlS29v0HmSRcIvo3iaWDjZ5hzwXIP_zuiKyreNzg_qG9Svyu09ZPnx3XRNZFAwyeI/s1600/B2CHosted.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="576" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqVtQ9NMqYsVh8fG_aeSWDCo8KcR5RrAobUFXrxPJYasI6nLe-IeyXunsy0XPmSu7m89t7bdpWCemlS29v0HmSRcIvo3iaWDjZ5hzwXIP_zuiKyreNzg_qG9Svyu09ZPnx3XRNZFAwyeI/s640/B2CHosted.jpg" width="640" /></a></div>
<br />
<b>B2C architecture after cloud:</b>
<br /><br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgjcbeofJcJuPdiywZ0PP-sup8dbRgaUHssQeJzi6R2gLphC0LIMqMDbIiFqSzq98tajIXOZDyom7IwVQsgNICeGNCRYj7ruoZdGnfDqzRMHA8lJmOE4-Nya-f4MmVLPDum7OH4hOhabs/s1600/B2CCloud.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgjcbeofJcJuPdiywZ0PP-sup8dbRgaUHssQeJzi6R2gLphC0LIMqMDbIiFqSzq98tajIXOZDyom7IwVQsgNICeGNCRYj7ruoZdGnfDqzRMHA8lJmOE4-Nya-f4MmVLPDum7OH4hOhabs/s640/B2CCloud.jpg" width="582" /></a></div>
<br />
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
This list is not complete, however the point is that typical B2C apps don’t have lots of technical requirements enforced upon them from consumers. This is why B2C service is so much cheaper, how much do you pay for Netflix and Spotify?</div>
<div style="text-align: left;">
<br /></div>
<br />
<h2 style="text-align: left;">B2B, Business Applications</h2>
<br />
<div style="text-align: left;">
Here are some technical requirements that businesses often enforce on to other businesses:</div>
<br />
<table border="0" cellpadding="5" cellspacing="0" class="table table-striped table-bordered b2brevo"style="text-align: left;">
<thead>
<tr>
<th>Requirement</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>Corporate Identity Integration (Single Sign On Authentication)</td>
<td>Medium/Large size companies have their own identity apps such as AzureAD, they don’t want to use your built in authentication mechanism if they can enable Single Sign On. This is where Federated identity comes in.</td>
</tr>
<tr>
<td>Audit</td>
<td>They would like assurance and full trail of what’s happening inside the system, so they will ask for audit.</td>
</tr>
<tr>
<td>Data restore & Backup</td>
<td>If company makes a mistake in their environment they will want to restore their data back to how it was.</td>
</tr>
<tr>
<td>Integration</td>
<td>Often they have their own systems and they will want to integrate their existing applications with your application.</td>
</tr>
<tr>
<td>Security clearance</td>
<td>Depending on the industry your business customers might want your ops staff to be security cleared, if this is not possible then sensitive data needs to be hidden or encrypted.</td>
</tr>
<tr>
<td>Escrow</td>
<td>Businesses need to ensure businesses continuity, to do this they need to be able keep going even your business will go out of business.</td>
</tr>
<tr>
<td>ISO 27001</td>
<td>Key management, Antivirus, Firewalls, Segregation of duties, Pen testing, etc.</td>
</tr>
<tr>
<td>Reporting</td>
<td>Access to data to be able to report on it.</td>
</tr>
<tr>
<td>Customisation & Configuration</td>
<td>Data retention, authorisation, workflow, etc.</td>
</tr>
<tr>
<td>Availability</td>
<td>Business sites don’t necessarily need to be as highly available as in a lot of industries people work 9am-5pm.</td>
</tr>
<tr>
<td>PCI DSS</td>
<td>When businesses make payments it’s important that card data is used & stored securely.</td>
</tr>
<tr>
<td>Recovery Point & Time</td>
<td>It’s critical that if data was corrupted or upgrade has gone wrong that app can recover quickly, business finance or patient data can’t be lost, huge amount of productivity can be lost if this not designed correctly.</td>
</tr>
<tr>
<td>Support</td>
<td>Be able to troubleshoot issues quickly.</td>
</tr>
<tr>
<td>On-prem installation</td>
<td>Company is able to install your software in their own environment.</td>
</tr>
</tbody>
</table>
<br />
<div style="text-align: left;">
This list is not complete, however if you compare B2B to B2C it’s easy to see that there is a lot more to it. So when companies charge other companies a lot more for their services we can appreciate why this happens.</div>
<div style="text-align: left;">
<br /></div>
<b></b><br />
<div style="text-align: left;">
<b>B2B architecture before cloud and multitenancy:</b></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4C45URM-v8bxatcrTcYqWVY0_CLa5fEbRLc8W7iQilZUpT6w7i5Mr6-G5K160gSwpW95lJG7lCDPSaCUjJWPqy89WhPIUA16zVEzde1yhkUo0irXw_gQMKdznREery5HdI7r4cJMdWqo/s1600/B2BHosted.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4C45URM-v8bxatcrTcYqWVY0_CLa5fEbRLc8W7iQilZUpT6w7i5Mr6-G5K160gSwpW95lJG7lCDPSaCUjJWPqy89WhPIUA16zVEzde1yhkUo0irXw_gQMKdznREery5HdI7r4cJMdWqo/s640/B2BHosted.jpg" width="640" /></a></div>
<div style="text-align: left;">
<br /></div>
<br />
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<b><b>B2B architecture after cloud and multitenancy:</b></b></div>
<b>
</b><br />
<div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg1-yjf1gcTYQUwQi8osfkB8Rkyeq5bno7bbuJAuLV45zF-uZI9VhGJc6N2818LMXI3jHyaG0gtuAVs413s0VbnxrArSsSbBlKGIFUvzPpjNJuPlzrKwmDj-fnBOpgAU2Cx7aspD4LAtE/s1600/B2BCloud.jpg" imageanchor="1"><img border="0" height="586" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg1-yjf1gcTYQUwQi8osfkB8Rkyeq5bno7bbuJAuLV45zF-uZI9VhGJc6N2818LMXI3jHyaG0gtuAVs413s0VbnxrArSsSbBlKGIFUvzPpjNJuPlzrKwmDj-fnBOpgAU2Cx7aspD4LAtE/s640/B2BCloud.jpg" width="640" /></a></div>
<br />
<h2 style="text-align: left;">
So where is the revolution?</h2>
<br />
Consumer facing sites were doing SaaS with multitenancy for years. Websites like YouTube and Ebay have been scaling their solutions for a long time. Cloud has just made it easier for them to scale and in turn it has reduced their hosting cost.
Business applications were doing SaaS for a long time, however multitenancy and cloud was just not a thing. Most service providers painfully installed software for each customer manually, then, they upgraded each customer one at the time and they monitored each customer one at the time. You get the picture. Customer provisioning took weeks and maybe months. Multitenancy offers something truly awesome, it removes slow per customer installation and upgrade problem and in turn it dramatically speeds up the software delivery and customer onboarding process. Cloud removes the infrastructure maintenance and provisioning burden. Together they simply transform business in a remarkable way. B2B apps can finally compete with B2C apps in terms of feature delivery speed and cost.
<br /><br />
<h2 style="text-align: left;">Conclusion</h2>
<br />
<div style="text-align: left;">
Let’s come back to the conversation with my friend. He thought that nothing was new here because he was thinking of B2C evolution. While I was thinking of B2B where multitenancy and cloud is transforming businesses and revolutionising business applications.</div>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-41025424312973287762017-02-12T04:12:00.003-08:002017-02-12T04:27:07.080-08:00Azure App Services Web App Antivirus - Concept<h2>
<a href="https://www.blogger.com/null" id="Introduction_1"></a>Introduction</h2>
<br />
Good web application antivirus solution needs to meet the following acceptance criteria:<br />
<ul>
<li>Real-time inbound and outbound file scanning</li>
<li>Low latency, sub second performance</li>
<li>Transparent, no need to make changes to your web app</li>
<li>Requires no maintenance and is easy to setup</li>
</ul>
<div>
<b><br /></b>
<b>If you are using PaaS service such as Azure Web Apps your antivirus options are:</b></div>
<style>
#azureappserviceswebappav tr th {
border: 1px solid #cccccc;
}
#azureappserviceswebappav tr td {
border: 1px solid #cccccc;
}
</style>
<br />
<table border="0" cellpadding="5" cellspacing="0" class="table table-striped table-bordered" id="azureappserviceswebappav" style="width: 100%px;">
<thead>
<tr>
<th width="30%"><span style="font-size: x-small;">Option</span></th>
<th width="30%"><span style="font-size: x-small;">Limitation</span></th>
<th width="10%"><span style="font-size: x-small;">Installation Effort</span></th>
<th width="10%"><span style="font-size: x-small;">Maintenance Effort</span></th>
<th width="10%"><span style="font-size: x-small;">Cost</span></th>
<th width="10%"><span style="font-size: x-small;">Meets criteria</span></th>
</tr>
</thead>
<tbody>
<tr>
<td><span style="font-size: x-small;">Firewall with ICAP integration</span></td>
<td><span style="font-size: x-small;">Expense and ongoing maintenance</span></td>
<td><span style="font-size: x-small;">High</span></td>
<td><span style="font-size: x-small;">Medium</span></td>
<td><span style="font-size: x-small;">High</span></td>
<td><span style="font-size: x-small;">Yes</span></td>
</tr>
<tr>
<td><span style="font-size: x-small;">Send files to the antivirus API from your app</span></td>
<td><span style="font-size: x-small;">Extra unnecessary development work</span></td>
<td><span style="font-size: x-small;">Medium/High</span></td>
<td><span style="font-size: x-small;">Medium</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">No</span></td>
</tr>
<tr>
<td><span style="font-size: x-small;">Background service that scans your files</span></td>
<td><span style="font-size: x-small;">Not real time, high possibility that viruses will be uploaded and downloaded</span></td>
<td><span style="font-size: x-small;">Medium</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">No</span></td>
</tr>
<tr>
<td><span style="font-size: x-small;">Reverse proxy binary content forwarding</span></td>
<td><span style="font-size: x-small;">Scans only uploads</span></td>
<td><span style="font-size: x-small;">Medium</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">No</span></td>
</tr>
<tr>
<td><span style="font-size: x-small;">IIS Filter (this solution)</span></td>
<td><span style="font-size: x-small;">PoC, not production ready</span></td>
<td><span style="font-size: x-small;">Medium</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">Low</span></td>
<td><span style="font-size: x-small;">Yes</span></td>
</tr>
</tbody>
</table>
<em><br /></em>
<em>In depth exploration of different antivirus options is beyond the scope this article.</em><br />
<em><br /></em>
<br />
<h2>
<a href="https://www.blogger.com/null" id="IIS_Filter_Solution_inspired_by_ModSecurity_20"></a>IIS Filter Solution, inspired by ModSecurity</h2>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJV8Wtk-C5fDWm_5v3XOauj2laXtsBFmeQoFOjiOdHWgBRLV0nj1WIQe6b5YXLMSZScMgS7LNZWCCbmDcWml73wZy0gvkoaZvFQPP2OK2dgRMAeAn_KHxF9YFp9qu2CJnb3pItX_d8Sk0/s1600/Antivirus.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="393" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJV8Wtk-C5fDWm_5v3XOauj2laXtsBFmeQoFOjiOdHWgBRLV0nj1WIQe6b5YXLMSZScMgS7LNZWCCbmDcWml73wZy0gvkoaZvFQPP2OK2dgRMAeAn_KHxF9YFp9qu2CJnb3pItX_d8Sk0/s640/Antivirus.jpg" width="640" /></a></div>
<br />
<br />
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Sounds interesting? <br /> Explore “Azure-Web-Apps-Antivirus” source code and instructions on Github.</span>
</a>
</div>
<br />
This article will only cover the <em>interesting parts</em> of the solution, if you have questions please do comment.<br />
<h3>
</h3>
<h3>
<br /></h3>
<h3>
AVFilter, File Upload/Download Filter</h3>
<br />
Filtering uploads and downloads is relatively straight forward thanks to IIS http modules, read more about them <a href="https://www.blogger.com/%5D(https://www.iis.net/learn/develop/runtime-extensibility/developing-iis-modules-and-handlers-with-the-net-framework)">here</a>.<br />
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Filtering Uploads</h4>
<br />
Here is an example of how you can detect file upload inside the http module:<br />
<pre class="brush: csharp"> private void Context_PreRequestHandlerExecute(object sender, EventArgs e)
{
HttpApplication httpApplication = (HttpApplication)sender;
HttpRequest httpRequest = httpApplication.Context.Request;
if (httpRequest.RequestType == WebRequestMethods.Http.Post)
{
if (httpRequest.Files.Count != 0)
{
//Check files
}
}
}
</pre>
<span style="font-size: x-small;">Take a look at the full implementation <a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus/blob/master/AVFilterIIS/Filter.cs">here</a>.</span><br />
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Filtering Downloads</h4>
<br />
When it comes to intercepting downloads it’s not as simple. Unfortunately outgoing traffic is written to the System.Web.HttpResponseStreamFilterSink and this stream is write only, so you can’t read it. So the only option that I and Google can think of is to proxy the outgoing stream. To make this happen I have created proxy class that changes write only steam to write/read stream, I have conveniently called it ReadWriteProxyStream, take a look at the implementation <a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus/blob/master/AVFilterIIS/ReadWriteProxyStream.cs">here</a>.<br />
<br />
Now that we have ReadWriteProxyStream class, we need to hook it up response filter, this is done when web request is first received:<br />
<pre class="brush: csharp"> private void Context_PreRequestHandlerExecute(object sender, EventArgs e)
{
HttpApplication httpApplication = (HttpApplication)sender;
httpApplication.Context.Response.Filter = new ReadWriteProxyStream(httpApplication.Context.Response.Filter);
}
</pre>
<span style="font-size: x-small;">Take a look at the full implementation <a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus/blob/master/AVFilterIIS/Filter.cs">here</a>.</span><br />
<br />
Before web response is returned back to user we need to check it, we can do this by reading the stream:<br />
<pre class="brush: csharp"> private void Context_PreSendRequestContent(object sender, EventArgs e)
{
HttpApplication httpApp = (HttpApplication)sender;
HttpResponse httpResponse = httpApp.Context.Response;
if(httpResponse.ContentType == "application/octet-stream")
{
ReadWriteProxyStream filter = httpResponse.Filter as ReadWriteProxyStream;
if (filter != null)
{
byte[] data = new byte[filter.Length];
filter.Position = 0;
filter.Read(data, 0, data.Length);
//Check files
}
}
}
</pre>
<span style="font-size: x-small;">Take a look at the full implementation <a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus/blob/master/AVFilterIIS/Filter.cs">here</a>.</span><br />
<h3>
</h3>
<h3>
<br /></h3>
<h3>
AVFileReceiver, Virus Scanning</h3>
<br />
For this proof of concept I have used Symantec virus scanner which comes with <a href="https://support.symantec.com/en_US/article.TECH104287.html">CLI</a>, and to test it I have used <a href="http://www.eicar.org/86-0-intended-use.html">EICAR test virus</a>. EICAR test virus is harmless, it can’t actually do anything to your machine.<br />
<br />
Symantec CLI will scan the specified file synchronously, so if the file is no longer there you know that it was not safe:<br />
<pre class="brush: csharp"> using (Process process = Process.Start(
"C:\\Program Files (x86)\\Symantec\\Symantec Endpoint Protection\\DoScan.exe",
"/ScanFile " + filePath))
{
process.WaitForExit();
}
if (!File.Exists(filePath))
throw new Exception("File was not safe!")
</pre>
<span style="font-size: x-small;">Take a look at the full implementation <a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus/blob/master/AVFileReceiver/Controllers/ReceiverController.cs">here</a>.</span><br />
<br />
<br />
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Azure-Web-Apps-Antivirus">
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" />
<span style="float: left; margin-left: 15px; margin-top: 20px;">Found this useful? <br /> Explore “Azure-Web-Apps-Antivirus” source code and instructions on Github.</span>
</a>
</div>
<h2>
</h2>
<h2>
<br /></h2>
<h2>
Final Thoughts</h2>
<br />
If you like this concept then please do share and rate this blog post. If this post will generate enough interest I will productionise this concept. However, if you just can’t wait, then here is what you need to do to make this production ready:<br />
<ul>
<li>Secure communication between AVFilter and AVFileReceiver</li>
<li>Secure AVFileReceiver access (authentication, network security group, etc)</li>
<li>Reduce VM maintenance through automation, use PowerShell files provided and place Cloud Service inside availability and upgrade group</li>
<li>Reduce memory consumption by intercepting binary responses only</li>
<li>Make debugging easier by logging exceptions</li>
<li>Improve performance by forwarding binary content using MTOM encoding to the AVFileReceiver and remove AVFilter temp storage</li>
<li>Improve scalability and responsiveness by using Async in the AVFileReceiver and AVFilter</li>
</ul>
<br />Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0tag:blogger.com,1999:blog-3391776337973860668.post-52537151494906287052016-12-20T03:00:00.003-08:002016-12-20T03:03:20.559-08:00Basic SaaS conversion guide, migrate your application to the Azure Cloud<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a></div>
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>A lot of organisations are currently converting their applications to SaaS. Unfortunately not much is written about “conversion & migration” (that I actually found useful), and I could have done with an article like this few months ago.<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>Objective of this article is to give you high level pragmatic overview of what you will need to think about, and do, when your company decides to convert / migrate your app to the Cloud.<br />
This article will expose the unknown unknowns, force you to rethink your assumptions, ask you some tough provoking questions and offer some tips so that you can give your company better estimates and an idea of what involved. <br />
<br />
Before we start, I would like to let you know that I was involved in the B2B SaaS conversion project, and migrated to Microsoft Azure. I will be drawing from my personal experience a lot, by doing so, hopefully I will be able to pass on some real world experience.<br />
<br />
<strong>This article is broken into two parts:</strong><br />
<ul>
<li>Business section is for executives and software architects, it introduces you to some of Cloud concepts, it’s suppose to make you think about the wider company, costs, why you are migrating and what’s going to be involved.</li>
<li>Technical section is for software architects and developers, it’s suppose to make you think about <strong>how</strong> you can go about the migration and what technical details you will need to consider. </li>
</ul>
<div>
<br /></div>
<h2 id="business-costs-why-and-impact">
Business: Costs, Why and Impact.</h2>
<div>
<br /></div>
<h3 id="1-understand-the-business-objectives">
1. Understand the business objectives</h3>
Converting your app to SaaS is not going to be cheap, it can take from few weeks to number of months (depending on the product size). So before you start working on this, understand why the business wants to migrate. <br />
<br />
<strong>Few good reasons to migrate:</strong> <br />
<ul>
<li>Removal of existing data center (cost saving) </li>
<li>International expansion (cost saving) </li>
<li>A lot of redundant compute that is used only during peak times (cost saving) </li>
<li>Cloud service provider provides additional services that can complement your product (innovation)</li>
</ul>
<br />
It’s vital that you understand the business objectives and deadlines, otherwise you will not be able to align your technical solution with the business requirements.<br />
<br />
<h3 id="2-understand-the-terminology">
2. Understand the terminology</h3>
<div>
<br /></div>
Here are some terms you need to get to know:<br />
<br />
<table style="text-align: left;">
<thead>
<tr>
<th style="border: 1px solid #cccccc;">Term</th>
<th style="border: 1px solid #cccccc;">Wikipedia Definition</th>
</tr>
</thead>
<tbody>
<tr>
<td style="border: 1px solid #cccccc;">Multi Tenancy</td>
<td style="border: 1px solid #cccccc;">The term “software multitenancy” refers to a software architecture in which a single instance of software runs on a server and serves multiple tenants. A tenant is a group of users who share a common access with specific privileges to the software instance.</td>
</tr>
<tr>
<td style="border: 1px solid #cccccc;">IaaS (Infrastructure as a Service)</td>
<td style="border: 1px solid #cccccc;">“Infrastructure as a Service (IaaS) is a form of cloud computing that provides virtualized computing resources over the Internet. IaaS is one of three main categories of cloud computing services, alongside Software as a Service (SaaS) and Platform as a Service (PaaS).”</td>
</tr>
<tr>
<td style="border: 1px solid #cccccc;">PaaS (Platform as a Service)</td>
<td style="border: 1px solid #cccccc;">“Platform as a service (PaaS) is a category of cloud computing services that provides a platform allowing customers to develop, run, and manage applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app.”</td>
</tr>
<tr>
<td style="border: 1px solid #cccccc;">SaaS (Software as a Service)</td>
<td style="border: 1px solid #cccccc;">Software as a service (SaaS; pronounced /sæs/) is a software licensing and delivery model in which software is licensed on a subscription basis and is centrally hosted. It is sometimes referred to as “on-demand software”. SaaS is typically accessed by users using a thin client via a web browser.</td>
</tr>
</tbody></table>
<br />
<div style="text-align: left;">
Here is a picture that explains IaaS, PaaS and SaaS very well:</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD_n_NTR5jeixoYGYOdB9LerREU_QCRNMiV02MRcs3-vHuhPpyUUwaE5M7GzUJX1MVIRrl2xMHhBY-InnVa1aTuptAR5zUUOHulynLgd3Nla10tYBqQ3a4-s3waGGoaxp3LsKcfhSOeYI/s1600/AA.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiD_n_NTR5jeixoYGYOdB9LerREU_QCRNMiV02MRcs3-vHuhPpyUUwaE5M7GzUJX1MVIRrl2xMHhBY-InnVa1aTuptAR5zUUOHulynLgd3Nla10tYBqQ3a4-s3waGGoaxp3LsKcfhSOeYI/s1600/AA.jpeg" /></a></div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<br /></div>
<br />
<h3 id="3-migration-approaches" style="text-align: left;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a>
3. Migration Approaches</h3>
<div style="text-align: left;">
<br /></div>
<table style="border: 1px solid #cccccc; text-align: left;">
<thead>
<tr>
<th style="border: 1px solid #cccccc;">Migration Approach</th>
<th style="border: 1px solid #cccccc;">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style="border: 1px solid #cccccc;"><a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a>Lift and shift</td>
<td style="border: 1px solid #cccccc;">Take your existing services and just migrate it to the IaaS (VMs), and in the future redesign your application for the Cloud.</td>
</tr>
<tr>
<td style="border: 1px solid #cccccc;">Multi tenancy</td>
<td style="border: 1px solid #cccccc;">Redesign your application so that it supports multi tenancy, however for now ignore other benefits. This migration approach might use IaaS with combination of PaaS. For example you might host your application in a VM, but use Azure SQL.</td>
</tr>
<tr>
<td style="border: 1px solid #cccccc;">Cloud native</td>
<td style="border: 1px solid #cccccc;">Redesign your application completely so that it’s Cloud native, it’s has built in multi tenancy and uses PaaS services. For example you will use Azure Web Apps, Azure SQL and Azure Storage.</td>
</tr>
</tbody></table>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
What approach you are going to take will depend on skills within the organisation, and what if any, products that you already have in the Cloud and how much time you have before you need to hit the deadline.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a>I am not going to focus on lift and shift in this article as it’s just taking what you already have and moving it to the Cloud, strictly speaking that is not SaaS, it is “Hosted” on the Cloud. </div>
<div style="text-align: left;">
<br /></div>
<h3 id="4-identify-technology-that-you-will-need" style="text-align: left;">
4. Identify technology that you will need</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Chances are your topology will be a combination of:<br />
<br /></div>
<table style="text-align: left;">
<thead>
<tr>
<th>Service</th>
<th></th>
<th>Technology Required</th>
</tr>
</thead>
<tbody>
<tr>
<td>Front-end</td>
<td>ASP.NET MVC, Angular, etc</td>
<td>IIS with .NET Framework</td>
</tr>
<tr>
<td>Back-end</td>
<td>Java, .NET, Node.JS, etc</td>
<td>Compute node that can host JVM, or WCF and can listen on a port</td>
</tr>
<tr>
<td>Persistence</td>
<td>SQL Server, MySQL, etc</td>
<td>SQL Server 2016, Azure Storage</td>
</tr>
<tr>
<td>Networking</td>
<td>Network segregation</td>
<td>Sub networks, VNET, Web.Config IP Security, Access Control List</td>
</tr>
</tbody></table>
<div style="text-align: left;">
<br /></div>
Understand what you actually need, don’t assume you need the same things. Challenge old assumptions, ask questions like:<br />
<div style="text-align: left;">
<ul>
<li>Do you really need all of that network segregation? </li>
<li>Do you really need to have front-end and back-end hosted on different machines? By hosting more services on a single machine you can increase compute density and save money. Need to scale? Scale the whole thing horizontally. </li>
<li>Do we really need to use relational data store for everything? What about NoSQL? By storing binary data in Azure Storage you will reduce Azure SQL compute <a href="https://docs.microsoft.com/en-us/azure/sql-database/sql-database-what-is-a-dtu">(DTUs)</a> which will save you a lot of money long term. </li>
<li>In this new world, do you still really need to support different on-prem platforms? SQL Server and MySQL? How about Linux and Windows? By not doing this, you can save on testing, hosting costs, etc.</li>
</ul>
</div>
<br />
<h3 id="5-scalability" style="text-align: left;">
5. Scalability</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The greatest thing about Cloud is scalability. You can scale effortlessly horizontally (if you architect your application correctly). How will your application respond to this? What is currently the bottleneck in your architecture? What will you need to scale? Back-end? Front-end? Both? What about persistence?</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Find out where the bottlenecks are now, so that when you redesign your application you can focus on them. Ideally you should be able to scale anything in your architecture horizontally, however time and budget might not allow you to do this, so you should focus on the bottlenecks. </div>
<div style="text-align: left;">
<br /></div>
<h3 id="6-service-level-agreements-sla-disaster-recovery" style="text-align: left;">
6. Service Level Agreements (SLA) & Disaster Recovery</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Services on the Cloud go down often, and often it’s fairly random what goes down, please do take a look at this <a href="https://azure.microsoft.com/en-us/status/history/">Azure Status History</a>. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The good news is that there is plenty of redundancy in the Cloud and you can just fail over or re-setup your services and you will be up and running again.</div>
<br />
<div style="text-align: left;">
<strong>Something to think about:</strong> </div>
<div style="text-align: left;">
<ul>
<li>What kind of SLA you will need, 99.9%? Is that per month or a year? </li>
<li>What kind of <a href="https://en.wikipedia.org/wiki/Recovery_point_objective">Recovery Point Objectives</a> will you need to reach? </li>
<li>What kind of <a href="https://en.wikipedia.org/wiki/Recovery_time_objective">Recovery Time Objectives</a></li>
<li>will you need to reach? </li>
<li>How much downtime did your prefered Cloud service provider has experienced recently? For example Azure had around 11 hours of downtime this year (2016). </li>
<li>If you store some data in relational database, some in NoSQL, some on the queue, when your services all go down, how are you going to restore your backups? </li>
<li>How will you handle data mismatch e.g. Azure SQL being out of sync with Azure Storage?</li>
</ul>
</div>
<br />
<div style="text-align: left;">
Higher SLA more likely you will need to move towards active-active architecture, but it’s complex, requires maintenance and lots of testing. Do you really need it? Will active-cold do?</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Please check out this great Microsoft article: <a href="https://docs.microsoft.com/en-gb/azure/guidance/guidance-resiliency-overview">Designing resilient applications for Azure</a>.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="7-security-operations" style="text-align: left;">
7. Security & Operations</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Security and Operations requirements are different on the Cloud. Tooling is not as mature, especially if you are looking to use PaaS services, and tooling that is available out of the box takes time to get used to. For example Azure <a href="https://azure.microsoft.com/en-gb/services/application-insights/">App Insights</a>, <a href="https://azure.microsoft.com/en-gb/services/security-center/">Security Center</a>, etc. </div>
<div style="text-align: left;">
Find out what security controls will need to be in place, find out what logs and telemetry ops team will need to do their job. </div>
<div style="text-align: left;">
<br /></div>
<h3 id="8-runtime-cost-model" style="text-align: left;">
8. Runtime cost model</h3>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Start putting together runtime cost model for different Cloud service providers. At this stage you know very little about these services, however you should start estimating the runtime costs. Cloud service providers normally provider tools to help you with this, for example Azure provides <a href="https://azure.microsoft.com/en-gb/pricing/calculator/">Azure Pricing Calculator</a>.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="9-staff-training" style="text-align: left;">
9. Staff Training</h3>
<div style="text-align: left;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><br /></div>
<div style="text-align: left;">
Now that you have a rough idea what service provider you will choose, start looking into training courses. Training can be official e.g. staff can get <a href="https://www.microsoft.com/en-us/learning/mcsa-cloud-platform-certification.aspx">Cloud Platform MCSA</a> certification, or it can be more unofficial. However, no matter what you do, I encourage you to give your staff time to play around with different services, put together few PoCs, read few books e.g. <a href="https://www.amazon.co.uk/70-532-Developing-Microsoft-Azure-Solutions/dp/0735697043">Developing Microsoft Azure Solututions</a>, watch some Azure videos e.g. <a href="https://azure.microsoft.com/en-gb/resources/videos/azure-friday/">Azure Friday</a>.</div>
<div style="text-align: left;">
This training will require some time, if you haven’t got time, consider consultancy.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="10-consultancy" style="text-align: left;">
10. Consultancy</h3>
<br />
<div style="text-align: left;">
If you would like to get some help there are number of things you can do with Azure: </div>
<div style="text-align: left;">
<ul>
<li>Hire Azure migration consultants </li>
<li>Contact “Microsoft Developer eXperience” team, they might be able to help </li>
<li>you with migration by investing in your project (free consultation & PoCs). </li>
<li>Sign up for Microsoft “Premier Support for Developers”, you will get a dedicated Application Development Manager who will be able to assist you and guide you and your team.</li>
</ul>
</div>
<br />
<div style="text-align: left;">
I strongly advise you to grow talent internally and not solely rely on consultants. Hybrid approach works best, get some consultancy and get them to coach your internal team.</div>
<div style="text-align: left;">
<br /></div>
<h2 id="technical-how-are-you-going-to-do-it" style="text-align: left;">
Technical: How are you going to do it?</h2>
<div style="text-align: left;">
<br /></div>
<h3 id="1-multi-tenancy-data-segregation" style="text-align: left;">
1. Multi tenancy data segregation</h3>
<br />
<div style="text-align: left;">
Hosted applications tend to be installed on a machine and used by one company, however SaaS applications are installed and used by lots of different companies. With sharing comes great responsibility, there are number of ways that you can handle multi tenancy data segregation: </div>
<div style="text-align: left;">
<ul>
<li>Shared Database, Shared Schema (Foreign Keys partition approach) </li>
<li>Shared Database, Separate Schema </li>
<li>Separate Database</li>
</ul>
</div>
<br />
<div style="text-align: left;">
What approach you go for depends on your requirements. Microsoft has written an article <a href="https://msdn.microsoft.com/en-us/library/aa479086.aspx">Multi-Tenant Data Architecture</a>, please read it.</div>
<br />
<div style="text-align: left;">
<b>What is my take? </b>If your app is very simple (To Do List, Basic document store, etc), go for Foreign Key partition. If you are app is more complicated e.g. finance or medical system I would go for the separate database / separate persistence. Separate persistence will reduce the chance of data cross tenant data leak, make tenant deletion simpler and more importantly it will makes ops easier, you can upgrade their persistence individually, you can back them up more often if you need to, and most likely you will not need to worry about sharding. However, separate persistence approach is not simple to implement, it requires a lot of development.</div>
<br />
<h3 id="2-login-experience" style="text-align: left;">
2. Login Experience</h3>
<br />
<div style="text-align: left;">
Now that users access your application there are number of ways for them to access their </div>
<div style="text-align: left;">
tenant area: </div>
<div style="text-align: left;">
<ul>
<li>Dedicated URL </li>
<li>Recognise Possible Tenants On Login</li>
</ul>
</div>
<br />
<div style="text-align: left;">
On the Cloud user experience will need to change, because many businesses are using it </div>
<div style="text-align: left;">
at the same time you need to somehow know who is who. There are number of ways you can do this:</div>
<br />
<div style="text-align: left;">
<strong>Dedicated URL</strong></div>
<div style="text-align: left;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>You can give your tenants dedicated URL so that they can login into their app. For example you can give them access to: yourapp.com/customername or customername.yourapp.com.</div>
<br />
<div style="text-align: left;">
However, this approach will require you to send out an email to the tenant, informing them </div>
<div style="text-align: left;">
what their URL is. If they forget the URL they will end up contacting your support team.</div>
<br />
<div style="text-align: left;">
<strong>Recognise Possible Tenants On Login</strong></div>
<div style="text-align: left;">
Tenant goes to yourapp.com and logs in. When they login they presented with possible tenants that they can access.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW3IxbSR9qN_Rg8yhYp1NX5r9XO3EPfsz62lL_uhletGujbAnSu7lCAtx1X-bMvNYQlnzIXvIiAbuouwXf0dXujUrEsiGUICy15IX-Sm7a9SFmaf8oFKueXTD6sqNIPIcehOviuoZBicU/s1600/RPTOL.png" imageanchor="1"><img border="0" height="408" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjW3IxbSR9qN_Rg8yhYp1NX5r9XO3EPfsz62lL_uhletGujbAnSu7lCAtx1X-bMvNYQlnzIXvIiAbuouwXf0dXujUrEsiGUICy15IX-Sm7a9SFmaf8oFKueXTD6sqNIPIcehOviuoZBicU/s640/RPTOL.png" width="640" /></a></div>
<div style="text-align: left;">
<br /></div>
<br />
<div style="text-align: left;">
With this approach tenants don’t need to remember their URL, however you need to introduce extra step before they login and you need to scan all possible tenants to see if the logged in user exists. This is more work, not to mention, what if your customer wants to use AzureAD? How do you know which AzureAD should be used? Now you will need to introduce mapping tables, extra screens, etc.</div>
<br />
<h3 id="3-application-state-is-your-application-already-stateless" style="text-align: left;">
3. Application state, is your application already stateless?</h3>
<div style="text-align: left;">
<br />
If you need your application to scale out then your compute instances should not keep any state in memory or disk. Your application needs to be stateless. </div>
<br />
<div style="text-align: left;">
However, if temporarily you need to keep some basic state, like session state then you are in luck, Microsoft Azure allows you to use ARR (Application Request Routing) so that all client requests get forwarded to the same instance every time. This should be a temporary solution as you will end up overloading single instance as it will take a while for cookies to expire and spread the load to other instances.</div>
<br />
<h3 id="4-what-cloud-services-should-you-use" style="text-align: left;">
4. What cloud services should you use?</h3>
<div style="text-align: left;">
<br />
This largely comes down to the security and performance requirements. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>Azure Web Apps are great, however they don’t support VNETs, which means you can’t stop OSI level 3 traffic to Front-end and Back-end. However you can use Web.Config IP Security to restrict access to front-end, back-end and Azure SQL supports Access Control List (ACL). Azure Web Apps also don’t scale very well vertically, so if you have lots of heavy compute it might make more sense economically for you to use Service Fabric or Cloud Services so that you can scale vertically and horizontally.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
However, Azure Web Apps require no maintenance, they are easy to use, they scale almost immediately and they are very cheap. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<b>What is my take? </b>Always go with PaaS services and avoid IaaS as much as you can. Let Azure do all of the operating system and service patching / maintenance. Go to IaaS only if you have no other choice.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="5-telemetry-logging" style="text-align: left;">
5. Telemetry & Logging</h3>
<br />
<div style="text-align: left;">
If you are going to use pure PaaS services then most like you will not be able to use your traditional ops tools for telemetry. The good news is that ops tooling on the Cloud is getting better.</div>
<br />
<div style="text-align: left;">
<b>Azure has a tool called App Insights it allows you to: </b></div>
<div style="text-align: left;">
<ul>
<li>Store / view your application logs (info, warnings, exceptions, etc). You will need to change your logging appender to make this work. </li>
<li>Analyse availability </li>
<li>Analyse app dependencies </li>
<li>Analyse performance </li>
<li>Analyse failures </li>
<li>Analyze CPU / RAM / IO (IaaS only)</li>
</ul>
</div>
<br />
<div style="text-align: left;">
Also there is a tool called Azure Security Center, it works very well if you have VMs in your subscription, it will guide you and give you best practices. It also has built in machine learning so it analyses your infrastructure and if it will see something unusual it will flag it up and and notify your ops team. It’s very handy.</div>
<br />
<h3 id="6-always-test-with-many-instances" style="text-align: left;">
6. Always test with many instances</h3>
<br />
<div style="text-align: left;">
You should be running your application with at least 2 compute instances. Last thing you want to happen is to come to the release date and scale out and find out that your application is not scaling. This means anything that can be scaled horizontally should be scaled so you are always testing.</div>
<br />
<h3 id="7-authentication" style="text-align: left;">
7. Authentication</h3>
<div style="text-align: left;">
<br />
Multi tenancy makes authentication hard, it makes it even harder if your application is B2B. Why? Well because all businesses have different authentication requirements, they might have AD and they want to integrate it with your app and enable Single Sign On, they might be a hip start-up and they will want to use Facebook to login into your app. This should not be overlooked, you might need to find open source authentication server or use service such as AzureAD. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
<b>What is my take?</b> Find an open source authentication server and integrate with it. Avoid AzureAD unless your customers pay for it. AzureAD is very expensive, basic tier cost around £0.60 per user and Premium 2 tier costs around £5.00 per user!</div>
<div style="text-align: left;">
<br /></div>
<h3 id="8-global-config" style="text-align: left;">
8. Global Config</h3>
<br />
<div style="text-align: left;">
Hosted applications tend to be installed, so they come with installers. These installers change app keys in the config. For example if your app integrates with Google Maps your installer might ask your customer to put in the Google Map app keys to enable Google Maps features. Now in the Cloud world this is very different, customers don’t care about this, they just want to login and use your software. So these keys are going to be the same for everyone, this means that most of these keys will need to be migrated out of the database and config files to App Settings (Environment variables).</div>
<br />
<h3 id="9-relational-database-upgrades" style="text-align: left;">
9. Relational Database Upgrades</h3>
<div style="text-align: left;">
<br />
After all these years the hardest things to update are still databases. This is especially the case if you go for the database per tenant approach. You release your software, services are now running using the new version, now you decide to upgrade all database schemas, what if one of them fails? Do you roll back the entire release? When do you upgrade the databases? Do you ping every tenant and upgrade them? Do you do it when they login?</div>
<br />
<div style="text-align: left;">
There are some tools that can help you with database upgrades: </div>
<div style="text-align: left;">
<ul>
<li><a href="https://flywaydb.org/">Flyway (Java)</a></li>
<li><a href="https://dbup.github.io/">DB Up (.NET)</a></li>
</ul>
</div>
<br />
<div style="text-align: left;">
These tools automatically patch your databases to the correct version level. </div>
<br />
<div style="text-align: left;">
However how you roll back, hot fix and upgrade your tenants is still up to you and your organisation.</div>
<br />
<div style="text-align: left;">
<b>What is my take?</b> Use tools like Flyway, <a href="http://martinfowler.com/articles/evodb.html">Evolutionary Database Design</a> approach and roll forward only. Find a problem, fix it and release the hotfix.</div>
<br />
<h3 id="10-encryption" style="text-align: left;">
10. Encryption</h3>
<br />
<div style="text-align: left;">
As you are storing your customers data in someone else’s data center chances are you will need to rereview your encryption strategy. Cloud providers are aware of this and to make your life easier they have released some features to assist you, for example here are few great encryption features from Azure: </div>
<div style="text-align: left;">
<ul>
<li>Encryption At Rest (<a href="https://msdn.microsoft.com/en-us/library/dn948096.aspx">Azure SQL</a>, <a href="https://docs.microsoft.com/en-us/azure/storage/storage-service-encryption">Azure Storage </a>) </li>
<li><a href="https://msdn.microsoft.com/en-GB/library/mt163865.aspx">Always Encrypted for Azure SQL Database and SQL Server</a></li>
<li><a href="https://msdn.microsoft.com/en-us/library/dn765131.aspx">Row Level Security for Azure SQL Database and SQL Server</a></li>
</ul>
</div>
<br />
<div style="text-align: left;">
Once again, you need to rereview your requirements and think about what will work for you and your organisation. You don’t want to encrypt everything using Always Encrypted because you will not be able to do partial searches and your application’s performance will be impacted.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Please be aware that Always Encrypted feature will most likely not prevent SQL Injection attack, and you might not be able to use some of the new features from Microsoft if you use this technology, for example Temporal Tables, they currently don’t support Always Encrypted. </div>
<div style="text-align: left;">
<br /></div>
<h3 id="11-key-store" style="text-align: left;">
11. Key Store</h3>
<div style="text-align: left;">
<br />
If you are using a key store, you will need to update it to support multi tenancy.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.blogger.com/blogger.g?blogID=3391776337973860668" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a></div>
<h3 id="12-fault-tolerance" style="text-align: left;">
12. Fault tolerance</h3>
<div style="text-align: left;">
<br />
There is a lot of noise out there about fault tolerance on the Cloud, you need to be aware </div>
<div style="text-align: left;">
of these three things: </div>
<div style="text-align: left;">
<ul>
<li>Integration error handling </li>
<li>Retry strategies </li>
<li>Circuit Breaker</li>
</ul>
</div>
<br />
<div style="text-align: left;">
<strong>Integration Error Handling</strong></div>
<br />
<div style="text-align: left;">
I throughly recommend that you do integration error handling failure analysis. </div>
<div style="text-align: left;">
If cache goes down, what happens? If Azure Service Bus goes down what will happen? It’s important that you design your app with failure in mind. So if cache goes down your app still works, it will just run a bit slower. If Azure Service Bus goes down, your app will continue to work but messages will not get processed.</div>
<br />
<div style="text-align: left;">
<strong>Retry Strategy (Optional)</strong></div>
<div style="text-align: left;">
Retry strategy can help you with transient failure. In the early cloud days I’ve been told that these errors were common due to the infrastructure instability and noisy neighbour issues. These errors are a lot less common these days, however they still do happen. Good news is that Azure Storage SDK comes with retry strategy by default and you can use ReliableSqlConnection instead of SqlConnection to get the retry strategy when you connect to Azure SQL.</div>
<br />
<div style="text-align: left;">
<i>A custom retry policy was used to implement exponential backoff while also logging each </i></div>
<div style="text-align: left;">
<i>retry attempt. During the upload of nearly a million files averaging four mb in size around 1.8% of files required at least one try</i> - Cloud Architecture Patterns - Using Microsoft Azure</div>
<br />
<div style="text-align: left;">
You don’t have to implement this when you migrate, you can accept that some of the users might get a transient failure. As long as you have integration error handling in place and you keep your app transactionally consistent your users can just retry manually.</div>
<br />
<div style="text-align: left;">
<strong>Circuit breaker (Optional)</strong></div>
<div style="text-align: left;">
<a href="http://martinfowler.com/bliki/CircuitBreaker.html">Martin Fowler Circuit Breaker</a></div>
<div style="text-align: left;">
<br /></div>
<h3 id="13-say-goodbye-to-two-phase-commit" style="text-align: left;">
13. Say goodbye to two phase commit</h3>
<div style="text-align: left;">
<br />
On the PaaS Cloud you can’t have two phase commmit, it’s a concious <a href="https://blogs.msdn.microsoft.com/rickatmicrosoft/2013/01/03/the-cap-theorem-why-everything-is-different-with-the-cloud-and-internet/">CAP theorem trade off</a>. This means no two phase commit for .NET or Java or whatever. This is not an Azure limitation.</div>
<div style="text-align: left;">
Please take a second and think about how this will impact your application. </div>
<div style="text-align: left;">
<br /></div>
<h3 id="14-point-in-time-backups" style="text-align: left;">
14. Point in time backups</h3>
<div style="text-align: left;">
<br />
Azure SQL comes with point in time backup, however Azure Storage doesn’t have point in time backup. So you either need to implement your own or use tool such as <a href="https://www.cherrysafe.com/">CherrySafe</a>.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="15-accept-technical-debt" style="text-align: left;">
15. Accept Technical Debt</h3>
<br />
<div style="text-align: left;">
During the migration you will find more and more things that you would like to improve. </div>
<div style="text-align: left;">
However, the business reality is going to be that you will need to accept some technical debt. This is a good thing, you need to get product out there, see how it behaves and focus on the areas that actually need improvement. Make a list of things that you want to improve, get the most important things done before you ship, and during the subsequent releases just remove as much technical debt as you can.</div>
<br />
<div style="text-align: left;">
<strong>What you think needs rewriting ASAP might change, as you convert your application you will find bigger more pressing things that you must improve, so don’t focus and commit to fixing the trivial things prematurely.</strong></div>
<div style="text-align: left;">
<strong><br /></strong></div>
<h3 id="16-new-persistence-options-optional" style="text-align: left;">
16. New Persistence Options (Optional)</h3>
<br />
<div style="text-align: left;">
As you migrate you should seriously think about persistence. Ask yourself the following questions: </div>
<div style="text-align: left;">
<ul>
<li>Do you really need to store everything in the relational database? </li>
<li>Can you store binary files in the Azure Storage? </li>
<li>What about JSON documents? Why not store them the DocumentDB?</li>
</ul>
</div>
<br />
<div style="text-align: left;">
This will increase migration complexity, so you might want to delay this and do this in the future. That’s understandable. But just so you know there are lots of reasons why you should do this.</div>
<br />
<div style="text-align: left;">
Azure SQL is not cheap and DTUs are limited. Less you talk to the Azure SQL the better. Streaming binary files to and from Azure SQL is expensive, it takes up storage space and DTUs, so why do it? Azure Storage is very cheap and it comes with Read-Access Geo Redundant Storage, so incase of a DR you can failover.</div>
<br />
<h3 id="17-monolith-service-based-or-microservice-architecture-optional" style="text-align: left;">
17. Monolith, Service-based or Microservice Architecture (Optional)</h3>
<div style="text-align: left;">
If you are migrating I will say this, don’t attempt at the same time to migrate over to Microservices. Do one thing, just get your application to the Cloud, enable multi tenancy, give business some value. After you have achieved your conversion milestone I would consider breaking your app down into smaller deployment modules. </div>
<br />
<div style="text-align: left;">
Why? Because it’s complex, here are just a few things that you will need to think about: </div>
<div style="text-align: left;">
<ul>
<li>Data Ownership - How do different modules share data? Who owns the actual data? </li>
<li>Integration - How are you going to integrate module A and B? </li>
<li>API to API communication? </li>
<li>Queues? </li>
<li>Database Integration? </li>
<li>Testing, how are you going to test the integration? Are you going to automate it? </li>
<li>Scalability - How are you going to scale different modules? </li>
<li>Transactions - In the Cloud there is no two phase commit, how are going to work around this?</li>
</ul>
</div>
<br />
<h3 id="18-infrastructure-as-code-optional" style="text-align: left;">
18. Infrastructure As Code (Optional)</h3>
<div style="text-align: left;">
Now that you are going to the Cloud you need to look after one more thing. Your infrastructure provisioning will now be a config file. Someone will need to maintain this. </div>
<br />
<div style="text-align: left;">
If you are using Azure you will define your infrastructure using Azure Resource Manager and PowerShell.</div>
<br />
<div style="text-align: left;">
This is optional because you could set up your infrastructure manually (I don’t recommend this).</div>
<div style="text-align: left;">
<br /></div>
<h3 id="19-gradual-release-optional" style="text-align: left;">
19. Gradual Release (Optional)</h3>
<br />
<div style="text-align: left;">
If you deploy your SaaS app and something goes wrong you will not want to take all of your customers down at the same time. This means you should consider having several installations. For example, you can have different URLs for different installations app.yourdomain.com and app2.yourapp.com. So when you deploy to app.yourdomain.com, if all goes well then you can promote your app to app2.yourapp.com. This is optional because you could just deploy out of hours to app.yourapp.com.</div>
<div style="text-align: left;">
<br /></div>
<h3 id="20-deployment-pipeline-optional" style="text-align: left;">
20. Deployment Pipeline (Optional)</h3>
<br />
<div style="text-align: left;">
You will need a <a href="http://martinfowler.com/bliki/DeploymentPipeline.html">deployment pipeline</a> if you want to have confidence in your deployments. Don’t underestimate this step, depending on what services you are going to use and how complex your pipeline is, it can take weeks to set this up. Chances are you will need to change your config files, transform them for different installations, tokenize them. Deploy to staging slots and VIP switch the production, add error handling, establish promotion process, etc.</div>
<br />
<div style="text-align: left;">
There are plenty of tools that can help you with this, I’ve been using Visual Studio Team Services to orchestrate builds and deployments. </div>
<div style="text-align: left;">
<br /></div>
<h3 id="21-avoid-vendor-lock-in-optional" style="text-align: left;">
21. Avoid Vendor Lock In (Optional)</h3>
<br />
<div style="text-align: left;">
Build your application in a such way that you can migrate with relative ease to another cloud provider. I know that no matter what you do, it will not be an easy journey, however, if you actually think about it and design with it in mind, it might take 1-3 months to migrate from Azure to AWS and not 1 year. How? When you redesign your application abstract infrastructure components out so there are no direct dependencies on Azure and Azure SDKs.</div>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com1tag:blogger.com,1999:blog-3391776337973860668.post-12614457598832253642016-12-10T15:21:00.000-08:002016-12-10T15:24:12.605-08:00Azure Web App Application InitializationIf you have an Azure Web App that needs initializing before it starts receiving traffic
and just calling the web server root "/" is not good enough (default behaviour) then you should be looking into
custom "applicationInitialization". Before you read this article please take a look at the
<a href="https://www.iis.net/configreference/system.webserver/applicationinitialization">IIS Application Initialization</a> documentation.<br />
<br />
This article will cover 9 things that are not
mentioned in the documentation.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2A57BQdebf-AswLVaaQsne7CSf6m_LrnFf7loOwtSqfFwxhgKmu1CTMF8CNdj1XEYEi2REBAGwq48Csrd2VyqHjnVRCMFjRjfMurzfGAmIfHowB2pdnah2wBVsK8NQYqNVkPzKVO1htA/s1600/webapp-logo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2A57BQdebf-AswLVaaQsne7CSf6m_LrnFf7loOwtSqfFwxhgKmu1CTMF8CNdj1XEYEi2REBAGwq48Csrd2VyqHjnVRCMFjRjfMurzfGAmIfHowB2pdnah2wBVsK8NQYqNVkPzKVO1htA/s400/webapp-logo.png" /></a></div>
<br />
<br />
This is how the web.config applicationInitialization XML config can look like:<br />
<pre><?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
<add initializationPage="/init1.aspx" hostName="somewebapp.azurewebsites.net"/>
<add initializationPage="/init2.aspx" hostName="somewebapp.azurewebsites.net"/>
</applicationInitialization>
</system.webServer>
</configuration>
</pre>
This config should be placed in the wwwroot.<br />
<br />
<h2 id="whattheydontmentioninthedocumentation">
What they don't mention in the documentation</h2>
<div>
<br /></div>
<h3 id="1youcanhavemanyinitializationpages">
1. You can have many initialization pages</h3>
As you can see in the example above you can actually initialize many pages e.g. init1.aspx, init2.aspx.<br />
<br />
<h3 id="2initializationsaredoneinorder">
2. Initializations are done in order</h3>
As per above example, init1.aspx will be called first, then as soon as init1.aspx responds or times out it will move on to init2.aspx<br />
<br />
<h3 id="3initializationsareresponseignorant">
3. Initializations are response ignorant</h3>
ApplicationInitilization doesn't care if init1.aspx returns 200, 404 or 503.
As long as it returns something it will consider it's job done.<br />
This means if your init1.aspx page times out, IIS will still think that you are good to go. <br />
<br />
<h3 id="4canthaveduplicateinitializationpage">
4. Can't have duplicate initialization page</h3>
You can't have duplicate keys, such as:<br />
<pre><?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
<add initializationPage="/init1.aspx" hostName="somewebapp.azurewebsites.net"/>
<add initializationPage="/init1.aspx" hostName="somewebapp.azurewebsites.net"/>
</applicationInitialization>
</system.webServer>
</configuration>
</pre>
This will make IIS error. You might think, why on earth would you want to have duplicate keys?
This could be considered a workaround for the timeouts, if init1.aspx takes too long to respond, call it again,
next time hopefully it will respond immediately. To workaround this you could just add query string:<br />
<pre><?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
<add initializationPage="/init1.aspx?call=1" hostName="somewebapp.azurewebsites.net"/>
<add initializationPage="/init1.aspx?call=2" hostName="somewebapp.azurewebsites.net"/>
</applicationInitialization>
</system.webServer>
</configuration>
</pre>
<h3 id="5initializationsaredoneinternally">
5. Initializations are done internally</h3>
IIS applicationInitialization requests are done internally so you will not see them in your W3C logs.
However if your init1.aspx page looks at the request and takes a look at the User Agent it will see the following:
"IIS Application Initialization Preload"<br />
<br />
<h3 id="6noclientstateissharedbetweencalls">
6. No client state is shared between calls</h3>
When IIS calls init1.aspx and then calls init2.aspx there is no common client state kept i.e. cookies, etc.
This means you can't tag the IIS applicationInitialization client.<br />
<br />
<h3 id="7initializationcanwarmupvirtualapplications">
7. Initialization can warm up virtual applications</h3>
<div>
<br /></div>
<h3 id="8hostnameisthesameforallslots">
8. hostName is the same for all slots</h3>
This has really surprised me, one Web App can have many slots, each slot has it's own name,
however you when you use initializationPage you can just specify the main slots hostName and this will work
for all slots. So for example, if you have production slot and you have staging slot you might consider doing something like this:<br />
Production Web.Confg <br />
<pre><?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
<add initializationPage="/init1.aspx?call=1" hostName="somewebapp.azurewebsites.net"/>
<add initializationPage="/init1.aspx?call=2" hostName="somewebapp.azurewebsites.net"/>
</applicationInitialization>
</system.webServer>
</configuration>
</pre>
Staging Web.Confg <br />
<pre><?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
<add initializationPage="/init1.aspx?call=1" hostName="somewebapp-staging.azurewebsites.net"/>
<add initializationPage="/init1.aspx?call=2" hostName="somewebapp-staging.azurewebsites.net"/>
</applicationInitialization>
</system.webServer>
</configuration>
</pre>
No need, this will work for both:<br />
<pre><?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<applicationInitialization doAppInitAfterRestart="true" skipManagedModules="true">
<add initializationPage="/init1.aspx?call=1" hostName="somewebapp.azurewebsites.net"/>
<add initializationPage="/init1.aspx?call=2" hostName="somewebapp.azurewebsites.net"/>
</applicationInitialization>
</system.webServer>
</configuration>
</pre>
This is good news, as you will need to perform less transformations. <br />
<br />
<h3 id="9loadbalancer">
9. Load Balancer</h3>
This might be one of the most important points.
IIS applicationInitialization and Azure Load Balancers work together.
As soon as IIS initializes all of the pages, internal Azure Web Apps load balancer is notified
and it starts sending traffic to the fully initialized instance.
So you can use applicationInitialization to warm up your application completely before you allow traffic to come in.<br />
<br />
<h2 id="conclusion">
Conclusion</h2>
If you have a large app and you need to invoke many different areas to warm up
caches, compilations, etc before your instance starts receiving traffic then IIS applicationInitialization is a good way to go!
<br />
<br />
<em>As I don't work for Microsoft I can't guarantee that my description of Azure Web App inner workings is 100% accurate.</em>Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com5tag:blogger.com,1999:blog-3391776337973860668.post-10494379453820704542016-08-29T06:47:00.000-07:002016-09-05T14:00:18.484-07:00Applied Domain-Driven Design (DDD) - Event Logging & Sourcing For AuditingIn this article I am going to explore the use of Event Logging and Sourcing as a solution for domain auditing. This article is not going to explore how to use Event Sourcing to obtain the current model state.<br />
<br />
<h3>
What is Event Logging?</h3>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg34rFVH_CAe0-1WLmZfsnE3G2FeE4prJ3PJUHZPiqJksLC3lcdpnlMu1wz4fJ6Q7qDclwNUvAX_IsSeafOJAcesFsq2s6wx2775i9eWrDkwJROPLGtqk_Up8OZgyloYOk7Y6giyqTF4wE/s1600/log-file-location-linux.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg34rFVH_CAe0-1WLmZfsnE3G2FeE4prJ3PJUHZPiqJksLC3lcdpnlMu1wz4fJ6Q7qDclwNUvAX_IsSeafOJAcesFsq2s6wx2775i9eWrDkwJROPLGtqk_Up8OZgyloYOk7Y6giyqTF4wE/s200/log-file-location-linux.png" width="200" /></a></div>
<br />
<br />
<br />
In my previous article I've <a href="http://www.zankavtaskin.com/2013/09/applied-domain-driven-design-ddd-part-2.html" target="_blank">explored domain events</a>. In that article they were synchronous unpersisted events. Aggregate root or Service would just raise an event and a handler would handle it. In this article we are going to change that, we are going to persist these domain events.<br />
<br />
<br />
<br />
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<h3>
</h3>
<h3>
</h3>
<h3>
<br /></h3>
<h3>
What is Event Sourcing? </h3>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8o-_ik1zGFqM7vHgDv7Ttn3CvPRV9LhhkNSyPr-lHu2pf8Ynr2RlIOV0WnnDMIkqi1ucr2x36tQQz7Exaa6-mFaCk_NKEPikhyphenhyphenevNEunga45Sq5ELyPlqbtiCq0h0QYi2XIT6PtUSZYM/s1600/Database_4.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8o-_ik1zGFqM7vHgDv7Ttn3CvPRV9LhhkNSyPr-lHu2pf8Ynr2RlIOV0WnnDMIkqi1ucr2x36tQQz7Exaa6-mFaCk_NKEPikhyphenhyphenevNEunga45Sq5ELyPlqbtiCq0h0QYi2XIT6PtUSZYM/s200/Database_4.png" width="200" /></a></div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">"Append-only store to record the full series of events that describe actions taken on data in a domain, rather than storing just the current state, so that the store can be used to materialize the domain objects. This pattern can simplify tasks in complex domains by avoiding the requirement to synchronize the data model and the business domain; improve performance, scalability, and responsiveness; provide consistency for transactional data; and maintain full audit trails and history that may enable compensating actions." - <a href="https://msdn.microsoft.com/en-gb/library/dn589792.aspx" rel="nofollow" rev="en_rl_minimal">Event Sourcing Pattern Microsoft</a></span><br />
<!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
</div>
<div>
<br /></div>
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
Requirements domain Event Logging and Sourcing can fulfil:</h3>
</div>
<div>
<div>
<br /></div>
<ul>
<li><span style="font-weight: normal;"><span style="font-family: inherit;">As a technical support member of staff I would like to be able to view audit log so that I can find out what my customers did i.e. did they get themselves in to a mess or is it that our software is buggy?</span></span></li>
<li><span style="font-weight: normal;"><span style="font-family: inherit;">As a system admin I would like to be able to view the audit log so that I can find out what my users are doing i.e. someone is not sure why something was changed, software admin needs to double check what happened. </span></span></li>
<li><span style="font-weight: normal;"><span style="font-family: inherit;">As a security analyst I would like to view audit log so that I can find out who has committed fraud. </span></span></li>
<li><span style="font-weight: normal;"><span style="font-family: inherit;">As a business expert I would like to find out how long it has taken someone to go through a process so that we can optimise it. </span></span></li>
<li>As a security analyst I would like audit log to be immutable so that no one can tamper with it </li>
<li>As a software engineer I would like to see what user has done so that I can re-produce their steps and debug the application. </li>
<li>As a software engineer I would like persisted domain events to be forwarded to the queue as we can't have 2 phase commit in the Cloud.</li>
</ul>
<br />
<br /></div>
<h3>
Why not just use CQRS with Event Sourcing? </h3>
<div>
<br /></div>
As it was mentioned by<a href="http://udidahan.com/2011/04/22/when-to-avoid-cqrs/" rel="nofollow" target="_blank"> Udi, CQRS is a pattern that should be used where data changes are competitive or collaborative</a>. A lot of systems don't fall in to this category, even if they do, you would only use CQRS potentially with Event Sourcing (CQRS != Event Sourcing) for a <a href="https://www.infoq.com/news/2016/04/event-sourcing-anti-pattern" rel="nofollow" target="_blank">part of the application and not everywhere</a>. This means you can't have automatic audit for your entire system by using CQRS with Event Sourcing.<br />
<br />
Event Sourcing is all about storing events and then sourcing them to derive the current model state.<br />
If you don't need "undo" and "replay" functionality, and if you don't need to meet super high scalability non-functional requirements (which most likely you don't) why over-engineer?<br />
<br />
This proposed solution is just logging events to get some of the benefits that Event Sourcing provides without the deriving the current model state. However, it will still be sourcing the events to obtain the audit log.<br />
<br />
<div>
<h3>
</h3>
<h3>
<br /></h3>
<h3>
Why is this a good solution for auditing? </h3>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-weight: normal;"><span style="font-family: inherit; font-size: small;">Your domain is rich and full of domain events (<a href="http://cqrs.nu/Faq/commands-and-events" rel="nofollow" target="_blank">domain event is something that has happened, it's an immutable fact and you are just broadcasting it</a>). It's also written using ubiquitous language. Because it describes what has happened and what was changed it's a great candidate to</span></span><span style="font-family: inherit;"> meet your <a href="http://martinfowler.com/eaaDev/EventSourcing.html" rel="nofollow" target="_blank">auditing, troubleshooting, debugging </a>and 2 phase commit Cloud requirements. </span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<span style="font-size: small;"><b>Pros:</b></span></div>
<ul>
<li><span style="font-size: small;"><span style="font-weight: normal;">It's fairly easy to create audit read model from domain events </span></span></li>
<li><span style="font-size: small;"><span style="font-weight: normal;">Domain events provide business context of what has happened and what has changed </span></span></li>
<li><span style="font-size: small;"><span style="font-weight: normal;">Reference data (Mr, Dr, etc) is stored in the same place so you can provide full audit read model </span></span></li>
<li><span style="font-size: small;"><span style="font-weight: normal;">Events can be written away to append only store </span></span></li>
<li><span style="font-size: small;"><span style="font-weight: normal;">Only useful event data is stored </span></span></li>
</ul>
<span style="font-weight: normal;"><span style="font-size: small;"><br /></span></span><span style="font-size: small;"><b>Cons:</b></span><br />
<ul>
<li>Every request (command) must result in domain event and you need to flatten it, it's more development work</li>
<li>Requires testing </li>
<li><span style="font-size: small; font-weight: normal;">Duplication of data. One dataset for current state. Second dataset for events. There might be mismatch due to bugs and changes. </span></li>
</ul>
<br />
<div>
</div>
<h3>
</h3>
<h3>
<span style="font-family: inherit;">
What about "proof of correctness"? </span></h3>
<h3>
<div style="font-weight: normal;">
<span style="font-family: inherit; font-size: small;">Udi, has already <a href="http://udidahan.com/2011/04/22/when-to-avoid-cqrs/" rel="nofollow" target="_blank">discussed this here</a> (scroll down to the "proof of correctness").</span><br />
<span style="font-family: inherit; font-size: small;"><br /></span>
<span style="font-family: inherit; font-size: small;">I recommend that you keep your storage transaction logs, it doesn't give you proof of correctness however it gives you extra protection. If someone bypasses your application and tampers with your data in the database at least it will be logged and you will be able to do something about it.</span><br />
<span style="font-weight: normal;"><br /></span>
<span style="font-weight: normal;"><br /></span></div>
</h3>
<h3>
Domain event logging implementation example </h3>
<div>
<br /></div>
<div>
I am going to take my <a href="http://www.zankavtaskin.com/2013/09/applied-domain-driven-design-ddd-part-2.html" target="_blank">previous article and build upon it</a>. I've introduced in the past this interface:</div>
<div>
<br /></div>
<pre class="brush: csharp">public interface IDomainEvent {}
</pre>
<br />
IDomainEvent interface was used like this:<br />
<br />
<pre class="brush: csharp">
public class CustomerCheckedOut : IDomainEvent
{
public Purchase Purchase { get; set; }
}
</pre>
<br />
<br />
We are going to change IDomainEvent to DomainEvent:<br />
<br />
<pre class="brush: csharp">
public abstract class DomainEvent
{
public string Type { get { return this.GetType().Name; } }
public DateTime Created { get; private set; }
public Dictionary<string, Object> Args { get; private set; }
public DomainEvent()
{
this.Created = DateTime.Now;
this.Args = new Dictionary<string, Object>();
}
public abstract void Flatten();
}
</pre>
<br />
This new DomainEvent will:<br />
<ol>
<li>Give you a timestamp for when domain event was created </li>
<li>Get the domain event name </li>
<li>Force events to flatten its payloads </li>
<li>Stores important arguments against the event </li>
</ol>
<br />
Here is example implementation:</div>
<pre class="brush: csharp">
public class CustomerCheckedOut : DomainEvent
{
public Purchase Purchase { get; set; }
public override void Flatten()
{
this.Args.Add("CustomerId", this.Purchase.Customer.Id);
this.Args.Add("PurchaseId", this.Purchase.Id);
this.Args.Add("TotalCost", this.Purchase.TotalCost);
this.Args.Add("TotalTax", this.Purchase.TotalTax);
this.Args.Add("NumberOfProducts", this.Purchase.Products.Count);
}
}
</pre>
<br />
Flatten method is used to capture important arguments against the event. How you flatten really depends on your requirements. For example if you want to store information for audit purposes, then above flatten might be good enough. If you want to store events so that you can "undo" or "replay" you might want to store more information.<br />
<br />
Why have Flatten method at all? Why not serialise and store the entire "Purchase" object? This object might have many value objects hanging of it, it might also have an access to another aggregate root. You will end up storing a lot of redundant data, it will be harder to keep track of versions (if your object shape changes, which it will) and it will be harder to query. This is why Flatten method is important, it strips away all of the noise.<br />
<br />
We don't want to handle all event flattening and persisting manually. To simplify and automate the event handling process I've introduced generic event handler:<br />
<pre class="brush: csharp">
public class DomainEventHandle<TDomainEvent> : Handles<TDomainEvent>
where TDomainEvent : DomainEvent
{
IDomainEventRepository domainEventRepository;
public DomainEventHandle(IDomainEventRepository domainEventRepository)
{
this.domainEventRepository = domainEventRepository;
}
public void Handle(TDomainEvent args)
{
args.Flatten();
this.domainEventRepository.Add(args);
}
}
</pre>
<br />
<div>
<br /></div>
<div>
</div>
<div style="border: 1px solid #ccc; overflow: hidden; padding: 15px;">
<a href="https://github.com/zkavtaskin/Domain-Driven-Design-Example">
</a><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJ-0GCQPZQNJ2G_UBKrw7MQ-yvMh14PMSYIaOaVjJNBcJ9ss5y9L9njtzEF0bpeLhXlcSPWKfz_NpAoUpDk6ig9buDK-LWZuMofPltM2Z_YRHVyvUe4VimVGCytA_jUT1EOWeSmMKIER0/s1600/GitHub-Mark-64px.png" style="-webkit-box-shadow: none; border: 0px; box-shadow: none; float: left;" /><a href="https://github.com/zkavtaskin/Domain-Driven-Design-Example">
</a><span style="float: left; margin-left: 15px; margin-top: 20px;"><a href="https://github.com/zkavtaskin/Domain-Driven-Design-Example">Would like to see full working example? <br /> Browse "Domain-Driven Design Example" Repository On Github</a> </span><a href="https://github.com/zkavtaskin/Domain-Driven-Design-Example">
</a>
</div>
<br />
<h3>
<br /><span style="font-family: inherit;">Extending this to meet additional security and operational requirements </span></h3>
<h3>
<div style="font-weight: normal;">
</div>
<div style="font-weight: normal;">
</div>
</h3>
<h3>
<div style="font-weight: normal;">
<span style="font-family: inherit; font-size: small;">You can take this few steps further and create a correlation id for the entire web request. This way you will be able to correlate IIS W3C logs, event logs and database logs. Find out how you can achieve this <a href="http://www.zankavtaskin.com/2016/09/how-to-correlate-iis-application-and.html">here</a>. </span><br />
<br /></div>
</h3>
<ul>
</ul>
<div>
<ul>
</ul>
<div>
<i>*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment. </i><br />
<ul>
</ul>
</div>
</div>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com11tag:blogger.com,1999:blog-3391776337973860668.post-58269068636105258732016-08-12T15:12:00.000-07:002016-09-05T14:32:02.704-07:00Creating Custom Key Store Provider for SQL Always Encrypted (Without Key Vault Example PoC)Recently we had to implement custom key store provider for always encrypted. We wanted it to access our own key store to retrieve the master key and to decrypt the column key. It was not very clear how this can be achieved. So I've decided to produce a PoC and write an article about it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPrLadziet8Fg_3TVywzo8BExhcDAdkwL_7N8tSHtpigVGkxSREgbt5SnNishDWcuj6thKqQY7s_Vhie6Qa72d9szCo4URsmsZnKzSKR9Wp9kyKykOotimRnQk42YWBQxfJPzPKFeMdRg/s1600/Screen+Shot+2016-08-12+at+22.44.42.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Custom Key Store Provider for SQL Always Encrypted" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPrLadziet8Fg_3TVywzo8BExhcDAdkwL_7N8tSHtpigVGkxSREgbt5SnNishDWcuj6thKqQY7s_Vhie6Qa72d9szCo4URsmsZnKzSKR9Wp9kyKykOotimRnQk42YWBQxfJPzPKFeMdRg/s1600/Screen+Shot+2016-08-12+at+22.44.42.png" title="Custom Key Store Provider for SQL Always Encrypted" /></a></div>
<br />
<h3>
</h3>
<div>
<br /></div>
<h3>
Setup your C# application</h3>
<h4>
</h4>
<h4>
</h4>
<div>
<br /></div>
<h4>
Step 1:</h4>
Make sure your project is set to .NET Framework 4.6.<br />
<br />
<h4>
Step 2:</h4>
Implement your own custom store provider by extending the SqlColumnEncryptionKeyStoreProvider and overriding the two methods:<br />
<br />
<pre class="brush: csharp"> public class MyOwnCustomKeyStoreProvider : SqlColumnEncryptionKeyStoreProvider
{
string masterKeyThatWillNotBeHardcodedInYourApp = "someMasterKey";
byte[] saltThatWillNotBeHardcodedInYourApp = UTF8Encoding.UTF8.GetBytes("someSalt");
//This will constantly get used
public override byte[] DecryptColumnEncryptionKey(string masterKeyPath, string encryptionAlgorithm, byte[] encryptedColumnEncryptionKey)
{
using (MemoryStream ms = new MemoryStream())
{
using (RijndaelManaged AES = new RijndaelManaged())
{
AES.KeySize = 256;
AES.BlockSize = 128;
Rfc2898DeriveBytes keyBytes = new Rfc2898DeriveBytes(
masterKeyThatWillNotBeHardcodedInYourApp,
saltThatWillNotBeHardcodedInYourApp,
1000
);
AES.Key = keyBytes.GetBytes(AES.KeySize / 8);
AES.IV = keyBytes.GetBytes(AES.BlockSize / 8);
AES.Mode = CipherMode.CBC;
using (CryptoStream cs = new CryptoStream(ms, AES.CreateDecryptor(), CryptoStreamMode.Write))
{
cs.Write(encryptedColumnEncryptionKey, 0, encryptedColumnEncryptionKey.Length);
cs.Close();
}
encryptedColumnEncryptionKey = ms.ToArray();
}
}
return encryptedColumnEncryptionKey;
}
//This will never get used by the app, I've used it just to encrypt the column key
public override byte[] EncryptColumnEncryptionKey(string masterKeyPath, string encryptionAlgorithm, byte[] columnEncryptionKey)
{
byte[] encryptedBytes = null;
using (MemoryStream ms = new MemoryStream())
{
using (RijndaelManaged AES = new RijndaelManaged())
{
AES.KeySize = 256;
AES.BlockSize = 128;
Rfc2898DeriveBytes keyBytes = new Rfc2898DeriveBytes(
masterKeyThatWillNotBeHardcodedInYourApp,
saltThatWillNotBeHardcodedInYourApp,
1000
);
AES.Key = keyBytes.GetBytes(AES.KeySize / 8);
AES.IV = keyBytes.GetBytes(AES.BlockSize / 8);
AES.Mode = CipherMode.CBC;
using (CryptoStream cs = new CryptoStream(ms, AES.CreateEncryptor(), CryptoStreamMode.Write))
{
cs.Write(columnEncryptionKey, 0, columnEncryptionKey.Length);
cs.Close();
}
encryptedBytes = ms.ToArray();
}
}
return encryptedBytes;
}
}
</pre>
<br />
<br />
<h4>
Step 3:</h4>
Register your provider with the SqlConnection:<br />
<br />
<pre class="brush: csharp"> //Register your encryption key strategies
Dictionary<string, SqlColumnEncryptionKeyStoreProvider> providerStrategies =
new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>();
providerStrategies.Add("MY_OWN_CUSTOM_KEY_STORE_PROVIDER", new MyOwnCustomKeyStoreProvider());
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providerStrategies);
</pre>
<br />
<h4>
Step 4:</h4>
Now, pay attention. Make sure that your connection is configured correctly, I've spent several hours trying to figure out why my setup was not working. It was all because I did not include "Column Encryption Setting=Enabled" in the connection string:<br />
<br />
<pre class="brush: csharp"> new SqlConnection("Server=tcp:some.database.windows.net,1433;Database=testing;User ID=testing@testing;Password=Password;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;Column Encryption Setting=Enabled")
</pre>
<br />
If you don't include Column Encryption Setting=Enabled, you will get unhelpful exception like this:<br />
<i><span style="color: red;"><br /></span></i>
<i><span style="color: red;">An unhandled exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll</span></i><br />
<span style="color: red;"><i><br /></i>
<i>Additional information: Operand type clash: nvarchar is incompatible with nvarchar(11) encrypted with (encryption_type = 'DETERMINISTIC', encryption_algorithm_name = 'AEAD_AES_256_CBC_HMAC_SHA_256', column_encryption_key_name = 'MO_CEK1', column_encryption_key_database_name = 'sometest')</i></span><br />
<span style="color: red;"><i><br /></i>
<i>Incorrect parameter encryption metadata was received from the client. The error occurred during the invocation of the batch and therefore the client can refresh the parameter encryption metadata by calling sp_describe_parameter_encryption and retry.</i></span><br />
<h4>
</h4>
<h4>
</h4>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
<br /></h3>
<h3>
Setup your database</h3>
<div>
<br /></div>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
Step 1:</h4>
Define your custom key store provider:<br />
<br />
<pre class="brush: csharp"> CREATE COLUMN MASTER KEY [MO_CMKSP] --Stands for My Own Custom Key Store Provider
WITH ( KEY_STORE_PROVIDER_NAME = 'MY_OWN_CUSTOM_KEY_STORE_PROVIDER',
KEY_PATH = 'MyKeyStoreWillNotUseThis')
</pre>
<br />
<h4>
Step 2:</h4>
Define the column encryption key that will get unwrapped by your own custom key store provider. Encrypted value needs to be some random value that gets encrypted by your master key and stored here as a hexadecimal:<br />
<br />
<pre class="brush: csharp"> CREATE COLUMN ENCRYPTION KEY [MO_CEK1] -- Stands for My Own Column Encryption Key 1
WITH VALUES
(
COLUMN_MASTER_KEY = [MO_CMKSP],
ALGORITHM = 'RSA_OAEP',
ENCRYPTED_VALUE = 0x29128e12266a71dd098bc3223b3bbf293a275b2ec8c13f97515f54dd7d2a54af46f37071e0e16e777d73f4a743ddb991
)
</pre>
<br />
<br />
<h4>
Step 3:</h4>
Encrypt columns by specifying the column encryption key:<br />
<br />
<pre class="brush: csharp"> CREATE TABLE [dbo].[Employee](
[Id] [int] IDENTITY(1,1) NOT NULL,
[SSN] [nvarchar](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (
COLUMN_ENCRYPTION_KEY = [MO_CEK1],
ENCRYPTION_TYPE = Deterministic,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) NOT NULL,
[Salary][int]
ENCRYPTED WITH (
COLUMN_ENCRYPTION_KEY = [MO_CEK1],
ENCRYPTION_TYPE = RANDOMIZED,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) NOT NULL,
PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[EmployeeExtraInformation](
[Id] [int] IDENTITY(1,1) NOT NULL,
[EyeColor] [nvarchar](11) NOT NULL,
[SSN] [nvarchar](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (
COLUMN_ENCRYPTION_KEY = [MO_CEK1],
ENCRYPTION_TYPE = Deterministic,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) NOT NULL,
PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
</pre>
<br />
<h3>
<br />PoC Code</h3>
<br />
<h4>
Program.cs</h4>
<pre class="brush: csharp">using System;
using System.Collections.Generic;
using System.Data.SqlClient;
namespace CustomKeyStoreProvider
{
class Program
{
static void Main(string[] args)
{
//Register your encryption key strategies
Dictionary<string, SqlColumnEncryptionKeyStoreProvider> providerStrategies =
new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>();
providerStrategies.Add("MY_OWN_CUSTOM_KEY_STORE_PROVIDER", new MyOwnCustomKeyStoreProvider());
//Apparently this works transparently with the Hibernate and the Entity Framework!
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providerStrategies);
using (SqlConnection connection = new SqlConnection({Your connection string};Column Encryption Setting=Enabled))
{
connection.Open();
string ssn;
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "INSERT INTO [dbo].[Employee] VALUES (@ssn, @salary)";
Random rand = new Random();
ssn = string.Format(@"{0:d3}-{1:d2}-{2:d4}", rand.Next(0, 1000), rand.Next(0, 100), rand.Next(0, 10000));
command.Parameters.AddWithValue("@ssn", ssn);
command.Parameters.AddWithValue("@salary", 18000);
command.ExecuteNonQuery();
}
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "INSERT INTO [dbo].[EmployeeExtraInformation] (eyecolor, ssn) VALUES (@eyecolor, @ssn)";
command.Parameters.AddWithValue("@eyecolor", "blue");
command.Parameters.AddWithValue("@ssn", ssn);
command.ExecuteNonQuery();
}
//Show stored data unencrypted
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT [id], [ssn], [salary] FROM [dbo].[Employee]";
using (SqlDataReader reader = command.ExecuteReader())
{
if(reader.HasRows)
{
Console.WriteLine("-- Showing all rows:");
while (reader.Read())
{
Console.WriteLine($"id : {reader["id"]}, ssn : {reader["ssn"]}, salary : {reader["salary"]}");
}
}
}
}
//Equals search, this actually works
using(SqlCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT [id], [ssn], [salary] FROM [dbo].[Employee] WHERE [ssn] = @ssn";
command.Parameters.AddWithValue("@ssn", ssn);
using (SqlDataReader reader = command.ExecuteReader())
{
if (reader.HasRows)
{
Console.WriteLine($"-- Showing found record for ssn {ssn}:");
while (reader.Read())
{
Console.WriteLine($"id : {reader["id"]}, ssn : {reader["ssn"]}, salary : {reader["salary"]}");
}
}
}
}
//Inner Join, this works as well
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = @"SELECT [dbo].[Employee].[salary], [dbo].[Employee].[ssn], [dbo].[EmployeeExtraInformation].[eyecolor] FROM [dbo].[Employee]
INNER JOIN [dbo].[EmployeeExtraInformation] ON [dbo].[Employee].[ssn] = [dbo].[EmployeeExtraInformation].[ssn]";
using (SqlDataReader reader = command.ExecuteReader())
{
if (reader.HasRows)
{
Console.WriteLine($"-- Showing all records inner joined:");
while (reader.Read())
{
Console.WriteLine($"eyecolor : {reader["eyecolor"]}, ssn : {reader["ssn"]}, salary : {reader["salary"]}");
}
}
}
}
try
{
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT [id], [ssn], [salary] FROM [dbo].[Employee] WHERE [ssn] like @ssn";
command.Parameters.AddWithValue("@ssn", ssn);
command.ExecuteReader();
}
}
catch (Exception ex)
{
Console.WriteLine("-- As expected, can't search on ssn using like:");
Console.WriteLine(ex.Message);
}
try
{
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT [id], [ssn], [salary] FROM [dbo].[Employee] WHERE [salary] = @salary";
command.Parameters.AddWithValue("@salary", 18000);
command.ExecuteReader();
}
}
catch(Exception ex)
{
Console.WriteLine("-- As expected, can't search on salary, it is a randomized field:");
Console.WriteLine(ex.Message);
}
connection.Close();
}
Console.ReadLine();
}
}
}
</pre>
<br />
<h4>
MyOwnCustomKeyStoreProvider.cs</h4>
<pre class="brush: csharp">using System.Data.SqlClient;
using System.IO;
using System.Security.Cryptography;
using System.Text;
namespace CustomKeyStoreProvider
{
public class MyOwnCustomKeyStoreProvider : SqlColumnEncryptionKeyStoreProvider
{
string masterKeyThatWillNotBeHardcodedInYourApp = "someMasterKey";
byte[] saltThatWillNotBeHardcodedInYourApp = UTF8Encoding.UTF8.GetBytes("someSalt");
//This will constantly get used
public override byte[] DecryptColumnEncryptionKey(string masterKeyPath, string encryptionAlgorithm, byte[] encryptedColumnEncryptionKey)
{
using (MemoryStream ms = new MemoryStream())
{
using (RijndaelManaged AES = new RijndaelManaged())
{
AES.KeySize = 256;
AES.BlockSize = 128;
Rfc2898DeriveBytes keyBytes = new Rfc2898DeriveBytes(
masterKeyThatWillNotBeHardcodedInYourApp,
saltThatWillNotBeHardcodedInYourApp,
1000
);
AES.Key = keyBytes.GetBytes(AES.KeySize / 8);
AES.IV = keyBytes.GetBytes(AES.BlockSize / 8);
AES.Mode = CipherMode.CBC;
using (CryptoStream cs = new CryptoStream(ms, AES.CreateDecryptor(), CryptoStreamMode.Write))
{
cs.Write(encryptedColumnEncryptionKey, 0, encryptedColumnEncryptionKey.Length);
cs.Close();
}
encryptedColumnEncryptionKey = ms.ToArray();
}
}
return encryptedColumnEncryptionKey;
}
//This will never get used by the app, I've used it just to encrypt the column key
public override byte[] EncryptColumnEncryptionKey(string masterKeyPath, string encryptionAlgorithm, byte[] columnEncryptionKey)
{
byte[] encryptedBytes = null;
using (MemoryStream ms = new MemoryStream())
{
using (RijndaelManaged AES = new RijndaelManaged())
{
AES.KeySize = 256;
AES.BlockSize = 128;
Rfc2898DeriveBytes keyBytes = new Rfc2898DeriveBytes(
masterKeyThatWillNotBeHardcodedInYourApp,
saltThatWillNotBeHardcodedInYourApp,
1000
);
AES.Key = keyBytes.GetBytes(AES.KeySize / 8);
AES.IV = keyBytes.GetBytes(AES.BlockSize / 8);
AES.Mode = CipherMode.CBC;
using (CryptoStream cs = new CryptoStream(ms, AES.CreateEncryptor(), CryptoStreamMode.Write))
{
cs.Write(columnEncryptionKey, 0, columnEncryptionKey.Length);
cs.Close();
}
encryptedBytes = ms.ToArray();
}
}
return encryptedBytes;
}
}
}
</pre>
<br />
<h4>
Setup.sql</h4>
<pre class="brush: csharp">CREATE COLUMN MASTER KEY [MO_CMKSP] --Stands for My Own Custom Key Store Provider
WITH ( KEY_STORE_PROVIDER_NAME = 'MY_OWN_CUSTOM_KEY_STORE_PROVIDER',
KEY_PATH = 'MyKeyStoreWillNotUseThis')
GO
CREATE COLUMN ENCRYPTION KEY [MO_CEK1] -- Stands for My Own Column Encryption Key 1
WITH VALUES
(
COLUMN_MASTER_KEY = [MO_CMKSP],
ALGORITHM = 'RSA_OAEP',
ENCRYPTED_VALUE = 0x29128e12266a71dd098bc3223b3bbf293a275b2ec8c13f97515f54dd7d2a54af46f37071e0e16e777d73f4a743ddb991
)
GO
CREATE TABLE [dbo].[Employee](
[Id] [int] IDENTITY(1,1) NOT NULL,
[SSN] [nvarchar](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (
COLUMN_ENCRYPTION_KEY = [MO_CEK1],
ENCRYPTION_TYPE = Deterministic,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) NOT NULL,
[Salary][int]
ENCRYPTED WITH (
COLUMN_ENCRYPTION_KEY = [MO_CEK1],
ENCRYPTION_TYPE = RANDOMIZED,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) NOT NULL,
PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
CREATE TABLE [dbo].[EmployeeExtraInformation](
[Id] [int] IDENTITY(1,1) NOT NULL,
[EyeColor] [nvarchar](11) NOT NULL,
[SSN] [nvarchar](11) COLLATE Latin1_General_BIN2
ENCRYPTED WITH (
COLUMN_ENCRYPTION_KEY = [MO_CEK1],
ENCRYPTION_TYPE = Deterministic,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) NOT NULL,
PRIMARY KEY CLUSTERED
(
[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
</pre>
<br />
<br />
<b>Useful links:</b>
<br />
<div>
<ul>
<li><a href="https://msdn.microsoft.com/en-GB/library/mt163865.aspx" rel="nofollow" target="_blank">Always Encrypted Database Engine</a></li>
<li><a href="https://blogs.msdn.microsoft.com/sqlsecurity/2015/09/25/creating-custom-key-store-providers-for-always-encrypted-azure-key-vault-example/" rel="nofollow" target="_blank">Creating Custom Azure Key Vault Key Store Provider for Always Encrypted </a></li>
<li>Always Encrypted works transparently with <a href="http://vladimirgoncharov.blogspot.co.uk/2015/07/using-always-encrypted-with-nhibernate.html" rel="nofollow" target="_blank">NHibernate</a> and <a href="https://blogs.msdn.microsoft.com/sqlsecurity/2015/08/27/using-always-encrypted-with-entity-framework-6/" rel="nofollow" target="_blank">Entity Framework, however there are limitations</a></li>
</ul>
<div>
<i><br /></i>
<i>*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment. </i><br />
<br /></div>
</div>
<a href="http://www.codeproject.com/" rel="tag" style="display: none;">CodeProject</a>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com6tag:blogger.com,1999:blog-3391776337973860668.post-30839105104298802882016-08-07T09:32:00.001-07:002016-08-07T09:39:35.626-07:00How to use TypeScript with FlotCharts or any other external JavaScript library<div>
I've started using TypeScript recently and I wanted to interact with FlotCharts. I had two questions on my mind:<br />
<ol>
<li>How does non TypeScript library interact with TypeScript app?</li>
<li>How do I use FlotCharts now? Has the interface completely changed? </li>
</ol>
This article will answer these questions and provide you with two implementation examples.<br />
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6O0BwtvZXmUmHUtV4YYN0YeXxQjGsMSF-yKOgez5MFYyAmKu2hHBk5JdU1DtVY_yhbzRGE4XSi9DuqE2pPqmR_1wy6nEYKo1oy7iidRe_Wx74QmZufXZYZ64oBIpiULGrEGtejolO5jA/s1600/ts.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6O0BwtvZXmUmHUtV4YYN0YeXxQjGsMSF-yKOgez5MFYyAmKu2hHBk5JdU1DtVY_yhbzRGE4XSi9DuqE2pPqmR_1wy6nEYKo1oy7iidRe_Wx74QmZufXZYZ64oBIpiULGrEGtejolO5jA/s320/ts.png" width="320" /></a></div>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Step 1:</h4>
<div>
Get the FlotChart Type definition file. You can do this via NuGet by invoking this command: </div>
<div>
Install-Package flot.TypeScript.DefinitelyTyped. </div>
<div>
<br /></div>
<div>
Type definition file is just an interface that is used to interact with the native JavaScript code. </div>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Step 2:</h4>
<div>
Get the actual FlotCharts library. You can do this via NuGet by invoking this command:</div>
<div>
Install-Package flot </div>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Step 3:</h4>
<div>
Check your "Scripts" folder structure, it should look something like this:</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTk10ocHb23EkvH1ox3tn9XskhOBhZs9B19efVnCeNjWQ0ZZDVPDbQ8RSEo_5c7SjAjmh_c_Tt0yHkVxZ7uBRkHYvlR-_ipIceiFtPSUiBqqh2xZSZ7FJAHh1hZYJI-Cl5yVSptlGNoGY/s1600/Capture.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTk10ocHb23EkvH1ox3tn9XskhOBhZs9B19efVnCeNjWQ0ZZDVPDbQ8RSEo_5c7SjAjmh_c_Tt0yHkVxZ7uBRkHYvlR-_ipIceiFtPSUiBqqh2xZSZ7FJAHh1hZYJI-Cl5yVSptlGNoGY/s400/Capture.PNG" /></a></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<h4>
</h4>
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Step 4:</h4>
<div>
Take a look at the query.flot interfaces, they can be found here: Scripts/Typings/flot/jquery.flot.d.</div>
<div>
<h4 style="-webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 1; word-spacing: 0px;">
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
</h4>
<h4>
<br /></h4>
<h4>
Step 5 - Implementation:</h4>
</div>
<div>
<b><br /></b>
<b>
Explicit Approach</b></div>
<div>
If you are a strong type purist you can go all the way and actually implement the defined interfaces like so:</div>
<pre class="brush: csharp">class DataSeries implements jquery.flot.dataSeries {
label: string;
data: Array<Array<number>> = new Array<Array<number>>();
constructor(label: string, data: Array<Array<number>>) {
this.label = label;
this.data = data;
}
}
class PlotOptions implements jquery.flot.plotOptions {
grid: jquery.flot.gridOptions;
constructor(grid: jquery.flot.gridOptions) {
this.grid = grid;
}
}
class GridOptions implements jquery.flot.gridOptions {
show: boolean;
constructor(show: boolean) {
this.show = show;
}
}
</pre>
<div>
Once you have implemented your classes, you can interact with the FlotChart library like this:
</div>
<pre class="brush: csharp"> let dataSeriesA: DataSeries = new DataSeries("A", [[0, 10], [1, 20], [2, 30]]);
let dataSeriesB: DataSeries = new DataSeries("B", [[0, 5], [1, 3], [2, 50]]);
let plotElement: JQuery = jQuery("#plot");
jQuery.plot(plotElement, [dataSeriesA, dataSeriesB], new PlotOptions(new GridOptions(false)));
</pre>
<div>
<b><br /></b>
<b>
Implicit Approach</b><br />
If you don't want to implement classes and just want to provide objects you can interact with the FlotChart library like this instead:<br />
<pre class="brush: javascript"> $.plot(
$("#plot"),
[
{ label: "A", data: [[0, 10], [1, 20], [2, 30]] },
{ label: "B", data: [[0, 5], [1, 3], [2, 50]] }
],
{
grid: {
show : false
}
}
);
</pre>
<br />
<h4>
Sample code</h4>
<b><br /></b>
<b>App.ts:</b><br />
<pre class="brush: csharp">
class DataSeries implements jquery.flot.dataSeries {
label: string;
data: Array<Array<number>> = new Array<Array<number>>();
constructor(label: string, data: Array<Array<number>>) {
this.label = label;
this.data = data;
}
}
class PlotOptions implements jquery.flot.plotOptions {
grid: jquery.flot.gridOptions;
constructor(grid: jquery.flot.gridOptions) {
this.grid = grid;
}
}
class GridOptions implements jquery.flot.gridOptions {
show: boolean;
constructor(show: boolean) {
this.show = show;
}
}
function explicit() {
let dataSeriesA: DataSeries = new DataSeries("A", [[0, 10], [1, 20], [2, 30]]);
let dataSeriesB: DataSeries = new DataSeries("B", [[0, 5], [1, 3], [2, 50]]);
let plotElement: JQuery = jQuery("#plotE");
jQuery.plot(plotElement, [dataSeriesA, dataSeriesB], new PlotOptions(new GridOptions(false)));
}
function implicit() {
$.plot(
$("#plotI"),
[
{ label: "A", data: [[0, 10], [1, 20], [2, 30]] },
{ label: "B", data: [[0, 5], [1, 3], [2, 50]] }
],
{
grid: {
show : false
}
}
);
}
window.onload = () => {
explicit();
implicit();
};
</pre>
<br />
<b><br /></b>
<b>Index.html:
</b><br />
<pre class="brush: javascript">
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>TypeScript With Flot Demo</title>
<script src="Scripts/jquery-1.4.1.js"></script>
<script src="Scripts/flot/jquery.flot.js"></script>
<script src="app.js"></script>
</head>
<body>
<h1>TypeScript with Flot demo</h1>
<div class="plot-container">
<div id="plotE" style="width:500px;height:500px;"></div>
<div id="plotI" style="width:500px;height:500px;"></div>
</div>
</body>
</html></pre>
<h4>
Summary:</h4>
</div>
<ol>
<li>TypeScript apps interact with non TypeScript libraries through definition files.</li>
<li>Library interfaces remain mostly the same. </li>
<li>You can interface explicitly by actually implementing reusable classes or implicitly by using objects.</li>
</ol>
<div>
<b><br /></b>
<b>Useful links:</b></div>
<div>
<ul>
<li><a href="https://www.typescriptlang.org/docs/handbook/interfaces.html" rel="nofollow" target="_blank">TypeScript interfaces </a></li>
<li><a href="https://www.typescriptlang.org/docs/handbook/writing-declaration-files.html" rel="nofollow" target="_blank">Writing Declaration files </a></li>
</ul>
<div>
<i><br /></i>
<i>*Note: Code in this article is not production ready and is used for prototyping purposes only. If you have suggestions or feedback please do comment. </i><br />
<ul>
</ul>
</div>
</div>
Zan Kavtaskinhttp://www.blogger.com/profile/10698403630279097421noreply@blogger.com0