1 00:00:00,140 --> 00:00:01,960 Hey, welcome back to part 2, 2 00:00:01,960 --> 00:00:05,290 Designing and Creating Windowed Aggregates. 3 00:00:05,290 --> 00:00:07,450 We are going to pick up where we left off 4 00:00:07,450 --> 00:00:10,460 from the last lesson, and we are going to be talking about 5 00:00:10,460 --> 00:00:11,910 the remaining 3 windows, 6 00:00:11,910 --> 00:00:14,460 sliding, session, and snapshot. 7 00:00:14,460 --> 00:00:17,190 So with that, let's dive in and start talking 8 00:00:17,190 --> 00:00:19,500 about sliding windows. 9 00:00:19,500 --> 00:00:24,143 Sliding windows output only when an event occurs, 10 00:00:25,010 --> 00:00:29,193 and every window has to have at least one event. 11 00:00:30,200 --> 00:00:31,470 A good way to think about this is 12 00:00:31,470 --> 00:00:33,570 these pictures over here on the right, 13 00:00:33,570 --> 00:00:36,680 when there's no people there, there's no event occurring, 14 00:00:36,680 --> 00:00:38,260 we're not going to get a window. 15 00:00:38,260 --> 00:00:41,670 We only get a window when there's an event occurring. 16 00:00:41,670 --> 00:00:45,300 And unlike the other 2 windows, 17 00:00:45,300 --> 00:00:48,590 sliding windows are going to continuously move forward 18 00:00:48,590 --> 00:00:53,010 by an epsilon, which is just some defined period of time. 19 00:00:53,010 --> 00:00:55,250 So basically instead of starting and stopping 20 00:00:55,250 --> 00:00:57,480 with a distinct 10-second window, 21 00:00:57,480 --> 00:01:01,010 it's going to continuously move forward in time. 22 00:01:01,010 --> 00:01:04,170 Now I know what you're thinking, this is complicated. 23 00:01:04,170 --> 00:01:07,460 This is definitely the most complicated of the windows, 24 00:01:07,460 --> 00:01:10,270 but stick with me and we will get through this together. 25 00:01:10,270 --> 00:01:13,270 So just hang on until the end and we'll get there. 26 00:01:13,270 --> 00:01:16,690 So, because we're moving forward in time 27 00:01:16,690 --> 00:01:19,820 with this epsilon and it's continuously moving, 28 00:01:19,820 --> 00:01:22,790 we could have an infinite number of windows in theory. 29 00:01:22,790 --> 00:01:26,080 And so to avoid this, Azure is only going to give us 30 00:01:26,080 --> 00:01:31,050 events when something enters or exits the window. 31 00:01:31,050 --> 00:01:32,920 Now, let me show you what I mean here. 32 00:01:32,920 --> 00:01:36,750 So this is an example of a 10-second sliding window. 33 00:01:36,750 --> 00:01:39,950 So let's start at this very first event, 34 00:01:39,950 --> 00:01:42,920 and we start with 0 seconds, this green bubbles, 35 00:01:42,920 --> 00:01:45,260 and we're going to go forward in time. 36 00:01:45,260 --> 00:01:47,310 At basically 9 or 10 seconds, 37 00:01:47,310 --> 00:01:50,620 we're going to get event #1 that occurs. 38 00:01:50,620 --> 00:01:52,530 Once event number one occurs, 39 00:01:52,530 --> 00:01:56,260 something entered the window, an event happened, 40 00:01:56,260 --> 00:01:58,060 so we're going to take from that time 41 00:01:58,060 --> 00:02:01,003 and we're going to look backwards in time, 10 seconds. 42 00:02:02,103 --> 00:02:04,520 So we're basically going to take from 0 to 10 seconds 43 00:02:04,520 --> 00:02:07,960 for the first window, event 1 occurred. 44 00:02:07,960 --> 00:02:10,330 We're still continuing on in time, 45 00:02:10,330 --> 00:02:13,120 and about 12 seconds event 5 occurs. 46 00:02:13,120 --> 00:02:15,620 So we're going to go backwards 10 seconds in time, and 47 00:02:15,620 --> 00:02:19,330 we're going to go from second 2 to basically second 12. 48 00:02:19,330 --> 00:02:22,793 And within that window, events 1 and 5 occurred. 49 00:02:23,760 --> 00:02:25,700 Time is still moving forward. 50 00:02:25,700 --> 00:02:30,700 At 15 seconds, we have events 9 and 7 that occur. 51 00:02:31,080 --> 00:02:34,610 So when that happens, we go from 15 seconds, 52 00:02:34,610 --> 00:02:37,250 back 10 seconds in time, and we create 53 00:02:37,250 --> 00:02:41,300 in a window with events, 1, 5, 9, and 7 in it. 54 00:02:41,300 --> 00:02:43,373 So those are our first 3 windows. 55 00:02:44,761 --> 00:02:49,220 We continue forward in time until we hit 20 seconds. 56 00:02:49,220 --> 00:02:52,330 At that point, a new window occurs, 57 00:02:52,330 --> 00:02:56,500 because event number one dropped off the map, 58 00:02:56,500 --> 00:02:58,920 because it's been longer than 10 seconds 59 00:02:58,920 --> 00:03:00,820 since that occurred. 60 00:03:00,820 --> 00:03:03,450 So we have a new window that contains events 61 00:03:03,450 --> 00:03:07,214 5, 9 and 7, event 1 dropped off. 62 00:03:07,214 --> 00:03:09,450 And so that's going to be from 10 to 20 seconds. 63 00:03:09,450 --> 00:03:12,670 Continue on in time, 22 seconds 64 00:03:12,670 --> 00:03:14,370 and then event 5 drops off. 65 00:03:14,370 --> 00:03:17,120 So we have a new window going from 66 00:03:17,120 --> 00:03:19,623 12 seconds to 22 seconds. 67 00:03:21,327 --> 00:03:26,200 We continue on in time, and once we reach 25 seconds, 68 00:03:26,200 --> 00:03:27,870 we would have another window, 69 00:03:27,870 --> 00:03:30,280 because events 9 and 7 dropped off, 70 00:03:30,280 --> 00:03:33,240 so that's our trigger, however, there's no new events 71 00:03:33,240 --> 00:03:37,670 from 15 to 25 seconds, so we don't get a window. 72 00:03:37,670 --> 00:03:39,960 We had an action that occurred, but no window, 73 00:03:39,960 --> 00:03:44,393 because there's no active events between 15 and 25 seconds. 74 00:03:45,310 --> 00:03:50,310 Continue on in time, 26 seconds, we get event number 8. 75 00:03:50,680 --> 00:03:53,650 And so we're going to go back and look 10 seconds back. 76 00:03:53,650 --> 00:03:56,690 So basically from 16 to 26 seconds 77 00:03:56,690 --> 00:03:59,533 we have event 8 and that's our new window. 78 00:04:00,600 --> 00:04:04,230 Continuing on in time, we move forward another 10 seconds, 79 00:04:04,230 --> 00:04:06,140 event 8 drops off, and we look, 80 00:04:06,140 --> 00:04:08,360 but hey, there's been no new events, 81 00:04:08,360 --> 00:04:11,450 so we don't get a new window at that point. 82 00:04:11,450 --> 00:04:12,750 So even though there was an action 83 00:04:12,750 --> 00:04:15,820 of 8 dropping off, there's no new window, 84 00:04:15,820 --> 00:04:19,100 because there's no new events between 26 seconds 85 00:04:19,100 --> 00:04:22,893 and whenever we look at this next, would be 36 seconds. 86 00:04:24,450 --> 00:04:25,870 Hopefully that makes sense. 87 00:04:25,870 --> 00:04:27,870 You may have to rewind just a little bit 88 00:04:27,870 --> 00:04:30,830 and watch this a couple of times if you're still struggling, 89 00:04:30,830 --> 00:04:33,160 but sliding windows are important 90 00:04:33,160 --> 00:04:35,840 and this is how sliding windows work. 91 00:04:35,840 --> 00:04:39,930 So jumping in and looking at the code for this, 92 00:04:39,930 --> 00:04:44,300 luckily way simpler, group by, here's our topic, 93 00:04:44,300 --> 00:04:46,650 sliding window, how many seconds 94 00:04:46,650 --> 00:04:49,850 or how many minutes or whatever time period we want. 95 00:04:49,850 --> 00:04:53,623 That's how we write code for sliding windows. 96 00:04:54,600 --> 00:04:59,230 Continuing on, our next one is session windows. 97 00:04:59,230 --> 00:05:03,740 Session windows produce an output only when an event occurs. 98 00:05:03,740 --> 00:05:06,210 Every window has at least one event. 99 00:05:06,210 --> 00:05:09,560 And like our last window, sliding windows, 100 00:05:09,560 --> 00:05:11,590 it's going to continuously move forward 101 00:05:11,590 --> 00:05:13,623 by that period epsilon. 102 00:05:14,840 --> 00:05:17,430 So with this example, we are going to create a window 103 00:05:17,430 --> 00:05:19,410 when event first occurs. 104 00:05:19,410 --> 00:05:22,410 So right there at 0 seconds, we have event number one, 105 00:05:22,410 --> 00:05:24,260 a window is created. 106 00:05:24,260 --> 00:05:27,490 We move forward in time, 5 seconds. 107 00:05:27,490 --> 00:05:29,640 If an event has occurred within 5 seconds, 108 00:05:29,640 --> 00:05:31,740 we continue on, in this case, 109 00:05:31,740 --> 00:05:34,870 event 6 happens within 5 seconds. 110 00:05:34,870 --> 00:05:37,600 So right there at the 5 second mark, event 6 happens. 111 00:05:37,600 --> 00:05:40,390 And so we continue on another 5 seconds. 112 00:05:40,390 --> 00:05:43,040 5 seconds goes on and we don't have any new events, 113 00:05:43,040 --> 00:05:47,043 so at 10 seconds our window closes with events one and 6. 114 00:05:47,940 --> 00:05:49,900 We continue forward in time. 115 00:05:49,900 --> 00:05:53,850 A new window is created as soon as the next event occurs. 116 00:05:53,850 --> 00:05:56,780 Event 9 happens and so we create a window. 117 00:05:56,780 --> 00:05:59,810 5 seconds moves forward, event 2 happens 118 00:05:59,810 --> 00:06:03,290 within that window, so we can extend for 5 seconds. 119 00:06:03,290 --> 00:06:05,960 5 seconds, event 3 occurs. 120 00:06:05,960 --> 00:06:09,170 We extend 5 seconds, event 4 occurs. 121 00:06:09,170 --> 00:06:12,540 We extend 5 seconds, event 8 occurs. 122 00:06:12,540 --> 00:06:14,750 5 more seconds, and then nothing happens, 123 00:06:14,750 --> 00:06:16,270 so we close the window with events 124 00:06:16,270 --> 00:06:18,563 9, 2, 3, 4, and 8 in it. 125 00:06:19,500 --> 00:06:24,500 Continuing on, our next window occurs with event 3. 126 00:06:25,490 --> 00:06:27,670 5 seconds and so on and so forth. 127 00:06:27,670 --> 00:06:29,960 So that's how a session window works. 128 00:06:29,960 --> 00:06:32,380 So basically we're looking and then we're just extending 129 00:06:32,380 --> 00:06:36,893 if we get an event that happens within that time period. 130 00:06:37,960 --> 00:06:39,820 Code for this one, also very simple, 131 00:06:39,820 --> 00:06:42,960 GROUP BY, topic, here's our session window, 132 00:06:42,960 --> 00:06:44,263 and then our timeframe. 133 00:06:46,740 --> 00:06:50,580 I have saved the easiest for last. You're welcome. 134 00:06:50,580 --> 00:06:53,090 So when we think snapshot, think camera, 135 00:06:53,090 --> 00:06:55,090 think moment in time. 136 00:06:55,090 --> 00:06:56,380 So with a snapshot window, 137 00:06:56,380 --> 00:06:57,760 we're going to get a group of events 138 00:06:57,760 --> 00:06:59,850 that have the exact same timestamp. 139 00:06:59,850 --> 00:07:03,340 So at 5 seconds we had events 1, 2, 3, and 4. 140 00:07:03,340 --> 00:07:05,260 10 seconds, we had events 5 and 6. 141 00:07:05,260 --> 00:07:07,220 At 12 seconds, event 7. 142 00:07:07,220 --> 00:07:10,170 And so those are all going to be unique windows. 143 00:07:10,170 --> 00:07:12,080 That's our snapshot window. 144 00:07:12,080 --> 00:07:14,170 And so with our snapshot window, 145 00:07:14,170 --> 00:07:17,390 code, also simple, GROUP BY, our topic, or X, 146 00:07:17,390 --> 00:07:19,900 and then it's just going to be system.timestamp 147 00:07:19,900 --> 00:07:21,840 and then some parentheses. 148 00:07:21,840 --> 00:07:24,993 That's how we're going to do the syntax for snapshot. 149 00:07:26,500 --> 00:07:30,350 All right, we have covered a ton of stuff here. 150 00:07:30,350 --> 00:07:33,100 In review, there are 5 window types: 151 00:07:33,100 --> 00:07:36,353 hopping, sliding, snapshot, tumbling, and session. 152 00:07:37,340 --> 00:07:40,080 You need to know when to use each type. 153 00:07:40,080 --> 00:07:41,770 The details are important here, 154 00:07:41,770 --> 00:07:44,030 I would suggest memorizing them, 155 00:07:44,030 --> 00:07:47,270 and you're going to need to understand the differences. 156 00:07:47,270 --> 00:07:49,982 So a good way to think about this 157 00:07:49,982 --> 00:07:52,690 is if I gave you an example of, 158 00:07:52,690 --> 00:07:54,410 I need to be able to look at 159 00:07:54,410 --> 00:07:57,210 5 minutes in time for a window, 160 00:07:57,210 --> 00:07:59,470 and it's very important that I don't have 161 00:07:59,470 --> 00:08:01,320 events in more than one window, 162 00:08:01,320 --> 00:08:03,850 and I can't have the windows overlap. 163 00:08:03,850 --> 00:08:06,880 What would be the best window type for me in this scenario? 164 00:08:06,880 --> 00:08:08,470 The answer would be hopping. 165 00:08:08,470 --> 00:08:10,640 So you need to know that level of detail 166 00:08:10,640 --> 00:08:11,890 for these window types. 167 00:08:11,890 --> 00:08:14,750 So make sure that you understand what they are, 168 00:08:14,750 --> 00:08:16,380 how they work, and you should have 169 00:08:16,380 --> 00:08:18,120 a general understanding of the syntax, 170 00:08:18,120 --> 00:08:20,200 although, that's not terribly complicated. 171 00:08:20,200 --> 00:08:23,370 If you understand that, you should be good to go. 172 00:08:23,370 --> 00:08:25,530 Again, this lesson may be one of those 173 00:08:25,530 --> 00:08:27,540 that you need to watch through more than one time 174 00:08:27,540 --> 00:08:30,660 to get a good feel of how windows work. 175 00:08:30,660 --> 00:08:33,320 If so, pop back in, watch it again. 176 00:08:33,320 --> 00:08:34,670 It's going to take a little repetition, 177 00:08:34,670 --> 00:08:37,960 because some of these concepts are rather complicated. 178 00:08:37,960 --> 00:08:39,120 So I hope this has been helpful 179 00:08:39,120 --> 00:08:41,040 and I will see you in our next lesson 180 00:08:41,040 --> 00:08:42,753 once you are good to go.