1
00:00:00,350 --> 00:00:01,580
Hey, what's up Gurus?

2
00:00:01,580 --> 00:00:04,970
So in this lesson, we are going to be talking about time,

3
00:00:04,970 --> 00:00:08,040
and specifically, how do we process time series data

4
00:00:08,040 --> 00:00:10,460
in Azure Stream Analytics?

5
00:00:10,460 --> 00:00:14,270
So in this lesson, we're going to talk about what time is,

6
00:00:14,270 --> 00:00:15,560
some ways that we can measure it,

7
00:00:15,560 --> 00:00:17,510
and then we're going to talk about some key concepts

8
00:00:17,510 --> 00:00:20,190
that you need to know for the 203.

9
00:00:20,190 --> 00:00:21,670
This is going to be a fairly short lesson

10
00:00:21,670 --> 00:00:24,070
so let's just dive right in and get started.

11
00:00:24,070 --> 00:00:26,730
To start off with, we need to talk about time

12
00:00:26,730 --> 00:00:31,400
and specifically, how do we calculate or measure time?

13
00:00:31,400 --> 00:00:33,750
So there's a couple of ways that we can do this.

14
00:00:33,750 --> 00:00:37,640
The first way is what's called the event time.

15
00:00:37,640 --> 00:00:39,370
Now, if we're looking at an event time,

16
00:00:39,370 --> 00:00:43,020
this would be when an event reaches the source.

17
00:00:43,020 --> 00:00:44,700
So the arrival.

18
00:00:44,700 --> 00:00:46,360
And a good way to think about this

19
00:00:46,360 --> 00:00:47,930
or I'll give you 2 different examples.

20
00:00:47,930 --> 00:00:50,360
The first way is if I'm writing a letter.

21
00:00:50,360 --> 00:00:55,360
So when I write the letter, I'm actively creating the event.

22
00:00:55,510 --> 00:00:58,710
That would be the event time while I'm writing the letter.

23
00:00:58,710 --> 00:01:01,030
Or for instance, if I want to go to the movies,

24
00:01:01,030 --> 00:01:03,350
when I go up and I purchase my ticket,

25
00:01:03,350 --> 00:01:06,330
that's the event time, when that physical ticket

26
00:01:06,330 --> 00:01:09,240
is first purchased at the counter.

27
00:01:09,240 --> 00:01:12,000
Now, there's another way that we can look at time.

28
00:01:12,000 --> 00:01:14,460
We can also look at processing time.

29
00:01:14,460 --> 00:01:15,840
So this is when the event

30
00:01:15,840 --> 00:01:18,060
is actually observed by the system.

31
00:01:18,060 --> 00:01:22,380
For instance, when I buy that ticket at the movie counter,

32
00:01:22,380 --> 00:01:26,500
there is a lag from when I physically buy the ticket,

33
00:01:26,500 --> 00:01:29,720
and when it goes through to cyberspace and it gets stored

34
00:01:29,720 --> 00:01:34,390
in the database in Azure or some sort of cloud, right?

35
00:01:34,390 --> 00:01:37,080
That point when it's stored in the cloud,

36
00:01:37,080 --> 00:01:40,113
that's actually when it's observed by this system.

37
00:01:41,210 --> 00:01:42,720
And so you got to think about this

38
00:01:42,720 --> 00:01:45,720
as data's coming across in the streaming fashion.

39
00:01:45,720 --> 00:01:48,380
Well, when did the event actually happen?

40
00:01:48,380 --> 00:01:49,880
Did it happen when I bought the ticket

41
00:01:49,880 --> 00:01:51,300
or when I wrote the letter,

42
00:01:51,300 --> 00:01:54,110
or did it happen when the letter was delivered,

43
00:01:54,110 --> 00:01:57,800
or when the ticket was actually recorded in the database

44
00:01:57,800 --> 00:02:00,580
in that processing time and event time?

45
00:02:00,580 --> 00:02:03,930
Lastly, we have the concept of watermarks.

46
00:02:03,930 --> 00:02:06,560
Watermarks are simply an arbitrary point in time,

47
00:02:06,560 --> 00:02:09,620
that simply shows what point the event

48
00:02:09,620 --> 00:02:14,170
has been ingressed or pulled into Azure Stream Analytics.

49
00:02:14,170 --> 00:02:17,800
That's all a watermark is, but it's a very important concept

50
00:02:17,800 --> 00:02:20,800
as we think about how we're looking at data

51
00:02:20,800 --> 00:02:22,603
in Azure Stream Analytics.

52
00:02:24,614 --> 00:02:26,680
So kind of putting everything in order,

53
00:02:26,680 --> 00:02:27,860
we have our watermark

54
00:02:27,860 --> 00:02:30,220
that is determining when the event arrived.

55
00:02:30,220 --> 00:02:33,240
And so this is going to be on the input side of things

56
00:02:33,240 --> 00:02:36,660
as we figure out when the event actually arrived.

57
00:02:36,660 --> 00:02:38,810
And then we have our windowing concept

58
00:02:38,810 --> 00:02:41,820
that is really more involved with querying.

59
00:02:41,820 --> 00:02:44,980
So watermark when it arrived, and then from there,

60
00:02:44,980 --> 00:02:46,490
we're going to take that window,

61
00:02:46,490 --> 00:02:48,860
and we're going to look at a bracket of time

62
00:02:48,860 --> 00:02:52,040
or a period of time, based upon that watermark

63
00:02:52,040 --> 00:02:54,970
to determine what sorts of transformations

64
00:02:54,970 --> 00:02:56,240
that we want to do to the data

65
00:02:56,240 --> 00:02:59,253
as it flows through Azure Stream Analytics.

66
00:03:00,250 --> 00:03:02,890
So in an ideal world, that's all you need.

67
00:03:02,890 --> 00:03:05,350
However, it's not an ideal world, right?

68
00:03:05,350 --> 00:03:07,090
We have early arriving events,

69
00:03:07,090 --> 00:03:09,490
late arriving events, missed events.

70
00:03:09,490 --> 00:03:10,500
And so because of this,

71
00:03:10,500 --> 00:03:12,740
we need to look at something called tolerance.

72
00:03:12,740 --> 00:03:14,270
And we'll talk more about this

73
00:03:14,270 --> 00:03:16,540
as we go over the next couple of lessons,

74
00:03:16,540 --> 00:03:18,400
but essentially what tolerance is,

75
00:03:18,400 --> 00:03:21,000
is just taking a look at your watermark and saying,

76
00:03:21,000 --> 00:03:23,710
okay, well how long are we going to wait for an event?

77
00:03:23,710 --> 00:03:26,140
So if I have an event that arrives

78
00:03:26,140 --> 00:03:29,000
in Azure Stream Analytics, what's the tolerance

79
00:03:29,000 --> 00:03:32,450
for a late-arriving or an early-arriving event?

80
00:03:32,450 --> 00:03:34,560
And so we'll talk more about those strategies

81
00:03:34,560 --> 00:03:36,290
in the next couple of lessons.

82
00:03:36,290 --> 00:03:39,000
What's really important for you in this lesson,

83
00:03:39,000 --> 00:03:42,710
is just simply to understand the 3 key concepts.

84
00:03:42,710 --> 00:03:45,003
What are the definitions for event,

85
00:03:45,890 --> 00:03:49,200
time, processing time, and watermarks?

86
00:03:49,200 --> 00:03:52,360
And then also understand that watermarking

87
00:03:52,360 --> 00:03:54,760
is the key to unlocking time issues.

88
00:03:54,760 --> 00:03:57,850
So the watermark is extremely important.

89
00:03:57,850 --> 00:03:59,640
So we'll be talking more about that

90
00:03:59,640 --> 00:04:02,310
and you're going to need to know the details for the 203.

91
00:04:02,310 --> 00:04:03,560
But if you understand the concepts

92
00:04:03,560 --> 00:04:05,690
of event processing and watermarks,

93
00:04:05,690 --> 00:04:08,230
you should be good to go on to the next lesson.

94
00:04:08,230 --> 00:04:09,547
I'll see you there.