Start as:

docker run --rm -p 80:80 --name httpbin kennethreitz/httpbin

Then run the curl commands

curl

To stop the Docker container open another terminal and execute

docker container stop -t0 httpbin

301 MOVED PERMANENTLY

$ curl -i http://localhost/status/301
HTTP/1.1 301 MOVED PERMANENTLY
Server: gunicorn/19.9.0
Date: Fri, 26 Sep 2025 10:32:08 GMT
Connection: keep-alive
location: /redirect/1
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 0

303 SEE OTHER (redirect)

$ curl -i http://localhost/status/303
HTTP/1.1 303 SEE OTHER
Server: gunicorn/19.9.0
Date: Fri, 26 Sep 2025 10:06:22 GMT
Connection: keep-alive
location: /redirect/1
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 0

404 NOT FOUND

$ curl -i http://localhost/status/404
HTTP/1.1 404 NOT FOUND
Server: gunicorn/19.9.0
Date: Fri, 26 Sep 2025 10:08:19 GMT
Connection: keep-alive
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Length: 0

I asked Chat GPT:

I have a Linux machine running ubuntu 2025.04. On it I have VirtualBox installed. I used to be able to run an MS Windows guest in it but not any more. It gives me the following error AMD-V is being used by another hypervisor (VERR_SVM_IN_USE). VirtualBox can't enable the AMD-V extension. Please disable the KVM kernel extension, recompile your kernel and reboot (VERR_SVM_IN_USE How can I fix this?

sudo systemctl stop libvirtd     # this gave an error
sudo modprobe -r kvm_amd kvm     # this helped me

$ python -c 'import os; print(os.getenv("NAME"))'
None

$ NAME=Foo python -c 'import os; print(os.getenv("NAME"))'
Foo

On Windows we need to do it separately:

In CMD:

> set NAME=Foo
> python -c "import os; print(os.getenv('NAME'))"
Foo

In PowerShell

> [Environment]::SetEnvironmentVariable("NAME", "Foo", "Process")
> python -c "import os; print(os.getenv('NAME'))"
Foo

We'll analyze the limitations of popular JavaScript frameworks, the advantages of using Vanilla JavaScript, and how Vanilla Boot offers a minimalist approach to building reusable UI components. The session will cover core concepts like direct DOM manipulation, component-based architecture, dynamic dependency loading, and full rendering control, followed by a walkthrough of Vanilla Boot’s source code and practical usage examples.

Eyal Lior is a Tel Aviv-based software developer with over 25 years of experience, specializing in complex GUIs built with vanilla JavaScript.

Transcript

1 00:00:02.130 --> 00:00:15.429 Gabor Szabo: Hello, and welcome to the code. Me. Event Channel and the Codema Meetup Group. My name is Gabor Sabo. I personally usually teach rust and python to companies and help companies to start using these languages.

2 00:00:15.770 --> 00:00:23.080 Gabor Szabo: And I have this meeting group, a Meetup group where I usually organize presentations about rust and python.

3 00:00:23.310 --> 00:00:26.109 Gabor Szabo: But this time it's going to be about Javascript.

4 00:00:26.340 --> 00:00:35.859 Gabor Szabo: And so I'm really happy that all agreed to give this presentation, and all the people who joined us in this meeting

5 00:00:36.010 --> 00:00:58.679 Gabor Szabo: feel free to ask questions. Just remember that we are putting it on Youtube. So we can ask in chat if you don't want to be on Youtube. And if you are watching this video on Youtube, then please, like the video and follow the channel and join us in our meeting group. So in our meet up groups, so you'll be able to join in one of our upcoming meetings.

6 00:00:58.960 --> 00:01:00.160 Gabor Szabo: Is that sad?

7 00:01:00.520 --> 00:01:06.620 Gabor Szabo: Yeah, it's your turn to introduce yourself and take over the screen.

8 00:01:06.990 --> 00:01:22.379 Eyal Lior: Okay, so Hi, everybody, my name is real. I am a full stack developer with 25 years of experience lots of experience in doing Web Gui. That's actually where I come from. That's why I have passion for Javascript.

9 00:01:22.510 --> 00:01:41.849 Eyal Lior: I started as a as a web developer web designer building websites when I was a teenager in the late nineties, and that's how I got into the field. So I always loved Javascript, and I loved the flexibility of it, the the reformers of it

10 00:01:42.510 --> 00:01:50.180 Eyal Lior: and and I'll start with a little bit introduction. So you'll see why I developed vanilla boot

11 00:01:51.066 --> 00:01:55.280 Eyal Lior: when when everybody started moving into Javascript frameworks.

12 00:01:55.590 --> 00:01:59.406 Eyal Lior: I was a little bit suspicious about it, because,

13 00:02:00.070 --> 00:02:10.080 Eyal Lior: I like vanilla Javascript. I'm not. I'm I don't like to be constrained, limited, and they felt that this is what the frameworks bring.

14 00:02:10.560 --> 00:02:18.800 Eyal Lior: I had a few years experience working with angular in its earliest versions. Version one and version 2.

15 00:02:19.510 --> 00:02:35.335 Eyal Lior: And so so anybody that has experience with Angle. I was going to be like, of course you didn't like it. It was version one and 2. It was horrible. But now we're in version. I don't know 18 or something, so it's completely different. And

16 00:02:36.090 --> 00:02:44.629 Eyal Lior: and 5 years ago I had, my 1st experience developing a mobile application with the react react. Yes.

17 00:02:44.850 --> 00:02:56.430 Eyal Lior: And since then I developed 3 more applications with React Js. And I reached a point that I'm I cannot handle it anymore and said, I'm not going to develop with react anymore. And I

18 00:02:56.550 --> 00:02:59.770 Eyal Lior: switched a few projects back to vanilla Javascript.

19 00:03:01.030 --> 00:03:16.759 Eyal Lior: Mind that some of my criticism of react is not relevant anymore because it did improve over time. But I'm going to present a project today that's going to enable you to keep using vanilla Javascript to develop

20 00:03:17.372 --> 00:03:31.140 Eyal Lior: applications with components with components just like you would. If you would use react or Anglo, but without the limitations. So let's let me just share the screen and start the presentation.

21 00:03:45.340 --> 00:03:52.620 Eyal Lior: Okay, so you must see. My, you should see my desktop. Now, right.

22 00:03:58.020 --> 00:04:00.150 Gabor Szabo: Yes, we see your whole desktop, I think.

23 00:04:00.150 --> 00:04:06.359 Eyal Lior: Oh, that! That's the vanilla boot website. That's my project. But let me just start the presentation.

24 00:04:09.040 --> 00:04:10.929 Eyal Lior: There it is. So

25 00:04:12.460 --> 00:04:30.180 Eyal Lior: vanilla boot. It's a minimal framework with maximum control, pure Javascript, pure freedom. That's the tagline. It's a lightweight javascript utility designed to streamline front end development without relying on extensive frameworks. When I say extensive frameworks, I mean, angular react view.

26 00:04:30.760 --> 00:04:39.930 Eyal Lior: You name them it enables developer to build application with reusable ui components. Yeah, how's it

27 00:04:40.140 --> 00:04:47.449 Eyal Lior: using standard Javascript and HTML, preserving the flexibility and simplicity of native web technologies?

28 00:04:48.330 --> 00:04:49.180 Eyal Lior: So

29 00:04:50.020 --> 00:04:58.990 Eyal Lior: when I ask people, why do you use frameworks? Because I don't get it like for me, I can. I can do anything with vanilla Javascript

30 00:04:59.370 --> 00:05:01.660 Eyal Lior: with ease, with extreme ease.

31 00:05:01.910 --> 00:05:08.310 Eyal Lior: So for me, it's not clear why people use frameworks, I can.

32 00:05:08.670 --> 00:05:25.839 Eyal Lior: I can guess, make a guess. But I asked people who do push for frameworks for people who I worked with people who are interviewed. Ask them, why do you use frameworks? And these are the 8 most popular answers that I got.

33 00:05:26.400 --> 00:05:33.510 Eyal Lior: The 1st one is the the fear of the free forms of the Javascript. So with Javascript, you can write

34 00:05:33.820 --> 00:05:38.859 Eyal Lior: many parts of code, and and there's no form to it. There's no

35 00:05:38.990 --> 00:05:47.940 Eyal Lior: beginning, and that there's no structural. There's No. 3 like structure for the code.

36 00:05:48.150 --> 00:05:53.166 Eyal Lior: and for someone who's managing a project and is going to give a lot of

37 00:05:53.860 --> 00:05:58.320 Eyal Lior: freedom to developers to to just build it any way they want.

38 00:05:59.050 --> 00:06:23.307 Eyal Lior: It is some somewhat scary, you know. You don't know where they're going to take it. And with the framework it's kind of like binds you to staying with a certain structure. The second thing is, you want to build modular, reusable ui components, and most people don't know how to do it with Javascript. And while it's very trivial and

39 00:06:25.550 --> 00:06:36.430 Eyal Lior: or should I say, it's very straightforward to build ui components with framework. And then there's the virtual dom which means efficient updates and rendering

40 00:06:37.556 --> 00:06:43.993 Eyal Lior: state management. People say Whoa! The framework gives me state management, and

41 00:06:44.820 --> 00:07:13.269 Eyal Lior: it can handle the more complex applications easily. It's easy when it's they, they would say, vanilla is good for small things. But when you have a complete complex app, I want state management and all that stuff built in reactivity, how to update ui on data change, you change the object and the form. Reflects the change to the data. And then there's the routing and navigation of the web app on the client side.

42 00:07:13.730 --> 00:07:25.099 Eyal Lior: And people say there's a rich ecosystem. There's a lot of support for it. There's plugins, extensions, add-ons, and tools. Blah blah! And the cross platform support.

43 00:07:25.330 --> 00:07:43.060 Eyal Lior: So people don't want to be like the old days where they have to test the everything that develop on different browsers and different operating system and the frameworks. It's 1 of the things that they promise. Now you. You might laugh when I switch to the second page.

44 00:07:44.330 --> 00:08:03.270 Eyal Lior: The 1st 2 reasons are the actual reasons. People opt to use frameworks like react and angular. Well, the number one reason is they're afraid of the freedom that Javascript in the browser brings with it, and second.

45 00:08:03.440 --> 00:08:11.231 Eyal Lior: they don't know how to build modular, reusable AI components without framework. Then there's the virtual dom

46 00:08:11.820 --> 00:08:20.439 Eyal Lior: thing that I think that nobody really like 99% of the people who I talk to actually 100% of the people

47 00:08:20.550 --> 00:08:44.109 Eyal Lior: I worked with projects I worked with. It wasn't relevant. State management, not easy at all was never easy. Maybe it's easier today, but certainly in in the previous years it was hell with frameworks rich ecosystem. If you if you if you look at libraries popular

48 00:08:44.624 --> 00:08:52.800 Eyal Lior: mature libraries thing things like aging grid, for example, that gives you the most sophisticated grid

49 00:08:52.960 --> 00:09:18.049 Eyal Lior: for web that you can put in your website or web apps. You can get it for angular for react, for view, for any kind of framework, and also for vanilla. So I think that the vast majority of libraries, certainly the most important ones, are available also for vanilla Javascript. So you don't need to use a framework in order to have those libraries.

50 00:09:19.660 --> 00:09:31.080 Eyal Lior: So let's talk about the state state the complicated world of State.

51 00:09:35.500 --> 00:09:37.566 Eyal Lior: if you want to.

52 00:09:39.220 --> 00:10:09.120 Eyal Lior: actually, you have to use this mechanism of state when you use react. And it's like a technology that you have to learn in order to do very simple things that I will soon show you how you can do it them without react. And we will see what is more simple and what's less simple. So here we have basically a button that says, add, and when you click it, it calls add value, and what it does it, it

53 00:10:09.230 --> 00:10:11.540 Eyal Lior: increments value 2,

54 00:10:11.850 --> 00:10:17.900 Eyal Lior: and then it sets value, one to the value of value 2. So when you read the code.

55 00:10:18.170 --> 00:10:31.229 Eyal Lior: you see that value, one is initialized 7, and we we expect to have the display. Show us Hello, 7. And when you click, it's going to increment value

56 00:10:31.340 --> 00:10:56.319 Eyal Lior: value 2. So it's going to be 4 is going to become 5 and then value one is going to get the value of value 2. So it's going. We expect to see 5. And then, in the subsequent presses of the button, we expect to see 6, 7, 8, and so on. So we start with 7. Then we jump to 5, 6, 7, 8, and so on. But this is not what actually happens.

57 00:10:56.670 --> 00:11:03.850 Eyal Lior: What actually is going to happen is that second?

58 00:11:06.780 --> 00:11:07.950 Eyal Lior: Here we have it.

59 00:11:08.490 --> 00:11:14.830 Eyal Lior: We have Hello, 7. Which is the value of value one. I click it once it says, Hello! 5 as expected.

60 00:11:15.090 --> 00:11:20.209 Eyal Lior: I click again, and it's 5 again, and it's 6 again, and it's 5 again.

61 00:11:20.370 --> 00:11:25.459 Eyal Lior: 5. Again I click. It's 6 again I click it. 5. What is happening?

62 00:11:25.640 --> 00:11:35.580 Eyal Lior: What? What is happening behind the scenes is that this function gets recalled again, everything is restarting.

63 00:11:35.910 --> 00:11:44.009 Eyal Lior: So it's not working as we think it's going to work. And we actually have to learn how

64 00:11:44.190 --> 00:12:11.309 Eyal Lior: what's happening behind the scenes or inside the black box. It's not the black box. It's open source, but no one's going to get into the source code of react. So we have to read a long document to understand how the Use State works behind the scenes. And when does the app being called? And when it's not in order to build something as simple as a counter that increments and integer and prints it.

65 00:12:12.350 --> 00:12:28.770 Eyal Lior: So this code that I've made here it's an example of a very common mistake. It's it has a name, the mistake. It's called reactive state mismanagement, or you can call it state initialization, anti pattern or

66 00:12:29.030 --> 00:12:39.050 Eyal Lior: stale state, an uncontrolled state mutation or redundant state updates so

67 00:12:39.310 --> 00:12:43.480 Eyal Lior: very complicated concepts in order to make something very simple.

68 00:12:45.570 --> 00:13:14.279 Eyal Lior: Now, this is how you would do it without react. If it's just HTML and standard native Javascript, the HTML would say, hello. And then there's a span which is just an element containing anything you put in it. Basically, text, and we can access it by the idea through with the idea of the span. Then there's new line and a button with title 2, and on click, there's an event. Handler, add one.

69 00:13:14.920 --> 00:13:20.389 Eyal Lior: and the Javascript would say this, we have value one initialized to 7 value, 2 initialized to 4,

70 00:13:20.700 --> 00:13:26.650 Eyal Lior: and on the start, on the loading of the page. We're going to get the element full.

71 00:13:26.780 --> 00:13:29.070 Eyal Lior: Do. By the way, do you see my mouth moving?

72 00:13:29.470 --> 00:13:31.540 Eyal Lior: No right. Can you see my mouse.

73 00:13:31.540 --> 00:13:35.110 Gabor Szabo: I I can see, and I'll have a question soon. Okay.

74 00:13:35.390 --> 00:13:52.119 Eyal Lior: And then we set it to value one to 7. Now, when we click the button, we expect Ed one to be called so in increment value 2, we say, value one equals value 2, and the text content of Foo shall be, value one.

75 00:13:52.490 --> 00:13:56.059 Eyal Lior: And let's see how it it's actually working.

76 00:13:57.330 --> 00:14:08.340 Eyal Lior: So this is the Javascript example. See? It's initialized to hell of 7. We click becomes 5, click, 6 clicks, 7, click 8, and so on. So

77 00:14:08.900 --> 00:14:14.160 Eyal Lior: there's less room to make mistakes because there's no

78 00:14:14.380 --> 00:14:21.320 Eyal Lior: complicated 100 pages document to read about how to use state. You don't have state. You just

79 00:14:21.770 --> 00:14:28.339 Eyal Lior: set the variable and set the context, the the content of the HTML element.

80 00:14:29.240 --> 00:14:31.270 Eyal Lior: So you don't need to learn.

81 00:14:31.270 --> 00:14:44.099 Gabor Szabo: I have a question because not I mean, I write Javascript and and stuff like this. But and I definitely not an expert. Why do you use on click? Is is this the

82 00:14:44.430 --> 00:14:51.470 Gabor Szabo: right? That's what you recommend, or or why not attach the event to the document? Whatever.

83 00:14:52.800 --> 00:14:56.740 Eyal Lior: That's the event development. I mean, you can go and attend.

84 00:14:57.158 --> 00:15:07.630 Gabor Szabo: You can. You can write in the HTML on Click, or you can write some Javascript that will attach the click events to the that.

85 00:15:07.770 --> 00:15:22.919 Eyal Lior: Like, if the button would have an idea, I can tell. Get element by idea and then connect the event. Handler. That's that's that's fine, and that's the that's the thing with vanilla Javascript. You have many ways to make

86 00:15:23.160 --> 00:15:40.210 Eyal Lior: things, and people are afraid of it. Some people are afraid of it. I'm not afraid of it, but you do need to have some consistency in a large project. That's true. So the free formness, the free form, nature of Javascript in the browser

87 00:15:40.920 --> 00:15:41.900 Eyal Lior: is.

88 00:15:43.090 --> 00:15:49.664 Eyal Lior: It's a good thing for some people and a bad thing for for others. It's truly a matter of

89 00:15:51.310 --> 00:16:09.650 Eyal Lior: personal preference. But when you, when you work with Javascript vanilla, you don't need to study refs function components. There used to be this thing called class components, and it's not deprecated. But it's not used anymore in react. This was causing a lot of those problems in the past.

90 00:16:10.140 --> 00:16:20.660 Eyal Lior: Higher order components render props, custom hooks, mounting, updating component trees. Lifecycle re-renders use effect, use context.

91 00:16:20.780 --> 00:16:28.410 Eyal Lior: You state all those concepts and and basically all those pieces of technology that you would have to read

92 00:16:28.670 --> 00:16:32.329 Eyal Lior: many, many pages on each one of them to understand.

93 00:16:32.550 --> 00:16:36.340 Eyal Lior: just to make something as simple as a button click that

94 00:16:36.630 --> 00:16:39.680 Eyal Lior: updates updates the data on the document.

95 00:16:41.260 --> 00:17:01.519 Eyal Lior: So the way I see those frameworks is, it's like a if you know the Ruby Goldberg machine, the super complicated machine that that eventually makes something very simple. So you have jsx, which is, it looks like HTML, but it really isn't. HTML, you can't write anything that you put in. HTML in it.

96 00:17:01.670 --> 00:17:22.980 Eyal Lior: and then it goes into the framework. And then you have a piece of code like react, create element and a reference to my button, and in a different way of writing the properties and the output eventually what the framework does it translate to Javascript? So you can write in Javascript or have the framework create Javascript for you

97 00:17:23.300 --> 00:17:27.859 Eyal Lior: that that's what happens behind the scenes. The the framework does this thing.

98 00:17:28.089 --> 00:17:38.680 Eyal Lior: It creates a new element called button. It sets the style to blue. It sets the text to. I'm a button. Click me and then document body append, child. So

99 00:17:39.760 --> 00:17:43.031 Eyal Lior: I say, why have all this big machine

100 00:17:43.730 --> 00:17:48.210 Eyal Lior: generate something as simple as as this simple Javascript.

101 00:17:50.140 --> 00:17:57.810 Eyal Lior: So let's talk about things that you can't do with frameworks which is direct and synchronous dom manipulations

102 00:17:58.340 --> 00:17:59.870 Eyal Lior: inline scripts.

103 00:18:00.570 --> 00:18:05.480 Eyal Lior: Work without the build system like a bundler like webpack

104 00:18:08.230 --> 00:18:14.030 Eyal Lior: and not even getting into typescript global event handling and delegation,

105 00:18:17.340 --> 00:18:23.939 Eyal Lior: accessing and modifying the full HTML structure like the the way it appears in the browser after it's rendered

106 00:18:24.200 --> 00:18:28.290 Eyal Lior: and real time dynamic script and style loading.

107 00:18:29.460 --> 00:18:43.649 Eyal Lior: I wouldn't know. Maybe you can do those things, but you probably have to start to study the framework really good, like. It's not enough that you know how to do it in vanilla. You'll have to study the framework

108 00:18:44.110 --> 00:18:48.509 Eyal Lior: deeply in order to make things that are pretty trivial without the foam, board

109 00:18:48.740 --> 00:18:52.050 Eyal Lior: and custom, event creation and dispatching.

110 00:18:54.690 --> 00:19:02.589 Eyal Lior: And you know, like they they say, focus only on your business and and app logic. And that's what I'm striving for.

111 00:19:02.690 --> 00:19:07.793 Eyal Lior: So let's talk about my project, and enough enough.

112 00:19:08.500 --> 00:19:31.550 Eyal Lior: saying bad things about the frameworks. Let's say some good things about the vanilla book project. It's a non pervasive the guiding. I'm talking about the guiding principles of the project. It is non-pervasive. It's not limiting how you write code, how you write your screens or your modules. You can do anything you could do with vanilla.

113 00:19:31.650 --> 00:19:38.929 Eyal Lior: There's nothing that you can do otherwise with vanilla Javascript that you cannot do when you use vanilla boot.

114 00:19:39.680 --> 00:19:46.929 Eyal Lior: It has a minimalist approach, meaning you don't need to learn the guts of the framework to have full control of your app.

115 00:19:47.600 --> 00:19:51.609 Eyal Lior: It gives you the ability to create reusable components.

116 00:19:51.900 --> 00:19:54.936 Eyal Lior: And one of the features is that

117 00:19:55.560 --> 00:19:58.709 Eyal Lior: it downloads only the components that are being used.

118 00:19:59.258 --> 00:20:09.549 Eyal Lior: You can decide to preload some components in if you want to. Cache them for to have faster loading times but

119 00:20:09.730 --> 00:20:15.330 Eyal Lior: basically you might have an application with a hundred screens.

120 00:20:15.680 --> 00:20:36.419 Eyal Lior: and the user is only using 2 or 3 of them. You don't want to download all the application. So that's 1 thing that the application does that the the vanilla boot does it only downloads the components that are that you need. And I'm not sure that frameworks or the the framework like react, does it?

121 00:20:36.610 --> 00:20:38.330 Eyal Lior: It has this separation.

122 00:20:38.840 --> 00:20:43.169 Eyal Lior: Okay, there's no. J, 6. This is something that I really hated when I worked with react

123 00:20:43.300 --> 00:20:48.800 Eyal Lior: and and not sure if angular is using J. 6 as well, or something like that. But

124 00:20:50.920 --> 00:20:58.260 Eyal Lior: standard. HTML, so this on click thing that you put inside the HTML,

125 00:20:58.390 --> 00:21:07.429 Eyal Lior: this is something you couldn't get away with. If you use J. 6, meaning the thing that looks like HTML, when you develop in react.

126 00:21:07.930 --> 00:21:10.329 Eyal Lior: It's J. 6. It's not the HTML, but

127 00:21:10.580 --> 00:21:20.230 Eyal Lior: you won't use standard. HTML, I like to use standard HTML, so I can use all those and pretty cool little things like on click, or

128 00:21:20.340 --> 00:21:21.070 Eyal Lior: oh.

129 00:21:21.180 --> 00:21:50.640 Eyal Lior: on mouse down, or any kind of thing that you can do in native HTML, I want to preserve those functionality, those abilities. There's no activity, full control of rendering. Nothing happens like we've seen in the example before of React. Nothing happens without you explicitly making it happen. So there's no mystery. Things. Why does it happen? Why does the value reset what's happening? There's nothing gets called behind the scene unless you make the call

130 00:21:52.318 --> 00:21:56.600 Eyal Lior: so nothing is going to go unplanned. Then you like.

131 00:21:56.750 --> 00:22:02.049 Eyal Lior: Well, I have to read a hundred pages document to see what happened

132 00:22:02.740 --> 00:22:27.969 Eyal Lior: this. And there's a simple flow of data between the embedded components. I will show you an example really so real soon. And I've built an example that has a component. And then within it. There are 2 components of the same type, and within each one of them does a child component also of the same type, and they can all talk. They they can all all communicate, and it's pretty simple and intuitive.

133 00:22:28.870 --> 00:22:32.859 Eyal Lior: and there is free form flow of data according to your needs.

134 00:22:32.980 --> 00:22:38.399 Eyal Lior: I don't limit you to a way to make this communication work

135 00:22:40.390 --> 00:22:47.079 Eyal Lior: free form, component architecture, no surprises, nothing to learn, no complex flow required for your code.

136 00:22:47.760 --> 00:22:52.999 Eyal Lior: Every component can be different, fully flex. Full flexibility to this is something that

137 00:22:54.300 --> 00:23:00.180 Eyal Lior: I like to have when I get into building a very complex application is that

138 00:23:00.380 --> 00:23:17.990 Eyal Lior: if I if I know and and I am doing it right now. At work, I I'm building an application that has about more than 150 screens, different screens, different kind of logic and complications. And I don't want to limit

139 00:23:18.499 --> 00:23:23.300 Eyal Lior: or to decide from the get go from the start of the project.

140 00:23:23.790 --> 00:23:30.920 Eyal Lior: How I'm going to build screens in general I don't know. I might get. I might build 30 screens and then

141 00:23:31.648 --> 00:23:39.180 Eyal Lior: bump into a screen. That's really really complicated and has completely different element and needs a different flow.

142 00:23:39.440 --> 00:23:47.940 Eyal Lior: And vanilla boot lets you have like a world of its own in each component. There's no

143 00:23:54.530 --> 00:23:58.540 Eyal Lior: how, how, I don't know how to to explain, but there's no

144 00:23:59.825 --> 00:24:04.289 Eyal Lior: they don't need to. Be like like like one another.

145 00:24:06.000 --> 00:24:11.909 Eyal Lior: Okay, so there's no virtual dom. There's no built in state management like like we talked about.

146 00:24:12.500 --> 00:24:19.558 Eyal Lior: So you don't need to learn the remarks in a walking. I know I'm repeating myself a little bit.

147 00:24:20.410 --> 00:24:23.820 Eyal Lior: The framework is written in short and simple code.

148 00:24:24.090 --> 00:24:31.809 Eyal Lior: There's no stratified object. Oriented hell. It's it's about 370 lines of code. That's it.

149 00:24:31.930 --> 00:24:54.260 Eyal Lior: Right now. I mean, it's a small. It's an early age. It's a it's early stage library. It will grow. But I'm not putting a lot of code in it. I'm trying to keep it minimalist and easy. So even if you would need to look into it inside of the library. It's short, it's simple, and it's not

150 00:24:54.590 --> 00:25:03.220 Eyal Lior: built in in too much of overly designed that you can't really tell. Well, when does this code get called by? Who

151 00:25:04.250 --> 00:25:05.949 Eyal Lior: no transpiling

152 00:25:06.550 --> 00:25:14.950 Eyal Lior: meaning? When when a code isn't transpired, you can just press F, 12 and and debug the code as it is written.

153 00:25:16.090 --> 00:25:23.770 Eyal Lior: So it's easier to investigate the code. In the browser environment. There's no chunking to incoherent little pieces of code

154 00:25:24.310 --> 00:25:31.860 Eyal Lior: and due to no framework as a base and fuel, the dependability

155 00:25:33.180 --> 00:25:40.849 Eyal Lior: and pilties. What is it right here we get long term reliability. So because this is such a small minimalist framework.

156 00:25:40.980 --> 00:25:48.770 Eyal Lior: and most of your code can eventually be even migrated to another library, or you can just

157 00:25:49.390 --> 00:26:00.640 Eyal Lior: move out and leave you. You can migrate from veneer, but really easily, because the code that you write for your application is not

158 00:26:01.630 --> 00:26:04.660 Eyal Lior: built in a very specific way

159 00:26:04.930 --> 00:26:11.829 Eyal Lior: for vanilla boot, vanilla boot give lets you build any way you want your code, your your application logic.

160 00:26:13.025 --> 00:26:16.520 Eyal Lior: So you don't marry the framework.

161 00:26:16.630 --> 00:26:38.999 Eyal Lior: You can leave it or customize it, and you still have your code detached from any kind of logic that you would have if you would use. For example, I use this example because I think 90% of the market use react today when you use react, you build the code around the structure of

162 00:26:39.360 --> 00:26:44.189 Eyal Lior: of those there are components

163 00:26:45.340 --> 00:26:53.760 Eyal Lior: and your code is split and the the flow of data is how a react program expects it to be.

164 00:26:53.940 --> 00:26:58.023 Eyal Lior: And here, rather, it's more

165 00:27:00.200 --> 00:27:07.220 Eyal Lior: you're you're free from this limitation, and you're not marrying the the library.

166 00:27:07.350 --> 00:27:15.230 Eyal Lior: and I want you to keep using it. But but it's I think it's will be much easier to keep being in independent.

167 00:27:16.240 --> 00:27:26.109 Eyal Lior: Okay? So you can create components completely different strategy for each one of them embed. You can embed components in palm components

168 00:27:26.600 --> 00:27:27.514 Eyal Lior: and

169 00:27:29.040 --> 00:27:38.660 Eyal Lior: communicate between components. You can load multiple components again, again, meaning multiple instances of instances of the same component.

170 00:27:38.980 --> 00:27:45.449 Eyal Lior: You can load dynamic javascript or css from any location.

171 00:27:47.151 --> 00:27:51.969 Eyal Lior: Let's let's move on. What about plugins and 3rd party stuff

172 00:27:52.410 --> 00:28:14.129 Eyal Lior: like I said before, there are plenty full of extensions available for vanilla Javascript equal or more than those available for framework. So here's an example of of libraries that I like, and you can use with vanilla Javascript, one js, which gives you actually the ability to create reactive reactive Ui elements

173 00:28:14.675 --> 00:28:17.920 Eyal Lior: without react. So you can, you can actually

174 00:28:18.540 --> 00:28:30.470 Eyal Lior: use Van Js with vanilla Van Js is not mine. It's another project you can like, create dynamically from code. HTML, elements like you would have like you would do with react.

175 00:28:30.770 --> 00:28:35.529 Eyal Lior: So you can add Van Js into vanilla boot.

176 00:28:35.660 --> 00:28:39.459 Eyal Lior: and have all those reactivity abilities

177 00:28:40.080 --> 00:28:50.870 Eyal Lior: in one model in your system or in many modules. But you would not be limited to choose. Okay, I'm I want to activity. So all my application has to be reactive.

178 00:28:51.460 --> 00:28:58.709 Eyal Lior: You can have. There's a project called Vanilla Router, which gives you the same routing mechanism that you have in in

179 00:28:58.830 --> 00:29:11.889 Eyal Lior: frameworks like angular and react ag grid, a great grid technology that I use a lot at work and and for other projects is available for vanilla.

180 00:29:12.410 --> 00:29:31.639 Eyal Lior: W. 2 ui and golden layout, 2 2 very good technologies for building windowing systems in Javascript, where you can have draggable windows that you can put your load your modules in into, and I'm plan. I plan to support them in in vanilla boot. So you can actually

181 00:29:31.850 --> 00:29:33.379 Eyal Lior: very easily write

182 00:29:34.572 --> 00:29:42.620 Eyal Lior: your own components that open inside those windows, you can integrate paypal stripe moment or yesterday

183 00:29:42.790 --> 00:29:47.429 Eyal Lior: digest and data visualization technology. So

184 00:29:48.020 --> 00:29:53.310 Eyal Lior: the idea that you need to use react or angular because they have 3rd party libraries.

185 00:29:53.700 --> 00:29:55.939 Eyal Lior: absolute nonsense. So let's

186 00:29:56.130 --> 00:30:04.859 Eyal Lior: get to the most important part. The example. Here's an example, and and I will. I will show it in a minute. In the browser.

187 00:30:05.020 --> 00:30:06.810 Eyal Lior: What we see in this example

188 00:30:07.150 --> 00:30:12.520 Eyal Lior: is, we see a root component, which is

189 00:30:13.460 --> 00:30:16.229 Eyal Lior: the the whole big black box.

190 00:30:16.610 --> 00:30:22.129 Eyal Lior: and within it there are 2 parent components, and each parent has 2 children.

191 00:30:22.710 --> 00:30:26.890 Eyal Lior: So there are 3 types of components here, there's the root, the parent and the child.

192 00:30:27.130 --> 00:30:32.960 Eyal Lior: and those concurrent instances like we had 2 parents and 2 childs within each parent.

193 00:30:34.010 --> 00:30:45.679 Eyal Lior: and we have here an ordered loader of dependencies. So suppose one of your application windows needs to load many javascript external Javascript libraries.

194 00:30:46.090 --> 00:30:53.209 Eyal Lior: and you need them to load in a certain order one before the other, and you maybe you want to execute

195 00:30:53.350 --> 00:31:04.149 Eyal Lior: some some custom Javascript code after you load some library, and before you load the next one, this is something that vanilla Boots lets you do, and I I show it in the example.

196 00:31:05.710 --> 00:31:13.910 Eyal Lior: Now let me just switch window for a moment and show you the example in the browser.

197 00:31:17.160 --> 00:31:24.950 Eyal Lior: Okay, so this is the live example. Every time I reload it it gets a random column. The random column comes from

198 00:31:25.280 --> 00:31:45.469 Eyal Lior: an external dependency. Now, I'm not going to get into the whole source code. It's not complicated at all. If you can get into the example is, live on the website, and you can do a control U or F 12, and get into the source and and view how it's done, and view how the components are loading. But

199 00:31:46.000 --> 00:31:58.800 Eyal Lior: the thing is, every child component has 2 buttons. Call myself, and call p. 2 c. 1. They they all have call p. 2, parent, 2, child, one. So if I call myself

200 00:31:58.990 --> 00:32:01.250 Eyal Lior: I called my. It says I called myself

201 00:32:01.740 --> 00:32:06.639 Eyal Lior: call myself call myself. Now, if I go here and I say, call p. 2 c. 1,

202 00:32:07.260 --> 00:32:10.510 Eyal Lior: it's going to say, call from child 2.

203 00:32:10.690 --> 00:32:14.159 Eyal Lior: It should say, call from child. Parent one, child, 2.

204 00:32:14.320 --> 00:32:15.510 Eyal Lior: Here it says.

205 00:32:16.140 --> 00:32:33.070 Eyal Lior: Hi, p. 1. i'm c, 1 or Hi, parent one. I'm child, one, and then Hi, parent one, I'm child 2, so we can see communication between the children and the parent and children and children in completely different place

206 00:32:33.990 --> 00:32:35.180 Eyal Lior: in the 3.

207 00:32:36.820 --> 00:32:39.620 Eyal Lior: Let's go back to that presentation.

208 00:32:42.480 --> 00:32:51.970 Eyal Lior: So just I'll get a little bit into the example. So once you want to investigate it yourself, you know what's going on.

209 00:32:52.110 --> 00:32:57.500 Eyal Lior: The 1st the the HTML that you're loading. It's it's called example. HTML,

210 00:32:58.110 --> 00:33:00.470 Eyal Lior: it loads the vanilla book library.

211 00:33:01.220 --> 00:33:07.810 Eyal Lior: It has a div that takes all the width and the height of the screen. We call it example layout.

212 00:33:08.010 --> 00:33:11.289 Eyal Lior: and that's the div where inside everything is loaded.

213 00:33:11.460 --> 00:33:18.579 Eyal Lior: And that's the start of the example. It says, it starts the library and gives it the reference.

214 00:33:18.770 --> 00:33:22.449 Eyal Lior: the to the Div, where we want everything to happen.

215 00:33:22.560 --> 00:33:27.119 Eyal Lior: and then we say, show module, and we pass the name of the module.

216 00:33:27.480 --> 00:33:32.120 Eyal Lior: It's called test grandparent. So basically, there's grandparent parent and child.

217 00:33:32.690 --> 00:33:38.110 Eyal Lior: and we want to 1st load a component called test grandparent.

218 00:33:38.310 --> 00:33:44.369 Eyal Lior: So what the lib. What vanilla boot does is it goes to a library to a folder called

219 00:33:44.650 --> 00:33:59.410 Eyal Lior: that name test grandparent, and it looks for 2 things. This is the the minimal. The minimal model has an HTML file and a Javascript file with the same name as as the folder, and it loads the HTML,

220 00:34:00.160 --> 00:34:07.050 Eyal Lior: and it scans it scans the HTML for divs that have class.

221 00:34:07.260 --> 00:34:35.570 Eyal Lior: The class name vanilla boot component. Whenever there's a div with vanilla boot component as a class. It knows that it's it's a component that needs loading, and it has a name and an idea. And then we go. We say, Okay, the name is test parent. So we go to test parent and load, test parent within test. Parent, you will have the same thing. You'll have a div with the class vanilla boot component, and the name test child. And this is how the child is being loaded.

222 00:34:39.179 --> 00:34:43.920 Eyal Lior: Each component has a vanilla boot component as a prototype.

223 00:34:45.050 --> 00:34:50.050 Eyal Lior: So, for example, if we have the child, this is the the code for the child component.

224 00:34:50.668 --> 00:34:53.109 Eyal Lior: The constructor gets an element.

225 00:34:53.480 --> 00:34:57.909 Eyal Lior: an in an element in the HTML and a link to the parent.

226 00:34:58.150 --> 00:35:00.750 Eyal Lior: and it caused the the

227 00:35:02.680 --> 00:35:08.632 Eyal Lior: the superclass constructor. And then here we see the dynamic loading of of

228 00:35:11.660 --> 00:35:20.370 Eyal Lior: of dependencies. So we're loading a Css file, and then we're loading a javascript. But

229 00:35:20.630 --> 00:35:31.030 Eyal Lior: right after the Javascript is loaded we have an event handle here a function. This will be called after this dependency sample dependency is loaded.

230 00:35:31.920 --> 00:35:39.530 Eyal Lior: and then we have the sample dependency, 2 loaded. Now, if you go to the sources, and you look what it does. Sample dependency dot Js

231 00:35:39.700 --> 00:35:45.730 Eyal Lior: colors of the children in one color. I think it's red or something, and then sample dependency, too.

232 00:35:46.970 --> 00:35:56.619 Eyal Lior: puts a random color to each child. So when the when the example is loading and you see a random color, you know that that

233 00:35:56.730 --> 00:36:01.500 Eyal Lior: sample dependency 2 is the last Javascript loaded.

234 00:36:02.080 --> 00:36:09.690 Eyal Lior: It's kind of like proving the the order of the dependencies and do start is the

235 00:36:09.920 --> 00:36:15.019 Eyal Lior: is a function that is called for every component. When all the dependencies finish to load.

236 00:36:18.140 --> 00:36:21.769 Eyal Lior: we are getting close to the end of the presentation.

237 00:36:23.190 --> 00:36:26.980 Eyal Lior: This is how communication between child and parent work.

238 00:36:27.220 --> 00:36:30.460 Eyal Lior: The component that has a link to the parent

239 00:36:30.780 --> 00:36:43.900 Eyal Lior: can simply call any method you choose to implement in the parent. So I call this method call from child. But this is any method that you would put in the parent component, you can access it directly.

240 00:36:44.190 --> 00:36:49.050 Eyal Lior: and then, once you access it. The parent has the reference to the child instance.

241 00:36:50.440 --> 00:37:00.009 Eyal Lior: So the parent has a function called call from child, and it gets a reference to this to the child and the text that we want to print.

242 00:37:01.660 --> 00:37:12.639 Eyal Lior: and then the the in the parent component you can call a function that you create in the child called parent component, we'll call. This

243 00:37:12.900 --> 00:37:16.959 Eyal Lior: could be anything you put anything, any functions you want to implement.

244 00:37:17.690 --> 00:37:34.300 Eyal Lior: But there is also a mechanism that a utility mechanism that I provide called find component. So if you want to communicate between components without any regard for where they are placed, you just give it the hierarchy. So you say, parent 2

245 00:37:34.660 --> 00:37:42.240 Eyal Lior: CPU, 2 c. 1. So it's the hierarchy of the the idea of the com components.

246 00:37:42.490 --> 00:37:55.110 Eyal Lior: So you can. You, you just every component that loads the library knows the structure, the the 3 of components that that are loaded, and you can access them by just the name

247 00:37:57.550 --> 00:38:01.700 Eyal Lior: final thoughts, vanilla vanilla boot is currently only

248 00:38:01.900 --> 00:38:08.910 Eyal Lior: 370 lines of code. The code is sourced from existing complex and mature projects which serve as repository

249 00:38:09.010 --> 00:38:26.380 Eyal Lior: phone, incremental dismantling and reuse. So basically, this is things that I've used when I built other projects and I dismantled them and brought into vanilla boot to make them more more general and and universal, so other people can use the same mechanism

250 00:38:31.050 --> 00:38:34.900 Eyal Lior: process is carried out methodologically, without haste with each addition.

251 00:38:35.540 --> 00:38:48.125 Eyal Lior: followed by careful observation of user adoption and identification of potential potential the factory needs before further investigation. Right now, it's an early stage project and

252 00:38:50.030 --> 00:39:01.600 Eyal Lior: I don't know where it's going to develop. We'll see who who adopts the project and what feature requests I'm getting. There's certainly things that will

253 00:39:01.930 --> 00:39:11.599 Eyal Lior: that it cannot do right now. I mean, it's not it's not ready to conquer the world, certainly, but for many, many users.

254 00:39:12.110 --> 00:39:24.220 Eyal Lior: for example, if, like me, you are building an application that's going to use be used inside an organization. And it's not going to be a website that's used by millions of concurrent users

255 00:39:24.620 --> 00:39:30.340 Eyal Lior: and most of your applications. Most most of your pages will have one or 2 components.

256 00:39:30.750 --> 00:39:35.209 Eyal Lior: But you're gonna have a hundred or 150 or 200 screens.

257 00:39:35.630 --> 00:39:39.090 Eyal Lior: Then it's perfect. It's perfect already.

258 00:39:39.590 --> 00:39:41.799 Eyal Lior: But if you're going to build a website.

259 00:39:42.300 --> 00:39:49.720 Eyal Lior: and you're gonna have like 50 components all loaded. Once you might want to think about how to

260 00:39:50.130 --> 00:40:09.729 Eyal Lior: pack the components together and not have it load, create 1, 50 or 100 htp. Requests for every component. So the framework is in is in early stage. It's not fit for any kind of use, but it it can be used for most most it projects

261 00:40:10.332 --> 00:40:14.959 Eyal Lior: especially ones that are not for public big websites.

262 00:40:15.190 --> 00:40:16.200 Eyal Lior: It's good.

263 00:40:16.940 --> 00:40:26.410 Eyal Lior: So is it too early to adopt? I would say, no, take it, adopt it. It's really easy and gives you the ability to you. Build in vanilla Javascript.

264 00:40:29.770 --> 00:40:36.459 Eyal Lior: So thank you for your attention, and please visit the website and look at the example. Try to look at the source of it

265 00:40:36.570 --> 00:40:39.409 Eyal Lior: and download it and play with it. And I'm

266 00:40:39.820 --> 00:40:44.440 Eyal Lior: you for any questions, any support that's needed. It's open source.

267 00:40:47.390 --> 00:40:49.249 Eyal Lior: Use it in any way you want.

268 00:40:52.000 --> 00:41:00.040 Gabor Szabo: Well, thank you for your presentation. If there are any questions, then please ask those from

269 00:41:00.260 --> 00:41:02.199 Gabor Szabo: from the people who are here in the

270 00:41:02.570 --> 00:41:07.190 Gabor Szabo: in the presentation. Please ask now. But as we didn't have any

271 00:41:07.520 --> 00:41:10.680 Gabor Szabo: questions during the talks, I guess.

272 00:41:11.760 --> 00:41:12.920 Gabor Szabo: People.

273 00:41:13.220 --> 00:41:18.679 Gabor Szabo: we're satisfied with what what you you said and how you presented it. So thank you very much.

274 00:41:18.890 --> 00:41:26.469 Gabor Szabo: and thanks for everyone who was here at the meeting.

275 00:41:26.720 --> 00:41:40.739 Gabor Szabo: And if you watch the video and you enjoyed it. Then please click on the like button and also follow the channel definitely. So you'll get notified below the video. You will find links to

276 00:41:41.580 --> 00:41:48.165 Gabor Szabo: where to find the the project and and how to get in touch with a and

277 00:41:49.150 --> 00:41:52.400 Gabor Szabo: I think that's it, for now any last words.

278 00:41:52.920 --> 00:41:58.284 Eyal Lior: Thank you for the opportunity to present it. I am very excited about this project, and

279 00:41:59.010 --> 00:42:04.220 Eyal Lior: I think a lot of people will be very happy about it, because it's

280 00:42:04.500 --> 00:42:12.049 Eyal Lior: it becomes hard today to be a vanilla Javascript developer when everybody's using react.

281 00:42:12.310 --> 00:42:16.310 Gabor Szabo: Yeah, I don't use any of the frameworks, so I'll build. I don't

282 00:42:16.430 --> 00:42:26.000 Gabor Szabo: build big things, usually a very lightweight. The the Javascript part. What I I do, but then I usually

283 00:42:26.150 --> 00:42:36.590 Gabor Szabo: either use vanilla Javascript, or or maybe I use Jquery, and but even that is not really necessarily in most cases

284 00:42:36.790 --> 00:42:42.930 Gabor Szabo: so, and thank you very much, and good luck with the project.

285 00:42:43.240 --> 00:42:43.960 Eyal Lior: Thank you.

286 00:42:44.210 --> 00:42:45.620 Gabor Szabo: And bye, bye, everyone.

287 00:42:46.180 --> 00:42:46.930 Eyal Lior: Joe.

This is really scary.

I am trying to figure out why would any developer want to limit the length of the password you use.

I can think of two explanations.

One is that someone thought that "it would be hard for you to remember a longer password so we limit it to 8 characters". Is this patronizing? Where would this come from?

The other, much worse is that they have allocated a given size of space (e.g. 8 characters) in the database where they store the passwords and they want to make sure yours will also fit in. This is really scary as this would indicated that they store the password as you typed in. In clear text. That's a huge security hole as that means anyone who can see the content of the database will have all the passwords. So for example if the admins store a backup of the database on some place that less secure than the original database server then someone might be able to access that.

The correct way to store passwords is to use some kind of a strong hashing algorithm. The nice feature of all the hashing algorithms is that they take any arbitrary long string and convert it to a fixed length string. For example sha256 will give you a 256 bit (32 bytes) long string. Regardless of the length of the input. So your password can be 10 characters or 20, after the conversion it will be 32 bytes long and that's what the developer needs to store in the database. Maybe with a few extra characters.

So the developers can fix the allocated place in the database and they can be sure that regardless of the length of the real password they will only need to store the expected number of bytes.

1. Personal Bookmarks
2. URL Shortener
3. TODO list
4. Shopping list
5. Repeated shopping list
6. Flight ticket vendor
7. Microblog
8. SSG - Static Site Generator
9. Job search manager

Add company, link to job ad.

Maybe includ details of the job ad (title, etc) copied from the ad.

List of events with the specific job post:

When you saw it.

When you submitted your CV.

How did you submit your CV? (Via LinkedIn, via the web site of the company, via a recruiter company, etc.)

When did you talk to them (with whom did you talk, notes you made)

Add a trigger, a TODO item to each company. (with specific date or without)

Allow the display of everything about a company.

Show the list of companies.

Show the list of TODO items.

Mark a TODO item as done. (and maybe turn into an even with notes.)

In this system we will have a "master list" where we only have the types of items. Each item can be either without the amount or with the range of amounts based on our past purchases.

When the user wants to start a new purchase s/he can do so either from an empty list, from the list of all the items in the master list or a list from a previous purchase.

The user can add new items to the current list. These items are also automatically added to the "master list".

The user can change the amount of each item in the current list. The user can also remove items from the current list.

The user can also direct add/remove items in the "master list".

There can can be other users in the system who can suggest items to the current list. (e.g. the kids want to get sweets). The main person will see who suggested each iterm.

Using Python

The Python SDK of SurrealDB

• Getting started with SurrealDB using Python and Docker
• CLI phone book in Python using SurrealDB as a database

Using Rust

• SurrealDB using Rust

Like this:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          7         2         5.129GB   4.806GB (93%)
Containers      3         3         12.46MB   0B (0%)
Local Volumes   4         3         7.627GB   230MB (3%)
Build Cache     165       0         2.378GB   2.378GB

Using the -v flag like this : docker system df -v shows the same information in way more details that can help you understand what part of Docker uses all that data.

Except in my case, the folder where Docker stored its data used about 20 times more space than reported by Docker. Naturally I only noticed it when my server ran out of disk-space putting me in panic-mode.

It took me a while till I realized there are some huge log-files in the containers folder.

I am sure you know that you can look at the output that was printed to the console of your container by running the command:

docker container logs NAME_OF_A_CONTAINER

As it turns out this output is saved in a file in the folder of the individual container and by default it is kept forever.

docker system df can provide detailed reports on disk usage, but it does not include the disk space used by the log files.

One of the services I used was very, I mean really talkative and printed tons of stuff on the console. Docker shoved all of that into an ever growing file.

Solution

The immediate solution was to stop docker and remove the huge file.

The longer-term solution was to configure Docker to keep the log files limited in size.

It can be done in the daemon configuration file which, at least in my installation is located in

/etc/docker/daemon.json

I already had this in the file:

{
    "data-root": "/home/data/docker",
}

That's because a while ago I moved all the data to a disk separate from the disk of the operating system.

Now I updated it:

{
    "data-root": "/home/data/docker",

    "log-driver": "json-file",
    "log-opts": {
        "max-size": "10m",
        "max-file": "3"
    }
}

Then I restarted the Docker daemon with

sudo systemctl restart docker

and I also had to stop and remove the existing containers and start new ones!

Without that the changed did not take effect.

To tell the truth I also rebooted the host machine. It is probably not necessary, but I like to do that after such changes to verify that the computer can be booted and all the services start as expected. I rather have a few seconds downtime when I still remember the changes I made than later when something breaks because of some changes a few weeks or month earlier.

For more details see how to configure logging drivers for Docker.

If you visit the web site the only indication as to how to contribute to it I found was a small GitHub logo at the bottom that leads to the GitHub repository of VS Code

As of this writing on 2023.06.18, it had more than 110,000 commits by 1,855 contributors. It also has 5K open issues and 405 open pull-requests. It is based on Electron and it is written in TypeScript.

In the README file in GitHub repository you can find a description on areas where one can contribute.

• Meta::CPAN (Perl, ElasticSearch)
• Visual Studio Code (Electron, TypeScript)

What are the goals?

Each 3rd-party registry provides some meta-data about the projects. e.g. information about the license of the project, the requirements of the projects, a link the public VCS (Version Control System) of the project. However, many of the libraries don't supply this data. Not including a link to the VCS makes it unnecessarily difficult to find it and to contribute to the project. Thus one of the first things we try to map is the list of libraries that do NOT have that link. Then we encourage contributors to find the repos and send a pull-request to include the link.

For the projects where we have a link to the VCS we can also check if they have CI (Continuous Integration) configured.

We can show how important each package is by counting all the other packages that depend on it.

Where it is available we also display the test-coverage report of each project.

Why separate projects?

Each Digger is written in the language of the libraries. Partially because I feel this as a good opportunity to learn the language and partially because that will make it easier for people who write those 3rd-party libraries to contribute to the respective projects. In addition each language has tools accessing the the registry and analyzing the code in their language. So writing in the "native" language gives us access to those tools.

Current projects

• Python - source
• Perl - source
• Ruby - source

Planned projects

• PHP - source

• Rust

• JavaScript / NodeJS source

This is not a problem during regular software development because you rarely need to install extra system-packages. Most (all?) programming languages have a tool to install their own packages as regular user. However, there are cases, especially when you first try to create a Docker image to run a particular piece of software, that you will need to install several system-level packages and also run the code of the project that might create files in the current working directory which is mapped to the external disk.

I found two solution for this. Both involves switching between regular user and root user but they work in different ways.

The first solution does not need any extra installation, but you probably need more self-discipline and in a way I feel it is more error-prone.

The second needs us to install sudo.

Without sudo

In this version we return to launching Docker and being user root inside.

$ docker run -it -w /opt -v$(pwd):/opt --name ubu ubuntu:23.04 bash

Now we are user root. At this time we can install extra packages. e.g. installing less:

# apt-get update
# apt-get install -y less

The we can switch to the regular ubuntu user and work as regular user inside Docker (e.g. create a file in the folder that was mapped to the external disk):

# su - ubuntu
$ cd /opt
$ touch hello

If we would like to install another system-wide package, we need to use exit to leave the ubuntu user, then we can install more packages and can get back to our regular user again with su - ubuntu. (The - tells Linux to set up the full environment of that user.)

In order to leave the Docker container we need to execute exit twice. First to exit the regular user to become root and then to exit the container.

With sudo

For this solution we need a little-bit of setup. We have the following Dockerfile:

examples/docker-sudu/Dockerfile

FROM ubuntu:23.04
RUN apt-get update && \
    apt-get install -y sudo && \
    usermod -aG sudo ubuntu && \
    echo "ubuntu  ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

• We install sudo.
• Add the user ubuntu to the group that can use sudo.
• We add ubuntu to the sudoers file and allow it to execute commands without providing a password.

We build the image:

docker build -t myubuntu .

Then run it as the user ubuntu

docker run -it -w /opt -v$(pwd):/opt --name ubu --user ubuntu myubuntu bash

Now we use it as the regular user ubuntu.

$ touch file_in_mounted_folder

If we would like to install as system-wide package we can use sudo:

$ sudo apt-get update
$ sudo apt-get install -y less

When we would like to exit we need to type exit only once.

When you run Docker on a linux host, mount an external folder, all the files created in the docker will be owned by user root on the host file-system.

Finally I think I understood how this works and found a solution. At least a partial solution for Ubuntu and CentOS-based images.

The Problem

Here is the problem again:

I run docker and mount the current folder as /opt inside the container:

$ docker run -it --rm -w /opt -v$(pwd):/opt ubuntu:23.04 bash

When inside the container I create a file and then exit the container:

# touch hello
# exit

If I look at the ownership of the file on my host computer (which is also Ubuntu):

$ ls -l hello

I get:

-rw-r--r-- 1 root root 0 Apr 26 20:35 hello

The file is now owned by user root.

Solution

Add the --user ubuntu parameter to the command:

$ docker run -it --rm -w /opt -v$(pwd):/opt --user ubuntu ubuntu:23.04 bash

Inside the container create a file and exit:

$ touch world
$ exit

Outside, on the host, check the ownership:

$ ls -l world
-rw-r--r-- 1 gabor gabor 0 Apr 26 20:38 world

It is now owned by gabor which is my regular user.

Explanation

In Linux each user has a user id. The user root has the id 0 on every Linux machine and I guess also on macOS, but I have not checked it. So both on my host computer and inside the Docker container the user root has the same id.

When I ran docker in the first example (without passing the --user parameter), inside the container I became user root. Any file I created then got the userID 0 as its owner. This is also the user root outside, so that's why the file is owned by root on the host system.

The first real user on Ubuntu (and I think in general on every Linux and Unix system) will get the id 1000. So the user ID of my user "gabor" on my host computer has the ID 1000. I could verify this by running the id command on my computer.

The Docker image 23.04 also comes with a default real user account which is called ubuntu and it also has the ID 1000. When in the 2nd example I used the --user ubuntu flag I entered the container as user ubuntu. If I ran the id command there I'd see:

$ id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev)

showing that this user in the container also has the ID 1000.

I found out about this by listing all the users using the cat /etc/passwd command. It is the last row.

So when I created a file on the mounted volume the file was created as user ubuntu (id 1000) inside the container. However on my host machine that id belongs to my regular user, so my computer sees that file as own by user gabor (id 1000).

Just as if I created it as my regular user on the host computer.

This solves the problem.

Notes and warnings

If on the host system your ID is not 1000, then this will not work, but you can create a user in the container with the ID you have and use that username. (See below for CentOs.)

Now that inside the container I am user ubuntu I don't have the privileges of user root so I cannot make changes to any of the system files. In particular I cannot install any extra software using apt. However, there are at least two ways to solve that. See Docker as both a regular user and as root (with and without sudo).

CentOS

After a little pause I though I'll try CentOS as well.

I ran

$ docker run -it --rm -w /opt -v$(pwd):/opt  centos:7 bash

and then inside the container

# cat /etc/passwd

This showed that all the user IDs are below 1000, they are all system-related users. So I need to create a user.

I created a Dockerfile with the following content:

FROM centos:7
RUN useradd centos

That RUN command will create a user with the username centos:

Built my own images:

$ docker build -t mycentos .

Ran docker using that images:

$ docker run -it --rm -w /opt -v$(pwd):/opt --user centos mycentos bash

Inside the containe I first checked the id then created a file, then exited:

$ id
$ cat /etc/passwd
$ exit

Outside I checked:

$ ls -l cent
-rw-rw-r-- 1 gabor gabor 0 Apr 26 22:26 cent

It looks fine.

Conclusion

On Ubuntu there is already a user called ubuntu, on CentOS there is no user, but we can create one easily. I assume on any other image you would either already had a user or you could create one.

This one actually had a lot of details in the README, but some parts were still missing, or at least were assumed to be obvious, which was not the case for us.

Clone the repo

git clone git@github.com:Rdatatable/data.table.git
cd data.table

Start Docker container

docker run -it --name data-table --workdir /opt -v$(pwd):/opt r-base:4.2.3 bash

Install external dependencies we'll need

apt-get update
apt-get install -y pandoc curl libcurl4-gnutls-dev texlive-latex-base texlive-fonts-extra texlive-latex-recommended texlive-fonts-recommended

Install R packages

Rscript -e 'install.packages(c("knitr", "rmarkdown", "pandoc", "curl", "bit64", "bit", "xts", "nanotime", "zoo", "R.utils", "markdown"))'

Run the build

R CMD build .

Run check on the generated file

In the README they mention data.table_1.11.5.tar.gz, but probably due to a change in the version number, now we have data.table_1.14.9.tar.gz

We can run this

R CMD check data.table_1.14.9.tar.gz

But if the README is updated with this then it will be out of date soon. Instead there could be an explanation that one needs to look at the generated file. Alternatively this command would pick up the current file, assuming there is only one of them.

R CMD check $(ls -1 data.table_*)

Exit the Docker container

exit

Restart the Docker container

docker container start -i data-table

Remove Docker container

docker rm data-table

Disclaimer: none of the participants had any experience in contributing to open source projects in R so we might have totally misunderstood how one should set up an environment.

First we looked for a project to play with. Checked the r projects, then looked at the awesome-R list and found r-yaml. We thought a library dealing with YAML files will be simple to install and test.

One thing we noticed quite quickly is the the compile target was missing from the Makefile. That's however is a minor issue in the documentation.

Then we spent quite a lot of time figuring out why things are failing. After about an hour of failures we felt it would be better to open an issue. Within a few minutes, maybe half an hour we got a response that solved the problem.

So here is how one could create a local development environment in a Docker container:

Clone the repo

git clone git@github.com:vubiostat/r-yaml.git
cd r-yaml

Start Docker container

Start a Docker container using an image based on R version 4.2.3 that was listed on Docker Hub.

Give the container a name "r-yaml" to make it easy to reuse it.

Designate the internal /opt folder as the workdir.

Map the current working directory to the internal /opt

I have not tried it, but I think MS Windows users will have to replace the -v$(pwd):/opt part by -v %cd%:/opt if using cmd or by -v ${PWD}:/opt if using PowerShell.

Run bash in the container.

docker run -it --name r-yaml --workdir /opt -v$(pwd):/opt r-base:4.2.3 bash

Install RUnit

Inside the container install the R dependency.

Rscript -e 'install.packages("RUnit")'

Install the external dependencies

apt-get update
apt-get install -y texlive-latex-base texlive-fonts-extra texlive-latex-recommended texlive-fonts-recommended

Run the tests

make check
make test

Exit from the container

exit

This will also stop the container.

Restart the container

We can restart the stopped container:

docker container start -i r-yaml

Remove the container

If we don't need the container any more we can remove it (after we exited and it was stopped).

docker rm r-yaml

Clean the environment

sudo git clean -dxf

This will print:

Removing build/

Conclusion

I hope this description will help someone else setting up the environment. I'll also suggest to include this, or something similar in the README file of the project.

This certainly helped me and next time it will be easier to set up a development environment for an R-based project.

For each GitHub project you can set up something they call GitHub Actions, even the link in every project is called "Actions". So if you happen to visit the source of the Code Maven web site then you'll have a link called Actions.

However what you see there is actually the output of the GitHub Workflows, or as I see they put it GitHub Actions workflows.

Each such workflow is a combination of some code and some configuration of GitHub Actions. However in this sentence the name "GitHub Actions" referred to the building blocks of the workflows.

In a nutshell GitHub Actions refer to two things. The workflows and the building blocks of the workflows.

Earlier, in most of my blog posts and training materials I used the name GitHub Actions to refer to the whole thing, but I'll need to reconsider this to make it easier for people to understand why the actions (workflows) are defined in a folder called workflows and why are they using building blocks that are also called "GitHub Actions" despite not being workflows.

Workflows

A GitHub Actions workflow is something you define in a YAML file in the .github/workflows/ folder of your git repository. It can be built from plain commands in Bash or basically any programming language, but it can also use a huge set of building blocks called Actions.

Actions

The Actions provided by Github are all stored in the action GitHub organization. Each action is a separate repository.

Everyone can create additional actions that can be private or can used by anyone. There is a whole Marketplace of Actions where you can locate more of these pre-built blocks.

As of this writing on 2023.03.19 there are 17,685 actions in the marketplace and probably there are lot more that were not published to the marketplace.

Having a good description of the contribution process and the culture of the development team is crucial, so is making it easy for a new potential developer to set up the local development environment and run the tests of a project locally.

It took me a while to find a PHP project that was simple enough for me to set up on a single Docker image, but after some trial and error I found the Twig project.

Funnily, only now, after managing the setup and staring to write this post did I realize that it is part of the Symfony project.

Anyway, I won't bother you with all the failed attempts I had till I get to this:

clone

Clone the Git repository of Twig

git clone git@github.com:twigphp/Twig.git

Docker container

Start a plain Ubuntu-based docker container in interactive mode and map the current working directory to the /opt directory inside the container.

My host system is itself an Ubuntu Linux machine. The command should also work on macOS.

I have not tried it, but I think MS Windows users will have to replace the "-v$(pwd):/opt" part by ```-v %cd%:/opt if using cmd or by -v ${PWD}:/opt if using PowerShell.

docker run -it --rm --workdir /opt -v$(pwd):/opt ubuntu:22.10 bash

Install PHP and tools

First we update the list of available Ubuntu packages.

Then we install tzdata. Normally it would ask questions during installation. The two environment variables provide the values to these questions.

Finally we install the command line version of php (no web server is needed for this project), composer which is a package management system for PHP and PHP Unit, the testing framework of PHP.

apt-get update
DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
apt-get install -y php-cli composer php-curl phpunit

Install dependencies

The projects uses composer to manage its dependencies so this is how we install them.

composer install

Run tests

Finally we can run the tests.

phpunit

Test results

As I write this post the tests fail with the following:

Time: 00:01.136, Memory: 36.00 MB

There were 2 failures:

1) Twig\Tests\Cache\FilesystemTest::testWriteFailMkdir
Failed asserting that exception of type "RuntimeException" is thrown.

2) Twig\Tests\Cache\FilesystemTest::testWriteFailDirWritable
Failed asserting that exception of type "RuntimeException" is thrown.

FAILURES!
Tests: 1663, Assertions: 4399, Failures: 2, Skipped: 4.

Legacy deprecation notices (3)

At this point I could use the editors and IDEs I have on my host system to edit the files of the project and then rerun the tests, but that I'll leave for someone else.

Quit the docker container

It is simple to exit from the Docker container. Because we used the ```--rm flag when we started the container, Docker will automatically remove it.

exit

Clean the git workspace

Running the composer and phpunit command created some files on the disk, the following command can remove them. I had to use sudo here as Docker created those files as user root.

sudo git clean -dxf

In case you were wondering which files and folders were added by those two commands, this was the output of the git clean command:

Removing .phpunit.result.cache
Removing composer.lock
Removing vendor/

Conclusion

I hope this will help someone set up the development environment.

As I think this information should be part of the project I'll open an issue pointing to this article and offering to send a Pull-Request to add the content of it to their documentation.

examples/r/counter.R

filename = "count.txt"
count = 0

if (file.exists(filename)) {
    as.numeric(readLines(filename)) -> count
}

count <- count + 1

cat(count, end="\n")

fh <- file(filename, "w")
cat(count, end="\n", file = fh)
close(fh)

After installing R we can run this script by typing in

Rscript counter.R

In R there are at least 3 ways to assign a value to a variable. One can use the boring = sign as we have in the first two assignments. We can also use the left arrow <- and we can also use the ->. In this script I used all 3, just to demo them. I think in a real script I'd stick to boring =, but as I can see code written by others, the also like to use the other signs.

The file.exists and the readLines functions seem to have very descriptive names.

The as.numeric function converts a string or "characters" as they are called in R to a number or "numeric" as it is referred to in R.

cat is a similar to print, but as I understand it is more generic function. Among other things it does not "number" the output lines. It also accepts various parameters such as the end parameter and the file parameter.

The file function with the w parameter opens the file for writing and returns a filehandle. We then use cat and direct its output to the file. At the end we close the file.

examples/javascript/improved-vanilla-javascript-counter.html

<form>
  <input id="counter" type="button" value="0">
  <input type="reset">
</form>

<script>
const form = document.forms[0]
const increment = () => console.log(++form.counter.value)
const reset = () => form.counter.value = 0

form.addEventListener('reset', reset)
form.counter.addEventListener('click', increment)
</script>

view

A few notes as I am trying to understand this example:

• The value attribute of an input element of type="button" is the value shown by the browser on the button.
• I am still not used to the fat-arrow notation of creating anonymous functions in JavaScript that are then assigned to a variable. () => code is a function definition.

It is indeed a very nice use of the various default capabilities of HTML and JavaScript.

A few of the most popular SSGs are

• Jekyll written in Ruby
• Hugo written in Go
• Docusaurus

Jamstack has a collection of 347 SSGs with various details categorized by language.

Description

Version 1: The user writes Markdown files and the proccsor generates HTML pages.

Version 2: The Markdown files can have some meta data at the top of the file. E.g called "front matter" Similar to what DEV.to allows.

Start implementing one of the fields and then one-by-one add the additional fields.

---
title: The title of the post
published: false
author: Name of the author
description: A short description of the article.
tags: some, words, or, even, multi word expressions
published_at: 2023-01-29T11:30:01
series: some-name
---

Here comes the content of the article.

This information is displayed on the web page.

The title is going to be both the HTML title tag in the head and the h1 element of the page.

The articles that have published: true are actually published. The ones that have published: false are considered drafts.

The published_at field can be used to create a page listing all the articles in the order the were published.

The generated front page could display the 3 most recently published articles.

The description can be the content of the HTML meta tag called description. (Look at the source of the current page.)

The tags can be used to crete pages, eg. t/some listing all the articles that have the same tag. There can also be a page called /tags listing all the tags and the number of articles with that tag.

Version 3:

Add a config file e.g. config.yaml that will contain some configuration information about the whole site. e.g.

title: The title of the site (on the main page)
front_limit: 3    (The number of articles to show on the front page)

Version 4:

Find other interesting features of the other processors list above.

I found two options:

Screenshot

The first one is to create a screenshot of the chart. I just click on the Print Screen button of my keyboard. Some tool opens that allows me to select an area of the screen. Then when I press enter in creates a screenshot and saves it in ~/Pictures/Screenshots/ as a png image. This is the result from a recent survey I ran among the participants of my Open Source Development Course particpants. The image that was saved was 31,362 bytes.

Copy to clipboard and save from there

The other option is to use that copy button we see in the top-right corner of each chart. When we click on it the chart is copied to the clipboard. We can paste that directly into an email or a document (I tried LibreOffice document, but I assume it would also work in Microsoft word and in Google docs.)

I can also us the xclip command to save the content of the clipboard in a file as was described in the Save clipboard post.

xclip -selection clipboard -t text/html -o > downloaded.html

The result was a file of 167,501 bytes tha looked like this:

<meta http-equiv="content-type" content="text/html; charset=utf-8">
<img
   src="data:image/png;base64,iVBORw0KGgoAAAA...mCC"
   alt="Forms response chart. Question title: How well do you know GitLab?. Number of responses: 11 responses."
   id="docs-internal-guid-cf3e7410-7fff-8a73-7757-c27f09525278">

It contains a png image in base64 encoding. I included the whole thing

I included it here, in this html file. If you click on view-source, you'll see the whole image in base64. It looks like one line from The Matrix.

Converting bas64 to png

I edited the downloaded html file and removed everything except the string representing the image:

iVBORw0KGgoAAAA...mCC

Then ran

base64 -d downloaded.html > out.png

The resulting file size was 125,402 bytes and this is the file:

I could then resize the image to make it smaller. Both at it is too high and wide and it is also a lot of bytes.

xclip -selection clipboard -t text/html -o > out.html

It saved the content of the clipboard the out.html. The format looked like this:

  <meta http-equiv="content-type" content="text/html; charset=utf-8">
  <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAC...

So there is an png image in base64 format.

If we open the HTML file in a browser we can see the image.

It has two buttons.

1. One of them is a counter. Every time you click on it the value on it will increase by one.

2. The other one is a reset button.

examples/javascript/vanilla-javascript-counter.html

<button id="counter">0</button>
<button id="reset">Reset</button>
  
 
<script>
function increment_counter() {
    let counter = document.getElementById('counter').innerHTML;
    counter++;
    console.log(counter);
    document.getElementById('counter').innerHTML = counter;
}

function reset_counter() {
    console.log('reset');
    document.getElementById('counter').innerHTML = 0;
}

document.getElementById('counter').addEventListener('click', increment_counter);
document.getElementById('reset').addEventListener('click', reset_counter);
</script>

view

At the top of the file we have two HTML buttons. Both have IDs for easier identification.

The JavaScript code has two functions and then two calls to document.getElementById. These two calls will locate the HTML element with the ID given as a parameter and then connect a function to the "click" event of each one of them. These are the two functions we have.

This means that when the user clicks on the HTML element with the ID counter the function increment_counter will be executed and when the user clicks on the element with the ID reset the reset_counter function will be called.

In this little animated gif you can see the two small buttons at the top and most of the view is taken up by the console where you can see the console.log messages.

Reset counter

The reset_counter function is the easier to explain:

console.log('reset');

Is only there so we can see on the console of the browser that something happened. Then we use the call

document.getElementById('counter')

to locate the HTML element and innerHTML to set the content of the element to 0.

Increment counter

Here we first copy the content of the HTML element to a variable we cleverly named counter.

Then we increment it by one using the ++ operator.

Then we have the console.log call. Just so we can see in the console what's happening.

Finally we write back the value the HTML document.

Issues

Keeping our data (the counter) as part of the HTML is probably not a very good idea in a project more complex than this one. However, in this case it has the advantage that we don't need to think how we keep our data (which is normally in JavaScript) in sync with the display (which is the HTML).

This counter is also not persistent. If you reload the page the counter will be reset.

For example, I have a reminder every month to back up my data from LinkedIn. Primarily the list of my connections though now that I am writing this I'll also check how could I backup all my other data from LinkedIn.

This backup is kept in a private git repository on GitHub as well.

Here is how I do it:

1. Visit LinkedIn
2. Click on "Me" (my avatar in the top right corner)
3. Click on Settings & Privacy
4. Click on Data Privacy
5. Click on Get a copy of your data
6. I am mostly interested in Connections so I check that box. You might want others as well.
7. Click on Request Archive

It will take some time for LinkedIn to prepare the zip file, but they will send you an email with a link.

Once I receive that email I click on the link and download the zipfile which is called something like Basic_LinkedInDataExport_12-05-2022.zip

Handle the downloaded file

1. cd to the workdir of the git repository where I keep my private data
2. unzip the file: unzip ~/Downloads/Basic_LinkedInDataExport_12-05-2022.zip
3. sort the content with sort Connections.csv -o Connections.csv
4. Remove the "empty" rows from the beginning. (For some reason there every time there are a few 10s of rows with only commas. I remove those.
5. Move the line that starts with First Name to the beginning of the file. This is the header of the file that is moved away by the sort.
6. Commit the file to git and push it out.

Automate this?

I am sure I could automate the process, but I don't think it is worth my time to do so.

At first I sent a "standard" configuration of GitHub Actions, but then I noticed 2 things. Some tests were dependent on some extra modules and some tests needed a threaded Perl. Apparently the one that I used in my "standard" configuration isn't. So In addition to the execution of the CI on Ubuntu, macOS and Windows natively, I've also added another job in which we use a perl-based Docker image and run the tests there. In this case one of the tags I used from Perl on Docker Hub was the 5.36-threaded.

I was happy to see that the tests passed on every platform and on every version of Perl I tried.

I sent the pull-request

GitHub Actions

examples/perl/data-alias/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        perl: [ '5.30', '5.36' ]
        exclude:
          - runner: windows-latest
            perl: '5.36'

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( startsWith( matrix.runner, 'windows-' ) && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm -v
        cpanm --installdeps --notest .
        cpanm --notest Devel::CallParser Test::Pod::Coverage Test::Pod

    - name: Show Errors on Windows
      if:  ${{ failure() && startsWith( matrix.runner, 'windows-')}}
      run: |
         ls -l C:/Users/
         ls -l C:/Users/RUNNER~1/
         cat C:/Users/runneradmin/.cpanm/work/*/build.log

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && startsWith( matrix.runner, 'ubuntu-')}}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Show Errors on OSX
      if:  ${{ failure() && startsWith( matrix.runner, 'macos-')}}
      run: |
         cat  /Users/runner/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        perl Makefile.PL
        make
        make test


  test-in-container:
    strategy:
      fail-fast: false
      matrix:
        perl: [ '5.30', '5.36', '5.36-threaded' ]
        # tags from https://hub.docker.com/_/perl

    runs-on: ubuntu-latest
    name: Perl ${{matrix.perl}}
    container: perl:${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm -v
        cpanm --installdeps --notest .
        cpanm --notest Devel::CallParser Test::Pod::Coverage Test::Pod

    - name: Show Errors on Ubuntu
      if:  ${{ failure() }}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        perl Makefile.PL
        make
        make test

Conclusion

Sometimes you need more than one job even for stand-alone 3rd-party libraries.

Pull-request

Conclusion

There are many projects that are low-hanging fruits where you can add GitHub Actions in a matter of minutes and get the benefits immediately.

It took me quite a while and some back-and-forth with the author to make GitHub Actions work and even now some of the tests are failing. Here is what happened:

Cloning

As always I started by cloning the timescaledb git repository.

git clone git@github.com:jonatas/timescaledb.git
cd timescaledb

Then I forked the repository on GitHub and added my fork as another remote:

git remote add fork git@github.com:szabgab/timescaledb.git

Using PostgreSQL as a service

At the beginning I thought the module needs a PostgreSQL database to run. As I did not have an example ready first I created an example of using PostgreSQL as a service in docker compose.

Then I created an example PostgreSQL as a service in GitHub Actions

With that armed I could create the first version of the GitHub Action for this Ruby Gem.

There at first I had some typos. What an unnecessary waste of time!

.env

Then, when finally I managed to get the syntax right and fixed the typos I git this error:

Errno::ENOENT: No such file or directory @ rb_sysopen - /__w/timescaledb/timescaledb/.env

Fail enough, even the README told me to create a .env file with content that looks like this:

PG_URI_TEST="postgres://<user>@localhost:5432/<dbname>"

So I added the following to the CI configuration file:

echo PG_URI_TEST="postgres://username:secret@database:5432/testdb" > .env

This made some progress and this time I got a different error:

PG::UndefinedFunction: ERROR:  function create_hypertable(unknown, unknown, chunk_time_interval => interval) does not exist

I did not know what to do. I thought I am not using the correct version of Ruby or PostgreSQL so I experimented with that a bit. Added code to print out the version number of PostgreSQL:

echo "SELECT version()" | psql -h postgres -U username testdb
echo "SELECT CURRENT_TIME" | psql -h postgres -U username testdb

even added this line:

tsdb postgres://username:secret@postgres:5432/<dbname> --stats

Timescale DB

Slowly I understood that instead of a vanilla PostgreSQL database I need to use to use Timescale which is based on PostgreSQL. I am sure others would have come to this conclusion much faster than I did.

So I switched to use a timescaledb Docker image.

This too involved several trials and errors, but I won't bore you with my small mistakes.

Instead I got this error:

Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
rake aborted!
Gemika::UnsupportedRuby: No gemfiles were compatible with Ruby 3.1.3

OK, so apparently it does not work with the latest version of Ruby.

then I got this error

Gemika::UnsupportedRuby: No gemfiles were compatible with Ruby 2.7.7

Finally I settled with Ruby 2.7.1.

By this time a lot of tests seemed to have passed, but I got the following error:

PG::InvalidSchemaName: ERROR: schema "toolkit_experimental" does not exist

This was the time when I decided to open an issue.

TimescaleDB-HA

The author quickly replied and suggested I use the timescaledb-ha Docker image.

I tried and failed and then tried another tag and after a while the tests started to fail at a new location. I even added several different tags so we will be able to see if any of those work, but no.

However it seems we are left with only two tests failing. So I pasted the failure reports to the open issue.

At this point the author suggested I send the PR and he'll try to deal with these failures. So I sent the pull-request.

GitHub Actions

examples/ruby/timescaledb/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
#  schedule:
#    - cron: '42 5 * * *'

jobs:
  test-in-container:
    strategy:
      fail-fast: false
      matrix:
        ruby: [ '2.7.1' ]
        database:
          - 'pg14.6-ts2.9.0-patroni-static-primary-latest'
          - 'pg14-patroni-static-primary-latest'
          - 'pg14.6-ts2.9.0-patroni-static-primary-p0'
          - 'pg14.6-ts2.9-oss-latest'
          - 'pg13.9-ts2.9-latest'

    services:
      database:
        image: timescale/timescaledb-ha:${{matrix.database}}
        env:
          POSTGRES_USER: username
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: testdb
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5


    runs-on: ubuntu-latest
    name: OS Ruby ${{matrix.ruby}} database ${{matrix.database}}
    container: ruby:${{matrix.ruby}}

    steps:
    - uses: actions/checkout@v3

    - name: Show Ruby Version
      run: |
        ruby -v

    - name: Install psql
      run: |
        apt-get update
        apt-get install -y postgresql-client

    - name: Show PostgreSQL version and time
      env:
        PGPASSWORD: secret
      run: |
        echo "SELECT version()" | psql -h database -U username testdb
        echo "SELECT CURRENT_TIME" | psql -h database -U username testdb

    - name: Setup
      run: |
        ./bin/setup

    - name: run tsdb
      run: ./bin/tsdb postgres://username:secret@database:5432/testdb --stats

    - name: Test setup
      run: |
        echo PG_URI_TEST="postgres://username:secret@database:5432/testdb" > .env
        cat .env
        bundle exec rake test:setup

    - name: Test
      run: bundle exec rake

Conclusion

Wow, this was really tiring even to describe what happened and I left out a lot of the smaller steps as those were primarily my mistakes and lack of reading the README of the project. However at the end I think I managed to put together something that will be useful for the developers and hopefully will also help others to set up CI for other libraries.

I was really surprised that my searches for this expression indicated that it is not used elsewhere. I wonder what is the expression in other languages for the same phenomenon.

prepper

So you might guess this was a funny case. A few days ago I tried to add GitHub Actions to the prepper Ruby Gem. The tests failed even locally so I reported the problem.

Soon the author replied that this Gem does not have any real tests. The only tests that are in the repository are the ones that were generated by bundle, the tool to jump-start the development of a Ruby Gem and for some reason they were failing.

So yes, I could have configure GitHub Action for a Gem that does not have any tests. That would be running the whole CI for every push without it actually checking anything. "full gas in neutral"

The author also indicated that he'll remove the failing test and soon will add real tests. I guess it would not be a total waste to add the GitHub actions now. That way when tests start to arrive to the repository they will be executed.

By the time I actually checked this the author has made the necessary changes to let the existing test pass.

I went ahead, added the GitHub Actions configuration file and sent the pull-request which the author promptly accepted.

GitHub Actions

examples/ruby/prepper/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
#  schedule:
#    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        ruby: [ '3.0' ]

    runs-on: ubuntu-latest
    name: Ruby ${{matrix.ruby}}
    container: ruby:${{matrix.ruby}}

    steps:
    - uses: actions/checkout@v3

    - name: Show ruby Version
      run: |
        ruby -v

    - name: Install Modules
      run: ./bin/setup

    - name: Run tests
      run: rake test

Conclusion

The author also said "thank you" in Hungarian. I could have guessed from his family name, but I was not really focused on who created the library, just wanted to add the Continuous Integration.

Default setup

I noticed that many Ruby Gems have the same instructions to set up the development environment and to run the tests. I guess this means they used some tool to generate the skeleton of the project that automatically created the bin/setup shell script and the test spec files that could be filled.

One one hand this is good as it probably encourages people to start writing test-specs.

On the other hand having a bash script to execute bundle install means we cannot use this on Windows natively. So for Windows we need different instructions. I think this isn't such a good idea.

Travis-CI

It seems that the script creating the Ruby Gem skeleton also include a configuration file for Travis-CI. Unfortunately Travis-CI does not work any more for free so this file just sits there and has no purpose. It even causes confusion because it gives the feeling that the the tests worked at least once in Travis, even for Ruby projects that were created after Travis-CI stopped its public service.

GitHub Actions

Anyway, for this particular Ruby Gem the instructions worked so I could configure GitHub Actions and send a pull-request

examples/draftjs-html/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
#  schedule:
#    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        ruby: [ '3.0' ]

    runs-on: ubuntu-latest
    name: Ruby ${{matrix.ruby}}
    container: ruby:${{matrix.ruby}}

    steps:
    - uses: actions/checkout@v3

    - name: Show ruby Version
      run: |
        ruby -v

    - name: Install Modules
      run: |
        ./bin/setup

    - name: Run tests
      run: |
        rake spec

Conclusion

There are many projects that don't have working CI (maybe have a Travis-CI configuration, but that does not do anything now), and their tests fail. I am actually a bit surprised about this even though this is one of the reasons I think it is important to have a functioning CI set up, so that you will be reminded if you forget to run the tests and they break.

Finally I found this Perl module called Plack-Middleware-LogAny.

I copied the CI file from Plack-Middleware-Greylist that I created on day 9.

The only difference is that this one passes on Windows as well.

I sent the Pull-request

Conclusion

It's better to start earlier working my CI.

Instructions

This project has instruction in its README file how to set it up locally and how to run its tests. This is awesome as it seems there are several ways to do this depending on the preferences of the author.

Error in tests

I cloned the repo, started my Docker container where I run the tests, installed the dependencies and then got an error while running the tests. I reported it.

Find the breaking change

Then I thought, if the author used Travis-CI then this surely worked at one point. So I looked at the history.

In my ~/.gitconfig I have an alias:

[alias]
  lg = log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit

So I just ran

git lg

And got this output:

* 8a160e5 - (HEAD -> main, origin/main, origin/HEAD) Bump version to 0.2.1 (7 hours ago) <Anton Kostyuk>
* 1ef292c - Add config dup method (7 hours ago) <Anton Kostyuk>
* 7a079f2 - Bump version (8 hours ago) <Anton Kostyuk>
* e21a35d - Bump version (8 hours ago) <Anton Kostyuk>
* a954ffd - Fix headers, params, callbacks inheritance (8 hours ago) <Anton Kostyuk>
* 6b1cdf7 - Fix config duplication (3 weeks ago) <Anton Kostyuk>
* 21381bf - Bump version (3 months ago) <Anton Kostyuk>
* c59c152 - Add default adapter method (3 months ago) <Anton Kostyuk>
* 97e6f11 - Fix initializing (3 months ago) <Anton Kostyuk>
* c7fc920 - Update README.md (3 months ago) <Anton Kostyuk>
* 4bf2717 - Bump version to 0.1.1 (3 months ago) <Anton Kostyuk>
* 4b06d19 - Update description and links (3 months ago) <Anton Kostyuk>
* c61eb20 - Update README.md (3 months ago) <Anton Kostyuk>
* 4ecb8e5 - Add tests (3 months ago) <Anton Kostyuk>
* 617a0c8 - Fix gem initialize (3 months ago) <Anton Kostyuk>
* ce87ffd - Fix files load (3 months ago) <Anton Kostyuk>
* 0cb0466 - Init (3 months ago) <Anton Kostyuk>

I noticed that there were a few changes 7-8 hours again. A change 3 weeks ago and the original code was written 3 months ago. So I said, I said let's try the commit from 3 weeks ago.

I checked it out, ran the tests and they all passed. So I checked out the next commit and the tests already failed. I did not even have to run git bisect and found and reported the problematic commit. I hope it will help the author fix the code and/or the tests.

GitHub Workflow file

The configuration file was straight forward. Almost exactly as we had yesterday. I copied it from there. The only difference is that now we run

./bin/setup
rake spec

in the docker containers but we run

bundler install
rake spec

when we run natively. Especially because Windows would not know what to do with the /bin/setup shell script.

Conclusion

The projects with the Travis-CI configuration might be even more in danger than those without any CI configured. They might be in the false security that they have a CI while it has not been working for quite some time.

It's a real pity that Travis-CI shut down their free service. I liked them. I don't know what has happened though I guess as GitHub Actions took over their business they could not afford it any more.

First I tried it locally. I ran the code that generates the HTML page and then opened the HTML file in my browser. The order was as I expected.

So how come it works correctly on my computer but incorrectly when it is on the GitHub Pages server.

This was really annoying.

I added some debugging code to print the version number of Ruby. As much as it did not seem reasonable I was desperate. I thought maybe we have some version difference and that causes the problem. I also added some code to print the dates to the screen while the HTML file is being generated.

I pushed out the code and after a few seconds looked at the results.

The version number was similar and the dates were printed in the correctly sorted order on the screen. So how come they are not sorted on the web site.

Then I saw it

When I refreshed the page it showed up properly and then there was a blink and the order was gone.

That's when I realized the HTML table is configured s a sortable table. I looked at the code and it was sorted by the wrong column.

I think I copied the code for this from the CPAN Digger, but we have different columns here so the code was sorting according to the names of the authors. Oh oh.

It did not happen when I looked at the HTML file on my computer as I was loading a plain file and it could not load the JavaScript that triggered the sorting.

I fixed the JavaScript code, well, update the configuration to point to the right column. This fixed the issue, but then I started to think, why do we even need to let the user sort the table? So I turned off the sorting.

Phew, that was a stupid bug.

four-pillars in Ruby

A class which tells fortune by Four Pillar astrology.

It is a very simple Gem without any dependencies and some plain tests. It was easy to configure so I configured both a Docker container based job and a job that runs on the 3 different operating systems.

Pull-Request

GitHub Actions configuration file

examples/ruby/four-pillars/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
#  schedule:
#    - cron: '42 5 * * *'

jobs:
  test-in-container:
    strategy:
      fail-fast: false
      matrix:
        ruby: [ '2.6', '2.7', '3.0' ]

    runs-on: ubuntu-latest
    name: OS Ruby ${{matrix.ruby}}
    container: ruby:${{matrix.ruby}}

    steps:
    - uses: actions/checkout@v3

    - name: Show Ruby Version
      run: |
        ruby -v

    - name: Run tests
      run: |
        rake test

  test-natively:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        ruby-version: ['3.0']

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Ruby ${{matrix.ruby-version}}

    steps:
    - uses: actions/checkout@v3
    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: ${{ matrix.ruby-version }}
    - name: Run tests
      run: rake test

Conclusion

Even Chinese Astrology can be tested.

I asked the author if he would be interested in a GitHub Actions configuration.

A warning during build

I tried to build the module locally and run its tests locally, but I encountered some issues:

First I noticed that there are some warning during the build emitted by one of the dependencies. As it turns out upgrading the dependency solved this issue, but the latest package of the dependency also had a minor issue. The version numbers in the different files were slightly confusing. So I reported that too.

Apparently that was already fixed, it was just not released yet.

An error - missing files

The next thing I encountered was that some files were missing. That made the tests fail. The author first thought that the issue was somehow cause by me mixing versions, but soon found that the files were not added to git.

This is one of the reasons to have CI.

This can happen to any one of us.

Maybe not to you, but someone else on your team or to a contributor to your project

Should the tip of the main branch be usable?

The author made an interesting comment on the issue about the missing files.

Here is what I think about it:

I would say it is a good practice to ensure all the test are always passing on the tip of the main branch. Otherwise one would risk starting to think that "some tests always fail" which could lead to some test failing that should not fail go unnoticed for a long period of time.

Also it upsets the CI and we don't want to do that.

Actually, one of the reasons of having a CI to have something look over our shoulders and make sure that all the tests are always passing on a clean environment.

And then comes the point, if all the tests are passing, doesn't that mean that the version is usable?

I am looking forward the continuation of this discussion.

Pull-Request

Once those issues were fixed adding the GitHub Action configuration file and converting the commands from the Travis-CI config file to GitHub Actions was not difficult. At one point the author commented that he wants the code to run on Windows and macOS as well, but it is difficult as he does not have access to those operating systems. I figured configuring GitHub Actions to run on those OS-es as well will help him with that task. I was also expecting the test or even maybe the compilation to fail on at lest one of them. So I was a bit surprised that everything worked.

Well, I still managed to make a few mistakes and thus had to try it several time, but after a while I got it right and sent the pull-request. To my surprise the author closed it as he is planning to move the repo to a GitHub organization.

It is unclear to me why that stops him from accepting the PR so I asked him on the PR.

GitHub Actions

In any case, I included the configuration files here. This time, I went a bit further than in the previous cases of my recent GitHub Actions pull-requests. I also went beyond what was in the Travis-CI configuration file.

This time I create two configuration file.

In the first one we use multiple version of perl in Docker containers.

examples/marpa-r2/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        perl: [ '5.10', '5.20', '5.30', '5.36', '5.36-threaded' ]
        # See options here: https://hub.docker.com/_/perl/

    runs-on: ubuntu-latest
    name: OS Perl ${{matrix.perl}}
    container: perl:${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm Config::AutoConf # optional dependency
        cd cpan
        cpanm --installdeps --quiet --notest .

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && startsWith( matrix.runner, 'ubuntu-')}}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: build Marpa and execute tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        set -x
        (cd cpan/xs/ && make)  # generate necessary files

        cd cpan/
        perl Build.PL
        ./Build
        ./Build test
        ./Build distmeta
        ./Build disttest
        MARPA_USE_PERL_AUTOCONF=1 ./Build disttest
        ./Build dist

In the second one we use the 3 different operating systems: Windows, Linux and macOS.

examples/marpa-r2/native.yml

name: Platforms

on:
  push:
  pull_request:
  workflow_dispatch:
#  schedule:
#    - cron: '42 5 * * *'

jobs:
  platform-test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        perl: [ '5.30' ]
        exclude:
          - runner: windows-latest
            perl: '5.36'

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( matrix.runner == 'windows-latest' && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm Config::AutoConf # optional dependency
        cd cpan
        cpanm --installdeps --quiet --notest .

    - name: Show Errors on Windows
      if:  ${{ failure() && matrix.runner == 'windows-latest' }}
      run: |
         ls -l C:/Users/
         ls -l C:/Users/RUNNER~1/
         cat C:/Users/runneradmin/.cpanm/work/*/build.log

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && matrix.runner == 'ubuntu-latest' }}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Show Errors on OSX
      if:  ${{ failure() && matrix.runner == 'macos-latest' }}
      run: |
         cat  /Users/runner/.cpanm/work/*/build.log

    - name: Make - generate necessary files
      run: |
        cd cpan
        cd xs
        make

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        cd cpan
        perl Build.PL
        perl Build
        perl Build test
        perl Build distmeta
        perl Build disttest

    - name: Run tests with autoconf
      env:
        MARPA_USE_PERL_AUTOCONF: 1
      run: |
        cd cpan
        perl Build disttest

    #- name: Build dist
    #  env:
    #    perl Build dist

Conclusion

You don't have to have a CI if you remember always running your tests. You don't even need to have access to Windows and macOS to make your code work there if you are a really good programmer as Jeffrey Kegler, the author of Marpa. And even then you might forget to add some of the files to git.

However, most of us aren't as focused on the details and most of would not be able to build a project like Marpa. For sure I know I wouldn't.

So we need the hand-holding and the discipline a CI can provide.

As always I started at the CPAN Digger. This time found the Log-Any distribution.

Pull-Request

Conclusion

Simple is good for work. Less exciting for the blog. Check out the previous entries of this series if you'd like to find more exciting cases.

Error reports

First I looked at a project called nezekan. I tried to run the tests locally, but failed. I am not sure if I was even trying to run them as the authors intended, but this error seem to be genuine. It is a case-sensitivity issue. After I fixed that locally I bumped into and reported another issue.

Only now, that I am writing this I realized that the project had a Travis-CI configuration file. Although Travis-CI does not provide the free service any more I could have looked at the configuration file and base my commands on that. I can get back to that later.

wp2txt

After a while I found the wp2txt project. Adding GitHub Actions to this was quite simple though it was different from the previous one. Here is the pull-request and below you'll find a copy of the configuration file as I sent it.

GitHub Actions

examples/wp2txt/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        ruby: [ '3.1' ]

    runs-on: ubuntu-latest
    name: Ruby ${{matrix.ruby}}
    container: ruby:${{matrix.ruby}}

    steps:
    - uses: actions/checkout@v3

    - name: Show Ruby Version
      run: ruby -v

    - name: Install dependencies
      run: bundle install

    - name: Install rspec
      run: gem install rspec

    - name: Run tests
      run: rspec

Conclusion

Slowly, but I learn how to run the tests in the various Ruby projects as well. Unfortunately there are not very many new uploads to RubyGems and this Ruby Digger is growing slowly.

A few days ago I sent a pull-request with a CI configuration to the RDF::KV module in Perl. I was not very happy with the solution, but sent the pull-request anyway. The author of the module was very responsive and soon pointed me to the step I missed, but also pointed at the Ruby port of rdf-kv.

Today finally I had the time to update the CI I wrote for the Perl version. (See the PR) and also to add the GitHub Actions configuration to the Ruby implementation and send the pull-request

examples/rdf-kv-ruby/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        ruby: [ '3.1' ]

    runs-on: ubuntu-latest
    name: Ruby ${{matrix.ruby}}
    container: ruby:${{matrix.ruby}}

    steps:
    - uses: actions/checkout@v3

    - name: Show Ruby Version
      run: ruby -v

    - name: Install dependencies
      run: bundle install

    - name: Run tests
      run: bundle exec rake

Conclusion

Maybe the earlier failures to set up CI for Ruby Gems wasn't even my fault. Maybe they really had failing tests.

Mojo-UserAgent-Cached

It was quite straight-forward. There were only two small issues. I had to install Module::Install before I could use the regular tools as the Makefile.PL relies on that. That's a known drawback of Module::Install for developers and maintainers.

The other problem was that tests were failing on Windows so I had to disable them. Anyway I sent a Pull-request it was merged soon after.

examples/mojo-useragent-cached/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest] # , windows-latest
        perl: [ '5.30', '5.36' ]
        exclude:
          - runner: windows-latest
            perl: '5.36'

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( startsWith( matrix.runner, 'windows-' ) && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm -v
        cpanm --notest Module::Install
        cpanm --installdeps --notest .

    - name: Show Errors on Windows
      if:  ${{ failure() && startsWith( matrix.runner, 'windows-')}}
      run: |
         ls -l C:/Users/
         ls -l C:/Users/RUNNER~1/
         cat C:/Users/runneradmin/.cpanm/work/*/build.log

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && startsWith( matrix.runner, 'ubuntu-')}}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Show Errors on OSX
      if:  ${{ failure() && startsWith( matrix.runner, 'macos-')}}
      run: |
         cat  /Users/runner/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        perl Makefile.PL
        make
        make test

Plack-Middleware-Greylist

Here too I had to disable the testing on Windows, this time because one of the dependencies was failing own Windows. The author then wanted to add testing more versions of Perl, but did it himself. I was contemplating a bit if I should send another PR where first we build the tar.gz on one version of Perl and then use that for testing, but there are no extra dependencies install for the building so I don't think that could reduce the time or the cpu usage.

pull-request

examples/plack-middleware-greylist/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest] # , windows-latest
        perl: [ '5.30', '5.36' ]
        exclude:
          - runner: windows-latest
            perl: '5.36'

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( startsWith( matrix.runner, 'windows-' ) && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install dependencies
      run: |
        cpanm -v
        cpanm --notest --installdeps .

    - name: Show Errors on Windows
      if:  ${{ failure() && startsWith( matrix.runner, 'windows-')}}
      run: |
         ls -l C:/Users/
         ls -l C:/Users/RUNNER~1/
         cat C:/Users/runneradmin/.cpanm/work/*/build.log

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && startsWith( matrix.runner, 'ubuntu-')}}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Show Errors on OSX
      if:  ${{ failure() && startsWith( matrix.runner, 'macos-')}}
      run: |
         cat  /Users/runner/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        prove -l

Conclusion

That was easy.

Today I picked another Perl module. As it turned out it wasn't such a "low-hanging-fruit" either as I was hoping for, but I managed to set up CI. At least on Ubuntu.

pEFL is the name of the Perl bindings for the Enlightenment Foundation Libraries.

According to the README of the Perl Module one needs to install the efl-dev package, but on Ubuntu 22.10 there is no such package. I found one called efl-doc, but that's the documentation. Running apt-cache search efl | grep dev helped me find the package called libefl-all-dev. It seem that this is what we needed. So I opened an issue.

At first I was trying to run the tests directly on the ubuntu-latest runner of GitHub Actions, but soon I gave up. Maybe the problem was that it is running an older version of Ubuntu. Maybe the problem was something else. I don't know, I have not researched it. Instead I switched to using the official Ubuntu 22.10 docker image.

After several trial and error I figured out that I have to also install the gcc and make packages for the compilation.

For Perl I needed ExtUtils::PkgConfig. The easiest seemed to be to install the libextutils-pkgconfig-perl package that is packaged and distributed by the developers of Ubuntu.

At the end I saw that the only test this package has is trying to load the module, but at least we can see that the compilation worked.

CentOS

After the success with Ubuntu I thought I'll also try to do the same on a CentOS based Docker container. I started with CentOS 7. I found out it does not even have Perl installed so I started with that. However, when I got to the installation of the efl library I found out that it is not available on CentOS. At least not in the default locations.

At this point I decided to stop for now and sent the Pull-Request with the Ubuntu configuration. Better to have one than none. Later we can try to figure out how to set this up on other Linux distributions. Maybe with different versions of Perl as well.

If the author even interested.

Why is this valuable?

While I was still composing this blog post, the author has updated the README file, closed the issue and commented on it indicating that it is unclear what is the purpose of compiling / running tests on every push? Unfortunately not all distributions have efl/Enlightenment. But compiling the libraries is not a hard job... Perhaps the Budgie Desktop switch to efl, then the distribution coverage could be better...

Let me answer it here as I think this could be interesting to others as well.

The point of CI running on every push is to get feedback as soon as possible.

It would be a lot more valuable if there were lots of tests than just loading the module as in this case, but even this way it is valuable to see if some change broke some part of the code that the developers have not noticed on their computer.

If the changes came as Pull-Request then the author does not even know if the sender has tried to compile the code on their computer. Maybe they just sent some changes? By letting the CI compile it both the author and the PR sender will quickly see the results. That can save time to the author and the PR sender might even fix the issue before the author has a chance to look at it.

It would be more useful if the compilation was done on various Linux and BSD distributions and maybe using various versions of Perl. If the specific distribution does not have a ready-made package of Efl then it could be possibly compiled from its source code. It is very unlikely that the author has many setups locally and it is even less likely that a potential contributor has all that setup. Even if they have it is probably easier to hand this job off to some external system (GitHub Actions in this case) than doing on your computer. So on the longer run it will be a lot more beneficial to set this up on other distributions as well. Consider this PR as the first step in that direction.

I have a video talking about this, though in that I was talking about Travis-CI.

GitHub Actions for Ubuntu

examples/perl-efl/ubuntu.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  ubuntu:
    runs-on: ubuntu-latest
    container: ubuntu:22.10

    steps:
    - uses: actions/checkout@v3

    - name: Install C dependencies
      run: |
        apt-get update
        apt-get install -y libefl-all-dev
        apt-get install -y gcc
        apt-get install -y make
        apt-get install -y libextutils-pkgconfig-perl

    - name: Show Perl Version
      run: |
        perl -v

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        perl Makefile.PL
        make
        make test

Partial configuration for CentOS

examples/perl-efl/centos.yml

  centos:
    runs-on: ubuntu-latest
    container: centos:7

    steps:
    - uses: actions/checkout@v3

    - name: Install C dependencies
      run: |
        yum install -y perl
        #yum install -y perl-ExtUtils-MakeMaker
        yum install -y perl-App-cpanminus
        cpanm Test::More
        cpanm ExtUtils::PkgConfig

    - name: Show Perl Version
      run: |
        perl -v

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        perl Makefile.PL
        make
        make test

Conclusion

Better one bird in the hand than two birds on the tree. Or some similar expression.

It wasn't supposed to be difficult but I encountered some issues and had to be pragmatic in the setup even if far from ideal.

IMHO it is better to have a working CI that already checks part of what can be checked than to have nothing.

I cloned the Git repository of the package and as it came with a Makefile.PL I ran the usual

perl Makefile.PL

just to see a nasty error:

include /home/gabor/os/p5-rdf-kv/inc/Module/Install.pm
String found where operator expected at Makefile.PL line 5, near "readme_from 'lib/RDF/KV.pm'"
	(Do you need to predeclare readme_from?)
syntax error at Makefile.PL line 5, near "readme_from 'lib/RDF/KV.pm'"
Execution of Makefile.PL aborted due to compilation errors.

I tried to understand the source of the problem, but I figured it is better to report it maybe the author will point to something I am doing wrong or maybe the author will confirm the problem.

As I could not use the regular process, but I still wanted to see if the tests pass I installed the dependencies that were listed in the Makefile.PL manually running cpanm. As I saw that the tests pass after I installed all the dependencies I set up GitHub Actions.

From the results I learned that the author tests need some additional modules, however after installing those one of the author tests failed. I think it is probably related to the same issue I already reported, so instead of trying to figure out what's going on I removed the test module that enabled that specific test.

At this point I could send the pull-request.

We'll see how the author responds.

GitHub Action configuration file

examples/rdf-kv/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        perl: [ '5.30', '5.36' ]
        exclude:
          - runner: windows-latest
            perl: '5.36'
          - runner: windows-latest
            perl: '5.34'

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( startsWith( matrix.runner, 'windows-' ) && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm -v
        #cpanm --installdeps --notest .
        cpanm --notest Moose URI::BNode RDF::Trine URI::NamespaceMap XML::RegExp Data::UUID::NCName Data::GUID::Any
        cpanm --notest Test::Pod::Coverage Test::Pod
        # The test using Test::CheckManifest fails without the Module::Install which now not used because of this issue:
        # https://github.com/doriantaylor/p5-rdf-kv/issues/2

    - name: Show Errors on Windows
      if:  ${{ failure() && startsWith( matrix.runner, 'windows-')}}
      run: |
         ls -l C:/Users/
         ls -l C:/Users/RUNNER~1/
         cat C:/Users/runneradmin/.cpanm/work/*/build.log

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && startsWith( matrix.runner, 'ubuntu-')}}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Show Errors on OSX
      if:  ${{ failure() && startsWith( matrix.runner, 'macos-')}}
      run: |
         cat  /Users/runner/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        prove -l
        #perl Makefile.PL
        #make
        #make test

Conclusion

It is better to have a CI running and executing some of the tests in a less-than-ideal way than to have no CI. A working CI can (and actually has to) be improved all the time.

While the indicates that this is a Windows-related thing, there are a number of packages on CPAN that are in the Win32 namespace, but also work on Linux. This one, it seems does not. It seems at least one of its dependencies, the Win32-API does not work on anything else besides Windows.

In the Makefile.PL of Win32-API I saw this code:

my $running_on_windows = $^O eq 'MSWin32' || $^O eq 'cygwin' || $^O eq 'msys';
die qq(OS unsupported\n)
    unless $running_on_windows
    or $ENV{WIN32_API_BUILD}     # So I can build it on Linux too
    ;

So it can also work on cygwin and msys and it can be packaged on Linux as well.

We might be able to setup a Linux machine to build the distribution of the module and then we can run the tests on various versions of Perl on Windows. However I did not want to invest that much time before I even see that it would be interesting to the maintainer of the package.

The setup was quite simple as you can see from the GitHub Action config file below.

I had to remove the two most recent versions of Perl as it seems they are not available in the GitHub Action I used. That could be improved.

The Pull-Request

GitHub Actions

examples/win32-wlan/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [windows-latest]
        perl: [ '5.30', '5.32' ] # It seems that perl 5.34 and 5.36 are not available

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Perl ${{matrix.perl}}

    steps:
    - uses: actions/checkout@v3

    - name: Set up perl
      uses: shogo82148/actions-setup-perl@v1
      with:
          perl-version: ${{ matrix.perl }}
          distribution: ${{ ( startsWith( matrix.runner, 'windows-' ) && 'strawberry' ) || 'default' }}

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm -v
        cpanm --installdeps --notest .

    - name: Show Errors on Windows
      if:  ${{ failure() && startsWith( matrix.runner, 'windows-')}}
      run: |
         ls -l C:/Users/
         ls -l C:/Users/RUNNER~1/
         cat C:/Users/runneradmin/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
      run: |
        perl Makefile.PL
        make
        make test

Conclusion

Sometimes you need Windows.

I looked at CPAN Digger where some Perl script collects and analyses the most recent uploads to CPAN, the central web site of of the Perl packages and MetaCPAN is the user-friendly place to view the content of all the packages.

When I looked at CPAN::Digger the first package that had a link to its GitHub repository but did not have CI was the Net-Async-Redis-XS distribution.

I thought that could be a nice challenge as it will probably require having Redis installed on the CI server.

Background work

Before I could start working on this project I wanted to create an example GitHub Action configuration where we have a Redis server and it is being access by curl. I wasted a lot of time on this and thought I should try to do something simpler that I can experiment with on my own computer and I don't need to send it to GitHub and wait for it do run my code.

So I started to work on a Docker Compose configuration where I had two containers. One is a Redis service and the other one is a simple client. Here to I failed to use curl as the client. However, I could make it work using the official Redis client called redis-cli. I added the example to my Docker slides.

Once I managed to create the Docker Compose configuration, creating the one running on GitHub Actions was easier. You can find my example in this repository.

I 08 2-3 hours on the preparations, but I also created the examples so people will be able to base their work on these examples.

Creating the GitHub Action using Redis

Once I had the Redis configuration the work on the Perl module was not that hard. I made a few typos and missed the fact that the repository has a Makefile.PL so at first I tried to use Dist::Zilla.

I also had to include the installation of a few Perl modules in the GitHub Action configuration. I am not sure why is that. Are they really missing from the list of dependencies of the module? If yes, how does it even get installed by users? Did I miss a step that would have installed them?

I'll leave the discussion about this for later, after the initial version of the CI is enabled.

Supported versions of Redis and Perl

I added 3 versions of Redis to the matrix and tried a number of versions of Perl, but it worked only with Perl 5.36.

Also I only tried this inside a Linux-based Docker container. Not on Windows and not on Mac OSX. I don't know if the module is even intended to support those platforms.

Adding a test

After sending the Pull-reques I went for a long walk. That helped me realize that I never told the tests what is the hostname of the Redis servers. So I guess the tests of the module never actually tried to access the server.

When I got back in-front of the computer I looked at the tests and indeed they were very shallow never trying to execute any operation that would need a server. So I created one, basically copying the example in the documentation of the module and put it in the t directory where it belongs. The test will received the name of the Redis host via an environment variable called REDIS_HOST. If this variable is not set these tests will be skipped. You can see the test-file here:

examples/Net-Async-Redis-XS/set_get.t

use strict;
use warnings;

use Test::More;
use Future::AsyncAwait;
use IO::Async::Loop;
use Net::Async::Redis::XS;

my $key = 'some-key';
my $value = 'some-value';

my $host = $ENV{REDIS_HOST};

plan skip_all => 'Set REDIS_HOST to run this test' if not $host;

my $loop = IO::Async::Loop->new;
$loop->add(my $redis = Net::Async::Redis::XS->new);
$redis->configure(host => $host);
await $redis->connect;
await $redis->set($key, $value);
my $result = await $redis->get($key);
is $result, $value;

done_testing();

I've also update the GitHub Action configuration file so it will set the REDIS_HOST environment variable to redis, the name of redis service that was included in the configuration file.

GitHub Actions configuration file

I've included here the configuration file for future reference as the one in the project, if it even gets accepted, will probably change.

examples/Net-Async-Redis-XS/ci.yml

name: CI

on:
  push:
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: '42 5 * * *'

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        # Perl 5.30, 5.32., 5.34 all failed with all 3 versions of Redis
        perl: [ '5.36' ]
        redis: ['6.0', '7.0', 'latest']
        #redis: ['latest']

    services:
      redis:
        image: redis:${{matrix.redis}}

    runs-on: ubuntu-latest
    name: Perl ${{matrix.perl}} Redis ${{matrix.redis}}
    container: perl:${{matrix.perl}}

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Show Perl Version
      run: |
        perl -v

    - name: Install Modules
      run: |
        cpanm --installdeps .
        cpanm --verbose Test::Fatal
        cpanm Net::Async::Redis
        cpanm Test::MemoryGrowth

    - name: Show Errors on Ubuntu
      if:  ${{ failure() && matrix.runner == 'ubuntu-latest' }}
      run: |
         cat /home/runner/.cpanm/work/*/build.log

    - name: Run tests
      env:
        AUTHOR_TESTING: 1
        RELEASE_TESTING: 1
        REDIS_HOST: redis
      run: |
        perl Makefile.PL
        make
        make test
        prove -b xt/

Pull-Request

I sent the Pull-Request. Let's see if it gets commented on or accepted.

I am going to combine the ideas and write an article every day on how I configured Continuous Integration for an Open Source project and I would like to challenge you to do the same!

This is going to be a series of articles (and videos) on setting up Continuous Integration (CI) for Open Source projects.

Each day during December 2022 (well, at least till the 25th) I am going to configure CI for an open source project.

Every day I'll try to find something different. A different programming language, a different VCS hosting service, maybe some different extras that are needed (a database, etc.)

However this challenge is not only for me. I hope that you will also join me on this crazy journey, find a project that does not have CI yet, write some test if there are non, set up the CI, send the pull-request, and then post a link to it as a comment on the days post.

The VCS hosting services: I know most of the open source projects are hosted on GitHub. A lot less on GitLab. I'd be glad to hear about other hosting services where there are Open Source projects.

Where can you find projects? You can search for them on the web site of the VCS hosting service or on one of the digger projects I've been building. The currently existing projects are PyDigger for Python, CPAN Digger for Perl, and the Ruby Digger.

Let's get started!

• Test and CI for MoreBeautifulPython
• Add GitHub Action CI to the Net-Async-Redis-XS Perl module
• CI for farmworld, a Python package - a failed attempt
• GitHub Action CI for wsblib Python
• GitHub Actions for Colombian Spanish. In Python
• CI for Win32-Wlan Perl module
• GitHub Actions CI for the pcr_optimizer Python package
• Be pragmatic setting up CI for the RDF::KV
• Adding CI to Perl::Efl - sometimes you need to do some extra work
• CI for Mojo-UserAgent-Cached and Plack-Middleware-Greylist
• GitHub Actions CI for the rdf-kv Ruby Gem
• CI for wp2txt Ruby project
• CI for sdk-py Python packages
• CI for a Python project of the National Institute of Standards and Technology
• CI For the Log::Any Perl module
• Failing CI to help fixing filesystem issues in a Python package
• Moving from Travis-CI to GitHub Actions for Marpa::R2
• A JavaScript bug and GitHub Workflow CI for the four-pillars Ruby gem
• GitHub Workflow for ruby-request-builder
• CI for Plack-Middleware-LogAny
• CI for the draftjs_html Ruby Gem
• CI for multi_string_replace Ruby Gem
• Full gas in neutral - CI for prepper
• CI using timescaledb a PostgreSQL based time series database
• CI for perl5-MIME-Types
• CI for Data::Alias in Perl - including threaded perl

For the full list visit the GitHub Actions book.

• Bash

• Crystal

• Perl with Build.PL

• Perl with Dist::Zilla

• Perl with Makefile.PL

• Perl with Makefile.PL using perldocker/perl-tester image

• PostgreSQL

• MySQL

• Python

• Redis

• Rust

• Solr

• GitHub Actions with parameters

• Run code if file changes

• Incremental caching demo using S3 compatibale object storage of Linode

• Cache restore and save (not doing what I wanted)

• Environment variables

• Check the owner - don't run in forks.

• Run only on the main branch and the pr/* branches to reduce CI runs in the organization, but allow CI runs for contributors in their forks

• Create Reusable Github Action workflow and use it in another workflow file

• Reuse a GitHub Action workflow from another (public) repository.

• GitHub jobs and workflows depending on each other

• Avoid duplicate triggers (push and pull_request for maintainers)

• List of files changed since the most recent push or in the current pull-request

• Try others

It is not like that. Each letter, each number, each punctuation (e.g. !?:) has a number and that number is stored.

Then the various programs you use to show that file will automatically convert the number to the drawing you are familiar with.

The generic name of all these "drawings" (digits, letters, punctuation, even emojies) is character or glyph. So we say we type in characters. Each character has a number associated with based on the standard called Unicode. Each such value is then stored on the disk in one of the encodings. What we see here is the most common representation called UTF-8.

We are going to use a program called xxd to show the real numbers representing each character in our files. It is available on Linux and I think on Mac OSX as well, and I am quite sure you can find something similar for Windows as well.

Actually this program shows the hexa value of each byte. A byte can hold a number between 0-255 or in hexa between 0-ff. As you'll later see some characters are represented by a single byte. Some by 2 and some even by 4.

Hexa numbers are represented by 0-9 and then a-f or A-F. The capitalization of these letters does not matter.

This program displays the hexadecimal values of each byte. That is, it uses base 16. We are going to also convert the numbers to base 10 as that's more familiar for most of us.

Hello World

This is the file I created. This is how you'd see it if you opened it with any text editor:

examples/hello_world.txt

Hello Wold!
Have a nice day!

This is how I run the xxd program:

$ xxd -c 1 -d examples/hello_world.txt

(-c 1 tells xxd to show only one byte per line, -d tells it to show the first column as decimal numbers)

This is its output:

00000000: 48  H
00000001: 65  e
00000002: 6c  l
00000003: 6c  l
00000004: 6f  o
00000005: 20
00000006: 57  W
00000007: 6f  o
00000008: 6c  l
00000009: 64  d
00000010: 21  !
00000011: 0a  .
00000012: 48  H
00000013: 61  a
00000014: 76  v
00000015: 65  e
00000016: 20
00000017: 61  a
00000018: 20
00000019: 6e  n
00000020: 69  i
00000021: 63  c
00000022: 65  e
00000023: 20
00000024: 64  d
00000025: 61  a
00000026: 79  y
00000027: 21  !

The left-most column shows the location of the bytes. It is just an index starting from 0. Then we see the hexadecimal (hexa) values in the file and finally, on the right hand side, we see our real characters. More-or-less.

Here you can see that the capital H is represented by the hexa number 48 which is decimal 72 (4*16+8) You can see a subset of the Unicode characters. I found it by searching for "Latin capital letter h" on that page.

Then the letter e is hexa 65 which is decimal 101.

The next interesting thing comes on the 6th row (index 5) where you can see that a space is represented by the hexa number 20 which is decimal 32.

A bit further (index 10) you can see an exclamation mark ! which is represented by hexa 21 which is decimal 33.

Then there is a dot . in this view. This dot is used by xxd to show that there is something, but that it cannot properly display it. There are several so-called control characters that have a number in the Unicode table, but that instead of representing a drawing (a character or glyph) the give an instruction to the computer what to do. For example, strange as it is, there is one called the "bell character" (hexa 7, decimal 7) that will make a beeping sound when the computer encounters it.

In our case it the dot is shown instead of a character called "line feed" or LF (hexa a or 0a, decimal 10).

This tells the computer, well, at least Linux and Mac OSX, that the next character should be displayed in the next line. If you look at how the file is displayed "normal" you will see that indeed the next thing, the letter H is displayed on a new line. On Windows this works slightly differently, but I don't want to get into that. At least not now.

The reset is easy as we only have Latin letters, spaces and an exclamation point. All of which we have already seen.

Hungarian

Next we'll see some text that uses Latin letters, but with a few additional characters.

English has 5 vowels: a, e, i, o, u, but depending on the word and the country where you are from, they might sound in various ways.

Hungarian has 14 vowels, each one has its distinct sound and they only slightly change based on the area the person is from. They are represented using the 5 Latin vowels with some extra accent marks on the top.

This is how they look like:

examples/hungarian.txt

aáeéiíoóöőuúüű

This is how they are represented in a file:

$ xxd -c 1 -d examples/hungarian.txt

00000000: 61  a
00000001: c3  .
00000002: a1  .
00000003: 65  e
00000004: c3  .
00000005: a9  .
00000006: 69  i
00000007: c3  .
00000008: ad  .
00000009: 6f  o
00000010: c3  .
00000011: b3  .
00000012: c3  .
00000013: b6  .
00000014: c5  .
00000015: 91  .
00000016: 75  u
00000017: c3  .
00000018: ba  .
00000019: c3  .
00000020: bc  .
00000021: c5  .
00000022: b1  .

The letter "a" did not cause any problems. It is hexa 61 which is decimal 97.

Given that a single byte can hold only up to 255 and there are way more characters in the world, the Unicode table has numbers that are higher than 255 and thus their will take up 2 or more bytes.

The á letter is represented by the bytes c3 and a1 (decimal 50081) in rows 1 and 2 above. See the row about UTF-8 on this page.

Because xxd shows us the content byte-by-byte it won't combine these two bytes and thus it uses dots in both of these lines to indicate that there is something it cannot really show.

Then comes the letter "e" hexa 65 decimal 101 which is easy again. See here.

The the letter "é" hexa c3 a9, decimal 50089. See here.

The rest is similar.

Spanish

Spanish has the famous ñ that actually sounds like the Hungarian ny, but let's not go into the Hungarian double and triple letters.

This is the text:

examples/spanish.txt

mañana

These are the numbers behind it:

$ xxd -c 1 -d examples/spanish.txt
00000000: 6d  m
00000001: 61  a
00000002: c3  .
00000003: b1  .
00000004: 61  a
00000005: 6e  n
00000006: 61  a

The only thing that stands out here is the ñ that is represented by c3 b1 (decimal 50097) See here.

Hebrew

Hebrew, Arabic, and Persian text can add an extra level of fun. Not just because they use a different alphabet, but also because they are written from right to left. Here is an example in Hebrew:

examples/hebrew.txt

שלום עולם

And here are the numbers behind the scene:

$ xxd -c 1 -d examples/hebrew.txt
00000000: d7  .
00000001: a9  .
00000002: d7  .
00000003: 9c  .
00000004: d7  .
00000005: 95  .
00000006: d7  .
00000007: 9d  .
00000008: 20
00000009: d7  .
00000010: a2  .
00000011: d7  .
00000012: 95  .
00000013: d7  .
00000014: 9c  .
00000015: d7  .
00000016: 9d  .
00000017: 0a  .

The only familiar item here is the space index 8 that is represented by the hexa 20 decimal 32 as in the other cases.

The letter ש is the first character, on the right-hand side of the text. It is represented by d7 a9 (decimal 55209). See it here.

Then comes ל the second character from the right. It is d7 9c (decimal 55196). See here

Korean, Japanese, Chinese, Thai, Urdu, ...

I won't go into that as I am not familiar with either of those

Emojies

However emojies are very common and well understood. Sort of.

examples/emojies.txt

👷👸👹👺👻✍👼👽👾👿💀💁💂

$ xxd -c 1 -d examples/emojies.txt
00000000: f0  .
00000001: 9f  .
00000002: 91  .
00000003: b7  .
00000004: f0  .
00000005: 9f  .
00000006: 91  .
00000007: b8  .
00000008: f0  .
00000009: 9f  .
00000010: 91  .
00000011: b9  .
00000012: f0  .
00000013: 9f  .
00000014: 91  .
00000015: ba  .
00000016: f0  .
00000017: 9f  .
00000018: 91  .
00000019: bb  .
00000020: e2  .
00000021: 9c  .
00000022: 8d  .
00000023: f0  .
00000024: 9f  .
00000025: 91  .
00000026: bc  .
00000027: f0  .
00000028: 9f  .
00000029: 91  .
00000030: bd  .
00000031: f0  .
00000032: 9f  .
00000033: 91  .
00000034: be  .
00000035: f0  .
00000036: 9f  .
00000037: 91  .
00000038: bf  .
00000039: f0  .
00000040: 9f  .
00000041: 92  .
00000042: 80  .
00000043: f0  .
00000044: 9f  .
00000045: 92  .
00000046: 81  .
00000047: f0  .
00000048: 9f  .
00000049: 92  .
00000050: 82  .

The first one is apparently called construction worker It is represented by 4 bytes. In hexa: F0 9F 91 B7 (in decimal 4,036,989,367. Yes that's a big number).

The rest are similar.

Unicode Table

You can copy any character, including an emoji from this article or from elsewhere and paste it in the search form of the Unicode table to find the information about it.

Conclusion

There is a lot more to say about representation of content in files, both historically and the current best practices, but I think this will be enough for now. I'll cover some related subjects later on.

• Slides
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

Go to the page of this article on DEV.to and press the END button on your keyboard. It will jump to the bottom of the page where you'll see a footer with a number of links including one to Forem the platform DEV runs on.

However, if you open the main page of DEV or the listings of one of the tags, you will notice that after a very short period of time more content is loaded and the links disappear. The automatic loading of additional content at the bottom of the page is called "infinite scrolling". Unfortunately this is rendering the footer unusable on these pages making the whole experience very frustrating.

So frustrating that I decided to write a whole post about it :-)

I seemed to remember that I already complained about it, but did not remember where. I was also wondering if it only frustrates me or others as well.

So I went to the GitHub repository of the project and found several related issues:

• Infinte scrolling needs to be fix on homepage
• Footer should be viewable

And a discussion Footer should be viewable

I have not been able to read all the comments, but I am surprised this issue still exists. I think I'd just remove the footer from all the pages that have infinite scrolling.

• Slides
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Docker Install curl
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• MetaCPAN
• Perl Advent
• Perl Advent on GitHub
• Is it down for me?
• What is My DNS

• Slides
• Docker Ubuntu htop
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Docker Ubuntu htop
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Create Docker image from Dockerfile
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Create your own Docker image
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Use Ubuntu Docker image
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Docker Hub
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Docker Exercise 1
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Run and remove Docker container
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

• Slides
• Docker busybox
• Source of slides on GitHub
• YouTube channel
• LinkedIn page

Finally we get to the point where we actually start to use Docker.

On Linux and macOS open the Terminal, on Windows open the Command Prompt (click on the Windows button on your keyboard, type in "cmd" and press ENTER). I'd also recommend you enlarge this window so you will see more content.

Type in the following command:

docker run hello-world

At this point Docker will try to start a container using the latest version of the image called hello-world. If this is the first time you try to do this, you won't have this images on your computer so you will see a warning:

Unable to find image 'hello-world:latest' locally

Then Docker will automatically try to download the image from the Docker HUB and will let you know about this:

latest: Pulling from library/hello-world

It shows you some progress and that it managed to download it:

2db29710123e: Pull complete
Digest: sha256:c77be1d3a47d0caf71a82dd893ee61ce01f32fc758031a6ec4cf1389248bb833
Status: Downloaded newer image for hello-world:latest

Then it will start a Docker container based on the hello-world:latest image.

This will print some text on the screen starting with

Hello from Docker!

The full output will look similar to this:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:c77be1d3a47d0caf71a82dd893ee61ce01f32fc758031a6ec4cf1389248bb833
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Once the text is printed to the screen the container will stop running.

This is not a very sophisticated image, it can only print this text and has no further functionality.

After Hello World - check the status

Before going on and trying to do more complex things with other images, let's look around to see what happened on the hard-disk of our computer.

Run

docker ps -as

This will show you a report with a number of columns. The -a tells it to show all the containers and the -s told it to include the SIZE column.

This list all the containers that have ran and are still around on your disk.

IF this is a clean computer where you only ran the above command then this is more or less what you will see.

In the first column you will see the CONTAINER ID. This will be different as it is the ID of the container which is the runtime version of the image. It changes from run to run.

The 2nd column is the name of the IMAGE which is hello-world as that's what we used.

COMMAND is the command that was executed inside the container. This was the default command baked in the image.

CREATED is the time when the container was created.

STATUS is, well the status of the container. Currently it reports that this container has stopped running.

PORTS is not interesting now.

NAME holds the name of the container. If the user does not provide a name, and we did not provide a name, than docker will pick a random name created from two random words.

SIZE is how much space the container takes up on the disk.

The command:

docker ps -as

The output:

CONTAINER ID   IMAGE        COMMAND   CREATED         STATUS                     PORTS  NAMES          SIZE
20dc20bcac8f   hello-world  "/hello"  31 minutes ago  Exited (0) 31 minutes ago         jovial_bouman  0B (virtual 13.3kB)

docker ps

If I ran docker ps without the extra parameters then it would only show the titles as the command without the -a (all) flag only shows the currently running containers and ours has already finished running.

docker ps -a

shows all the containers that are running and all the containers that have stopped but have not been deleted yet.

docker images

If you type in

docker images

you will see the list of images on your computer.

REPOSITORY              TAG                IMAGE ID       CREATED         SIZE
hello-world             latest             feb5d9fea6a5   14 months ago   13.3kB

Here you can see the images that were downloaded or that were created locally.

You remember as we ran the docker run ... command docker told us that it cannot find the images locally and so it downloaded the image called hello-world with the latest tag.

In the first column you can see the name of the image. The second column is the tag. In this case it is latest.

The IMAGE ID is a short version of the unique ID of the image.

CREATED shows when was the image created. In our case this is an image that was created 14 month ago.

SIZE is the size of the whole image.

As you can see this is a really, really small image. Basically just the text we saw.

We can see that there are images and containers which are the runtime copies of the images.

Run Hello World again

docker run hello-world

This time it does not say that it can't find the image, and it does not need to download the image as we already have it locally. It only prints the text starting with Hello from Docker!.

Now we take a look at the status of the containers:

docker ps -as

The result will look something like this:

CONTAINER ID   IMAGE        COMMAND   CREATED        STATUS                    PORTS NAMES             SIZE
f4b4ef0de525   hello-world  "/hello"  7 seconds ago  Exited (0) 6 seconds ago        focused_margulis  0B (virtual 13.3kB)
20dc20bcac8f   hello-world  "/hello"  8 minutes ago  Exited (0) 8 minutes ago        jovial_bouman     0B (virtual 13.3kB)

Now we have two lines. The second line is the same as we had earlier, except of the CREATED and STATUS columns that shows relative time and that has changed.

As you can see the CONTAINER ID is different in the two rows and the NAMES are different, but the IMAGE and the COMMAND are the same.

If we list the images using docker images we can see that the list of images has not changed. We still has that single image on our computer.

Remove container

The last thing we would like to see now is how to get rid of these containers that were created and saved on the disk.

We can use the CONTAINER ID to remote a container.

docker rm 20dc20bcac8f

If we run again

docker ps -as

Then you can see that only one line remained:

CONTAINER ID   IMAGE        COMMAND   CREATED        STATUS                    PORTS NAMES             SIZE
f4b4ef0de525   hello-world  "/hello"  27 seconds ago  Exited (0) 26 seconds ago      focused_margulis  0B (virtual 13.3kB)

We can also remove the other container in the same way after which there are no more containers left on the system as you can verify by running docker ps -as again.

Remove a Docker image

The image is still there as you can see by running docker images.

In order to remove that we can use the rmi command:

docker rmi hello-world

Untagged: hello-world:latest
Untagged: hello-world@sha256:c77be1d3a47d0caf71a82dd893ee61ce01f32fc758031a6ec4cf1389248bb833
Deleted: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
Deleted: sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359

If we run again docker images then we can see it is gone.

Next

Next we are going to see a slightly more complex example using the busybox, but before you go there I'd recommend you try the commands you just saw.

In this episode we are discussing the Docker Registry. (See all the Slides)

A Docker Registry is a place where people can upload their Docker images so others can use those images. There are private and public registries. The main public registry is the Docker Hub where you can upload public images for free and private images after certain payment.

The big cloud providers also have their own container registries which are integrated with their cloud offering.

For example there is the GCR - Google Cloud Container Registry, the ECR - AWS Elastic Container Registry and the Azure Container Registry. The smaller players also offer it, for example there is the Digital Ocean Container Registry.

In these registries you can search for various Docker images.

For example visit the Docker Hub and search for mongodb. You'll find a number of images. You can download them and use them and you will not have to worry yourself how to create a Docker image for many of the common services.

Then search for Wordpress. You'll find images that will make it easy to run Wordpress on your computer either locally or on a server. You won't have to worry much about installation and configuration and relatively easily you'll be able to upgrade it to newer versions.

You can also create your account on Docker Hub and upload your own Docker images. As long as the images are public you don't have to pay and I think you can also have one private image, but if you'd like to have more than there is some fee involved.

The idea would be that you create and upload an image and then later you, some other members of your team, or even on your production server you can download use that image.

Docker Hub is for the general use, but if you use one of the cloud providers then probably you should also use their Docker registry.

Set up your own registry

If neither of those choices are good for you, for example because you'd like to have all the images in-house, you can deploy your own registry server. It can be on premise, in the offices of your company or you can put it on some server elsewhere.

That's all for now about Docker registries. We are going to see them later on as all the images we are using are based on some other image that is in the Docker hub.

Hello World!

The next episode will cover the Hello World! of Docker containers.

These are 3 entities involved in running Docker.

Docker host

Docker host is where at the end the Docker container will run. If your computer runs Linux then the Docker host is native. It runs directly on your Linux machine. However if your container is a Linux-based container then on Windows and Mac OSX it needs a Linux-based virtual machine to run on. So in this case we will have a virtual machine running on top of the operating system that runs on our computer.

This means the performance is not going to be as good as if you were running it natively on Linux, but then on top of this host, on the top of this virtual machine that runs on Windows or on OSX, you can have several Docker containers running.

Docker Daemon

There is a docker daemon which is running on the host itself so it does not matter if your operating system is Linux or Windows or OSX it is running on top of the host.

Docker Client

Finally there is the Docker client, the one that you are actually interacting with, so when you type in docker, that's the Docker client and that too runs on your Operating System that can be either Linux, OSX, or Windows.

So docker Daemon runs on the host, but the Docker client can run on another machine. So one machine can have the client and then it can connect to the daemon running on some other host.

In this course we are going to run client and daemon on the same computer. So it's easier.

How do you start the daemon? You need to start the daemon in order to talk to it. The client you just type in docker and press ENTER. That's the client.

So we need to start the daemon which is just the Docker server. (For our purposes a daemon is just another word for a server.)

How do you do that?

It is depending on your installation.

In some cases the installation will configure the Docker daemon to start automatically when the computer boots. Depending on your use-case this might be the preferable solution.

Sometimes you might prefer that it won't start together with the Operating System, so it won't use resources when you don't need it. In that case you need to manually start the daemon.

On OSX you need to type in

open -a Docker

to launch the Docker daemon.

On Linux you'd probably write

sudo service docker start

On Windows you'll have an icon of the Docker Desktop. Just run that. It takes some time, maybe a minute to start it, but then the daemon is running.

Then you can use the Docker client (the docker command) in order to connect to the daemon and give it instructions. Run those command that we saw in the help.

I think we have one more episode (video) before we start actually doing things with Docker because I'd like to talk a bit about the Docker Registry before we start using Docker.

In order to get help with Docker you can use the following commands:

docker --help
docker help run
docker help ps
docker help images
docker help rm
docker help rmi

If you type in on the command line:

docker --help

then you will see a list of the command available for Docker. You might need to scroll-up to see the beginning of the output or on Linux and OSX you could run it through less

docker --help 2| less

You can ask for help by typing docker help and then the name of one of the commands so docker help rmi for example and then it gives you some kind of an explanation of what does the rmi command do.

You will find a lot more detailed documentation on the web site of the documentation of Docker. There you can search using the name of the command and find all kinds of documents. For example the reference page if docker rmi.

That's basically how to get help, though you can just search on your favorite search engine

• Source of slides on GitHub
• YouTube channel
• LinkedIn page

Hello! In this episode ofthe Docker course of Code-Maven we are going to check out the first command in Docker. Basically to see the version number and some information about Docker.

Normally, as usual the slides are here and we can get to the slide about Docker version where you can copy-paste the commands from.

At this time I am going also to show you on my command line.

docker --version

So here you can see that if you type in docker --version in your terminal then you are supposed to get the output of Docker:

Docker version 20.10.14, build a224086

In Windows you would probably open the CMD so you click on the "Windows" key and then you type in cmd and press ENTER. That's how you get into the Windows terminal (Command Prompt).

On Linux you open a terminal.

On Mac you also open a terminal.

Just as I have it in the video.

In there you can type docker --version.

If you type in correctly, and sometime I do, then you get to see the current version number of Docker and that indicates that Docker is available to you on the system now.

So that's one thing that you can do.

docker version

Then you can get some more information by typing in docker version without the dashes.

In the previous command we had two dashes, this time we don't have dashes. This will show the details of the curently installed Docker. This shows all kinds of information about the operating systema and so on.

docker info

The last one to show is docker info. It will show a lot more information about the machine that hopefull you will not need any time soon.

The slides are in the usual place and we are jumping ahead to the current slide: Install Docker.

In order to use Docker you need to install it on your computer. On your local computer.

There are instructions on how to install Docker on all 3 of the major operating systems. Here is the link to them. Just click on it to get to the place where you can download and install Docker from.

Obviously this page will change as time goes on. This time you can see you can install

• Docker Desktop for Mac.
• Docker Desktop for Windows
• Docker for Linux.

The best is to follow the instructions there.

I'd like to point out a couple of things.

You could install Docker using apt-get or yum depending on the distribution of your Linux Operating System, but probably it is better to use the installation method suggested in the Docker link because that way you'll get a better, a newer version of Docker.

An one more thing for Linux. There is a link to the post installation for Linux.

This is especially important if you start running Docker.

Normally you should be using your Linux machine as a regular user and not as root, but Docker needs special rights so it can access the kernel in a special way. It is better not to run Docker as root so here some instructions on how to run Docker as non-root user. So I am recommending that you follow these instructions. That's for Linux.

For Windows there are other things. Depending on the version of Windows you might be able to install the Docker Desktop for Windows but there might be problems, especially if you are using older versions of Windows. In that case you might need to use the older version.

You have to check which version is working for you. In a minute or so we are going to see how you can actually check it out.

You can use the Command Prompt by running cmd and there you can type the docker command to see that Docker actually runs on your system.

We will have a separate recording especially for Windows, but for now let's hope that it already works for you.

I provide help to individuals and teams with training, techniques, and tools for test automation, CI, and CD.

I try to help Open Source projects with the same services. This is helped by my supporters. Both individuals and corporations.

In a separates series you can read and watch small contributions to Open Source projects. Mostly configuring CI systems and adding automated tests.

Generic CI-related

• CI notification strategy
• DevOps, CI/CD, and Automation assessment questions

GitHub

• DevOps work - setting up CI and Docker for the course management project with Thomas Klausner

GitLab

• Exploring GitLab CI
• GitLab CI job building a Docker image
• Testing a Docker image created by GitLab CI
• Build Docker image in GitLab CI/CD pipeline

BitBucket

• Limit Bitbucket pipelines to specific branches

Travis-CI

Since writing these post Travis has stopped supporting Open Source projects free of charge.

• Travis-CI: Why use Continuous Integration?

• Enable Travis-CI for Continuous Integration

Create a new project with the brilliant name "abcd"

cargo new abcd

cd abcd

The directory layout:

.
├── Cargo.toml
└── src
    └── main.rs

examples/rust/abcd/Cargo.toml

[package]
name = "abcd"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

examples/rust/abcd/src/main.rs

fn main() {
    println!("Hello, world!");
}

Compile and run your program:

cargo run

I thought this could be a good opportunity to see how is the GitHub Actions configuration file offered by GitHub itself.

I made a few changes to the default file, specifically I have removed some comments and enabled the job an all branches, not only on the main branch of the project.

This was also important as it is usually better to work on a branch before sending a pull-request.

I committed the change to a branch I called CI.

The first run of the GitHub Actions failed for Ruby 3.0 and was cancelled for Ruby 2.6 and 2.7. Then I added fail-fast: false to allow all the jobs to run till completition (or failure).

This allowed me to see that the tests pass on Ruby 2.6 and 2.7.

In the next change I've disabled Ruby 3.0 and sent the pull-request with the following file:

examples/scale_rb_ci.yml

name: Ruby

on:
  push:
  pull_request:

permissions:
  contents: read

jobs:
  test:

    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        ruby-version: ['2.6', '2.7'] # , '3.0'

    steps:
    - uses: actions/checkout@v3
    - name: Set up Ruby
    # To automatically get bug fixes and new Ruby versions for ruby/setup-ruby,
    # change this to (see https://github.com/ruby/setup-ruby#versioning):
    # uses: ruby/setup-ruby@v1
      uses: ruby/setup-ruby@0a29871fe2b0200a17a4497bae54fe5df0d973aa # v1.115.3
      with:
        ruby-version: ${{ matrix.ruby-version }}
        bundler-cache: true # runs 'bundle install' and caches installed gems automatically
    - name: Run tests
      run: bundle exec rake

I found xhash package.

As this is the first time I am adding GitHub Actions to a project written in Go I got a bit luck as this one had a Dockerfile and a Makefile. Thus it was quite obvious how to run the tests.

make
make test

I also looked at a more popular Golang project to see what they have and based on that I got the following config file that I saved as .github/workflows/ci.yml. Besides a small typo I got it right and now the CI runs and tests the code on 6 different platforms.

examples/xhash-ci.yml

name: CI

on:
  push:
  pull_request:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        runner: [ubuntu-latest, macos-latest, windows-latest]
        go: [ '1.18', '1.19' ]

    runs-on: ${{matrix.runner}}
    name: OS ${{matrix.runner}} Go ${{matrix.go}}

    steps:
    - uses: actions/checkout@v2

    - name: Install Go
      uses: actions/setup-go@v3
      with:
        go-version: ${{ matrix.go }}
        check-latest: true

    - name: Show Go Version and environment
      run: |
        go version
        go env

    - name: Install dependencies
      run: |
        make

    - name: Run tests
      run: |
        make test

Welcome back to the Code-Maven Docker course.

In this video we are going to learn about the difference between Docker containers and Docker images.

When people are talking about Docker, they usually use word container, but there are basically two main things: containers and images.

They are very similar, but slightly different.

The slides as usual and jumping right to the current slide: Docker container vs. image.

So the big difference, or the way you could express this is basically that they are the same thing in different phases in their life.

A container can be seen as the run-time version of an image.

So when you create something, at the end you create an image and you store it on the disk and when you use it, it becomes a container. It is called a container.

So basically just like a Linux image, like an ISO file, for a Virtual Machine and that's frozen, you can save it on the disk and then when you start running it, it becomes a container.

It's a copy of the image, but when it is running.

If you are thinking in programming terms then some people like to compare it to the class and the instance. In Python it is the class object and the instance object in other languages it's class and instance.

The class is the definition of the things, and the instance is what being used during the runtime. So this is sort of the same with containers and images.

If it wasn't clear now, that's ok. You can come back and understand it later and I am sure as we go along it will be clearer, much clearer later on.

So thank you for listening and goto the next video.

Welcome back to the Code-Maven Docker course.

This time we'll cover the question What is really Docker?.

A reminder the links to the slides and my name is Gabor Szabo.

I am jumping right to the current slide which is quite short. The question: What is Docker?.

Usually people who create a slide like this to talk about the subject, put an image of a container on it because they talk about containerization so Docker, ....

If you are familiar with Virtual Server then Docker is basically a light-weight Virtual Server (VirtualBox).

It is described as a container. It is something you can put into stuff, your software basically. You put in different pieces of your software, your libraries, your code and so on. In a different container you'd put your database. In a different container you'd put your web-server and so on.

And then you'd use together all these containers.

So basically it is, if you are familiar with VMware, it is a very light-weight version of it. The point of being lightweight is that you can put many of these on the same machine, many more than you would be able to run with Virtual Servers.

So basically that's it. It's not some kind of a magic. You'll see in the course how you can create these Docker containers and how can you use them eventually.

Hello and welcome back to the Code-Maven Docker course. My name is Gabor Szabo.

In this episode we are going to ask the question Why to use Docker at all?

Remember this is the place where you can find the slides and I ma going directly to the page where we are discussing Why to use Docker?.

One of the biggest problems that software development encounters is the different types of environments where we run our software.

So there are the developers who develop the application on some desktop system which is usually either Windows or Mac. Sometimes Linux, but usually it is either Windows or Mac and then there might be some QA people who are checking the software - Quality Assurance (QA) and they might install it on some environment which might resemble more the actual production environment, but it's still different from the production environment. Which is probably some Linux server somewhere in the cloud and which has different capabilities.

The Operating System is different of course, the computers have different capabilities, have different rights, different rights to transfer data and so on.

So these differences make it really hard to make sure that the software that was running on the developer's machine will actually behave well on the production server as well.

In addition there is what we usually call the "dependency hell". An application depends on something and then another part of the same application depends on the same thing, but might depend on a different version of that thing. So there is a common library let's say.

How can you run these two parts of the application on the very same server?

So this issue is really, really difficult. One of the solutions is of course to force everyone to use the same exact version, but that usually means that people use the old version of something. A much better way is to use some kind of a virtualization like Docker that allows that allows us to separate the various parts of the application and each application (each part) can declare and use its own dependencies and can move forward with the dependencies as they feel fit. At their speed. So that's another thing.

And the third thing that's really hard in many development companies or many companies is what we call on-boarding of new employees or new developers.

In a reasonably large application to set up an environment, a development environment, might take days or weeks even and it might have been documented, maybe people need to go around and ask other people how to setup things and usually most of the people who get in these companies they will have a long long time, maybe months or years till they figure out all kinds of issues that don't work on their environment because they did not set it up the way it should have been set up. Or the ways other set up.

Docker, or in general virtualization can help us solve all of these problems. So the question of on-boarding.

So Docker and in general containerization or virtualization, but specifically Docker can allow us to create a standard environment in which we develop the code, check the code (quality assurance), and deploy the code. And then the differences, well they won't disappear, but they will be much smaller.

The same with on-boarding people. Once we have the standardized environment we can run on it, we can just tell the people to just download this standardized environment and run it.

They still have plenty of things to set up. Maybe their editor or IDE or all kinds of other things, but at least they don't have to waste so much time and energy by setting up their environment.

So that's about the Why Docker and we can go on now trying to figure out what Docker is.

Now what you are seeing is the opening page with a couple of links and these links are important.

The first link you can see here leads to the slides that we are going to utilize during the course.

I am going to click on this now so you can see. The link will be under the video as well so you'll be able to find it easily, but I wanted to show you how it looks like.

So these are the slides which are on my web-site and these slides are available for everyone and will be available all the time. One of the things that you can do on these slides is press the question mark (?) that you can see at the top, so I pressed the question mark and you can see all kinds of buttons for navigation.

I am going to go ahead a little bit here because one of the most important thing that you might want to click on is the h key which you can also see here at the bottom. So the h button will open some extra text here.

In many slides, but not all of them, but in many slides there will be extra text and this text is going to grow during the recording of the video course and therefore it will be there afterwards as well. So you'll be able to read this extra text and then it will be able to help you even beyond the videos. So that's part of the slides.

Then another thing is that in these slides you'll see a lot of examples. Each one of them is a file. Many of them will be files. You'll see the names of the files somewhere in the slide and instead of copy-pasting from the slide that might be less comfortable in many cases, I made it available.

So the link at the bottom (of the main page) you can see a link to the source of slides on GitHub.com. So all the slides and all the examples from all the courses I have are available in this repository and you can download by either just regularly cloning it: You click on this link and then clone it:

git clone https://github.com/szabgab/slides.git

If you are familiar with git and know how to do it.

Or, if you are not familiar with git, then you can also click on the big green button called Clone or download and download it as a zip file.

This way you'll get the most recent version of everything, but it will be a little bit harder to update things because you'll have to come here and download it again and again, but you don't need to clone it.

So that's it.

Inside you'll see the material of all of my courses, specifically the docker is the one you are going to look at at this time.

So that's it basically at the beginning. Then you'll see various videos feel free to jump around and see if some part you already know then jump ahead if you feel that you are looking for something specific then look at that video and then come back later as you prefer.

Good luck with this training course.

It is nince. It first goes over the directory structure I give it on the command line and then starts to filter files.

• First it will eliminate all the files that have a unique size
• Then it will compare the first few bytes of the files that have the same size
• Then it will compare the last few bytes of the file that have the same size
• Finally, among the still remaining candidates for equality it will compare their checksum using md4 or sha1

The problem I had is that I have many huge files. Quite a few are larger than 1 Gb and many over 10 Gb. I have about 2TB data.

It takes a lot of time to go over all the files, especially computing the checksums takes a lot of time as the files need to be read.

Add to the injury that many of these files are still located on slow-to-access external disks.

I thought it would be better if I could skip the checksum step, but there was no such flag.

So what can I do.

First I opened a ticket explaining my issue, but then I thought, what if I try to implement it?

I clones the source code. It seems it is written in C++. I've never written C++ and even with C I have very little experience.

git clone git@github.com:pauldreik/rdfind.git

Try to compile

The next thing I had to find is how to compile the code. After a few failed experiments I ran

grep release *

and found the release_new_version.txt that contained the instructions:

git clean -xdf .
./bootstrap.sh
./configure
make dist-gzip

The first problem I encountered was a missing package. I had to install it:

sudo apt-get install nettle-dev

Then I bumped intor the following error when running make:

$ make
make  all-am
make[1]: Entering directory '/home/gabor/work/rdfind'
g++ -DHAVE_CONFIG_H -I.     -g -O2 -MT rdfind.o -MD -MP -MF .deps/rdfind.Tpo -c -o rdfind.o rdfind.cc
rdfind.cc: In function ‘Options parseOptions(Parser&)’:
rdfind.cc:225:30: error: ‘numeric_limits’ is not a member of ‘std’
  225 |     o.maximumfilesize = std::numeric_limits<decltype(o.maximumfilesize)>::max();
      |                              ^~~~~~~~~~~~~~
rdfind.cc:225:45: error: expected primary-expression before ‘decltype’
  225 |     o.maximumfilesize = std::numeric_limits<decltype(o.maximumfilesize)>::max();
      |                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:658: rdfind.o] Error 1
make[1]: Leaving directory '/home/gabor/work/rdfind'
make: *** [Makefile:537: all] Error 2

Luckily a little search for the error message brought me to this page. Not surprisingly I am not the first one to see this error. Better yet, apparently it has already been fixed in a branch.

So I using git branch -a I checked and saw that there is a branch called devel. I switched to that branch and ran the whole process again. This time make worked and it create a file called rdfind in the root directory of the project.

Now that I knew I can compile and build the executable I started to look at the code.

First I wanted to find where it processes the command-line parameters so I'll be able to add a parameter called -nochecksum.

I searched for the string "argv" as that is the name of the variable that holds the command-line parameters in many languages. Quite soon I found that is it in the rdfind.cc file.

examples/rdfind-checksum.diff

commit d3c6a523bf531fe4bf46c8b13397b111b3fb6634
Author: Gabor Szabo <gabor@szabgab.com>
Date:   Thu Sep 15 14:37:50 2022 +0300

    allow for '-checksum none' to disable checksum filtering. #118

diff --git a/rdfind.cc b/rdfind.cc
index 64dd8f6..cd35314 100644
--- a/rdfind.cc
+++ b/rdfind.cc
@@ -61,7 +61,7 @@ usage()
     << " -followsymlinks    true |(false) follow symlinks\n"
     << " -removeidentinode (true)| false  ignore files with nonunique "
        "device and inode\n"
-    << " -checksum           md5 |(sha1)| sha256\n"
+    << " -checksum           md5 |(sha1)| sha256 | none\n"
     << "                                  checksum type\n"
     << " -deterministic    (true)| false  makes results independent of order\n"
     << "                                  from listing the filesystem\n"
@@ -103,6 +103,7 @@ struct Options
   bool followsymlinks = false;        // follow symlinks
   bool dryrun = false;                // only dryrun, don't destroy anything
   bool remove_identical_inode = true; // remove files with identical inodes
+  bool checksum = true;      // use some checksum
   bool usemd5 = false;       // use md5 checksum to check for similarity
   bool usesha1 = false;      // use sha1 checksum to check for similarity
   bool usesha256 = false;    // use sha256 checksum to check for similarity
@@ -174,8 +175,10 @@ parseOptions(Parser& parser)
         o.usesha1 = true;
       } else if (parser.parsed_string_is("sha256")) {
         o.usesha256 = true;
+      } else if (parser.parsed_string_is("none")) {
+        o.checksum = false;
       } else {
-        std::cerr << "expected md5/sha1/sha256, not \""
+        std::cerr << "expected md5/sha1/sha256/none, not \""
                   << parser.get_parsed_string() << "\"\n";
         std::exit(EXIT_FAILURE);
       }
@@ -237,8 +240,10 @@ parseOptions(Parser& parser)
   // done with parsing of options. remaining arguments are files and dirs.
 
   // decide what checksum to use - if no checksum is set, force sha1!
-  if (!o.usemd5 && !o.usesha1 && !o.usesha256) {
-    o.usesha1 = true;
+  if (o.checksum) {
+    if (!o.usemd5 && !o.usesha1 && !o.usesha256) {
+      o.usesha1 = true;
+    }
   }
   return o;
 }
@@ -356,17 +361,19 @@ main(int narg, const char* argv[])
     { Fileinfo::readtobuffermode::READ_FIRST_BYTES, "first bytes" },
     { Fileinfo::readtobuffermode::READ_LAST_BYTES, "last bytes" },
   };
-  if (o.usemd5) {
-    modes.emplace_back(Fileinfo::readtobuffermode::CREATE_MD5_CHECKSUM,
-                       "md5 checksum");
-  }
-  if (o.usesha1) {
-    modes.emplace_back(Fileinfo::readtobuffermode::CREATE_SHA1_CHECKSUM,
-                       "sha1 checksum");
-  }
-  if (o.usesha256) {
-    modes.emplace_back(Fileinfo::readtobuffermode::CREATE_SHA256_CHECKSUM,
-                       "sha256 checksum");
+  if (o.checksum) {
+    if (o.usemd5) {
+      modes.emplace_back(Fileinfo::readtobuffermode::CREATE_MD5_CHECKSUM,
+                         "md5 checksum");
+    }
+    if (o.usesha1) {
+      modes.emplace_back(Fileinfo::readtobuffermode::CREATE_SHA1_CHECKSUM,
+                         "sha1 checksum");
+    }
+    if (o.usesha256) {
+      modes.emplace_back(Fileinfo::readtobuffermode::CREATE_SHA256_CHECKSUM,
+                         "sha256 checksum");
+    }
   }
 
   for (auto it = modes.begin() + 1; it != modes.end(); ++it) {

At the end I felt it is better to change the already existing -checksum</hl flag and allow it to receive the value "none" to avoid the checsum calculations.

I was lazy and did not add any new tests, but it could be done later if the author of the code requests it.

After making the change I pushed out the code to my forked repository of the project. The GitHub Actions were triggered and I saw that all the tests pass.

I was very pleasantly surprised by the ease of finding the instructions how to build the code, the existing tests and the ease to make the adjustments.

We'll see if the Pull-Request is accepted.

• Introduce Continuous Integration (CI) to the starpy Python project

• Fix URL to repository of the glob-linters Python project

• Patch rdfind, a C++ project, allowing the user to disable checksum comparision

• Testing interval-timer, a Python module

• Make it easy to setup CI environment - python (once-utils)

• Python helper: Let all CI jobs fail separately

• Automated test for sssimp, a simple static site generator in Python

• Add GitHub actions to the xhash project written in Go

• Add GitHub Actions to the scale_rb Ruby project

• All the Perl-related projects, videos, and blog posts are on the Perl Maven web site

How to find Open Source projects

A few ideas on how to find projects to contribute to.

• Python: On PyDigger find projects that have Link to GitHub, but no CI
• Perl: The CPAN Digger project has a script called bin/cpan-digger that can create a report with some logs of recently uploaded Perl modules.
• In general: Visit the Explore topics page of GitHub, select a topic, select the language, sort by "Recently uploaded". In the list look for projects with few stars. They are probably the newer ones with less things already in place. For example this is the link for Go

The video started at the PyDigger project.

The points I made

• Make sure you have the python requirements listed in the requirements.txt file
• Make sure you can run pytest and it will find your tests. (test filenames start with test_*
• Having a README.md is also a nice thing with instruction how to set up an environment and how to run the tests.

This is the content of the .github/workflows/ci.yaml file from the project that configures GitHub Actions.

examples/starpy/ci.yaml

name: CI

on:
  push:
  pull_request:

jobs:
  build_python:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.7", "3.8", "3.9", "3.10"]

    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install dependencies
      run: pip install -r requirements.txt

    - name: Check Python version
      run: python -V

    - name: Test with pytest
      run: pytest -vs

There are many well known services that either run on your computer by default or you can install them. The former might be the service waiting for you to type in your username and password on the terminal to log in, or waiting for an ssh connection, or running your cron jobs. The latter might be a web-server, a database server. (Things that are called servers are usually installed as services so they would run in the background.)

Services are usually configured to start when the computer boots up to be able to provide service even without the login of any user.

How can we create a service / daemon?

The following file is just a very simple script that will print the timestamp in a file every 1 second. There is noting special about it.

examples/daemon/timestamp.py

import datetime
import time
import sys

param = ""

if len(sys.argv) > 1:
  param = sys.argv[1]

while True:
   with open("/tmp/timestamp.log", "a") as fh:
       fh.write(param + ' ' + str(datetime.datetime.now()) + "\n")
   time.sleep(1)

You can run the script on the command line:

python examples/daemon/timestamp.py

and then in a separate terminal you can observe the file growing by running the following command:

tail -f /tmp/timestamp.log

In order to stop the script you need to switch back to the first terminal and press Crtl-C.

Then you can try to run the script with some parameter. e.g.:

python examples/daemon/timestamp.py hello

And switch to the 2nd terminal to observe the output.

The service / daemon configuration file

In order to turn this into a service we need to create a service configuration file:

examples/daemon/timestamp.service

[Unit]
Description=Demo Service

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/bin/python3 /home/gabor/work/code-maven.com/examples/daemon/timestamp.py "my daemon"
Restart=always
WorkingDirectory=/tmp
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

The most important field here is:

ExecStart

Create a symbolic link from the system-wide location of all the services in /usr/lib/systemd/system/ to the place where our service configuration file is. (Alternatively you could also copy the timestamp.service file.

sudo ln -s /home/gabor/work/code-maven.com/examples/daemon/timestamp.service  /usr/lib/systemd/system/timestamp.service

Then we need to tell the daemon-service no reload all the configuration files. We need to do this every time we add a new service or change the configuration file of a service.

sudo systemctl daemon-reload

Then we can start the service

sudo systemctl start timestamp.service

At this time we should be able to use the tail command to observe how the service works and the file grows one line every second.

tail -f /tmp/timestamp.log

Troubleshooting pitfalls, potential issues and fixes

If you the timestamp.log file does not grow you it means we had some issues.

You can observe the status of the service by running the following command:

sudo systemctl status timestamp.service

You can also look at the log in the syslog file:

less /var/log/syslog

While writing this article I had two issues:

Issue one: typo in command

At first I had a typo in the command on the ExecStart line. That was easy to fix.

Issue two: rights on the log file

Even after the typo was fixed I still did not see the log file growing.

sudo systemctl status timestamp.service

got me this output:

× timestamp.service - Demo Service
     Loaded: loaded (/lib/systemd/system/timestamp.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-08-21 09:35:46 IDT; 7s ago
    Process: 2046747 ExecStart=/usr/bin/python3 /home/gabor/work/code-maven.com/examples/daemon/timestamp.py mydaemon (code=exited, status=1/FAILURE)
   Main PID: 2046747 (code=exited, status=1/FAILURE)
        CPU: 119ms

Aug 21 09:35:46 code-maven systemd[1]: timestamp.service: Scheduled restart job, restart counter is at 5.
Aug 21 09:35:46 code-maven systemd[1]: Stopped Demo Service.
Aug 21 09:35:46 code-maven systemd[1]: timestamp.service: Start request repeated too quickly.
Aug 21 09:35:46 code-maven systemd[1]: timestamp.service: Failed with result 'exit-code'.
Aug 21 09:35:46 code-maven systemd[1]: Failed to start Demo Service.

Then I looked at the syslog:

less /var/log/syslog

and saw this:

Aug 21 09:37:44 code-maven python3[2047136]:   File "/home/gabor/work/code-maven.com/examples/daemon/timestamp.py", line 11, in <module>
Aug 21 09:37:44 code-maven python3[2047136]:     with open("/tmp/timestamp.log", "a") as fh:
Aug 21 09:37:44 code-maven python3[2047136]: PermissionError: [Errno 13] Permission denied: '/tmp/timestamp.log'

Apparently the daemon that is currently configured as user root does not have permissions to open the log file for writing.

$ ls -l /tmp/timestamp.log
-rw-rw-r-- 1 gabor gabor 504 Aug 21 09:35 /tmp/timestamp.log

At this point I could have changed the configuration file of the service to run as user gabor, I could have changed the right on the file so everyone can write to it.

I decided to get rid of the file and let the daemon create it by itself:

$ rm -f /tmp/timestamp.log

Then I started the daemon again. This time it started to work.

$ sudo systemctl status timestamp.service

● timestamp.service - Demo Service
     Loaded: loaded (/lib/systemd/system/timestamp.service; disabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-08-21 09:54:13 IDT; 3s ago
   Main PID: 2049558 (python3)
      Tasks: 1 (limit: 18981)
     Memory: 3.5M
        CPU: 26ms
     CGroup: /system.slice/timestamp.service
             └─2049558 /usr/bin/python3 /home/gabor/work/code-maven.com/examples/daemon/timestamp.py "my daemon"

Aug 21 09:54:13 code-maven systemd[1]: Started Demo Service.

I can observe the file as well. Now it is owned by user root:

$ ls -l /tmp/timestamp.log
-rw-r--r-- 1 root root 4508 Aug 21 09:54 /tmp/timestamp.log

Stop the service

sudo systemctl stop timestamp.service

Restart the service

sudo systemctl restart timestamp.service

Conclusion

It is very easy to convert a script to a daemon, but one needs to be careful with the rights on files and also the environment that is different for a daemon and for a script that runs on the command line.

We only scratched the surface of the daemonization. There are tons of other options you can deal with, but this should already get you started.

For more details see the documentation of the sidenav

examples/materializecss/side-nav-with-hierarchy.html

<!DOCTYPE html>
<html>
  <head>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css">
    <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">

    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <title>Materialize: Sidenav with vanilla JavaScript</title>
  </head>

  <body>
    <div class="container">
      <!-- start the definition of the sidenav -->
      <ul id="slide-out" class="sidenav">
        <li><a href="#!">First Sidebar Link</a></li>
        <li><a href="#!">Second Sidebar Link</a></li>
        <li class="no-padding">
          <ul class="collapsible collapsible-accordion">
            <li>
              <a class="collapsible-header">Dropdown<i class="material-icons">arrow_drop_down</i></a>
              <div class="collapsible-body">
                <ul>
                  <li><a href="#!">First</a></li>
                  <li><a href="#!">Second</a></li>
                  <li><a href="#!">Third</a></li>
                  <li><a href="#!">Fourth</a></li>
                </ul>
              </div>
            </li>
          </ul>
        </li>
      </ul>
      <!-- end the definition of the sidenav -->

      <div class="row">
        <!-- the button to open the sidenav -->
        <div class="col s1"><a href="#" data-target="slide-out" class="sidenav-trigger">show main menu</a></div>
        <div class="col s11">&nbsp;</div>

        <div class="col s12">
          "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et
           dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip
           ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
           fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
           mollit anim id est laborum."
        </div>
      </div>


    </div>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
  </body>
</html>

<script>
document.addEventListener('DOMContentLoaded', function() {
    const elem = document.getElementById('slide-out');
    const options = {};
    const instance = M.Sidenav.init(elem, options);
    // instance.open()

    const collapsibles = document.querySelectorAll('.collapsible');
    const coptions = {};
    const collapsible_instances = M.Collapsible.init(collapsibles, coptions);
});
</script>

view

In order to open the side navigation bar the user needs to click on the "show main menu" link.

We had to initialize both the sidenav and the collapsible in the JavaScript code.

In order to see the little triangle to open the second level of items we had to include the Material fonts (the second stylesheet include at the top of the file).

For more details see the documentation of the sidenav

examples/materializecss/2-side-nav-vanilla-javascript.html

<!DOCTYPE html>
<html>
  <head>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css">
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <title>Materialize: Sidenav with vanilla JavaScript</title>
  </head>

  <body>
    <div class="container">

      <!-- start the definition of the sidenav -->
      <ul id="main-slide-out" class="sidenav">
          <li><a href="#!">Link</a></li>
          <li><div class="divider"></div></li>
          <li><a class="subheader">Subheader</a></li>
          <li><a href="#!">Other link</a></li>
          <li>Outdented Title
            <ul>
                <li><a href="#!">Alfa</a></li>
                <li><a href="#!">Beta</a></li>
            </ul>
          </li>
          <li><div class="divider"></div></li>
          <li><a href="">Title</a>
            <ul>
                <li><a href="#!">Alfa</a></li>
                <li><a href="#!">Beta</a></li>
            </ul>
          </li>
      </ul>

      <ul id="secondary-slide-out" class="sidenav">
          <li><a class="subheader">Secondary</a></li>
          <li><div class="divider"></div></li>
          <li><a href="#!">Yet another link</a></li>
      </ul>
      <!-- end the definition of the sidenav -->

      <div class="row">
        <!-- the button to open the sidenav -->
        <div class="col s1"><a href="#" data-target="main-slide-out" class="sidenav-trigger"><i class="material-icons">show main menu</i></a></div>
        <div class="col s1"><a href="#" data-target="secondary-slide-out" class="sidenav-trigger"><i class="material-icons">show secondary menu</i></a></div>
        <div class="col s11">&nbsp;</div>

        <div class="col s12">
          "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et
           dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip
           ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
           fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt
           mollit anim id est laborum."
        </div>
      </div>


    </div>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
  </body>
</html>

<script>
document.addEventListener('DOMContentLoaded', function() {
    const elem1 = document.getElementById('main-slide-out');
    const options1 = {
        'edge': 'left'
    };
    const instance1 = M.Sidenav.init(elem1, options1);
    // instance1.open()


    const elem2 = document.getElementById('secondary-slide-out');
    const options2 = {
        'edge': 'right'
    };
    const instance2 = M.Sidenav.init(elem2, options2);
    // instance2.open();
});
</script>

view

In order to open the left-side navigation bar the user needs to click on the "show main menu" link. In order to open the right-side navigation bar the user needs to click on the "show secondary menu" link.

The bar automatically closes when the user clicks on the content of the page outside the navbar.

The main by sidenav is aligned to the left side of the screen. The secondary is aligned to the right-side of the screen.

At the bottom of the file you can see the JavaScript code. In this example we searched for the navbars one-by-one using their ID and then initialized them separately using (slightly) different options.

I also included a commented out line of code that would open the navbars when the page is loaded.

You could also use the open method if you wanted to open the secondary navbar when the user selected something on the primary navbar.