Politreco

Lazy libkmod compression library loading

2025-03-03T10:00:00-08:00

One new feature in kmod 34 is related to lazily loading the decompression libraries. In other words, dlopen them when/if they are needed. Although it’s desired for a tool like modinfo to be able to inspect a .ko.xz, .ko.zst or .ko.gz module, other daemons linking to libkmod would benefit from never loading them. This is the case for systemd-udevd that will load the kernel modules during boot when they are requested by the kernel.

Testing on Archlinux, it goes from this:

$ cat /proc/$(pidof systemd-udevd)/maps | grep -e libz -e liblzma
7f27a9ae8000-7f27a9aeb000 r--p 00000000 00:19 15656                      /usr/lib/liblzma.so.5.6.4
7f27a9aeb000-7f27a9b0e000 r-xp 00003000 00:19 15656                      /usr/lib/liblzma.so.5.6.4
7f27a9b0e000-7f27a9b19000 r--p 00026000 00:19 15656                      /usr/lib/liblzma.so.5.6.4
7f27a9b19000-7f27a9b1a000 r--p 00031000 00:19 15656                      /usr/lib/liblzma.so.5.6.4
7f27a9b1a000-7f27a9b1b000 rw-p 00032000 00:19 15656                      /usr/lib/liblzma.so.5.6.4
7f27a9b1b000-7f27a9b27000 r--p 00000000 00:19 15985                      /usr/lib/libzstd.so.1.5.7
7f27a9b27000-7f27a9bed000 r-xp 0000c000 00:19 15985                      /usr/lib/libzstd.so.1.5.7
7f27a9bed000-7f27a9bfe000 r--p 000d2000 00:19 15985                      /usr/lib/libzstd.so.1.5.7
7f27a9bfe000-7f27a9bff000 r--p 000e3000 00:19 15985                      /usr/lib/libzstd.so.1.5.7
7f27a9bff000-7f27a9c00000 rw-p 000e4000 00:19 15985                      /usr/lib/libzstd.so.1.5.7
7f27aa892000-7f27aa895000 r--p 00000000 00:19 7852                       /usr/lib/libz.so.1.3.1
7f27aa895000-7f27aa8a3000 r-xp 00003000 00:19 7852                       /usr/lib/libz.so.1.3.1
7f27aa8a3000-7f27aa8a9000 r--p 00011000 00:19 7852                       /usr/lib/libz.so.1.3.1
7f27aa8a9000-7f27aa8aa000 r--p 00017000 00:19 7852                       /usr/lib/libz.so.1.3.1
7f27aa8aa000-7f27aa8ab000 rw-p 00018000 00:19 7852                       /usr/lib/libz.so.1.3.1
$

to this:

$ cat /proc/$(pidof systemd-udevd)/maps | grep -e libz -e liblzma
$

… even if all modules in Archlinux are zstd-compressed.

systemd itself started doing this a few releases ago and published https://systemd.io/ELF_PACKAGE_METADATA/. That spec is also used for this new support in kmod to annotate what libraries are possibly dlopen’ed. However although it prevented libkmod from being loaded in other binaries, it didn’t prevent all these decompression libraries from being mapped in systemd-udevd since it uses libkmod.

One might wonder why not even libzstd.so is mapped. That’s because when loading the modules and the kernel supports decompressing that format, libkmod just skips any decompression: it opens the file and passes the descriptor to the Linux kernel via finit_module. /sys/module/compression shows what algorithm the Linux kernel can use for module decompression.

In kmod 34 all that is needed is to setup the build with -Ddlopen=all. More fine-grained options are also supported, allowing to specify individual libraries to dlopen. It may go away in a future release if distros just choose an all-or-nothing support.

Linux module dependencies

2024-08-13T10:00:00-07:00

With the imminent release of kmod 33, I thought it’d be good to have a post about the different types of module dependencies that we have in the Linux kernel and kmod. The new version adds another type, weak dependency, and as the name implies, is the weakest of all. But let’s revisit what are the other types first.

Hard (symbol) dependency

This is the first dependency that every appeared in kmod (and module-init-tools). A hard (or as some call, “symbol”) dependency occurs when your module calls or uses an exported symbol of another module. The most common way is by calling a function that is exported in another module. Example: the xe.ko calls a function ttm_bo_pin() that is provided and exported by another module, ttm.ko. Looking to the source:

$ nm build/drivers/gpu/drm/ttm/ttm.ko | grep -e "\bttm_bo_pin\b"
0000000000000bc0 T ttm_bo_pin
$ nm build64/drivers/gpu/drm/xe/xe.ko | grep -e "\bttm_bo_pin\b"
             U ttm_bo_pin

It is not possible to insert the xe.ko module before ttm.ko and if you try via insmod (that doesn’t handle dependencies) it will fail with the kernel complaining that the ttm_bo_pin symbol is undefined.

The manual invocations to nm illustrates what the depmod tool does: it opens the modules and reads the ELF headers. Then it takes note of all the symbols required and provided by each module, creating a graph of symbol dependencies. Ultimately that leads to module dependencies: xe ➛ ttm. This is recorded in the modules.dep file and its sibling modules.dep.bin. The former is human-readable and the latter is used by libkmod, but they contain the same information: all dependencies for all the modules. Also note that each line reflects indirect dependencies: module A calls symbol from B and B calls symbol from C will lead to A depending on both B and C. Real world example:

$ cat /lib/modules/$(uname -r)/modules.dep | grep kernel/drivers/gpu/drm/xe/xe.ko.zst:
kernel/drivers/gpu/drm/xe/xe.ko.zst: kernel/drivers/gpu/drm/drm_gpuvm.ko.zst kernel/drivers/gpu/drm/drm_exec.ko.zst kernel/drivers/gpu/drm/scheduler/gpu-sched.ko.zst kernel/drivers/gpu/drm/drm_buddy.ko.zst kernel/drivers/i2c/algos/i2c-algo-bit.ko.zst kernel/drivers/gpu/drm/drm_suballoc_helper.ko.zst kernel/drivers/gpu/drm/drm_ttm_helper.ko.zst kernel/drivers/gpu/drm/ttm/ttm.ko.zst kernel/drivers/gpu/drm/display/drm_display_helper.ko.zst kernel/drivers/media/cec/core/cec.ko.zst kernel/drivers/acpi/video.ko.zst kernel/drivers/platform/x86/wmi.ko.zst

So the xe.ko (with .zst extension since it’s compressed) directly or indirectly depends on drm_gpuvm.ko, drm_exec.ko, gpu-sched.ko, drm_buddy.ko, i2c-algo-bit.ko, drm_suballoc_helper.ko, drm_ttm_helper.ko, ttm.ko, drm_display_helper.ko, cec.ko, video.ko, wmi.ko. Same information, but using the kmod tools rather than looking at the raw index:

$ modinfo -F depends xe
drm_display_helper,ttm,drm_gpuvm,drm_suballoc_helper,video,drm_buddy,drm_exec,drm_ttm_helper,gpu-sched,cec,i2c-algo-bit

Soft dependency

There are situations in the kernel where it’s not possible or desired to use a symbol directly from another module - they may interact by registering in a subsystem, scanning a bus etc.. In this case depmod doesn’t have enough information from the ELF file about that. Yet the user would have a more complete support if both modules were available - it may even cause failures visible to the end user and not only “partial support for features”.

The softdep implementation contains 2 parts: pre and post dependencies. The post dependencies are not very much used in practice: they instruct kmod to load another module after loading the target one.

They can come from a configuration file like e.g. /etc/modprobe.d/foo.conf or from the kernel itself by embedding that info in the module. From the kernel source this is achieved by using the macro MODULE_SOFTDEP(). Example:

lib/libcrc32c.c:MODULE_SOFTDEP("pre: crc32c");

When libkmod is loading a module it will first load, in order:

hard dependencies
soft pre dependencies
target module
soft post dependencies

Historically softdeps were also a way to (mostly) get rid of install rules, in which the configuration instructs libkmod to execute something instead of loading the module - people would add an install rule to execute something and then call modprobe again with --ignore-install to fake a dependency. That could easily lead to a runtime loop which is avoided with softdep since kmod can (and does) check for loops.

Weak dependency

After explaining the other types of dependencies, back to the new addition in kmod 33. These are very similar to pre softdep: they come either from a configuration file or embedded in the module and they express a dependency that wouldn’t cause the target module to fail to load, but that may cause the initialization to export less features or fail while initializing. There is one important difference: weak dependencies don’t cause libkmod to actually load the module. Rather the dependency information may be used by tools like dracut and other tools responsible for assembling an initrd to make the module available since it may or may not be used. Why are they called “weak”? This was a borrowed terminology from “weak symbols”: a weak symbol is there, waiting to be used, but it may or may not be, with the final decision happening in the final link stage.

With weak dependencies, hopefully some of the pre softdep embedded in the kernel may be replaced: if the target module is already doing a request_module() or in some way getting the other module to be loaded, it doesn’t need a softdep that would serialize the module load order and possibly load more modules than required.

Tracepoints and kernel modules

2022-03-15T10:00:00-07:00

While debugging, it’s invaluable to be able to trace functions in the Linux kernel. There are several tools for that task: perf, ftrace (with helper tool like trace-cmd), bpftrace, etc.

For my own debug of kernel internals and in the last few years to debug the i915 module for Intel graphics, I usually use tracepoints or dynamic tracepoints (with perf probe) as the trigger for something I want to trace.

One question I get some times is “how do I trace the initial probe of a kernel module?”. At that time you (usually) don’t yet have the module loaded and as a consequence, the symbols of that module are still not available for using tracepoints. For example:

# trace-cmd list -e '^i915.*'
#

One thing to realize is that a lot of time we don’t want to debug the module loading, but rather the module binding to a device. These are 2 separate things, that happen in a cascade of events. It probably also adds to the confusion that the userspace tools to load the modules are not named very consistently: modprobe and insmod, and they end up calling the [f]init_module syscall.

The various buses available in the kernel have mechanisms for stopping the autoprobe (and hence binding to a device), when a module is loaded. With i915, all we have to do is to set a few things in the /sys/bus/pci/:

# echo 0 > /sys/bus/pci/drivers_autoprobe
# modprobe i915

With the i915 module, but not attached to any device, we are ready to attach to any tracepoint or create new dynamic probes. Let’s suppose I want to debug the i915_driver_probe()() function, and any function called by it during the initialization. This is one of the functions to initialize the GPU in i915 called when we are binding to a device.

F=i915_driver_probe

echo 0 > /sys/bus/pci/drivers_autoprobe 
modprobe i915

cd /sys/kernel/debug/tracing
cat /dev/null >  trace
echo $F > set_graph_function
echo _printk > set_graph_notrace
echo 10 > max_graph_depth
echo function_graph > current_tracer
echo 1 > tracing_on

echo 0000:03:00.0 | tee /sys/bus/pci/drivers/i915/bind
cp trace /tmp/trace.txt

echo 0 > tracing_on
cat /dev/null >  trace

With the snippet above we will start tracing whenever function i915_driver_probe() is executed. We also set a few additional parameters: set the maximum call depth to graph, disable graphing on printk() since it’s usually not very interesting. Depending on the size of your trace you may also need to increase the buffer_size_kb to avoid the ring buffer to rotate, or have something to pump data out off the ringbuffer to a file.

Even after we enable tracing by echo’ing 1 to the tracing_on file, nothing happens as we are not automatically binding to the device. In the snippet above we bind it manually, by writting the pci slot to the /sys/bus/pci/drivers/i915/bind file. We then should start seeing the function to be traced:

# head /tmp/trace.txt
# tracer: function_graph
#
# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
  3)               |  i915_driver_probe [i915]() {
  3)               |    __devm_drm_dev_alloc() {
  3)               |      __kmalloc() {
  3)               |        kmalloc_order_trace() {
  3)               |          kmalloc_order() {
  3)               |            __alloc_pages() {

Another thing very useful to do is to attach to a specific tracepoint. For example, to trace the MMIO read/write we can do the commands below. Now I’m using trace-cmd directly and omitting the device bind, that we should do after start recording:

# trace-cmd record -e i915:i915_reg_rw
Hit Ctrl^C to stop recording
^C
# trace-cmd report
...
tee-2239  [009]   778.693172: i915_reg_rw:          read reg=0x51000, len=4, val=(0xc1900, 0x0)
tee-2239  [009]   778.693174: i915_reg_rw:          write reg=0xc6204, len=4, val=(0x10251000, 0x0)
tee-2239  [009]   778.693796: i915_reg_rw:          read reg=0xa26c, len=4, val=(0x4, 0x0)
tee-2239  [009]   778.693805: i915_reg_rw:          read reg=0xd00, len=4, val=(0x14, 0x0)
tee-2239  [009]   778.693808: bprint:               intel_gt_init_clock_frequency: 0000:03:00.0 Using clock frequency: 19200kHz, period: 53ns, wrap: 113816ms
tee-2239  [009]   778.693817: i915_reg_rw:          read reg=0x913c, len=4, val=(0xff00, 0x0)
tee-2239  [009]   778.693825: i915_reg_rw:          read reg=0x9144, len=4, val=(0xff00, 0x0)
tee-2239  [009]   778.693908: i915_reg_rw:          read reg=0x9134, len=4, val=(0xff, 0x0)
tee-2239  [009]   778.693918: i915_reg_rw:          read reg=0x9118, len=4, val=(0xd02, 0x0)
tee-2239  [009]   778.693933: i915_reg_rw:          read reg=0x9140, len=4, val=(0x30005, 0x0)
tee-2239  [009]   778.693941: i915_reg_rw:          read reg=0x911c, len=4, val=(0x3000000, 0x0)

Side-load on embedded Linux with Buildroot

2019-09-22T10:00:00-07:00

Embedded Linux boards and devices often come with a shortage of tools installed. Often enough they don’t use a “normal” Linux distribution that can be easily extended by the end user by means of installing additional packages via dnf, apt, pacman, etc. Old/ancient versions of the Linux kernel is also very common.

In my previous post I wrote about using evemu to record data coming from the input subsytem and replay it on my computer in order to develop a Remote Controller software for ArduPilot. Without surprise, evemu isn’t included in the tools that come with Disco or SkyController 2. They also don’t have a package manager neither. I could use their alchemy build system to regenerate the entire distro, but that is much more work than I wanted: just add a few binaries there to extend rather than to replace what is currently there.

I turned back to Buildroot for that. It has a great variety of software already “packaged”, it’s pretty straight forward to use and has good integration for toolchains. I have 2 use cases for it:

Using pre-packaged software
Piggyback on Buildroot toolchain integration

The first allows me to build software already integrated in Buildroot to extend the ones available in Disco while with the second it’s possible to build my own external projects: dema-rc and ArduPilot.

Using pre-packaged software

You can head to https://buildroot.org/download.html and download a stable release, but I prefer to just clone their git repository, which allows me to switch to different branches/tags as the need arises.

I’m building additional software for Disco and SkyController 2 from Parrot. Both of them use a P7 processor that is an ARM Cortex A9. To configure Buildroot we use kconfig, which will be familiar for those who work(ed) in the Linux kernel. We just call make menuconfig and select the options we want.

$ git clone git://git.buildroot.net/buildroot
$ cd buildroot
$ make menuconfig

The options for me are:

- Target options -->
  - Target Architecture: ARM little endian
  - Target Architecture variant: cortex-A9
  - Target ABI: EABIhf
  - Floating point strategy: VFPv3-D16

On the SkyController 2 I have very little space for the additional software (8 MB) so I like to enable these options, too:

- Build options -->
  - Strip target binaries
  - gcc optimization level: optimize for size
  - libraries: shared only

Now the toolchain part. You can either bring your own toolchain (e.g. provided by the manufacturer of your board) or you can use Buildroot’s own integrated toolchains. I prefer the second since it’s much easier than trying to find where the different vendors make them available for download and making sure that toolchain can build correctly the software already packaged. Also, Buildroot often has a more recent and better maintained version.

You may face some problems, though: if you use a much more recent version, your libc may contain different symbol versions than the libc available on the board. You will either have to make the new libc available on the board together with your programs (more on that later) or you may simply choose an older toolchain version to avoid the problem. Another problem that may happen is if the libc you are using or other software make use of syscalls that are not available in the (ancient) Linux kernel the vendor made available for that board. In my case it’s either version 3.10 or 4.0, depending on the version of the firmware I’m using on Disco/SK2. My advice: stay with a version close to what is used on the board, or be In my case I select the toolchain like below:

- Toolchain
  - Toolchain type: External toolchain
  - Toolchain: Linaro ARM 2018.05

After that, just select the software you want under the Target packages section and build with make. All needed software will be downloaded and built.

Since we are not really creating a complete rootfs for a board, I don’t care about building a kernel or creating a filesystem image. All I want is to build a few packages and have the toolchain available for my own projects.

After building those packages you can inspect output/target/ for the binaries and libraries compiled. In the end you want to bring (some of) those to the target board. You can copy the ones you need to the root filesystem in the board, making them available right away, or you can dedicate a separate directory for those. I prefer the second approach (hence the name “side-load” in this post) since then I’m sure I’m not overwriting a system library/binary and changing its behavior. When I want to execute them, setting LD_LIBRARY_PATH is sufficient in most of the cases, with the exception of when a configuration file is needed and the program tries to search for it in e.g. /etc. If you don’t want to copy everything, readelf helps to find the dependent libraries:

$ arm-linux-gnueabihf-readelf -d output/target/usr/bin/evemu-record  | grep NEEDED
0x00000001 (NEEDED)                     Shared library: [libevemu.so.3]
0x00000001 (NEEDED)                     Shared library: [libevdev.so.2]
0x00000001 (NEEDED)                     Shared library: [libm.so.6]
0x00000001 (NEEDED)                     Shared library: [libc.so.6]

For my needs I can use the system’s libm and libc, and just copy libevemu, libevdev, and evemu-record to /data/rootfs/usr/{lib,bin}/. Alchemy build system used by Disco comes from Android and inherits the filesystem layout: / is read-only and /data/ read-write. As said before, setting LD_LIBRARY_PATH=/data/rootfs/usr/lib is all I need most of the time.

Piggyback on Buildroot toolchain integration

Once we have Buildroot configured we could just integrate our software in there and make it part of the normal “distro”. However I prefer keeping my development software outside of any package management or rootfs builder: even though an incremental build in Buildroot is fast, it’s slower than an incremental build of just my single program. Also, having the toolchain setup to build simple test programs while I’m hacking on the board helps a lot.

The toolchain we set up in Buildroot is a cross toolchain. In order to use it all we need is to follow the instructions for our build system to cross-compile our software. Each build system has a slightly different way to do that. We also need a small tweak in Buildroot’s toolchain, because for some reason we don’t get a symlink for pkg-config in the usual triplet scheme like for other binaries:

$ cd ~/p/buildroot/output/host/bin
$ ln -s pkg-config arm-linux-gnueabihf-pkg-config

dema-rc uses meson as build system. You can find its instructions for cross compilation in its manual. I also suggest taking a look on instructions for other projects using meson. Examples: mesa and also its specific instructions for RPI. My configuration file for meson (`~/.local/share/meson/cross/disco.txt) looks like this:

[binaries]
c =         '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-gcc'
cpp =       '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-g++'
ar =        '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-ar'
strip =     '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-strip'
pkgconfig = '/home/lucas/p/buildroot/output/host/bin/arm-linux-gnueabihf-pkg-config'

[properties]
sys_root = '/home/lucas/p/buildroot/output/target/'

[host_machine]
system = 'linux'
cpu_family = 'arm'
cpu = 'armv7a'
endian = 'little'

For ArduPilot, since each build targets a certain board, we already set the extra configuration. The only thing you need is to have the compiler with the triplet scheme in your PATH. Then you compile for the disco board as usual:

$ ./waf configure --board disco
$ ./waf plane

More information about using a cross toolchain and generate a rootfs can be found in the crosstool-ng instructions.

Using evemu to simulate input devices

2019-04-30T01:15:00-07:00

I’m doing some experiments to make the remote controller of Parrot’s Disco drone (SkyController 2) to work with ArduPilot. I will talk more about this in another post. This one is basically about a nifty tool I’ve found that I didn’t know existed: evemu. I always used evtest to debug the input system in Linux.

SkyController 2 runs Linux and exposes all the sticks and buttons as an evdev device. I developed a tool to serve as a general-purpose software to be used in this kind of device: dema-rc. Ultimate goal is to have a software that can replace what comes on these controllers instead of parsing their packets like I did for 3DR SoloLink.

Testing it remotely on the target every time is time-consuming and requires me to leave both the drone and controller powered up. When I started it I thought about recording the events with evtest and then create a tool to expose that via uinput on my laptop. It turned out that tool already existed and it does in that exact way.

On SkyController 2 I run once:

$ evemu-record /dev/input/event0 skycontroller2.evemu

After copying that file to my laptop I can re-create a similar device locally by using:

$ sudo evemu-play skycontroller2.evemu
www.parrot.com MPP GAMEPAD: /dev/input/event23
Hit enter to start replaying

Then whenever I want the sequence of events to be re-played to test in my software I just hit enter on that window. You may want to use xinput to tell X to ignore that device, but in my case I didn’t need as it was already ignoring:

(II) config/udev: Adding input device www.parrot.com MPP GAMEPAD (/dev/input/event23)
(II) No input driver specified, ignoring this device.

This approach proved very useful as it worked on the remote device on the first try after debugging it with the replays.

Per-project configuration for vim

2019-04-03T21:57:00-07:00

For a long time I have accumulated different configurations for vim for the different open source projects I’m contributing or have contributed in the past. It became a giant unmaintained file for ftplugin/ftdetect with regexes to match the dir name and try to apply every important thing a project needs: from coding style to commands to build the source code.

After changing my machine and re-installing several softwares, I’m trying to keep my dotfiles more organized. For vim, after looking at some options available I settled in this one, that I think is very short and elegant:

" Git specific configuration
let git_path = system("git rev-parse --git-dir 2>/dev/null")
let git_vimrc = substitute(git_path, '\n', '', '') . "/vimrc"
if !empty(glob(git_vimrc))
    exec ":source " . git_vimrc
endif

Unfortunately I don’t remember where this came from. But it allows you to have an additional vimrc to be loaded depending on the project you are. Since nowadays 99% of the projects I have are using git, it tries to load the vimrc from the git-dir of that project, which is often .git/vimrc.

I also have a ~/.vim/repo_vimrcs/ in which I keep the configuration files for projects I may have more than one tree or checkout. Inside each of those trees I only need to have a symlink to point to the “one source of truth” for configuring that project:

.git/vimrc -> ~/.vim/repo_vimrcs/linux.vimrc

In this file I can keep the coding style and also configure makeprg to do the right thing when trying to build that project. For example, this is what I have for igt, that uses meson as build system:

let &makeprg='ninja -C build'

git helper for “recursive rebase”

2019-03-26T10:57:00-07:00

Very often I’m in the middle of an interactive rebase and while editing a commit I remember I should have changed a commit that I initially didn’t mark on the rebase todo. If you tried to git-rebase again you would notice it’s not possible due to git’s bookkeeping of the current rebase.

In the past what I usually did was to either 1) Continue the rebase and then rebase again to fix the previous commit or 2) Create a fixup commit with git commit --fixup and then rebase again with --auto-squash.

Now I have a helper that I’m calling “recursive rebase”. Under the hood it’s not actually a recursive rebase, but just a way to edit the git-rebase’s todo file so you can rewind it. See it here: git-recursive-rebase. So let’s assume our git log looks like below:

aa7a42b (HEAD -> tip) cast-terminal: wrap asciinema and asciinema-player
0c75f7a Add recursive-rebase for poor souls depending heavily on git-rebase-i
6e70b2c mkosi-resize: rework and fix partition table recreation
60658c8 custom-kernel-install: also support installkernel
615eb02 custom-kernel-gen: also copy System.map
e683b4c git-rebase-compile: log commits being tested
f60f71c Allow removal of symlinks to fail
67c76ee (master) Fix missing char in root partition uuid

I start the interactive rebase on 60658c8 with git rebase -i 60658c8^ and mark to edit that commit.

e 60658c8 custom-kernel-install: also support installkernel
pick 6e70b2c mkosi-resize: rework and fix partition table recreation
pick 0c75f7a Add recursive-rebase for poor souls depending heavily on git-rebase-i
pick aa7a42b cast-terminal: wrap asciinema and asciinema-player

While editing I realize I also want to fixup 2 commits before that. I can do that by using git recursive-rebase HEAD~2 (I actually usually give the commit id by looking at git-log/tig. The new todo file looks like below:

$ git recursive-rebase HEAD~2
...
HEAD is now at e683b4c git-rebase-compile: log commits being tested

$ git rebase --edit-todo
pick 615eb02 custom-kernel-gen: also copy System.map
pick 60658c8 custom-kernel-install: also support installkernel
exec echo "recursive-rebase previously stopped here"
break
pick 6e70b2c mkosi-resize: rework and fix partition table recreation
pick 0c75f7a Add recursive-rebase for poor souls depending heavily on git-rebase-i
pick aa7a42b cast-terminal: wrap asciinema and asciinema-player

Just to help people keep the context on where they are in the rebase we add a line saying that it was previously there and stop the rebase. The command break was recently added to git (v2.20), but changing that to x false should make it work on previous versions as well. Bellow a asciinema recording of my terminal session:

“Throw away” linux images in seconds

2015-08-10T12:44:00-07:00

Generating a new rootfs from scratch in order to test changes to early parts of the software stack or just to have a pristine environment is something I needed several times in the past.

Since I use Archlinux in my desktop something that I like is to have a similar environment in the target test rootfs. I decided to re-use and improve a script from Kay Sievers to create an installer that can be booted as a VM, as a container or in bare metal: arch-installer.sh. Originally it was a script to bootstrap a Fedora image and I think that with some small changes that would still be possible.

$ time sudo arch-installer.sh -l ~/vm/test.img
...
real 0m31.238s
user 0m22.277s
sys 0m2.473s

30 seconds later I have a complete pristine image that can be used as a VM with qemu, as a container with systemd-nspawn or just copied to a pendrive/sdcard to boot for example a Minnow Board Max.

Container:

$ sudo systemd-nspawn -b -i ~/vm/test.img

VM:

sudo kvm-that ~/vm/test.img

Note: ‘kvm-that’ is also a script available in the same repository so I don’t have to type all the options to qemu.

In order to boot another computer or a board like Minnow Board Max just dd the image to a usb disk or sdcard. You can also generate the image directly to the final destination:

$ sudo arch-installer.sh -l /dev/mmcblk0

The script has also some nice options to make it easy to customize the final image. One thing that I’m often doing is giving an overlay directory with configuration files for wpa_supplicant. This way I can already access my WiFi networks in the target image.

If you always need certain packages you can use the example debug-tools hook that is executed before the image is finalized. By mixing hooks like that and the overlay directory mentioned above it’s possible to add your local repository to pacman.conf and install packages not available in Archlinux. Or packages that you’d like to maintain on your own. In my use cases with Minnow Board Max I maintain my own kernel with configurations suited to run ardupilot on it.

Taking maintainership of dolt

2015-03-09T19:27:00-07:00

For those who don’t know, dolt is a wrapper and replacement for libtool on sane systems that don’t need it at all. It was created some years ago by Josh Triplett to overcome the slowness of libtool.

Nowadays libtool should be much faster so the necessity for dolt should not be as big anymore. However as can be seen here, it’s not true (at least in libtool > 2.4.2). Yeah, this seems a bug in libtool that should be fixed. However I don’t like it being any more complicated than it should.

After talking to Josh and to Luca Barbato (who maintains an updated version in github) it seemed a good idea to revive dolt on its original repository. Since this file is supposed to be integrated in each project’s m4 directory, there are several copies out there with different improvements. Now the upstream repository has an updated version that should work for any project — feel free to copy it to your repository, synchronizing with upstream. I don’t really expect much maintenance. Credit where it’s due: most of the missing commits came from the version maintained by Luca.

So, if you want to integrate it in your repository using autotools, it’s pretty simple, feel free to follow the README.md file.

Hacking the Intel fan, for fun

2013-10-01T17:43:00-07:00

An alternative headline is: “how to show your wife how much you love her, the geek way”.

From 17 to 22 of September I was in New Orleans participating in the discussions of the Linux Plumbers Conference, which has already turned into one of my favorite conferences. Lots of fun, talking to great people and good discussions about systemd, containers, cgroups, kernel modules, etc. However as the headline indicates this blog post is not to talk about the conference but rather about a toy the Intel booth was giving out: a fan with 7 leds in its propeller. See below:

Fan distributed to attended during LPC

When turned on it shows a text message: “We’re hiring!”, “01.org/jobs”. So, if you are looking for a job and want to come to work with me, you already know where to apply ;-). The fun part is that in its box it’s written “programmable message fan”. The guys from the booth told me that the first question people were asking was how to change the message appearing there, but they had no idea. This post is to show how I did that.

Some days after arriving in Brazil I saw a post from Steven Rostedt on G+ regarding this fan and a blog post he found: http://hackingwithgum.com/2009/10/06/hacking-the-cenzic-pov-fan/. Disassembling our fan showed that it’s a little bit different than that one, changing the EEPROM and with 1 extra pin.

Disassembling the fan

However looking carefully at the board we can see it’s pretty similar: It’s a T24C04A EEPROM that is programmable via I2C. I’m not sure if the extra pin is the the write-protect feature that is present in this EEPROM or if it’s to select the address (in which case we would just have another address on our side). Hence we are safe connecting it to ground. From T24C04A’s datasheet we figure it can work in the range 1.8V to 5.5V. So, instead of using a serial connector like in the other blog post, we can use any development board that has an I2C bus available for us to play with it, particularly BeagleBone Black that has a 3.3V I2C bus, which I’m using here. From the picture below you can notice that a) I didn’t have many HW components available and b) my drawing skills are horrible. I just did a quick hack to get it to work, i.e.: connect GND, VCC and the pull-up resistors (since in I2C the bus is high when nobody is transmitting) [ See UPDATE 2 ].

Wiring the fan to beaglebone

For reading from and writing to the EEPROM I’m using i2cdump and i2cset respectively. And i2cdetect to show me where the device was plugged and its address. Beware that in beaglebone the devices in /dev don’t match the ones in the HW schematics (thanks Koen for pointing out this to me).

Now the software part. Like in the other fan we have a column of 7 leds and each letter is “rendered” using 5 rows. However the way the strings are written is different. I tried to use the python script that was provided, but after some tests I figured I’d need to do some modifications. Below is how the strings are stored in our fan’s EEPROM:

First byte is the number of strings present in the EEPROM. Each string then has as its first byte the string length. Then we have 5 bytes for each char, in reverse order. They encode the state of the LEDs in each column: 0 means ON and 1 is OFF. After some try and errors we realize that not only the string is reversed, but also the columns. So the first byte in the character encodes the right-most column. In the end we have a 7x5 matrix for each char. I started to draw all chars and change the python script to use them, ~~but I got lazy and just finished the letters I was interested in~~ (see UPDATE 1). The final result is the video shown above that says “Talita, I love you”, in Portuguese :-).

I used the following commands to dump the EEPROM, encode the text and write to it.

root@beaglebone:~# # dump what's in address 0x50 on bus 1 (use i2cdetect to find out the bus and address of your device)
root@beaglebone:~# i2cdump 1 0x50
root@beaglebone:~# # encode the message given as args
root@beaglebone:~# /tmp/ascii2fan "string1" "string 2 with space" "string 3" > ~/message.bin
root@beaglebone:~# # write the content of ~/message.bin into the EEPROM
root@beaglebone:~# i=0; od -An -t x1 ~/message.bin | while read line; do   
                  for c in $line; do   
                      cmd=$(printf "i2cset -y 1 0x50 0x%x 0x$c b" $i); $cmd; ((i++));
                   done;
               done

You can download my modified ascii2fan. I was using it on /tmp and lost it after power cycling, so I needed to change the file again and I didn’t confirm it’s still working. It’s almost the same as the one provided in hackingwithgum.com, it’s just the table that really changes.

UPDATES:

I uploaded a new version of ascii2fan, containing all the letters.
As Matt Ranostay pointed out, the beaglebone black has already an internal pullup resistor, so this is not really necessary.

Optimizing hash table with kmod as testbed

2013-09-25T07:32:00-07:00

One thing that caught my interest lately was the implementation of hash tables, particularly the algorithms we are currently using for calculating the hash value. In kmod we use Paul Hsieh’s hash function, self entitled superfast hash. I fell troubled with anything that entitles itself as super fast, especially given the benchmarks provided are from some years ago, with older CPUs.

After some time spent on benchmarking and researching I realized there were much more things to look after than just the hash function. With this post I try to summarize my findings, showing some numbers. However do take them with a grain of salt.

The hash functions I’m using here for comparisons are: DJB, Murmur32, Paul Hsiesh and CRC32c (using the crc32c instruction present in SSE4.2). My goal is to benchmark hash functions when strings are used as key (though some of the algorithms above can be adpated for blobs). Also the use of these hash tables are only for fast lookups, so any cryptography-safe property of the functions or lack thereof is irrelevant here. For all the benchmarks I’m using depmod’s hash tables. There are 3 hash tables:

module names: keys are very small, 2 to 20 chars
symbol names: keys are mid-range, 10 to 40 chars
module path names: keys are the largest: 10 to \~60 chars

On my benchmarks I identified the following items as important points to look after optmizations:

Hash functions: how do they spread the items across the table?
Time to find the bucket
Time to find the item inside the bucket
Hash functions: time to calculate the hash value

Whilst the first and the last ones depend on the algorithm chosen for calculating the hash value, the second and third are more related to the data structure used to accommodating the items and the size of the table.

Hash functions: how do they spread the items across the table?

It’s useless to find a hash function that is as fast as it can be if it doesn’t cope with it’s role: spreading the items across the tables. Ideally each bucket would have the same number of items. This ideal is what we call perfect hash, something is possible only if we known a priori all the items and can be accomplished by tools like gperf. If we are instead looking for a generic function that does its best on spreading the items, we need a function like the ones mentioned above: given any random string it calculates a 32 bits hash value used to find the bucket in which the value we are interested in is located. It’s not only desirable that we are able to calculate the hash value very fast, but also that this value can be in fact useful. A naive person could say the function below is a super fast constant time hash function.

uint32_t naive_hashval(const char *key, int keylen)
{
        return 1;
}

However as one can note it isn’t used anywhere because it fails the very first goal of a hash function: to spread the items on the table. Before starting the benchmarks I’ve read in several places that the crc32c instruction is like the function above (though not that bad) and couldn’t be used as a hash function for real. Using kmod as a test bed this isn’t what I would say. See the figures below.

We used all 3 hash tables used in depmod, dumping all the items just before destroying them. The graph shows how many items were in each bucket. For all algorithms we have almost the same average and standard deviation. So, which functions should we use for the next benchmarks? The answer is clearly all of them since they provide almost the same results, being good contenders.

Time to find the bucket

Once we calculated the hash value, it’s used to find in which bucket the item lies on. For starters, a bucket is nothing more than a row in the table as depicted below:

b0 it1 → it2 → it3 b1 it4 b2 it5 → it6 → it7 → it8 b3 it9 → it10 → it11

The hash table above has size 4, which is also the number of buckets. So we need to find a way to convert the (32-bit) hash value to a 3 bits. The most common way used in hash table implementations is to just take the value’s modulo:

uint32_t hashval = hashfunc(h, key, keylen);
int pos = hashval % h->size;

The size above is set when the hash table is created (some hash table implementations use a grow-able approach, which is not treated in this post). The modulo above is usually implemented by taking the rest of a division which uses a DIV instruction. Even if nowadays this instruction is fast, we can optimize it away if we pay attention to the fact that usually the hash table size is a power of 2. Since kmod’s inception we use size=512 for module names and paths; and size = 2048 for symbols. If size is always a power of 2, we can use the code above to derive the position, which will lead to the same result but much faster.

uint32_t hashval = hashfunc(h, key, keylen);
int pos = hashval & (h->size - 1);

The DEC and AND instructions above are an order of magninute faster than the DIV on today processors. However the compiler is not able to optimize the DIV away and use DEC + AND since it can’t ensure size is power of 2. Using depmod as a test bed we have the following clock cycle measurements for calculating the hash value + finding the bucket in the table:

keylen      before   after
2-10          79.0    61.9 (-21.65%)
11-17         81.0    64.4 (-20.48%)
18-25         90.0    73.2 (-18.69%)
26-32        104.7    87.0 (-16.82%)
33-40        108.4    89.6 (-17.37%)
41-48        111.2    91.9 (-17.38%)
49-55        120.1   102.1 (-15.04%)
56-63        134.4   115.7 (-13.91%)

As expected, the gain is constant regardless of the key length. The time to calculate the hash value varies with the key length, which explains the bigger gains for short keys. In kmod, to ensure the size is power of 2, we round it up in hash_init() to the next multiple with the following function:

static _always_inline_ unsigned int ALIGN_POWER2(unsigned int u)
{
    return 1 << ((sizeof(u) * 8) - __builtin_clz(u - 1));
}

There are other ways to calculate it, refer to kmod’s commit as to why this one is used.

Time to find the item inside the bucket

As noticed in the previous section we use a MOD operation (or a variation thereof) to find the bucket in the table. When there are collisions and
a bucket is storing more than 1 item, hash table implementations usually resort to a linked list or an array to store the items. Then the lookup ends up being:

Calculate the hash value
Use the hash value to find the bucket
Iterate through the items in the bucket comparing the key in order to find the item we are interested in.
Return the value stored.

So often the item is a struct like below:

struct hash_entry {
        const char *key;
        void *value;
};

In the struct above I’m considering the key is a string, but it’s also possible to have other types as key.

So once we have the bucket, we need to go through each item and strcmp() the key. In kmod since we use an array to store the items we have a little better approach: the array is kept sorted and during lookup it’s possible to use bsearch(). However as one can imagine, keeping the array sorted doesn’t come for free. We are speeding up lookup with the downside of slowing down insertion.

Thinking on this problem, the following came to mind: we use and benchmark complicated functions that do their best to give a good 32 bits value and then with the modulo operation we use just throw away most of it. What if we could continue using the value? If we don’t mind the extra memory used to store one more value in the struct hash_entry above, we can. We store the hash value of each entry and then compare them when searching inside the bucket. Since comparing uint32 is very fast, there’s no much point in keeping them sorted anymore and we can just iterate very fast through all items in the bucket checking first if the hash values match and only then strcmp() the key. With this we can drastically reduce the amount of string comparisons we do in a lookup-intensive path, the time to add and item (since it doesn’t need to be sorted anymore) and also the complexity of the code. The downside is the memory usage, with one extra 32 bit value for each entry. The table below shows the results.

keylen      before   after
2-10         222.8   127.7 (-42.68%)
11-17        231.2   139.1 (-39.85%)
18-25        273.8   181.3 (-33.78%)
26-32        328.7   236.2 (-28.13%)
33-40        366.0   306.1 (-16.34%)
41-48        354.0   341.7 (-3.48%)
49-55        385.1   390.5 (1.40%)
56-63        405.8   404.9 (-0.21%)

Hash functions: time to calculate the hash value

This was my original intention: to benchmark several hash functions and choose the best one based on data from real world, not random or fabricated strings. My desire was also to profit as much as I could from nowadays processors. So if there’s a new instruction, let’s use it for good.

Based on that I came to know the crc32c instruction in SSE4.2. By using it we have a blazing fast way to calculate crc32, capable of a throughput of around 1 char per clock cycle. I was also optimistic about it when I checked that its distribution was as good as the other contenders. The figure below shows the time to calculate the hash value for
each of the algorithms.

As can be seen on the benchmark above, the winner is indeed crc32c. It’s much faster than the others. It’s noteworthy the fact that it increases much less than the other as the length goes up.

From the other contenders, DJB is the worst one reaching much higher times. It’s the simplest one to implement, but in my opinion the others are small enough (though not that simple) to take its place.

How much does the hash function affect the hash table lookup time? By lookup time I mean:

The figure below answers this question:

We can see from the figure above that the crc32 implementation is the faster one. However, the gain is not as big as when we consider only the time to calculate the hash value. This is because the other operations take much more.

Conclusion

Although the CRC32 implementation is faster than its contenders and that I was tempted to change to it in kmod, I’m not really doing it. There would be a significant gain only if the key was big enough. As noted above, the time to calculate the hash value in CRC32 grows much more steadily than the others. If the keys were big enough, like 500 chars, then it could be a change to make. It’s important to note that if we were to change to this hash function we would need to add an implementation for architectures other than x86 as well as introduce the boilerplate code to detect in runtime if there’s support for the crc32c instruction and use the generic implementation otherwise.

The other 2 optimizations, although not ground breaking are easy to make and not intrusive. Nonetheless if people want to check the hash functions in their workload I will make available the code and benchmarks in a repository other than kmod’s.

Speeding up build on autofoo projects

2012-11-19T14:23:00-08:00

First of all, a little digression about build systems.

I’d like to clarify that I’m no lover of autotools. But I’m no lover of any other build system, neither, and autotools is used on several open source projects, including most of the ones I participate. It’s easily copied and pasted by new projects. It’s known by distro maintainers how to hack on it to work on different scenarios like cross-compilation, distribute build across machines, different compilers, rpath, etc etc. Moreover, from my experience, project maintainers usually dislike changing the build system because it causes several headaches not related to the raison d’être of the project. So in general I prefer to modernize the build system rather than radically change it to another one.

Enlightenment window manager is about to be released and I was really bored by the amount it was taking to compile. I use icecream to spread the build across the machines on the local network, so I usually do things like “make -j40”. No matter how many jobs I could put there to parallelize the build it was still painfully slow, particularly while compiling the modules. Taking a look in the Makefile.am files, my suspicion was true: it was using recursive makefiles and since each module has a few source files (circa 3 \~ 6), the build was parallelized only among the files in the same directory. This is because the build is serialized to build each directory. There are plenty of links on the web about why projects should use non-recursive automake. I will enlist some:

So I decided to convert E17’s automake files to non-recursive ones. At least the modules part. After hours of repetitive tasks for converting it, fixing bugs, building out-of-tree, in-tree, distcheck, etc, I committed it and the build time was improved like below:

[table ID=1 /]

So, after configuring it we can build E17 in roughly 1/4 of the previous time.

After the commit introducing the change there were several others to improve it even more, prettify the output, fix some other bugs. It also got reverted once due to causing problems to other developers, but in the end it was applied back. The worst bug I found was related to subdir-objects option to Automake and Gettext’s Automake macros. That option means that the objects built are kept in the same directory as the correspondent source file. This is needed, particularly in a non-recursive automake scenario, so the objects from different modules don’t conflict due to being put in the same directory. However, letting this option in configure.ac made “make distcheck” fail in some obscure ways and I later tracked it down to be gettext’s fault. A simple “fix” was to remove it from configure.ac and set it in the “AUTOMAKE_OPTIONS” variable of the modules’ Makefile.am. I really hope someone has the time and will to fix gettext macros - they are a horrible mess and I don’t want to play with them.

Analyzing chess.com tournaments

2012-10-16T19:54:00-07:00

This year at ProFUSION we started to create chess tournaments to play using chess.com so we have fun not only coding but playing this game. However it’s even more fun when we put both together: chess and code :-)!

During the second tournament we realized chess.com was missing a nice feature: to allow participants to predict who would be the champion based on the current championship status and future results. To show the current state, Chess.com presents us with a table like this:

Not the missing games with a “_”. What we would like is to predict who can still win the tournament based on these missing games. One trick here is how to calculate the tie break, but it’s really straightforward to implement once we understand the formula:

So, for each sum up the result of each game multiplied by the current score of the opponent (“opp” in the formula above) the game was played against.

With that in mind I wrote a small tool, ccs, to allow you to predict the next results:

$ ./ccs.py data/example2.html 
css> state
Number of players: 8

                  1.      2.      3.      4.      5.      6.      7.      8.      |  Score | Tie Break
-------------------------------------------------------------------------------------------------------
1. demarchi        X      1 1     1 0     1 0     0       1 1     1 1     1 1     |     10 |        54
2. ulissesf       0 0      X      1 1     0 1     1       1 1     1 1     1 1     |     10 |        52
3. lfelipe        1 0     0 0      X      1 1     1 1     1 0     1 1     1 1     |     10 |        52
4. hdante         1 0     0 1     0 0      X      1 0     0 1     1 1     1 1     |      8 |        38
5. Gastal         1       0       0 0     1 0      X      1 1     1 1     1 1     |      8 |        34
6. marinatwp      0 0     0 0     1 0     0 1     0 0      X      1 1     1 1     |      6 |        22
7. yanwm          0 0     0 0     0 0     0 0     0 0     0 0      X      1 1     |      2 |         0
8. antognolli     0 0     0 0     0 0     0 0     0 0     0 0     0 0      X      |      0 |         0
ccs> push 1x5=1
Simulation added: demarchi beats Gastal
Number of players: 8

                  1.      2.      3.      4.      5.      6.      7.      8.      |  Score | Tie Break
-------------------------------------------------------------------------------------------------------
1. demarchi        X      1 0     1 1     1 0     0 1     1 1     1 1     1 1     |     11 |        62
2. lfelipe        1 0      X      0 0     1 1     1 1     1 0     1 1     1 1     |     10 |        53
3. ulissesf       0 0     1 1      X      0 1     1       1 1     1 1     1 1     |     10 |        52
4. hdante         1 0     0 0     0 1      X      1 0     0 1     1 1     1 1     |      8 |        39
5. Gastal         1 0     0 0     0       1 0      X      1 1     1 1     1 1     |      8 |        35
6. marinatwp      0 0     1 0     0 0     0 1     0 0      X      1 1     1 1     |      6 |        22
7. yanwm          0 0     0 0     0 0     0 0     0 0     0 0      X      1 1     |      2 |         0
8. antognolli     0 0     0 0     0 0     0 0     0 0     0 0     0 0      X      |      0 |         0
ccs>

Ccs parses the pairings table created by chess.com (given you saved it somewhere and passed as argument to the tool) and then gives you a “ccs> ” prompt, waiting for commands. Type ‘help’ to see the list of available commands. Basically it allows you to a. see the current state of the game (‘state’ command) and b. push and pop result simulations (sigh! ‘push’ and ‘pop’ commands).

A nice feature that I’d like to introduce soon is to export to a big svg with all the state transitions, marking leaf nodes when there’s a champion. I’m also releasing the source as open source, so anybody wanting to help can implement it :-). Code is available on my github: https://github.com/lucasdemarchi/ccs. GPL-2 as usual.

What can we predict in the example above?

No matter the pending games, gastal can not win anymore, since he will reach at most 54 as tie break, leaving lfelipe with 56. That also implies lfelipe would be the champion if gastal wins all his pending games;
If demarchi wins his last game he wins the tournament, with score=11 and tie-break=62. If ulissesf also wins, he will have the same score, but his tie-break will be 60, pushing demarchi’s tie-break to 64.
If ulissesf wins and demarchi loses, ulissesf is the champion.

Since I am demarchi on the table above, now what I have to do is either win the last game or convince Gastal to give up his pending games :-).

Back from Linux Plumbers

2012-09-03T18:29:00-07:00

I’m back from USA after one week attending Linux Plumbers Conference. This was my first time in LPC, in which I was part of the CoreOS, talking about “From libabc to libkmod: writing core libraries”.

It was a very good experience and I’m glad to meet so many developers, both kernel and userspace hackers. Some of them I only knew from IRC, mailing-lists, etc and it was great time to share our experiences, discuss the current problems in Linux and even fix bugs :-). We seem finally to have reached a consensus on how module signing should be done - the outcome of Rusty Russel’s talk is that he will now start applying some pending patches. There will be no required changes to kmod, except a cosmetic one in modinfo to show if a module is signed or not.

Rusty was also very helpful in fixing a long-standing bug in Linux kernel: a call to init_module() returns that a module is already loaded, even if it didn’t finish it’s initialization yet. This used to be “fixed” in module-init-tools by a nasty hack adding a “sleep(10000)” if the module state (checked on sysfs) is “coming”. I mean “fixed” because this approach is still racy, even though the race window is much shorter than without it. So we finally sat down and wrote a draft patch to fix it. This will probably reach Linus tree in the next merge window.

The above example only seconds what Paul McKenney said on his blog yesterday: “A number of the people I informally polled called out the hallway track as the most valuable part of the conference, which I believe to be a very good thing indeed!” - I was one of the people he informally polled ;-). I’d like to thank all the Committee and people involved in organizing this conference - it was a very great experience.

Finally, you can find my slides below (or download from Google Docs). I think soon the audio will be published. Meanwhile you may enjoy Lennart’s picture when he was a child in slide #5 (during the talk he claimed it’s not him, but I don’t believe - they are too similar :-)).

ConnMan in Archlinux

2012-08-28T23:37:00-07:00

For sometime (I think it’s almost 2 years) I was maintaining the ConnMan package in AUR, the user repository in Archlinux.

After talking to Dave Falconindy and Daniel Wallace, the later accepted to maintain it in community repository. As a result I’m dropping the package in AUR. All of you that were using my package should upgrade to the latest version from the official Archlinux repository.

A great news is coming for Enlightenment users, too: a new ConnMan module, written from scratch, that works properly with recent versions. This is reaching e-svn very soon. Stay tunned. Thanks to Michael Blumenkrantz, too, who declared me a “hero” for doing this :-).

BlueZ to move to standard D-Bus interfaces

2012-07-19T18:28:00-07:00

During the last week I was in Recife, Brazil, together with Henrique Dante, Ulisses Furquim and other fellows attending the BlueZ meeting. We discussed several topics, among them the upcoming BlueZ 5 API. That discussion started by Johan and Marcel saying BlueZ would not move to DBus.Properties and DBus.ObjectManager anymore for the next version. Main reason being that BlueZ now will have more frequent releases than it had in past and therefore there wasn’t enough time to convert the API. However I and Luiz von Dentz already had an almost working solution: I implemented DBus.Properties and he did the DBus.ObjectManager, so we received the news with much regret.

Since these changes break the API, not accepting them now means we’d need to wait for BlueZ 6 in order to have these interfaces, which expected to happen only on later next year. Thus we argued that and Marcel Holtmann challenged us to make a demo on Wednesday. Challenge accepted! After working one night to put the implementations together and finish the DBus.PropertiesChanged signal we could present such a demo. And it worked without any issues. To say the truth we only converted 2 interfaces (Adapter and Manager), but that was enough to convince them we can finish this in time for BlueZ 5.0. Or at least that’s what we hope. Final consensus: this change is back on track for the upcoming major release of BlueZ.

Now we are polishing the implementation and start sending the patches. Thefirst patch set was already sent and hopefully soon all the others will reach BlueZ’s mailing list.

So, why is this important? By implementing these interfaces, that are part of the D-Bus specification, it becomes much easier to write client code to talk to bluetoothd and since D-Bus bindings already have implementations for them it’s much less error prone, too. At the same time we are aiming to simplify the code needed in bluetoothd to support our use-cases, so for both people who write bluetoothd and for those who write client applications this will be beneficial.

ELC 2012

2012-03-20T19:30:00-07:00

Hey, this is my feedback of ELC 2012. If you didn’t read the first part, about ABS 2012, you can read the previous post first.

ELC is one of my favorite conferences as I can meet several talented people and have good talks about Linux in embedded devices. This time was not an exception and I enjoyed very much. The main reason I was there was because I was going to present kmod, the new tool to manage kernel modules. But that would be only on the last day of the conference. Let’s start from the beginning.

To open the conference Jon Corbet gave his usual kernel report starting from January 2011 and going on through the events in each month: the mess in ARM, death of the big kernel lock, userspace code in kernel tree (should we put libreoffice there, too?) and so on. Following this keynote I went to see Saving the Power Consumption of the Unused Memory. Loïc Pallard from ST-Ericsson talked about how memory consumption in increasingly important in embedded devices for the total power consumption. We are going from the usual 512 MB on smartphones to 2 \~ 4 GB of DDR RAM. There are some techniques to reduce this the power drained and he presented the PASR framework, that allows the kernel to turn on/off specific banks/dies of memory since not all of them is used all the time. Later on talking to the guys from Chromium OS I realized that this is especially true when the device is sleeping. We may want to discard caches (therefore use much less memory when in sleep mode) and then turn off banks not used. In my opinion the battery consumption is one of the most important today for embedded Linux devices: I’m tired to have to charge my smartphone every day or every X hours. I hope we can improve the current state by using techniques as the one presented in this talk.

In Embedded Linux Pitfalls Sean Hudson from Mentor Graphics shared his experience while coming from closed embedded solutions to open source ones. Nice talk! I think people doing closed development should see presentations like this: one of the main reasons for failing in opensource is not being able to talk to each other: HW guys not talking to SW guys, NIH, not playing the rules of the communities and therefore having to carry a lot of patches, etc. I’ve always been involved with opensource so I don’t know very well how things work for companies doing closed development, but I do know that more often than not we see those companies trying to participate in communities/opensource and failing miserably. In my opinion one of the main reason is because they fail to talk, discuss and agree on the right solution with communities.

One of the best talks in ELC 2012 was Making RCU Safe for Battery-Powered Devices. Paul McKenney is one of the well known hackers of the Linux kernel, maintaining the RCU subsystem. Prior to this talk I had no idea RCU had anything to do with power consumption. He went through a series of slides showing how and why RCU got rewritten several times in the past years, how he solved the problems reported by community and how things get much more complicated with preemption and RT. He finished his presentation saying that the last decade was the most important of his carrier and that is because of the feedback he got from RCU being used in real life. I’d really love to see more people from Academia realizing this.

The next day Mike Anderson gave a great keynote about The Internet of things. Devices in Internet are surpassing the number of people connected and soon they will be much more important. It’s a great opportunity for embedded companies and for Linux to become the most important Operating System in the world. Recent news about this are already telling us that 51% of the Internet traffic is non-human (although we can’t classify all of that as “good traffic”). Following his keynote I went to see Thomas Petazzoni from Free Electrons talk about Buildroot. I like Buildroot’s simplicity and from what Thomas said this is one thing they care about: Buildroot is a rootfs generator and not a meta-distro like openembedded. There were at least 3 people asking if Buildroot could support binary packages and he emphasized that it was a design decision not to support them. I like this: use the right tool for the each job. I already used Buildroot before to create a rootfs using uClibc and it was great to see that it was already packaging the last version of kmod before I went to ELC.

In the end of the second day I participated in Real-Time (BoFs) with Frank Rowand. It was great to have Steven Rostedt and Paul McKenney there as they contributed a lot to the discussion, pointing out the difficulties in RT, the current status of RT_PREEMPT patches regarding mainline and forecasts of when it will be completely merged. There were some discussions about “can we really trust in RT Linux? How does it compare with having an external processor doing the RT tasks?”. In the end people seemed to agree that it all boils down about what do you have in your kernel (you probably don’t want to enable crappy drivers), how do you tolerate fails (hard-RT vs soft-RT) and that RT is not a magic flag that you turn on and it’s done: it demands profiling, kernel and application tuning and expertise in the field. People gave several examples of devices using the RT_PREEMPT patches: from robots and aircrafts in the space to cameras (the Sony cameras given away on the last day were 1 of the examples).

On Friday, the last day of the conference, I was much more worried about my presentation in the end of the day than with other talks. Nonetheless I couldn’t miss Koen Kooi from Texas Instruments talking about Beaglebone. It’s a very interesting device for those who like to DIY: it’s much smaller than its brothers like Beagleboard and Pandaboard and still has enough processing power for lots of applications. Koen was displaying his slides using node.js running on a Beaglebone. What I do like to see though is barebox replacing u-boot as the bootloader. If you attended Koen’s talk on ELCE last year, you know u-boot is one of the culprits for a longer boot. Jason from TI kindly gave me a Beaglebone so I can use it for testing kmod; when I have some spare time I’ll take a look on the things missing for using barebox on it, too.

The last talk of the conference was mine: Managing Kernel Modules With kmod. I received a good feedback from people there: they liked the idea behind kmod - designing a library and then the tools on top of that. I had some issues with my laptop in the middle of my presentation, but it all went well. I could show how kmod works, the details behind the scenes, the short history of the projects and how it’s replacing a well known piece of userspace tools of Linux in all major desktop and embedded distros. When I was showing the timeline of the project I remember Mike Anderson saying: “tell us when it will be done”. I can’t really say it’s done now, but after the conference we already had versions 6 and 7 and contrary to other releases in the latest versions the number of commits is very small. After 3\~4 months the project is reaching a maintenance phase as I said it would. If you would like to see my slides, download it here or see it online below. You can also watch the video of my talk as well as all the others in LF’s video website.

Android Builders Summit 2012

2012-03-17T14:25:00-07:00

Four weeks ago, from Februart 13rd to Februrary 17th I was at Android Builders Summit and Embedded Linux Conference in San Francisco. I was a bit busy these last weeks so I didn’t have an opportunity to write here about the conferences as I usually do. I was going to do a post about both the conferences, but after writing a little bit I realized it would be very big. So I split in 2 and here is the one for ABS 2012; the other is coming soon.

This was my first time at Android Builders Summit. Since in the end of last year I participated in a project modifying Android internals, I felt it would be really good to be in touch with people doing the same things and learn with them. Before going there I was already surprised that Google was not sponsoring the conference, but there I was astonished that there was nobody from Android’s team and I don’t remember talking to Googlers, too. I don’t know what happened but it would be really good for the next conference if Google could be part of the conference since for the very nature of how they manage the project they are the people pushing the platform forward.

In the first day of the conference Greg Kroah-Hartman, Tim Bird and Zach Pfeffer answered the question “Android and the Linux Kernel Mainline: Where Are We“: it’s done. Well, not totally done, but most of the code needed in kernel is already in mainline: except for some pieces that render your device useful it’s already possible to boot Android userspace with a mainline kernel. I think the main point of this effort is to allow companies and enthusiasts to use features from the mainline kernel and newer versions than the ones available on AOSP. As the diff between mainline and Android’s kernel decreases it’s much easier to deploy an Android device with a different kernel. More details can be found in http://lwn.net/Articles/481661/.

From the other talks I attended on the first day, the one that caught my eyes was USB Device Support Workshop. Bernie Thompson from Plugable talked a bit about the lack of proper support in Android to deal with kernel modules: it’s really hard for device maker companies like his own to have products working on Android. And it’s not because they aren’t committed to developing Linux device drivers but because of the lack of support in Android to easily deal with kernel drivers: either the external device is supported by the company shipping the Android product or there’s no way for example to plug in an external camera and get it to work. Audience was a bit confused saying that that was a Linux problem and some voices telling that in Windows lands that doesn’t happen. Not true. Linux supports more devices that any other operating system in the world, however Android is currently missing some tools to profit from it. After some discussion Bernie prepared some tables with USB devices that people could hack on, get it supported in Linux/Android, etc.

On the second day I attended Real-Time Android, particularly because of my involvement with real-time since I graduated at university and because I was curious about applying it to Android. As I said one of the benefits of having Android kernel closer to mainline is that it’s easier to do things like this. Wolfgang Mauerer from Siemens applied the RT_PREEMPT patches to Android’s kernel so you could have a real-time embedded system and still use Android’s app. As I was expecting RT would be applied for native applications, not java based ones.

Topics in Designing An Android Sensor Subsystem: Pitfalls and Considerations was advanced talk about Sensors in Android and how one would choose one strategy over another and the tradeoffs between battery life, sample rate, external co-processor, DIY or license the algorithms used, etc. It was not a talk for the regular Joe doing an app that uses the Android’s Sensors API (that was what I knew about it) but more for people creating devices that would like to use sensors.

It was a conference different from the conferences I’m used to attend like ELC/LinuxCon: there was very few people who I already knew and I had the feeling that we were talking about a product from someone else, not a product we were helping to develop - instead we were having talks about how to hack a platform we do not own. In general I liked the talks I could attend and talking to people at the corridors. They even gave me some insights for my talk about kmod, later on Friday at ELC. I’ll talk more about it on the next post.

For those wanting to see the slides/videos, Linux Foundation made them available at their site - go on and see for yourselves.

ANNOUNCE: codespell 1.4

2012-03-03T02:06:00-08:00

codespell 1.4 is out! Nothing really new, just a maintenance release: 1 bug fix and some new entries to the dictionary. See the entire announcement on its mailing list.

As per patches I’m receiving it seems that codespell is being successfully used by opensource projects. I’m glad codespell can help those projects, particularly people who don’t have English as their mother tongue as I don’t. It’s also an opportunity to people starting on a project, as I said in last LinuxCon Brazil.

I’m not submitting patches anymore to Linux kernel myself using codespell because after doing that twice I started to receive a lot of emails from people using get_maintainer script. It’s very annoying to filter the good emails (that were indeed addressed to me) from that that I was in CC just because there was a misspelling that my patch fixed. Since that patch touched 2463 files, it’s very common to have my name in the output of get_maintainer :-(. I’m still trying to figure out how to properly filter that without losing important emails. Any tips (I’m a GMail user)?

Back to codespell announcement, the only missing item in TODO is to be able to separate comments, strings and source code in order to fix misspellings only in the first 2. Nonetheless codespell seems to be doing a good job without that feature and unless someone step in to implement it without impacting the parse time too much, my plan is to keep as is.

Go get it (and package for your distro) while it’s fresh!

ANNOUNCE: kmod 3

2012-01-06T06:50:00-08:00

Hey, kmod 3 is out. Really nice to finish this release. I was hoping to have it between the holidays, but there were some major bugs pending. It’s nice to see udev from git already using it instead of calling modprobe for each module. Kay reported a hundred less forks on bootup after start using libkmod and libblkid.

It’s nice too receive feedback about other architectures that we don’t have access, too. With kmod 3, sh4 joined the other architectures that were tested with kmod.

Since I’m already doing the announcements to the mailing lists, I’ll not repeat the NEWS here. Just look at the archives if you didn’t receive the email.

Happy new year!

ANNOUNCE: kmod 2

2011-12-21T04:40:00-08:00

I’m glad to announce the second version of kmod. I’m sorry for not sending the first version to the mailing lists. Now I’m both writing it here and sending to the mailing list.

I thank very much the feedback received for the first version and that now Jon Masters, the maintainer of module-init-tools, is helping us with kmod and already announced that kmod will replace module-init-tools in future.

I’d like to especially thank Tom Gundersen, Dave Reisner, Marco d’Itri, Jon Masters, Luis Strano, Jan Engelhardt and Kay Sievers who have been extensively testing kmod and helping with compatibility with previous tools. Right now kmod is tested in i686, x86_64, sparcv9, powepc64, s390 and ARM. More testers are greatly appreciated, especially for architectures not mentioned here.

News for this version are (copying from NEWS file):

Some bugs fixed: the worst of them was with an infinite loop when an alias matched more than one module.

New APIs in libkmod to:

ELF

Support to load gzipped kernel modules: kmod can be compiled with support to gzipped modules by giving the —enable-zlib flag
Support to forcefully load modules, both vermagic and modversion
Support to force and nowait removal flags
Configuration files are parsed in the same order as modprobe: files are sorted alphabetically (independently of their dir) and files with the same name obey a precedence order
New tool: kmod-modinfo
kmod-modprobe gained several features to be a 1:1 replacement for modprobe. The only missing things are the options ‘—showconfig’ and ‘-t / -l’. These last ones have been deprecated long ago and they will be removed from modprobe. A lot of effort has been put on kmod-modprobe to ensure it maintains compabitility with modprobe.
linux-modules@vger.kernel.org became the official mailing list for kmod

For the next version, we plan to migrate our git repository to kernel.org. Meanwhile, git repository can be found at http://git.profusion.mobi/cgit.cgi/kmod.git/ and packages at http://packages.profusion.mobi/kmod/

Given enough eyeballs, all bugs are shallow

2011-12-16T08:59:00-08:00

So, in last post I said kmod 2 could be released sooner than expected if there were major bugs. Not as much as a surprise, there was 1: depending on the alias passed to the lookup function we were blocked iterating a list.

It’s now fixed in git tree. Thanks to Ulisses Furquim for fixing it and Dave Reisner for the bug report. We already have some other great stuff implemented so we’ll soon have another release.

Another great news is that now we have the maintainer of module-init-tools (Jon Masters) cooperating with us. We will discuss how the two projects will co-exist/merge. So, for now on the official mailing list of the project is linux-modules@vger.kernel.org.

ANNOUNCE: kmod 1

2011-12-15T04:05:00-08:00

For some weeks now I and Gustavo Barbieri at ProFUSION have been working on a new library and a set of tools, libkmod and kmod respectively. This is the announcement of its first public release.

Overview

The goal of the new library libkmod is to offer to other programs the needed flexibility and fine grained control over insertion, removal, configuration and listing of kernel modules. Using the library, with simple pieces of code it’s possible to interact with kernel modules and then there’s no need to rely on other tools for that. This is a thing lacking on Linux for a while and it’s one of the items in the Plumber’s Wish List for Linux. Quoting it:

provide a proper libmodprobe.so from module-init-tools:
Early boot tools, installers, driver install disks want to access
information about available modules to optimize bootup handling.

We went one step further and not only we are able now to give an API to load and remove kernel module, but also all the other common operations are being added to this API. The first user for this API will be udev. In a recent Linux Desktop (and also several embedded systems) when computer is booting up, udev is responsible for checking available hardware, creating device nodes under /dev (or at least configuring their permissions) and loading kernel modules for the available hardware. In a kernel from a distribution it’s pretty common to put most of the things as modules. Udev reads the /sys filesystem to check the available hardware and tries to load the necessary modules. This translates in hundreds of calls to the modprobe binary, and in several of them just to know the module is already loaded, or it’s in-kernel. With libkmod it’s possible for udev with a few lines of code to do all the job, benefiting from the configurations and indexes already opened and parsed. We’ve been talking to Kay Sievers (udev’s mantainter) and Lennart Poettering (systemd’s maintainer) regarding this and we are looking forward to have udev using libkmod soon.

Example code:

To insert a module by name without any options and strange configurations it’s sufficient to do as following (without treating errors for easy of comprehension - see the documentation for possible errors):

struct kmod_ctx *ctx = kmod_new(NULL, NULL);
struct kmod_module *mod;
kmod_module_new_from_name(ctx, name, &mod);
kmod_module_insert_module(mod, 0, NULL);
kmod_module_unref(mod);
kmod_unref(ctx);

Tools

Besides doing the library, we are re-designing the module-init-tools tools on top of the new API we created. With this first version we are already providing compatible binaries for lsmod, rmmod, insmod and modprobe, the last one with some functionality missing. Next versions we plan to fill the gaps with the provided tools and provide all the others like depmod and modinfo.

License

We try to avoid issues regarding licences: the library is licensed under “LGPLv2 or later” and the tools are under “GPLv2 or later”. There’s still lots of work to be done and places to optimize. We greatly appreciate contribution from other developers.

Roadmap

The API is not set on stone and is going to suffer some changes in future releases as we see fit to finish implementing all the tools. Below is the list of the features already implemented

kmod 1

libkmod provides the necessary API for:

List modules currently loaded
Get information about loaded modules such as initstate, refcount, holders, sections, address and size
Lookup modules by alias, module name or path
Insert modules: options from configuration and extra options can be passed, but flags are not implemented, yet
Remove modules
Filter list of modules using blacklist
For each module, get the its list of options and install/remove commands
Indexes can be loaded on startup to speedup lookups later

Tools provided with the same set of options as in module-init-tools:

kmod-lsmod
kmod-insmod
kmod-rmmod
kmod-modprobe, with some functionality still missing (use of softdep, dump configuration, show modversions)

Following is a rough roadmap for future releases:

kmod 2

Provide the API for features missing in kmod-modprobe, namely: dump configuration and indexes, soft dependencies, install and remove commands. Features relying on ELF manipulation will still be missing;
Provide all the tools available in module-init-tools. Some of them like depmod may be entirely copied from module-init-tools for later convertion;

kmod 3

Provide a single kmod tool that will abstract all the others, accepting commands like “kmod list”, “kmod remove”, “kmod insert”. Distributions may then use symlinks from current tools to the kmod binary and we can kill the ‘kmod-*’ test tools that we are introducing in kmod 1;

We thoroughly test the features implemented in kmod, but like any other software it’s possible to contain bugs that we didn’t find; we may decide to release new versions between the versions above and then this numbers change. Otherwise kmod 2 will already be sufficient for udev to pick it up as a dependency and start benefiting from the fine grained control over its operations with kernel modules.

Repositories

The repository for this project is located at http://git.profusion.mobi/cgit.cgi/kmod.git/

Package with kmod 1 source code can be downloaded from: http://packages.profusion.mobi/kmod/

Thanks

Last I’d like to thank Kay Sievers for his support in reviewing code, giving advices and helping to design kmod.

Having fun in the sky

2011-12-12T21:56:00-08:00

No words to describe free fall. If you never did, you should be rethinking your next vacations ;-)

Becoming social…

2011-12-02T15:12:00-08:00

I’ve just installed a plugin in my blog to display the common social sites below each post. From now on, if you like a post you can share it with the world using your preferred social media.

I’m using the Socialize plugin in WordPress. I’m not sure if it’s the best one: I just installed and checked it works. Any advices?

AndroidConf 2011

2011-11-26T21:17:00-08:00

Hoje dei uma palestra na AndroidConf sobre “Modificando a API do Android”. Referências ao projeto que falei sobre AVRCP podem ser encontradas em um post meu anterior. Estou disponibilizando abaixo os slides.

EDIÇÃO 02/12/2011: coloquei uma nota no slide 7, relatando o que falei durante a apresentação sobre uso de IDE.

Para aqueles que não conseguem visualizar acima, segue o link direto para a apresentação.

LinuxCon Brazil

2011-11-19T21:54:00-08:00

I’m back from LinuxCon Brazil, that was held in Sao Paulo on 17 and 18 November. Before the first keynote, ProFUSION was announced as becoming member of Linux Foundation :-)! Our logo is already in their members page.

It was also a great time to talk again to some developers I met in LinuxCon Europe last month and some that were not present there. One talk I really like was given by Eugeni Dodonov about the Intel Linux Graphics stack. It was a good overview of all the graphics stack in Linux, paying attention to Intel’s boards and drivers. Gustavo Barbieri talked about HTML5 and WebKit and other 2 ProFUSION’s employees — Rafael Antognolli and Bruno Dilly — presented “Application Development using Enlightenment Foundation (EFL)”

This time I also gave a presentation entitled “How to become an open source developer”. My focus was on the Brazilian crowd out there, willing to start to contribute to open source projects, looking for a job or just trying to understand why we do open source development. I hope it was useful for them and for you reading this blog, too. So, below are the slides of my presentation:

For those of you who can not see the file embedded above or want the direct link, here it’s in PDF format.

I also talked to some important people regarding a new project of mine. Stay tuned for a new library soon.

Back from Kernel Summit, LinuxCon Europe and ELCE

2011-11-04T00:20:00-07:00

Last week from 23-Oct to 28-Oct I was at 3 conferences in Prague, Czech Republic, together with Gustavo Barbieri, Gustavo Padovan and Ulisses Furquim: the ProFUSION crew in Prague.

Starting from Kernel Summit, I had the opportunity to join the Bluetooth Summit and participate in the discussions regarding this subsystem in Linux, both in kernel and user space. We had a lot of hot topics to discuss, including the upcoming BlueZ 5.0, Bluetooth 3.0 (high speed), Bluetooth 4.0 (low energy) and I could also demonstrate the work I’ve been doing with the AVRCP profile. I’m glad it received a good acceptance from other developers. Some of them I didn’t know personally such as Luiz von Dentz, Claudio Takahasi, Vinicius Gomes. Others I had the pleasure to meet again like Marcel Holtmann and Johan Hedberg.

(We didn’t discuss only bluetooth related things. We noticed that more than 1/3 of the people there, working in the core of Bluetooth in Linux, was Brazilian and soon we were discussing with Samuel Ortis - a French, maintainer of ConnMan - who is the best soccer player :-).)

Daniel Wagner from BMW also brought up some interesting scenarios of multiple devices connected through Bluetooth in car kits and helmets (like this one): HFP, A2DP, HSP (and maybe also AVRCP?). All of them interacting and working together at the same time. Since the gstreamer conference was also taking place at the same facility we could also discuss with PulseAudio developers. In the end, it seems BlueZ and PulseAudio are working pretty well together, though we still have to polish some rough edges for some use cases like this.

Being at Kernel Summit was a great time to meet developers of other parts of the kernel too, such as Steven Rostetd and Peter Zylstra, with whom I had more contact some time ago when I was working in the Linux scheduler.

When the Kernel Summit was over (on Tuesday), LinuxCon and ELCE were taking off. It was great to have once more these two conferences collocated and being able to attend talks on both of them. There were several talks I’d like to attend but some of them were overlapping. ~~I’m looking forward to see the recorded talks later this year~~[1]. It would be too extensive to detail each one here, so I’m just detailing some of them that grabbed more attention from me.

Gustavo Barbieri and Sulamita Garcia talked about Demistifying HTML5 and how it can be used to develop Apps. Gustavo focused on the EFL port of WebKit (in which I’m one of the developers ;-)) and the underlying technologies. It seems like the mentality of “let’s do apps in a very high-level language” instead of “providing a native language in a sdk” is coming back. Differently from what happened some years ago, this time maybe it will work out. Only future will show us.

Since this year I got involved with Android and development of the platform, I went to several Android-related talks. Leveraging Android’s Linux Heritage was really good stuff, showing how to replace some parts of the Android platform: bash instead of the I-wanna-be-a-shell that comes with Android by default, putting gstreamer in, optimizing some parts of the code, etc. In the same tune there was another talk entitled Build Community Android Distribution and Ensure the Quality. Interesting (but not surprising) to see how hard is to contribute to AOSP and how Android is much different from other open source projects we are used to.

Another interrelated areas that I have interest in (maybe because I work for a company related to embedded systems :-)) are system initialization, fast boot and development boards (such as Pandaboard). Therefore I attended systemd Administration in the Enterprise and Integrating systemd: Booting Userspace in Less Than 1 Second. The former, given by Lennart and Kay, focused on detailing some systemd features for guys running enterprise servers while in the latter Koen told us about his experience reducing boot time by using systemd in a Pandaboard. In this last talk I also met Jean Christophe, one of the developers of barebox (a bootloader aiming to replace U-Boot). Last time I checked, pandaboard was not in the list of supported boards but I was greatly surprised that now it is. Barebox has the advantages of running with caches enabled, having an architecture much more beautiful and being much faster than u-boot. In summary, IMHO it’s a bootloader done right.

Other interesting talk was Tuning Linux For Embedded Systems: When Less is More, in which Darren Hart gave instructions to reduce boot time and image size in very resource constrained scenarios (he was aiming a rootfs of only 4MB and total boot time under a second). Some key things to know is how to investigate what is not important to the application, what can be removed from kernel/userspace in order to fit the requirements and when to replace, why to replace and what to replace. Last but not least, in Developing Embedded Linux Devices Using the Yocto Project and What’s new in 1.1 David Stewart gave a status quo of the Yocto project. Interesting how the project evolved over this year and next time someone doing embedded systems think about ruling out its own distro from scratch, it would be good to look at Yocto.

I met a lot of other people for whom I apologize not citing their name here. This post would be yet bigger than it already is. I had a really great time their and I hope to continue going to these conferences. And the next one is LinuxCon Brazil, in which I’ll talk about How to Become an Open Source Developer. I look forward to seeing all of you there.

I’d like to thank the Linux Foundation for organizing such a great event and ProFUSION to allowing and sponsoring me to be there.

Side note: the problem is that now I want to do a lot of things in different projects without having time to to: systemd, Linux kernel, BlueZ, pandaboard, barebox, Android, etc :-)

[1] UPDATE: videos have been published - http://free-electrons.com/blog/elce-2011-videos/

AVRCP 1.3 on BlueZ

2011-10-11T21:42:00-07:00

During the past weeks I’ve been working again on the BlueZ project and now we can finally announce that the AVRCP 1.3 profile is officially supported.

Technical background

For those who don’t know what I’m talking about, here comes a little background for those buzzwords:

BlueZ is the user-space part of the Bluetooth® technology stack used on Linux and Android. It has support for several Bluetooth profiles such as RFCOMM, HID, PAN, PBAP, OBEX, HFP, A2DP (some of them are implemented as separate projects) that are defined by the Bluetooh SIG. In simpler terms, BlueZ is what allows your Linux device to do amazing things with Bluetooth technology such as stream stereo music, make phone calls and other wireless magics.

One of these profiles supported by BlueZ is AVRCP (Audio/Video Remote Control Profile), which allows two devices to communicate through Bluetooth technology and exchange commands/messages to control the music/video being played.

New features

Until some weeks ago BlueZ only had support for version 1.0 of the AVRCP protocol. This early version allows a Controller device (e.g. a Bluetooth technology-based car kit) to tell the Target device (e.g. a smartphone) to play, pause, go to next music and go to previous music. We’ve now upstreamed an implementation of AVRCP Version 1.3, which adds some nice features to the previously supported version, such as:

Transmitting metadata of the music being played;
Change Application Settings such as Equalizer, Repeat, Shuffle and Scan modes;
Set current status of media playback: playing, stopped, paused, forward-seeking, reverse-seeking.

Some time ago I bought a Bluetooth stereo car kit. How boring it was having the ability to stream music from my phone but not see any information regarding who was playing, which album, etc. This is no more. Now we have proper support for AVRCP 1.3 :-). Our ProFUSION team utilized the open source baseport for the OMAP™ processor-based Blaze™ mobile development platform from Texas Instruments Incorporated (TI) to help achieve this milestone. Additionally, we worked together with TI on testing and debugging to make this AVRCP 1.3 support a reality. Below you can see yours truly holding a Blaze™ mobile development platform from TI, sending music metadata to a Bluetooth technology-enabled car kit.

ANNOUNCE: codespell 1.2

2011-09-30T17:24:00-07:00

Since I created a mailing list for codespell, the announcements here will not have as many details as before. Checkout the new version of codespell: http://groups.google.com/group/codespell/browse_thread/thread/bce1a6f83d4bbd85

One of the issues I with codespell was that it was trying to parse cscope.out, since it’s a text file. On Linux Kernel this file can get very big and besides taking much longer, sometimes it was running out of memory :-). Now codespell has an option to ignore files, even text ones. It’s as easy as passing —skip=”*.eps,cscope.out” (notice this is useful for ignoring eps images too). Another useful thing (not much for Linux Kernel though) is proper detection of encoding by using chardet.

Attending AI classes

2011-09-21T05:30:00-07:00

For those who didn’t know, Stanford is offering some online courses, starting from next month. During my graduation I already have some AI classes but I thought it would be interesting to participate in this class with thousand of students.

One area that I like on AI is the on to teach computers how to play games. I like that so much that during my AI classes in Politecnico di Milano I developed a computer game to play tic-tac-toe in NxNxN dimensions. It’s actually fun to play, and you can find some screenshots (and the code) on a past blog post. I didn’t have time to continue maintaining it, but the part that I liked more was actually designing the algorithm, using MinMax, and eventually finding out that I couldn’t win the game anymore when playing against the computer. This was both motivating and demotivating because I knew that if I did developed it right, I couldn’t possibly win anymore. Maybe I’ll update this project when I have some time. If you would like to try it and find any problem, send me a patch and I will happily apply. If you think you have a better algorithm, it’s very easy to develop your own Player and plug it in the game. I challenge you to win my MinMax player ;-).

During the time I was developing this game, a friend of mine that lived with me in Italy had a dream of making money playing poker. Because of this he was attending the AI classes too and eventually playing online to develop his strategies (in sites such as partypoker). It was very fun talking to him and his plans and strategies, though I don’t know if he succeeded.

Even if you frequented some AI class before like me and my friend, I think it will be very interesting to participate on this one. If you didn’t register yet, you still have some time: go to http://www.ai-class.com and subscribe.

Seminário de Linux Embarcado 2011

2011-08-10T20:30:00-07:00

No último final de semana participei do Seminário de Linux Embarcado, em que palestrei sobre “systemd: repensando a inicialização”. O feedback que tive do pessoal foi positivo, mostrando bastante interesse nos diversos tipos de inicialização sob demanda, exclusivos desse sistema de init. Os slides estão disponíveis aqui no blog.

Gostei bastante do evento, que teve também participação do Antognolli, que trabalha comigo, sobre interfaces gráficas em sistemas embarcados usando o conjunto de bibliotecas EFL. Outro conhecido meu de outras conferências, Glauber Costa, falou sobre QEMU.

codespell 1.1

2011-06-18T20:32:00-07:00

codespell 1.1 is just released.

Since I was receiving emails about bugs and suggestions I decided to create a mailing list. Check out the announcement of v1.1 there.

Benchmarking Javascript engines for EFL

2011-06-14T19:25:00-07:00

The Enlightenment Foundation Libraries has several bindings for other languages in order to ease the creation of end-user applications, speeding up its development. Among them, there’s a binding for Javascript using the Spidermonkey engine. The questions are: is it fast enough? Does it slowdown your application? Is Spidermonkey the best JS engine to be used?

To answer these questions Gustavo Barbiericreated some C, JS and Python benchmarks to compare the performance of EFL using each of these languages. The JS benchmarks were using Spidermonkey as the engine since elixir was already done for EFL. I then created new engines (with only the necessary functions) to also compare to other well-known JS engines: V8 from Google and JSC (or nitro) from WebKit.

Libraries setup

For all benchmarks EFL revision 58186 was used. Following the setup of each engine:

Spidermonkey: I’ve used version 1.8.1-rc1 with the already available bindings on EFL repository, elixir;
V8: version 3.2.5.1, using a simple binding I created for EFL. I named this binding ev8;
JSC: WebKit’s sources are needed to compile JSC. I’ve used revision 83063. Compiling with CMake, I chose the EFL port and enabled the option SHARED_CORE in order to have a separated library for Javascript;

Benchmarks

Startup time: This benchmark measures the startup time by executing a simple application that imports evas, ecore, ecore-evas and edje, bring in some symbols and then iterates the main loop once before exiting. I measured the startup time for both hot and cold cache cases. In the former the application is executed several times in sequence and the latter includes a call to drop all caches so we have to load the library again from disk

Runtime - Stress: This benchmark executes as many frames per second as possible of a render-intensive operation. The application is not so heavy, but it does some loops, math and interacts with EFL. Usually a common application would do far less operations every frame because many operations are done in EFL itself, in C, such as list scrolling that is done entirely in elm_genlist. This benchmark is made of 4 phases:

Phase 0 (P0): Un-scaled blend of the same image 16 times;
Phase 1 (P1): Same as P0, with additional 50% alpha;
Phase 2 (P2): Same as P0, with additional red coloring;
Phase 3 (P3): Same as P0, with additional 50% alpha and red coloring;

The C and Elixir’s versions are available at EFL repository.

Runtime - animation: usually an application doesn’t need “as many FPS as possible”, but instead it would like to limit to a certain amount of frames per second. E.g.: iphone’s browser tries to keep a constant of 60 FPS. This is the value I used on this benchmark. The same application as the previous benchmark is executed, but it tries to keep always the same frame-rate.

Results

The first computer I used to test these benchmarks on was my laptop. It’s a Dell Vostro 1320, Intel Core 2 Duo with 4 GB of RAM and a standard 5400 RPM disk. The results are below.

First thing to notice is there are no results for “Runtime - animation” benchmark. This is because all the engines kept a constant of 60fps and hence there were no interesting results to show. The first benchmark shows that V8’s startup time is the shortest one when considering we have to load the application and libraries from disk. JSC was the slowest and Spidermonkey was in between.

With hot caches, however, we have another complete different scenario, with JSC being almost as fast as the native C application. Following, V8 with a delay a bit larger and Spidermonkey as the slowest one.

The runtime-stress benchmark shows that all the engines are performing well when there’s some considerable load in the application, i.e. removing P0 from from this scenario. JSC was always at the same speed of native code; Spidermonkey and V8 had an impact only when considering P0 alone.

Next computer to consider in order to execute these benchmarks was a Pandaboard, so we can see how well the engines are performing in an embedded platform. Pandaboard has an ARM Cortex-A9 processor with 1GB of RAM and the partition containing the benchmarks is in an external flash storage drive. Following the results for each benchmark:

Once again, runtime-animation is not shown since it had the same results for all engines. For the startup tests, now Spidermonkey was much faster than the others, followed by V8 and JSC in both hot and cold caches. In runtime-stress benchmark, all the engines performed well, as in the first computer, but now JSC was the clear winner.

There are several points to be considered when choosing an engine to be use as a binding for a library such as EFL. The raw performance and startup time seems to be very near to the ones achieved with native code. Recently there were some discussions in EFL mailing list regarding which engine to choose, so I think it would be good to share these numbers above. It’s also important to notice that these bindings have a similar approach of elixir, mapping each function call in Javascript to the correspondent native function. I made this to be fair in the comparison among them, but depending on the use-case it’d be good to have a JS binding similar to what python’s did, embedding the function call in real python objects.

ESC Brazil - Realtime Linux with RT_PREEMPT

2011-06-03T20:34:00-07:00

Two weeks ago I’d give a talk about realtime Linux at ESC Brazil: “Usando Linux como Sistema de Tempo Real” (Using Linux as a realtime OS). Sadly some days before while playing soccer I broke my fibula and I had to have a surgery. I regret I couldn’t attend this conference.

At least in the company I work for there are more people with knowledge in this area. Gustavo Barbieri went there in my place and had a good feedback from the attendees.

Now I have stay home. At least for 1 or 2 months :-(.

codespell 1.1-rc1

2011-06-02T16:30:00-07:00

I’m glad to announce the first RC of codespell 1.1. I decided to let the biggest feature for the next version and release 1.1 with the small but important features that were already implemented. This new version comes with the following features:

Verbosity level: tired of seeing so many things printed while the fixes are taking place? Now you can filter what you see
Exclusion list: there are cases in which the codespell spots a false positive, but disabling the entry in the dictionary will prevent it from fixing many other places. This is particularly true when there are names in source code. In Linux kernel I’ve seen some names with “Taht” and “Teh” that were incorrectly fixed to “That” and “The”. Now we have a file with lines that are exclude from the ones codespell will fix. Hopefully such lines will not change very often and we can maintain a file per project for future executions of codespell in each project. I’m maintaining one for the Linux kernel. It’s in data/linux-kernel.exclude.
Interactive mode: for those fixes that are not done automatically (because they have more than one possible fix), now we can interactively decide on each one. I recommend for everyone interested in this feature to run codespell once without this option to fix the automatic ones and another time to go through the other fixes.
Stats (summary) of the changes: are you interested how many times a word was misspelled? Now codespell can display a summary of all the fixes it has done.

I was particularly worried about the increase in runtime when using the exclusion list. However it proved to be very fast when excluding the lines by using their hashes. I can successfully parse the entire Linux kernel tree within 1min30s in my laptop with slow HD. The biggest feature that I’ve left for the next version is to allow changes to be applied only to parts of the source code like comments and strings. I expect to implement this for a future 1.2 version.

Besides these new features, there are some fixes to the dictionary. Thanks to all of you who have sent me fixes and suggestions. I’m glad to see patches generated by codespell been applied to other opensource projects that I’m not the one to send. As of now, I’ve seen patches been applied to: Linux Kernel, oFono, ConnMan, FreeBSD, LLVM, clang, EFL and others that I don’t remember right now.

For those who prefer to wait for a stable release, I’m also releasing codespell 1.0.2 with fixes only to the dictionary.

As always, you can download codespell packages from:

http://packages.profusion.mobi/codespell/

Repositories are available at:

http://git.profusion.mobi/cgit.cgi/lucas/codespell/

https://github.com/lucasdemarchi/codespell

Por uma web melhor e mais segura

2011-05-28T01:37:00-07:00

Hoje adicionei no site um script para alertar usuários que usam versões antigas (e inseguras) de browsers. O projeto é o “Salve a Web, por favor”. Ele é opensource e pode ser visto no repositório do github. Basta carregar o script com a seguinte entrada na sua página:

<script type="text/javascript" src="http://sawpf.com/1.0.js"></script>

Acho que pela própria natureza dos posts no meu blog, a divisão dos browsers não é parecida com aquela global, em que o Internet Explorer ainda é o browser mais usado. Porém acho importante alerr os usuários. Quem sabe poderemos ter uma web melhor e mais segura futuramente? Veja abaixo as estatísticas de acesso a esse blog no últimoo ano.

Os poucos visitantes com browsers antigos que leem esse blog vão ver o seguinte banner acima (obtido mudando o user agent do Chromium para um IE 6):

Cross-compiling with icecc/icecream

2011-05-13T16:28:00-07:00

Very common questions I hear when dealing with compiling open source projects are:

How do I cross-compile a project using icecc/icecream?
How to use a different compiler version for compiling my project and still benefiting from icecc/icecream?

Note: from now on I’ll always refer to icecream instead icecc/iceccd/distcc for the name of the project.

Given you already created your cross toolchain (or downloaded from somewhere else, e.g. CodeSourcery/Linaro) these two questions are essentially the same. All you have to do is to follow the two steps below:

1. Create the “compiler environment”

Understanding this part is really understanding how this magic remote-compiling works. When you want to compile a source remotely, what icecream does is sending a copy of your compiler and the things it needs to the remote machine, executing the process and getting back the result. By “things it needs” I mean: assembler, linker, libc, libgcc and some other libraries like libm, libgmp, libstdc++, libz, etc. Creating this environment with icecream is dead easy: call “icecc —build-native”. Following is the output I get on my Archlinux box with gcc 4.6.0 as default compiler:

$ icecc --build-native
adding file /usr/bin/gcc
adding file /lib/libc.so.6
adding file /lib/ld-linux-x86-64.so.2
adding file /usr/bin/g++
adding file /usr/bin/as
adding file /usr/lib/libopcodes-2.21.0.20110430.so
adding file /usr/lib/libbfd-2.21.0.20110430.so
adding file /lib/libdl.so.2
adding file /usr/lib/libz.so.1
adding file /usr/bin/cc1=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/cc1
adding file /usr/lib/libcloog-isl.so.1
adding file /usr/lib/libisl.so.6
adding file /usr/lib/libgmp.so.10
adding file /usr/lib/libppl_c.so.4
adding file /usr/lib/libppl.so.9
adding file /usr/lib/libgmpxx.so.4
adding file /usr/lib/libstdc++.so.6
adding file /lib/libm.so.6
adding file /usr/lib/libgcc_s.so.1
adding file /usr/lib/libpwl.so.5
adding file /usr/lib/libmpc.so.2
adding file /usr/lib/libmpfr.so.4
adding file /usr/bin/cc1plus=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/cc1plus
adding file /usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/liblto_plugin.so
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confPaVA2Q
creating 6b1d2b44080a004a88c0746fe128172e.tar.gz

Note that in the last line it created a .tar.gz file. This is the environment that will be sent to other machines. If you want to use another compiler, you need to create another environment that will be later passed to icecream in the second step.

To create an environment for a compiler that is not the default in your machine, first thing you need is to have it in your PATH, pointing to the icecc binary. Here sometimes I use GCC 4.4 instead of the default compiler, so I’ll use it as an example. In my machine, GCC 4.4 is installed in /usr/bin/gcc-4.4 and icecream is installed in /opt/icecream/bin:

$ which gcc-4.4
/usr/bin/gcc-4.4
$ which icecc
/opt/icecream/bin/icecc

Go to where icecc is installed and make a symlink to icecc with the same name of the compiler you want to use:

$ sudo ln -s icecc gcc-4.4
$ sudo ln -s icecc g++-4.4

Now, tell icecream to create the environment for this compiler:

$ cd /tmp
$ export ICECC_CC=gcc-4.4
$ export ICECC_CXX=g++4.4
$ icecc --build-native

Now your environment is ready. Copy the file generated to somewhere you’ll remember later (you can give it whatever name you like):

$ sudo cp my-environment-built-above.tar.gz /var/icecream/gcc-4-4.4_x86-64.tar.gz

This step will be done only once, as opposed to the second step below that is repeated whenever you compile a new source.

2. Tell icecream which environment to use

When compiling a source code, tell icecream which environment it will send to other hosts. You do this by exporting some env vars:

$ export ICECC_CC=gcc-4.4
$ export ICECC_CXX=g++-4.4
$ export ICECC_VERSION=/var/icecream/gcc-4-4.4_x86-64.tar.gz

Now you can compile your source code as usual, be it calling gcc directly or through makefiles or other build systems. For example:

$ gcc-4.4 helloworld.c -o helloworld

If you manage a handful of machines running icecream, I’d recommend a software we developed at ProFUSION called Picolé.

UPDATE: if you want a recommendation on how to build a cross toolchain, crossdev it is. The steps are the same as above, replacing gcc-4.4 with the name given to your compiler (e.g. arm-elf-gcc-4.6.0)

codespell 1.0.1

2011-04-16T16:34:00-07:00

I released codespell 1.0.1 after a small fix that is really necessary for codespell working right. Thanks to Luis Felipe Strano that spot the bug and generated another giant patch to LLVM.

Grab the latest version below:

http://packages.profusion.mobi/codespell/

ANNOUNCE: codespell 1.0

2011-04-01T15:43:00-07:00

I’m glad to announce codespell 1.0! After 3 RCs and patches submitted to several projects, I thought it was stable enough to call it 1.0. You can download the 1.0 version below:

http://packages.profusion.mobi/codespell/codespell-1.0.tar.bz2

Seemy previous post if you are willing to know what codespell is or read the README file inside the package.

I have already filled a TODO file with ideas for the next version. They came to mind after I’ve generated a giant patch for the Linux kernel. It is the biggest patch I’ve ever produced with codespell. Really, I think it’s the biggest patch I’ve produced ever. I hope Linus accept that patch as is because without the changes I’m planning for codespell 1.1, it’s a pain to fix some corner cases.

Some people are sending me suggestions and more misspellings to my email. I appreciate those emails and seeing they are successfully using codespell in other projects. If you want a faster way to have your changes incorporated into codespell, you may also send patches through git-send-email or just use my repository on github to send me pull requests.

UPDATE (04/11/2011): Linus accepted the patch. There’s also a discussion with further improvements for codespell on LKML.

On typos and misspellings

2011-02-07T03:56:00-08:00

This week I’m in a release mood, so I’m releasing several projects I’m involved with. If you lost the first two, checkout dietsplash 0.3 and genslide 0.3 (though the announcement was in Portuguese).

After developing for several projects I’ve noticed most of them contain typos and misspellings. Even if this does not directly affect the source quality (unless the misspellings are in documentation), if we left the comment there, we’ve left it for a reason: because we want the reader of that code to stop and read it. It’s particularly good to have the correct spelling of each word when there are people from several parts of the word that maybe do not have English as their mother tongue (as I don’t). This way we can be more sure the correct message is being given through code, comments and documentation.

Thinking a bit on this I made some bash and awk scripts to fix misspellings based on the list of common misspellings available on wikipedia. I’ve successfully sent patches for projects like the Linux kernel, ConnMan, oFono and EFL. After some of them were accepted and after I decided to run the scripts again, I noticed how slow they were (if you are curious what they did, you can google on the oFono mailing list, in which I explain the scripts). So, I started a new, very short project: codespell. Measuring against the Linux kernel tree, it runs circa 20x faster than the previous scripts. Its current version is 1.0-rc1 and I’d like to have some more testers before I release the final 1.0.

Codespell is designed to fix misspellings in source code, but it can be applied to any type of text files. When possible, codespell will automatically fix the misspelling. Otherwise it will give some suggestions about possible changes. For example, running against the Linux kernel tree, it gives me several lines like below:

drivers/target/target_core_transport.c:2528: competion ==> competition, completion

drivers/edac/cpc925_edac.c:186: MEAR ==> wear, mere, mare

WARNING: Decoding file drivers/hid/hid-pl.c

WARNING: using encoding=utf-8 failed.

WARNING: Trying next encoding: iso-8859-1WARNING: Decoding file drivers/hid/hid-pl.cWARNING: using encoding=utf-8 failed. WARNING: Trying next encoding: iso-8859-1

drivers/net/niu.c:3276: clas ==> class | disabled because of name clash in c++

FIXED: ../kernel/drivers/scsi/aacraid/aacraid.h

FIXED: ../kernel/drivers/scsi/lpfc/lpfc_sli.c

FIXED: ../kernel/drivers/scsi/aacraid/aacraid.hFIXED: ../kernel/drivers/scsi/lpfc/lpfc_sli.c

(This is all in beautiful colored lines! Test it to see the true output)

The first two illustrate some changes that cannot be automatically done because that misspelling is a common one for more than one word. So, codespell gives you the file and line where they occur.

The WARNINGs are related to the encoding of the file. Codespell will default to parse files in UTF-8 encoding, which will handle ‘ascii’ as well. If it fails to decode any line, it will try the next available encoding, i.e. ISO-8859-1. Using these two encodings I have successfully ran codespell with all the projects I care about.

Codespell allows some changes to be disabled. This is shown by the “clas => class” fix, that are not always safe to do because of name clash with C++ code.

The lines prefixed with “FIXED” show the files that were automatically fixed. In current Linus’ master branch, this resulted in:

2545 files changed, 5007 insertions(+), 5007 deletions(-)

These were the automatic fixes, that may contain some false positives. The funniest one is the on found in Documentation/DocBook/kernel-hacking.tmpl:

/*

* Sun people can’t spell worth damn. “compatability” indeed.

* At least we *know* we can’t spell, and use a spell-checker.

*/

As can be seen by the number above, this is not really true ;-).

So, there it’s: **codespell 1.0-rc1**. Get it. Test it. Report problems. Tell me about projects that were successfully patched.

git repository is down

2011-02-01T20:13:00-08:00

I’m turning down my repository at politreco.com. All my projects are now available either on ProFUSION’s repo or on github.

ANÚNCIO: genslide 0.3

2011-02-01T03:14:00-08:00

É com satisfação que anuncio a primeira versão do programa genslide, genslide 0.3. É um simples programa que criei para facilitar a conversão de letras de música em slides, salvando-os em pdf através de Latex/Xelatex. Esses slides podem ser usados durante shows e cultos e venho usando-os já faz 6 meses na igreja que eu frequento. Embora ainda tenha bastante coisa para ser feita, ele já se mostra bastante útil e facilita bastante na hora de criar novos slides.

A ideia é que o usuário não se preocupe com muito com a formatação, apenas com a divisão de título, estrofes e coro. Se você é da área de computação, isso se traduz dizendo que a gramática é bem simples. Uma símples música, contendo o título e 2 estrofes:

Esse é um título de exemplo

A primeira linha da primeira estrofe

A segunda linha da primeira estrofe

A terceira linha da primeira estrofe

A primeira linha da segunda estrofe

A segunda linha da segunda estrofe

A terceira linha da segunda estrofe

Existem alguns recursos adicionais, como marcação de uma estrofe como coro e marcação de um trecho da música para repetição. Além disso, nem sempre a frase cabe nos espaço disponível do slide. A divisão em várias linhas é feita de forma automática, bem como a divisão entre os slides. Assim, se você tiver uma linha da forma abaixo, ela será automaticamente dividida em 2 ou mais linhas.

Essa é uma linha realmente longa que não caberia em uma única linha no slide

Outro recurso interessante é a integração com o site de letras do Terra. Dessa forma é possível gerar os slides simplesmente passando o link para a letra nesse site. Por exemplo, para gerar um slide da música “Pra cima Brasil”, do João Alexandre:

\$ genslide-bin http://letras.terra.com.br/joao-alexandre/46505/

A aparência dos slides finais é dada por um conjunto de arquivos que formam um tema. O tema que uso atualmente é bem simples, mas nada impede que temas mais sofisticados sejam criados. Atualmente a linha acima geraria o conjunto de slides abaixo. Espero que esse programa seja útil não só para mim, mas para todos que precisam fazer esse tipo de conversão.

**[Pra cima brasil](http://www.slideshare.net/lucasdemarchi/pra-cima-brasil “Pra cima brasil”)**

Download

Você pode fazer o download de genslide na página de downloads do programa. A última versão, 0.3, pode ser obtida em formato tar.bz2 ou zip. Tudo o que você precisa é ter python3 instalado (testado somente em Linux).

Hacking e feedback

O código-fonte da última versão está disponível acima ou, para a última versão de desenvolvimento, você pode fazer o download no repositório do projeto.

ANNOUNCE: dietsplash 0.3

2011-01-27T14:37:00-08:00

I’m pleased to release a new version of dietsplash. This is the first announcement for dietsplash, though it has always been open source software. Here it is, dietsplash 0.3.

dietsplash is a small boot splash screen for Linux. Its aim is to do the minimal work possible so you can see your system booting without getting bored about the messages going on (or better, your client doesn’t complain to you that it’s ugly to see all those messages). Its main target is embedded systems, where running a full/big splash like plymouth is an overkill. Special attention is given to keep it small: as of now, it produces a binary of just 14KB without images and 34KB using the images that come as example bundled into the binary.

Running it is very simple, requiring very little configuration if you don’t have any splash screen yet. If you are already running another splash like plymouth or psplash, you’ll first have to disable it. Please, see the README file that details what you have to do in order to have it up and running.

DOWNLOAD

Releases are available at http://packages.profusion.mobi/dietsplash/. Here is the latest release: dietsplash 0.3.

HACKING

Patches and feedback are very welcome. You can either send me an email or use the infrastructure at github, where there’s a “mirror” of the git repository.

Git repositories: git://git.profusion.mobi/users/lucas/dietsplash.git (where the fun happens) and git://github.com/lucasdemarchi/dietsplash.git (synchronized every now and then).

ProFUSION work on oFono and ConnMan

2011-01-18T12:33:00-08:00

As some of you know, I’m working on some open source projects at ProFUSION. Among them there are two projects started by Intel that are now part of Meego. If you look at Meego Architecture you’ll see these components are a great part of the Communications domain.

In ConnMan ProFUSION made mainly some bug fixes and minor improvements while we were creating a binding for e_dbus and a module for E17. It’s working pretty well, it’s fast and reliable. Examining my logs in my desktop system I can see that in less than a second ConnMan is up with wired and wireless interfaces and the IP is already discovered through dhcp.

In oFono we did a bit more work, implementing missing features in the core and also in drivers of Huawei and MBM (sony-ericsson) modems as well as on the phone simulator, phonesim.

It’s great to see on Marcel’s reports last month that in 2010 ProFUSION was the 3rd major contributor for oFono and the 5th for ConnMan. I’m happy to be one of the developers contributing to these 2 great projects.

Open source hardware - Arduino

2011-01-18T04:34:00-08:00

This post is just forward of an amazing documentary I saw today. It’s Arduino The Documentary. How open source hardware became cheap and fun. It’s worth watching all its 27min.

I think universities in Brazil should learn about and join projects like this.

EFL 1.0 Beta 2

2010-11-12T19:52:00-08:00

EFL team is proud to announce beta2 of the coming EFL 1.0. WebKit-EFL also received a new snapshot as well as several other related projects. For more information, take a look at the announcement.

Criando imagens customizadas de endereços de email

2010-11-02T07:15:00-07:00

Endereços de email são muitas vezes mascarados na web para que não sejam facilmente obtidos por programas que tem por objetivo enviar spam. É relativamente fácil escrever um programinha que fica vasculhando a web em busca de endereços de email, cadastrá-los em um banco de dados e depois utilizá-los como quisermos. Para que isso não ocorra, várias vezes nos deparamos com endereços de email passados como “joao AT gmail DOT com” ou alguma coisa do tipo. Outra técnica muito utilizada é transformá-lo em uma imagem. Dessa forma o programinha malicioso que a princípio era fácil se torna bem mais difícil.

Eu acho que hoje em dia temos bons programas de anti-spam, o que torna os métodos acima não tão relevantes. Porém, acredito que converter o endereço de email em uma imagem o deixa bonito. Veja por exemplo a minha página de contatos. Fazer imagens desse tipo apara domínios conhecidos como “gmail.com”, “hotmail.com” e “yahoo.com” é bem fácil: vários sites permitem que você escolha um desses domínios e escreva o seu email. Um que eu gostei foi o do freetechjournal, pois tem vários domínios ali dos serviços de email mais populares. É fácil encontrar serviços semelhantes que permitem alguma customização.

Procurei bastante na web algum site que permitisse criar um email com uma imagem de domínio customizada, porém não achei nenhum. Como é uma operação bem simples, resolvi fazer um script em python para isso. Estou disponibilizando o iconifymail, que contém também a imagem usada para a empresa em que trabalho, ProFUSION, que pode servir como base para que você faça a sua própria imagem (obrigado, Marina). O uso é bem simples, basta passar como primeiro argumento a imagem a ser utilizada como domínio e, como segundo, o seu endereço de email (sem o domínio). Algumas coisas eu deixei hard-coded no script, mas creio que seja fácil adaptá-lo para o tamanho de imagem que você quiser. O resultado final é o que pode ser visto abaixo e também na minha página de contatos:

Essa é a versão 0.1 do iconifymail, open source, disponibilizado sob GPLv3. Se você quiser contribuir, pode usar os repositórios git: oficial ou mirror.

Easily embedding WebKit into your EFL application

2010-10-23T23:53:00-07:00

This is the first of a series of posts that I’m planning to do using basic examples in EFL, the Enlightenment Foundation Libraries. You may have heard that EFL is reaching its 1.0 release. Instead of starting from the very beginning with the basic functions of these libraries, I decided to go the opposite way, showing the fun stuff that is possible to do. Since I’m also an WebKit developer, let’s put the best of both softwares together and have a basic window rendering a webpage.

Before starting off, just some remarks:

I’m using here the basic EFL + WebKit-EFL (sometimes called ewebkit). Developing an EFL application can be much simpler, particularly if you use an additional library with pre-made widgets like Elementary. However, it’s good to know how the underlying stuff works, so I’m providing this example.
This could have been the last post in a series when talking about EFL since it uses at least 3 libraries. Don’t be afraid if you don’t understand what a certain function is for or if you can’t get all EFL and WebKit running right now. Use the comment section below and I’ll make my best to help you.

Getting EFL and WebKit

In order to able to compile the example here, you will need to compile two libraries from source: EFL and WebKit. For both libraries, you can either get the last version from svn or use the last snapshots provided.

EFL:

Grab a snapshot from the download page. How to checkout the latest version from svn is detailed here, as well as some instructions on how to compile

WebKit-EFL:

A very detailed explanation on how to get WebKit-EFL up and running is available on trac. Recently, though, WebKit-EFL started to be released too. It’s not detailed in the wiki yet, but you can grab a snapshot instead of checking out from svn.

hellobrowser!

In the spirit of “hello world” examples, our goal here is to make a window showing a webpage rendered by WebKit. For the sake of simplicity, we will use a default start page and put a WebKit-EFL “widget” to cover the entire window. See below a screenshot:

[caption id=”attachment_324” align=”aligncenter” width=”490” caption=”hellobrowser - WebKit + EFL”][/caption]

The code for this example is available here. Pay attention to a comment in the beginning of this file that explains how to compile it:

gcc -o hellobrowser hellobrowser.c   
    -DEWK_DATADIR="\"$(pkg-config --variable=datadir ewebkit)\""   
    $(pkg-config --cflags --libs ecore ecore-evas evas ewebkit)

The things worth noting here are the dependencies and a variable. We directly depend on ecore and evas from EFL and on WebKit. We define a variable, EWK_DATADIR, using pkg-config so our browser can use the default theme for web widgets defined in WebKit. Ecore handles events like mouse and keyboard inputs, timers etc whilst evas is the library responsible for drawing. In a later post I’ll detail them a bit more. For now, you can read more about them on their official site.

The main function is really simple. Let’s divide it by pieces:

    // Init all EFL stuff we use
    evas_init();
    ecore_init();
    ecore_evas_init();
    ewk_init();

Before you use a library from EFL, remember to initialize it. All of them use their own namespace, so it’s easy to know which library you have to initialize: for example, if you call a function starting by “ecore_”, you know you first have to call “ecore_init()”. The last initialization function is WebKit’s, which uses the “ewk_” namespace.

    window = ecore_evas_new(NULL, 0, 0, 800, 600, NULL);
    if (!window) {
        fprintf(stderr, "something went wrong... :(\n");
        return 1;
    }

Ecore-evas then is used to create a new window with size 800x600. The other options are not relevant for an introduction to the libraries and you can find its complete documentation here.

    // Get the canvas off just-created window
    evas = ecore_evas_get(window);

From the Ecore_Evas object we just created, we grab a pointer to the evas, which is the space in which we can draw, adding Evas_Objects. Basically an Evas_Object is an object that you draw somewhere, i.e. in the evas. We want to add only one object to our window, that is where WebKit you render the webpages. Then, we have to ask WebKit to create this object:

    // Add a View object into this canvas. A View object is where WebKit will
    // render stuff.
    browser = ewk_view_single_add(evas);

Below I demonstrate a few Evas’ functions that you use to manipulate any Evas_Object. Here we are manipulating the just create WebKit object, moving to the desired position, resizing to 780x580px and then telling Evas to show this object. Finally, we tell Evas to show the window we created too. This way we have a window with an WebKit object inside with a little border.

    // Make a 10px border, resize and show
    evas_object_move(browser, 10, 10);
    evas_object_resize(browser, 780, 580);
    evas_object_show(browser);
    ecore_evas_show(window);

We need to setup a bit more things before having a working application. The first one is to give focus to the Evas_Object we are interested on in order to receive keyboard events when opened. Then we connect a function that will be called when the window is closed, so we can properly exit our application.

    // Focus it so it will receive pressed keys
    evas_object_focus_set(browser, 1);

    // Add a callback so clicks on "X" on top of window will call
    // main_signal_exit() function
    ecore_event_handler_add(ECORE_EVENT_SIGNAL_EXIT, main_signal_exit, window);

After this, we are ready to show our application, so we start the mainloop. This function will only return when the application is closed:

    ecore_main_loop_begin();

The function called when the application is close, just tell Ecore to exit the mainloop, so the function above returns and the application can shutdown. See its implementation below:

static Eina_Bool
main_signal_exit(void *data, int ev_type, void *ev)
{
    ecore_evas_free(data);
    ecore_main_loop_quit();
    return EINA_TRUE;
}

Before the application exits, we shutdown all the libraries that were initialized, in the opposite order:

    // Destroy all the stuff we have used
    ewk_shutdown();
    ecore_evas_shutdown();
    ecore_shutdown();
    evas_shutdown();

This is a basic working browser, with which you can navigate through pages, but you don’t have an entry to set the current URL, nor “go back” and “go forward” buttons etc. All you have to do is start adding more Evas_Objects to your Evas and connect them to the object we just created. For a still basic example, but with more stuff implemented, refer to the EWebLauncher that we ship with the WebKit source code. You can see it in the “WebKitTools/EWebLauncher/” folder or online at webkit’s trac. Eve is another browser with a lot more features that uses Elementary in addition to EFL, WebKit. See a blog post about it with some nice pictures.

Now, let’s do something funny with our browser. With a bit more lines of code you can turn your browser upside down. Not really useful, but it’s funny. All you have to do is to rotate the Evas_Object WebKit is rendering on. This is implemented by the following function:

// Rotate an evas object by 180 degrees
static void
_rotate_obj(Evas_Object *obj)
{
    Evas_Map *map = evas_map_new(4);

    evas_map_util_points_populate_from_object(map, obj);
    evas_map_util_rotate(map, 180.0, 400, 300);
    evas_map_alpha_set(map, 0);
    evas_map_smooth_set(map, 1);
    evas_object_map_set(obj, map);
    evas_object_map_enable_set(obj, 1);

    evas_map_free(map);
}

See this screenshot below and get the complete source code.

[caption id=”attachment_335” align=”aligncenter” width=”490” caption=”EFL + WebKit doing Politreco upside down”][/caption]

Understanding RT by graphics

2010-08-29T03:30:00-07:00

A lot of people seem not to understand very well what real-time means. They usually tend to think RT has anything to do with performance and the raw throughput. It doesn’t. It’s all about determinism and guarantees.

This post’s title has “by graphics” words. I think the graphics below are worth a thousand words. I obtained them while porting the RT tree to Freescale boards some weeks ago at ProFUSION. Basically I’m running a task that wakes up every 40ms, run a tiny job, send some numbers through the network and sleeps again. The time in the graphics is the difference between the total time (sleep + wake-up + execution) and 40ms.

One important fact: while this is running, I’m running some CPU-intensive jobs in background and the board is receiving a ping flood from another host.

See that when the task is running with real-time priority, it doesn’t matter there’s a hugger job in background or someone is trying to take your board down with a ping flood. It’s always possible to draw a line and say it will never* go beyond that limit. In the other hand, when running with normal priority, the total time varies much more.

[caption id=”attachment_298” align=”aligncenter” width=”550” caption=”Task running with normal priority”][/caption]

[caption id=”attachment_299” align=”aligncenter” width=”550” caption=”Task running with real-time priority”][/caption]

PS.: In the graphics above I’m using an oscilloscope made by Arnaldo Carvalho de Melo. Thanks, acme.

* Well, never is a strong word. You better test with several scenarios, workloads, etc before saying that.

WebKit

2010-08-28T05:15:00-07:00

After some time working with the EFL port of WebKit, I’ve been nominated as an official webkit developer. Now I have super powers in the official repository :-), but I swear I intend to use it with caution and responsibility. I’ll not forget Uncle Ben’s advice: ”with great power comes great responsibility”.

I’m preparing a post to talk about WebKit, EFL, eve (a new web browser based on WebKit + EFL) and how to easily embed a browser in your application. Stay tuned.

strace, valgrind and gdb as a strace-like tool

2010-07-30T06:18:00-07:00

A long time since I do not post some technical thing. Time to share something I learnt during latest months.

My boss at ProFUSION always says there are only 2 debugging tools worth using: strace and valgrind. Indeed, they are very helpful while developing software in C. If you don’t know them, hurry! Google it. Basically what I can do is to summarize their functionality:

strace traces all system calls a program does. You don’t even need the source code. Suppose a software is misbehaving and you want to know what files it is opening: all you have to do is to intercept calls to your kernel, i.e. trace all open syscalls. You have to do as following:

bash $ strace -e trace=open ./your-program-to-trace

This will output all the system calls of type open, with their parameters;
valgrind helps you to debug problems related to memory accesses. It’s a common mistake to free an alloc’d variable and after trying to access its content. Valgrind will tell you exactly where you freed the memory an then you may catch the bug much more easily

However I don’t think gdb is not worth using and I’ll show here it’s so powerful that you can even simulate strace with gdb. Let’s say you want to do the same thing as above with strace. You’ll need to use the ‘catch’ command in gdb:

 (gdb) catch syscall open

Done! Your program will stop for every open syscall. But wait, strace does not break, it just prints the syscall and continues exection. Let me continue… One of the features gdb has that I see really few people using is commands. Hey! GDB is not a tool made only for seeing the backtrace when a program reaches a certain line in your source code! You can automate several things, print memory locations, break if a variable assumes a certain value etc. So, if you just want to print something, you can improve our previous example:

(gdb) catch syscall open
(gdb) commands 1
> bt
> continue
> end

Now it will print the backtrace every time an open syscall is issued. If you pay attention to the calling convention of your architecture[1] [2], you could check the parameters to that syscall and print its value or stop if it is something interesting. For example, in x86_64 the following snippet would print the filename being opened:

(gdb) catch syscall open
(gdb) commands 1
> silent
> x/s $rdi
> continue
> end

Yes, you could use strace and grep for the file you are interested in. But, where’s the fun ;-)? The thing is that all these debugging tools rely on ptrace syscall, that allows a process to trace another one. So, choose your debugging tool and go fix your code :-)

Brasil x Holanda

2010-07-02T23:37:00-07:00

Brasil perdeu. Infelizmente estamos fora da copa.

O Juca Kfouri resumiu praticamente tudo o que eu acho desse jogo. Só pra complementar: o time de laranja era uma equipe de teatro ou de bonecas?

Now, officially an EFL developer

2010-05-05T02:36:00-07:00

Last week I was officially added to EFL developers list. After contributing some patches to eina, edbus, elementary and E17 (especially to connman module), Gustavo Barbieri, who is also my boss at ProFUSION, added me to developers list giving me commit rights on EFL svn.

He said me some weeks ago that the only thing missing to add me as developer was that I’d have to use E as my window manager. Fair enough. If one wants to be a developer of a certain program, it’s better to first be an active user. So, last Friday I wiped out my Gnome and started using E17 as my default window manager. If you don’t know E, you should check its site. It’s a lightweight Desktop Environment designed on top of EFL, the Enlightnement Foundation Libraries.

I don’t think the current theme is that good, but I’m getting used to it. Hopefully, it will be replaced some time soon. Below, my current desktop with an animated wallpaper with a clock:

[caption id=”attachment_266” align=”aligncenter” width=”553” caption=”Enlightenment”][/caption]

Linux Collaboration Summit 2010

2010-04-21T22:29:00-07:00

And this was the Linux Collaboration Summit 2010:

Linux Collaboration Summit 2010 from Kenny Moy on Vimeo.

I’m appearing on this video exactly at 2:00 :-D. After it shows Google giving Nexus One phones to everybody. Really nice.

Embedded Linux Conference 2010

2010-04-15T09:17:00-07:00

So, today was the last day of the Embedded Linux Conference. Now, I’ll be here in USA for the Linux Collaboration Summit. It was really cool and it was amazing to meet people you are only used to chat, exchange some emails or that you heard about. Just to name some: Steven Rostedt, Greg Kroah-Hartman, Frank Rowand, Mike Anderson, Andrew Morton, Jon Corbet and others. I was really impressed too seeing Rostedt programming or Greg answering emails. What a great guys

And yesterday I presented my work, talking about the optimization of the Linux scheduler for soft real-time when running on multi-core architectures. I must admit I was a bit nervous, but it seems that people liked it. Following some pictures:

Banco do Brasil + Linux 64 bits

2010-03-30T03:06:00-07:00

Faz 1 mês mais ou menos que liguei no Banco do Brasil reclamando que a “solução de segurança” do site deles não estava funcionando no meu Linux 64 bits. Sem grandes surpresas, depois de me perguntarem se eu tinha a VM java da sun instalada, versão, etc, eles me passaram para o suporte de segundo nível. Pediram o meu telefone, dizendo que entrariam em contato (isso foi no sábado).

Logo na segunda-feira de manhã me ligaram e após mais um tempo conversando, ele disse: “o problema é que a nossa solução de segurança não é homologada para a jre 1.6. Acontece que em algumas plataformas ela funciona, mas não é garantido. A gente sabe que em Linux 32 bits funciona, mas a jre 1.5 é que é oficialmente homologada”. Então eu tinha duas opções: ou instalar a versão 1.5 ou rodar um browser 32 bits. Felizmente no meu sistema, archlinux, já tem um pacote firefox32 que faz tudo o que preciso e foi o a solução que adotei. Até hoje.

Hoje quando fiz meu usual “sudo pacman -Syu” para instalar atualizações de todos os pacotes, vi que tinha uma atualização da jre para ser feita: jre-6u18-2. Aí pensei… quem sabe nessa versão mudou alguma coisa que fez funcionar o applet do BB? Sem muitas esperanças, abri o site no chromium e para minha surpresa, funcionou! Abri no Firefox, e funcionou também. Pra tirar a dúvida, fiz o downgrade para a versão anterior do java, jre-6u17-1. Abri novamente e continuou funcionando. Logo, não foi causada pela atualização da VM, mas sim por alguma mudança que o pessoal do Banco do Brasil fez. Não sei se eles chegaram a homologar a versão 1.6 da JVM (o que já estaria mais do que na hora), só sei que agora funciona com Linux 64 bits.

Felizmente agora posso acessar a minha conta no Banco do Brasil sem precisar do browser 32 bits. Já estou removendo-o. Deixe seu comentário pra eu saber se funcionou pra você também (inclusive aqueles que usam FreeBSDs e outros unixes).

Amazing kernel related blog

2010-03-23T03:45:00-07:00

Today while reading “posts” on Buzz and digging in my RSS reader, I found this blog: http://smackerelofopinion.blogspot.com/ made by an ubuntu engineer. How many amazing posts there are on this blog. Particularly interesting is the one about ACPI Debugging and the other about perf, the tool I posted about some days ago (although it covers just a tiny amount of things this tool is capable of).

When I have some spare time I’ll try to use that first post to debug the ACPI on my buggy asus laptop, where temperature goes sometimes as high as 90 °C. Last year I tried to adjust the hysteresis curve of the fan control in order to have a cooler laptop but had no success on it.

PS.: I wished a day had 48h to be able to read all the cool stuff out there

Pescaria 2010

2010-03-21T21:33:00-07:00

Final de semana pra relaxar com a família. Seguindo a sugestão de Ataíde e Alexandre, fomos pescar. Que pena que o meu peixe de 12kg acabou escapando. Algumas fotos:

[caption id=”” align=”aligncenter” width=”640” caption=”Respiração”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”Peixão”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”Meu pai pegando um peixe”][/caption]

Mais fotos em: http://picasaweb.google.com.br/lucas.de.marchi/Pescaria?feat=directlink

Stripping perf out from kernel source

2010-03-10T14:19:00-08:00

You may have heard about perf, a tool made by Linux kernel developers to help debugging, benchmarking and profiling.

The problem is you have to download the whole kernel source in order to compile perf. If you are not used to follow kernel developping, most likely you will not have its source. But it’s still possible to compile perf.

As I’m helping to maintain perf package in Archlinux, I made the following script to strip perf from a given kernel version:

#!/bin/bash

function usage() {
    cat - <
This script works fine with latest 2.6.33 kernel. Regularly I'm uploading to this site the stripped version of perf. The latest one, you find here.

Mais um infeliz cliente da Telefônica

2010-02-25T05:01:00-08:00

É impressionante como a Telefônica pode ser uma operadora tão ruim. Não bastasse ser ruim e cara, você fica preso a ela por diversos motivos. Explico: tentando diminuir os gastos da linha telefônica, hoje liguei lá para contratar uma segunda linha telefônica, mas do tipo Linha Lazer. A intenção era começar a usar essa segunda linha só para o speedy e depois migrar (usando a portabilidade numérica) a outra linha para uma operadora melhor (e mais barata, muito mais barata).

A diferença de assinatura entre a Linha Clássica, que temos agora, contra a Linha Lazer: R\$ 40,35 contra R\$ 14,90. E todos os outros serviços (em ambas) são mais caros do que no Livre.

Depois de passar todos os documentos pedidos, repetir e repetir mais uma vez, a atendente tentou empurrar a Linha Clássica pra mim. Expliquei que queria a Linha Lazer, e não a Clássica. Sabendo que a gente já tinha uma outra linha (ela tinha perguntado antes), disse que a Linha Lazer não estava disponível para a minha região. Mentira! Aposto que se não tivéssemos nenhuma linha não teríamos problema em contratá-la.

Graças à magnífica privatização do setor de telefonia no Brasil, ela é a única provedora de internet banda (quase) larga por aqui. Não fosse isso, eu já teria cancelado minha linha. Sendo isso, penso seriamente em cancelá-la de qualquer jeito e ficar sem internet mesmo, pelo menos por um tempo.

Alguém tem sugestão?

PS.: Não é que descobri que tem gente na mesma situação que eu?

Coming back, with git tips

2010-02-10T23:52:00-08:00

So, it has been some months I last wrote on my blog. Not because I don’t have things to write, but because sometimes I’m too lazy do empty my “queue of things to write about”. Nothing better than start with one thing at a time.

Since january I’m working at ProFUSION. As before when I was mainly developing for Linux kernel and finishing my thesis (@ USP / Polimi), I’m using git a lot. Even in projects in which upstream is using a centralized approach like subversion, I simply can’t live without a local tree and thus, git. So here I’ll post somethings I’ve been using:

1. Incrementally add your changes to commit

One thing that is essential to keep your tree clean is to commit each new feature in a different commit and write a proper description of what you are doing and perhaps why you are doing that (often in case of a bug fix). But, sometimes you don’t code “atomically” several logically separated parts. Instead, you may prefer to code a lot (whilst you still have a great amount of caffeine in blood) and separate them when you finish. Here comes the savior command:

$ git add --patch file-to-partially-add-to-commit

So you will be presented to a series of options where you can choose which parts are going to next commit and the ones that will remain on unstaged area.

See The Thing About Git for more details.

2. Using gmail to receive patches

While our googler friends do not implement a proper way to export emails as mbox, with this tip we can circunvent the problem without relying on standalone clients such as mutt. Steps:

Create a ‘patch’ label in Gmail (or whatever name you want).
Install fetchmail and procmail from your distro repository:
- Archlinux: sudo pacman -S fetchmail procmail
- Gentoo: sudo emerge …
- Ubuntu: sudo apt-get install …
Configure fetchmail to use your google account. Put this in your \$HOME/.fetchmailrc

set postmaster "YOUR_LINUX_USER"
set bouncemail
set no spambounce
set softbounce
set properties ""
set no showdots

poll imap.gmail.com with proto IMAP
user 'YOUR_EMAIL@gmail.com' there with password  'YOUR_PASS' is 'YOUR_LINUX_USER' here
options keep fetchall ssl mda "formail -c >> /PATH/TO/SAVE/patches.mbox"
folder patch

Workflow

Find patches to be applied in gmail;
Mark them with ‘patch’ label;
Use fetchmail to download patches:
- bash $ fetchmail
Now you can happily apply patches saved in /PATH/TO/SAVE/patches.mbox to your project with:
- bash $ git am /PATH/TO/SAVE/patches.mbox

Parallelizing

2009-12-09T19:00:00-08:00

Sometimes we discover that we don’t have to parallelize only the software. We must also parallelize the way it is written.

Thus, one of the last days of my graduation in Computer Engineering:

[caption id=”attachment_220” align=”aligncenter” width=”560” caption=”One day of hard work”][/caption]

Almost finished!

Welcome Windows 7

2009-10-24T04:36:00-07:00

Welcome from a Linux user…

Python, bpython and my university’s email

2009-10-21T02:15:00-07:00

Almost a year ago I posted in my past blog (in portuguese, automated translation) a tutorial on how to use greasemonkey to workaround the ugly interface of my university’s webmail. More specifically, in Firefox it looses the ability to select all emails. I took that just for an example… the real goal was to delete all emails since I access them through Gmail and periodically I get the email: “your mailbox is full…”. So, how could I delete them without having to download them all? Simple: connect through POP3 and issue the “dele” command. I could do that even with plain telnet, but I decided to come back to python, since it was awhile I didn’t use it.

In the same week I saw two interesting posts in my RSS reader:

the first came from BR-Linux, and it’s about bpython, a better CLI to python. It’s amazing!! You can get a lot of IDE-like functionalities in your command line. You can view a short video made by its author.
The second is about pydev and came from “Linux Today”. With PyDev you can use Eclipse to edit your python code and get all functionalities you may out of an IDE. I still didn’t use it (and I think I’ll continue with vim) but it’s an interesting post to read as it talks about IDE vs vim/eclipse/other_text_editors.

And now, the really-stupid-delete-all-mail-from-poli.py:

#!/usr/bin/python

import getpass, poplib

strServer = 'pop3.poli.usp.br'
user = raw_input('user: ')
server = poplib.POP3(strServer)
server.user(user)
server.pass_(getpass.getpass())
numMsg = len(server.list()[1])
while True:
    resp = raw_input('%d messages in Inbox. Delete all? (yes/no) ' % numMsg)
    if resp == 'yes':
        for i in range(1,numMsg+1):
            server.dele(i)
        break
    elif resp == 'no':
        break
    else:
        print 'Answer yes or no\n'

server.quit()

It will ask the user and password to connect to pop3.poli.usp.br, show the current number of messages and ask for confirmation.

Trissa - update status

2009-10-14T16:26:00-07:00

Just to let you all know some updates to Trissa: currently it’s in a hold state, waiting for next stable Ogre release, which is version 1.7. So until they release this version I’ll hold Trissa in a beta status, although it’s working great right now and all fixes for 1.0 are in place.

The biggest problem with current version of Ogre is that it doesn’t have proper support for multi-threaded games, or better, only one thread is able to make calls to Ogre functions which change the rendering states. In the design of Trissa this is not currently possible as both core’s and player’s code may change the rendering.

Also, I’d like to clarify a thing about the name: in contrary to what I said, “trissa” does not mean “tic-tac-toe” in italian. The right name in italian is “tris” as a friend of mine told me some months ago. However I decided to keep the name, because I liked it, it’s now properly indexed by Google and the game is not the original tic-tac-toe.

TinyOS

2009-10-07T01:32:00-07:00

As I did with previous projects I had at my university, I’d like to share another one: it’s a project using TinyOS. It’s mainly intended for education purposes, so if you are trying to learn TinyOS, it’s a good example to look at.

What does it do?

It’s a platform to monitor temperature and humidity of various rooms in a house. All sensors must collect data every X seconds and send them to sink, a predefined node. Sink can also change the value X and sensors may not reach sink in a single hop, so nodes have to use their own routing protocol to send forward data from other nodes to sink.

If you are interested in this project, send me an email and so I could help you to understand it. I think it’s a good exercise to look at source code and discover which routing protocol I was talking about (before reading the README file).

As I think I won’t modify this code anymore, rather than put it in my git repository I made a .tar.bz2 which is available here: House Monitor source code.

Testando o google translate com o último post

2009-10-02T13:57:00-07:00

Escrevi o último post em italiano e queria testar o Google Translate. Aqui está o link do último post traduzido para o Português.

Não digo que foi ruim para um tradutor automático. Em geral creio que dê para entender, mas algumas partes foram bem divertentes. Um exemplo é a frase “Eu e meu amigo fomos para Mateus Pomatt”, tradução de “Io e il mio amico Matteo siamo andati a Val Formazza”. Perdeu-se todo o sentido, que seria: Eu e meu amigo Matteo fomos a Val Formazza.

De qualquer forma achei legal poder manter a forma do site só traduzindo-o com o Google Translate. Antes quando precisava de uma tradução eu copiava e colava o trecho que queria traduzir.

Val Formazza!

2009-10-02T13:44:00-07:00

E’ da un po’ di tempo che non scrivo niente sul blog, ma è perché era un po’ preso con le cose dell’università. Spero poter aggiornarlo più spesso ora.

Questo post è sulla passeggiata che ho fatto un mese fa. Io e il mio amico Matteo siamo andati a Val Formazza. E’ stato davvero spettacolare. Non posso immaginare una persona ritornare dall’Italia senza essere andato al meno una volta in montagna. È ancora più difficile da capire che tanti italiani non siano mai andati. Me lo raccomando a tutti!

Vi lascio qualche foto.

[caption id=”” align=”aligncenter” width=”640” caption=”Prima foto a Val Formazza”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”Spettacolare!”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”La marmotta, che pochi sono riusciti a vedere così vicina”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”Io e Matteo”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”Camminando sopra un ghiacciaio”][/caption]

[caption id=”” align=”aligncenter” width=”640” caption=”La discesa”][/caption]

Sono tante foto belle. È difficile da scegliere quale mettere qui. Allora vi lascio anche il link dell’album se voleste guardare altre: Montagna.

L’ultima foto che vorrei lasciare qui è della famiglia di Matteo. Vorrei anche ringraziare loro per avermi accolti così bene. Peccato che mancano un suo fratello e la sua sorella.

[caption id=”” align=”aligncenter” width=”640” caption=”Matteo’s family”][/caption]

Una frase che sentivo spesso in Italia era “beati i brasiliani per avere le foreste, le spiagie, ecc…”. Io però vi dico: beati gli italiani per avere, tra altre cose, le montagne bellissime!

Countdown restarted

2009-09-15T12:00:00-07:00

Por motivos de força maior, a contagem regressiva foi reiniciada. Nova data: 27 de setembro.

12 dias, e contando…

Suse studio

2009-08-16T00:49:00-07:00

I was really impressed about the demos I saw in last Novell’s announcement : Suse Studio. So last week I applied to have an account since it’s not open for everybody. Today I received the following email:

Thank you for your interest in SUSE Studio, a new service that makes
it quick and easy to create and test software appliances from your web
browser. We are pleased to invite you to try SUSE Studio.

To accept this invitation and take SUSE Studio for a spin, please
visit this link:

****************************************

Alternatively you can go to http://www.susestudio.com/account/invitation and manually
enter your invitation code there. Your invitation code is:

**********************

We hope that you will enjoy using SUSE Studio. If you encounter bugs,
have general feedback or want to discuss SUSE Studio you can reach us
at feedback@susestudio.com.

Have fun using SUSE Studio!

Best Regards,
The SUSE Studio Team

Woww… amazing!! I’ll try it out and let you know what I think of it in a few weeks. In the meantime you can see some screencasts on their website or on youtube.

Do you have an account?

The final countdown

2009-08-15T16:03:00-07:00

Em 30 dias estarei pousando no Brasil!!

A contagem final começou.

Preparem a picanha que eu estou chegando.

Compiler’s compiler version

2009-08-15T02:06:00-07:00

Today I was just wondering… what’s the version of the compiler which compiled my compiler. Quite a strange question to make myself and I really don’t know where this curiosity came from.

Looking in Wikipedia:

Early compilers were written in assembly language. The first self-hosting compiler — capable of compiling its own source code in a high-level language — was created for Lisp by Tim Hart and Mike Levin at MIT in 1962.^[2]^ Since the 1970s it has become common practice to implement a compiler in the language it compiles, although both Pascal and C have been popular choices for implementation language. Building a self-hosting compiler is a bootstrapping problem — the first such compiler for a language must be compiled either by a compiler written in a different language, or (as in Hart and Levin’s Lisp compiler) compiled by running the compiler in an interpreter.

Interesting, don’t you think? So let’s see the version of your compiler’s compiler. If you use GCC, it will put a comment in section named (surprise!) .comment. Generate the assembly correspondent to a C source code and you are going to see in the end of the file an entry like this:

.size main, .-main
.ident “GCC: (GNU) 4.4.1”
.section .note.GNU-stack,”“,@progbits

So, let’s play with our already compiled compiler. First we have to check the compiler version:

[lucas@skywalker tmp]\$ gcc -v
Using built-in specs.
Target: i686-pc-linux-gnu
Configured with: ../configure —prefix=/usr —enable-shared —enable-languages=c,c++,fortran,objc,obj-c++ —enable-threads=posix —mandir=/usr/share/man —infodir=/usr/share/info —enable-__cxa_atexit —disable-multilib —libdir=/usr/lib —libexecdir=/usr/lib —enable-clocale=gnu —disable-libstdcxx-pch —with-tune=generic
Thread model: posix
gcc version 4.4.1 (GCC)

Ok! Version 4.4.1. Let’s use the readelf command to see the content of .comment section:

[lucas@skywalker tmp]\$ readelf -p .comment /usr/bin/gcc

String dump of section ‘.comment’:
[     1] GCC: (GNU) 4.4.0 20090630 (prerelease)
[    29] GCC: (GNU) 4.4.0 20090630 (prerelease)
[    51] GCC: (GNU) 4.4.1

[    63] GCC: (GNU) 4.4.1

(…)

[   237] GCC: (GNU) 4.4.1
[   249] GCC: (GNU) 4.4.1
[   25b] GCC: (GNU) 4.4.0 20090630 (prerelease)
[   283] GCC: (GNU) 4.4.1
[   295] GCC: (GNU) 4.4.0 20090630 (prerelease)

I didn’t understand if it’s 4.4.1 or 4.4.0, i.e. if it was used a prior version to compile the current version or if current was recompiled afterwards with this new compiler produced. Testing random binaries in /usr/bin seems to produce similar effects, having more than one version.

So… no answers yet. Any clues?

Trissa 3D is arriving

2009-07-29T21:59:00-07:00

I’m proud to release version 0.98 of one of my projects, Trissa. Whilst this is not a version intended for the end-user, I’d like to have some more users to find bugs before releasing 1.0. Be warned: this is intended for developers only. There’s no package yet, so you have to compile it yourself. Some screenshots:

I’m putting online also a presentation I gave about this project in my university, Politecnico di Milano. Trissa is the name of the Tic-tac-toe game in italian and I made it as part of a course: Laboratory of Artificial Intelligence and Robotics. Thanks to professor Francesco Amigoni who helped me. [ .odp / .pdf ]

Source code is available at Github. To get the last version with git:

git clone git://github.com/lucasdemarchi/trissa.git

You will need some libraries that may or may not be pointed out by the building system:

scons
OIS
Ogre3D
CEGUI
Boost

Let me know if you have any trouble to compile it and if you find any bugs. In “TODO” file there are some known bugs.

Awesome!

2009-07-13T17:34:00-07:00

Just a short post to share a beautiful Brazilian song. Two days ago my friend was playing it, now it just doesn’t get out of my head :-)

Have fun.

Sharing some homeworks

2009-07-11T23:01:00-07:00

So, I’ve spent some time last weeks doing some homeworks for courses at Politecnico di Milano. One of them is for the course “Image analysis and synthesis” in which I did together with a friend of mine some animations in Blender.

The goal was to make some animations of sport scenes and after apply a kind of “filter” in Matlab to simulate a high exposure of the camera, or, if you prefer, to draw the trace of the ball. As the only object moving in scene was the ball, these two are roughly the same thing. This is done by taking the mean of W frames and making a new one from them, where W is the “window” of the algorithm.

We took four sports: basketball, table tennis, golf and bowling. The amount of things produced is \~ 400MB, so I’m not putting it online for everyone. If anyone is interested in the Blender models, ask me privately (you find my email in the “About” link above). I put some animations in youtube, sou you can see them:
Bowling: http://www.youtube.com/watch?v=YrgkLCTrsVg
Table tennis: http://www.youtube.com/watch?v=HmrDPdDevRM
Basketball: http://www.youtube.com/watch?v=TjeOXhxceWA

3D animation is one of the areas I admire, but definitely isn’t an area I’m good at. Also, this was the first time I did something serious with blender, so don’t expect superb results (and continue reading this post if you gave up after seeing the videos ;-) ).

After thinking a while about the time Matlab took to make the final results I decided that it could be better… Two things could be improved: (i) use more threads to calculate the new images and (ii) use CPU optimized CPU instructions. I think it’s a shame to have a dual core with lot of specialized instructions to do things faster and don’t use them. These two goals brought me to code the algorithm in C.

I used pthreads to divide the work among threads (you say at command line the number of threads you want, or just say it to use the number of available processors) and freeimage to load and save images. I did made another version that uses gdk-pixbuf to load and store images and it took me so disappointed. To simply load and save images DIFFERENT images (so, things that are not correlated) it needs to use synchronization mechanisms. This leads to an algorithm not scaling very well as it could. I also tested another one with CImg… and the result was so disappointing too. I think not every developer is performance-ish as I am. Or not everyone cares about it as long as it is easy to use. So, stop talking and show the results:

[caption id=”attachment_125” align=”aligncenter” width=”640” caption=”Time to compute in each implementation”][/caption]

The best version run in less than half of the time required by Matlab! I could also optimize more the algorithm in C to get even better results, but I was already ok with that (read: I have other things to do). It’s also worth noting the improvement when optimizing the code for my CPU: with one thread I got a boost of \~20% and with 2 threads of \~18%. Particularly in this type of algorithm, it’s important to use SSE/SSE2 instructions to get a faster mean of the images. In fact, viewing the assembly code generated in this case we find these instructions as below (the xorps, movaps are SSE instructions that operate with xmmN registers).

 8049517:       0f 57 f6                xorps  %xmm6,%xmm6
 804951a:       8b 41 04                mov    0x4(%ecx),%eax
 804951d:       8b 4d 08                mov    0x8(%ebp),%ecx
 8049520:       39 41 08                cmp    %eax,0x8(%ecx)
 8049523:       0f 82 36 06 00 00       jb     8049b5f <worker_thread+0x6ff>
 8049529:       8d 7d cc                lea    -0x34(%ebp),%edi
 804952c:       0f 29 b5 08 ff ff ff    movaps %xmm6,-0xf8(%ebp)
 8049533:       89 bd 04 ff ff ff       mov    %edi,-0xfc(%ebp)
 8049539:       89 3c 24                mov    %edi,(%esp)
 804953c:       89 44 24 08             mov    %eax,0x8(%esp)
 8049540:       c7 44 24 04 64 a2 04    movl   $0x804a264,0x4(%esp

Awesome!! GCC automatically generated a SSE optimized code. If you got interested, see the wikipedia article about SSE.

The code is available under GPLv2 license at github: http://github.com/lucasdemarchi/imgmean. Let me know if it’s useful to you.

The browser war continues…

2009-07-04T11:55:00-07:00

One month ago I wrote a post talking about the new Chrome for linux, or better yet, that the Chromium (this one is opensource) is finally working well under Linux, and so fast.

Well, while the Chromium guys are working hard to get a decent browser, the firefox guys didn’t stop working and last week Mozilla released the so waited Firefox 3.5.

My 2 biggest complaints about the prior version were that (i) the awesome bar gets slow as time pass and you visit a lot of sites and (ii) the start up time. The first one I learned that can be solved still in that release optimizing the database of the bar. It uses SQLite and a cleaning is all it needs after a while. The second problem was solved installing the new version. Its speed is not as amazing as Chromium but it’s fast enough.

The only plugin that I had that is not updated yet to this new version is Google Gears. But frankly, I’m thinking in disabling it. Although it’s amazing to access my Gmail while I’m offline, I think this plugin is heavier than it could be.

So, for now I’m happy with my new Firefox, but Chromium is still installed and I update it often. Ohh, talking about it, my issue with ‘dead keys’ in Chromium is gone. Now I can write portuguese, english and italian always with the same keyboard configuration.

Returning to Firefox, another feature I liked so much is the new support to HTML5’s video tag. I hope soon there will be no more need to use flash plugin. I think any other browser support it yet, therefore only my friends who use Firefox 3.5 will be able to see the promotion videos below. If you didn’t upgraded yet, it’s a good time to do it.

What’s new:

Your browser doesn’t support this feature yet. Update to a better one: [Firefox 3.5](http://www.getfirefox.net)

Nice promotion video @ dailymotion. There’s even Linus Torvalds on this video (@ 7s) and it was nice to see people saying “obrigado” (portuguese word to “thanks”), but I didn’t recognize where they are from. Any clue?

Your browser doesn’t support this feature yet. Update to a better one: [Firefox 3.5](http://www.getfirefox.net)

You can see more open videos at dailymotion.

Happy browsing.

Benvinda de volta Oficina G3

2009-06-28T14:35:00-07:00

Creio que nem todos conheçam a banda Oficina G3. Em poucas palavras, é uma banda evangélica de rock muito boa.

O rock pesado nunca foi meu estilo favorito. Passei a ouvir Oficina principalmente quando lá pelo ano 2000 (acho) eles partiram para um estilo mais light com CDs como “Acústico” e “Ao Vivo no Olímpia”. O último CD deles que ouvi bastante foi o “Humanos” quando eles voltaram para o rock que apesar de pesado, era ainda “ouvível”.

Com a saída do vocalista PG do grupo eles deram uma “recaída” e depois de algum tempo sem lançar nenhum CD lançaram o “Além do que os olhos podem ver”. Que decepção! Rock terrivelmente pesado e muito longe do meu gosto.

Bom… esse post é pra falar que, navegando esses dias na internet descobri que eles foram eleitos a melhor banda de rock gospel no “Troféu Talento 2009”. Aí resolvi procurar qual tinha sido o último CD deles e descobri que tinha um novo: “Depois da Guerra”. E que maravilha de CD! Continua no estilo pesado, mas muito, muito melhor do que o anterior que tinha ouvido. As letras não são tão bonitas e poéticas como as do João Alexandre (um dos meus favoritos tanto na parte da letra quanto musicalmente), mas são deveras boas.

Dou destaque para a música “Continuar” que você pode ouvir abaixo, uma das mais calmas do CD.

Descobri também que quando estou programando, ouvir uma música mais agitada como essas me deixa mais acordado - razão pela qual Oficina G3 voltou para a minha playlist e tenho ouvido esse CD várias vezes durante o dia.

Pergunta para meus amigos programadores: vocês escutam alguma música enquato programam? O que?

Meu Brasil brasileiro

2009-06-22T01:00:00-07:00

Questo post è dedicato ai miei amici italiani. Mi dispiace, ma l’Italia ha giocato troppo male e il Brasile… beh, è il Brasile!!

Sentite la differenza tra il cronista brasiliano e l’italiano.

Italiano:

Brasiliano:

Ci vediamo domani. Io sarò con la maglietta della nazionale brasiliana ;-)

Chromium

2009-06-05T22:57:00-07:00

After reading this post at “Linux Today” I decided to install the so expected Linux version of Chrome, the Google’s browser.

As I use Arch Linux, I haven’t expected to have a compiled version linked directly by the chromium’s website. Instead, I was hoping Arch’s developers already packaged it. And, with no big surprises, they did. Just to clarify a thing before continuing: Chromium is the open source projected behind Chrome (which is not opensource).

I’m not writing this post to repeat what’s already written there. So read it too. As opposed to his impression however, I’m running it without big problems: pages are rendered well formatted, without html or css problems. Sure, it’s not a browser to use everyday yet, but as a pre-alpha version it’s doing pretty good. Below, a picture with this post been written.

[caption id=”attachment_92” align=”aligncenter” width=”553” caption=”Using Chromium to post on my blog”][/caption]

Not everything works for me: I was not able to read my emails on Gmail, since there’s one error on getting Google’s certificate (see “UPDATE” below); my keyboard with deadkeys doesn’t work well, I had to switch to a configuration with no dead keys in order to this post; a bunch of other things are incomplete or missing (for example in the options window) and there are some other issues as can be seen here.

Ohhh, and I almost forgot to say: yes, it’s fast… sooo fast. When I first opened it the first word I said was: “owwwwhhh”. Also to render the webpages I noted a speedup comparing it to Firefox (Swiftfox). Let’s see if it will continue to be fast like this when it gets bloated with a lot of extensions. One architectural difference among Chromium and Firefox I noted is that every tab is a different process. Whether this is good or not, I don’t know yet… We have to wait the next versions to make a fair comparison.

UPDATE:

Today I upgraded to version 3.0.184.0 (17842) and gmail is working! The problem with the keyboard remains

Saudade…

2009-05-28T21:04:00-07:00

No final do ano passado meu amigo Fadinha Ricardo publicou um post em seu blog falando sobre saudade. Concordo com o sua afirmação. Como escrevi nos comentários daquele post, em 2004 foi feita uma sondagem a tradutores profissionais por uma empresa britânica e a palavra “saudade” do português foi eleita como uma das mais difíceis de se traduzir.

O bom porém da saudade é quando você mata ela. Que sensação boa!!

Resolvi tocar nesse assunto porque meu irmão veio me visitar. Nos últimos dois finais de semana pude estar com ele e dar uma passeada pela Itália. Tantos lugares, tantas emoções, tantas lembranças… enfim, nos divertimo à beça. A lista de lugares por onde passamos seria longa de mais pra contar, então vou deixar somente os nomes das cidades (os nomes que eu sei em português sem ficar procurando no google eu coloquei em português, caso contrário deixei em italiano):

Milão, Como, Turim e Monza no primeiro final de semana.

[caption id=”” align=”aligncenter” width=”288” caption=”Eu e meu irmão… A cidade de Como lá em baixo”][/caption]

No segundo final de semana (em itálico as cidades pelas quais só passamos brevemente): Verona, Padova, Rovigo, Badia, Giaciano con Baruchela, Castelmassa, Ferrara, Bologna, Firenze, Pisa. Na última ficamos em um camping muito bom e depois nos separamos: eu voltei pra Milão e ele continuou sua exploração turística da Itália indo pra Lucca.

Neste exato momento o avião dele está partindo de Roma e aposto que está morrendo de vontade de mostrar as fotos… Vou dar uma de spoiler e adiantar algumas.

+———————————————————+———————————————————+ |

Apesar de morar em Milão há cerca de 1 ano e meio, nunca tinha subido em cima do Duomo. Fiquei pensando que tem coisas que a gente faz só como turista e muitas vezes não aproveita a cidade aonde vive. Dois exemplos: 1. Morei na cidade de São Paulo por 4 anos e não visitei quase nenhum ponto turístico; 2. Conheço pessoas que moram/moravam em Paris e nunca subiram na torre Eiffel. Sendo guia turístico do meu irmão eu finalmente pude subir no Duomo (ainda falta eu ver o afresco “A última Ceia”, já que estava lotado e não pudemos ver).

[caption id=”” align=”aligncenter” width=”216” caption=”Em cima do Duomo de Milão”][/caption]

Sábado à noite, jantar com os brasileiros (e “agregados”). Acho que deu pra ele conhecer como é o dia-a-dia/final de semana dos estudantes brasileiros na Itália.

[caption id=”” align=”aligncenter” width=”288” caption=”Jantar”][/caption]

Também fizemos um passeio de barco no belíssimo Lago de Como.

[caption id=”” align=”aligncenter” width=”288” caption=”Lago de Como”][/caption]

Antes que me perguntem “o que vocês foram fazer em cidadezinhas não-turísticas como Badia, Giaciano con Baruchela, etc?”. Foi em Giaciano con Baruchela que meu bisavô nasceu. Foi emocionante ver o livro com seu registro de nascimento em 1874!! Mas as fotos daí pra frente ficaram só com ele… vou ter que esperar ele chegar no Brasil.

Pra terminar, volto ao título desse post: saudade é ruim, mas quando a gente mata é bom de mais. Boa viagem, Rafa.

E a contagem regressiva pra volta pro Brasil começou!

Playing with your cache

2009-05-20T21:00:00-07:00

Another possible titles could be “how to really slow down your computer” or “why caches are so important in computer architectures”. I started playing with turning my cache on and off last week using Linux because there are some situations in which you have to know why a piece of software is not working as expected. A possible problem could be the well known cache trashing, in which the contents of the cache is thrown away very often. Turning it down may give an answer if this is really the problem since now even running very slowly this variable is eliminated.

For example in my stage we had some tests showing that in certain scenarios a quad-core machine is much slower than an equivalent single or dual core. Next post I’ll show how to play with your cores, activating and deactivating them, so you may create your own programs and test them against a 1, 2, 3,… cores machine.

Also I think it’s a good exercise to students of computer science/engineering who are enrolled in courses as “computer architectures” and “operating systems”. For those, two good books: “Modern Operating Systems” by Tanenbaum and “Computer Architecture: A Quantitative Approach” by Hennessy and Patterson.

In an Intel processor there are 3 ways to disable the cache (see section 10.5 of [1] for more details):

Setting and unsetting the CD bit in register CR0;
Using page-level cache control flags;
Using MTRR (Memory Type Range Registers);

Since it’s possible to play with disabling and enabling the caches only in kernel mode and inside Linux MTRRs are already exported to user-space through the /proc/mtrr pseudo-file, the third method is the easiest and the one I chose to demonstrate here.

Verifying the availability of MTRRs

Before using MTRRs it’s important to see if the target processor support it. You can confirm it searching for a mtrr flag int the /proc/cpuinfo pseudo-file, which when read uses the CPUID assembly instruction to discover the processor’s capabilities. The following code shows an example.

[lucas@skywalker]$ cat /proc/cpuinfo | grep flags flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm ida tpr_shadow vnmi flexpriority

Other two tools that can be used to see details of the processor: dmidecode[3] (must run as root) and the stand-alone cpuid[4] program. Neither of them come installed by default on most distributions, but they give a lot of information about the processor which turn them worth installing. The first parses a table named SMIBIOS (some say DMI) reported by the BIOS while the second uses the CPUID instruction to query the processor.

Reading MTRRs

After verifying you have a supported CPU, the current way of accessing the caches can be seen reading the file /proc/mtrr. Example:

[lucas@skywalker ~]$ cat /proc/mtrr reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back reg01: base=0x0e0000000 ( 3584MB), size= 128MB, count=1: write-combining

The first field indicates the rule’s number. It’s used for deleting or changing a rule afterwards. Base is the start memory address, size is (sic) the size of the region in memory, count is the number of regions with this size, and the last field is the current “policy” of the cache.

In this example all the physical RAM (2 GB) is accessed using a write-back mechanism for caches. The second rule references to video memory and is automatically configured by the X server. The following code is a piece of <linux_source>/arch/x86/include/mtrr.h containing the available mechanisms for cache accesses. It can also be verified in table 10-8 of [1].

#define MTRR_TYPE_UNCACHABLE 0 #define MTRR_TYPE_WRCOMB 1 /*#define MTRR_TYPE_ 2*/ /*#define MTRR_TYPE_ 3*/ #define MTRR_TYPE_WRTHROUGH 4 #define MTRR_TYPE_WRPROT 5 #define MTRR_TYPE_WRBACK 6 #define MTRR_NUM_TYPES 7

Besides this way of obtaining MTRRs’ information, the /proc/mtrr pseudo-file has also an IOCTL interface (the one use above is named ASCII interface) that define some calls that could be used inside a program. [2] has some examples how this could be done. I stick with the first option to continue this short tutorial.

Writing MTRRs

You can create a new rule writting to /proc/mtrr as if it was a regular file, indicating the base, size and type fields. New rules are allowed to overlap existing ones, and the final result depends on the processor. For Intel processors there’s a precedence among types as below:

Uncachable > Write-combining > Write-through > Write-back.

Turning the caches off for a region in memory is just a matter of changing that region’s type to uncachable. You do so by writing the desired fields to /proc/mtrr. The following example creates a third rule setting all the memory to uncachable. Even if still exists a rule setting all memory to write-back, because of the precedence mentioned above the final result will be the application of this last rule.

[root@skywalker ~]# echo "base=0x00000000 size=0x80000000 type=uncachable" > /proc/mtrr [root@skywalker ~]# cat /proc/mtrr reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back reg01: base=0x0e0000000 ( 3584MB), size= 128MB, count=1: write-combining reg02: base=0x000000000 ( 0MB), size= 2048MB, count=1: uncachable

Congratulations!! Now your system will start to run really really slowly and the it will look a little irresponsive, even in the simple task of writing in the terminal to enable the caches again. Be patient an issue the following command to have your system back again:
[root@skywalker ~]# echo "disable=2" > /proc/mtrr
It uses the number of the rule you want to delete. In our case is the one we have just created. The new set of rules:

[root@skywalker ~]# cat /proc/mtrr reg00: base=0x000000000 ( 0MB), size= 2048MB, count=1: write-back reg01: base=0x0e0000000 ( 3584MB), size= 128MB, count=1: write-combining

Conclusion

Note you don’t have to set the whole memory to uncachable. You may set another base and size fields so to have just a piece of it as uncachable. Thus only programs that use this region would be running very slowly.

Although the applications of turning the caches off are really small, I think it’s very cool. There is one limitation however that I’d like to surpass but that is not possible with x86 (at least I didn’t find out a way to do it): to turn caches on and off independently of their level, that is, to have a level 1 cache turned off while still having it on on level 2 and vice versa.

[1] Intel 64 and IA-32 Architectures Software Developer’s Manual Volume 3A: System Programming Guide, Part 1. Intel. Available at: http://www.intel.com/Assets/PDF/manual/253668.pdf

[2] <linux_source>/Documentation/x86/mtrr.txt

[3] Dmidecode. Available at http://www.nongnu.org/dmidecode/

[4] Cpuid. Available at http://etallen.com/cpuid.html

Hello world - Again!

2009-05-06T17:06:00-07:00

I’m back again!!

Soon I’ll post some news and explain why I’m changing home.

Update your RSSs as I’ll not update my past blog anymore. So, as the title says:

HELLO WORLD!!

Driver radio STLC2590

2008-10-03T15:52:00-07:00

This is a re-post of a project I did. It’s still the same, no version upgrade. I’m putting it on this blog as I don’t update or track comments on the old one and people generally go there through google.

—

First of all I’d like to thank ST Microeletronics for give us the opportunity to work on a Linux driver for their chip STLC2590. All the documentation needed to write it was given by them. Thanks also to Patrick Bellasi, our tutor, which has tough us so much in regard to Linux driver development, APIs of ST, tools used in this project etc. At last, I’d like to thank William Fornaciari, which indicated us to ST so we could do this project as part of the course “Laboratory of Operating Systems and Software Design” at Politecnico di Milano.

This is a driver intended to be used mainly on Embedded Systems and SoC. I’ve never heard about a common PC using this chip within multimedia cards. So, all the instructions and documentation were made in the scope of crosscompiling it for ARM processors.

Here I publish the code under the GPLv2 License. I remember all of you that may use this driver that it’s dependent on the I2C API of STLinux (thought this limitation can be removed with few lines of code, but it requires knownledge of the vanilla I2C API). Also, I recommend to use the kernel and tools (for crosscompiling) from Poky Linux.

Current version: 0.0.12
Description and features:
- Radio driver for chip STLC2520 using V4L2 API.
- Configure the chip through I2C.
- RDS stuff is implemented. You have to use an userspace utility to get it working: rdsd.
- Kernel Linux 2.6.20 and tools for crosscompiling from Poky Linux. Patches to kernel from STLinux that include the I2C API used.
- Tools used for debugging and testing: rdsd and rdsquery, fmtools.

Still somethings to do:
- Remove the “exclusive_open” behavior (taken from V4L2) so more than one client at a time can connect to it (for example, one could control radio stations, volume, etc and the other the rds stuff, that it’d be the case of fmtools and rdsd working together).
- Review the sleep calls inside the code to render it faster and better.
- Review the need of locks on interrupt handlers of rds data.

Download the source code and documentation (in italian, as a final report for my course): stlc2590-0.0.12.tar.bz2.

If you want to know more about the driver or its development, I really encourage you to:
1) Read the documentation (if you are Italian please translate it - documentation is also under GPLv2 and the .odt and image files are given);
2) Contact me (either through the comments space or by email)
3) Hire me to work with you ;-)
As an academic work I think it can also be useful for those which are learning Linux and Kernel Programming. If you want to learn more you should get this driver as example and also read books (at least some chapters) like Linux Device Drivers (available online) and Advanced Linux Programming.

Enjoy it.