Woof progress
I have been quiet lately, as busy coding.I'm writing the hardest part of Woof, which is to rip apart the Ubuntu packages and create slim packages for Woof. The difficulty is that it has to be completely automated.
There are going to be three scripts, '1download', '2createpackages' and '3builddistro'. I'm currently writing the middle one, as it is the most difficult.
Very slow, heaps of fiddly details. I have to examine every package that gets processed, file-by-file, to see that a sensible woof package is getting created. I'm probably 3/4 done.
So, soon I'll write '3builddistro', then we shall see... I fully expect the live-CD to crash in a flaming heap, for all kinds of reasons, especially missing dependencies.
Anyway, here is something to look at, for anyone interested:
Nimrod
I have been scouting around checking out more Python-like languages, and found this:
http://nimrod.ethexor.com/
it's a true compiler. I compiled a couple of console apps, "hello world" was 82KB, but then I tested another much larger app and it also was a 82KB executable.
Nimrod has GTK2 bindings, but "Hello world" was 878KB. But, all the GTK2 libs were compiled statically -- very interesting, in the docs the author seems to prefer this so that GTK2 apps will work anywhere. I sent an email a few days ago asking howto/if I can change that to link all libs shared -- he hasn't yet replied.
No comments - Edit - Delete
4.1.2: .deb expansion bugfix
We have been discussing this recently on the forum. The utilities 'undeb' and 'dpkg-deb' are unable to expand some .deb files (Debian and Ubuntu packages). Note that 'undeb' is a script that calls 'dpkg-deb', and I personally prefer to use the latter directly.I found the reason is that some recent .deb packages use LZMA compression. A more recent 'dpkg-deb' is required, but it internally calls the 'lzma' utility -- but once again the one available as a Puppy .pet package is inadequate -- 'dpkg-deb' calls 'lzma' with commandline options that the latter does not understand.
I have upgraded dpkg-deb from version 1.10.18 to 1.14.20, and got the binary executable from a Ubuntu Intrepid repository.
The Puppy4 repo has 'lzma-458', but I got version 4.32.0beta3 from the Ubuntu Intrepid repository and upgraded the Puppy package.
How to manually upgrade Unleashed 4.1.1:
1. Download the new 'dpkg-deb' and 'lzma' from here: http://puppylinux.com/test/4.1.2/
2. Update the packages: puppy-unleashed/packages/dpkg-deb-1.14.20 and lzma
3. Update the entries in puppy-unleashed/packages.txt -- change the version numbers and the lzma entry to "on".
Of course, if you are running Puppy right now, any recent version, just grab those utilities from the above URL and substitute them in place of the originals. Then you'll be immediately ok to handle all .debs.
Note that in ROX-Filer if you click on a .deb then Xarchive is launched and it brings up the contents of the .deb in a window -- however if unable to do so, you get a blank window -- which these new utilities should fix.
No comments - Edit - Delete
Python miscellania
Disciple asked me on the forum why I am "going after Python now". Here is part of my response:I'm thinking about including the "full" Python in Woof, as cutdown as I can get away with, because Python is so heavily used in Linux these days. So many applications require Python, like Inkscape for example. The Rox Filer people have other apps than need Python. If you go to www.gnome-files.org and look around, almost every second app is written in Python and uses PyGTK.
So, I am just bowing to the reality. But, whether I do take the plunge and go for the "full" Python, I still don't know.
Tinypy has nothing to do with the above. It is something that I am playing with in my ongoing search for something to upgrade to from Bash coding. In particular, something that I can myself hack on.
So, tinypy + gtkdialog + low-level-system-hacks is of interest to me.
It could be that I just cannot hack the "full" Python down enough for Woof. Well, we shall see. Python is in the Puppy4 'devx' file, but without extra libraries like PyGTK.
It really is nice to have a printed textbook, so this morning I ordered "Learning Python" from Barnes and Noble (as they accept PayPal):
http://search.barnesandnoble.com/Learning-Python/Mark-Lutz/e/9780596513986/?itm=1
Amazon.com has lots of reviews of that book, it rates well, cheaper too, but they don't accept payPal. My local bookshop had no books on Python, which is why I resorted to on-line. For any other Aussies reading this, the airmail charge was US$12.98 -- so in total it probably cost about the same as I would have paid locally.
10 Comments - Edit - Delete
New names for files in Woof and derivatives
I have implemented this in Woof. File woof-unleashed/DISTRO_SPECS has some overall specifications:DISTRO_VERSION="600"
DISTRO_NAME="Woof"
#prefix for distro files... exs: woof_600.sfs, woof_save.2fs
DISTRO_FILE_PREFIX="woof"
When the distro is built, this file ends up at /etc/DISTRO_SPECS.
If built with a 'zdrv' file, it is named 'woof_zdrv_600.sfs', or whatever the variables specify.
I have modified all the scripts accordingly.
One advantage of this system is that if you wanted to create a derivative of Woof, say 'Bark', change these:
DISTRO_FILE_PREFIX='bark'
All files will be prefixed with "bark", for example 'bark_save.2fs', so they can't get mixed up with any other derivative.
DISTRO_NAME='Bark'
All messages in dialog boxes and elsewhere will display this as the distro name. For example, the main Help page will have "Welcome to Bark Linux".
5 Comments - Edit - Delete
4.1.2: ssb module added to initrd
I mentioned this in an earlier post, but here it is formally announced as a bugfix to be in 4.1.2.The 'createpuppy' script in Unleashed has to have "ssb.ko " added to the 'NEEDEDINITRDMODS' variable, at line 938.
How to manually update Unleashed 4.1.1:
1. Open puppy-unleashed/createpuppy in a text editor.
2. Modify line 938 so that it looks like this:
NEEDEDINITRDMODS=' aufs.ko cdrom.ko fuse.ko ide-cd.ko ide-floppy.ko nls_cp437.ko nls_iso8859-1.ko sqlzma.ko squashfs.ko sr_mod.ko unionfs.ko unlzma.ko aes.ko aes_generic.ko blkcipher.ko crypto_blkcipher.ko cbc.ko cryptoloop.ko rsrc_nonstatic.ko yenta_socket.ko ehci-hcd.ko ohci-hcd.ko uhci-hcd.ko usb-storage.ko usbcore.ko usbhid.ko scsi_wait_scan.ko ssb.ko '
Note that the order of the modules in the list does not matter.
No comments - Edit - Delete
4.1.2: sort applet, init fixed
As I make changes to 4.1.1, for the next bugfix release, I'll put the text "4.1.2:" in the title.Busybox fix
I announced recently that as I needed the 'sort' utility in the initrd, I had upgraded Busybox to 1.12.1. Bad move, some problems, so I have rolled back to 1.4.2 -- Puppy 4.1 - 4.1.1 has Busybox 1.4.1, and 1.4.2 is the last in the 1.4 series.
I compiled 1.4.2 with the same '.config' file as before, except that I enabled the 'usleep' and 'sort' applets, also enabled 64-bit arithmetic for 'test' and 'expr'.
To manually upgrade Unleashed 4.1.1:
1. Grab the new 'busybox' executable from here: http://puppylinux.com/test/4.1.2/
2. Gunzip and place it in puppy-unleashed/boot/initrd-tree0/bin
3. In the same directory create symlinks 'sort' and 'usleep'.
usb-storage probe fix
I recently reported a bug when probing for USB storage drives, testing the 2.6.27.4 Ubuntu kernel. The fix required 'sort -u' to be inserted into the 'init' script.
To manually upgrade Unleashed 4.1.1:
4. Edit puppy-unleashed/boot/initrd-tree0/init, at line 267:
#v412 bug, ubuntu kernel, got duplicate 'device found at 2', need 'sort -u'...
USBSTORAGES=`/bin/dmesg | grep "usb-storage: device found at" | sort -u | wc -l | sed -e 's/ //g'`
For the record:
There is a fatal bug in the Busybox 1.12.1 'modprobe' utility. Puppy4 uses the full 'modprobe', however as Busybox claims to now support gzipped modules and modaliases I enabled modprobe/insmod and removed the "full" modprobe/insmod. However, feeding modaliases to Busybox (modprobe <modalias>), it failed to load the matching module.
Also, the up/down arrow keys no longer worked when running at the shell prompt.
I couldn't be bothered any more, just went back to what works.
3 Comments - Edit - Delete
Tinier tinypy
I have downloaded the latest tinypy from SVN and compiled with dielibc. I posted the 'tinypy' executable here:http://murga-linux.com/puppy/viewtopic.php?t=35446
'tinypy' is now smaller, just 127KB. I examined the source, and very interesting, quite a lot of functions built-in. The core interpreter and some functions are written in C, but many functions are written in Python -- these are converted to bytecode and embedded in the 'tinypy' executable.
A very big chunk of that 127KB is composed of embedded bytecode. It is nice for the programmers to code in Python, and as I understand it the "proper" Python has lots of modules (libraries) written in Python -- however it is bad from the point of view of size and execution speed.
It would be a very interesting exercise to see how small tinypy could really become, if the functions were all written in C!
It is a plus point for me that tinypy is written in C, as I find that generally more readable than C++, and scanning through the code it looks well written and well commented.
Ha ha, my vaporware Woof could also have a specialised Python-lookalike, based on a modified tinypy (or whatever) ....and guess what it will be called ...wait for it... PupPy
2 Comments - Edit - Delete
Tiny-Python
tinypyI reported recently the problem of Wirbel generating huge executables, so I looked around at what other Python compilers/interpreters are out there. Specifically, what I'm looking for is small size. An interpreter is still within my acceptable range, as they compile to bytecode first -- in fact, I think that the "full" Python has an option to generate a bytecode file. Bytecode is kind of half-way between a pure source-code-interpreter and a native-code-compiler.
Anyway, the first one that I have tested is 'tinypy'. It took a lot of messing around, but I managed to compile it statically in the uClibc rootfs -- the interpreter executable is just 131KB! -- that is static, totally standalone, so can reside in the initial ramdisk and will work in any Linux distro and all old versions of Puppy.
Homepage: http://tinypy.org/
I have packaged it up as 'tinypy-1.1-static.pet':
http://murga-linux.com/puppy/viewtopic.php?t=35446
So far I have only tested "hello world" -- like Wirbel, 'print' is a function, so you need brackets:
print("hello world")
The main problem with testing it is the lack of documentation -- I will have to study the source code to find out what functions etc. are available.
pymite
This is another one, but I can't compile it.
Shed-Skin
This is another Python compiler, like Wirbel, that generates intermediate C++ code. I'm going to test that next.
2 Comments - Edit - Delete
Busybox upgraded in initrd
I was testing USB storage detection at bootup, using the 2.6.27.4 kernel (source from Ubuntu), and found something really odd. The 'init' script runs 'dmesg' to see how many USB drives are found, and waits in a loop until the scan is complete. Inside the initrd, I ran 'dmesg' and it gave these lines:<7> usb-storage: device found at 2
<7> usb-storage: device found at 2
<7> usb-storage: device found at 3
<7> usb-storage: device scan complete
<7> usb-storage: device scan complete
The above is interspersed with other lines. The weird thing is the duplicate line, and that causes the wait loop to wait for the timeout, which is 25 seconds, as the loop just compares the number of "device found" lines and waits until there are the same number of "device scan complete" lines.
What the 'init' script does is loop for a full 25 seconds, whereas a couple of seconds would have most likely been enough. I fixed it by putting a 'sort -u' to get rid of the duplicate lines, but the problem was that the initrd did not have a 'sort' application.
That leads me to Busybox. The initrd has Busybox version 1.4.1, however the latest is 1.12.1, so I downloaded and compiled that statically in a uClibc root f.s. I enabled a lot more applets, including 'sort'.
No comments - Edit - Delete
Major bug found in 4.1.x
This bug affects the "normal" Puppy with 2.6.25.16 kernel, not the "retro". It will account for why some people experienced trouble with USB, but it was ok in the retro.I am currently playing with the 2.6.27.4 kernel, and I suddenly noticed that the 'ohci-hcd' module has two dependencies: 'usbcore' and 'ssb'. Er, 'ssb' is not in the initial ramdisk.
I checked 4.1 and 4.1.1, the same situation with the 2.6.25.16 kernel.
What this means is that USB storage that requires the 'ohci-hcd' module will not be detected in the initial ramdisk, which means that you can't boot from USB nor have a 'pup_save' in USB.
Note that the 'ohci-hcd' driver will get loaded later on (after the switch_root) so you can access the USB drives then. Just not at bootup.
The 'uhci-hcd' driver is the most common USB host driver, and that only requires the 'usbcore' driver, so all is well there.
Amazing that such a serious bug managed to get through all those alpha and beta stages of testing!
Hmmm, does this mean I have to bring out a 4.1.2 second bugfix release? Note, if you have 4.1 or 4.1.1 Unleashed, it is easy to fix this bug -- just edit 'createpuppy' script and add "ssb.ko " to the 'NEEDEDINITRDMODS' variable.
17 Comments - Edit - Delete