About

Michael Zucchi

 B.E. (Comp. Sys. Eng.)

Tags

android (44)
beagle (63)
biographical (82)
business (1)
code (56)
cooking (29)
dez (6)
dusk (30)
ffts (3)
forth (3)
free software (4)
games (32)
gloat (2)
globalisation (1)
gnu (4)
graphics (16)
gsoc (4)
hacking (414)
haiku (2)
horticulture (10)
house (23)
hsa (6)
humour (7)
imagez (28)
java (216)
java ee (3)
javafx (48)
jjmpeg (67)
junk (3)
kobo (15)
linux (3)
mediaz (27)
ml (15)
nativez (3)
opencl (117)
os (17)
parallella (97)
pdfz (8)
philosophy (26)
picfx (2)
politics (7)
ps3 (12)
puppybits (17)
rants (134)
readerz (8)
rez (1)
socles (36)
termz (3)
videoz (6)
wanki (3)
workshop (2)
zedzone (13)
Wednesday, 06 May 2015, 21:48

post google code post

Well nobody bothered to comment about the stuff i removed from google code apart from the one lad or lass who lamented the loss of some javafx demos.

I had comments open+moderated for a few weeks but got hit by spammers a couple of days ago so had to go back to id+moderated. Maybe something got lost in those 500 bits of snot but i don't think so. The spam was quite strange; most mentioned web sites but didn't provide links or weren't very readable so i'm not sure what the point was. Perhaps they're just fishing for open sites or naive moderators they can then exploit. Like the "windows computer department" that keeps calling and calling hoping i'll not tell them to fuck off every time (sigh, no i don't normally say that although i would tonight).

I've still got the subversion clones but i'm not inclined to do much with any of it for the forseeable future and i'm not even sure if i'm going to continue publishing other bits of code i play with going forward.

Desktop Java, OpenCL, ARM assembly language; these things are just not very common in the Free Software world. Server Java is pretty common but that's just, well, `open sauce' companies sharing costs and not hobbyists. So i think all i'm really doing is providing hints or solutions for some student's homework or help for graduate programmers to keep their jobs. And even then it's so niche it wouldn't be many, if any.

As an example of niche, I was looking up some way to communicate with adobe photoshop that doesn't involve psd format and one thing i came across was someone linking to one of my projects for some unfinished experiments with openraster format - on the first page of results. This happens rarely but still too often. Of course it could just be the search engine trying to be smart and tuning results to the user, which is a somewhat terrifying possibility (implications beyond these types searches of course). FWIW I came to the conclusion photoshop is just one of those proprietary relics from the past which intentionally refuses to support other formats so it's idiot users can continue to be arse-reamed by its inflated price.

It's just a hobby

As a hobby i have no desire to work on larger projects of my own or other established projects in my spare time. Occasionally i'll send in a patch to a project but if they want a bunch of fucking around then yeah, ... naah. In hindsight i somewhat regret how we did it on evolution but i think i've mentioned that before. Neither do i need to solicit work or build a portfolio or just gain experience.

I'm not sure how many hobbyists are around; anyone with remotely close to enough skill seems to be jumping into the wild casinos of app-stores or services and expecting to make billion$ and not just doing it for the fun of it. Some of those left over just seem to be arrogant egotistical fuckwits (and some would probably think the same of me). Same as it ever was I guess.

I suppose I will continue to code-drop even if it's just out of habit.

For another hobby I made kumquat marmalade on the weekend. Spent a couple of hours in the sun slicing the tiny fruit and extracting seeds (2-3 cups worth of seeds) and cooked it the next day. Unfortunately after all that effort it looks like it wasn't cooked quite enough and it probably wont set - it's a bit runny but at least it tastes good. Not sure what i'll do with 2-odd litres of the stuff though.

Tagged beagle, code, imagez, jjmpeg, mediaz, parallella, pdfz, puppybits, readerz, socles, videoz.
Friday, 13 March 2015, 13:25

ahh shit, google code is being scrapped

Joy. I guess it's not making google enough money.

I'll make a copy of all my projects and then delete them. Probably sooner rather than later (probably immediately because i'd rather get it out of the way, i have a script going now). I'm keeping all the commit history for now although i never find it particularly useful. Links in old posts may break.

Some of them may or may not appear later somewhere else, or they may just sit on my hdd until i forget about them and delete them by accident or otherwise.

Wherever they end end up though it wont be in one of the other commercial services because i'm not interested in doing this again.

I need to do the same to this blog, ... but that can wait.

Update: So ... it seems someone did notice. Don't worry i've got a full backup of everything. In part I was pissed off with google and in part I wanted to make it immediately obvious it was going away to see if anyone actually noticed. Who could know from the years of feedback i've never received.

If you are interested in particular projects then add a comment about them anywhere on this blog. All comments are moderated so they wont appear till i let them through but there is no need to post more than once (i'll delete any nonsense or spam unless it makes you look like a bit of a wanker). I will see about enabling anonymous comments for the moment if they are not already on. I can then decide what to do depending on the interest.

I will not be using github or sourceforge nor the like.

In a follow-up post i'll see where i'm going with them as well as pondering the future location and shape of the blog itself. I've already written some stuff to take the blogger atom stuff, strip out the posts, download images, and fix the urls.

(in resp to Peter below) I knew the writing was on the wall when google code removed binary hosting: it's kinda useless for it's purpose without it. This is why all my new projects subsequent to that date are hosted differently.

Tagged android, beagle, code, dusk, games, gloat, hacking, imagez, jjmpeg, mediaz, pdfz, puppybits, readerz, socles.
Friday, 24 May 2013, 11:50

on google

So google have decided to disable downloads on google code.

So I have decided to stop using it.

... although as yet I have no concrete plans or timeline for when this decision will take effect.

Whilst they claim it's about abuse, one can only assume that is just a "likely-sounding excuse" for what in reality is just another straight-up lie from the PR department of a supra-national conglomerate, and it's really just a way to cut costs and promote their 'drive' service (a useless microsoft/apple only service as far as i'm concerned).

Nobody seems to have reported that they have also gimped their POP interface to gmail a couple of days ago. No more UID support. This makes POP a lot less reliable/useful as a mail store (although in honesty it was never designed for that purpose). I proceeded to delete all the mail in gmail to help them free up some disk space.

I guess over-all the writing is on the wall. We all know that at some point 'google account' will mean 'google+', and blogger may be retired at any time.

So it seems my on-going-but-totally-lax search for alternatives to 'everything google for convenience' just got another big kick up the rump-side.

As my projects are all pretty small and low-volume I might look at a local solution because every network based solution faces the same problem. I have a couple of beagleboards doing nothing although getting a running and secure-enough system might be more pain than it's worth.

It's a bit of a pain to have to deal with.

Tagged beagle, dusk, hacking, imagez, java, javafx, jjmpeg, mediaz, pdfz, puppybits, rants, readerz, socles, videoz.
Monday, 19 April 2010, 14:00

Hello world

Well I integrated a timer interrupt into WoofƆs and got that to work eventually - the MMU makes things like this very fun as I have to map everything I need to use. A few silly bugs in the MMU code didn't help either.

After a 7 hour stint last night I finally booted up an executed a 'user process', including sending it the initial welcome message. It doesn't do much other than call a debug printf function to say hello, but given that it is running in it's own address space successfully that seems a reasonable achievement.

Interrupts are going to be a bit of a pain. Because I want user code to deal with interrupts the kernel has to clear the hardware interrupt status bits itself - which means it needs to know about all the devices that might produce interrupts and how to turn them off. I can't see any way around this right now. I also need some way of letting the user-code know what triggered the interrupt, which is another issue - even with 96 interrupts there is some double-up on the interrupt source. I guess there's always the mailbox idea, or a syscall to query it. There might be some hardware where the interrupt needs processing immediately too, but i'll come to that when I find it.

Hmm, next thing might be to think about devices and get a serial device working.

Tagged os, puppybits.
Saturday, 17 April 2010, 22:01

Timers

Finally got timers `worked out' - i'd played with it before but couldn't get it to work - but it turned out that the code was basically fine, but I had a bug in the irq `enable' routine ... der. They are really rather simple devices that work much the same in almost all microprocessors ever made ..., so it was a bit embarrassing it took this long.

It's using the 32 768Hz clock - which means you have to use some extra hardware logic to get a true 1Khz divisor, but the manual explains that pretty clearly as referenced in the source.

Source is in irq-timer.c.

I'm at the point in WoofƆs that I need at least one timer to get much further so it seemed a good time to poke it into puppybits. I managed to squeeze in a little time to play with WoofƆs over the last few nights and got the code that i'd written a few days ago to load and run - after a stack of mucking about as usual. It doesn't do much - with the trivial round-robin scheduler and no interrupts once the memserver task goes to sleep the idle task starts and is never pre-empted :). I had some linking issues, but the main problem was that for some silly reason I was calling some assembly from C as a function *pointer* not a function and it took me far too long to spot it.

But there's an excuse this time. It's been a really long week - 49 hours of work, quite a few very late nights, an on-site meeting I had to wake up at 7am for, a couple of midnight meetings for GSOC - one of which I missed because I fell asleep, and then I woke up at 2am and didn't crash again till after 8am this morning. Not sure 'tired' covers it. The sunrise wasn't bad though.

Tagged os, puppybits.
Monday, 05 April 2010, 20:39

That's it for summer I guess.

Daylight savings just finished here - and it's already throwing me out. After plugging away at the computer all afternoon I thought it must've been 10 or something as it seemed to have been dark for so long, but it was only 7:30 ... Good time to call it a day I guess.

Yesterday I got a whole lot of work out of the way on the retaining wall, I should've done more today but I was too lazy. The weather was too nice too - although then I just spent most of the day inside instead of taking advantage of it. Once I sit down to hack it's pretty well game over ... Probably should've made the effort though - looks like weather is turning wet soon, and I need to get some of the stuff done whilst it's still try. Maybe an hour or two over the next two days might do it.

Well that's Easter done, back to work tomorrow.

WoofƆs

Was up pretty late hacking away on some WoofƆs stuff, and then continued it for most of today. For the most part it's just re-arranging stuff I already have from puppybits, or my previous x86 hacking. Pretty time consuming though, trying to tie it all together. It also needed quite a bit of re-thinking and re-jigging along the way.

I've got it all building, and it currently launches two tasks, one which immediately goes to sleep waiting for work to do, and the idle task which blinks the led. But that isn't really any more than I already have in puppybits, so I need to start testing the other stuff - per-process virtual memory, message passing, and so on. These things are a bit hard to play with in isolation since so much support is needed first. Hmm, I really need to get timers and timer interrupts working too, but I can test that in isolation.

The architecture which i've thought about so far does seem to be holding together at least - no big surprises have come along ... yet. Although that assumes the stuff i've written actually works too. So far I have a 'memserver' which is tightly bound to the kernel - it is basically a kernel thread since it accesses the kernel memory directly. It is used to create all in-kernel objects - directly adding them to the kernel in some cases, or where necessary through lightweight system calls to register the new resource. This lets me avoid any dynamic memory in the kernel itself, and by using the right data structures or the occasional lightweight system call I don't have to worry about serialisation either (well, once i've got it right).

I had been thinking about all sort of exotic data structures like trees or hash tables to locate resources based on an object id - but these all have various issues. Execution time, serialisation, and so on (I have implementations that use no dynamic memory, so that wasn't an issue). So in the end I settled for a simple array for many of the objects - it probably uses less memory anyway, and certainly needs less code. It also allows me to update and access it atomically from multiple threads of execution without trouble.

The whole 'kernel' is only 10 system calls so far - and although I still need a few more for interrupts, it shouldn't be many ... I think. Of course most of the work is in the servers, and the kernel is just passing data around. Even in those 10 I already have some 'helper' syscalls too - they aren't strictly necessary but combine a couple of system calls into one, there is probably scope for some more of those.

I had a little chicken and egg problem with message allocating - you need to send a message to the memory server to allocate new messages before they can be sent via the kernel. I think I have worked out a solution - part of the process start-up will be to send a system message to the new process. This message can then be used to ask for more, or just as a general purpose message container. Still, I might need some other more direct mechanism since message ownership passes every time they are forwarded, and it is possible for them to get `lost'.

I might have to look into a mailbox mechanism I guess, although I don't want to have too many options for IPC. After I'd written the basic message allocation system I thought it would be a bit bulky to do too often - well the point is not to in the first place, but it seemed a bit heavy. I looked into a simpler mechanism of message passing using limited registered arguments. But it just didn't seem that useful - because you can only pass primitive non-pointer types. As soon as you want to pass buffers around the mechanism falls down, and you need even more complex support code (like 'far' copies), which then needs to perform extra security checks, and so on. One `freebie' of the message passing mechanism as i've envisioned it so far is it sort of self-checks. Nothing the kernel deals with needs checking in detail because any addresses are from trusted sources or have already been verified, or just integer representations of virtual addresses it never looks at. And servers can perform fairly cheap checking since all data must fit in a fixed bound.

Might sit on all that for a while now.

Tagged beagle, os, puppybits.
Sunday, 04 April 2010, 12:02

New project

Finally cleaned up and checked in the kernel start code, and a simple demo which uses it.

I still need to do some prototyping of various low-level components before I can make a whole, but I might kick off the part of the whole I can do already because i'm getting a bit bored of just working on these small fragments.

Keeping with the canine theme, i'm going to call it ...

WoofƆs

Or maybe WʊfƆs, although I think the larger one is more aesthetically pleasing (the letters are based on the international phonetic alphabet if you were wondering).

I'll probably just keep it in the puppybits project, although keep the code separate. With work and home being as they are I haven't had much time nor energy to work on this stuff, so things aren't going to progress at any great rate. Speaking of which I think it's time to hit the yard and get dirty shoveling. Two days of Easter down and all i've managed so far is ~97km of cycling, a hangover, and an improved tan.

Tagged beagle, os, puppybits.
Wednesday, 10 March 2010, 00:10

Syscalls and IPC

The next thing i've been looking at is system calls, and using them to implement IPC.

I wanted system calls to lead to a direct context switch efficiently, so that for example, in the simple case an IPC call to a server process acts synchronously. But on the other hand, there are other tasks for system calls which are really just isolated or system-level function calls, and they don't need the overhead.

So I came up with the idea of two levels - simple system calls which are invoked and return much any other function invocation, and heavier ones which might lead to a context switch. I just then use the system call number to chose what code to execute.

It's actually easier just showing the code I came up with than trying to explain it.

svc_entry:      
        cmp     r7,#SVC_SIMPLE_LAST
        push    { r12, lr }
        adrlo   r12,svc_vectors
        adrlo   lr,svc_exit             @ low vectors, call and return to svc_exit
        ldrlo   pc,[r12, r7, lsl #2]
  ...
svc_exit:
        ldmia   sp!,{r12, pc }^

Well that's the entirety of the code-path for 'simple' system calls. It doesn't strictly need to save r12, but it makes the context-switching case a bit simpler, and keeps the stack aligned properly too. It doesn't need to save the normal EABI work registers, because the target vector will save any it needs to.

At the svc_exit label I initially had `pop { r12, pc }^' thinking that the pop pseudo-op would work the same as ldmia as it does normally, but implement an exception return because of the trailing `^' ... but it doesn't ...

The "we might context switch" case is then much the same as every other exception handler, and re-uses some of the code too. It saves the state to the current TCB, and then invokes the vector. It also handles invalid system call numbers by suspending the task.

        srsdb   #MODE_IRQ
        cps     #MODE_IRQ
        stm     sp,{r0-r14}^
        cps     #MODE_SUPERVISOR

        cmp     r7,#SVC_LAST
        adr     r12,svc_vectors
        adr     lr,ex_context_exit
        ldrlo   pc,[r12, r7, lsl #2]
        b       task_error_handler      @ syscall # invalid - suspend task

One of the simplest useful IPC mechanism I can think of is simply a signal based one (the AmigaOS idea of signals, not the Unix one, although they can be made to work the same). If you're waiting on a signal and it arrives you wake up, otherwise it just gets registered asynchronously and if you wait later, it no-ops.

It only required 2 basic operations, and another 2 for a bit more flexibility.

Signal signals another task with a single signal bit, and Wait puts the current task to sleep until any one of a given set of signals arrives, or does nothing if it already has. Since either may trigger a context switch, these are implemented using the heavier-weight system call mechanism.The other two functions are AllocSignal and FreeSignal whose usage is pretty obvious. These are implemented using non-context switching system calls.The functions are very simple. First Signal, all it does is set the signal bit in the task control block, and checks if the task was waiting for it. If so it sets the return value the function expects, adds the task back to the run queue and reschedules the task, which may then execute depending on the scheduling algorithm.

void svc_Signal(int tid, int sigNum) {
        struct task *tcb = findTask(tid);

        if (!tcb)
                return;

        sigmask_t m = (1<<sigNum);
        sigmask_t sigs;

        tcb->sigReceived |= m;
        if (tcb->state == TS_WAIT) {
                sigs = tcb->sigReceived & tcb->sigWaiting;
                if (sigs) {
                        tcb->tcb.regs[0] = sigs;

                        ... move tcb to run queue ...
                        ... reschedule tasks ...
                }
        }
}

Wait is just as simple. In the simplest case, if any of the signals it is waiting on have already arrived, it just returns the intersection of the two sets. Otherwise, the task is put to sleep by placing it on the wait queue, storing the signals it is waiting on. The `trick' here is that when it wakes up, it will just act like it has returned from a context switch - and start executing immediately after the svc instruction. The Signal code above uses that to return the return-code in the delayed case via register r0 - the standard ABI register for function return values.

sigmask_t svc_Wait(sigmask_t sigMask) {
        struct task *tcb = thistask;
        sigmask_t sigs;

        sigs = tcb->sigReceived & sigMask;
        if (sigs) {
                tcb->sigReceived &= ~sigs;
        } else {
                tcb->sigWaiting = sigMask;
                tcb->state = TS_WAIT;

                ... move current task to wait queue ...
                ... reschedule tasks ...
        }

        return sigs;
}

Invoking system calls uses the svc instruction. This encodes the system-call number in the instruction, but you can also pass it in a register - and I follow the linux EABI which puts the system call number in r7.

With this mecanism it is trivial to implement a system-call in assembly language - tell the compiler what the args are, but the assembly just passes them to the system.

C code:

extern int Wait(int sigmask);

ASM:

Wait:   push { r7 }
        mov  r7,#3
        svc  #3
        pop  { r7 }
        bx   lr

But this wont let the compiler in-line the calls, which would be desirable in many cases. So one must resort to inline asm in the C source.

sigmask_t Wait(sigmask_t sigMask) {
        register int res asm ("r0");
        register int sigm asm("r0") = sigMask;

        asm volatile("mov r7,#3; svc #3"
                     : "=r" (res)
                     : "r" (sigm)
                     : "r7", "r1", "r2", "r3", "r12");

        return res;
}

(I'm not sure if that is 100% correct - but it seems to compile correctly.)

Code in ipc-signal.c and another version of context.S.

Other IPC

Well beyond simple signals there are a few other possibilities:

I'm fond of the async message passing model, particularly with non-copying of messages - I used it in Evolution and other threaded code i've written since. So i'd like to try and get that working efficiently in a protected/virtual memory environment if I can - I had it working before on x86. I don't know if it worked terribly efficiently, but I don't see why it shouldn't be too bad.

Mailboxes also look interesting - because of their simplicity mostly. They can convey more information than a single bit as with signals, but i'm not sure they could replace them (the CELL BE hardware implements mailboxes plus a bit-based system much the same as signals - and if they spent the hardware to do them both there's probably a good reason).

Once I get some mechanism to pass data between tasks I guess i'll look at something a bit more meaty, like a device driver - and go back to banging my head against that gigantic OMAP manual.

Tagged beagle, hacking, puppybits.
Older Posts
Copyright (C) 2018 Michael Zucchi, All Rights Reserved.Powered by gcc & me!