I think the proposed ban is: don't develop weapons with software that can make the killing decisions without humans in the loop. That itself isn't so impossible a ban to make something like the CWC out of, but nations couldn't even stop 3D gun printing, despite the same sort of worries.

Additionally, even with that ban, you could have the exact same video with the minor difference that each drone sends its video stream to a human somewhere else who presses a button to authorize each suggested-by-the-drone kill. And if we got that far, the pressure to get rid of the ban would be immense, because tool AIs are better as agent AIs.

And if we can't even handle regular old microsystems technology, how are we going to handle nanotech? https://www.amazon.com/Military-Nanotechnology-Applications-... (and briefly covered in this talk by the same author http://www.youtube.com/watch?v=MANPyybo-dA) discusses it but in the 11 years since it was published I see no reason to think the world is any better prepared.

You might be interested in reading this little known book: Military Nanotecnology: Potential Applications and Preventive Arms Control (http://www.amazon.com/Military-Nanotechnology-Applications-P...) I see many parallels with concerns about nanotechnology being written off as scaremongering and little action being done in preemptive measures of control. No one on the danger side of AGI thinks monte carlo tree search will doom humanity, just as no one on the danger side of nanotech thinks that being able to move a single atom up and down in a crystal will turn that atom into a humanity-killing pathogen. But they are steps, and the future dangers are totally ignored.