The most obvious storage type for observers (stored in attach()) is array. In order to be able to detach attached objects, you should be able to identify it inside observer storage.
I suggest you attach observers this way:
$this->_observers[spl_object_hash($observer)] = $observer;
so you can detach it later if you need:
unset($this->_observers[spl_object_hash($observer)]);